WMOD00478

• Title: Field Experimentation in Weather Modification
• Date: 3/1/1979
• Weather Modification - Doc Type: Report

~8

plied" versus "theoretical" research, how to usefully
resolve different viewpoints on the same experiment, how
to best utilize two measurement systems on the same
response dimension, how to properly investigate inad-
vertent weather modification situations, how to promote
productive collaboration between statisticians and mete-
orologists), but time, space, and format fortunately im-
pose constraints. Hence, I shall focus my comments on
. the following three issues: (1) the roles of exploratory and
confirmatory experiments; (2) the design, analysis, and
interpretation of "nonrandomized projects," and (3) the
enlargement of the productive intersection of statistics
and meteorology.

.

2. EXPLORATORY VERSUS CONFIRMATORY
EXPERIMENTS

The physical and biological sciences (and, I trust, all
science) utilize the discovery (exploration) and verifica-
tion (confirmation) dichotomy in admitting new results
to the accepted body of knowledge. Thus, one often sees
statements such as "first discovered by A and subse-
quently verified by B." Hence, it is not unusual to find
that some experiments are primarily exploratory, some
are confirmatory, and some are both. In weather modifica-
tion research, the Skyline Conference Report (NAS-NRC
1959) appears to he one of the first publications to em-
phasize this exploratory-confirmatory dichotomy; more
recently, there have been attempts to fully describe and
classify each of these types of experiments (e.g., Flueck
1977; Flueck and Mielke 1978).
An exploratory (E) experiment can be defined as "an
attempt at publicly 'staking a claim' on a planned or
unplanned set of relations among events based on a
plauSible (though often crude) conceptual model and
appropriate scientific evidence" (Flueck 1977). It is per-
mitt~d, in fact encouraged, to search (in an orderly man-
ner) for the treatment effects; and consequently, changes
in the design and implementation of an E experiment are
allowable. Of course, all such searching and changes
should be carefully recorded and publicly reported along
with at least the experiment's response data set (pre-
ferably by experimental unit).
Alternately, a confirmatory (C) experiment has been
defined as "an attempt at securing an 'independent' con-
firmation of the results that already have been discovered
and reported by an.E experiment" (Flueck 1977). Thus a
C experiment should be more tightly focused, must
clearly and unequivocally state what is to be confirmed
and how, may utilize a smaller sample size than the cor-
responding E experiment, and should not allow changes
in design, implementation, or analyses.
Lastly, many experiments are E-C experiments; part
of the experiment attempts to confirm a relation, and
part explores new ones. In fact, "both the economics of
experimentation and the desire to uncover the 'full
picture' often dictate E-C experiments" (Flueck 1977).
However, care must be exercised to clearly designate and
separate that part which is C and that which is E.

'~"-WL';Z.c.;.:~JW.~.:/,~_..__.j,~,;;;.~

;"',.;1:;, ~.

".",~-~

,'" ~'=-l..:..:"YA.'i.::<'-"Jli:-,..:....,.idd;"-~'_'_'__"'-'--""~'_,_5''-'_e~ _

Journal ,of the American Statistical Association, March 1979

In view of these descriptions, Whitetop clearly was an -
exploratory experiment in the modification of summer
convective clouds. As Braham has demonstrated, it did
"stake a claim" (i.e., the indicated overall treatment
effect was a decrease in estimated precipitation with
addiHonal indication of increases in some situations and
decreases in others); it proposed a crude conceptual
model for the results (summarized in Braham's four con-'
clusions in Section 3.1); the changes in design .and im-
plementation were reported (Braham 1966 and Flueck
1971) ; all of the attempted and completed analyses were
listed (Flueck 1971) ; and the rainfall response data were
publiished by experimental unit (Decker and Schickedanz
1966).
It'is not unusual-in fact it should be anticipated-
that important exploratory experiments in any discipline
are l~ccorded considerable professional discussion, and
Whitetop (as Braham points out) is no exception. How-
ever, some of the resulting discussion of Whitetop and its
results seems to have become "personalized" and "pro-
tected" (e.g., Neyman 1977), and in this regard, Whitetop
is an undesired exception. Clearly the generation of new
knowledge is a competitive business, but fair and open
competition should be the accepted format.
Perhaps some of the present differences of viewpoint
on the Whitetop results could be reconciled through a
joint venture of extensive reanalysis, but full convergence
of views will probably have to await a confirmatory
experiment. In this regard, the ethical issue raised by
Braham of repeating a field experiment that strongly sug-
gests a 50cietalliability (e.g., less rainfall) is troublesome,
but perhaps not insurmountable. For example, an experi-
ment might be designed which would attempt to produce
decreases in rainfall from the subset of clouds that were
most suggestive of decreases in Whitetop, and increases
from the subset most suggestive of increases. The overall
rainfl!l.ll results possibly could be controlled by the pro-
portions of the two subsets admitted for experimentation.
In attempting to confirm a conceptual model of treat-
ment-produced rainfall, it is important to investigate both
sides of the predicted changes.
Two other cloud seeding experiments mentioned by
Braham. appear to qualify as proper exploratory experi-
ments; Climax I and FACE I. Each experiment seems to
have fulfilled the aformentioned criteria. In addition,
each produced strong indications of a beneficial treatment
effect (though this is not a requirement for a successful
exploratory experiment), and each has been important
enoul~h to attract attention and discussion from both
meteorologists and statisticians. Perhaps the amount of
serious professional attention accorded an ~xploratory
expeliment is one measure of its "success" ; however, such
attention should not interfere with the need to confirm
its results.

3. DESIGN AND ANALYSIS OF
"NONRANDOMIZED PROJECTS"

In Section 5, Professor Braham raises at least two
important issues under the heading of "Nonrandomized