WMOD00478

• Title: Field Experimentation in Weather Modification
• Date: 3/1/1979
• Weather Modification - Doc Type: Report

~ __..A;;'i ~f

Cook and Holschuh: Field Experimentation in Weather Modification

69

of R.A. Fisher (1926, p. 504) offer some guidance:

A scientific fact should be regarded as experimentally established
, only if a properly designed experiment rarely faiu to give [a
0.05] level of significance.

Admittedly, some portions of this statement are open to
interpretation. Nevertheless, one important implication
is that a single experiment can never serve to completely
establish a scientific fact. No single experiment, no matter
how well conceived and implemented, should ever be
regarded as definitive. As pointed out by Professor
Braham, replication of weather modification experiments
is likely to be difficult. Cost and the notorious variability
of the weather are two obvious complications. Moreover,
replication of the results of future weather modification
experiments is likely to be most difficult for the same
reason that analyses of past experiments have proven
difficult-treatment-unit nonadditivity.
Treatment-unit nonadditivity is present when the
true response to a treatment varies over subsets of the
experimental units. The most extreme form is when the
direction of the response may vary from unit to unit.
Evidently, this is the case in White top. The overall effect
of seeding in Whitetop was a decrease in both rain a.nd
radar echo cover, while qualitatively different results
were obtained when the experimental units (days) were
. partitioned on maximum heights of clouds, number of
burner hours, and mean wind speed. Might the same
phenomenon be observed on other relevant but as yet
unknown partitions of the experimental"units? Treat-
ment-unit nonadditivity is not isolated to Whitetop but
is, in fact, a recurring notion throughout Professor
Braham's article. .
We suspect that treatment-unit nonadditi vity is
probably present to one degree or another in many past
experiments. For example, there is some evidence in the
data as presented by Woodley et 0.1. (1977) to suggest that
in Project FACE the treatment response varies with the
seeding suitability criteria. Until the ability to predict,
. a priori, additive subsets of the experimental units is
developed (assuming this is possible), interpretation and
comparison of the results of weather modification experi-
ments will end in moot conclusions. At a minimum, it is
necessary to develop the ability to predict subsets in
which the individual units all respond in the same direc-
tion. Exploratory data analysis tempered with est.ablished
meteorological physics is one way to approach this
problem. However, any conclusions so obtained must be
subject to independent verification, as pointed out in
Fisher's comment.
We recently completed (Cook and Holschuh 1978) a
report for the Science and Technology Subcommittee of
. the Minnesota Legislature on the feasibility of conducting
confirmatory evaluations of commercial cloud seeding
projects in Minnesota. The responses to our report were
strong and clearly indicate that some meteorologists feel
they presently have the ability to predict additive sub-
sets. Whether or not this opinion is widespread, we do not
know.

The existence of treatment-unit nonadditivity also
prompts us to reconsider the appropriateness of relying
on an overall effect as a measure of probable treatment
response. Suppose, {or example, it is known that seeding
produces, on the average, a positive response. Would this
knowledge be sufficient to justi{y the implementation o{
seeding technology over a large region where it is simul-
taneously suspected that, because of treatment-unit non-
additivity, seeding is likely to produce negative effects
in some (perhaps small) subregion? Should the farmer
who experiences a decrease in rainfall due to seeding be
content with the knowledge that most others have
experienced increases?
A second important implication of Fisher's comment
which warrants special emphasis is that experiments
which are to contribute to the establishment of a scientific
fact must be properly designed. In our view, randomiza-
tion is essential to such experiments. Randomization a.nd
the ll.Ssociated randomization distribution eliminate com-
pletely the need to rely on a presumption about the dis-
tribution offered by nature (at least in the case of addi-
tive effects). Perhaps equally important is the fact that
randomization allows the elimination of any other factor
that might have caused an observed association. For
example, one problem in analyzing the results of a com-
mereial project is the possibility that the operator may
have inadvertently chosen naturally rainy days for
seeding. Randomization dispenses with such concerns.
Analyses of nonrandomized projects and exploratory.
analyses of randomized experiments are open also to the
problems of multiplicity and creditability. This does not
mean that such inquiries should be ignored. Certainly
they may provide important information for the formula-
tion of future hypotheses. However, until better .models
for the physical processes involved in weather modifica-
tion are proven, the results of such inquiries should not
be counted as confirmatory.
Inferences based on randomization are occasionally
complicated when there seems to be an association be-
tween the actual randomization and a presumably rele-
vant covariate that was not available a priori. The results
of Whitetop pose such an inference problem. Before dis-
cussing the randomization controversy surrounding
Whitetop, it is worthwhile to briefly consider the issue
of 8, "bad" randomization. The- validity of significance
tests derived from the randomization distribution is
based on repeated sampling: under the assumption o{ no
treatment effects and regardless of constraints on the
randomization, 5 percent, say (assuming this is an
achievable level), of the possible randomizations will, by
definition, yield p-values not greater than 0.05. A ran-
domization may be termed "bad" if it is suspected that,
in the absence of treatment effects, the associated p-value
would not be greater than 0.05. This suspicion arises
us\ul.lly either from an observed association between a
cova.riate and the particular randomization, or {rom the
empirical notion that overly patterned randomizations
tend to confound influential environmental factors with