WMOD00478

• Title: Field Experimentation in Weather Modification
• Date: 3/1/1979
• Weather Modification - Doc Type: Report

86

As yet we have no general way to think clearly and
sharply about these matters. Multiple-comparison
methods may provide a Band-Aid. but not a fundamental
remedy.
An excellent brief treatment of the multiplicity problem
is given by John W. Tukey (1977) in the setting of clinical
trials, a setting with similarities to that of weather modifi-
cation. Others have written about aspects of it, and there
are discussions of it, with bibliography, in at least four
articles of the Internatio1!-al Encyclopedia of the Social
Sciences under such rubrics as data-dredging andhy-
potheses suggested by the data (Lehmann 1968, p. 42;
Selvin 1968, p. 418; Kruskal 1968a, p. 212; and Kruskal
1968b, pp. 247-~8. These articles, with extensive revi-
sions and postscripts, appear in the I nternatwnal En-
cyclopedia of Statistics, 1978).

4. MULTIPLICITY AND CONCERNS ABOUT
RANDOMIZATION

The problems of mUltiplicity arise in many ways, and
one of those ways is especially relevant to the present
discussion: the expression of concerns about the ran-
domization procedure used in an experiment. To give
concrete circumstances, suppose that an experiment con-
sists of paired observations, say a control plant variety
and a new plant variety, both grown in pots that are
paired as to soil, position in greenhouse, etc. Which pot
of a pair gets the control and which the new variety is
decided by the flip of a fair coin, with independence of
the flips over pairs.
The primary observations are on, let us say, the dry
weight of the harvested plants. Yet there are many pos-
sible contextual observations of quantities that may
affect growth: the two pots of a pair will generally differ
in humidity of soil; amounts of iron, zinc, etc. in the soil;
friability of soil; micro-climate: temperature, humidity,
breezes, etc.; amount of soil, etc. An agronomist might,
I suppose, easily write down 25 or 30 such variates. If
there are k such variates, if they are stochastically in-
dependent, and if we do a suitable paired two-sample
test of the null hypothesis of no difference between con-
trol and new variety for each, at, say, the .05 level, then
the null probability of finding at least one statistically
significant difference is 1 - (.95)1:. As we all know, that
is only .05 for k = 1, but it grows fairly rapidly, and even
if k is only 12, the probability of at least one suchmis-
leading result is about one-half.
Of course in practice the variates will not be inde-
pendent, but the moral is unchanged. Only the details

? I
. _",.h ,.~_." J,_"...<

, ,,:4i,-,:-..._"

~,"i',--~"}.;...:.'

.''',~ "-'" _ -J., ~

'-;.'

Journal of the American Statistical Association, March 1979

wouldl be different-if we knew how to do the calculations.
In general we do not know that, although usable bounds
may <<lxist in favorable situations.
Consider a critic of an experiment who expresses
doubts about the validity of its randomization procedure
by finding a statistically significant effect in the con-
textw~l variates, an effect that cannot come from the
treatment. Such a critic is under a special obligation to
layout all the effects examined, and to point out as
quantitatively as possible the extent to which multiplicity
degrades P-values or other properties based on single-
analysis models. .
The problem is difficult to think about because un-
doubtedly some putative randomization procedures really
are faulty, either by malice or inadvertently. We need not
dwell on the marketing firm with a financial interest in
how results turn out, or on the statistically innocent
medi(:al assistant who wants the doctor's treatment to do
well, or on the experimenter himself who quite uncon-
SCiOWlly peeks at the randomly generated numbers or
ex post facto and tendentiously discards as aberrant some
experimental units; these are all prototypes that surely
have existed. In this post-Freudian era, it is not shocking
to note that sincere assertion of honor and honesty may
coexi.ut with selective slips of the hand or tongue.
And yet there are thoroughly honest randomizations
as well, although no randomization can stand up in-
definitely to attack when a moderate to large number of
partly independent contextual characteristics (together
with functions of these characteristics) are examined.
In this arena abound unsolved, even unstated, prob-
lems of scientific philosophy and public policy. The latter
arise with special force when the randomized experiments
are very expensive or are likely to have relatively im-
medilLte effects on national policy.

REFERENCES

Bunker, J.P., Hinkley, D., and McDermott, W.V. (1978), "Surgical
Innovation and Its Evaluation," Science, 200, 937-941.
Krusk:ilJ, William H. (1968110), "Statistics I. The Ji'ield," in Inter-
nati<mal EncycJqpedia of the Social Sciencea, Volume 15, ed. David
L. Sills, New York: Macmillan, 206-224.
-- (1968b), "Significance, Tests of," in International Encyclo-
pedia of the Social Sciencu, Volume 14, ed. David L. Sills, New
York: Macmillan, 238-250.
Lehmnnn, E.L. (1968), "Hypothesis Testing," in International
Ent<rJcJqpedia of the Social Sciencu, Volume 7, ed. David L. Sills,
New York: Macmillan, 40-47.
Belvin, Hanan C. (1968), "Survey Analysis 1. 'Methods of Survey
Analysis," in International EncycJqpedia of the Social Sciencu,
Volume 15, ed. David L. Sills, New York: Macmillan, 411-419.
Tukey', John W. (1977'), "Some Thoughts on Clinical Trials, Espe-
cially Problems of Multiplicity," Science, 198,679-684.

I
j