WMOD00464

• Title: The Management of Weather Resources - Volume II
• Prepared For: The Weather Modification Advisory Board
• Date: 6/30/1978
• Weather Modification - Doc Type: Report

A-ll

b) the experiment may well be invalid because potential biases were not checked, most com-
monly less-than-conscious biases of judgment (or data collection) by those not blind to
the treatment. (Some problems with control areas may fall here,)

c) the design or, more likely, the analysis was not as sensitive as it should have be~m" thus
wasting a part of the information so laboriously gathered.

, d) doing either the experiment or the analysis in 31 certain way has caused difficulties that
might have been avoided.

e) the conduct of the experiment or the analysis deviates or the analysis deviates from some
critic's standard of reasonableness to the extent that he or she "would never have done it
that way".

Confusion of levels here can give an utterly false impression -- an impression that there is, was,
or ever will be, a single right way to conduct and analyse an experiment -- and that any devia-
tion from this pattl~rn is Just absolutely wrong. In truth, the design and analysis of an experi-
ment involves many compromises, about which honest and well-informed persons win differ,
and whose relative merit will inevitably be differently judged years later, in the presenc~: of new
knowledge.

Criticisms of clear invalidity deserve the utmost attention, since invalidity could destroy
the entire value of the experiment.

Criticisms of potential invalidity need to be given much more attention than they have
received. Here, too, the entire value of the experiment may be lost.

Criticisms of insensitivity deserve careful attention. If there are insensitivities in the
design, they are likely to be irretrievable and to be regretted. If there are insensitivities in the
analysis, finding thlem may well stimulate a more careful reanalysis.

Criticisms of avoidable difficulty, below the levels of invalidity or insensitivity, deserve
attention, especially for the future. In considering them, however, we must be prepared to
recognize the need for compromise and the possibility that they are the result of making the
best available choke.

Criticisms of the form "no reasonable meteorologist (or no reasonable statistician) would
have done that" should be matters for professional discussion. Unless general consensus is
attainable, they ,probably should not be allowed to confuse the understanding or evaluation of
the experiment's results by a broader public.

Instances where these levels have been adequately distinguished are not us easy to find as
they should be.

We shall consider below certain statistical issues which lead to criticisms of still another
form:

f) the investigaltors are not justified in being as certain as they claim.

If we concentrate on what is appropriately concluded from a given experiment, such issues are
indeed issues of validity and deserve the corresponding utmost attention. Their consequences
are, however, not the complete destruction of the value of the experiment but only the weak-
ening of its conclusions. 'They cannot be neglected, but they ordinarily identify how thl~ prob-
lem they point to c;an be dealt with.