WMOD00464 - Laserfiche WebLink

Home Browse Search

A-10 down"! If just this happens, ought we then be encouraged, and go on to confirmatory experi- ment checking each of the 25 that seemed to go up? Only if we have nothing better to do. (Sometimes, indeed, it may be that we have nothing better to do.) What has happened is exactly what pure chance should give us on average; we ought to evaluate it accordingly. One counterargument is likely to be "but I didn't ask all 1000 questions", to which the automatic cross-examination should be "Is there anyone of them you wouldn't have asked if it appeared that its answer was likely to be favorable?" In confirmatory phases, we cannot live with 1000 questions, or 100, or, usually, even 10 or 5. There is trouble enough with 2 or 3. We MUST strive to focus our confirmatory phases as nearly on a single main question as we can. (Once the main question has been "carved in tablets of stone" we may be able to be seri- ous about a few further auxilliary questions.) In confirmatory phases, there is no substitute for the sharpest possible focusing. In an exploratory phase, the situation is different, but usually not as different as we might like. (We have discussed some of these issues earlier under ,,* the transition to confirmation *".) * problems of analysis * While we do have suggestions to make about future analyses of weather modification experiments, these are more appropriately discussed in Chapter D below. All that we need say here is that most serious questions of analysis raised by recent experiments are: 1) questions of multiplicity -- how many questions were asked (actually or potentially). 2) questions of sensitivity -- would other analyses have been expected to be much more sen- sitive, and if so, which? To the extent these arise in. connection with individual experiments, it seems best to first take them up in that context. * confidence intervals * We urge upon all concerned with the design or analysis of weather modification experi- ments the importance of answering "By how much?" and not just trying to answer "Up or down?". Techniques which are now available and have already been used in analyzing major experiments, specifically re-randomization, can be relatively easily extended to provide confidence intervals (and not just tests of significance). A summary of a weather modification experiment consisting only of an apparent effect and a "P-value" should now be considered inadequate, as compared to a statement of an interval of amounts of change corresponding to a chosen degree of confidence associated with that interval (such as 95%). 6. Kinds of Grounds for Criticism In criticizing experiments, including their analysis, it is important to recognize different levels of criticism -- something that has not always been done. We need to distinguish carefully at least the following levels of criticism: a) the experiment is clearly invalid either because of identifiable biases or because what was measured was not what it was said to be (or the analysis may be clearly invalid because it does not correctly reflect the actual design).