WMOD00478 - Laserfiche WebLink

Home Browse Search

82' , tions about exploratory findings is that of multiplicity. The chance that, in multiple analyses of the same data, some "effect" will be found to be "one-percent signifi- cant" increases rapidly with the number of analyses per- fonned, as well as with the number of other, a priori reasonable, analyses which were not carried out be- cause the data were counter-indicative. Running a large number of analyses and treating them as a priori single tests could well be described as artificial stimula- tion of significance. A P-value of one percent for some obs~rved pattern has quite a different import from a one- percent significant result on the test of the principal hypothesis of the experiment. And yet, it has some weight, if only to lead one to reanalyze other experi- ments for repetitions of the same pattern, or to design new experiments for that purpose. As regards methodology, one must be aware that the multiplicity of explorat.ory analyses, mostly unforeseen at the design stage, does not fit into the Neyman-Pearson two-decision framework. Nor is it possible to "optimize" all these analyses simultaneously, or even to ascertain all the distributions involved. It is therefore necessary to use robust techniques which are more generally valid and give rough indications of the extent of statistical support. Nonparametric tests, especially permutation tests which do not rely on distributional assumptions or independence of observations, are suitable here, as is jackknifing. Braham comments that meteorolgists tend to evaluate findings in terms of the physical sense they seem to make, rather than merely in terms of the statistical support they have. :Meteorologists are completely right in doing so. If an exploratory analysis reveals one pattern at P = '0.10 that makes good physical sense and another pattern at P ,; 0.05 that does not fit in with anything the meteorologist knows, he will obviously, and justifi- ably, concentrate on the first pattern rather than on the second. A single experiment's P-value is only one element of all the evidence the meteorologist must muster in evaluating a finding. To return to Braham's question, the extent to which a statistician should be involved in the meteorology of a cloud seeding experiment depends on the stage of experi- mentation and analysis. The design stage must be a joint venture in which the meteorologist determines the goals of the experiment and proposes the inain variables and hypotheses to be studied. The statistician can then bring the methods of mathematical statistics to bear on efficient experimental design. The more the statistician under- stands the cloud physics involved and the characteristics of meteorological data to be studied, the better he can serve the meteorologist's purpose. If the statistician does not understand the meteorological rationale of the experi- ment, or if he is ignorant of the special features of the variables considered, his design is likely to remain a largely irrelevant txercise in mathematical statistics. An example is the derivation of optimal parametric tests under assumptions of unit-to-unit independence-even , . :t~._r~~J"r~-t~~~;.)~'o.-';,;d4:-'-,,_.~~r_~i~._,i_&_'~;;;',,;_ t",,~,~_._~ Journal of the American Statistical Association, March 1979 though it is well-known that there is considerable serial correIa.tion in precipitation. After the design (1) comes the execution (2), followed by calculation of the preordained test and estimate (3). In theory, all this is determined at the design stage, and the experimenter may follow an unequivocal protocol through stage (2) to the conclusion of his experiment. In practice, things do not work quite that way-measure- ments often cannot be taken as planned, treatment methods may vary, some definitions are likely to be found impractical, some units must be changed, and test statistiics can almost always be improved upon. Who is to decide whether these changes are permissible within the predetermined design? To illustrate from the Israeli .rainfall stimulation ex- periment: The 8 P.M. to 8 P.M. daily unit initially used required continuous-time recording gages. When these proved difficult to obtain, was it permissible to substitute ordinary gages and thereafter continue the experiment on an 8 A.M. to 8 A.M. basis (Gabriel and Neumann 1978)? When the seeding plane pilots were found to prefer flying within sight of the coa.stline rather than farther out, as originally planned, was it appropriate to shift the definition of the target accordingly? When a suitable concomitant variable was found, was it permissible to redesi~:n the significance test to take that variable into account? When local observations, as well as publica- tions from abroad, suggested that simple rank analyses would not efficiently take into account occasional large seeding; effects, was it permissible to change from the proposed Wilcoxon-Mann- Whitney test to one which now looked more powerful? When the . pilots went on strike, could the lost days be omitted from the analyses, and how was the randomization to continue after they returned to work-as allocated for that date or as al- located for the first day they were on strike? This role of the statistician is not foreseen in the texts. He has to serve as an interpreter and arbiter on the design and on changes made during experimentation. Ap- parently his understanding of the function and rationale of randomization is also required in supervising that this is actually carried out correctly. For similar reasons, the statistician may also be asked to supervise the collation of datl~ and calculation of the teEt statistic. This quality control on the statistical aspects of ex- perimentationis a function which statisticians cannot carry lOut if they are too deeply involved in an experi- ment. If they are on the experimental team, or, worse yet, employed by the meteorologists, their objectivity may be suspect. Indeed, their ability to insist on exact adherence to protocol requires not only independence but also suitablle status-it may be difficult for a fresh statistics Ph.D. to insist that an eminent cloud physicist stick to some seemingly trivial but possibly crucial point in a randomization procedure. And indeed the authority and presti~:e of the statisticians involved may well be adduced as evidence that the procedures were properly carried,