WMOD00464 - Laserfiche WebLink

Home Browse Search

A-2 purpos~s tq be met by our standards remain the same, but the means for meeting these stan- dards may vary somewhat -. but not seriously -- from those appropriate for pure experimenta- tion, as we shall notice from time to time. 2. Quality Data * blindness of judgment * The facts that (j) not every day, not even every rainy day, is a likely candidate for seed- ing, and that (ij) the judgment of day quality still seems to have a large component of subjec- tive judgment (as opposed to checking whether or not agreed-upon conditions are met) leave us in an unusually difficult situation. Given that the judgment wit! be made in total honesty, we know enough about less-than-conscious processes to be deeply concerned about possible differences in judgment as to the suitability of a day (or situation) between instances when the judgment-maker knows that this day, if judged suitable, will be seeded and those instances when he knows that it will not be seeded. Furthermore, the principle that "justice must be seen to be done" demands great care in avoiding any semblance that subjective judgments might have contributed appreciably to the observed results (even small biases can be serious, since it now seems reasonable to expect only small to moderate effects of seeding). This is not an easy problem to attack. We cannot expect, yet, to go all the way toward objectivity of decision. We cannot yet be explicit enough about the important aspects of a situation. We must depend to a degree on expert judgment. But we must plan, carefully and vigorously, both (a) to use objective, auditable criteria as far as possible, and (b) to keep from anyone making the judgments the knowledge of whether this day (or situation), if declared "suitable", will indeed be seeded. And we also must do this in a way that does not give a false impression of distrust of individuals. Without experts performing at their highest skill, our experiments are likely to be doomed to misleading failure. To be careful about subjective aspects of decision should be viewed as reflecting deep respect for the proper role of the experts -- as allowing them to perform at their highest skill, using all the available weather data, but undlstraded by a knowledge of whether seeding is to happen or not if they make the decision of suitability. These situations where decisions of suitability are made in process, as for example by air- borne scientists, raise the problem of blindness in its most difficult form. We regret to say that any confirmatory study whose results are to be taken seriously MUST confront this issue. Admittedly such judgments may enhance performance fairly as well as unfairly. Admittedly objective criteria for in-process changes of suitability may not perform as well as expert's judg- ment. The use of the objective criteria, however, is part of the price that must be paid if sup- posedly confirmatory studies are to be taken seriously. Like the other aspects of data quality to which we turn next, maintaining adequate blind- ness is uncomfortable but very important. * oversight and auditing * Equal care is needed in the collection and handling of data. The historically common pat- tern of "collect the data now, organize and scan it next winter (or after the last year of the experiment)" cannot be commended and must be judged quite unsafe. Data should be scanned for plausibility and completeness almost at once. At the same time it can and should be entered into whatever archive or data management system is to be used. Gaps and implausible values .should be the subject of immediate inquiry into causes -- and into steps that could be taken to