IntermtnClimateSummaryJan2008 - Laserfiche WebLink

Home Browse Search

INTERMOUNTAIN WEST CLIMATE SUMMARY, JANUARY 2008 Observation Yes hits Yes Forecast No Total misses No false alarms correct negatives Total ~tQi[7~{~J&i )~/@~ ~ 19{t~~ CQ{Q)@i~rr\~/:~ [F\J@ Figure 1 b. A contingency table shows what types of errors are being made. A perfect forecasting system would only produce hits and correct negatives. person verifying the forecast uses expert judgment to decide what makes a forecast accurate. For example, a forecast for a high temperature of 750F might be considered inaccurate either when the observed high temperature was 760F or when the high temperature was 850F. The second common forecast verification measure is skill. Skill is the accuracy of a forecast over a reference forecast. The reference forecast might be random chance, persistence forecasts, climatology, or even another forecast. A random chance forecast would be like flipping a coin to decide whether or not to forecast precipitation. Persistence forecast is forecasting the same condi- tions that are happening at the time of the forecast. For example, if it is currently snowing, a persistence forecast is for snow to continue. A forecast of climatology is forecasting the average conditions for the forecast period. A "skillful" forecast must show improvement over a reference forecast. Other measures of forecast quality besides accuracy and skill include bias, resolution, and sharpness. Bias measures if forecasts on average are too high or too low relative to the truth. Resolution measures the ability of a series of forecasts to discrim- inate between distinct types of events, even if the forecast itself is wrong. Sharpness indicates if the forecasts can predict extreme values. Sharpness is important because forecasters can some- times achieve high skill scores by predicting average conditions but in some cases the occurrence of extreme events may be more important to users. In general, focusing on just one measure of forecast quality may be misleading. For example, in the case of Findley's forecasts, their apparent high accuracy obscured the fact their skill was less than a constant forecast of no tornado. Methods of Forecast Verification Forecast verification methods are chosen depending on the type of verification (accuracy or skill) and the type of forecast (dichotomous, continuous, probabilistic, etc.). Examples of verification methods range from simply "eyeballing" the fore- cast compared to observations, to statistically and numerically advanced methods. Eyeballing a forecast is as simple as it sounds and can be use for a variety of forecasts. A forecaster simply looks at the fore- cast and the observations side by side to see how well they match up (Figure la). "Eyeballing" verification is very subjective and can lead to different outcomes depending on the judgment of the indi vidual forecasters looking at the data. A contingency table is typically used to verify dichotomous forecasts, like the tornado example above, over a period of time. The table shows the "yes" and "no" forecasts and observations (Figure 1 b). To find the accuracy of the forecasts, one must sum "hits" and "correct negatives" and divide by the "Total". This will give a number between 0 and 1; the closer to 1, the more accurate the forecast. This type of score can be very misleading in rare events when forecasting "No" will lead to a high "correct negatives" category such as the occurrence of tornados as in the Findley Affair. Numbers in the contingency table can be com- bined in many other ways than just accuracy. For example, the False Alarm Ratio is the number of events that were forecasted to occur but did not. One can numerically verify or calculate the error between the forecast and the observed values with the help of graphical repre- sentations. Graphical displays, such as scatter or box-and-\vhis- ker plots, are used to verify forecasts of continuous variables such as maximum temperature over a period of days. Scatter plots show the observed amount plotted against the forecast amount. An accurate forecast in this case would lie along the diagonal of the scatter plot. Box-and-whisker plots can show the distribution of the observed values relative to the forecasted values, which can provide a measure of the resolution of the forecast. In a well- resolved forecast, the box plot of the forecast would appear to have the same spread as the observed values. Skill scores can be calculated for almost all types of forecasts, but they are most often used for categorical and probabilistic forecasts, like the seasonal climate outlooks issued by NOAA's Climate Prediction Center (CPC) (see pages 13 and 14). All skill scores measure the fraction of correct forecasts to total forecasts FEATURE ARTICLE I 3 i~