What is Forecast Verification?
Weather models are verified to monitor and improve the quality of their forecasts and to compare the quality of different forecast systems. To verify a forecast, it must be compared to a 'truth' such as a corresponding observation set or other estimate of the true weather like a forecast analysis, a very short-range forecast or the previous day’s weather conditions (known as the persistence method).
Verification of forecasts is important for administrative, scientific and economic reasons, and any verification scheme should add value by being informative (Joliffe and Stephenson, 2012). From an administrative viewpoint, measuring the performance of forecasts allows decision makers to evaluate previous upgrades to computing resources, forecasting models and training. Scientifically, forecast verification is used to improve understanding of the forecast model performance and help scientists improve the forecast model. Economically, forecast verification takes into account the different needs of different types of forecast users. For example, a farmer could use precipitation forecasts in very different ways to an insurance company. The farmer may be interested in how much rainfall his farm will get over the course of the summer, whereas the insurance company may be interested only in very heavy rainfall on certain days that could cause flooding damage or concert cancellation. If the verification is informative, the administrator, scientist or other user gets useful information on the quality of the forecasts, which can then help to inform their decisions.
The last 20 years of Verification
In the past 20 years weather forecasts have changed greatly and forecast verification methods have also changed. In the early 1990s, faster computers allowed ensemble forecasting to develop. Ensemble forecasting represents the uncertainty inherent in weather prediction by producing multiple forecasts, each of which has a small alteration to the model starting conditions and/or to the forecast model. This meant that different verification methods, such as the continuous ranked probability skill score (CPRSS), had to be used. Figure 1 shows how ECMWF’s ensemble forecast of temperature at 850hPa for Europe has improved from 1995 to 2014 using the CRPSS. Ensemble forecasts produced today for 8.5/9 days ahead for this parameter are equivalent in skill to ensemble forecasts produced in 1996 for 4 days ahead.
Research is ongoing to develop new forecast verification methods for specific users (such as emergency services, aviation, agriculture etc.) and specific weather parameters (such as precipitation, wind speed, etc.). While this research is mostly taking place in weather centres and universities, you can also get involved. The World Meteorological Organisation's Joint Working Group on Forecast Verification Research (JWGFVR) has set a challenge to individuals or teams to develop and demonstrate new user-oriented forecast verification metrics. All applications of meteorological and hydrological forecasts relevant to user sectors such as agriculture, energy, emergency management, transport, will be considered. The metrics must be new and can be quantitative scores or diagnostics (for example, diagrams). The deadline for completed entries is 31 October 2016 and full details are at the JWGFVR pages on the WMO website.