Met Éireann - met.ie - The Irish Meteorological Service Online
 
met.ie - The Irish Meteorological Service Online     
The Irish Meteorological Service Online  

Monday, 11 December 2017
  
FINE
1 Celsius
W Moderate 
Search:   
Print this webpage
  Home - Weather News

 

Weather News

Forecast Verification
11 November 2015

Forecast Verification


“Forecasts possess no intrinsic value. They acquire value through their ability to influence the decisions made by users of the forecasts.” Allan Murphy, (1931-1997) Professor of Atmospheric Sciences at Oregon State University and pioneer in the field of probabilistic weather forecast verification.

What is Forecast Verification?

Weather models are verified to monitor and improve the quality of their forecasts and to compare the quality of different forecast systems. To verify a forecast, it must be compared to a 'truth' such as a corresponding observation set or other estimate of the true weather like a forecast analysis, a very short-range forecast  or the previous day’s weather conditions (known as the persistence method).

Why verify?

Verification of forecasts is important for administrative, scientific and economic reasons, and any verification scheme should add value by being informative (Joliffe and Stephenson, 2012). From an administrative viewpoint, measuring the performance of forecasts allows decision makers to evaluate previous upgrades to computing resources, forecasting models and training. Scientifically, forecast verification is used to improve understanding of the forecast model performance and help scientists improve the forecast model. Economically, forecast verification takes into account the different needs of different types of forecast users. For example, a farmer could use precipitation forecasts in very different ways to an insurance company. The farmer may be interested in how much rainfall his farm will get over the course of the summer, whereas the insurance company may be interested only in very heavy rainfall on certain days that could cause flooding damage or concert cancellation. If the verification is informative, the administrator, scientist or other user gets useful information on the quality of the forecasts, which can then help to inform their decisions.

The last 20 years of Verification

In the past 20 years weather forecasts have changed greatly and forecast verification methods have also changed. In the early 1990s, faster computers allowed ensemble forecasting to develop. Ensemble forecasting represents the uncertainty inherent in weather prediction by producing multiple forecasts, each of which has a small alteration to the model starting conditions and/or to the forecast model. This meant that different verification methods, such as the continuous ranked probability skill score (CPRSS), had to be used. Figure 1 shows how ECMWF’s ensemble forecast of temperature at 850hPa for Europe has improved from 1995 to 2014 using the CRPSS. Ensemble forecasts produced today for 8.5/9 days ahead for this parameter are equivalent in skill to ensemble forecasts produced in 1996 for 4 days ahead.


Figure 1.  ECMWF ensemble forecast performance (1995-2014) for temperature at 850 hPa for Europe – each point on the curves is the forecast range at which the 3-month mean (blue lines) or 12-month mean centred on that month (red line) of the continuous ranked probability skill score (CPRSS) falling below 25%. From Haiden et al 2014 p22.


Future advances - you can get involved!

Research is ongoing to develop new forecast verification methods for specific users (such as emergency services, aviation, agriculture etc.) and specific weather parameters (such as precipitation, wind speed, etc.). While this research is mostly taking place in weather centres and universities, you can also get involved. The World Meteorological Organisation's Joint Working Group on Forecast Verification Research (JWGFVR) has set a challenge to individuals or teams to develop and demonstrate new user-oriented forecast verification metrics. All applications of meteorological and hydrological forecasts relevant to user sectors such as agriculture, energy, emergency management, transport, will be considered.  The metrics must be new and can be quantitative scores or diagnostics (for example, diagrams). The deadline for completed entries is 31 October 2016 and full details are at the JWGFVR pages on the WMO website.


Footnote
If you want to find out more about forecast verification methods, a good overview is available on the CAWCR website. The area is comprehensively covered in Joliffe and Stephenson (2012) and Wilks (2011). Here is the link to some of Met Éireann's verification.

 
References

  • Jolliffe, I.T., and D.B. Stephenson, 2012: Forecast Verification: A Practitioner's Guide in Atmospheric Science. 2nd Edition.  Wiley and Sons Ltd, 274 pp.
  • Wilks, D.S., 2011: Statistical Methods in the Atmospheric Sciences. 3rd Edition.  Elsevier, 676 pp.
  • Haiden, T., M. Janousek, P. Bauer, J. Bidlot, L. Ferranti, T. Hewson, F. Prates, D.S. Richardson and F. Vitart, 2014, Evaluation of ECMWF forecasts, including 2013-2014 upgrades. Technical Memorandum 742, European Centre for Medium-Range Weather Forecasts. Available from: <http://old.ecmwf.int/publications/library/do/references/list/14>. [4 November 2015]
  • World Meteorological Organisation (JWGFVR) 2015, Challenge to Develop and Demonstrate the Best New User-Oriented Forecast Verification Metric. Available from: <http://www.wmo.int/pages/prog/arep/wwrp/new/FcstVerChallenge.html>. [4 Nov 2015]
  • Collaboration for Australian Weather and Climate Research (CAWCR) 2015, Forecast Verification. Available from:  <http://www.cawcr.gov.au/projects/verification/>. [4 November 2015]

 
MET ÉIREANN, Glasnevin Hill, Dublin 9, Ireland Tel: +353-1-8064200 Fax: +353-1-8064247

© The material on this site is Met Éireann copyright. The name Met Éireann and the Met Éireann logo are registered trademarks. Copyright Notice.
Times indicated in forecasts and reports may be local or UTC.
twitter World Weather Services


Site Design and Development by FUSIO