Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds Tilmann Gneiting 1 , Larissa I. Stanberry 1 , Eric P. Grimit 2 , Leonhard Held 3 and Nicholas A. Johnson 4 Technical Report no. 537 Department of Statistics, University of Washington June 2008 Abstract We discuss methods for the evaluation of probabilistic predictions of vector-valued quantities, that can take the form of a discrete forecast ensemble or a density forecast. In particular, we propose a multivariate version of the univariate verification rank histogram or Talagrand diagram that can be used to check the calibration of ensemble forecasts. In the case of density forecasts, Box’s density ordinate transform provides an attractive alternative. The multivariate energy score generalizes the continuous ranked probability score. It addresses both calibration and sharpness, and can be used to compare deterministic forecasts, ensemble forecasts and density forecasts, using a single loss function that is proper. An application to the University of Washington mesoscale ensemble points at strengths and deficiencies of probabilistic short-range forecasts of surface wind vectors over the North American Pacific Northwest. Keywords Calibration · Density forecast · Ensemble postprocessing · Exchangeability · Forecast verification · Probability integral transform · Proper scoring rule · Sharpness · Rank histogram 1 Introduction One of the major purposes of statistical analysis is to make forecasts for the future, and to provide suitable measures of the uncertainty associated with them. Consequently, forecasts ought to be issued in a probabilistic format, taking the form of probability distributions over future quantities or events (Dawid 1984). Stigler (1975) gives a lucid account of the 19th century transition from point estimation to distribution estimation. Today, we may be witnessing what future generations might 1 Department of Statistics, University of Washington, Seattle, Washington, USA 2 3Tier Environmental Forecast Group, Seattle, Washington, USA 3 Institut f¨ ur Sozial- und Pr¨aventivmedizin, Abteilung Biostatistik, Universit¨at Z¨ urich, Switzerland 4 Department of Statistics, Stanford University, Stanford, California, USA 1