17.11 CONCLUSION

Forecasters often work with multiple competing models or observe multiple forecasts from survey data. In such a situation it is not only of interest to ask whether the individual forecasts are “optimal,” using the methods from chapters 15 and 16, but also to ask whether there exists a single dominant forecasting model, or perhaps a set of models that are better than the other ones (as in the model confidence set).

Evaluating Density Forecasts

Chapter 12 and 13 considered the situation where a predictive distribution, , rather than a point forecast, was provided. This chapter examines how we can evaluate such predictive densities. For a single density forecast this means evaluating whether the density is correctly specified and leads to good forecasting performance. If there are multiple density forecasts, we may be interested in comparing them and perhaps selecting the best. Although the ideas of density forecast evaluation mirror those for point forecasts, the literature on density forecast evaluation is less well developed.

The primary difficulty that arises when evaluating density forecasts is that we never actually observe the density of an outcome; only a single draw from the distribution is observed. Further complicating matters, when we observe outcomes over a period of time the predictor variables change and so we obtain density forecasts ${\hat{p}}_{Yt}(y_{t+h})={\hat{p}}_Y(y_{t+h}|z_1,...,z_t)$ and outcomes $y_{t+h}$ conditional on different values of $z_t$ .

One way to evaluate density forecasts is to explicitly rely on a loss function that maps the density forecast and outcome to a single number that can be the basis for an examination of average loss. Such loss functions are typically known as scoring rules, especially in the statistics literature, and were discussed in chapters 2 and 13. Loss functions should be chosen to address the particular forecaster’s problem as different forecasters may use different loss functions. In practice, however, most studies rely on a small set of loss functions, the most popular being the log score, i.e., the average likelihood given certain distributional assumptions. Other methods that map the density forecast and outcome to a single number do not have an explicit loss function in mind but result in statistics that trade off different types of errors.

A second approach looks for desirable properties of the density forecast such as calibration or sharpness or matching the distribution function of the outcome. These features are often examined separately, with “good” density forecasts possessing as many desirable properties as possible.

Section 18.1 discusses density evaluation methods based on loss functions. Density forecasts are sometimes evaluated using features such as calibration, resolution, and sharpness. We cover this in section 18.2. Probability integral transform tests use information on the full density and are covered in section 18.3. Section 18.4 covers multicategory forecasts, while section 18.5 discusses how to evaluate interval forecasts, and section 18.6 concludes. Again, we focus on the case with a single-period forecast horizon, $h=1$ , to keep notation simple.