18.1 EVALUATION BASED ON LOSS FUNCTIONS

18.1 EVALUATION BASED ON LOSS FUNCTIONS

A direct approach to the evaluation of density forecasts would be to apply loss functions to a sequence of observed values of the difference between what actually occurs and the density forecasts. As we noted in the introduction to this chapter, what we observe is not a sequence of distributional outcomes but, rather, a single draw from that distribution. So the loss function applied to this setup needs to map single outcomes and density forecasts to the real line. One option is to transform the density forecast into a point forecast which can then be evaluated using standard procedures. Specifically, suppose we have in mind a particular loss function that allows us to map the density forecast into a point forecast. We can then, first, transform the density forecast to a point forecast and second, evaluate this point forecast with the same loss function. Alternatively, we could use a scoring rule which takes density forecasts as inputs.

For the first of these approaches, discussed more fully in section 18.1.1 below, it is important to note that the results will depend on the loss function outside the special (and unlikely) case that the density forecast is correctly specified. The comparison of misspecified density forecasts is discussed in Diebold, Gunther, and Tay (1998). The correctly specified density forecast is preferred over a misspecified density forecast by any loss function. In contrast, when the models generating the density forecasts are misspecified, the ranking of the associated forecasts may well differ across different loss functions—i.e., two forecasters with different loss functions will rationally disagree on their ranking of two sets of density forecasts. For example, consider the MSE and lin-lin loss functions. If one of the density forecasts is accurate for the conditional mean but not for the conditional variance, whereas the other is accurate for the conditional variance but is a little off-center, it could easily be that the MSE forecaster prefers the first density forecast whereas the lin-lin forecaster prefers the second.

Under the second approach, discussed in section 18.2 below, different scoring rules will prioritize accuracy at different points of the outcome distribution, and hence may differ in their ranking of misspecified density forecasts. For the binary forecasting problem there is a simple relation between loss functions applied to point forecasts and scoring rules. However, for more general problems there is not an obvious relationship between the two formulations (point forecasts evaluated using loss functions versus density forecasts evaluated using scoring rules). In practice, in situations where it is not straightforward to convert a density forecast into a point forecast, the scoring rule approach would seem appropriate.