18.1.1 Evaluation of Individual Density Forecasts

18.1.1 Evaluation of Individual Density Forecasts

To empirically evaluate a density forecast, we require an observed sequence of outcomes and density forecasts $\{y_{t+1},{\hat{p}}_{Yt}(y_{t+1})\},t=1,\dots ,T-1$ . One way to proceed would be to convert the density forecast into a point forecast and use the methods for point forecast evaluation covered in chapter 16. Using this approach, we would first generate the optimal point forecast, $f({\hat{p}}_{Yt}(y_{t+1}))$ , implied by the density forecast, ${\hat{p}}_Y(y_{t+1}|Z_t)$ , and the loss function, $L(f({\hat{p}}_{Yt}(y_{t+1})),y_{t+1})$ . For example, under MSE loss, the point forecast $f(z_t)$ is simply the conditional mean of $y_{t+1}$ computed using ${\hat{p}}_{Yt}(y_{t+1})$ . The average loss associated with the density

forecast then becomes

$${\hat{L}}_T=(T-1{)}^{-1}\sum _{t=1}^{T-1}L(f({\hat{p}}_{Yt}(y_{t+1})),y_{t+1}),\text{\hspace{1em}(18.1)}$$

which is in the form of an average loss, as examined in chapter 16.

This simple approach to evaluating density forecasts might be appropriate for a number of reasons. First, as we saw in chapter 13, density forecasts can be justified on the grounds that there are multiple users with different loss functions. Any one of these users might examine the performance of a density forecast with reference to the specific loss function deemed appropriate for their problem. The relevant measure of forecast performance is the average loss calculated from each user’s specific loss function. Second, even if a density forecast works well for some loss functions, it need not do so for all loss functions. For example, a density forecast that works well under MSE loss—meaning it is well centered—may be very poor for loss functions based on quantiles near the tails. This is just another way of saying that evaluation is specific to the loss function assumed in (18.1).

Understanding the sampling properties of (18.1) is difficult since the realized loss is a complicated function of the underlying density forecast and loss function. Moreover, the density forecasts typically depend on models whose estimates are recursively updated. The assumptions of West (1996) may not apply to situations with density forecasts, so actually assessing the correct sampling distribution and its dependence on the underlying estimates used to construct the density forecasts requires extensions of these methods, at least for situations where recursive updating in parameter estimates introduces nonstationarities. A typical approach is to ignore estimation error and simply evaluate the sampling error treating the observations on the realized loss, $L(f(z_t),y_{t+1})$ , as data. However, this may understate the true sampling variability of the sample mean.

Rather than converting the density forecast to a point forecast, an alternative is to use loss functions that are directly related to densities, such as the scoring rules discussed in chapter 2, sometimes known as skill scores. Scoring rules are functions that map the outcome $y_{t+1}$ and the density forecast ${\hat{p}}_{Yt}(y_{t+1})$ to the real number line. If proper, they are maximized in expectation when the density forecast and the distribution of the outcome are the same.

For binary outcomes, distributional forecasts are equivalent to conditional mean forecasts. Since many loss functions for point forecast result in proper scoring rules, the two approaches are often equivalent. For example, under MSE loss the sample average loss associated with the distributional forecast of a binary variable is

$$Q_T^{\mathrm{MSE}}=(T-1{)}^{-1}\sum _{t=1}^{T-1}(y_{t+1}-{\hat{p}}_{Yt}(y_{t+1}){)}^2.\text{\hspace{1em}(18.2)}$$

This is often referred to as the Quadratic Probability Score (QPS) statistic or the Brier QPS after Brier (1950). As discussed in chapter 12, there is a direct link between scoring rules for the binary forecasting problem and the underlying loss functions for the point forecasts in which the scoring rule can be viewed as a weighted average taken over the loss of the underlying point forecasts.

For variables with continuous outcomes, there is no simple relationship between the scoring rule and the underlying loss function of the point forecasts. Thus, forecast evaluation usually rests on an arbitrarily chosen scoring rule. The most popular of these is the log scoring rule which leads to the sample average

$$Q_T^{\mathrm{LS}}=(T-1{)}^{-1}\sum _{t=1}^{T-1}\log ({\hat{p}}_{Yt}(y_{t+1})),\text{\hspace{1em}(18.3)}$$

where ${\hat{p}}_{Yt}(y_{t+1})$ is the conditional density forecast evaluated at the outcome $y_{t+1}$ For binary variables (18.3) becomes

$$Q_T^{\mathrm{LS}}=(T-1{)}^{-1}\sum _{t=1}^{T-1}\left(y_{t+1}\log ({\hat{p}}_{Yt}(y_{t+1}))+(1-y_{t+1})\log (1-{\hat{p}}_{Yt}(y_{t+1}))\right)\text{\hspace{1em}(18.4)}$$

This is effectively the average likelihood. As with most scoring rules, we use the negative of the loss and so look for a large average likelihood or average log score. More generally, any scoring rule $S(y_{t+1},{\hat{p}}_{Yt}(y_{t+1}))$ mapping $(y_{t+1},{\hat{p}}_{Yt}(y_{t+1}))$ to a single number results in the average score

$$Q_T=(T-1{)}^{-1}\sum _{t=1}^{T-1}S(y_{t+1},{\hat{p}}_{Yt}(y_{t+1})).\text{\hspace{1em}(18.5)}$$

As with the average loss in (18.1), understanding the sampling properties of these objects is complicated by their dependence on estimated objects.