18.1.2 Comparing Density Forecasts

18.1.2 Comparing Density Forecasts

The method of Amisano and Giacomini (2007) compares two density forecasts, ${\hat{p}}_{Y1}(y|z)$ and ${\hat{p}}_{Y2}(y|z)$ , using the log scoring rule. Using a setup similar to Giacomini and White (2006) for their distribution theory, Amisano and Giacomini (2007) assume that each successive forecast is based on a rolling window of observations which ensures that estimation error does not disappear asymptotically. Letting $y_{t+1}$ be the outcome variable whose density is being predicted, Amisano and Giacomini (2007) base density evaluations on the studentized weighted average differences in log scores, i.e.,

$$t_{\mathrm{AG}}={\hat{\sigma }}_{T_P}^{-1}\sqrt{T_P}\sum _{t=T_R}^{T-1}w\left(\frac{y_{t+1}-{\hat{\mu }}_t}{{\hat{\sigma }}_t}\right)\left(\log \left[{\hat{p}}_{Y1t}(y_{t+1})\right]-\log \left[{\hat{p}}_{Y2t}(y_{t+1})\right]\right),\text{\hspace{1em}(18.6)}$$

where log $\lceil {\hat{p}}_{Yit}(y_{t+1})\rceil$ is the log of the i th predictive distribution evaluated at the outcome ${\mathbf{y}}_{t+1},{\hat{\mathbf{\mu }}}_{\mathbf{t}}$ and ${\hat{\sigma }}_t$ are estimates of the mean and variance of $y_{t+1}$ , respectively, based on the same rolling estimation window used to construct the forecasts, and ${\hat{\sigma }}_{T_P}^2$

is an estimate of the long-run variance of

$$\left\{w\left(\frac{y_{t+1}-{\hat{\mu }}_t}{{\hat{\sigma }}_t}\right)\left(\log \left[{\hat{p}}_{Y1t}(y_{t+1})\right]-\log \left[{\hat{p}}_{Y2t}(y_{t+1})\right]\right)\right\}$$

for $t=T_R,\dots ,T-1$ . Using notation from chapters 16 and 17, $T_P=T-T_R$ is the length of the evaluation sample, and $T_R$ is the start of the prediction sample.

Amisano and Giacomini (2007) suggest estimating ${\hat{\sigma }}_{T_P}^2$ using standard methods such as Newey and West (1987). Should it be desired, w(·) can be chosen to put more or less weight on different parts of the distribution. For example, a symmetric zeromean density for $w(\cdot )$ would weight observations near the mean of $y_{t+1}$ more heavily than the tails. This might be appropriate for a forecaster with MSE loss. The authors suggest a range of other weighting functions corresponding to alternative choices for where the focus of the test should be. Under fairly general assumptions they provide results showing that the limiting distribution for $t_{\mathrm{AG}}$ is standard normal and so can be compared to standard critical values.

Gneiting and Ranjan (2011) propose an adjustment to the approach of Amisano and Giacomini (2007) which is based on the sum of a different function of the data and density forecasts. Their approach towards weighting regions of the support of the density is guaranteed to be proper, i.e., it ranks the true predictive density highest, if this is included among the density models being compared. Once again, this leads to a test statistic that is asymptotically normally distributed.

Chapter 17 showed that pairwise comparisons of point forecasts have been extended to comparisons of multiple point forecasts. Similar extensions have been proposed for comparing the accuracy of multiple density forecasts. Employing the Reality Check of White (2000), Corradi and Swanson $(2006\mathsf{a},\mathsf{b})$ propose ways to test whether the density forecast generated by one out of possibly many misspecified models is better in the sense that it more closely approximates the data than some benchmark model. Their approach considers parametric density models, ${\hat{p}}_{Yit}(y_{t+1},{\theta }_i),i=1,...,n$ , whose parameters are updated recursively using, e.g., quasi-maximum likelihood estimators ${\hat{\theta }}_{it}$ that minimize $-{\sum }_{\tau =1}^{t-1}$ ln ${\rho }_{Yi\tau }(y_{\tau +1},{\theta }_i)$ . Each density model is associated with a conditional distribution, ${\hat{F}}_{Yit}(u,{\theta }_i)=$ $\mathrm{Pr}(y_{t+1}\le u|Z_t,{\theta }_i)$ , where $F_{0t}(u,{\theta }_0)$ is the true conditional probability distribution. Corradi and Swanson propose to use squared errors to measure the distance between probability distributions and the data and assume that the first model is the benchmark we are interested in beating. Proceeding as in White (2000), they test the composite null that even the best among the alternative models, $k=2,\dots ,m$ cannot generate distribution forecasts that are closer to the “true” distribution than the distribution forecasts implied by the benchmark model:

$$\underset{k=2,\dots ,m}{\max }{\int }_{U}E\left[{\left({\hat{F}}_{1t}(u,{\theta }_1^{\ast })-F_{0t}(u,{\theta }_0^{\ast })\right)}^2-{\left({\hat{F}}_{kt}(u,{\theta }_k^{\ast })-F_{0t}(u,{\theta }_0^{\ast })\right)}^2\right]\phi (u)du\le 0,\text{\hspace{1em}(18.7)}$$

where ${\theta }_i^{\ast }$ is the probability limit of ${\hat{\theta }}_{it}$ . The alternative is that this expression exceeds 0. Here $\phi (u)\ge 0$ and $\begin{array}{r}{\int }_U\dot{\phi }(u)du=1\end{array}$

The test statistic in (18.7) tests whether the mean squared error between the cumulative distribution function values for the kth model measured relative to the true conditional distribution are smaller than that of the first (benchmark) model. Negative values indicate that none of the models perform better than the benchmark and so suggest that the benchmark model should be preferred. In practice, the true conditional distribution is of course unknown—otherwise we would not have a model selection problem in the first place—so Corradi and Swanson propose testing the null in (18.7) through the statistic

TABLE 18.1:
Diebold–Mariano test applied to GARCH(1,1), risk metrics, and rolling window forecasts of daily US stock market volatility.

Note: The out-of-sample range is 2000–2010 using daily data.

$$\begin{array}{rl}& W_{T_P}=\underset{k=2,\dots ,m}{\max }\frac{1}{\sqrt{T_P}}{\int }_U\sum _{t=T_R}^{T-1}[{\left({\hat{F}}_{1t}(u,{\hat{\theta }}_{1,t})-1(y_{t+1}\le u)\right)}^2\\ & -{\left({\hat{F}}_{kt}(u,{\hat{\theta }}_{k,t})-1(y_{t+1}\le u)\right)}^2]\phi (u)du,\end{array}\text{\hspace{1em}(18.8)}$$

where the indicator function 1(·) equals 1 if the condition inside the brackets is true, and 0 otherwise. Hence, the probability forecasts are directly compared to the rate at which outcomes of a certain magnitude occur in the evaluation sample. Corradi and Swanson (2006c) survey the theory and implementation of inference with this test statistic which, as in White (2000), is based on bootstrap methods.