18.1.3 Empirical Application to Volatility Forecasting

18.1.3 Empirical Application to Volatility Forecasting

Applications of density forecasting in finance have often focused on volatility modeling. As discussed in chapter 2, Patton (2011) establishes that the MSE and QLIKE loss functions can be used to consistently rank volatility forecasts even when the outcome is measured with noise but can be proxied through the realized variance. Patton (2011) suggests using Diebold–Mariano tests applied to these loss functions.

Table 18.1 provides an illustration of this approach applied to daily stock market returns measured by the S&P500 index. As benchmark we use the risk metrics model which downweights past squared returns exponentially by a factor $\lambda =0.94$ along with a 300-day rolling window volatility estimator. These forecasts are compared to a GARCH(1,1) and an AR model with lag length selected by AIC estimated on the realized variance series, using MSE loss. The initial estimation window uses 10 years of data from 2000 to the end of 2009, while the out-of-sample evaluation period runs from 2010 through 03/31/2015 and so covers more than five years. Forecast precision is measured using the realized variance as a proxy for the outcome. Positive values of the Diebold–Mariano test show that the squared errors of the model listed first are higher than those of the model listed last. The results show that the AR model performs best and in fact beats the GARCH and rolling window models with $p-$ values close to 5%. Conversely, the rolling window method performs quite poorly in this application.

18.2 EVALUATING FEATURES OF DISTRIBUTIONAL FORECASTS

An alternative to examining the density forecast by means of some loss function is to review certain features of the density forecast. The idea is to check whether certain features of the distribution of the outcomes line up well with features implied by the distributional forecasts. This can be done either by directly relating the predicted and “actual” distributions and examining whether they are identical, or by considering certain features of the probability distribution.

Feature-based evaluation has been most fully developed for the case where Y is either 0 or 1 and so is a binary random variable with a probability forecast ${\hat{p}}_t=p(Y_{t+1}=1|z_1,...,z_t)$ . Murphy (1973) showed that the standard quadratic probability score, $E[(Y_{t+1}-{\hat{p}}_t{)}^2]$ , can be decomposed as follows (see equation (15.3)):

$$\begin{array}{rl}& E\left[(Y_{t+1}-{\hat{p}}_t{)}^2\right]=\mathrm{Var}(Y_{t+1})+E[({\hat{p}}_t-E\left[Y_{t+1}|{\hat{p}}_t\right]{)}^2]\\ & -E[E\left[Y_{t+1}|{\hat{p}}_t\right]-E[Y_{t+1}]{]}^2.\end{array}\text{\hspace{1em}(18.9)}$$

The first term in (18.9) is the unconditional variance of the outcome variable and so is independent of the forecast; the second term is the average squared calibration error; the last term is known as the resolution term and is the squared average difference between the conditional mean given the forecast and the unconditional mean. Calibration and resolution are examined below; the hope is that calibration is small (0) while resolution is large, leading to a smaller MSE.

To compute the decomposition in (18.9), for the case where $b_{\mathcal{P}i}$ follows a discrete distribution we divide the line from 0 to 1 into m bins with centers $p_i,i=1,\dots ,m$ Suppose again that we have $T_P$ (out-of-sample) observations to evaluate forecasting performance and observe $n_i$ observations in the i th bin so that $T_P={\sum }_{i=1}^m{n}_i$ . For each bin we predict ${\nabla }_{Pi}{n}_i$ of the outcomes to be equal to 1. For each i, let the outcomes be $y_{ij},j=1,\dots ,n_i,$ and let ${\bar{y}}_i=n_i^{-1}{\sum }_{j=1}^{n_i}{y}_{ij}$ be the associated sample mean of the outcomes. For the special case with $Y=0,1$ , Murphy (1973) suggests the following decomposition:

$$T_P^{-1}\sum _{t=T_R}^{T-1}(y_{t+1}-p_t{)}^2=T_P^{-1}\sum _{i=1}^m{n}_i({\bar{y}}_i-p_i{)}^2-T_P^{-1}\sum _{i=1}^m{n}_i({\bar{y}}_i-\bar{y}{)}^2+\bar{y}(1-\bar{y}).\text{\hspace{1em}(18.10)}$$

A good forecast, i.e., one with a small MSE, has calibration as close to 0 as possible and a resolution as high as possible. Calibration and resolution are correlated and it can be unclear how to adjust these measures to minimize the mean squared loss since any attempts to minimize the calibration term might also decrease the resolution.

To derive the decomposition in (18.10), note that

$$\begin{array}{rl}& \mathbb{E}\frac{\frac{\lambda }{2}}{\lambda }{\sum }_{i=0}^N-{\lambda }^2{\sum }_{j=1}^N-{\lambda }^2{\sum }_{i=1}^N\\ & -\mathbb{E}\frac{\lambda }{2}{\sum }_{i=0}^N-{\lambda }^2{\sum }_{j=1}^N-{\lambda }^2{\sum }_{i=1}^N{\sum }_{j=1}^N{\lambda }^2\\ & -{\lambda }^2\frac{\lambda }{2}{\sum }_{i=0}^N{\lambda }^2\left({\sum }_{j=1}^N-{\lambda }^2{\sum }_{i=1}^N-{\lambda }^2{\sum }_{j=1}^N{\lambda }^2\right)\\ & -\mathbb{E}\frac{\lambda }{2}{\sum }_{i=0}^N{\lambda }^2\left({\lambda }^2-{\sum }_{j=1}^N-{\lambda }^2{\sum }_{i=1}^N{\lambda }^2\right)\\ & -\mathbb{E}\frac{\lambda }{2}{\sum }_{i=0}^N{\lambda }^2\left({\lambda }^2-{\sum }_{j=1}^N-{\lambda }^2{\sum }_{i=1}^N{\lambda }^2\right)\\ & -\mathbb{E}\frac{\lambda }{2}{\sum }_{i=0}^N{\lambda }^2\left({\sum }_{j=1}^N-{\lambda }^2{\sum }_{i=1}^N{\lambda }^2\right)\\ & -\mathbb{E}\frac{\lambda }{2}{\sum }_{i=0}^N{\lambda }^2\left({\lambda }^2-{\sum }_{j=1}^N-{\lambda }^2{\sum }_{i=1}^N{\lambda }^2\right)\\ & -\mathbb{E}\frac{\lambda }{2}{\sum }_{i=0}^N\end{array}$$

Here we used that $\begin{array}{r}{T}_P^{-1}{\sum }_{i=1}^m{n}_i{\bar{y}}_i=\bar{y}\end{array}$ . The first term is the calibration (ignoring the scaling by $p_i(1-p_i))$ , while the second term is the resolution.

Other decompositions of the Brier score exist (see, e.g., Sanders, 1963), one of which results in a property known as sharpness which refers to how concentrated the distributional forecast is. Sharpness is a feature of the forecasts themselves rather than how they relate to the outcomes. It is commonly evaluated based on a histogram of the probability forecasts on [0, 1]. A sharp forecast has a lot of probability mass near 1 or 0, the idea being that the forecast is giving decisive signals on which outcome will occur.

Extensions from the binary problem to general distributions have only recently been tackled. Our discussion reflects the small formal literature on such extensions and ignores some less used properties of density forecasts. We first cover calibration, then resolution, and finally sharpness.