18.5 EVALUATING INTERVAL FORECASTS

18.5 EVALUATING INTERVAL FORECASTS

Interval forecasts take the form of a statement that the future outcome will fall in some interval $[p_{t+1|t}^1(\alpha ),p_{t+1|t}^{\mathrm{u}}(\alpha )]$ with probability α for $\alpha \in (0,1)$ . Construction of such forecasts is examined in section 13.5.

To evaluate interval forecasts, define the indicator variable $1_{y_{t+1}}=1(y_{t+1}\in$ $[p_{t+1|t}^1(\alpha ),p_{t+1|t}^{\mathrm{u}}(\alpha )])$ which equals 1 if the outcome falls inside the forecast interval and 0 otherwise. A testable implication that could serve as a null hypothesis is that

$$E\left[1_{y_{t+1}}\right]=\alpha .\text{\hspace{1em}(18.25)}$$

For any given sample of outcomes and interval forecasts, this suggests testing correct unconditional coverage, $\begin{array}{r}(T-1{)}^{-1}{\sum }_{t=1}^{T-1}{1}_{y_{t+1}}=\alpha \end{array}$ . This leads to a test that the outcome falls in the interval with the stated probability, precisely the same idea as calibration but for a fixed probability. Since the test statistic is a simple sample mean, one can use a standard t-test, perhaps with robust standard errors. Again this property cannot form the basis for a comparison of forecasts based on different conditioning information. For example, a confidence interval based on the unconditional distribution of $y_{t+1}$ should pass this test.

Christoffersen (1998) extends the idea from correct unconditional coverage to correct conditional coverage given a set of information. This leads to a null hypothesis $E\left[1_{y_{t+1}}|Z_t\right]=\alpha$ , for any $z_t$ . Specifically, Christoffersen refers to the sequence of interval forecasts, $\{p_{t+1|t}^1\dot{(}\alpha ),p_{t+1|t}^{\mathrm{u}}(\alpha ){\}}_{t=1}^{T-1}$ , as being efficient with respect to the conditioning information, $Z_t=z_1,\dots ,z_t,\mathrm{if}$

$$E\left[1_{y_{t+1}}|Z_t\right]=\alpha .\text{\hspace{1em}(18.26)}$$

The key requirement is that conditional efficiency requires (18.26) to hold for all t. This is a much stronger requirement than the property of correct unconditional coverage in (18.25). In fact, under the null of conditional efficiency (18.26), Christoffersen shows that a test of $E\left[1_{y_{t+1}}|z_1,\dots ,z_t\right]=E\left[1_{y_{t+1}}|1_{y_t},1_{y_{t-1}},\dots ,1_{y_1}\right]=\alpha$ for all $t,$ is equivalent to testing that the sequence $\{1_{{\mathbf{y}}_t}\}$ is identically and independently distributed Bernoulli with parameter α, $\{{\underline{1}}_{y_t}\}\sim i.i.d$ . Bern(α). This suggests a simple test. Under the null hypothesis, the likelihood for the indicators $(1_{y_1},\dots ,1_{y_{T-1}})$ is

$$L(\alpha ;1_{{\gamma }_1},\dots ,1_{{\gamma }_{T-1}})=(1-\alpha {)}^{n_0}{\alpha }^{n_1},$$

where $n_0$ is the number of cases for which $1_{y_t}=0.$ , and $n_1=T-n_0-1$ . No parameters need to be estimated under the null. Under the alternative of firstorder Markov dependence in the indicators, we can define $n_{ij}$ as the number of observations where value i was followed by value $j,i,j\in \{0,1\}$ and ${\pi }_{ij}=\mathrm{Pr}(1_{y_t}=$ $j|1_{y_{t-1}}=i)$ . Then the approximate likelihood function becomes

$$L({\pi }_{00},{\pi }_{11};1_{y_1},\dots ,1_{y_{T-1}})={\pi }_{00}^{n_{00}}(1-{\pi }_{00}{)}^{n_{01}}(1-{\pi }_{11}{)}^{n_{10}}{\pi }_{11}{}^{n_{11}}.$$

This likelihood requires estimating two parameters, ${\pi }_{00}$ and ${\pi }_{11}.\mathrm{A}$ joint likelihood ratio test for correct coverage and independence is then (Christoffersen, 1998)

$$\mathrm{LR}=-2\log \left[L(1-\alpha ,\alpha ;1_{y_1},\dots ,1_{y_{T-1}})/L({\pi }_{00},{\pi }_{11};1_{y_1},\dots ,1_{y_{T-1}})\right],$$

which asymptotically has a ${\chi }_2^2$ distribution, provided that we ignore issues related to estimation error due to how the forecasts were generated.

The ideas of resolution and sharpness have also been applied to evaluate interval forecasts, at least informally. Sharpness means having short intervals. Resolution means that the length of the interval forecasts varies with conditioning information.