19.7 MODEL INSTABILITY AND FORECAST EVALUATION

19.7 MODEL INSTABILITY AND FORECAST EVALUATION

Clark and McCracken (2005b) consider how structural breaks affect encompassing tests as well as tests for equal predictive accuracy. Building on the insight that the predictive accuracy of a model with time-varying parameters depends on the timing of the break(s) relative to the point of the forecast, they find that out-of-sample tests can miss genuine predictability in the presence of breaks even in situations where in-sample tests would detect such predictability. This situation is particularly likely to occur if the parameters break towards 0 during the part of the sample that is used for out-of-sample forecast evaluation. Their analysis suggests that parameter instability may be one reason for the common finding that in-sample predictability fails to translate into out-of-sample predictability.

Rossi (2005a) provides optimal tests for the joint null of no predictability (Granger causality) and model stability. Rossi’s tests address whether $x_t$ has predictive power over $y_{t+1}$ in linear regression models $y_{t+1}={\beta }_t{x}_t+{\epsilon }_{t+1}$ in which ${\beta }_t$ might be time varying. One form of parameter instability that is particularly tractable is when ${\beta }_t$ shifts from $\beta$ to $\hat{\beta }$ at an unknown point in time. Rossi’s exponential Wald test proceeds as follows. Let ${\hat{\beta }}_1$ τ and ${\hat{\mathbf{\beta }}}_2$ τ be the OLS estimators before and after the break,

posited to occur at time τ ,

$$\begin{array}{l}{\hat{\beta }}_{1\tau }={\left(\frac{1}{\tau }\sum _{t=1}^{\tau -1}{x}_{t-1}{x}_{t-1}^{\prime }\right)}^{-1}\left(\frac{1}{\tau }\sum _{t=1}^{\tau -1}{x}_{t-1}{y}_t\right),\\ {\hat{\beta }}_{2\tau }={\left(\frac{1}{T-\tau }\sum _{t=\tau }^{T-1}{x}_{t-1}{x}_{t-1}^{\prime }\right)}^{-1}\left(\frac{1}{T-\tau }\sum _{t=\tau }^{T-1}{x}_{t-1}{y}_t\right).\end{array}$$

The test proposed by Rossi uses two components. One component, $\hat{\beta }=(\tau /T){\hat{\beta }}_{1\tau }+$ $(1-\tau /T){\hat{\beta }}_{2\tau }$ , is simply the full-sample estimate of $\beta$ . Testing that this component is nonzero amounts to a test that $\beta$ is constant, but nonzero. However, such a test is unable to detect variation in $\beta$ that causes this parameter to be ${}_0$ on average, even in situations where $\beta$ at times is significantly different from 0. The other component, ${\hat{\beta }}_{1\tau }-{\hat{\beta }}_{2\tau }$ , can detect such variation. This component measures the difference between the parameters estimated on two different subsamples and thus can identify variation in $\beta$ over time.

The exponential Wald test statistic proposed by Rossi (2005a) takes the form

$$\begin{array}{c}\mathrm{Exp}-{\mathsf{W}}_T^{\ast }=\frac{1}{T}\sum _{\tau =0.15T}^{0.85T}\frac{\exp (\frac{1}{2})}{0.7}\left({\left({\hat{\beta }}_{1\tau }-{\hat{\beta }}_{2\tau }\right)}^{\prime },{\left(\frac{\tau }{T}{\hat{\beta }}_{1\tau }+\left(1-\frac{\tau }{T}\right){\hat{\beta }}_{2\tau }\right)}^{\prime }\right){\hat{V}}^{-1}\\ \times \left(\begin{array}{c}{\hat{\beta }}_{1\tau }-{\hat{\beta }}_{2\tau }\\ \frac{\tau }{T}{\hat{\beta }}_{1\tau }+\left(1-\frac{\tau }{T}\right){\hat{\beta }}_{2\tau }\end{array}\right),\end{array}$$

where

\begin{array} { r l } & { \hat {V} = ( \begin{array} { l l } { \frac { \gamma } { 6 } } & { S _ { m } ^ { \prime } S _ { 1 } ^ { - 1 } S _ { 3 } } & { 0 } \\ { 0 } & { \frac { \gamma - 1 } { 2 } S _ { m } ^ { \prime } S _ { 2 } ^ { \prime} \frac { \hat { \delta } _ { 2 } } { S _ { 3 } } S _ { 2 } ^ { \prime} \frac { \hat { \delta } _ { 2 } } { S _ { 3 } } S _ { 3 } } \end{array} ) , } \\ & { S _ { m } = \displaystyle \frac { 1 } { T } \sum _ { \tau = - 1 } ^ { T } S _ { m - 1 } x _ { \tau - 1 } ^ { \prime} , } \\ & { \hat {S} _ { 1 } = ( \displaystyle \frac { 1 } { T } \sum _ { \tau = 2 } ^ { T } \mathrm { a } _ { \tau } \hat { \delta } _ { \tau} \hat { \delta } _ { \tau - 1 } ^ { \prime} ) + \displaystyle \sum _ { j = 2 } ^ { T - 1 } ( 1 - \displaystyle \frac { | { \frac { j } { 2 } } | } { \tau ^ { 1 / N } } ) ( \displaystyle \frac { 1 } { \tau } \sum _ { \tau = 2 } ^ { \tau } \mathrm {A} _ { \tau} \hat { \delta } _ { \tau - j } ^ { \prime} \frac { \hat { \delta} _ { j} } { S _ { j - j} ^ { \prime} \hat { \delta} _ { j - j - 1 } ^ { \prime} } ) , } \\ & { \hat{S} _ { 2 } = ( \displaystyle \frac { 1 } { T - \tau} \sum _ { \tau = + 1 } ^ { T - 1} S _ { m - 1} \hat { \delta} _ { \tau} \hat { \delta} _ { \tau} \hat { \delta} _ { \tau - 1} ^ { \prime} ) } \\ & \quad \quad + \displaystyle \sum _ {j = - 1 } ^ {T - 1} ( 1 - \displaystyle \frac {j} { ( T - \tau ) ^ { 1 / N } } ) ( \displaystyle \frac { 1 } {T - \tau} \sum _ { \tau = j + 1 } ^ { \tau - 1} \mathrm {a} _ \end{array}

and ${\hat{\mathbf{\epsilon }}}_t=y_t-x_{t-1}^{\prime }\hat{\mathbf{\beta }}$ are the regression residuals from the fitted model. In the absence of serial correlation in the data, only the first term in ${\hat{\mathbf{S}}}_1$ and ${\hat{\mathbf{S}}}_2$ matter.

Rossi (2005a, table B1) tabulates the critical values for the ${\mathrm{Exp}-\mathrm{W}}_T^{\ast }$ test under the joint null hypothesis that $x_t$ does not Granger cause $y_{t+1}$ and no time-variation in the parameters, i.e., ${\beta }_t=\beta =0$

Empirically, Rossi (2013a) finds that Granger causality tests that are robust to instabilities can detect stronger empirical evidence of predictability—including outof-sample evidence—for many macroeconomic series.

Rossi (2013a) undertakes a large-scale comparison of forecasting procedures that account for model instability. Like Pesaran and Timmermann (2007), she finds that the choice of estimation window is very important for forecasting performance. Overall, the empirical evidence presented by Rossi suggests that a simple equalweighted average tends to perform as well as or better than several more sophisticated approaches.

Giacomini and Rossi (2010) suggest fluctuation tests for examining the stability of forecasting models using their out-of-sample performance. The basic idea is that, following Giacomini and White (2006), we can consider the differences in the outof-sample losses generated by two different forecasting methods as raw data that can be examined in a test. Using this approach, Giacomini and Rossi present a number of tests. For example, their fluctuation test is a sequence of rolling Diebold– Mariano tests for equal predictive accuracy. The DM test is constructed from a window of observations around each point in the sample and so will fluctuate as the window moves. If the fluctuations are too large relative to the critical values from an asymptotic distribution, this is considered evidence of instability in the relationship between the two forecasting methods.

Rossi and Sekhposyan (2013) extend the analysis of model instability for point forecasts to cover evaluation of distribution forecasts. Their tests, which include Kolmogorov–Smirnov and Cramer–von Mises tests such as those described in chapter 18, have power against misspecified density models even if the misspecification affects only part of the sample due to model instability. Rossi and Sekhposyan (2015) develop tests for forecast rationality that are robust to instabilities in the forecast errors. If forecasts are inefficient in only part of the sample, traditional rationality tests lack power. By basing their tests on rolling windows of the sample, Rossi and Sekhposyan (2015) can detect more local evidence of deviations from forecast rationality.