17.4.1 Complications Arising from Nested Models

17.4.1 Complications Arising from Nested Models

West (1996) established that two nonnested models’ relative predictive accuracy will asymptotically be normally distributed, albeit with standard errors that need to account for estimation error. When models are nested, this result, or the results of West and McCracken (1998), need no longer hold. Under the null that the additional predictors of the larger model have zero coefficients and so are irrelevant, the forecasts from the large and small model will be identical. In the limit as the effect of estimation error vanishes, the standard error of the difference between the two models’ forecast performance will therefore be 0. Test statistics based on differential MSE performance will therefore have nonstandard limiting distributions.

To deal with such situations, Clark and McCracken (2001) consider the t-test for $\beta =0$ in (17.7) along with the ENC-T and ENC-NEW statistics for testing whether one model encompasses another smaller model that it nests. The distributions of such tests are generally nonstandard and the limit distributions for the ENC-T and ENC-REG methods in (17.9) and (17.10) are asymptotically equivalent under the null. The nonstandard distributions depend on the difference in the number of parameters in the two forecasting models, the estimation method used to generate out-of-sample forecasts (i.e., static, rolling, or recursive windows), and the proportion of out-ofsample to in-sample observations, $T_P/T_R$ . As this last ratio becomes small, the test statistic becomes more like a standard normal distribution. Clark and McCracken provide critical values constructed from Monte Carlo approximations to the null distribution for a number of cases.

To see how such nonstandard distributions arise, we follow the analysis in Hansen and Timmermann (2012) and consider a simple regression model with only a constant:

$$y_{t+1}=\beta +{\epsilon }_{t+1},{\epsilon }_{t+1}\sim (0,{\sigma }_{\epsilon }^2).\text{\hspace{1em}(17.24)}$$

Suppose that $\beta$ is estimated recursively by least squares, so that $\begin{array}{r}{\hat{\beta }}_t=t^{-1}{\sum }_{s=1}^t{y}_s\end{array}$ Using this model, the one-step-ahead forecast of $y_{t+1}$ given information at time t becomes the sample average, i.e., ${\hat{f}}_{t+1|t}={\hat{\beta }}_t$ . Following Hansen and Timmermann, we compare this forecast to a simple benchmark forecast ${\hat{f}}_{t+1|t}^b=0$ , which does not require any parameters to be estimated.

Once again, suppose that $T_R$ observations are used for initial estimation, while the remainder of the sample, $T_P=T-T_R$ , is used for forecast evaluation, and let $\lambda =T_R/T$ be the fraction of the sample used for initial estimation, while $1-\lambda$ is used for out-of-sample evaluation.

We evaluate the forecasts through their out-of-sample MSE values measured relative to those of the benchmark forecasts:

$$D_T(\mathbf{\lambda })=\sum _{t=T_R}^{T-1}{\left(y_{t+1}-{\hat{f}}_{t+1\mid t}^b\right)}^2-{\left(y_{t+1}-{\hat{f}}_{t+1\mid t}\right)}^2.\text{\hspace{1em}(17.25)}$$

Using the form of the two forecast errors in (17.25), we have

$$\begin{array}{l}{D}_T(\lambda )=\sum _{t={\bar{T}}_{\epsilon }}^{T-1}{\left(y_{t+1}-{\hat{f}}_{t+1\ell }^b\right)}^2-{\left(y_{t+1}-{\hat{f}}_{t+1\ell }\right)}^2\\ =\sum _{t={\bar{T}}_{\epsilon }}^{T-1}(y_{t+1}-\beta +\beta {)}^2-{\left(y_{t+1}-\beta -({\hat{\beta }}_t-\beta )\right)}^2\\ =\sum _{t={\bar{T}}_{\epsilon }}^{T-1}({\epsilon }_{t+1}+\beta {)}^2-{\left({\epsilon }_{t+1}-\frac{1}{t-1}\sum _{s=1}^{t-1}{\epsilon }_{s+1}\right)}^2\\ =\sum _{t={\bar{T}}_{\epsilon }}^{T-1}{\beta }^2+2\beta {\epsilon }_{t+1}-{\left(\frac{1}{t-1}\sum _{s=1}^{t-1}{\epsilon }_{s+1}\right)}^2+2\left(\frac{1}{t-1}\sum _{s=1}^{t-1}{\epsilon }_{s+1}\right){\epsilon }_{t+1}.\end{array}$$

Simulated critical values for out-of-sample MSE test statistic for nested model comparison (large model has two extra regressors).

Next, define the partial sum,

$$W_T(u)=\frac{1}{\sqrt{T}}\sum _{s=1}^{\lfloor uT\rfloor }{\epsilon }_s,u\in [0,1].$$

From Donsker’s theorem, $W_T(u)\Rightarrow {\sigma }_{\epsilon }B(u)$ , where $B(u)$ is a standard Brownian motion. Hence, as shown by Hansen and Timmermann (2012),

$$\sum _{t=T_R}^{T-1}{\left(\frac{1}{t-1}\sum _{s=1}^{t-1}{\epsilon }_{s+1}\right)}^2=\frac{1}{T}\sum _{t=T_R}^{T-1}{\left(\frac{T}{t-1}{W}_T\left(\frac{t-1}{T}\right)\right)}^2\text{\hspace{1em}(17.26)}$$

$${\to }^d{\sigma }_{\epsilon }^2{\int }_{\lambda }^1{u}^{-2}B(u{)}^{2}du;$$

$$\sum _{t=T_R}^{T-1}\left(\frac{1}{t-1}\sum _{s=1}^{t-1}{\epsilon }_{s+1}\right){\epsilon }_{t+1}=\sum _{t=T_R+1}^T\frac{T}{t-1}{W}_T\left(\frac{t-1}{T}\right)\left[W_T\left(\frac{t}{T}\right)-W_T\left(\frac{t-1}{T}\right)\right]$$

$${}^d{\sigma }_{\epsilon }^2{\int }_{\lambda }^1{u}^{-1}B(u)dB(u).\text{\hspace{1em}(17.27)}$$

Let $\begin{array}{r}{\hat{\sigma }}_{\epsilon }^2=(1-\lambda {)}^{-1}{T}^{-1}{\sum }_{t=T_R}^{T-1}(y_{t+1}-{\hat{f}}_{t+1\mid t}{)}^2\end{array}$ be a consistent estimator of ${\sigma }_{\epsilon }^2$ . Under the null, $H_0:\beta =0;$ , it follows from (17.27) that

$$\frac{D_T(\lambda )}{{\hat{\sigma }}_{\epsilon }^2}{\to }^{d}2{\int }_{\lambda }^1{u}^{-1}B(u)dB(u)-{\int }_{\lambda }^1{u}^{-2}B(u{)}^{2}du,\text{\hspace{1em}(17.28)}$$

where $B(u)$ is a standard Brownian motion. Hence the test statistic in (17.28) converges to an integral of Brownian motion and will not have a standard normal distribution. This happens because of the recursive updating in the parameter estimates and the fact that innovations to the dependent variable are correlated with future revisions to the parameter estimates.

This simple derivation from Hansen and Timmermann (2012) is a special case of the general results in McCracken (2007). McCracken tabulates the critical values for cases with fixed, rolling, and expanding estimation windows and covers regressions with multivariate extensions to the baseline model. Using the earlier notation, let $\pi =T_P/T_R$ , while $\lambda =T_R/(T_R+T_P)$ . Table 17.1 shows a few select critical values for McCracken’s OOS-F test, simulated using the methods of Hansen and Timmermann (2015), for the linear regression model with two additional variables in the large prediction model for 95 and 99% critical values.

Note, first, how sensitive the critical values are to the sample split parameter, λ, varying from around 2 to 4 at the 99% critical level as the sample split fraction decreases from around 0.90 to one-third. Second, and as a result, these critical values can be very different from conventional critical values used in hypothesis testing under the assumption that $\pi =0$ so $\lambda =1$ . With two additional predictor variables in the large model, these critical values are 4.826 for $\alpha =0.95$ and 7.910 for $\alpha =0.99$ ; see McCracken (2007, table 4).