16.2 SIMULATING OUT-OF-SAMPLE FORECASTS

16.2 SIMULATING OUT-OF-SAMPLE FORECASTS

Assuming that the forecasts, $f_{t+h|t}$ , are based on model estimates, $\hat{f}(x_t,{\hat{\beta }}_t)$ , statistics that are a function of the forecasts will have sampling properties that depend on the estimation scheme used to generate such forecasts. The key observation is that the estimated parameters of the forecasting model, ${\hat{\mathbf{\beta }}}_t$ , typically get updated as time passes since they are based on information in the data set available for making the forecast. Time variation in ${\hat{\beta }}_t$ introduces a new source of variation in the forecast. Moreover, recursive updating of the parameters of the forecasting model induces a correlation between shocks to the outcome variable, $y_{t+h}$ , and future parameter estimates, ${\hat{\beta }}_{t+h},{\hat{\beta }}_{t+h+1}$ , etc. These effects extend the problem from one of simply evaluating a sample mean to one that depends in a more complicated way on the data.

Two possible scenarios arise. First, sampling errors in the parameters of the forecasting model may be asymptotically irrelevant. In this case inference can be based on average loss assuming that the forecast errors come from the true model. Second, how the forecast was constructed may matter asymptotically, in which case estimation of the parameters of the forecasting model provides a second source of randomness in the average loss. For the second scenario, several different methods for how the parameters of the forecasting model are estimated have been studied. Each method varies in the sampling distribution of the loss $L(\hat{f}(x_t,{\hat{\beta }}_t),y_{t+h})$ due to differences in how the estimates ${\hat{\mathbf{\beta }}}_t$ are arrived at. Often such differences are either not asymptotically negligible or, even when they are, still have different second-order asymptotic effects.

To account for the criticism that in-sample forecasts are not feasible in real-time, out-of-sample (OoS) forecasts impose the constraint that the parameter estimates of the forecasting model use only information available at the time the forecast was computed, which we denote time $t$. Hence, only information included in $Z_t$ can be used to estimate the forecasting model at time $t$ and generate forecasts $f_{t+h|t}$ . Advancing one period to time $t+1$ , only information included in $Z_{t+1}$ can be used to generate forecasts $f_{t+h+1|t+1}$ , and so forth. The same argument holds for choice of forecasting model, i.e., only information known at time $t$ can be used to select a model for generating a forecast $f_{t+h|t}$ . Only at the very end of the sample, at time $T$ , is the full sample available for selecting and estimating the parameters of the forecasting model.

The many variants of OoS forecast estimation methods differ in how they account for possible model instability by discounting past data more or less strongly. Differences between the various weighting methods can be illustrated in the context of the linear regression model with a one-period forecast horizon $(h=1)$ ,

$$y_{t+1}={\beta }^{\prime }{x}_t+{\epsilon }_{t+1},$$

which leads to one-step forecasts of the form ${\hat{\mathbf{f}}}_{t+1|t}={\hat{\mathbf{\beta }}}_t^{\prime }{\mathbf{x}}_t$ , where

$${\hat{\beta }}_t=(\sum _{s=1}^t\omega (s,t)x_{s-1}{x}_{s-1}^{\prime }{)}^{-1}(\sum _{s=1}^t\omega (s,t)x_{s-1}{y}_s).\text{\hspace{1em}(16.7)}$$

Different methods are defined by different weight functions $\omega (s,t)$ which determine the importance of recent versus older data. We next describe the most common weighting schemes.