6.7.1 Statistical Measures of Forecast Performance

6.7.1 Statistical Measures of Forecast Performance

We study out-of-sample results even though the methods we use are based on penalized in-sample performance. In part, this is because we wish to see how much the selected forecasting models change over time and across different economic environments. A second reason is that this allows us to simulate the forecasting models’ “real-time” performance. The advantages and disadvantages of this approach are discussed in more detail in chapter 17 of the book.

All methods considered here use linear forecasting models. The forward stepwise approach uses a maximum p-value of 0.05 for a predictor to be added to the existing forecasting model. The backward stepwise approach uses a minimum p-value of 0.10 for a predictor to be removed from the existing model.

For the Lasso method, we choose two very small values of the penalty factor, λ. Specifically, we use the formula of Belloni and Chernozhukov (2011),√ $\lambda =2c\sigma \sqrt{T}{\Phi }^{-1}(1-\alpha /2n_k)$ , setting $c=0.002$ (which we label $c_1)$ and $c=0.006$ (labeled $c_2).$

Figure 6.2: Quarterly US stock market returns and recursively generated out-of-sample forecasts based on a variety of forecasting and model selection methods, including the AIC and BIC information criteria, forward (frw) and backward (bck) stepwise selection, Bagging with different significance levels, Lasso with different penalties, cross validation (CV), and the prevailing mean (PM).

Our implementation of the Bagging scheme uses B = 1,000 bootstrapped samples of the original data. The data are arranged in $\{y_{t+1:T},X_{t:T-1}\}$ tuples. Autocorrelation in the predictors is preserved through the circular block bootstrap of Politis and Romano (1994) with block size chosen optimally according to Politis and White (2004). Dependence across observations is preserved by using the same blocks for all variables. For each bootstrapped sample the estimate, ${\hat{\beta }}^b$ , is used to compute forecasts, ${\hat{\mathbf{f}}}_{t+1|t}^b=(x_t^{\prime }{S}_t){\hat{\mathbf{\beta }}}^b$ , where $S_t$ is a stochastic selection matrix whose (i, i ) element equals the indicator function $1(|p-{\mathrm{val}}_i|<\alpha )$ . A predictor is added only if its p-value is less than α. Hence, as α grows larger the criterion becomes more liberal and the selected model becomes larger. We consider two different values of α, namely $\alpha =1\%$ , and α = 5%. The final Bagging forecasts are obtained by averaging across the bootstrap draws

$${\hat{\mathbf{f}}}_{t+1|t}^{\mathrm{BAGG}}=\frac{1}{B}\sum _{b=1}^B{\hat{\mathbf{f}}}_{t+1|t}^b.\text{\hspace{1em}(6.34)}$$

The cross-validation scheme uses a validation sample of $v=0.5(0.5t-\sqrt{t}-1)$ observations for a sample of size t and discards 0.25t observations when constructing the estimation sample.

Figure 6.2 shows the recursively generated forecasts of stock returns for several of the model selection procedures. The realized series, shown in the top left panel, is highly volatile and shows no evidence of autocorrelation. Turning to the forecasts, the first point to note is that many of the forecasts are strongly correlated across the different model selection methods. Even so, there are also important differences on some occasions. For example, the AIC and Lasso method with the smallest penalty $\left(c_1\right)$ generate more volatile forecasts than selection methods such as the BIC or the Lasso with a larger penalty term $\left(c_2\right)$

As a benchmark, we follow Welch and Goyal (2008) and use the prevailing mean. This is simply the historical average, and so forecasts $y_{t+1}$ by means of ${\sum }_{\tau =1}^t{y}_{\tau }$ . As shown in the bottom right window, the prevailing mean forecast is far more stable than the other forecasts.

Table 6.1 shows the extent of the similarity in the forecasts produced by the different methods. Below the diagonal this table shows the correlations between the forecasts, while above the diagonal it shows the correlations between the forecast errors. For most methods—the key exception being the prevailing mean—the correlation between the forecasts is strongly positive. Because of the presence of a large common unexplained component in stock returns, correlations among the forecast errors (shown above the diagonal in table 6.1) are very large and exceed 0.90.

The first two columns of table 6.2 compare the time-series behavior of the forecasts by reporting their volatility and bias. Among the information criteria and the stepwise approaches the forecasts generated by the models selected by the AIC or the backward stepwise approach are more volatile than those generated by models selected by the BIC or the forward stepwise approach. In turn, these methods lead to more volatile forecasts than the forecasts based on the Bagging or cross-validation methods. How we set the parameters of the Bagging or Lasso affects the volatility of the forecast since this influences how many predictors get included by these methods. The lower the penalty for including additional variables, the more variables get included in the forecasting model, and the more volatile the forecasts.

Figure 6.3 plots recursive inclusion frequencies. Specifically, for each variable selection method the plots show which predictors get included, indicated by an asterisk, and which ones get excluded. Such plots provide an informative way to display how different variables enter and exit from the forecasting model. We present results for four variables, namely the dividend–price ratio, the T-bill rate, the term spread, and the default yield. Clearly which variables get included varies a great deal over time and across selection methods. For example, the dividend–price ratio gets included a large proportion of the time by the backward stepwise, Akaike, and 5% Bagging methods but is never selected by the forward stepwise approach, the Lasso method with the highest penalty term, or the cross-validation method. There is also evidence of instability across time in the models being selected. For example, the term spread is selected by many of the criteria in the early-to-midseventies, but rarely afterwards, with the exception of the Lasso method with the smallest penalty factor and the AIC towards the end of the sample.

In light of these differences, it may seem surprising that the forecasts are so strongly correlated. However, many of the predictors are strongly correlated and so can substitute for each other. Different linear combinations of the predictor variables can therefore be strongly correlated.

Forecast evaluation is discussed in detail in part III of the book. However, as an indication of the methods’ predictive accuracy, we report root mean squared error (RMSE) values in the third column of table 6.2. By this criterion, the model selection methods associated with the least volatile forecasts perform best, suggesting that the predictors are not very powerful and parameter estimation error is important in this application. In fact, the prevailing mean model that does not include any timevarying predictors generates the lowest RMSE value among all the methods.

ABLE 6.1 Correlations among forecasts (shown below the diagonal) and forecast errors (shown above the diagonal) for different approaches to forecasting quarterly US stock arket return

TABLE 6.2:
Out-of-sample forecasting performance for different forecasting methods applied to quarterly US stock market returns. Std and Bias are the standard deviation and bias of the recursive forecasts, while RMSE and $R^2$ are the root mean squared forecast error and the $R^2$ , the latter measured relative to the prevailing mean model. DM lists the Diebold–Mariano test for equal predictive accuracy. Sign and $\mathrm{PT}.$ -test give the percentage of correctly predicted signs of quarterly excess returns and the associated value of the Pesaran–Timmermann (1992) test statistic, along with its $p\cdot$ -value.

An alternative measure of predictive accuracy is provided by the out-of-sample $R^2$ measure shown in the fourth column of table 6.2, measured relative to the prevailing mean model which therefore gets a score of $0.{}^{10}$ Negative values show underperformance under this metric relative to the simple prevailing mean forecast, while positive values suggest better performance against this benchmark. In this case all models generate negative out-of-sample $R^2$ -values, confirming that the size of the predictable component in stock returns is small.

As we make clear in our discussion of forecast evaluation methods in chapter 17, caution has to be exercised when conducting inference on the sampling distribution of out-of-sample measures of relative forecast precision in situations such as here, where model parameters are estimated using an expanding estimation window and the selected models change over time. With this caveat in mind, as an indication of the statistical significance of the difference in forecasting models, the fifth column of table 6.2 reports Diebold–Mariano tests computed relative to the prevailing mean model which thus gets a value of 0 by this criterion. The Diebold–Mariano test of Diebold and Mariano (1995) is explained in greater detail in chapter 17, but is computed here as a t-test on the differences in two models’ mean squared errors. Positive values suggest that the prevailing mean model produces a lower MSE value than the alternative method listed in each row, while negative values suggest

$$1-\frac{T^{-1}{\sum }_{t=0}^{T-1}(y_{t+1}-f_{it+1|t}{)}^2}{T^{-1}{\sum }_{t=0}^{T-1}(y_{t+1}-{\bar{y}}_{t+1|t}{)}^2},$$

Figure 6.3: Recursive variable inclusion plots for the dividend–price ratio, the T-bill rate, the term spread, and the default yield in models fitted to quarterly US stock market returns. Variable inclusion is marked by an asterisk, while empty spaces indicate that the variable is not selected in a given period.

the opposite. The large negative values of the Diebold–Mariano test statistic obtained here indicate that the performance of the forecasts based on the AIC and BIC model selection criteria is particularly poor.

As argued previously, statistical measures of forecast performance are not necessarily closely aligned with economic measures of forecasting performance. To highlight this, we report two alternative measures of performance. First, we report the percentage of correctly predicted signs of excess returns or the “hit rate.” This is shown in column 6 of table 6.2. The numbers range between 51% for the models selected by the backward stepwise approach versus 61% for the prevailing mean and 60% for the Lasso method with the largest penalty term.

The highest hit rate is obtained simply by always predicting a positive mean excess return. This is a so-called “broken clock” forecast that never changes and therefore carries little or no information. A genuine ability to predict the sign of excess returns must entail some ability to forecast both positive and negative signs. The sign test proposed by Pesaran and Timmermann (1992), and explained in more detail in chapter 18, accounts for the forecasts’ ability to predict both positive and negative signs. Results for this test, which is asymptotically normally distributed, are shown in the seventh and eighth columns of table 6.2. These reveal no significant ability of any of the forecasting methods to pick models that correctly forecast the direction of the stock market.