6.7.2 Economic Measures of Forecast Performance

6.7.2 Economic Measures of Forecast Performance

We have emphasized the potential for differences between statistical and economic measures of predictive performance. To evaluate this issue in the present context, we consider the portfolio allocation decisions of an investor with mean–variance preferences

$$E\left[U(y_{t+1})|Z_t\right]=E\left[y_{t+1}|Z_t\right]-\frac{a}{2}\mathrm{Var}(y_{t+1}|Z_t),\text{\hspace{1em}(6.35)}$$

where $E\left[y_{t+1}|Z_t\right]$ is the expected portfolio return, $\mathrm{Var}(y_{t+1}|Z_t)$ is the conditional variance of the portfolio return and $\alpha =5$ captures the investor’s risk aversion. The investor chooses between a risk-free asset paying the three-month T-bill rate and the risky stock market portfolio. This investor’s portfolio return is given by $y_{t+1}=y_{ft+1}+{\omega }_t{y}_{st+1}$ , where ${\mathbf{y}}_{ft+1}$ is the risk-free rate, ${\mathbf{y}}_{st+1}$ is the excess returns on stocks, net of the T-bill rate, and ωt is the weight on stocks selected on the basis of information available at time t, $Z_t=\{y_{\tau },x_{\tau }{\}}_{\tau =1}^t$ . To avoid bankruptcy concerns, we restrict $f_t$ to lie between 0 and 1 and so rule out short selling and leverage. The portfolio weight is then determined from ${\omega }_t(Z_t)=\max (\min ({\hat{\beta }}_t^{\prime }{x}_t/a{\hat{\sigma }}_{\epsilon t}^2,1),0)$ , where ${\hat{\beta }}_t$ is the recursively estimated OLS estimate from a linear return regression model and ${\hat{\sigma }}_{\epsilon t}^2$ is the recursive estimate of the variance of the residuals from this regression.

Table 6.3 shows the mean return and volatility (standard deviation) of the resulting portfolio, along with the annualized certainty equivalent return (CER) and Sharpe ratio. The CER is defined as the annualized risk-free return which, over the evaluation sample, would have made an investor indifferent between following the benchmark strategy suggested by the prevailing mean model that assumes constant mean returns versus alternative strategies based on time-varying predictors.

TABLE 6.3:
Portfolio performance based on out-of-sample forecasts of quarterly US stock market returns using the forecasting methods listed in the rows. The portfolio performance is reported for a risk-averse mean–variance investor who chooses between stocks and T-bills based on the predicted excess return on stocks. Mean return and Std measure the mean and volatility of the resulting portfolio returns while CER is the certainty equivalent return and Sharpe is the Sharpe ratio. All measures are annualized.

The Sharpe ratio is the mean portfolio return in excess of the T-bill rate divided by the standard deviation of the portfolio return, both estimated over the evaluation sample.

The results suggest that the methods associated with the most volatile return forecasts do not always translate into portfolio strategies with more volatile returns. In fact, the most volatile portfolio returns are associated with the excess return forecasts that are mostly positive, i.e., the Bagging and Lasso methods with the smallest penalty factor, the AIC, and the prevailing mean. These methods do not, however, generate the highest certainty equivalent returns. The BIC, backward stepwise method, and the two Lasso methods generate CER values above 1% and Sharpe ratios somewhat higher than the Sharpe ratio associated with the prevailing mean model.

Figure 6.4 plots the recursively estimated allocation to stocks. This is the forecaster’s choice variable in this setting. A value of 1 corresponds to allocating all money to stocks, while a value of 0 corresponds to holding only T-bills. The portfolio weights fluctuate a great deal and frequently fall on the 0–1 bounds that we impose in our analysis. This is a result of our assumption of no transaction costs and would change if we assumed that large changes to the portfolio weights were costly. The upper bound of 100% invested in stocks is binding more often than the lower bound of zero allocation to stocks. This is a reflection of the equity risk premium, i.e., the higher historical mean of stock returns compared with returns on T-bills, whose average is positive during the sample and so it takes relatively strong evidence to persuade the investor not to allocate any money to stocks.

Figure 6.5 (left) shows a scatterplot of the RMSE versus the hit rate, i.e., the proportion of correctly predicted signs of excess returns, across the nine different model selection methods considered here. There is a negative correlation between the two measures, suggesting that the models with the highest proportion of correctly predicted signs tend to generate the smallest RMSE values.

Figure 6.5 (right) shows the potential disconnect between statistical measures of forecast performance such as RMSE and economic measures such as the CER.

Figure 6.4: Mean–variance investor’s allocation to stocks based on recursively generated predictions using different ways to select the forecasting model. A value of 1 on the vertical axis shows a 100% allocation to stocks, while 0 shows a 100% allocation to Treasury bills.

Interestingly, the methods with the highest RMSE values tend to generate the highest CER values. For example, the models selected by the BIC and backward stepwise methods both generate high RMSE values, but also generate mean CER estimates of 1% per year or higher.

These findings suggest that forecasting methods with poor statistical performance can still yield forecasts that are valuable from an economic perspective. The reason is that large forecast errors can be punished more heavily by convex statistical loss functions such as MSE compared with economic loss functions which may put less weight on the large forecast errors.

Figure 6.6 plots the cumulative wealth over time from investing 1 in 1970 and then using the recursively generated forecasts to optimally allocate between the stock market portfolio and three-month T-bills. We show results for the prevailing mean strategy which does not model any time variation in the conditional equity risk premium. This strategy ends up with \35 in 2013. The best performing strategy is the Lasso method using the larger penalty term (c2) which generates a cumulated wealth above $50 at the end of 2013. Such plots should be interpreted with caution given the wide standard errors that surround the cumulative values. Nevertheless, they do serve as an illustration that economic and statistical measures of forecasting performance can yield quite different results.

Figure 6.5: Relation between the percentage of correctly predicted signs of excess stock returns versus the root mean squared error (RMSE) (left) and the relation between a mean–variance investor’s certainty equivalent return (CER) versus the RMSE (right).