8.5.1 Empirical Comparisons

8.5.1 Empirical Comparisons

Table 8.2 reports out-of-sample root mean squared forecast errors for a subset of the models considered here and in the previous chapter, including random walk, prevailing mean, EWMA, AR, STAR, and Markov switching models. For the inflation series, the EWMA model with $\delta =0.8$ or δ chosen optimally (labeled ${\delta }^{\ast })$ by recursively minimizing the MSE as well as the AR model with lag length selected by the BIC, produce the lowest RMSE values while the STAR and prevailing mean model perform very poorly. The poor performance of the prevailing mean is perhaps to be expected since this model fails to capture the persistence of the underlying variable. This is not a concern for stock returns where the prevailing mean model comes out on top, along with the AR model selected by BIC which, as we have seen, rarely includes any lags.

While the nonlinear models do not perform well for the inflation and stock return series, they produce quite accurate predictions for the unemployment rate series, although even here they are bettered by the AR models. In the case of the interest rate series, we would expect the regime switching model to perform quite well because of the presence of periods with very different dynamics such as the Federal Reserve’s monetarist experiment during 1979–1982 and the period with zero lower bound on the short interest rate following the global financial crisis. This is indeed what we find as this model produces more accurate point forecasts than the linear models, although the random walk model performs even better for this variable.

TABLE 8.2:
Out-of-sample root mean squared forecast errors for different forecasting methods. For methods with estimated parameters, the parameters are estimated using a recursively expanding window. The sample evaluation period is 1970–2014.

TABLE 8.3:

Diebold–Mariano tests comparing the mean squared error performance of different forecasting methods to the benchmark random walk model (for inflation, unemployment and interest rates) or the prevailing mean model (for stock returns). Positive values indicate that a model produces lower mean squared errors than its benchmark.

Table 8.3 shows Diebold–Mariano tests for different models’ mean squared error performance measured against the performance of a benchmark which, for the persistent inflation, unemployment and interest rate series is the random walk model, while the benchmark is the prevailing mean for the stock return series. Positive values indicate superior performance relative to the benchmark and the Diebold–Mariano test, discussed in detail in chapter 17, gives some indication of the significance of differences in forecasting performance. By this measure, the EWMA approach with optimal choice of δ, the AR model with lag length selected by BIC, and the Markov switching model perform somewhat better than the random walk benchmark for the inflation rate. For the unemployment rate series, linear autoregressive models perform quite well as does the STAR model with lag length selected by BIC. Conversely, none of the models produce accurate forecasts, and many produce distinctly worse forecasts than their benchmark, for stock returns and the interest rate series.

The stock return series is quite different from the other series considered here. This variable is not persistent at all and so it is perhaps unsurprising that the random walk model (using the past quarter’s stock return to forecast next quarter’s stock return) does extremely poorly. Interestingly, the STAR models also perform very poorly for this series, and only the AR(BIC) method applied to stock returns produces lower RMSE values than the prevailing mean.