18.6 CONCLUSION

Approaches to evaluating distributional forecasts differ in whether they conduct direct comparisons guided by a loss function (often called accuracy) or whether they examine properties of the density forecast as it relates to outcomes. Evaluations based on loss functions allow us to compare pairs of distribution forecasts which is useful in situations where some benchmark model exists. The unconditional distribution of the outcome is an obvious benchmark and it corresponds to the no-change forecast often used in point forecasting. In other areas, the difference between a distribution forecast and the unconditional forecast (evaluated by means of some loss function) is known as “skill.”

Rather than using loss functions directly to examine density forecasts, various characteristics of distributional forecasts are often examined. Calibration refers to the “reliability” of a forecast and measures whether forecasts of various probabilities result in outcomes occurring with the same frequencies. This measure ignores the spread in the forecast distribution and well-estimated unconditional distributions of the outcome will have this property. Resolution refers to how the forecast varies with conditioning information and is usually gauged relative to the unconditional distribution. Sharpness refers to how well the distributional forecast manages to rule out unexpected events. By themselves, none of these measures delivers a verdict on whether the distributional forecast is good. However, each property is worth examining at the model-building stage, as they can suggest regions where the distributional forecast might be lacking.

IV Refinements and Extensions ----------------------------------------

Forecasting under Model Instability

specified in the sense that it differed from the data-generating process. Assuming a sufficiently stationary environment, the estimated parameters of the misspecified model will converge to their pseudo-true values and the resulting estimates are often useful for forecasting.

This chapter examines the frequently encountered situation where the parameters of the forecasting model change over time in such a way that the full-sample estimates of the forecasting model provide a poor representation of the forecasting model at the end of the sample when the forecast is computed. Model parameters may evolve over time due to phenomena such as shifting market conditions, changing regulations, new technologies, or changes in government policies.

Empirical evidence strongly suggests that parameter instability plagues commonly used forecasting models. In a large-scale study of forecasting models fitted to a wide range of macroeconomic variables, Stock and Watson (1996) find evidence of parameter instability for the majority of the variables. In a frequently cited study, McConnell and Perez-Quiros (2000) document a sharp drop in the volatility of output and various components of GDP around 1984. This phenomenon is known as the “Great Moderation” and is reflected in a wide variety of macroeconomic variables and surely affects volatility forecasts for such variables.

Model instability, when neglected, can show up in the form of a disparity between a forecasting model’s in-sample and out-of-sample performance or even in differences in the model’s out-of-sample forecasting performance in different subsamples. Stock and Watson (2003a) conclude, “Forecasts based on individual indicators are unstable. Finding an indicator that predicts well in one period is no guarantee that it will predict well in later periods. It appears that instability of predictive relations based on asset prices (like many other candidate leading indicators) is the norm.” Ang and Bekaert (2007), Paye and Timmermann (2006), and Rapach and Wohar (2006) find evidence of instability for prediction models fitted to stock market returns.

This chapter focuses on forecasting models of the form $f(z,{\beta }_t)$ , where ${\beta }_t$ follows some exogenous process, i.e., parameter changes are nonrandom or follow a random process that is independent of the observed data. This contrasts with “observation driven models” (Cox, 1981) which allow the parameters to depend on observable data. Assuming that the parameters change in a stable manner, such models sometimes give rise to nonlinear forecasting models with stable parameters. For such cases, the modeling issues that arise are similar to those examined in chapter 8 on forecasting with nonlinear models.

Three difficulties arise when constructing good forecasts in the presence of model instability. First, it is generally very difficult to determine the exact form of the parameter instability. Tests of the null of model stability have power against many potential types of parameter instability and so rejections of the null of stability tend to be uninformative about the nature of the instability. Second, as a result, the forecaster often does not have a good idea of how to model parameter instability. We examine different approaches to modeling parameter instability in the second section of this chapter.

Finally, when the forecasting model is unstable, future values of the parameters might change again over the forecast horizon. In this situation, forecasting procedures—and even methods for calculating the risk of a particular forecasting approach—require modeling both the probability and magnitude of future breaks. Models with rare changes to the parameters—e.g., a single break—have little or nothing to say about the chance of a future break. For such models it can be reasonable not to consider the chance of additional breaks over the forecast horizon, assuming that the horizon is sufficiently short. For models with multiple breaks, such an approach is far less reasonable.

Section 19.1 provides a broad discussion of how breaks affect forecasting performance and section 19.2 follows up by pointing out some limitations to in-sample tests for model instability. Procedures for handling single and multiple breaks are covered in sections 19.3 and 19.4, respectively. Forecasting methods that posit a model for the process generating past and future breaks are discussed in section 19.5, while the opposite approach—ad hoc methods for dealing with breaks—are covered in section 19.6. Section 19.7 discusses model instability and forecast evaluation and section 19.8 concludes.