15.7 CONCLUSION

Forecast evaluation is an important part of constructing, monitoring, and improving individual forecasting models. In-sample methods for forecast evaluation are similar to diagnostic testing in econometric analysis and include assessing whether forecast errors are serially correlated (or, more generally predictable), in which case a better model could be constructed. Out-of-sample methods simulate models’ “real-time” performance, using only part of the sample to generate a sequence of forecasts so as to reduce the possible effect of data-mining on the forecasting performance.

Forecast evaluation methods should reflect the forecaster’s loss function. In fact, given a sequence of observed forecasts, it is sometimes possible to reverse engineer the forecaster’s unknown loss function and to conduct tests of forecast rationality under much weaker assumptions than knowledge of the forecaster’s specific loss function. In the case with mean squared error (quadratic) loss, a large literature has suggested tests for forecast optimality based on unbiasedness and lack of serial correlation in forecast errors. More broadly, assumptions about the underlying data generating process and the forecaster’s loss function can also be traded off so as to establish testable implications of forecast rationality.

Evaluation of Individual Forecasts

Measurement of predictive accuracy is closely related to forecast evaluation. Both absolute and relative measures of forecasting performance can be considered. Absolute performance measures are concerned with the accuracy of an individual forecast relative to the outcome, using either an economic (loss-based) or a statistical metric. Relative performance measures compare the performance of one or several forecasts against some benchmark. The present chapter is concerned with individual models’ absolute forecasting performance, while the next chapter deals with comparisons of predictive accuracy across multiple forecasts.

Forecast evaluation amounts to understanding whether the loss from a given forecast is “small enough.” Both formal and informal methods can be employed. Informal methods such as the graphical approach of Theil (1961) described in chapter 15 yield “big picture” answers to this question and can help point to directions in which a set of forecasts is clearly deficient. For example, a plot of the realized value against the predicted value could reveal a systematic tendency for over- or underprediction.

More formal methods that use the data to estimate average loss can also be examined. Examination of such averages has a long history, but only in the last few years have researchers begun to place standard errors on loss estimates in a way that facilitates rigorous inference about predictability. A key complication is that the distribution of the test statistic for these sample averages may depend on how the forecasts were constructed, specifically which estimation method the forecasts were based on. This has led to the recognition that not only the forecasting model, but also the forecasting method matters. For example, whether an expanding or a fixed estimation window was used to generate the forecast can affect the subsequent inference concerning predictive accuracy.

Formal evaluation of an individual forecast is concerned with testing whether the forecast is optimal with respect to some loss function and a specific information set. If forecast optimality is rejected, the implication is that the forecast can be improved upon. Forecast optimality tests have a long history reflected in a vast literature with its own nomenclature. As with the calculation of standard errors on average loss, construction of such tests requires understanding the method (and information set) used to generate forecasts and forecast errors. Often this is ignored in practice, and for many interesting situations the precise method for constructing the tests is unknown. One exception is the use of so-called pseudo out-of-sample forecasting methods which simulate the process by which forecasts could have been generated in “real time.” By construction, this setup assumes that the methods used for model estimation and forecast construction are known to the evaluator.

Sampling properties of forecast evaluation statistics, such as the consistency and standard error, may well depend not only on the model but also on how the parameters were estimated and what data were used to construct the forecast. This is important because there are several different methods for constructing forecasts. A natural approach is to simulate the real-time forecasting problem by using only historically available data to construct forecasts. This backtesting approach contrasts with an approach of using the full data sample for estimation and forecast evaluation purposes.

The outline of the chapter is as follows. Section 16.1 reviews some issues that arise in analyzing the sampling distribution of average losses. Section 16.2 turns to different schemes for simulating out-of-sample forecasts, while section 16.3 pursues more formal methods for evaluating the magnitude of average loss from a sequence of forecasts and conducting inference on out-of-sample loss. These methods reflect both the properties of the data-generating process and the method used to construct the forecasts, where the latter explicitly accounts for the fact that the forecasts are themselves generated from some forecasting model and so parameter uncertainty must be taken into account. Section 16.4 discusses asymptotics for out-of-sample forecast rationality tests with generated forecasts. Section 16.5 covers evaluation of aggregate and disaggregate forecasts and Section 16.6 concludes.