14.8 CONCLUSION

Forecast combination has been widely found to be a reliable, and often simple, way to improve on the predictive accuracy of individual models. Moreover, simple combination methods such as equal weighting often perform well and can be difficult to beat in situations where combination weights are difficult to estimate with more precision or the precision of the underlying forecasts is broadly similar. This situation is often encountered with survey data for which individual forecasters’ track records can be short and forecasts tend to cluster around similar points.

Model combination can be used in situations where the forecaster has more information about the underlying models used to generate the forecasts and so is not constrained to treat the observed forecasts simply as any other data, but can evaluate the likelihood of each model. Bayesian model averaging techniques have shown promise in many empirical applications as have predictive pooling methods for generating density forecasts.

Because forecasts from individual models are likely to be generated by misspecified models, it is unlikely that any particular model will dominate all other models across time and different economic states. This helps explain why, empirically, model and forecast combinations have been found to be a reliable device for generating “pooled” forecasts that are often more robust than the individual forecasts entering into the combination.

III Forecast Evaluation ----------------------------

Desirable Properties of Forecasts

${\text{ W }}_{f_{t+1/t},t=1,...,T}^{\mathrm{at}\mathrm{constitues}\mathrm{a}{}^{\alpha }\mathrm{g}}$ ood” forecast? When considering a sequence of forecasts, , this question is related to how “close” the forecasts are to the outcome, $y_{t+h}$ . If we have more than a single sequence of forecasts we can ask whether one sequence provides the best forecasts in the sense that it is, on average, closer to the outcomes that were subsequently observed. Quantifying what we mean by “closeness” requires a metric that trades off different forecasting mistakes. Empirical work often relies on somewhat arbitrary statistical measures although, as we emphasize throughout this book, it is preferable to use an economically motivated loss function to measure the distance between the forecast and outcome. In examining distributional forecasts one still requires some metric for closeness; the loss function provides a useful foundation although more arbitrary statistical measures such as the Kullback–Leibler distance are also popular. A related question that has occupied the literature is whether the forecasts are obviously deficient in the sense that they are systematically biased on average or in certain states of the world. Before admonishing a forecast because it is biased, recall, though, that even good forecasts may be biased—for example if they trade off bias against a reduction in the estimation error or due to asymmetries in the loss function.

We break these questions up over the following chapters according to the assumed loss function, the types of data available (including restrictions assumed on the datagenerating process), and the questions being asked. A key issue is which testable implications follow from different assumptions on the loss function. We examine this in the present chapter. In situations where we observe additional data that were available to the forecaster, we can examine whether a particular sequence of forecasts efficiently exploited all information embedded in such data. Chapter 16 examines the case where a single sequence of forecasts is available for forecast evaluation and considers the estimation of expected loss.

When we observe more than a single set of forecasts or forecasting models, additional questions arise. We may be interested in examining whether one sequence of forecasts is better than others. Alternatively, even if one forecasting model is found to be better than individual alternatives, combining its forecasts with other forecasts might lead to improvements. Chapter 17 examines the evaluation and comparison of more than a single sequence of forecasts.

For forecast distributions, the issue of how close the forecasts are to the outcome becomes a question of how close the forecast distribution is to the conditional distribution of the outcomes. We examine this in chapter 18. Binary forecasting problems give rise to many interesting special case results. For the binary prediction problem, chapter 12 showed that forecasts of the probability of an outcome are distribution forecasts and so these are also considered in chapter 18.

There are many reasons why we need to evaluate forecasts. First, forecast evaluations can be valuable in their own right. Many statistical agencies provide forecasts as part of their mission, and it is important to examine whether such forecasts are any good. For example, an understanding of whether data are employed optimally to generate forecasts has implications for economic models of how decision makers behave.

A second reason for evaluating forecasts is that we are usually interested in examining whether we can improve the forecasts. Rejections of the rationality tests in this chapter provide some indication of how the forecasts can be improved and hence of how better forecasts can be constructed. In this sense forecast evaluation is simply part of the forecast estimation process. Similarly, if a comparison of multiple forecasts suggests that one set of forecasts contains information not available from another set of forecasts, this suggests either choosing the former or combining the information in the two sets so as to generate a better forecast.

Section 15.1 considers a variety of informal methods for examining forecasts, while section 15.2 presents loss decomposition methods which serve the same purpose. Section 15.3 describes testable efficiency properties that forecasts should have under known loss, while section 15.4 covers efficiency properties under unknown loss. By forecast efficiency we refer to population-optimal properties of the forecasts, ignoring the effect of estimation errors on the forecast. Sections 15.5 and 15.6 cover efficiency properties when the outcome is unobserved as well as the interpretation of efficiency tests. Section 15.7 concludes.