2.6 CONCLUSION

2.6 CONCLUSION

In any nontrivial forecasting situation, any forecasting method is going to make errors as the forecast will not equal the outcome with probability 1. As a consequence, forecasters need to assess the impact these errors will have on decisions based on imperfect forecasts. The loss function quantifies how costly forecast errors are for the decision maker. Formally, loss functions map the space of outcomes and decisions (forecasts) to the real number line (which is typically normalized to the nonnegative part of the real number line, although this is just a convenience) and so allow us to directly measure the economic effects of forecast errors.

In many situations the step of formally constructing a loss function that is relevant to the particular forecasting problem is skipped, and instead an informal statistic— often an intuitive function of the outcomes and forecasts assessed over a number of forecast situations or time periods—is used to evaluate forecasting performance. We showed above (using the Kuiper’s score as an example) that such approaches can be difficult to interpret and often the resulting measures of loss have poor properties. This approach is therefore to be avoided.

Instead, any approach to a real decision or forecasting problem should carefully consider the relevance of the loss function to the real costs of the errors that are sure to arise when the forecasting method is put into practice. In some cases this involves constructing a loss function that is specific to a particular problem. In situations with a financial outcome that can be directly measured, this can and should be employed as the loss function. In other situations care needs to be taken to ensure that the loss function employed approximates to a reasonable extent the actual costs associated with the forecast errors.

In the next few chapters, as well as later in the book when we examine forecast evaluation, we show that loss functions matter for every step of the classical forecasting process. This includes estimation of parameters, choice of models, and the eventual evaluation of the forecasting method. It follows that the best forecasting method plausibly will depend on the choice of the loss function and a forecasting model built for one loss function may be inferior when evaluated on a different loss function. This is highly suggestive of taking seriously the step of constructing the loss function when building a forecasting model.

There are situations where it might be relevant to simply choose an “off-the-shelf” method, many of which we discuss in this chapter. Often forecasts are “intermediate” inputs provided to higher-level decision makers (e.g., the Greenbook forecasts computed by the Federal Reserve) or to the public (e.g., public weather forecasts provided by the government). Sometimes the end use is either not known with sufficient precision or the end uses are diverse enough across different agents that it is simply not possible to construct the loss function of the end user. In these cases it is typical to use mean squared error or similarly simple loss functions. This is a useful approach, although (a) perhaps a density forecast in this situation makes more sense, and (b) there are costs from not matching the forecast with the loss function. Indeed, the density forecasting approach is becoming more prevalent in these situations as weather forecasts are given as probabilities and government agencies make increasing use of fan charts, etc.