15.3 EFFICIENCY PROPERTIES WITH KNOWN LOSS

15.3 EFFICIENCY PROPERTIES WITH KNOWN LOSS

To more rigorously discuss and conduct tests on the relationship between actual and predicted values, it is necessary to establish a benchmark for what constitutes an optimal or “good” forecast. As emphasized throughout the book, this issue is best addressed with reference to the forecaster’s loss function and information set. We begin by discussing the properties that an optimal forecast should have for general loss functions and then address more specialized cases such as quadratic or homogenous loss.

The idea of rational or optimal forecasts originates from the rational expectations literature starting with Muth (1961) which was subsequently brought to the analysis of forecasts by Mincer and Zarnowitz (1969) and followed by papers such as Nordhaus (1987) and Batchelor and Dua (1991). The original notion was that a rational agent constructs forecasts of the outcome variable given knowledge of the true data-generating process (DGP) for the outcome—including the model and its parameters—using all currently available information. Rationality means optimal use of all information available to the forecaster in constructing the forecast, so “rational,” “efficient,” and “optimal” forecasts are synonymous in this setting. Forecasts are deemed rational if, given a loss function and a particular set of data, they fully capture all available information in the sense that these data cannot be used to construct a better forecast. Various flavors of rationality have been proposed, such as weak versus strong rationality, unbiasedness of forecasts and orthogonality of the forecast error to all available information. The resulting tests imply similar constructs from the perspective of the optimality property and amount to testing the first-order condition that the forecast cannot be improved upon using ex ante available information.

In the spirit of the rational expectations literature, the early literature assumed that efficient forecasts were based on knowledge of the true data-generating process, including the true parameter values. More recent work has relaxed this assumption to allow for parameter estimation errors. In the presence of uncertainty about the true data-generating process, the notion of efficient forecasts is that they are not obviously misspecified and thus dominated by other forecasting methods.

By definition, an efficient forecast has the property that no other forecast or combination of the forecast and the available data in the forecasters information set, $Z_t.$ , can be used to generate a smaller expected loss so that, consistent with (3.3),

$$f_{t+h|t}^{\ast }=\underset{f(z_t)}{\mathrm{argmin}}E\left[L(f(z_t),y_{t+h})|Z_t\right].\text{\hspace{1em}(15.4)}$$

Equivalently, efficiency holds if $f^{\ast }$ is sufficient for the best forecast given the information available in the extended set of data $z_t\in Z_t$ , i.e., for any ${\tilde{f}}_{t+h|t}=f(z_t)$ ,

$$\begin{array}{r}E\left[L(f_{t+h|t}^{\ast }(z_t),y_{t+h})|Z_t\right]\le E\left[L({\tilde{f}}_{t+h|t}(z_t),y_{t+h})|Z_t\right].\end{array}\text{\hspace{1em}(15.5)}$$

If additional information in $Z_t$ can be used to provide a better forecast than $f^{\ast }{}_{:}$ then the forecast is deemed not to be optimal in the sense that it fails to effectively utilize all data in $Z_t$ . The definition is explicitly conditional on the information set considered, $Z_t,$ , and so any test of efficiency is conducted conditional on a particular information set. Finding that a prediction could be improved by adding information not available when the original forecast was computed does not prove that the original (smaller) information set was inefficiently utilized. It may of course help improve the original forecast if this information subsequently becomes available.

The composition of the information set lies behind the commonly used terminology of “weak” versus “strong” forecast efficiency. Tests of weak-form forecast efficiency refer to an information set that includes only past forecasts and past outcomes (and hence past forecast errors), i.e., $Z_t=\{y_{t-i},f_{t+h-i|t-i},e_{t-i|t-h-i}{\}}_{:}$ $i=0,1,\dots ,t.$ Strong-form efficiency refers to extensions of the information set to include other variables, often all variables that were publicly available at the time the forecast was made.

Rationality is also defined in the context of a given loss function: a forecast that is efficient under one loss function, $L_1$ , need not be efficient under another loss function, $L_2\ne L_1$ . For example, one loss function may require predicting only the mean, while another loss function may require forecasts of both the mean and the variance. Forecast models that use a correct specification of the mean but not for the variance would then be efficient for the first loss function, but not for the second. In situations where the loss function underlying the forecast is unknown, evidence that a given forecast is inefficient with respect to one loss function, while it seems efficient with regard to another loss function might help establish the shape of the loss function that the forecaster implicitly used, at least under the null of forecast rationality.

Assuming that the loss function is once differentiable with regard to the forecast, the first-order condition for an optimal forecast becomes

$$E[L^{\prime }(f_{t+h\mid t}(z_t),y_{t+h})|Z_t]=0,\text{\hspace{1em}(15.6)}$$

where $L^{\prime }(f_{t+h\mid t}(z_t),y_{t+h})$ is the derivative of the loss function with respect to the forecast, evaluated at the optimum forecast, $f_{t+h|t}^{\ast },i.e.$ ,

$$L_{t+h|t}^{\prime }(.)\equiv {\frac{\partial L\left(f_{t+h|t},y_{t+h}\right)}{\partial f_{t+h|t}}|}_{f_{t+h|t}^{\ast }}.\text{\hspace{1em}(15.7)}$$

The first-order condition in (15.6) forms the basis for many forecast rationality tests. Granger (1999) refers to the derivative in (15.7) as the generalized forecast error. Under a broad set of conditions that allow interchange between the expectation and differentiation operators, an implication of (15.6) is that the generalized forecast error is a martingale difference sequence and so has zero conditional mean with respect to all elements in the forecaster’s information set. These conditions do not require that L is everywhere differentiable with respect to the forecast, nor is a unique optimum assumed.

The conditional moment condition (15.6) is written with respect to the information set $Z_t$ and so does not lend itself directly to testing. If $Z_t$ is generated by the variables $z_t.$ , then the conditional moment condition in (15.6) can be converted into an unconditional moment condition of the form

$$E[L^{\prime }(f_{t+h\mid t},y_{t+h})g(z_t)]=0,\text{\hspace{1em}(15.8)}$$

for all possible functions $g(z_t){.}^3$ We use the notation $v_t=g(z_t)$ for a given choice of such functions and refer to these as “test functions” following White (2000). Different choices of test functions, $v_t.$ , change the direction of the power of tests of the moment conditions in (15.8). A specific test function, $v_t$ , yields power in the direction of correlation between the derivative of the loss function and $v_t$ but has zero power in directions that are orthogonal to $v_t$

The most common way to implement a test of the orthogonality condition in (15.8) is to consider a regression,

$$L^{\prime }(f_{t+h\mid t}(z_t),y_{t+h})={\beta }^{\prime }{v}_t+u_{t+h},\text{\hspace{1em}(15.9)}$$

and testing $H_0:\beta =0$ versus $H_1:\beta \ne 0$

For MSE loss $L^{\prime }(f_{t+h\mid t}(z_t),y_{t+h})$ is just the difference between the outcome and the forecast, i.e., the forecast error ${\mathbf{e}}_{t+h|t}$ . The main challenge in conducting inference lies in understanding the sampling distribution for $\hat{\mathbf{\beta }}$ from such a regression. For example, inference generally depends on how the forecasts are generated. We discuss these issues below.

The implication that can be derived from rejections of rationality is that there is information in $v_t$ that could potentially be used to improve the forecast. Including $v_t$ in the information set for constructing the forecast and recomputing the optimal forecast could result in a forecast that has lower expected risk and could also result in a revised forecast for which $L^{\prime }(f_{t+h\mid t}(z_t),y_{t+h})$ is orthogonal to $v_t$ and hence is deemed rational.

While forecast efficiency tests are contingent upon the available information, they also critically depend on the assumed loss function. In the special case of MSE loss, the first-order condition in (15.6) does not involve any unknown parameters. However, for most other loss functions, additional parameters are involved. Even if these parameters are known, the first-order condition in (15.6) will typically depend on additional parameters capturing the shape of the loss function and so does not only involve the forecast error. Conversely, if the parameters of the loss function are unknown and have to be estimated, efficiency tests generalize to whether any parameters exist within a particular family of loss functions for which the forecast can be rationalized. Estimating these additional parameters then becomes part of the rationality test. Yet a third case arises if the shape of the loss function is unknown, or at least not known up to a few shape parameters. While the unrestricted case is untreatable, rationality tests are possible if restrictions can be imposed on the loss function and/or on the underlying data-generating process. We next discuss a variety of such cases.