15.6 INTERPRETING EFFICIENCY TESTS

15.6 INTERPRETING EFFICIENCY TESTS

Care needs to be exercised when interpreting the outcome of forecast efficiency tests. Suppose we fail to reject the null hypothesis of forecast efficiency. Because any efficiency test is conditional on a particular choice of information set, it is possible that the information set simply was too limited and that a well-chosen augmentation of the information set could overturn the result. In other words, even if we fail to reject $H_0:{\beta }_{1h}=0$ in the regression

$$e_{t+h|t}={\beta }_{1h}{v}_{1t}+u_{1t+h},\text{\hspace{1em}(15.42)}$$

it is possible that we would reject the null hypothesis $H_0:{\beta }_{1h}={\beta }_{2h}=0$ in the regression

$$e_{t+h|t}={\beta }_{1h}{v}_{1t}+{\beta }_{2h}{v}_{2t}+u_{2t+h},\text{\hspace{1em}(15.43)}$$

where $v_{1t},v_{2t}\in Z_t$ . This simple point, that tests of forecast efficiency are contingent upon the specified information set, raises several issues. First, in the common situation where we do not observe the specific information available to the forecaster at the time when the forecast was generated, it can be very difficult to conduct a strong-form test of efficiency, i.e., a test of whether forecasters efficiently exploited all information at their disposal.

Conversely, conducting the efficiency test based on data that actually were not available to the forecaster in real time could lead us to wrongly reject that forecasts were efficient. This issue arises in forecast rationality tests with macroeconomic data which are often based on revised data as opposed to the original vintage data. If forecast errors and data on realized values were not in the forecaster’s information set, then an apparent rejection of forecast rationality is of course meaningless. At a minimum, current and past values of the forecasts are known to the forecaster which suggests using this variable, or possibly the change in the forecast, $f_{t+h|t}-f_{t+h-1|t-1}$ as a predictor.

Weak power of the forecast rationality test is a second important issue. Macroeconomic data are typically recorded at monthly or quarterly frequency which limits us to small sample sizes and reduces the power of efficiency tests. This small sample concern is particularly strong in survey data such as the Survey of Professional Forecasters or Blue Chip forecasts where the forecast record of individual forecasters typically is very short and often not contiguous.

Short sample sizes and lack of power is also a concern with pseudo out-of-sample forecast experiments. These experiments typically split the full data sample into estimation and evaluation samples so that only a portion of the full data is held back for the actual forecast evaluation test. Ideally, the estimation sample is large enough to obtain precise estimates, while the evaluation sample is sufficiently large to obtain high power for the test. However, macroeconomic and many finance applications clearly impose severe limits on the available data. Another issue is the fact that macroeconomic data get revised and so an evaluation that uses data that were unavailable to the forecaster in “real time” can lead to wrong conclusions. Clark and McCracken (2009) address the importance of this issue both theoretically and for an empirical application involving inflation forecasts.

A related issue arises when the outcome distribution perceived by the forecaster differs from the empirical distribution observed in a particular evaluation sample, as in the case of the famous “peso” problem. Suppose that a currency forecaster puts a nonzero weight on the possibility that there will be a catastrophic event such as a substantial devaluation of the Mexican peso against the US dollar, perhaps due to an economic meltdown leading to a run on the currency. Forecasts that account for this possibility will lead to peso predictions that are on average lower than if this possibility was ignored. This makes rational forecasts appear biased if the meltdown does not occur in a given evaluation sample. The bias arises because the evaluation sample is not sufficiently representative in the sense that it does not contain a currency crash as stipulated by the forecaster. In this sense the peso problem is a small sample problem in which the evaluation sample differs significantly from the population distribution. If a very long sample was available with sufficiently many crash events to match the forecaster’s prediction, the bias would vanish.

We might also be limited by not trusting data or forecasts too far in the past to have been generated by the same process as current values. The forecast methods used by survey participants or institutions can undergo change, leading to a failure to statistically reject the null of forecast rationality, even though the forecasts are poor from an economic perspective. Conversely, if rejections based on full-sample information reflect the forecaster’s learning process, it is not clear that the forecaster used historically available information inefficiently.

A final issue in interpretation of forecast rationality tests is that any test of rationality is a joint test of rationality and the assumed loss function. As in the case with the forecaster’s information set, the forecaster’s loss function is typically unknown. This can lead to wrong rejections or failures to reject the null of forecast rationality. An immediate implication of a rejection of forecast rationality is that the forecaster did not optimally use all information and that a better forecast could have been produced. Suppose that the loss function that was actually used by the forecaster differs from that assumed by the evaluator. If the forecaster has asymmetric loss so the cost of over- and underpredictions are not the same, it would typically have been rational to generate biased forecasts; see section 15.2. Forecast efficiency tests based on MSE loss would reject the null of forecast rationality due to such bias and so lead to the wrong conclusion. We illustrate this point in the following example.

Example 15.6.1 (MSE evaluation of asymmetric quadratic loss). Suppose a forecaster has the following asymmetric quadratic loss function:

$$L(e_{t+1\mid t})\equiv [\alpha +(1-2\alpha )1(e_{t+1\mid t}<0)]e_{t+1\mid t}^2.$$

This gives rise to the following first-order condition:

$$E\left[v_t\left(\alpha -1\left(e_{t+1\mid t}<0\right)\right)\mid e_{t+1\mid t}\mid \right]=0.\text{\hspace{1em}(15.44)}$$

The generalized forecast error in (15.7) now takes the form

$$\begin{array}{rl}& 2\left((\alpha -1(e_{t+1}<0))|e_{t+1\mid t}|\right)=2\left(\alpha |e_{t+1\mid t}|-1(e_{t+1\mid t}<0)|e_{t+1\mid t}|\right)\\ & =2\alpha |e_{t+1\mid t}|-\left(|e_{t+1\mid t}|-e_{t+1\mid t}\right)\\ & =(2\alpha -1)|e_{t+1\mid t}|+e_{t+1\mid t}.\end{array}$$

Hence the first-order condition in (15.44) can be rewritten as

$$E\left[{\mathbf{v}}_t\left((2\alpha -1)|{\mathbf{e}}_{t+1|t}|+{\mathbf{e}}_{t+1|t}\right)\right]=0,$$

which is the first-order condition from the regression

$$e_{t+1|t}+(2\alpha -1)|e_{t+1|t}|=v_t\delta +u_{t+1}.$$

Efficiency tests based on MSE loss simply use the forecast error in the regression $e_{t+1|t}=v_t\delta +{\tilde{u}}_{t+1}$ . They therefore omit the regressor $(1-2\alpha )|e_{t+1|t}|$ and induce an omitted variable bias in the estimate of δ . The omitted variable is nonzero except when loss is quadratic $(\alpha =0.5)$ and hence will cause the constant term $(if{v}_t$ includes an intercept) to be nonzero, inducing a bias. The omitted variable could also be correlated with other terms in $v_t$ and in a large enough sample the null of rationality will be rejected due to the use of an incorrect loss function.

Should we conclude from these considerations that forecast rationality is essentially empirically untestable due to the twin problems arising from joint hypothesis tests (the forecaster’s loss function and information set are unknown) and weak power due to short samples or model instability? While these issues clearly complicate forecast rationality tests, this is taking the critique too far. For example, flexible loss functions of the type in (15.28) can be used to capture most interesting deviations from quadratic error loss, at least in so far as error-based loss is concerned. Issues related to unobservable or nonstationary data can in part be dealt with by basing the forecast rationality test on observables such as forecast revisions; see (15.40).

While we here consider tests of forecast rationality given the loss function and show how to extend the tests to allow for flexibility in the loss function, an alternative question might be to consider whether we can recover the forecaster’s loss function, assuming that rationality holds. In the context of a parametric set of loss functions, this is what Elliott, Komunjer, and Timmermann (2005) allows. More generally, we might be interested in finding out whether it is possible to recover the forecaster’s loss function without constraining this to be parameterized up to a finite set of unknown parameters. This question is addressed in Lieli and Stinchcombe (2013) who provide conditions under which identification of the loss function is possible.