15.3.2 Optimality Tests with Known Loss Shape, but Unknown Parameters

15.3.2 Optimality Tests with Known Loss Shape, but Unknown Parameters

All of the results in section 15.3.1 rely on MSE loss which justified our use of the forecast error e in place of $L^{\prime }(f,y)$ . If the loss function were not in the MSE family, then orthogonality between the forecast error $e_{t+h}$ and the test instruments $v_t$ need not imply that the forecasts are efficient. Conversely, a rejection of orthogonality between $e_{t+h}$ and $v_t$ does not imply that the forecasts are inefficient if the loss function lies outside the MSE family.

In practice, the forecaster’s loss function is often unknown. Such a situation arises when we evaluate forecasts that are constructed by third parties that have built their forecasts for their own (unknown) purposes. While the loss function may not be known, it is possible that it could be well approximated by a family of loss functions that are known apart from a few shape parameters. Alternatively, the loss function may be entirely unknown. We cover the first case here and subsequently turn to the second case.

When the forecaster’s loss function is known up to a finite number of parameters, rationality tests that account for the unknown parameters can be used. For example, consider optimality tests in the context of the two-parameter loss function proposed by Elliott, Komunjer, and Timmermann (2005) and described in detail in chapter 2:

$$L(e_{t+1\mid t},\mathbf{p},\mathbf{\alpha })\equiv [\alpha +(1-2\alpha )1(e_{t+1\mid t}<0)]|e_{t+1\mid t}{|}^{\mathbf{p}},\text{\hspace{1em}(15.28)}$$

where $\mathbf{p}$ is a positive integer, while $\alpha \in (0,1)$ . Elliott, Komunjer, and Timmermann (2005) consider the case where $p=1$ or $p=2$ . In their setting, rationality can only be tested jointly with an estimate of α and so their analysis addresses whether there exists an α for which forecast rationality is not rejected.

To obtain an estimate of α given a set of $d$ instruments, $v_t,\mathbf{a}$ linear instrumental variable estimator can be used for the unknown loss parameter, ${\hat{\alpha }}_T:$

$${\hat{\alpha }}_T\equiv \frac{{\left[\frac{1}{T-1}{\sum }_{t=1}^{T-1}{v}_t|e_{t+1\mid t}|P^{-1}\right]}^{\prime }{\hat{S}}^{-1}\left[\frac{1}{T-1}{\sum }_{t=1}^{T-1}{v}_{t}1(e_{t+1\mid t}<0)|e_{t+1\mid t}|P^{-1}\right]}{{\left[\frac{1}{T-1}{\sum }_{t=1}^{T-1}{v}_t|e_{t+1\mid t}|P^{-1}\right]}^{\prime }{\hat{S}}^{-1}\left[\frac{1}{T}{\sum }_{t=1}^T{v}_t|e_{t+1\mid t}|P^{-1}\right]},\text{\hspace{1em}(15.29)}$$

where $\hat{\mathbf{s}}$ is a consistent estimate of the positive-definite weight matrix, $\mathbf{S}=$ $E\left[v_t{v}_t^{\prime }(1\left(e_{t+1\mid t}<0\right)-\alpha {)}^2|e_{t+1\mid t}{|}^{2p-2}\right]$ , and the forecasts run from $t=1$ to $t=T$ Elliott, Komunjer, and Timmermann (2005) establish conditions under which, for a given value of $\hslash =1,2$ , a joint test of forecast rationality and the flexible loss function in (15.28) can be conducted based on the test statistic

$$\begin{array}{c}J=\frac{1}{T-1}[{\left(\sum _{t=1}^{T-1}{v}_t[1(e_{t+1\mid t}<0)-{\hat{\alpha }}_T]|e_{t+1\mid t}{|}^{p-1}\right)}^{\prime }{\hat{S}}^{-1}\\ \times \left(\sum _{t=1}^{T-1}{v}_t[1(e_{t+1\mid t}<0)-{\hat{\alpha }}_T]|e_{t+1\mid t}{|}^{p-1}\right)]\sim {\chi }_{d-1}^2.\end{array}\text{\hspace{1em}(15.30)}$$

The test requires that the number of instruments, $d>1$ (the test is a test for overidentification), and assumes that the forecasts are constructed using a recursive estimation scheme. As with the earlier rationality tests, the test has power only for directions implied by the instruments chosen for the test.

Tests based on an assumption of MSE loss arise as a special case when $p=2$ and $\alpha =0.5$ in (15.28). The difference is that if indeed $\alpha =0.5$ , tests based on MSE loss impose this restriction, whereas the test in (15.30) uses a consistent estimate of α which is treated as unknown and hence has lower power than a test that imposes $\alpha =0.5$ . On the flip side, if $\alpha \ne 0.5$ then conventional tests based on MSE loss will asymptotically wrongly reject the null of forecast rationality with probability 1.

In contrast, the test in (15.30) is consistent and avoids this problem, controlling for size if the forecaster’s loss function reflects a different value of the true α. Asymptotically there is no loss from relaxing the assumption that $\alpha =0.5$ , but there is clearly a gain in terms of directing power in the desired direction.

When we apply the approach of Elliott, Komunjer, and Timmermann (2005) to a sample of current-quarter GDP growth forecasts from the Survey of Professional Forecasters, using a constant and the lagged forecast error as instruments, the estimated value for α is 0.35 with an associated J -test statistic from (15.30) of 0.25 which is statistically insignificant. A test of the overidentifying restriction that $\alpha =0.5$ yields a statistic of 8.93 which means that the null of rational forecasts and quadratic error loss is strongly rejected. For current-quarter inflation forecasts the estimated value of α is 0.55 and the associated tests are 12.3 and 13.2, which in both cases reject the null of forecast rationality irrespective of whether α is constrained to equal 0.5. This illustrates how different shapes of the loss function can lead to different inference on whether the forecasts were rational or not.

The test in (15.30) has been applied to a number of forecasting problems, including applications to energy forecasts (Auffhammer, 2007), economic growth (Elliott, Komunjer, and Timmermann, 2008), and EU commission forecasts (Christodoulakis and Mamatzakis, 2009).