2.2.1.6 Binary Loss

2.2.1.6 Binary Loss

When the space of outcomes Y is discrete, the forecast errors typically take on only a small number of possible values. Hence in constructing a loss function for such problems, all that is required is to evaluate each of a small number of possibilities. The simplest case arises when forecasting a binary outcome so that $\mathcal{V}=\{-1,1\}$ or $\mathcal{V}=\{0,1\}$ . In this case there are only four possible pairings of the point forecast and outcome: two where the forecast gives the correct outcome and two errors. If we restrict the loss function to not depend on Z (this case is examined below) and also restrict the problem so that a correct forecast has the same value regardless of the value for Y, then the binary loss function can be written ${\mathsf{as}}^8$

$$L(f,y)=\{\begin{array}{ll}0& \mathrm{if}f=y=0,\\ (1-c)& \mathrm{if}f=0,y=1,\\ c& \mathrm{if}f=1,y=0,\\ 0& \mathrm{if}f=y=1.\end{array}$$

Here we have set the loss from a correct prediction to 0 and normalized the losses from an incorrect forecast to sum to 1 by dividing by their sum; see Schervish (1989), Boyes, Hoffman, and Low (1989), Granger and Pesaran (2000), and Elliott and Lieli (2013).

For (2.18) to be a valid loss function, we require that $0<c<1$ . This ensures that the properties of the loss function listed in (2.3) hold. Notice that the binary loss function can be written as $L(e)$ , since the loss is equal to

$$L(f,y)=c1(e<0)+(1-c)1(e>0).$$

2.2.2 Level- and Forecast-Dependent Loss Functions

Economic loss is mostly assumed to depend on only the forecast error, $e=Y-f.$ This is too restrictive an assumption for situations in which the forecaster’s objective function depends on state variables such as the level of the outcome variable Y. More generally, we can consider loss functions of the form $L(f,y)\ne L(e)$ . The most common level-dependent loss function is the mean absolute percentage error (MAPE), given by

$$L(e,y)=a\left|\frac{e}{y}\right|.$$

Since the forecast and forecast error have the same units as the outcome, the MAPE is a unitless loss function. This is considered to be an advantage when constructing the sample analog of this loss function and employing it to evaluate forecast methods across outcomes measured in different units. If the loss function is well grounded in terms of the actual costs arising from the forecasting problem, dependence on units does not seem to be an important issue—comparisons across different forecasts with different units should be related not through some arbitrary adjustment but instead in a way that trades off the costs associated with the forecast errors for each of the outcomes. This is achieved by the multivariate loss functions examined in the next section.

Scaling the forecast error by the outcome in (2.19) has the effect of weighting forecast errors more heavily when y is near 0 than when y is far from 0. This is difficult to justify in many applications. Moreover, if the predictive density for Y has nontrivial mass at 0, then the expected loss is unlikely to exist, hence invalidating many of the results from decision theory for this case. Nonetheless, MAPE loss remains popular in many practical forecast evaluation experiments.

More generally, level- and forecast-dependent loss functions can be written as $L(f,y)$ but do not reduce to $L(e)$ or $L(e,y)$ . Although loss functions in this class are not particularly common, there are examples of their use. For example, Bregman (1967) suggested loss functions of the form

$$L\left(f,y\right)=\varphi \left(y\right)-\varphi \left(f\right)-{\varphi }^{\prime }\left(f\right)\left(y-f\right),$$

where $\varphi$ is a strictly convex function, so ${\varphi }^{\prime \prime }>0$ . Squared error loss is nested as a special case of 2.20.

Differentiating (2.20) with respect to the forecast, f , we get

$$\begin{array}{c}\frac{\partial L\left(f,y\right)}{\partial f}=-{\varphi }^{\prime }\left(f\right)-{\varphi }^{\prime \prime }\left(f\right)\left(y-f\right)+{\varphi }^{\prime }\left(f\right)\\ =-{\varphi }^{\prime \prime }\left(f\right)\left(y-f\right),\end{array}$$

which generally depends on both y and f . This, along with the assumption that ${\varphi }^{\prime \prime }>0.$ , ensures that the conditional mean is the optimal forecast. Bregman loss is further discussed in Patton (2015).

In an empirical application of level-dependent loss, Patton and Timmermann (2007b) find that the Federal Reserve’s forecasts of output growth fail to be optimal if their loss is restricted to depend only on the forecast error. Rationalizing the Federal Reserve’s forecasts requires not only that overpredictions of output growth are costlier than underpredictions, but also that overpredictions of output are particularly costly during periods of low economic growth. This finding can be justified if the cost of an overly tight monetary policy is particularly high during periods with low economic growth when such a policy may cause or extend a recession.9