2.1.4 Loss Functions Not Based on Expected Loss

2.1.4 Loss Functions Not Based on Expected Loss

So far we have characterized the loss function $L(f,Y)$ for a univariate outcome, and defined its properties with reference to a “one-shot” problem. This makes sense when forecasting is placed in a decision-theoretic or utility-maximization context. This approach to forecasting is internally consistent, from initially setting up the problem to defining the expected loss and conducting model estimation and forecast evaluation.

Some loss functions that have been used in practice are based directly on sample statistics without relating the sample loss to a population loss function. In cases where such a population loss function exists and satisfies reasonable properties, this does not cause any problems. Basing the loss function directly on a sample of losses can, however, sometimes yield a loss function that does not make sense in population or for fully specified decision problems. Loss functions that do not map back to decision problems often have poor and unintended properties. We consider one such example below.

Example 2.1.2 (Kuipers score for binary outcome). Let $f=\{1,-1\}$ be a forecast of the binary variable $y=\{1,-1\}$ and let $n_{j,k},j,k\in \{-1,1\}$ be the number of observations for which the forecast equals j and the outcome equals k. The Kuipers score is given by

$$\frac{n_{1,1}}{n_{1,1}+n_{-1,1}}-\frac{n_{1,-1}}{n_{1,-1}+n_{-1,-1}}.\text{\hspace{1em}(2.9)}$$

This is the positive hit rate, i.e., the proportion of times where $\gamma =1$ is correctly predicted less the “false positive rate,” i.e., the proportion of times where $\gamma =1$ is wrongly predicted. This can equivalently be thought of as

$$\mathrm{KuS}=\frac{n_{1,1}}{n_{1,1}+n_{-1,1}}+\frac{n_{-1,-1}}{n_{1,-1}+n_{-1,-1}}-1,\text{\hspace{1em}(2.10)}$$

which is the hit rate for $y=1$ plus the hit rate for $y=-1$ minus a centering constant of 1. The Kuipers score is positive if the sum of the positive and negative hit rates exceeds 1. For a sample with a single observation, this definition makes no sense, as one of the denominators in (2.10) is 0: either $n_{1,1}+n_{-1,1}=0or{n}_{1,-1}+n_{-1,-1}=0.$

For a single observation, this sample statistic does not follow from any obvious loss function. The first term in $(2.10)$ is the sample analog of $P[f=1|Y=1]$ and the second is the sample analog of $P\left[f=-1|Y=-1\right]$ . However, they do not combine to a loss function with this sample analog. This failure to embed the loss function into the expected loss framework results in odd properties for the objective. For example, the definition of KuS in (2.9) implies that the marginal value of an extra ${}^{\infty }hit{,}^{†}i.e.$ , a correct call, depends on the sample proportion of hits. To see this, consider the improvement in KuS from adding a single successfully predicted observation $y=1,f=1$

The resulting improvement in the hit rate is

$$\begin{array}{r}\Delta \mathrm{KuS}=\frac{n_{1,1}+1}{n_{1,1}+1+n_{-1,1}}-\frac{n_{1,1}}{n_{1,1}+n_{-1,1}}\\ =\frac{n_{-1,1}}{\left(n_{1,1}+n_{-1,1}\right)\left(n_{1,1}+1+n_{-1,1}\right)}.\end{array}$$

Thus the marginal value of a correct call depends on the total number of observations and the proportion of missed hits prior to the new observation. The Kuipers score’s poor properties arise from the lack of justification of its setup for a population problem.

2.2 SPECIFIC LOSS FUNCTIONS

We next review various families of loss functions that have been suggested in the forecasting literature. The vast majority of empirical work on forecasting assumes that the loss function depends only on the forecast error, $e=Y-f,$ , i.e., the difference between the outcome and the forecast. In this case we can write $L(f,Y,Z)=L(e)$ In general, loss functions can be more complicated functions of the outcome and forecast and take the form $L(f,Y)$ or $L(f,Y,Z)$

2.2.1 Loss That Depends Only on Forecast Errors

The most commonly used loss functions, including squared error loss and absolute error loss, depend only on the forecast error. For such loss functions, $L(f,Y,Z)=$ L (e), so the loss function takes a particularly simple form.

2.2.1.1 Squared Error Loss

By far the most popular loss function in empirical studies is squared error loss, also known as quadratic or mean squared error (MSE) loss:

$$L\left(e\right)=a{e}^2,a>0.\text{\hspace{1em}(2.11)}$$

This loss function clearly satisfies the three Granger properties listed in ( 2.3). When viewed as a family of loss functions—corresponding to different values of the scalar a—squared error loss forms a homogeneous class.7 It is symmetric, bowl shaped, and differentiable everywhere and penalizes large forecast errors at an increasing rate due to its convexity in |e|. The loss function is not bounded from above. Large forecast errors or “outliers” are thus very costly under this loss function.