18.2.2 Resolution

18.2.2 Resolution

Resolution asks the following question: Is the conditional distribution different from the unconditional distribution? We saw in the example that calibration is unable to distinguish between correct specification of a conditional density and correct specification of the unconditional density. Resolution attempts to resolve this issue. Resolution is usually examined through the difference in the conditional and unconditional means, $E\left[Y|\hat{p}\right]-E\left[Y\right]$ , thus making it a function of the forecast, $\hat{\mathbf{p}}$ . Average resolution relies on taking the average of this squared difference over different values of $\hat{\mathbf{p}}$ . High resolution for a binary outcome requires that the density forecast can differentiate the chance that $Y=1$ in different situations. Resolution equals 0 if the density forecast is equivalent to the unconditional distribution for the outcome.

Following the same path as for calibration, we can construct binary outcomes from any continuous distribution by focusing on some event, A. For any such event, $A,$ we can then consider the difference between $\begin{array}{r}P\left[y_{t+1}\in A|{\int }_A{\hat{p}}_{t+1}(y)dy=p\right]\end{array}$ and $P[y_{t+1}\in A]$

Example 18.2.3 (Comparing Gaussian density forecasts, continued). When $A=$ $\{y_{t+1}>0\}$ , we know that for the second forecast, $\begin{array}{r}{\int }_A{p}_{Y2t+1}(y)dy=\Phi ({\mu }_t)\end{array}$ . Unconditionally $P\left[y_{t+1}\in A\right]=0.5$ . The difference between the conditional and unconditional forecasts of A is then $\Phi ({\mu }_t)-0.5$ . Clearly the conditioning information (knowledge of ${\mu }_t)$ results in different forecasts, which is the idea behind resolution. This example also shows that squaring the difference makes sense since the difference can be both positive $(if{\mu }_t<0)$ and negative $(if{\mu }_t>0)$ .

Once again, “bins” or ranges of values for $\mathbf{p}$ can be used to test resolution. The use of bins allows for multiple observations for each comparison. Using the same definitions of terms as in (18.11), resolution with m bins is measured as

$$\frac{1}{T}\sum _{i=1}^m{n}_i({\bar{y}}_i-\bar{p}{)}^2,\text{\hspace{1em}(18.13)}$$

where $\bar{p}$ is the (unconditional) sample average of the event that $y_{t+h}\in A$ . As for calibration, nonparametric procedures can be used to estimate the resolution as a function of $p.$ .

18.2.3 Sharpness

Sharpness is a property of the distributional forecast and so does not depend on the outcome $y_{t+1}$ . This property gets at whether the distributional forecasts discriminate relatively clearly between different outcomes. For binary outcomes, forecasts of $y_{t+1}=1$ with a high probability such as 80% would be sharper than forecasts assigning a 55% chance to this event. When directed at continuously distributed outcomes, the forecasts are considered “sharp” if they assign zero probability to a large set of all possible outcomes, and a high probability to those outcomes predicted to happen. Another way to state this is that $\mathrm{if},$ for any probability $P,$ there exists an interval A on the support of $y_{t+1}$ , where ${\int }_A{p}_{t+1}(y)dy=p$ , then the length of the interval A is short.

Example 18.2.4 (Comparing Gaussian density forecasts, continued). Let $p=0.9$ For the first forecast, $\overline{{\int }_{A_1}{p}_{Y1t+1}(y)dy}=0.9:for::A_1=\{-2.33,2.33\}$ . For the second forecast, $\begin{array}{r}{\int }_{A_2}:p_{Y2t+1}(y)d\dot{y}=0.9:for::A_{2t}=\{{\mu }_t-1.645,{\mu }_t+1.645\}\end{array}$ . Clearly the second forecast is sharper than the first.

Since one can always present very sharp forecasts by assigning a probability of 1 to a single point, the sharpness property must be (and is in practice) used in concert with the other properties of a desirable forecast. Mitchell and Wallis (2011) raise other issues with the use of sharpness in practice.