18.2.1 Calibration

18.2.1 Calibration

Calibration requires that if a density forecast assigns a certain probability to an event, then the event should occur with the stated probability over successive observations. This idea first arose in predictions of binary outcomes. Consider forecasting whether it will rain or shine, so a density forecast is just the probability that it will rain. If the weather forecaster predicts a 70% chance of rain, then over all the days for which this is the density forecast, we should see rain occurring 70% of the time. More generally, for a well-calibrated forecast, $E\left[Y|p(z)\right]=p(z)$

This idea can be extended by transforming any event for the outcome to a binary variable. For any such event, A, if the associated density forecast $\begin{array}{r}{\int }_A{p}_Y(y|z)(y)dy=p,\end{array}$ calibration requires that $P[y_{t+1}\in A]$ is indeed equal to $P,$ conditional on the same information. For example, let A be the set of outcomes for which we predict an increase in the value of the outcome, $y_{t+1}>y_t$ . Then, for all periods where the predicted probability that $y_{t+1}-y_t>0$ is, say, $1/4.$ , we should see an increase in $\mathbf{y}$ exactly 25% of the time. For a well-calibrated forecast this should hold for all values of $\mathbf{p}$ and any event, A. Clearly, this only holds if the forecast distribution is precisely the distribution of the outcomes.

Example 18.2.1 (Comparing Gaussian density forecasts). Following Gneiting and Raftery (2007), consider two possible density forecasts for $y_{t+1}$ . The first is the true unconditional density, $p_{Y1t+1}\sim \mathrm{N}(0,2)$ . The second is a correctly specified conditional density, $p_{Y2t+1t}\sim \mathrm{N}({\mu }_t,1)$ , where the marginal distribution of ${\mu }_t$ is drawn from a standard Gaussian distribution, $i.e.,{\mu }_t\sim \mathrm{N}(0,1)$ . Let $A=\{y_{t+1}>0\}$ · Then ${\mathcal{P}}_{Y1t+1}$ implies a probability forecast ${\int }_A{p}_{Y1t+1}(y)dy=0.5$ in every period. Correct calibration requires that $y_{t+1}>0$ holds half the time on average. For the second model, $\begin{array}{r}{\int }_A{p}_{Y2t+1}(y)dy=\Phi ({\mu }_t)\end{array}$ , which varies over time. Hence, calibration requires that $\begin{array}{r}E\left[\widetilde{{\gamma }_{t+1}}>0|\Phi ({\mu }_t)=p\right]=E\left[{\gamma }_{t+1}>0|{\mu }_t={\Phi }^{-1}(p)\right]=p.\end{array}$ for all periods where $\Phi ({\mu }_t)=\mathbf{p}$ . This example has sufficient variation in ${\mu }_t$ so the condition can be checked for many different values of $p.$

Most attempts to examine calibration lead to informal rather than formal hypothesis tests. An issue that arises is that $\begin{array}{r}P\left[y_{t+1}\in A|{\int }_A{p}_{t+1}(y)dy=p\right]\end{array}$ is a function of $p\in [0,1]$ but often a given sample will have very few observations for any specific value of $p.$ . In the binary forecasting literature this has often been overcome by examining the property for a range of values for $\mathbf{p}$ to create a histogram as in the Murphy decomposition in (18.10). Using this approach, calibration can be measured as

$$\frac{1}{T}\sum _{i=1}^m{n}_i({\bar{y}}_i-p_i{)}^2,\text{\hspace{1em}(18.11)}$$

where a value of 0 means perfect calibration. Formal tests based on this statistic have a nonpivotal distribution. To get around this, Seillier-Moiseiwitsch and Dawid (1993) suggest the following standardized statistic:

$$\sum _{i=1}^m\frac{n_i({\bar{y}}_i-p_i{)}^2}{p_i(1-p_i)}.\text{\hspace{1em}(18.12)}$$

Using a martingale difference sequence central limit theorem, these authors show that the statistic in (18.12) is approximately ${\chi }_m^2$ . Their result does not account for estimation error in the construction of the distributional forecast, however.

For forecasts converted into binary predictions, Galbraith and van Norden (2011) suggest using kernel methods. Standard kernel approaches to estimate the conditional mean are appropriate for this problem; see Pagan and Ullah (1999) for a review of the general problem. For general continuously distributed outcomes, $y_{t+1}$ , a large number of choices exist for A. Calibration means that the result should hold for any such choice, although it is not feasible to try out every single choice for A.

Calibration suffers from the shortcoming that it ignores information in the conditioning variables. A forecast can be well calibrated even though the density forecast is not the best available one.

Example 18.2.2 (Comparing Gaussian density forecasts, continued). For $A=$ $\{y_{t+1}>0\}$ , the first density forecast, $\begin{array}{r}{p}_{Y1t+1},\end{array}$ , is well calibrated since $P\left[y_{t+1}>0\right]=0.5$ which is the predicted probability each period. The second density forecast, pY2t+1, is also well calibrated. Whenever ${\mu }_t=\mu$ , then $y_{t+1}\sim \mathrm{N}(\mu ,1)$ and so $P\left[y_{t+1}>0\right]=$ $\Phi (\mu )$ , resulting in a well-calibrated model.

Calibration thus does not measure whether the density forecast is good in the sense that it assigns high probability to events that are likely to happen. Rather, it can be used to measure whether the density forecast makes systematic mistakes in some situations. For this reason, calibration is often termed “reliability”; a well-calibrated forecast is reliable in the sense that it avoids predicting events that do not happen, even if it can be a poor indicator of what will actually happen. In this sense calibration is really more a tool for specifying a good distribution rather than a way to persuade a user that the density forecast is optimal.