14.5.1 Classical Approach to Density Combination

14.5.1 Classical Approach to Density Combination

One approach combines densities by means of some loss function. Unlike in the point forecasting problem, we do not directly observe the outcome density—we observe only a draw from this—and so cannot directly choose the weights to minimize the loss between this object and the combined density. One way around this is to use a loss function that does not require knowing the density for the outcome variable; this leads directly to Kullback–Leibler loss.

The Kullback–Leibler (KL) loss for a linear combination of densities ${\sum }_{i=1}^m{\omega }_i{p}_i(y)$ relative to some unknown true density $p(y)$ is given by

$$\begin{array}{l}\mathrm{KL}=\int p(y)\ln \left(\frac{p(y)}{{\sum }_{i=1}^m{\omega }_i{p}_i(y)}\right)dy\\ =\int p(y)\ln (p(y))dy-\int p(y)\ln \left(\sum _{i=1}^m{\omega }_i{p}_i(y)\right)dy\\ =C-E\ln \left(\sum _{i=1}^m{\omega }_i{p}_i(y)\right),\end{array}$$

where C is constant across all choices of the weights, ${\omega }_i$ . Hence, minimizing the KL distance is the same as maximizing the log score in expectation.

The use of the log score to evaluate the density combination is one of the more popular approaches in the literature. Geweke and Amisano (2011) use this approach to combine GARCH, hierarchical Markov normal mixture, and stochastic volatility models for predicting the density of daily stock returns. See also Hall and Mitchell (2007).

Under the log score criterion, estimation of the combination weights becomes equivalent to maximizing the log-likelihood. Given a sequence of observed outcomes for y, $\{{\mathbf{y}}_t{\}}_{t=1}^T$ , the sample analog is to maximize

$${\hat{\omega }}_T=\arg \underset{{\omega }_T}{\max }(T-1{)}^{-1}\sum _{t=1}^{T-1}\ln \left({\omega }_{iT}{p}_{it+1|t}(y_{t+1})\right),$$

subject to

$${\omega }_{iT}\ge 0,\sum _{i=1}^m{\omega }_{iT}=1,\mathrm{for}\mathrm{all}i.$$

Estimation of these weights can be undertaken using a constrained optimization program that treats $p_{it}(y)$ —the density evaluated at the outcome—as the data. In the case with $m=2$ , they note that if one model nests another and the smaller model is correctly specified, in the limit the two densities will be identical. Thus, in the limit the second-order condition is 0 for all sets of weights. In practice this might not be a problem since in this case the two densities are the same and so any weighting will provide the same results for the combined density.

The log score (or KL distance) is not the only candidate scoring rule that could be employed. Any proper scoring rule can be employed. Other scoring rules that have been suggested in the literature include quadratic scoring (QS) and the continuous ranked probabilistic score (CRPS). See Gneiting and Raftery (2007) for further discussion.