正在学习

2.4 SCORING RULES FOR DISTRIBUTION FORECASTS

So far we have focused our discussion on point forecasts, but forecasts of the full distribution of outcomes are increasingly reported. Just as point forecasting requires a loss-based measure of the distance between the forecast $f$ and the outcome $Y,$ distribution forecasts also require a loss function. These are known as scoring rules and reward forecasters for making more accurate predictions, i.e., predictions that are “closer” to the observed outcome get a higher score, where closeness depends on the shape of the scoring rule. Gneiting and Raftery (2007) provide a survey of scoring rules and discuss their properties.

Scoring rules, $S (p, y)$ , are mappings of predictive probability distributions, $P,$ and outcomes, $y,$ to the real line. Suppose a forecaster uses the predictive probability distribution, $p,$ while the probability distribution used to evaluate the “goodness of $fit^{†}$ of $p$ is denoted $p_{0}$ . Then the expected value of $S (p, y)$ under $p_{0}$ is denoted $S (p, p_{0})$ . A scoring rule is called strictly proper if the forecaster’s best probability distribution is $P 0;$ , i.e., $S (p_{0}, p_{0}) \geq S (p, p_{0})$ with equality holding only if $p = p_{0}$ In this situation there will be no incentive for the forecaster to use a probability distribution $p \neq = p_{0}$ since this would reduce the score. The performance of a given candidate probability distribution, $P,$ relative to the optimal rule, can be measured through the so-called divergence function

$d (p, p_{0}) = S (p_{0}, p_{0}) - S (p, p_{0}) . (2.25)$

Notice the similarity to the normalization in equation (2.3a) for loss functions based on point forecasts in (2.3): the divergence function obtains its minimum value of 0 only if $p = p_{0}$ , and otherwise takes a positive value. The forecaster’s objective of maximizing the scoring rule thus translates into minimizing the divergence function.

Several scoring rules have been used in the literature. Many of these have been considered for categorical data limited to discrete outcomes $y = (y_{1}, \dots, y_{m})$ with associated probabilities ${p_{1}, \dots, p_{m}}$ . Denote by $b_{P i}$ the predicted probability that corresponds to the range that includes $γ_{i}$ . The logarithmic score,

$S (p, y_{i}) = lo g (p_{i}), (2.26)$

gives rise to the well-known Kullback–Leibler divergence measure,

$d (p, p_{0}) = j = 1 \sum m p_{0 j} lo g (p_{0 j} / p_{j}) . (2.27)$

Similarly, the quadratic or Brier score,

$S (p, y_{i}) = 2 p_{i} - j = 1 \sum m p_{j}^{2} - 1, (2.28)$

generates the squared divergence

$d (p, p_{0}) = j = 1 \sum m (p_{j} - p_{0 j})^{2} . (2.29)$

For density forecasts defined over continuous outcomes the logarithmic and quadratic scores take the form

$lo g S (p, y) = lo g p (y), S (p, y) = 2 p (y) - (\int p (y)^{2} μ (d y))^{1/2},$

where $μ (\cdot)$ is the probability measure associated with the outcome, y. Both are proper scoring rules. By contrast, the linear score, $S (p, y) = p (y)$ , can be shown not to be a proper scoring rule; see Gneiting and Raftery (2007).

Which scoring rule to use in a given situation depends, of course, on the underlying objectives for the problem at hand and the choice should most closely resemble the costs involved in the decision problem. To illustrate this point, we next provide an example from the semiconductor supply chain.

Example 2.4.1 (Loss function for semiconductors). Cohen et al. (2003) construct an economically motivated loss or cost function for a semiconductor equipment supply chain. Supply firms are assumed to hold soft orders from clients which may either be canceled (with probability π ) or get finalized (with probability $1 - π)$ at some later date, $y_{N},$ when the final information arrives. Given such orders, firms attempt to optimally determine the timing of the production start, $y_{π .}$ , where $y_{N} > y_{π}$ due to a production lead-time delay. If an order is canceled, the supplier incurs a cancelation cost, c, per unit of time. Let y denote the final delivery date in excess of the production lead time. If this exceeds the production date, the supplier will incur holding (inventory) costs, h, per unit of time. Conversely, if the production start date, $y_{π}$ , exceeds $y,$ the company will not be able to meet the requested delivery date and so incurs a delay cost of $\dot{g}$ per unit of time. Cohen et al. (2003) assume that suppliers choose the production date, $y_{π}$ , so as to minimize the expected total cost

$E [L (y_{π}, y, y_{N})] = π \times c \int_{y_{π}}^{\infty} (y_{N} - y_{π}) d P_{N} (y_{N}) + (1 - π) [h \int_{y_{π}}^{\infty} (y - y_{π}) d P_{y} (y) + g \int_{- \infty}^{y_{π}} (y_{π} - y) d P_{y} (y)],$

where $P_{y} (y)$ and $P_{N} (y_{N})$ are the cumulative distribution functions of y and $y_{N},$ , respectively. Provided that this expression is convex in $y_{π}$ , the cost-minimizing production time, $y_{π}^{*}$ , can be shown to solve the first-order condition

$π \times c \times P_{N} (γ_{π}^{*}) + (1 - π) (g + h) P_{y} (γ_{π}^{*}) = π \times c + (1 - π) h, (2.30)$

and so implicitly depends on the cancelation probability, cancelation costs, inventory and delay costs, in addition to the predictive distributions for the finalization and final delivery dates. Cohen et al. (2003) use an exponential distribution to model the arrival time of the final order, $P_{N} .$ , and a Weibull distribution to model the distribution of the final delivery date, $P_{Y}$ . To estimate the model parameters and predict the lead time, the authors use data on soft orders, final orders, and order lead time. Empirical estimates suggest that $\overset{g}{^} = 1.0, \hat{h} = 3.0, \overset{c}{^} = 2.1$ , indicating that holding costs are three times greater than delay costs, while cancelation costs are twice as high as the delay costs. This in turn helps the manufacturer decide on the optimal start date for production, $y_{π}^{*}$

练习题

Which of the following best describes a scoring rule $S (p, y)$ ?

A. A function that maps predictive probability distributions and outcomes to a non-negative integer

B. A function that maps predictive probability distributions and outcomes to the real line

C. A function that maps only outcomes to the real line

D. A function that maps only predictive probability distributions to the real line

What is the condition for a scoring rule to be strictly proper?

S (p_{0}, p_{0}) \leq S (p, p_{0})

with equality holding for all

p

S (p_{0}, p_{0}) \geq S (p, p_{0})

with equality holding only if

p = p_{0}

S (p_{0}, p_{0}) = S (p, p_{0})

for all

p

S (p_{0}, p_{0}) \neq = S (p, p_{0})

for all

p \neq = p_{0}

What is the divergence function $d (p, p_{0})$ defined as?

d (p, p_{0}) = S (p, p_{0}) - S (p_{0}, p_{0})

d (p, p_{0}) = S (p_{0}, p_{0}) + S (p, p_{0})

d (p, p_{0}) = S (p_{0}, p_{0}) - S (p, p_{0})

d (p, p_{0}) = S (p, p_{0}) \times S (p_{0}, p_{0})

The logarithmic score $S (p, y_{i}) = lo g (p_{i})$ is a proper scoring rule.

The linear score $S (p, y) = p (y)$ is a proper scoring rule.

The quadratic or Brier score is given by $S (p, y_{i}) = 2 p_{i} - \sum_{j = 1}^{m} p_{j}^{2} - 1$ . The squared divergence generated by this score is $d (p, p_{0}) = \sum_{j = 1}^{m} (p_{j} - p_{0 j})^{2}$ . The missing term in the Brier score formula is ___.

For density forecasts defined over continuous outcomes, the logarithmic score takes the form $lo g S (p, y) = lo g p (y)$ . The quadratic score takes the form $S (p, y) = 2 p (y) - (\int p (y)^{2} μ (d y))^{1/2}$ . The missing operator in the quadratic score formula is ___.

Explain the purpose of the divergence function $d (p, p_{0})$ .

Which of the following are proper scoring rules? (Select all that apply)

A. Logarithmic score

B. Quadratic or Brier score

C. Linear score

D. Squared error score

What factors should be considered when choosing a scoring rule for a given situation?

Which of the following is the formula for the Kullback–Leibler divergence measure generated by the logarithmic score?

d (p, p_{0}) = \sum_{j = 1}^{m} p_{0 j} lo g (\frac{p _{0 j}}{p _{j}})

d (p, p_{0}) = \sum_{j = 1}^{m} p_{j} lo g (\frac{p _{j}}{p _{0 j}})

d (p, p_{0}) = \sum_{j = 1}^{m} lo g (\frac{p _{0 j}}{p _{j}})

d (p, p_{0}) = \sum_{j = 1}^{m} p_{0 j} lo g (p_{0 j})

Which of the following statements about the divergence function are true? (Select all that apply)

A. It obtains its minimum value of 0 only if

p = p_{0}

B. It takes a positive value if

p \neq = p_{0}

C. It is defined as

d (p, p_{0}) = S (p, p_{0}) - S (p_{0}, p_{0})

D. It is used to measure the performance of the forecast distribution relative to the true distribution.

Which of the following statements correctly describes the relationship between a strictly proper scoring rule and the divergence function?

A. The divergence function is minimized when the scoring rule is not strictly proper.

B. The divergence function

d (p, p_{0})

is minimized when

p = p_{0}

for a strictly proper scoring rule.

C. The divergence function is independent of the scoring rule's properness.

D. The divergence function increases as

p

approaches

p_{0}

for a strictly proper scoring rule.

Which of the following are proper scoring rules for discrete outcomes?

A. Logarithmic score:

S (p, y_{i}) = lo g (p_{i})

B. Quadratic or Brier score:

S (p, y_{i}) = 2 p_{i} - \sum_{j = 1}^{m} p_{j}^{2} - 1

C. Linear score:

S (p, y) = p (y)

D. Squared error loss:

L (f, y) = (y - f)^{2}

The divergence function for a strictly proper scoring rule is always non-negative and equals zero only when $p = p_{0}$ .

登录后解锁笔记、知识点解析、AI 问答

立即登录

2.4 SCORING RULES FOR DISTRIBUTION FORECASTS