On the multibin logarithmic score used in the FluSight competitions

The FluSight challenges (1) represent an outstanding collaborative effort and have “pioneered infectious disease forecasting in a formal way” (ref. 2, p. 2803). However, I wish to initiate a discussion about the employed evaluation measure.

The competitions feature discrete or discretized targets related to the US influenza season. E.g., for the peak timing $Y$, a forecast distribution $F$ consists of probabilities $p_1,\dots ,p_T$ for the $T=33$ wk of the season. Such forecasts can be evaluated using the log score (3, 4)

$$\text{logS}\left(F,y_{\text{obs}}\right)=\text{log}(p_{y_{\text{obs}}}),$$

where $y_{\text{obs}}$ is the observed value. This score is strictly proper; i.e., its expectation is uniquely maximized by the true distribution of $Y$. In the FluSight competitions the logS is applied in a multibin version,

$$\text{MBlogS}\left(F,y_{\text{obs}}\right)=\text{log}\left(\sum _{i=-d}^d{p}_{y_{\text{obs}}+i}\right),$$

to measure “accuracy of practical significance” (ref. 1, p. 3153). Depending on the target, $d$ is either 1 or 5. Following the competitions, this score has become widely used (5–10), even though as also mentioned in ref. 1, it is improper. This may be problematic as improper scores incentivize dishonest forecasts. Assume $T>2d$ and

$$p_1=···=p_d=p_{T-d+1}=···=p_T=0,$$ [1]

i.e., probability 0 for the $2d$ extreme categories. Now define a “blurred” distribution $\tilde{F}$ with

$${\tilde{p}}_t=\frac{{\sum }_{i=-d}^d{p}_{t+i}}{2d+1},t=1,\dots ,T,$$ [2]

where $p_t=0$ for $t<1$ and $t>T$ and Eq. 1 ensures ${\sum }_{t=1}^T{\tilde{p}}_t=1$. This implies

$$\text{MBlogS}\left(F,y_{\text{obs}}\right)=\text{logS}(\tilde{F},y_{\text{obs}})+\text{log}\left(2d+1\right);$$

i.e., the MBlogS is essentially the logS applied to a blurred version of $F$. To optimize the expected MBlogS under their true belief $F$, forecasters should therefore not report $F$, but a sharper forecast $G$ so that the blurred version $\tilde{G}$ (with ${\tilde{p}}_{G,1},\dots ,{\tilde{p}}_{G,T}$ derived from $p_{G,1},\dots ,p_{G,T}$ as in Eq. 2) is close or equal to $F$. This follows from the propriety of the logS. An optimal $G$ is found by maximizing ${\sum }_{t=1}^T{p}_t\cdot \text{log}\left({\tilde{p}}_{G,t}\right)$ with respect to $p_{G,1},\dots ,p_{G,T}$.

This optimal $G$ can differ considerably from the original $F$, as Fig. 1 shows for forecasts of the 2016 to 2017 national-level peak timing by the Los Alamos National Laboratory (LANL) team (9) (downloaded from https://github.com/FluSightNetwork/cdc-flusight-ensemble/). The optimized $G$s (with $d=1$) often have their mode shifted by 1 wk and tend to be multimodal, even for unimodal $F$. Averaged over the 2016 to 2017 season they yield improved MBlogS for the peak timing (−0.434 vs. −0.484). This illustrates that the MBlogS may be gamed, even though we strongly doubt participants have tried to. The logS, like any other proper score, could avoid such pitfalls.

Fig. 1.

Forecasts $F$ for the peak week, submitted by the LANL team in weeks 6 to 7, 2017, and optimized versions $G$. Diamonds mark the observed peak week. Expected scores are computed under $F$.