Fig. 5. Model performance metrics overall and stratified by week of projection and epidemic phase.

The performance metrics are a the percentage of weeks for which the 95% forecast interval contained the observed incidence, b relative mean absolute error, c bias, and d sharpness. In each panel, the bounds of the box represent the 25th and 75th percentiles and the line corresponds to the median of the distribution of the respective metric. The whiskers extend to 1.5 * Inter-quartile range in both directions. In forecasting ahead, we assumed transmissibility to be constant over the forecast horizon. If the 97.5th percentile of the R estimate used for forecasting was less than 1, we defined the epidemic to be in the declining phase during this period. Similarly, if the 2.5th percentile of R was greater than 1, we defined the epidemic to be in a growing phase. The phase was set to stable where the 95% Credible Interval of the R estimates contained 1. See Methods for definitions of each metric.