Table 1.
Detailed summary of forecast evaluation for Germany (based on ECDC data). C0.5 and C0.95 denote coverage rates of the 50% and 95% prediction intervals; AE and WIS stand for the mean absolute error and mean weighted interval score.
| Germany, cases | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1-week ahead incident | 2-week ahead incident | 1-week ahead cumulative | 2-week ahead cumulative | |||||||||||||
| Model | AE | WIS | C0.5 | C0.95 | AE | WIS | C0.5 | C0.95 | AE | WIS | C0.5 | C0.95 | AE | WIS | C0.5 | C0.95 |
| epiforecasts-EpiExpert | 12,333 | 8,781 | 5/10 | 7/10 | 30,329 | 22,157 | 2/9 | 3/9 | 12,334 | 8,781 | 5/10 | 7/10 | 42,667 | 30,669 | 2/9 | 4/9 |
| epiforecasts-EpiNow2 | 11,171 | 7,932 | 5/10 | 7/10 | 37,338 | 27,293 | 4/9 | 7/9 | 11,171 | 7,932 | 5/10 | 7/10 | 47,738 | 34,253 | 4/9 | 6/9 |
| FIAS_FZJ-Epi1Ger | 7,798 | 5,709 | 7/10 | 9/10 | 29,190 | 21,058 | 5/9 | 7/9 | 17,255 | 11,264 | 3/10 | 7/10 | 38,925 | 29,937 | 5/9 | 6/9 |
| ITWW-county_repro | 34,425 | 28,906 | 0/10 | 2/10 | 64,378 | 53,136 | 0/9 | 2/9 | 34,077 | 28,558 | 0/10 | 2/10 | 101,184 | 84,276 | 0/9 | 2/9 |
| LANL-GrowthRate | 38,970a | 23,379a | 5/7 | 7/7 | 77,438a | 42,294a | 2/6 | 6/6 | 39,042 | 26,794 | 5/10 | 7/10 | 116,494 | 78,224 | 3/9 | 5/9 |
| LeipzigIMISE-SECIR | 20,019 | 2/5 | 3/5 | 51,115 | 0/4 | 1/4 | 35,901 | 31,690 | 1/10 | 1/10 | 93,111 | 83,670 | 1/9 | 2/9 | ||
| MIT_CovidAnalytics-DELPHI | 41,313a | 29,004a | 1/8 | 4/8 | 78,872a | 61,447a | 2/7 | 2/7 | ||||||||
| SDSC_ISG-TrendModel | 10,963 | 10,963 | ||||||||||||||
| UCLA-SuEIR | 25,012 | 47,747 | 25,012 | 69,800 | ||||||||||||
| USC-SIkJalpha | 20,028 | 1/1 | 1/1 | 30,891 | 21,567 | 1/1 | 1/1 | 49,640 | ||||||||
| KIT-baseline | 18,475 | 12,998 | 5/10 | 9/10 | 32,690 | 25,543 | 3/9 | 6/9 | 18,475 | 12,998 | 5/10 | 9/10 | 47,472 | 37,155 | 3/9 | 5/9 |
| KIT-extrapolation_baseline | 12,016 | 10,522 | 7/10 | 10/10 | 36,498 | 26,195 | 6/9 | 7/9 | 12,016 | 10,522 | 7/10 | 10/10 | 47,145 | 35,142 | 7/9 | 7/9 |
| KIT-time_series_baseline | 15,383 | 11,014 | 5/10 | 9/10 | 44,481 | 28,625 | 4/9 | 8/9 | 15,383 | 11,014 | 5/10 | 9/10 | 61,489 | 39,244 | 4/9 | 8/9 |
| KITCOVIDhub-inverse_wis_ensemble | 14,017 | 9,358 | 5/10 | 9/10 | 42,063 | 27,993 | 2/9 | 5/9 | 13,464 | 9,265 | 6/10 | 9/10 | 52,972 | 35,194 | 2/9 | 8/9 |
| KITCOVIDhub-mean_ensemble | 16,649 | 10,677 | 4/10 | 8/10 | 42,214 | 27,290 | 1/9 | 6/9 | 15,771 | 10,630 | 4/10 | 8/10 | 57,125 | 37,356 | 1/9 | 6/9 |
| KITCOVIDhub-median_ensemble | 11,534 | 8,094 | 5/10 | 9/10 | 37,620 | 25,017 | 3/9 | 7/9 | 12,877 | 9,243 | 6/10 | 7/10 | 49,438 | 34,461 | 2/9 | 6/9 |
| Germany, deaths | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1-week ahead incident | 2-week ahead incident | 1-week ahead cumulative | 2-week ahead cumulative | |||||||||||||
| Model | AE | WIS | C0.5 | C0.95 | AE | WIS | C0.5 | C0.95 | AE | WIS | C0.5 | C0.95 | AE | WIS | C0.5 | C0.95 |
| epiforecasts-EpiExpert | 187 | 131 | 5/10 | 7/10 | 333 | 234 | 3/9 | 6/9 | 187 | 131 | 5/10 | 7/10 | 442 | 295 | 3/9 | 7/9 |
| epiforecasts-EpiNow2 | 180 | 120 | 5/10 | 7/10 | 376 | 235 | 3/9 | 7/9 | 180 | 120 | 5/10 | 7/10 | 539 | 336 | 4/9 | 7/9 |
| FIAS_FZJ-Epi1Ger | 256 | 223 | 3/10 | 4/10 | 525 | 433 | 2/9 | 3/9 | 215 | 175 | 0/10 | 4/10 | 684 | 564 | 1/9 | 4/9 |
| Imperial-ensemble2 | 254 | 195 | 5/10 | 5/10 | 252 | 192 | 5/10 | 5/10 | ||||||||
| ITWW-county_repro | 371 | 355 | 1/10 | 2/10 | 537 | 483 | 1/9 | 2/9 | 370 | 354 | 1/10 | 2/10 | 824 | 757 | 2/9 | 2/9 |
| LANL-GrowthRate | 195a | 128a | 3/7 | 7/7 | 457a | 313a | 2/6 | 5/6 | 189 | 134 | 3/10 | 7/10 | 560 | 441 | 3/9 | 5/9 |
| LeipzigIMISE-SECIR | 621 | 0/5 | 1/5 | 768 | 1/4 | 1/4 | 1,167 | 991 | 0/10 | 1/10 | 2,184 | 1,799 | 0/9 | 1/9 | ||
| MIT_CovidAnalytics-DELPHI | 474a | 357a | 1/8 | 3/8 | 403a | 306a | 0/7 | 5/7 | 449a | 331a | 1/8 | 3/8 | 639a | 525a | 0/7 | 3/7 |
| SDSC_ISG-TrendModel | 357 | 357 | ||||||||||||||
| UCLA-SuEIR | 456 | 827 | 456 | 1,177 | ||||||||||||
| USC-SIkJalpha | 489 | 0/1 | 0/1 | 600 | 500 | 0/1 | 0/1 | 963 | ||||||||
| KIT-baseline | 479 | 263 | 2/10 | 9/10 | 835 | 510 | 0/9 | 5/9 | 479 | 263 | 2/10 | 9/10 | 1,155 | 707 | 0/9 | 5/9 |
| KIT-extrapolation_baseline | 202 | 134 | 7/10 | 9/10 | 383 | 246 | 5/9 | 8/9 | 202 | 134 | 7/10 | 9/10 | 493 | 330 | 5/9 | 8/9 |
| KIT-time_series_baseline | 238 | 190 | 6/10 | 9/10 | 624 | 415 | 4/9 | 8/9 | 238 | 190 | 6/10 | 9/10 | 866 | 577 | 4/9 | 8/9 |
| KITCOVIDhub-inverse_wis_ensemble | 180 | 114 | 4/10 | 9/10 | 255 | 147 | 2/9 | 8/9 | 174 | 108 | 5/10 | 9/10 | 370 | 224 | 3/9 | 8/9 |
| KITCOVIDhub-mean_ensemble | 204 | 138 | 3/10 | 9/10 | 298 | 174 | 2/9 | 8/9 | 216 | 147 | 3/10 | 9/10 | 441 | 262 | 2/9 | 8/9 |
| KITCOVIDhub-median_ensemble | 200 | 135 | 4/10 | 8/10 | 334 | 216 | 3/9 | 7/9 | 202 | 133 | 4/10 | 8/10 | 440 | 270 | 3/9 | 8/9 |
aEntries where scores were imputed for at least 1 week. Weighted interval scores and absolute errors were imputed with the worst (largest) score achieved by any other forecast for the respective target and week. Models marked thus received a pessimistic assessment of their performance. If a model covered less than two-thirds of the evaluation period, results are omitted.