Table 1.
Comparison of different methods for regional %ILI estimation.
| Whole period | ‘09-’15 | H1N1 | ‘10-’11 | ‘11-’12 | ‘12-’13 | ‘13-’14 | ‘14-’15 | ‘15-’16 | ‘16-’17 | ‘17-’18 | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| MSE | |||||||||||
| ARGO2 | 0.149 | 0.165 | 0.628 | 0.121 | 0.047 | 0.240 | 0.101 | 0.248 | 0.059 | 0.157 | 0.278 |
| VAR | 0.212 | 0.215 | 0.832 | 0.148 | 0.066 | 0.285 | 0.164 | 0.334 | 0.109 | 0.328 | 0.505 |
| GFT | — | 0.932 | 1.237 | 0.573 | 0.359 | 5.686 | 0.301 | 0.362 | — | — | — |
| GFT+VAR | — | 0.328 | 1.332 | 0.163 | 0.070 | 0.751 | 0.161 | 0.387 | — | — | — |
| naive | 0.231 | 0.242 | 0.961 | 0.179 | 0.064 | 0.317 | 0.182 | 0.400 | 0.090 | 0.231 | 0.669 |
| MAE | |||||||||||
| ARGO2 | 0.224 | 0.229 | 0.509 | 0.251 | 0.156 | 0.295 | 0.202 | 0.283 | 0.186 | 0.277 | 0.344 |
| VAR | 0.256 | 0.257 | 0.576 | 0.260 | 0.179 | 0.346 | 0.251 | 0.313 | 0.248 | 0.324 | 0.431 |
| GFT | — | 0.521 | 0.696 | 0.556 | 0.490 | 1.454 | 0.411 | 0.368 | — | — | — |
| GFT+VAR | — | 0.290 | 0.681 | 0.275 | 0.178 | 0.447 | 0.259 | 0.391 | — | — | — |
| naive | 0.268 | 0.271 | 0.636 | 0.289 | 0.182 | 0.355 | 0.268 | 0.372 | 0.230 | 0.327 | 0.519 |
| MAPE | |||||||||||
| ARGO2 | 0.155 | 0.158 | 0.206 | 0.140 | 0.108 | 0.120 | 0.110 | 0.115 | 0.112 | 0.132 | 0.104 |
| VAR | 0.164 | 0.166 | 0.230 | 0.128 | 0.123 | 0.147 | 0.130 | 0.125 | 0.145 | 0.146 | 0.121 |
| GFT | — | 0.354 | 0.253 | 0.270 | 0.347 | 0.626 | 0.260 | 0.162 | — | — | — |
| GFT+VAR | — | 0.175 | 0.246 | 0.130 | 0.120 | 0.171 | 0.140 | 0.163 | — | — | — |
| naive | 0.161 | 0.163 | 0.237 | 0.142 | 0.124 | 0.143 | 0.137 | 0.140 | 0.133 | 0.149 | 0.141 |
| Correlation | |||||||||||
| ARGO2 | 0.963 | 0.954 | 0.936 | 0.951 | 0.825 | 0.935 | 0.938 | 0.952 | 0.934 | 0.938 | 0.974 |
| VAR | 0.951 | 0.944 | 0.924 | 0.943 | 0.793 | 0.934 | 0.914 | 0.941 | 0.882 | 0.923 | 0.961 |
| GFT | — | 0.833 | 0.946 | 0.934 | 0.778 | 0.905 | 0.936 | 0.947 | — | — | — |
| GFT+VAR | — | 0.930 | 0.914 | 0.934 | 0.817 | 0.895 | 0.931 | 0.935 | — | — | — |
| naive | 0.942 | 0.933 | 0.902 | 0.928 | 0.791 | 0.914 | 0.895 | 0.908 | 0.897 | 0.912 | 0.937 |
The evaluation is based on the average of ten US HHS regions in multiple periods and multiple metrics, respectively. The MSE, MAE, MAPE and correlation are reported, comparing the estimates by different methods to CDC’s reported %ILI (the prediction target) over each period. The method with best performance is highlighted in boldface for each metric in each period. Methods considered here include ARGO2, VAR, GFT, GFT+VAR, and the naive method. All comparisons are conducted on the original scale of CDC’s %ILI. The whole period is March 29, 2009 to March 17, 2018. “2009–2015” is March 29, 2009 to August 15, 2015 following GFT’s availability. Columns 4 to 12 correspond to the 2009 off-season H1N1 outbreak, and every post-2009 regular flu reason (week 40 to week 20 next year, defined by CDC’s Morbidity and Mortality Weekly Report, 17′–18′ season up to March 17, 2018). Note that 2017–2018 is the validation period as the methodology of ARGO2 was frozen on December 26, 2016. All methods and metrics are defined in Methods section.