Table 4. Model applications.

Only published model applications were included. Each line corresponds to a separate model test; therefore, some models appear more than once. References are listed for further details.

Model	Study	Prediction Target	Sample Size	Spatial Domain	Time Domain	Testing Method1	Metric Score2
B. Spatial Risk Random Forest	[29]	Mean annual incidence per 100,000 population	43,512 county-years	Conterminous US (3,108 counties)	2005–2018, averaged	Bootstrapping	R2pred = 0.59 [0.44–0.70], RMSE = 3.7
D. Spatial Risk High Resolution BRT	[30]	Ranked relative risk (0–1)	1,378 human cases	South Dakota	2004–2017	Out of sample data	AUC = 0.727
E. RF1	[28]	Annual human cases	882 county-years	New York and Connecticut	2000–2015	LOYOCV	R2pred = 0.72, RMSE = 1.6
E. RF1	[28]	Seasonal mosquito MLE	218 county-years	New York and Connecticut	2000–2015	LOYOCV	R2pred = 0.45, RMSE = 2.3
E. RF1	[28]	Seasonal mosquito MLE	2,596 trap-years	New York and Connecticut by trap	2000–2015	LOYOCV	R2pred = 0.53, RMSE = 1.0
F. NE_WNV County-years	[34]	2018 human cases3	1,472 county-years	Nebraska	2002–2017	Out of sample data	CRPS = 1.90
F. NE_WNV County-years	[34]	2018 WNV positive counties3	1,472 county-years	Nebraska	2002–2017	Out of sample data	Accuracy = 0.717
G. GLMER Ensemble	[20]	MLE mosquito infection rate	225 grid-years	Suffolk County, New York	2001–2015	LOYOCV	RMSE = 4.27
H. Harris County	[21]	MLE mosquito infection rate (1-month lead)	130,567 trap-nights	Harris County, Texas	2002–2016	Out of sample data	R2pred = 0.8
H. Harris County	[21]	Mosquito abundance (1-month lead)	10,533,033 mosquitoes	Harris County, Texas	2002–2016	Out of sample data	R2pred = 0.2
I. ArboMAP	[35]	Positive county-weeks	Approximately 9,504 county-weeks (training) Approximately 792 county-weeks (testing)	South Dakota	2004–2015 (training) 2016 (testing)	Out of sample data	AUC = 0.836–0.8564
I. ArboMAP	[36]	Positive county-weeks	Approximately 11,088 county-weeks	South Dakota	2004–20175	Fit to training data only	AUC = 0.876, Rs = 0.84
J. Chicago Ultra-Fine Scale	[22–24]	Human case probability (by hexagon)	1,346,940 hexagon-weeks	Variable, up to 5,345 1-km hexagons	2005–20166	Fit to training data only	R2 > 0.85; RMSE < 0.02; AUC > 0.90
K. Model-EAKF System	[25]	Annual human cases; peak mosquito infection rates; peak timing of infectious mosquitoes; annual infectious mosquitoes	21 county-years	2 counties (Suffolk, New York and Cook, Illinois)	Weekly, Varied by location	Retrospective data assimilation	Threshold-based accuracy 7
K. Model-EAKF System	[26]	Multiple8	110 outbreak-years	12 counties	Weekly, Varied by location	Retrospective data assimilation	Threshold-based accuracy 7
K. Model-EAKF System	[39]	Multiple8	4 county-years	4 counties	Weekly, 2017	Real-time data assimilation	Threshold-based accuracy 7
L. Temperature-forced Model-EAKF System	[26]	Multiple8	110 outbreak- years	12 counties	Weekly, Varied by location	Retrospective data assimilation	Threshold-based accuracy 7
M. California Risk Assessment	[31]	Historical outbreaks of western equine encephalomyelitis and St. Louis encephalitis as proxy for WNV	14 agency-years	California	Half-months	Temporal correspondence	Early detection of arbovirus risk prior to outbreaks
M. California Risk Assessment	[32]	Onset and peak of human cases by geographic region	12 half-months in 3 regions	California	Half-months	Retrospective data assimilation	Early detection of WNV risk prior to onset and peak of human cases
M. California Risk Assessment	[33]	Emergency planning threshold (risk ≥ 2.6)	11,476 trap-nights	Los Angeles Country, California	2004–2010	Retrospective data assimilation	AUC = 0.982

¹LOYOCV: leave-one-year-out cross-validation; Out of sample data: accuracy based on data not used to develop the model; Fit to training data only: accuracy based on the same data used to develop the model; Retrospective data assimilation: finalized data until the time of forecast; Real-time data assimilation: data processed and available at the time of forecast.

²R²_pred: predictive R², i.e., an R² calculated on data outside the sample, R_s: Spearman correlation coefficient, AUC: area under the curve, Threshold-based accuracy: +/−25% of peak week, human cases, total infections over the season; +/−25% or 1 human case, RMSE: Root Mean Squared Error, CRPS: Continuous Ranked Probability Score.

³Results for 2018 reported here, validation was also performed separately for 2012–2017, see [34] for details.

⁴Three analyses presented: short-term: AUC = 0.856, annual made on July 5: AUC = 0.836, annual made on July 39: AUC = 0.855.

⁵Restricted to July–September for each year.

⁶Restricted to 21 epi weeks per year.

⁷Varied by analysis and lead time.

⁸Prediction targets: human cases in next 3 weeks; annual human cases; week with highest percentage of infectious mosquitoes; peak mosquito infection rate; annual infectious mosquitoes.

AUC, area under the curve; CRPS, Continuous Ranked Probability Score; LOYOCV: leave-one-year-out cross-validation; RMSE, Root Mean Squared Error; WNV, West Nile virus.