Building a Better Syndromic Surveillance System: the New York City Experience

Objective

To evaluate temporal and spatial aberration detection methods for implementation in a local syndromic surveillance system.

Introduction

The New York City (NYC) syndromic surveillance system has monitored syndromes from NYC emergency department (ED) visits since 2001, using the temporal and spatial scan statistic in SaTScan for aberration detection. Since our syndromic system was initiated, alternative methods have been proposed for outbreak identification. Our goal was to evaluate methods for outbreak detection and apply the best performing method(s) to our daily analysis of syndromic data.

Methods

We tested six temporal aberration detection methods: a modified C2 algorithm [1], a CUSUM algorithm, the Holt-Winters (HW) smoothing method, a generalized linear model (GLM), a temporal scan statistic in SaTScan [2], and an autoregressive integrated moving average (ARIMA) model. We tested four spatio-temporal methods: a generalized linear mixed model (GLMM), a Bayesian model [3], and the spatial scan statistic using the Bernoulli probability model and space-time permutation scan statistic in SaTScan [2]. We analyzed ED visits that occurred in NYC during 2010-2011. To test a method’s ability to detect different outbreak types, we added simulated outbreaks to NYC syndromic data. A total of 180 datasets were created, each containing one spike of varying duration and magnitude, allocated either to a single ZIP code, a cluster of ZIP codes, or citywide. To compare the performance of these methods, we estimated sensitivity, specificity, timeliness, and positive predictive value (PPV) of signals. Timeliness was estimated by calculating the earliest day of detection as a proportion of the outbreak length before the peak number of excess cases (timeliness ranged from 1 to 0, with detection on the first day of an outbreak resulting in a timeliness measure of 1). To determine how difficult it would be to implement these methods into our system, we recorded programming time, computer run time, and skills (detailed coding or statistical) needed to program the method.

Results

The temporal scan statistic, our current method for temporal outbreak detection, performed the best in regard to sensitivity and timeliness of detection (Table 1). Specificity was low however, suggesting the method may signal often, even when there is no outbreak. All methods had poor PPV, so when a method did signal, it was unlikely to be an outbreak. Among the spatial methods, the GLMM had the highest sensitivity (Table 2), though it signaled the most often. Of note, the methods performed better when the outbreaks were of longer duration (>5 days), both in detecting outbreaks at all and the timeliness of outbreak detection. The easiest method to code was the modified C2 and the most difficult was the Bayesian spatial model.

Conclusions

While there was variability in method performance, we feel there is no one best method for outbreak detection, as we have to weigh issues like signal frequency, specificity, and timeliness. We also found certain methods required time and skill to code, which is useful when deciding which methods to implement into a daily system. Our poor PPV may be explained in that methods are detecting unlabeled outbreaks given we used actual syndromic data. Our next step is to continue comparing results between these methods, and select the most promising methods to run prospectively in our current system.

Table 1. Metrics of the temporal models running at a fixed alert rate of 0.01 (1 alarm/100 days)

Table 2. Metrics of the spatial models