Evaluation of Syndromic Surveillance Systems in 6 US State and Local Health Departments

Abstract

Objective:

Evaluating public health surveillance systems is critical to ensuring that conditions of public health importance are appropriately monitored. Our objectives were to qualitatively evaluate 6 state and local health departments that were early adopters of syndromic surveillance in order to (1) understand the characteristics and current uses, (2) identify the most and least useful syndromes to monitor, (3) gauge the utility for early warning and outbreak detection, and (4) assess how syndromic surveillance impacted their daily decision making.

Design:

We adapted evaluation guidelines from the Centers for Disease Control and Prevention and gathered input from CDC subject matter experts in public health surveillance to develop a questionnaire.

Participants:

We interviewed staff members from a convenience sample of 6 local and state health departments with syndromic surveillance programs that had been in operation for >10 years.

Results:

Three of the 6 interviewees provided an example of using syndromic surveillance to identify an outbreak (ie, cluster of foodborne illness in 1 jurisdiction) or detect a surge in cases for seasonal conditions (eg, influenza in 2 jurisdictions) prior to traditional, disease-specific systems. Although all interviewees noted that syndromic surveillance has not been routinely useful or efficient for early outbreak detection or case finding in their jurisdictions, all agreed that the information can be used to improve their understanding of dynamic disease control environments and conditions (eg, situational awareness) in their communities.

Conclusion:

In the jurisdictions studied, syndromic surveillance may be useful for monitoring the spread and intensity of large outbreaks of disease, especially influenza; enhancing public health awareness of mass gatherings and natural disasters; and assessing new, otherwise unmonitored conditions when real-time alternatives are unavailable. Future studies should explore opportunities to strengthen syndromic surveillance by including broader access to and enhanced analysis of text-related data from electronic health records.

Introduction

Evaluation of public health surveillance systems is a critical component of ensuring that conditions of public health importance are appropriately monitored. ¹ U.S. Centers for Disease Control and Prevention’s (CDC) original Guidelines for Evaluating Surveillance Systems, published in 1988 and updated in 2001, center on promoting the best use of resources by ensuring efficiency and focusing on surveillance systems for conditions of public health importance. ^2,3 In 2004, increased interest in addressing the threat of bioterrorism and the then nascent development of systems for early outbreak detection led to the publication of the Framework for Evaluating Public Health Surveillance Systems for Early Detection of Outbreaks.⁴ The framework supplemented the existing CDC evaluation guidelines and specifically highlighted the need to evaluate timeliness, sensitivity, predictive value positive (PVP), and predictive value negative (PVN) for early outbreak detection, including syndromic surveillance systems.

Syndromic surveillance systems were originally developed to speed early warning of and rapid response to bioterrorism-related events. ⁵ The purpose of syndromic surveillance is to use data, primarily those collected from emergency department (ED) records (eg, patient chief complaints), and statistical methods (eg, aberration calculations) to detect, monitor, and characterize patterns of illness in the community. Early syndromic surveillance systems, such as the one developed in New York City in 1995, focused specifically on waterborne illness (ie, Cryptosporidium and Giardia), but were neither fully automated nor linked to patient records.⁶ The attacks of September 11, 2001, and the subsequent anthrax attacks substantially increased the pressure to obtain real-time information that would inform and provide early warning to health departments of bioterrorism events.⁷ For example, CDC launched the BioSense program in 2003 under the authority of the Public Health Security and Bioterrorism Preparedness and Response Act of 2002, establishing an integrated national public health surveillance system for early detection and rapid assessment of bioterrorism-related events.^8,9 During the past decade, syndromic surveillance programs expanded to include improving public health situational awareness (ie, information that helps one understand a dynamic disease control environment).¹⁰ In 2014, BioSense evolved into the National Syndromic Surveillance Program to emphasize the collection and timely sharing of data and the role of practitioners at the state and local level who collaborate to improve the nation’s responsiveness to outbreaks and public health hazards, while advancing the science of syndromic surveillance.¹¹

Reviews and evaluations of syndromic surveillance systems in the literature have focused on a specific infectious disease (eg, influenza) or syndrome (eg, gastrointestinal illness).^12.13 In order to better understand the current state of syndromic surveillance we conducted structured interviews with staff among 6 syndromic surveillance programs of pioneering state and local health departments in the United States. These jurisdictions were early-adopters of syndromic surveillance and now have greater than 10 years of syndromic surveillance practice and innovation. Interviewees included the syndromic surveillance program managers of all 6 jurisdictions, and program analysts, in addition to the managers, of 2 jurisdictions. Among these, 4 are currently affiliated with the National Syndromic Surveillance Program (NSSP).

Currently, approximately 60 state, county, or local health departments in the United States operate syndromic surveillance programs. Of these, 47 sites are part of the NSSP, representing some portion of 40 states. (Michael Coletta, MPH, email communication, April 14, 2017). With our small sample, our intent was not to extrapolate our findings nationally, but to qualitatively assess the attributes of 6 experienced syndromic surveillance programs to provide insight into opportunities to strengthen syndromic surveillance in the US.

Methods

To understand the procedures, policies, and outcomes associated with syndromic surveillance, we interviewed staff members from a convenience sample of 6 health departments with mature syndromic surveillance systems operating for >10 years: 2 state health departments (Michigan and Washington) and 4 local health departments (Boston, Denver, New York City, and Seattle). We selected an initial 4 interviewees based on our knowledge of the health departments’ long-standing syndromic surveillance systems and their willingness to participate. We selected the other 2 interviewees based on recommendations from the initial 4 interviewees.

We adapted CDC’s standard methods for evaluating public health surveillance systems, including the Guidelines for Evaluating Surveillance Systems, the Framework for Evaluating Public Health Surveillance Systems for Early Detection of Outbreaks, and the Epidemic Intelligence Service Standard Evaluation Form, and gathered input from CDC public health subject matter experts to develop standard questions used for each of the interviews.^14–18 We used closed- and open-ended questions that focused on (1) understanding the characteristics and current uses of syndromic surveillance systems, (2) identifying the most and least useful syndromes to monitor, (3) gauging the utility of their syndromic surveillance as an early warning and outbreak detection system, and (4) assessing how syndromic surveillance impacted their decision making. We categorized the responses based on primary use, early warning, and decision making. This brief report highlights the findings specific to the attributes that we believe provide the greatest insight into the unique qualities of these systems as well as areas that present opportunities to strengthen syndromic surveillance and ultimately public health surveillance.

We conducted 5 interviews by telephone and 1 interview in person between November and December 2015. Each interview lasted 60 to 75 minutes. The interviewer recorded responses to questions on the survey form. Qualitative data was collected and compiled for review. In February and March 2016, we sent follow-up questions to 4 health departments to clarify responses and to ask additional system-specific questions. We compiled our data in tables using Microsoft Word. This work was determined by CDC officials to be program evaluation and was deemed not to be human subjects research.

Results

All 6 health departments used the text of ED chief complaints as their primary data source, and 2 health departments included at least 1 additional data source, such as triage notes or International Classification of Diseases (ICD) codes.

Syndromic surveillance systems at all 6 health departments included 4 functional domains: data collection, data transmission, data analysis, and reporting. Each health department defined the algorithms for identifying patterns of text in the ED chief complaint before receiving and analyzing the data. The 6 health departments developed syndrome definitions using keywords found in chief complaints and ICD code fields.

Below are highlights of system attributes from our interviews.

Simplicity:

Because the data being collected and transmitted are part of the routine hospital data collection, there is minimal reporting burden on the hospitals and their providers.

Timeliness:

In 5 health departments, the data were transmitted as they became available from the ED in near-real time, delayed only by automated processing time or network transmission. One health department reported delays of 24 to 96 hours because of batching by a third-party data aggregator. There is essentially no reporting lag in a patient entering the ED and the chief complaint data being transmitted to the health department, which allows analysis to take place promptly. Three health departments analyzed the previous days’ ED data each weekday to detect aberrations in the occurrence of syndromes in the community; 1 conducted automated hourly analysis, because syndromic data were received continuously; and 2 conducted analysis only upon request or when an alert from the system prompted review of the data by syndromic surveillance staff members to determine if follow-up was required.

Data Quality:

Since the primary purpose of data entered in the EHR is for patient care data quality depends on the health care provider. There is no standardized manner in which the hospitals are encouraged to capture the data. Within each jurisdiction, hospitals varied according to the person documenting the chief complaint (registration clerk v triage nurse), the mechanism for documenting findings in electronic health records (EHRs) (free text v selecting from a list), and the populations (adult v pediatric) that influenced the data content, quality, or readiness for use.Quality and content of data varies across the multiple data entry fields in the EHR; some fields are required while some are optional, some facilities report chief complaint exactly as the patient reports, whereas others paraphrase or select terms from a drop-down menu. Other free text fields such as clinical impression and triage note are often left empty or include boilerplate language. Sometimes a field will cut-off mid-sentence.

Flexibility:

The syndromic surveillance systems in the jurisdictions evaluated were highly flexible. All 6 interviewees highlighted the ability to easily edit the algorithms as needed. This flexibility allowed for refinement over time through trial and error and in near-real time, as new diseases or conditions emerged. The ad hoc nature of syndromic surveillance systems dictated that each of the six health departments’ algorithms was unique.

Representativeness/Coverage:

Representativeness is difficult to accurately assess presently because the true utilization of EDs for the types of conditions and events related to outbreaks of infectious disease. These 6 syndromic surveillance systems were representative of the populations seeking ED care with 60–100% (median: 96) of EDs submitting syndromic data to the health departments in these jurisdictions. In two jurisdictions all non-federal hospitals with EDs were participating in the syndromic surveillance program.

Perceived Predictive Value:

Outbreaks of various conditions and magnitudes occurred in all these jurisdictions over 10 years, and during this time the method was infrequently successful in the identification of outbreaks. Only 1 health department reported efforts to identify potential cases of reportable diseases using syndromic surveillance data. All 6 health departments reported that the textual data of the ED chief complaint were not specific enough for notifiable case finding of most conditions. Three interviewees provided an example of syndromic surveillance identifying an outbreak or a surge in cases more quickly than traditional systems. One jurisdiction identified a foodborne outbreak when a group of patients sought care in the same ED with symptoms of gastrointestinal illness. The outbreak likely would have gone unnoticed. It was investigated and the original cluster of chief complaint data led to identification of other cases through query and analysis of the syndromic surveillance system.

Five health departments reported using syndromic surveillance data to monitor and track seasonal influenza and pandemic 2009 H1N1. In 2 jurisdictions, surges in the number of patients with influenza-like illness (ILI) seeking care in EDs provided several days’ earlier warning than did established systems, which allowed for targeted vaccination and messaging for at-risk populations. However, all 6 respondents agreed that syndromic surveillance has not been routinely beneficial or efficient for early detection of outbreaks or case finding; additionally these 6 health departments indicated that the predictive value of using syndromic surveillance to identify outbreaks was low, primarily due to the low signal to noise ratio of the chief complaint data for reportable conditions.

Most and Least Useful Syndromes:

Five of the 6 health departments indicated that the most useful syndromes to monitor were ILI and gastrointestinal (GI) as a potential indicator of Norovirus. Three of the 6 listed both ILI and GI as priority syndromes to monitor; 2 prioritized GI; and 1 listed neither, relying on tailored queries of the syndromic data depending on the situation (ie, rash during a measles outbreak).

Usefulness:

Five health departments indicated that syndromic surveillance has become an accepted and primary component of improving situational awareness and has provided information regarding conditions for which previously no other timely surveillance system existed, or for which other data were not available in a timely manner (eg, opioid overdoses). Four health departments also indicated using syndromic surveillance to rapidly assess rumors of outbreaks or other unusual events. Below we highlight 3 specific examples of syndromic surveillance use in these health departments.

Adverse Events associated with Opioids and Marijuana:

Increased prescription opioid and heroin overdoses and adverse events associated with marijuana use have become public health concerns in recent years.^19,20 The lack of formal surveillance systems to track these events makes it challenging to rapidly obtain timely information about opioid overdose trends and examine the public health impact of decriminalized marijuana use. All 6 health departments had developed syndrome definitions to examine data on ED visits related to these drug exposures and used the results of increased ED usage for these conditions to inform the development of drug overdose prevention and outreach programs; collaborate with law enforcement anti-drug efforts; inform clinicians who prescribe opioids to improve awareness, prescribing practices, delivery of prevention messages to patients, and treatment; inform boards of health about the impact of marijuana legalization on ED usage; and educate purveyors and users of edible marijuana about risks and practices associated with severe health outcomes.²¹

Mass Gatherings and Special Events:

All 6 interviewees used syndromic surveillance systems to identify evolving health needs and incidents associated with mass gatherings, special events, or manmade and natural disasters (eg, mass casualty events). All 6 health departments indicated using syndromic surveillance to assess syndromes and potential conditions of interest in local and regional hospital ED chief complaint data associated with events of potential public health importance. For example, just after the Boston Marathon bombing in April 2013, the Boston syndromic surveillance system indicated an increase in mental health-related visits to local EDs. The health department provided this information to the Boston Office of Public Health Preparedness, which in turn offered additional mental health services and worked with health care providers to prepare for a surge in patients with mental health-related needs.

Bioterrorism Alerts:

During a 2008 national political convention in Denver that gathered tens of thousands of people from across the nation, an environmental sensor produced a signal indicating the presence of an organism that might be used as a bioterrorism agent. The local public health authority used public health surveillance data, including syndromic surveillance data, to determine if there had been an increase in patient ED visits for syndromes that could be associated with this agent. Findings indicated no increase in syndrome presentations related to the perceived threat. This information prevented a large-scale response and decontamination effort, thereby saving time and resources for response and public relations staff members.

Discussion

In the United States, syndromic surveillance has become an additional data source to monitor the spread and intensity of outbreaks (eg, influenza) and enhance public health situational awareness during mass gatherings, during natural disasters, and when new, unmonitored conditions arise for which there is no formal disease reporting requirement or legislation. Health departments have also found novel uses (eg, heroin and opioid overdoses) for these near real-time data by developing aberration detection methods and leveraging natural language processing tools.

Among the 6 interviewees, syndromic surveillance was seen as useful for monitoring new conditions and shows promise when the analysis of textual data improves decision-making utility. Other jurisdictions and experts have also considered broadening the use of syndromic surveillance to monitor conditions such as heat-related illness. ^22,23 Based on the information collected in our interviews we believe that additional coded and free-text contextual information, such as triage notes, history of present illness, and other EHR fields, may provide useful data that could improve attributes of syndromic surveillance systems, such as the PVP, in these six jurisdictions and to the practice in general.

Our interviews indicated that the utility of syndromic surveillance for early outbreak detection has been low among these 6 health departments. This seems primarily due to the signal-to-noise ratio that was not high enough in these jurisdictions to indicate the presence of an outbreak or other event of public health importance.²⁴ The ability of syndromic surveillance to detect an outbreak depends on several local features, including the magnitude and shape of epidemic curve; the likelihood that the resulting syndrome will lead to an ED visit for outbreak-related symptoms; the background frequency of the syndrome among ED patients; and the timeliness, completeness, and accuracy of transmitting and consuming ED data.

Much has been written about syndromic surveillance sensitivity, with the consensus that syndromic surveillance systems are highly sensitive but have low specificity for infectious conditions. ^25,26 The specificity of syndromic surveillance can be substantially impacted by media attention. One health department in our study observed “worried well” patients seeking care in the ED during highly publicized public health events, including events involving Ebola virus and Legionnaire’s disease. This type of care-seeking behavior was seen in 2001 after anthrax cases were confirmed in a New Jersey jurisdiction. This behavior has the potential to obscure a true syndromic surveillance signal during an event such as a bioterrorism attack as the increase of ED visits would trigger an alert, but there would not be a true increase of cases.²⁷ However, large increases in worried well patients seeking care in EDs can be an indicator of increased anxiety and social stress in the population, which may trigger such public health interventions as risk communication, educational material, or community outreach.

Based on the findings from our interviews with these 6 early-adopting jurisdictions, we offer the following considerations, which we hypothesize may help to strengthen the early outbreak detection capability of the 6 systems evaluations as well as similar systems in other jurisdictions:

Access and Use Text Better for Syndromic Surveillance

Health departments should consider expanding access to other text fields from the EHR into the syndromic surveillance data stream. We speculate that to improve sensitivity and predictive value of syndromic surveillance, access to additional data from the EHR, such as coded diagnoses and free-text fields (eg, triage notes), will be needed to separate the signal indicating the presence of a condition of interest from the noise of the chief complaint. More adaptive methods in which the model learns over time might allow for the detection of syndromes that were not prespecified. Advanced methods of working with text data may also help with discovery of other patterns in the high volume data of the EHR.

Enhance Broader Data Exchange Between Health Care and Public Health

To more directly address the need for rapidly identifying outbreaks, health departments should explore further leveraging of data with more specificity within the EHR. This could include the automated generation and electronic transmission of reports to state and local public health authorities in real time for review and action based upon laboratory and diagnostic codes for reportable cases or conditions. Implementation of this electronic case reporting could provide the timely, sensitive, specific, and actionable information needed for early warning systems for some conditions.

Conclusions

Syndromic surveillance has provided many lessons and adjunctive capabilities, which have expanded modern public health’s surveillance armamentarium. Public health is benefiting from systems and data integration between the public health and health care sectors and is strengthening relationships at the local and state levels. General experience with syndromic surveillance during the past 2 decades has varied; some health departments have used it to reassure the public that no nefarious act has occurred, and others have used it as a routine component of overall public health surveillance. The 3 unpublished examples of uses of syndromic surveillance mentioned by interviewees have not been described in the literature. We surmise that many other interesting uses of syndromic surveillance data, in many jurisdictions, also have never come to light. Facilitating publication of the routine, episodic, and unique uses of syndromic surveillance would enable a fuller evaluation, discovery of innovative methods, and expanded use of syndromic surveillance systems.

Implications for Policy & Practice

To improve the predictive value and strengthen the early outbreak detection capability of these systems, exploration of additional data from the EHR and methods for analyzing these data are needed. By focusing on key enhancements such as broadening text-related data acquisition and enhancing computing and text analytic resources, health departments may accelerate the development and use of syndromic surveillance systems to get the right information to the right people at the right time, which is the overarching goal of CDC’s Surveillance Strategy.²⁸