Adapting Syndromic Surveillance Baselines After Public Health Interventions

Abstract

Background

Public health surveillance requires historical baselines to identify unusual activity. However, these baselines require adjustment after public health interventions. We describe an example of such an adjustment after the introduction of rotavirus vaccine in England in July 2013.

Methods

We retrospectively measured the magnitude of differences between baselines and observed counts (residuals) before and after the introduction of a public health intervention, the introduction of a rotavirus vaccine in July 2013. We considered gastroenteritis, diarrhea, and vomiting to be indicators for national syndromic surveillance, including telephone calls to a telehealth system, emergency department visits, and unscheduled consultations with general practitioners. The start of the preintervention period varied depending on the availability of surveillance data: June 2005 for telehealth, November 2009 for emergency departments, and July 2010 for general practitioner data. The postintervention period was July 2013 to the second quarter of 2016. We then determined whether baselines incorporating a step-change reduction or a change in seasonality resulted in more accurate models of activity.

Results

Residuals in the unadjusted baseline models increased by 42%-198% from preintervention to postintervention. Increases in residuals for vomiting indicators were 19%-44% higher than for diarrhea. Both step-change and seasonality adjustments improved the surveillance models; we found the greatest reduction in residuals in seasonally adjusted models (4%-75%).

Conclusion

Our results demonstrated the importance of adjusting surveillance baselines after public health interventions, particularly accounting for changes in seasonality. Adjusted baselines produced more representative expected values than did unadjusted baselines, resulting in fewer false alarms and a greater likelihood of detecting public health threats.

Public health surveillance is a key part of protecting the health of a nation. Statutory authorities, such as Public Health England (PHE), maintain surveillance services to identify potential threats to public health, including infectious disease, bioterrorism, and environmental hazards.¹ Public health surveillance can provide early warning and information about the impact of threats; these warnings help authorities plan their response to protect the population’s health.

Increasingly, public health organizations are using syndromic surveillance in addition to traditional surveillance, such as laboratory reporting.^2-9 A syndromic surveillance system refers to a data source (eg, emergency department [ED] visits) within a syndromic surveillance service (eg, the PHE real-time syndromic surveillance service). Syndromic surveillance service also includes the informatics and analytical processes and public health actions undertaken. Syndromic surveillance uses prediagnostic data; that is, the data do not specify a single particular threat or pathogen but instead use broad aggregations of diagnoses that can detect a range of threats. Syndromic surveillance often involves monitoring aggregated population-level data rather than person-level data. Although less specific than data from laboratory-based reports, syndromic surveillance data are timelier and can detect potentially emerging threats against which routine diagnostic tests have not yet been developed.

Syndromic surveillance often involves the use of “big data”: large, unvalidated, complex health data sets that need to be analyzed on a near–real-time basis to enable timely public health action. Therefore, surveillance services apply statistical algorithms to automatically identify unusual spikes, trends, or other aberrations in health data. Researchers have developed a wide range of statistical methods for aberration detection in syndromic surveillance.¹⁰ These methods all require a baseline to establish usual, expected activity in the absence of any public health threats.

A large national public health intervention (eg, the introduction of a new vaccine into the routine immunization schedule) may lead to a long-term change in morbidity. Consequently, the level of activity monitored by syndromic surveillance will also change. Therefore, it is important that historic baselines are adjusted to account for the effect of these interventions. If these adjustments are not made, modeled baselines could appear too low, resulting in false alarms, or too high, resulting in a failure to detect increases in activity.

The vast majority of studies on public health interventions and surveillance focus on the important areas of measuring the impact^11-18 or the effectiveness and safety of interventions.^19,20 To our knowledge, no studies in the peer-reviewed literature have investigated how public health interventions have affected surveillance services.

One recent example of a successful public health intervention was the introduction of a rotavirus vaccine in England in July 2013.²¹ This intervention led to reductions in the number of laboratory reports of rotavirus and the number of people presenting to health care services with diarrhea and vomiting.^18,22-24 Research in several countries showed that the introduction of a rotavirus vaccine can change the seasonality of illness.^12,24,25

PHE maintains a suite of syndromic surveillance systems, used to complement existing public health surveillance programs, with each system using a historical baseline. We assessed the accuracy of these baselines after a public health intervention (using the introduction of rotavirus vaccine as an example) to determine changes required in the baseline models used for aberration detection. Thus, the aim of our work was to improve the effectiveness of surveillance baselines postintervention by accounting for any changes in seasonality and absolute reductions in illness.

Materials and Methods

Baseline Models

PHE uses the Rising Activity Multi-level Mixed effects Indicator Emphasis (RAMMIE) method, described elsewhere,²⁶ to generate baselines and alarm thresholds for routine surveillance. RAMMIE creates regression models for baselines, which are refreshed approximately every 6 weeks as new data become available. We compared 3 models created by the RAMMIE method. The first model, the original model, did not take into account any changes caused by the public health intervention; this model was used prospectively for surveillance at the time of the introduction of the rotavirus vaccine. The second model, a step-change model, included a single binary variable that was 0 before the intervention and a constant value afterward. The third model, a seasonality model, included a different model for seasonality before and after the intervention by using 2 sets of variables for seasonality (based on month of the year), one of which had values of 0 before the intervention.

Original Model

The original model (Equation [1]) is a negative binomial regression model that includes coefficients for seasonality, day of the week, and public holidays.

$$\begin{array}{c}\mathrm{I}\mathrm{n}\left(coun{t}_t\right)=\mathrm{I}\mathrm{n}\left(tota{l}_t\right)+{\beta }_{0}B{H}_t+\sum _{i=1}^7{\beta }_i{D}_{it}\\ & +\sum _{j=1}^{12}{\alpha }_j{M}_{jt}+\gamma H_t\end{array}$$ (1)

where count _t is the predicted syndromic count on day t for the modeled indicator; total _t is the total syndromic count on day t for all diagnoses received by the system; BH _t is a binary variable that is 1 when day t is a public holiday, 0 otherwise; and D _it are 7 binary variables for the day of the week, in which on day t, 6 of these will be 0 and the other variable will be equal to 1, so that the sum of the 7 variables on any day is equal to 1.

M _jt are 12 weighted variables for the months of the year. If a single coefficient value (α_j) for each month is used to model seasonality, then discontinuities can result, with large step changes in the daily models between the end of one month and the beginning of the next. To avoid these step changes in the model, the monthly coefficient values are weighted to smooth the transition from one month to the next (Equations [2] - [5]). Each day t is assigned 1 or 2 non-0 weights for these month variables. Day t has a non-0 value for the month in which it falls and for the previous month if it is in the first half of the month or for the following month if it is in the second half of the month.

When t is the kth day of month j,

If k > 16 (second half of the month), then

$$M_{jt}=(46-k)∕30$$ (2)

$$M_{(j+1)t}=(k-16)∕30$$ (3)

Otherwise, when k < 17 (first half of the month), then

$$M_{(j-1)t}=(16-k)∕30$$ (4)

$$M_{jt}=(14+k)∕30$$ (5)

These monthly weights are designed so that in the middle of the month (defined as day 16) M _jt is 1 for the current month and 0 for all other months. At the start of the month (eg, day 1), M _jt is 0.5 for the current month, 0.5 for the previous month, and 0 otherwise. Consequently, the sum of the 12 weights on any particular day is always equal to 1.

H _t is a binary variable for the telehealth models after the introduction of a new telehealth service (a national free telephone health information advice line for England, National Health Service [NHS] 111) in September 2013,²⁷ otherwise 0. β ₀ , β _i , α _j, and γ are the model coefficients.

Step-Change Model

The step-change model (Equation [6]) adds a binary variable to the original model to account for the change after the public health intervention (in our example, the introduction of a rotavirus vaccine).

$$\begin{array}{c}\mathrm{I}\mathrm{n}\left(coun{t}_t\right)=\mathrm{I}\mathrm{n}\left(tota{l}_t\right)+{\beta }_{0}B{H}_t+\sum _{i=1}^7{\beta }_i{D}_{it}\\ & +\sum _{j=1}^{12}{\alpha }_j{M}_{jt}+\theta I_t\end{array}$$ (6)

where I _t is 0 before the intervention and 1 afterward.

Seasonality Model

The seasonality model (Equation [7]) adds a new set of seasonality variables to account for a change in seasonality after the public health intervention.

$$\begin{array}{c}\mathrm{I}\mathrm{n}\left(coun{t}_t\right)=\mathrm{I}\mathrm{n}\left(tota{l}_t\right)+{\beta }_{0}B{H}_t+\sum _{i=1}^7{\beta }_i{D}_{it}\\ & +\sum _{j=1}^{12}{\alpha }_j{M}_{jt}+\gamma H_t+\sum _{j=1}^{12}{\delta }_j{I}_t{M}_{jt}\end{array}$$ (7)

where I _t is a binary variable equal to 1 after the intervention and equal to 0 before the intervention. Consequently, although α_j measures the preintervention seasonality as in the original and step-change models, the new model coefficient δ_j estimates the difference in seasonality between the postintervention and preintervention periods.

Effectiveness of Models

We retrospectively compared the models with the observed syndromic counts seen each day. We measured model fit by comparing the average square of the daily differences between observed values and baseline values (residuals) for the postintervention periods.

Worked Example: Introduction of Rotavirus Vaccine

We selected syndromic indicators for gastrointestinal illness from 3 national syndromic surveillance systems that have several years of data available before and after the introduction of the rotavirus vaccine in July 2013. We included syndromic indicators based on clinical diagnostic codes (eg, Systematized Nomenclature of Medicine Clinical Terms)²⁸ for diarrhea and vomiting and, for non-telehealth systems, a combined syndromic indicator for gastroenteritis. We used data from 3 national syndromic surveillance systems: NHS 111, which captures data on telehealth calls²⁷; the General Practitioner Out-Of-Hours and unscheduled care Syndromic Surveillance (GPOOHSS) system²⁹; and the Emergency Department Syndromic Surveillance System (EDSSS), which captures data on ED visits.³⁰ Data from different periods were available for each system, with the period depending on when the systems were established and when we refitted the surveillance models during 2016. For NHS 111, data were available for June 1, 2005, to March 28, 2016; GPOOHSS, November 7, 2009, to February 15, 2016; and EDSSS, July 27, 2010, to April 16, 2016. During this study, EDSSS was a sentinel system and, therefore, it had a smaller volume of data than the other 2 systems; it became a national system in 2018.

Results

Approximately 3.5 million telephone calls were documented to NHS 111 for diarrhea or vomiting; more than 2 million consultations were documented by the GPOOHSS for diarrhea, vomiting, or gastroenteritis; and more than 120 000 ED visits were documented by EDSSS for diarrhea, vomiting, or gastroenteritis. Daily counts were several times higher during weekends and public holidays than during weekdays for NHS 111 and GPOOHSS (Table 1).

Table 1.

Observed daily counts of consultations, by syndromic surveillance system, indicator, weekday mean, and weekend and public holiday mean, England, 2005-2016

System/indicator	Weekday, mean no.	Weekend and public holiday, mean no.	Total during period, no.
NHS 111a (June 1, 2005, to March 28, 2016)
Diarrhea	293	560	1 469 937
Vomiting	438	742	2 082 731
GPOOHSSb (November 7, 2009, to February 15, 2016)
Diarrhea	76	280	318 392
Vomiting	129	329	436 928
Gastroenteritis	353	1050	1 302 659
EDSSSc (July 27, 2010, to April 16, 2016)
Diarrhea	11	13	24 085
Vomiting	9	10	18 713
Gastroenteritis	37	43	81 807

^aNational Health Service (NHS) 111 is a national free telephone health information advice line for England.²⁷

^bGeneral Practitioner Out-Of-Hours and unscheduled care Syndromic Surveillance (GPOOHSS) system.²⁹

^cEmergency Department Syndromic Surveillance System (EDSSS) captures data on emergency department visits.³⁰

The original model fit the observed data better before the intervention than after the intervention; we found bigger differences between the baseline data and the observed data after the intervention. Consequently, the average square of the daily residuals was larger after the intervention than before the intervention for each system and indicator. The smallest percentage increase in residuals was for ED visits for diarrhea (42%), and the largest was for calls to NHS 111 for vomiting (198%). For each system, the percentage increase in residuals after the intervention was greater for the vomiting-related models than for the diarrhea-related models (Table 2).

Table 2.

Residual errors for the original model before and after the introduction of a rotavirus vaccine in July 2013, by surveillance system and indicator, England, 2005-2016

System and indicator	Average of square of daily residualsa		Percentage increase
	Preintervention	Postintervention
NHS 111b : vomiting	9004.5	26 829.7	198
NHS 111b : diarrhea	3309.9	9113.0	175
GPOOHSSc : gastroenteritis	5999.4	15 533.4	159
EDSSSd : gastroenteritis	42.7	89.5	109
EDSSSd : vomiting	7.4	13.7	86
GPOOHSSe: vomiting	685.5	1160.1	69
GPOOHSSe: diarrhea	431.7	649.1	50
EDSSSd : diarrhea	14.8	21.1	42

^aA residual is the difference between baseline and observed counts.

^bNational Health Service (NHS) 111 is a national free telephone health information advice line for England.²⁷

^cGeneral Practitioner Out-Of-Hours and unscheduled care Syndromic Surveillance (GPOOHSS) system.²⁹

^dEmergency Department Syndromic Surveillance System (EDSSS) captures data on emergency department visits.³⁰

The step-change model had lower postintervention residuals than the original model for every system and indicator (Table 3). However, the percentage improvements in residuals in the step-change model compared with the original model were much smaller in the NHS 111 and EDSSS (percentage improvement, 0%-7%) than in the GPOOHSS (percentage improvement, 40%-60%).

Table 3.

Comparison of average square of residual model errors after an intervention (introduction of a rotavirus vaccine in July 2013), by surveillance system and indicator, England, 2005-2016

System and indicator	Average of square of daily residuals			Percentage improvement compared with original model
	Original model	Step-change model	Seasonality model	Step-change model	Seasonality model
GPOOHSSa : gastroenteritis	15 533.4	6239.6	3835.0	60	75
NHS 111b : vomiting	26 829.7	26 786.7	11 632.7	0	57
NHS 111b : diarrhea	9113.0	8814.6	4013.0	3	56
GPOOHSSa : diarrhea	649.1	391.7	309.9	40	52
GPOOHSSa : vomiting	1160.1	646.4	573.4	44	51
EDSSSc : gastroenteritis	89.5	83.2	79.8	7	11
EDSSSc : diarrhea	21.1	20.6	19.5	3	8
EDSSSc : vomiting	13.7	13.3	13.1	3	4

^aGeneral Practitioner Out-Of-Hours and unscheduled care Syndromic Surveillance (GPOOHSS) system.²⁹

^bNational Health Service (NHS) 111 is a national free telephone health information advice line for England.²⁷

^cEmergency Department Syndromic Surveillance System (EDSSS) captures data on emergency department visits.³⁰

The seasonality model had lower residual errors than both the original model and the step-change model for every system and indicator. In particular, the residuals for the NHS 111 indicators in the seasonality model were 56%-57% lower than the residuals in the original model (Table 3).

Visual inspection of the data as time series for the original model and seasonality model showed a clear change in the seasonality after the public health intervention in July 2013 (Figure 1). Before the intervention, a large peak in vomiting and diarrhea took place annually around week 13. After the intervention, much less seasonal variation occurred. The original model predicted that these peaks would continue after the intervention, whereas the seasonality model was much closer to the observed seasonality.

Figure 1.

Comparison of selected gastrointestinal indicators in an original model and in a seasonality model before and after a public health intervention (introduction of a rotavirus vaccine in July 2013), England, 2011-2015. Solid lines indicate observed rates, dashed lines indicate models, and the solid vertical line in mid-2013 indicates introduction of the rotavirus vaccine. Abbreviation: ED, emergency department. Data sources: Harcourt et al,²⁷ Harcourt et al,²⁹ Elliot et al.³⁰

The NHS 111 system indicators had both more seasonal variation and higher levels of activity before than after July 2013 (Figure 2). The original model and the step-change model both accounted for the reduction in activity but not the change in seasonality. Therefore, after 2013 these models overestimated activity during the spring peak (weeks 1-17) and underestimated activity during the summer and autumn (weeks 26-40). For example, the percentage of vomiting-related NHS 111 calls was lower on average each week of the year after the intervention than before the intervention. However, this drop was greater during the weeks when vomiting-related calls were highest before the intervention. Therefore, we observed less seasonal variation after the intervention. The step-change model had a similar annual average to the observed postintervention average, but because it still modeled the old seasonal variation, this model overestimated activity during the spring and underestimated activity during the summer.

Figure 2.

Seasonality of telehealth vomiting-related calls before and after a public health intervention (introduction of a rotavirus vaccine in July 2013), England, 2011-2015. Observed weekly data were averaged during the preintervention and postintervention periods, showing both a decrease in calls and a change in seasonality. Both models have an annual average call rate similar to the postintervention period, but only one has corrected for the change in seasonality. Abbreviation: ED, emergency department. Data sources: Harcourt et al,²⁷ Harcourt et al,²⁹ Elliot et al.³⁰

Discussion

We found that the successful introduction of a national public health intervention in England negatively affected the residuals of syndromic surveillance model baselines. Furthermore, we found that the effect was not just a step-change reduction but also a change in the seasonal trends monitored by the syndromic indicators. Once adjusted for seasonality, the surveillance models continued to accurately model gastrointestinal activity.

The reductions in the numbers of patients accessing health care services after the introduction of the rotavirus vaccine were greatest during the spring (weeks 4-16), when the rotavirus incidence was previously the highest.¹⁸ Therefore, there was a decrease in the seasonal variation of the syndromic indicators studied. When models assumed a step-change decrease rather than a reduction in seasonal variation, the reduction in seasonality resulted in an overestimation of counts during part of the year and an underestimation during other periods. Thus, step-change models resulted in a decrease in detection sensitivity for part of the year and an increase in false alarms at other times. Changes in seasonality after introduction of a vaccine have been described in other countries^12,25; as such, our results will have relevance to public health authorities worldwide.

Although all our models improved by including changes to seasonality postintervention, the differences in the amount of improvement were sometimes counterintuitive. For example, the models for vomiting improved more than the models for diarrhea. However, these differences may reflect differences in scale or specificity of syndromic indicators across systems. The syndromic indicators cover broad diagnoses of diarrhea, vomiting, and gastroenteritis that are not specific to rotavirus infection. Also, the definitions of these indicators and hence their specificity varied by system.

Limitations

Potential confusion exists between various causes of changes in syndromic data that occur at similar times. In our study, we highlighted the big changes in data provision for the NHS 111 system in September 2013, shortly after the introduction of the rotavirus vaccine. Modeling the effect of 2 or more changes that happen concurrently is difficult. Another limitation when using aberration detection methods such as RAMMIE, which explicitly model seasonality, is implementing changes to models in real time. The full effect of a change in seasonality cannot be modeled until at least 1 year after the public health intervention. This delay is a considerable disadvantage in a system that is required for ongoing surveillance.

A further limitation in this study was the assumption of a single point in time when the intervention took effect. The effect of an intervention, such as the introduction of a vaccine, may gradually increase or decrease as the vaccine’s effectiveness or uptake changes. More sophisticated models could apply a weighting based on time since introduction.

Practice Implications

Failure to account for major public health interventions in surveillance models can negatively affect the ability of surveillance systems to detect events. If background activity decreases, and thresholds are unadjusted, then an emerging outbreak of disease takes longer to detect or is undetected. In addition, failure to account for changes in seasonality can result in periods of false alarms, when thresholds decrease but activity is stationary, thereby rendering the surveillance system less effective.

As syndromic surveillance systems become more established, they will likely encounter large changes that require recalculation of thresholds, caused by either public health interventions or other factors, such as changes in the health care systems underpinning the syndromic surveillance system.²⁷ Therefore, adjustments to syndromic surveillance systems such as those we described in this study are needed. PHE uses syndromic indicators for diarrhea, vomiting, and gastroenteritis to detect a range of short-term health threats, including rotavirus, norovirus, and cryptosporidium outbreaks.^31-33 Consequently, if baselines are too high because the background incidence of rotavirus is lower than the preintervention period, then the capacity of these systems to detect outbreaks of gastrointestinal illness is reduced. Improvements to the RAMMIE model for syndromic surveillance will, therefore, increase its effectiveness at detecting public health threats.

Our study highlights one aspect of public health interventions that is less well covered in the literature, namely, the effect on ongoing surveillance systems. Furthermore, we demonstrated this effect with a real-world example that did not rely on the modeling assumptions inherent in simulation and scenario studies. Also, the example of the national introduction of a rotavirus vaccine illustrates the complexities that arise when an intervention results in a change in the seasonality of surveillance indicators. The lack of adjustment to baselines can result in a reduced confidence in surveillance systems and a failure to detect outbreaks. Therefore, we recommend that in the weeks after introduction of a major public health intervention, surveillance systems should be checked for a step-change in underlying rates and that, a year after the introduction, they should be checked for a change in the seasonality of rates.

Alternative surveillance detection methods do not require as much historical data as do regression model methods such as RAMMIE. For example, a control chart method usually creates a simple baseline based on the activity of one variable during the previous few days. Therefore, a simple control chart approach that does not explicitly model seasonality does not require seasonality adjustments postintervention.

ORCID iD

Roger Antony Morbey, PhD https://orcid.org/0000-0001-8543-477X