Novel public health risk assessment process developed to support syndromic surveillance for the 2012 Olympic and Paralympic Games

Abstract

Background

Syndromic surveillance aims to provide early warning and real time estimates of the extent of incidents; and reassurance about lack of impact of mass gatherings. We describe a novel public health risk assessment process to ensure those leading the response to the 2012 Olympic Games were alerted to unusual activity that was of potential public health importance, and not inundated with multiple statistical ‘alarms’.

Methods

Statistical alarms were assessed to identify those which needed to result in ‘alerts’ as reliably as possible. There was no previously developed method for this. We identified factors that increased our concern about an alarm suggesting that an ‘alert’ should be made.

Results

Between 2 July and 12 September 2012, 350 674 signals were analysed resulting in 4118 statistical alarms. Using the risk assessment process, 122 ‘alerts’ were communicated to Olympic incident directors.

Conclusions

Use of a novel risk assessment process enabled the interpretation of large number of statistical alarms in a manageable way for the period of a sustained mass gathering. This risk assessment process guided the prioritization and could be readily adapted to other surveillance systems. The process, which is novel to our knowledge, continues as a legacy of the Games.

Introduction

Syndromic surveillance is defined as ‘a real-time (or near real-time) collection, analysis, interpretation and dissemination of health-related data to enable the early identification of the impact (or absence of impact) of potential human or veterinary public health threats that require effective public health action’.¹ The broad aims of the syndromic surveillance systems are to: provide early warning of health protection incidents; provide real-time estimates of the size and extent of health protection incidents; and provide reassurance about lack of impact of incidents e.g. during mass gatherings.¹ As this surveillance utilizes data on non-specific health complaints such as ‘vomiting’, ‘influenza-like-illness’ or ‘rash’, it is particularly useful for the surveillance of unexpected threats that may be associated with mass gatherings. During 27th July until 12th August and 29th August to 9th September 2012 the United Kingdom (UK) hosted the London 2012 Olympic and Paralympic Games, respectively. As part of the public health preparations for the Games, the Health Protection Agency (HPA, now Public Health England (PHE)) further developed its suite of syndromic surveillance systems² to meet the enhanced surveillance requirements of the Games.

By the time of the Games, the HPA syndromic surveillance service monitored in and out of hours general practitioner consultations, calls for remote health advice and attendances at sentinel emergency departments.^3,4 Although much has been published on the methods for statistical analyses for syndromic surveillance systems,^5–7 there has been little description of how in practice the statistical alarms generated through these systems should be assessed, prioritized and communicated in order to facilitate appropriate and effective public health action.

We describe the development and use of a novel public health risk-assessment tool to ensure those leading the public health response to the Games were alerted to the statistical alarms considered to be of public health importance, but were not inundated with multiple daily statistical alarms.

Methods

Syndromic surveillance during the Olympics

At the time of the Games, two of the surveillance systems were well established and had several years’ worth of historical data (the remote health advice surveillance and the GP in hours syndromic surveillance) while the Emergency Department Syndromic Surveillance System (EDSSS) and the GP out of hours surveillance System (GPOOHSS) were being developed and expanding in geographical coverage across England. Thus the amount of baseline historical data varied between surveillance system, and the statistical methods used to identify unusual increases in call/consultation rates also differed for this reason. For the established systems, a combination of control chart and the ‘Stroup’⁶ methods were used. For the systems with less historical data, Shewhart control charts were used,⁵ with the baselines derived from the previous few weeks. All statistical methods were automated. In addition to the automated statistical analyses, the HPA real-time syndromic surveillance team (ReSST) also visually examined daily graphs of key syndromes for London and other English regions.

The outputs of the daily analyses and interpretation of syndromic data (described below) were included in a daily syndromic surveillance bulletin which formed part of a wider HPA syndromic surveillance assessment and the overall HPA situation report (SITREP) for the Games.

Risk assessment terminology

In order to standardize the terminology used within the public health risk assessment, and thereby reduce confusion within the team and between other HPA departments and outside the organization, we developed a terminology to define the different aspects of the risk assessment creating the terms ‘signal’, ‘alarm’ and ‘alert’ to describe the indicator being monitored, detection threshold(s) and public health communication respectively (Table 1).

Table 1.

Terminology to describe aspects of the risk assessment.

Term	Description
Signal	Refers to a syndrome being monitored by geography  and system (e.g. GP consultations for influenza-like illness in the North West Region on first August 2012)
Alarm	A statistically significant increase in a signal (e.g. a  statistically significant rise in GP consultations for influenza-like illness in the North West Region on first August 2012)
Investigation	The process by which the HPA syndromic team  investigates and assesses the alarm to determine whether it has the characteristics that may warrant communication outside the team for assessment of potential public health action
Alert	Communication of an alarm and investigation, i.e.  unusual syndromic activity to those outside the syndromic surveillance team, which may warrant further public health action

Development of the public health risk assessment

During the planning for the Games, those leading the syndromic surveillance service, in conjunction with those leading the HPA public health response to the Games, agreed that it was not useful or manageable for the Game leaders within HPA and externally to be presented with multiple statistical alarms each day. Rather, a public health risk assessment should be developed to prioritize those alarms considered to be of public health importance. We developed this risk assessment tool incrementally by working with statistical, epidemiological, syndromic surveillance scientists and public health colleagues. The starting point for initiating the risk assessment was either an automatically generated statistical alarm from one of the national syndromic systems or an epidemiological assessment of a time-series graph suggesting an increase in a signal (e.g. vomiting in London) with or without a statistical alarm.

The public health risk assessment tool

The aim of developing the risk assessment tool was to identify those alarms which needed to be prioritized to result in an ‘alert’, as reliably and consistently as possible. There was no previously documented method for this (especially encompassing the wide variety of syndromes monitored), and the team identified the factors that increased our concern about an alarm and suggested that an ‘alert’ should be made. Not all factors would apply for all alarms and signals, but we developed a framework for risk assessment which is novel in syndromic surveillance, to our knowledge.

The risk assessment was divided into two stages:

1. An assessment of the statistical alarm by the syndromic surveillance scientist

   For each alarm the risk assessment encompassed:

   1. the size of the excess over expected (baseline) levels and how unusual this was for the syndrome;
   2. whether the alarm was a repeat alarm within the previous few days;
   3. whether the alarm was different to the national trend;
   4. whether a similar alarm has occurred independently over several different syndromic systems.

   This assessment was ‘scored’ using a Likert-type scale (available from authors) and those alarms scoring highly were investigated further with a consultant epidemiologist.

2. The alarms meeting the scoring criteria on the initial risk assessment (or an alarm of particular concern to any member of the team) were discussed by the scientist with a consultant epidemiologist and a further ‘final’ risk assessment performed

   This final part of the risk assessment included:

   1. assessment of the type of syndrome and the time of year—was there anything which gave concern? (e.g. a rise in influenza-like illness in summer; a rise in bloody diarrhoea);
   2. anything in the geographical distribution which suggested a local incident;
   3. evidence of age group clustering;
   4. assessment of evidence of increased severity of a syndrome (e.g. a greater proportion of Emergency Department attendances for respiratory disease being admitted).

Any alarm scoring highly on the final risk assessment (or any alarm of particular concern for any of the team) would result in an ‘alert’, that is, a communication from the syndromic surveillance team to other relevant public health colleagues. It was agreed that the type of alert would depend on the assessment of seriousness of the rise in the syndrome(s), and could range from a mention in the daily syndromic bulletin, to an urgent phone call and discussion with the Olympics Incident Director (e.g. in the event of an apparent emerging respiratory syndrome and a shift to increased severity of outcomes).

This risk assessment process was put into operation in late March 2012, well in advance of the Games so as to gain experience in both the production of the outputs and the risk assessment process. The team, as part of their preparation, met periodically to review the scoring for the risk assessments and to ensure consistency. This included independent scoring of example alarms by the consultant epidemiologists and discussion of any apparent differences.

All of the scores from the risk assessments were recorded in a database along with any relevant notes (e.g. what action was taken, why no alert was felt warranted).

We additionally, and at the end of the Games, informally asked team members approximately how long the scoring took and how much it helped or hindered the team. No formal time estimation was made.

Results

Syndromic surveillance alarms

The public health risk assessment process was applied throughout the period of enhanced surveillance for the Games (2 July to 12 September 2012). A summary of the outcomes from the risk assessments is shown in Fig. 1. During the period, 73 daily syndromic surveillance reports were produced and fed into the overall HPA daily SITREP.

Fig. 1.

Assessment of daily signals to identify activity of public health concern, 2nd July to 12th September 2012.

Over the same period, a total of 350 674 signals were analysed representing a daily mean of 4803. There were 4118 alarms generated from automated statistical analyses, representing a mean of 56 alarms per day interquartile range (33, 68). The most alarms generated on one day was 254 on 16th July. (A nation-wide increase in an indicator like asthma can lead to alarms in a large number of locations.)

A total of 122 alerts were issued during the period, interquartile range (0, 4). (Separate alerts are counted for each location affected.)

Risk assessments: interpretation and dissemination

The daily Olympic syndromic surveillance output (bulletin) contained: an overall summary and key messages from the four surveillance systems; a table including counts of clinical consultations for selected signals and systems (Fig. 2); and a number of time-series graphs covering selected indicators for each English region.

Fig. 2.

Syndromic signals overview table displaying signals, and the corresponding number of cases each day, for selected syndromes over a 14-day period. Signals were highlighted in green (shown as white here) if they were typical for the time of year, amber if there (shown as grey) was a statistically significant increase in activity ‘an alarm’ and red (shown as black) if there was considered to be a potential public health impact and ‘an alert is being issued’. Excess cases: - the number of cases above the upper threshold (shown for the last date in the date range only).

Risk communication

Of the 122 alerts issued during the Olympics reporting period, the majority were of the form of repeated messages contained in the daily bulletin, only four alerts necessitated a telephone call to public health professionals. Examples of two risk assessments, one resulting in an alert and one not, are provided in Fig. 3, and examples of written alerts in the daily syndromic bulletin are included in Fig. 4.

Fig. 3.

Examples of the risk assessment process for an alarm which resulted in an alert , and for an alarm which did not result in an alert.

Fig. 4.

Examples of written alerts within daily syndromic surveillance Olympic report.

Although no major health incidents were detected, we did identify an unusual increase during June 2012 that resulted in alarms in the asthma/difficulty breathing signal across several syndromic systems.⁸ This resulted in an alert that was telephoned through to the HPA Olympic coordinating committee and other relevant public health departments within the HPA e.g. respiratory and environmental teams. Following a review of available evidence respiratory pathogens and environmental pollutants were discounted as principle causes, though the increase may have been associated with a prolonged cold and wet spring and an increase in fungal spores.

A further example of an ‘alert’ occurred during late July 2012 when a period of hot weather triggered a number of alarms associated with heat-related consultations such as heat stroke. The signals were prospectively analysed to establish whether there was evidence of changes in the severity of presenting symptoms, or if the age distribution had shifted. This analysis determined that the number of heat/sun stroke cases was within levels expected during periods of hot weather, and this alert was classified as of no significance to the Games.

Acceptability/performance of risk assessment

The scientists estimated that the assessment took on average a couple of minutes per alarm (and the second stage assessment a couple of minutes further).

The risk assessment process was appreciated by the team as a structured and systematic way of assessing the alarms and ensuring a clear record of the decision making process.

Discussion

Main findings of this study

This paper describes the development and operationalization of a novel public health risk assessment process for syndromic surveillance data gathered during the Summer Olympics and Paralympic Games 2012. The risk assessment process provided a structured framework that streamlined the workload, improved the consistency of epidemiological interpretation of the data, and appropriateness of alerts issued to key groups during the Games.

Although we, fortunately did not experience a major incident during the Games, the syndromic systems did detect an increase in several syndromes (e.g. increase in pertussis consultations and increase in heat-related syndromes during a period of increased temperatures) demonstrating the ability of the systems to detect such increases. The systems are routinely used outside mass gatherings for monitoring the impact of infections with seasonal peak activity such as norovirus and seasonal influenza and in assessing the public health impact of incidents such as flooding.^9,10

What is already known of this topic

One of the purposes of the routine statistical analysis of syndromic surveillance data is to support the timely detection and reporting of significant changes in trends of defined syndromic indicators so as to facilitate the implementation of appropriate public health response. Syndromic surveillance is not concerned with a single disease or system and has multiple and sometimes diverse aims (e.g. detecting the onset of influenza-like illness in the community, assessing the impact of cold weather). Alarms generated from automated statistical analyses supported the identification of syndromes that should be investigated more robustly, but this investigation requires the input of personnel with relevant public health knowledge and experience in order to prioritize the reporting of alarms of concern/public health relevance.

What this study adds

The key role of the real-time syndromic surveillance service during the Games was, in the absence of any overt major incidents being detected or reported, to provide some reassurance of the absence of impact (i.e. outbreaks) in the population linked to the Games. The importance of this assurance function should not be underestimated, particularly during a period of increased media scrutiny, when an informed response to rumours and press speculation is required. The value of this was highlighted in the review of the Games response.¹¹

The inclusion of a risk assessment process ensured that additional public health interpretation was undertaken which reduced the high volume of daily (un-interpreted) statistical alarms (which would be of little use to incident directors) to much smaller numbers of appropriate, meaningful alerts to key public health groups.

The use of a novel risk assessment process enabled the scrutiny and interpretation of a large number of statistical alarms in a manageable and auditable way during the Olympics and Paralympic Games. The risk assessment was meant to facilitate the prioritization and reporting of important signals, rather than be prescriptive, and it could be readily adapted to other surveillance systems/situations. We are unaware of such a process linking the statistical alarms generated through a syndromic surveillance system with a contemporaneous public health focused risk assessment to improve the specificity of agreed public health action(s). The use of the risk assessment process continues as an integral part of the national syndromic surveillance service, and the systems are maintained as part of a legacy of the Games. The fact that the team was involved in the development of the risk assessment tool would almost certainly have engendered a sense of ownership which should increase its acceptability.

Limitations of this study

Although the process has not been formally validated, in the lead up to the Games the work to refine the process and the tools that underpin it led to improvements in the inter-rater reliability (consistency) and face validity of the process.

Anecdotally, the number of alerts issued during the Games period was higher the number routinely communicated outside of the Games period. This probably reflects the precautionary approach adopted by the team in keeping with the intense scrutiny and heightened awareness during the Games period and the role of the syndromic surveillance service.

This novel syndromic public health assessment process should be continually refined and improved through periodic audits, and also provides auditable documentation of the real-time decision making in the event of a review of missed incidents or false alarms. The systems and the risk assessment tool are evolving, with amendments being made to the statistical thresholds¹² to ensure the alarms are more sensitive. Further work is planned, using Bayesian techniques, to explore whether and which steps in the process could be automated, and which are dependent on public health knowledge and skills.

Funding

G.E.S., A.E. and R.M. receive support from the National Institute for Health Research Health Protection Research Unit in Emergency Preparedness and Response and J.H. from the National Institute for Health Research Health Protection Research Unit in Gastrointestinal Infections. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, the Department of Health or Public Health England.