Risk Assessment During the Pan American and Parapan American Games, Toronto, 2015

Abstract

During the summer of 2015, the Pan American and Parapan American Games took place in the Greater Toronto area of Ontario, Canada, bringing together thousands of athletes and spectators from around the world. The Acute Care Enhanced Surveillance (ACES) system—a syndromic surveillance system that captures comprehensive hospital visit triage information from acute care hospitals across Ontario—monitored distinct syndromes throughout the games. We describe the creation and use of a risk assessment tool to evaluate alerts produced by ACES during this period. During the games, ACES generated 1420 alerts, 4 of which were considered a moderate risk and were communicated to surveillance partners for further action. The risk assessment tool was useful for public health professionals responsible for surveillance activities during the games. Next steps include integrating the tool within the ACES system.

The 17th Pan American and Parapan American Games (hereinafter, PPAG) were held in Toronto, Ontario, Canada, from July 10 to 26, 2015 (the Pan American Games), and from August 7 to 15, 2015 (the Parapan American Games). More than 12 000 athletes and team officials from 41 countries attended. The PPAG also drew >16 000 volunteers and an estimated 250 000 visitors to the Greater Toronto area and associated PPAG areas.¹

Mass-gathering events such as the PPAG in Toronto can pose increased risks to public health because they have the potential to amplify the transmission of infectious diseases, cause and exacerbate respiratory or gastrointestinal outbreaks, and even serve as a potential target for mass casualty violence or bioterrorism.^2,3 In addition, large-scale events may restrict access to patients during a medical emergency^4,5 and can threaten the stability of a region’s health system.² Timely and reliable surveillance is necessary to address the increased health risks and potential terrorist threats to ensure participant and spectator safety during large-scale events.^3,6

Syndromic surveillance can fulfill several functions—including detecting changes in patterns of disease, tracking proxy syndromes during known outbreaks (eg, influenza season, food recalls) to track the spread of disease, and monitoring for and possibly detecting instances of bioterrorism.⁷ The Acute Care Enhanced Surveillance (ACES) system is a syndromic surveillance system created by Kingston, Frontenac and Lennox & Addington Public Health (hereinafter, KFLA) in 2015 that gathers emergency department (ED) data in real time across Ontario. The system currently has >150 users in the province. Most EDs are open 24 hours per day and act as an open point of access for health care. EDs can act as a sentinel health service for monitoring potential health events in community residents and nonresidents, which is important when monitoring an international mass gathering. Additionally, international visitors can carry and transmit infectious diseases that are not common or easily recognized by health care professionals in the country in which a mass-gathering event is occurring.³ Higher-acuity visits (eg, injuries and trauma) that could be associated with a terrorist event at a mass gathering can also be captured when ED activity is monitored.⁸

ACES collects comprehensive hospital visit triage information from 124 acute care hospitals across Ontario to monitor 84 distinct syndromes. Visits are sorted into syndrome categories based on the patient’s chief complaint, through use of natural language-processing algorithms. Statistical algorithms detect variations and trends in these hospital visits. At the time that the PPAG started, the ACES system was receiving data from 53 of the 61 hospitals in the geographic footprint of the PPAG events.

This article describes the development of a risk assessment tool for interpreting surveillance alert data from the ACES system and provides preliminary results on use of the risk assessment tool by the ACES team during the PPAG.

Methods

Recognizing that even highly optimized aberration detection methods generate false positives that are costly and time prohibitive to investigate, a further risk screen was developed for use during the PPAG at the request of ACES system users and provincial surveillance partners. We modeled our risk assessment tool after the risk assessment process developed by Public Health England for syndromic surveillance during the London 2012 Summer Olympics⁹ and based its content on the surveillance experiences of previous mass-gathering events.¹⁰ We determined the process for using the tool, and its context in the overall surveillance strategy and communication cycle, months before the PPAG began.

We created the paper-based risk assessment tool (Figure) to help ACES users assess the severity of each alert raised by the system during the PPAG to determine when public health experts should take further and escalating action. The tool asks 15 questions about the frequency, timing, and severity of the alert, as well as the syndrome’s potential association with a mass-gathering event. A total score ranging from 0 to 25 is assigned to each alert once all 15 questions have been answered. This score can then categorize each alert into 1 of 3 possible recommendations, ranked by acuity. An overall score of 0-14 indicates that the alert is not a risk to public health and should be documented and reviewed at the user’s discretion. A score of 15-19 indicates that the alert may be of public health concern and should be reported internally for continued monitoring. A score of 20-25 indicates that the alert is likely of public health concern and requires immediate action.

Figure.

Risk assessment tool developed to evaluate ACES alerts during the Toronto Pan American and Parapan American Games, July 7–August 18, 2015. Abbreviations: ACES, Acute Care Enhanced Surveillance; CDC, Centers for Disease Control and Prevention; CNPHI, Canadian Network for Public Health Intelligence; CTAS, Canadian Triage and Acuity Scale; CUSUM, cumulative sum; FSA, forward sortation area; GOARN, Global Outbreak Alert and Response Network; PCR, polymerase chain reaction; PHAC, Public Health Agency of Canada; PHIMS, Public Health Information Management System; SPC, statistical process control; WHO, World Health Organization.

For the PPAG, we identified 15 syndromes as being potential indicators of (1) terrorist threats (smoke inhalation, injury, trauma, burn, mental health issues, cellulitis, carbon monoxide poisoning, orthopaedic fractures), (2) spread of communicable disease (respiratory illness, fever/influenza-like illness, gastroenteritis, meningitis, sepsis), or (3) environmental/air quality-related events (asthma/chronic obstructive pulmonary disease, heat-related illness).¹¹ We used 3 aberration detection methods during this assessment: (1) a cumulative-sum control chart, which takes the current day’s syndrome count and compares it with the previous 7-day average (an alert is generated when the cumulative-sum value exceeds a threshold value of 3); (2) an extreme statistical process control method, which triggers when a daily count exceeds 3 SDs from a 14-day moving average; and (3) a trend statistical process control method, which triggers when a daily count is the sixth daily count in a row of increasing value. The 15 syndromes were monitored daily by all local public health agencies within the PPAG events area. A team of health analysts from KFLA were each assigned 2 or 3 of the 15 syndromes monitored by the ACES system daily during the PPAG. If an alert was triggered for 1 of the syndromes, an analyst alerted 1 of 2 resident physicians from the KFLA team who were responsible for investigating the alert further and deciding if the alert would be scored with the risk assessment tool. If it was decided that an alert would be scored, we also noted the amount of time that it took our resident physicians to complete the risk assessment tool.

Daily surveillance based on the ACES system began on July 7, 2015, and ran continuously through August 18, 2015. Surveillance between the end of the Pan American Games (July 26) and the start of the Parapan American Games (August 7) was still conducted daily because the same public health threats could have applied during the transition between the 2 events. The ACES system has 84 syndromes for which a chief complaint record could be assigned. Any alerts occurring for syndromes other than the 15 described here were not eligible for investigation after a preliminary assessment. It is also worth noting that the ACES system alerts in real time, through a shifting 24-hour window. This process can result in numerous alerts for the same syndrome in a short period. Therefore, if an initial alert was assigned a score with our assessment tool, subsequent alerts for the same syndrome in the same local public health agency were simply monitored for further situational awareness.

Results

During the 43-day period, ACES produced 1420 alerts for local public health agencies within the PPAG region, nearly half of which occurred in the Toronto area local public health agency. For each alert, the risk assessment process was implemented when the syndrome in question had the potential for public health concern or when ED visit counts were rising. A preliminary assessment by the KFLA health analysts identified 182 of the 1420 alerts for further investigation with the assessment tool. Scoring an alert with the tool took 2 to 5 minutes to complete. However, because the same 2 resident physicians scored the alerts during the PPAG, the time to complete the tool decreased over time.

The assessment tool scored 4 alerts as a moderate threat and identified none as requiring immediate investigation. One of the 4 moderate alerts was for carbon monoxide exposure. This syndrome was monitored as a terrorist event indicator with the potential for resulting side effects from a power outage. Typically, this syndrome averages approximately 5 visits per week across all hospitals in the ACES system; however, 1 local public health agency saw 31 visits for the syndrome in <12 hours. Because of how closely connected the ACES team was with provincial and local partners during the PPAG period, it investigated these visits locally and determined that they were related to a potential carbon monoxide exposure in a school setting. The remaining 3 moderate alerts were for asthma-related visits in 3 local public health agencies on 3 consecutive days (July 28-30). Increased asthma-related visits in the summer are common, and because these alerts did not coincide with other factors (eg, heat or smog advisories or PPAG-related events), these alerts were shared with PPAG partners for situational awareness.

Discussion

Although this study centered on a mass-gathering event during a fixed time frame and covering a large geographic footprint, we believe that these preliminary results show that our risk assessment tool can be used daily at a local level by ACES users. The minimal amount of time to complete the tool once a user becomes familiar with the questions can really pare down the number of false positives that users often say is a limitation for them in their daily use of the system. Our next step will consider how the tool can be converted into an electronic version and integrated within the ACES system so that it is easier to use and save those scores within a user’s account. This prospective feature will also allow us to evaluate how the tool is being used and how scoring decisions may differ among users and geographies.

Even though we created the tool based on experience gleaned from previous mass-gathering events and tailored it to the ACES system, we feel that its framework could easily be adapted for use by other syndromic surveillance systems for other purposes. Another future goal for our team would be to partner with other hospital triage systems outside of Canada to test how our tool could be modified and applied in different jurisdictions that may have slightly different data elements. Ultimately, the decision to act on any alert generated by a syndromic surveillance system is up to the individual user of that system. The risk assessment tool is a guide rather than a definitive ruling, and human judgment is always needed when analyzing this type of data. Further use of the tool and ongoing analysis of how local public health agencies use it is warranted to better understand where improvements can be made and to ascertain how it can improve the efficiency and workflow of ACES users.