Designing and executing a functional exercise to test a novel informatics tool for mass casualty triage

Sara B Donevant; Erik R Svendsen; Jane V Richter; Abbas S Tavakoli; Jean B r Craig; Nicholas D Boltin; Homayoun Valafar; Salvatore Robert DiNardi; Joan M Culley

doi:10.1093/jamia/ocz087

. 2019 Jun 27;26(10):1091–1098. doi: 10.1093/jamia/ocz087

Designing and executing a functional exercise to test a novel informatics tool for mass casualty triage

Sara B Donevant ^1,^✉, Erik R Svendsen ², Jane V Richter ¹, Abbas S Tavakoli ¹, Jean B r Craig ³, Nicholas D Boltin ⁴, Homayoun Valafar ⁴, Salvatore Robert DiNardi ¹, Joan M Culley ¹

PMCID: PMC6748810 PMID: 31246255

Abstract

Objective

The testing of informatics tools designed for use during mass casualty incidents presents a unique problem as there is no readily available population of victims or identical exposure setting. The purpose of this article is to describe the process of designing, planning, and executing a functional exercise to accomplish the research objective of validating an informatics tool specifically designed to identify and triage victims of irritant gas syndrome agents.

Materials and Methods

During a 3-year time frame, the research team and partners developed the Emergency Department Informatics Computational Tool and planned a functional exercise to test it using medical records data from 298 patients seen in 1 emergency department following a chlorine gas exposure in 2005.

Results

The research team learned valuable lessons throughout the planning process that will assist future researchers with developing a functional exercise to test informatics tools. Key considerations for a functional exercise include contributors, venue, and information technology needs (ie, hardware, software, and data collection methods).

Discussion

Due to the nature of mass casualty incidents, testing informatics tools and technology for these incidents is challenging. Previous studies have shown a functional exercise as a viable option to test informatics tools developed for use during mass casualty incidents.

Conclusion

Utilizing a functional exercise to test new mass casualty management technology and informatics tools involves a painstaking and complex planning process; however, it does allow researchers to address issues inherent in studying informatics tools for mas casualty incidents.

Keywords: chlorine exposure, disaster, functional exercise, informatics, mass casualty incident

INTRODUCTION

Due to the nature of mass casualty incidents (MCIs), testing informatics tools and technology for MCIs can be difficult. First, there is no readily available victim population or identical setting for validation and usability testing and using untested informatic tools during a real-life MCI may potentially place patients’ lives at risk.¹ Previous studies have shown the functional exercise (FE) as a viable option for 1) testing a wireless electronic health record,²^,³ 2) evaluating instruction and critical thinking,⁴ 3) comparing human actors to high-fidelity simulators,⁵ 4) testing equipment and tools in the field,^6–8 and 5) comparing triage systems.⁹ Following the success of these studies, we decided to use the FE methodology to test the Emergency Department Informatics Computational Tool (EDICT), an informatics tool designed to efficiently and accurately triage patients exposed to chemicals.¹⁰ The purpose of this article is to describe the process of designing, planning, and executing an FE to accomplish the research objective of validating the novel informatics tool, EDICT.

During any MCI, emergency departments (EDs) are overwhelmed by the sheer number of patients, the severity of injuries, and limited staff to triage and care for patients, all significant public health concerns.^11–13 MCIs involving chemicals greatly increase the complexity of triaging, treating, and managing patients exposed to an unknown chemical, which was the situation on January 6, 2005 in Graniteville, SC. A train derailment released chlorine into the surrounding area killing 9 people and sending hundreds to the local EDs with no forewarning of the incident or the chemical involved.¹⁴^,¹⁵ This lack of knowledge left ED providers without appropriate guidance for the chemical involved and evidence-based triage methods for the arriving victims. The Graniteville incident demonstrated the need for a chemical MCI triage method that harnesses the power of informatics to provide early detection of an MCI, to identify an Irritant Gas Syndrome Agent (IGSA), and to deliver accurate and efficient ED triage to improve patient outcomes and potentially save lives.¹⁵^,¹⁶ Informatics tools have the potential to improve these outcomes through the use of integrated informatics tools specifically designed for these ED purposes and capable of providing situational awareness to emergency responders.

Emergency department informatics computational tool

Due to the lack of IGSA informatics tools for MCIs,¹⁶ the research team embarked on the development of EDICT, a native mobile application that incorporates an artificial intelligence (AI) algorithm to provide decision support in triaging and treating patients exposed to an IGSA. The development of EDICT occurred using a robust and systematic approach to produce a solid chain of evidence. First, the data for 298 patients from 1 local ED (ie, 198 exposed to chlorine and 100 not exposed for comparison) was abstracted using Research Electronic Data Capture (REDCap), a secure web-based research platform.¹⁷^,¹⁸ Next, 3 clusters of signs/symptoms (ie, respiratory, chest discomfort, eyes/nose/throat) were found to be statistically significant with a chlorine exposure (p < .010) and provided the basis for the algorithm to guide the development of EDICT.¹⁷^,¹⁹ The algorithm was tested using a computer simulation exercise with 26 experienced nurses and was found to be more precise (weighted Kappa-0.82, CI 0.78–0.85) and accurate (weighted Kappa-0.81, CI 0.77–0.85) than traditional triage methods (weighted Kappa-0.73, CI 0.69–0.77; 0.32 CI 0.26–0.37).²⁰ Once the research team’s computer scientists incorporated the validated algorithm into EDICT, the team was ready to assess the accuracy and efficiency of EDICT using a FE.

EDICT is a mobile application designed to triage and manage patients during a chemical MCI using a client-server transaction model.²¹ Users interacted with EDICT through unique modes, kiosk mode and nurse mode. Kiosk mode is designed for patient data entry while nurse mode utilizes the AI algorithm to provide decision support on exposure status and triage recommendations. Furthermore, EDICT’s mobile interface allowed nurses to move about the ED while remaining connected to the decision support system for situational awareness information available through summary screens.

OBJECTIVE

Due to the limited patient population (ie, patients exposed to chlorine) and lack of access to real-time MCIs, the research team determined there was a need to conduct an FE for the testing and validation of EDICT. Therefore, the objective is to describe our methodological process of designing and implementing an FE to serve as a guide for other research involving informatics tools that lack a readily available population for testing. The 3-year EDICT planning and development process identified needs for designing and implementing an FE to meet the rigorous research requirements. We describe the 5 key components of the FE: 1) contributors, 2) venue, 3) hardware, 4) IT elements, and 5) data collection methods.

MATERIAL AND METHODS

The university Institutional Review Board and the local hospital reviewed and approved the study as exempt from the Protection of Human Subjects Regulations. The research team developed rigorous protocols to assure validity, reliability, and reproducibility of EDICT. This commitment to research integrity contributed to the complexity and considerations of the FE design and execution. For example, the research team used multiple data collection methods (ie, client-server model, REDCap, video, and paper surveys) to ensure a robust research data set.

The FE included a simulated ED area complete with patient kiosk stations and nurses’ stations to triage patients. The simulated patients were given standardized information cards with the relevant victim details from the abstracted and de-identified clinical data. During the FE, as the patients arrived in the ED, they were quickly observed by a triage nurse to identify those patients in need of life-threatening intervention and immediate processing. Noncritical patients proceeded to the kiosk area where they entered data into EDICT via preprogrammed android tablets. Next, patients proceeded to an area where they were summoned for evaluation and triage by a nurse. Using EDICT, the nurse accessed the kiosk data in real-time and reviewed the AI recommendation for exposure status and triage recommendation. After the nurse confirmed or modified the decision support recommendation, the patient exited the ED. Figure 1 provides the layout of the simulated ED triage area.

Figure 1. — Simulated ED for the testing of EDICT.

Contributors to the FE methodology

External advisory committee

The research team selected an External Advisory Committee (EAC), which included 19 individuals representing organizations within the community as well as national experts to assist in the design of EDICT and the FE planning and execution. Experts from nursing (n = 7, 35%), medicine (n = 1, 5%), toxicology (n = 1, 5%), military (n = 1, 5%), emergency response (n = 4, 20%), policy/community (n = 4, 20%), and law enforcement/health and safety (n = 2, 10%) were involved. Beginning in 2015, the EAC assisted the research team in assessing the needs (ie, venue, participants, IT, data collection) to successfully plan and execute the FE.

FE participants

Key FE coordinators and facilitators. Key personnel involved in the FE execution met monthly before the exercise to plan the logistics and data collection methods. Early in the design process, it was crucial to identify and define the essential roles. Based upon the needs assessment, the team identified 7 key positions: 1) exercise research director, 2) safety officer, 3) exercise controller, 4) information technology (IT) coordinator, 5) student coordinator, 6) exercise observer supervisor, and 7) registration in/out coordinator. Job action sheets (JAS) were developed which described the role, duties, and chain of command for each position, as well as the information presented in the Just-In-Time (JIT) training the morning of the FE. The responsibilities for the 7 Key FE roles are described in Supplementary Appendix A.

Roles for patients, ED nurses, and support staff. Over 390 individuals participated in the FE. Junior and senior nursing students (n = 298, 76%), faculty (n = 41, 10%), and staff (n = 15, 4%) from 2 universities participated as patients, ED nurses, and support staff. Additional nurses and individuals from the community (n = 29, 7%) were recruited to assist in various roles as well as the Reserve Officers’ Training Corps (n = 10, 3%) who assisted with logistics. Before the FE, students completed online emergency preparedness training modules in Blackboard Learn (Washington, DC).

The students portrayed the 298 patients (referenced as patients throughout this article). Before the FE, each student was randomly assigned a number that correlated with a patient ID and a time to enter the ED in groups modeled to replicate a surge associated with an MCI (ie, starting with a slow and steady stream and increasing to create a surge).

The morning of the FE, all participants received a packet at registration that contained essential information including JAS and schedule. Each student participant also received a patient ID sticker and patient card with de-identified patient data. The patient card included fictitious name, address, and date of birth; patient ID, bar code of patient ID, chief complaint, and description of how to play the assigned patient (ie, coughing, leaning forward to breathe, etc.). Before the start of the FE, JIT training and a safety briefing were provided to all participants specific to their role.

Faculty served as kiosk helpers (n = 4, 10%), primary triage nurses (n = 2, 5%), supervisor nurses (n = 2, 5%), secondary triage nurses (n = 10, 24%), observers (n = 19, 46%), and ED expeditors (n = 4, 10%). Additional ancillary staff assisted with a variety of supporting roles such as directing students, food distribution, registration, communications, information technology, and clean-up.

Key IT methodology components of the FE

Venue

When selecting the venue, the needs of the FE dictated that sufficient space was essential to accommodate a large number of participants including parking and adequate facilities (ie, restrooms, audio-video equipment, kitchen, and holding area). The research team decided on an indoor venue to accommodate any weather condition and was allowed access to the university football stadium and its 11 000 square foot banquet facility for the FE (Figure 1) as well as 6 classrooms for JIT training. Throughout the planning process, the research team visited the venue to plan the layout and logistics of the FE. Other FE needs included the location of restrooms, electrical outlets, tables/chairs, and audio-visual equipment.

Hardware/software architecture and standards

EDICT’s user interface is a mobile application running on an android operating system (OS). Android OS was selected because of the flexibility of open source development, the ability to customize the user interface, and ease of adoption. Android OS accounts for nearly 65% of mobile devices and tablet market share, meaning users would most likely have previous experience using android applications.²² EDICT was developed using the Android Studio integrated development environment (IDE) architecture running on Linux Ubuntu 16.04.3 LTS. Bitbucket was integrated as the software version control, and GIT was used for local and remote repository access.

The Samsung Galaxy Tab A, model SM-T550 with Android Marshmallow 6.0.1, was selected for the FE. Based upon the need assessment for the FE, the research team’s computer engineers found Samsung’s larger 10-inch screen allowed for favorable human-computer interaction (HCI) elements, such as larger text and buttons, and provided 12 hours of use on a single charge. However, the research team assured redundancy by having additional tablets and charging capabilities. The Galaxy’s 16 GB with 2 GB of RAM provided sufficient data storage, backup, and expanded memory for quicker access to patient information via the real-time client-server communication. Also, the research team purchased protective cases to safeguard the integrity of the tablets in the event of dropping or falling. Another need was barcode scanning, which is a commonly used feature within healthcare facilities. The Samsung Galaxy tablet included a camera for barcode scanning.

The on-site server allowed for real-time bidirectional communication with EDICT to receive patient data from the kiosk stations and, based upon the tested AI algorithm discussed above, provided AI decision support to the nurse stations.²¹ The server used the patients’ symptoms (ie, type and number) and vitals (ie, heart rate, respiratory rate, oxygen saturation) along with the patients’ geographical locations to provide the secondary triage nurse with a recommended exposure status (ie, exposed, potentially exposed, not exposed) and triage category (ie, exit, monitor, urgent, immediate). The interoperability allowed for data integrity in the event of a hardware failure. The stadium’s bandwidth allowed for real-time and secure data collection/transmission as well as storage to accommodate EDICT data and research desktop computers.

Data collection methods

EDICT data collection

A critical data collection need was the discrete data entered directly into EDICT by the patients and nurses. User data was collected in real-time and stored on a Linux Apache server Ver. 2.4.18 using a MySQL Ver. 14.14 relational database. The database schema is shown in Figure 2.

Figure 2. — Relational database schema used to store user data collected from EDICT.

Observer data collection

The research team determined the need to observe the following locations: 1) kiosk stations (ie, where patients entered data into EDICT), 2) Nurse stations (ie, primary triage, secondary triage, and supervisor), and 3) entire simulated ED. When designing the observer data collection tools, the research team was driven by the need to capture information electronically in a quick, easy-to-use method. REDCap was chosen as the data collection tool due to its ease of use and ability to function in an offline mode when the internet is not available.¹⁸ The user interface requires a web browser for the online mode and the REDCap Mobile App²³ with the data collection tool downloaded and configured for the offline mode to enhance interoperability.

When developing the data collection tools, the research team identified a list of necessary features and data elements for capture, which guided the data collection design in REDCap. Supplementary Appendix B lists the data collected with these tools.

Patients wore a readily visible ID number which allowed the observers to view and record the number when documenting observations without interacting with the participants (ie, patients, nurses) or interfering with FE activities. Similarly, Kiosk and Secondary Triage stations were labeled with unique letters corresponding to the EDICT tablets. Figure 3 is a representation of a Kiosk Observer Tool. The observers were instructed to interface only with the Exercise Observer Coordinator or the Exercise Controller when needing assistance. Prior to the FE, the Exercise Observer Coordinator installed the REDCap mobile app on 15 Apple iPads which were configured so no login was required and had the ability to be used in the event of no or poor internet availability.

Figure 3. — Example of Kiosk Observer Collection Tool.

Figure 4. — Secondary triage nurses’ station with patients, observers, and IT staff.

Figure 5. — Student with GoPro^TM at kiosk station.

Participant data collection

All participants were asked to complete paper surveys at the end of the FE that included a combination of 5-point Likert questions and open-ended questions. This process provided feedback on the participants’ perception of their role in the FE. Data collected on this survey is listed in Supplementary Appendix C.

HCI data collection

An HCI expert developed paper surveys for the participants to record their perception of EDICT’s usability. We utilized the widely used and validated System Usability Scale (SUS) with a 5-point Likert rating scale as well as 3 yes/no questions, and 1 open-ended question (see Supplementary Appendix D).²⁴

Video data collection

The research team opted to use videos to collect additional data. A videographer was hired to record the entire FE using multiple cameras focused on different areas of the simulated ED. In addition, students and nurses were randomly assigned to wear a GoPro to provide visualization of the students’ and nurses’ activities.

RESULTS

Planning a large-scale FE required substantial time and attention to details as the research team only had this 1 opportunity to complete the aims of the research using the FE. The prior 3 years of planning and EDICT development culminated in the careful execution of this FE. The research team learned several lessons throughout the planning and execution process, which are shared below to assist future researchers with developing an FE to test informatics tools.