Educating First Responders to Provide Emergency Services to Individuals with Disabilities

Abstract

Objective

Individuals with disabilities experience more negative outcomes due to natural and manmade disasters and emergencies than do people without disabilities. This vulnerability appears due in part to knowledge gaps among public health and safety emergency planning and response personnel (responders). The research assessed the effectiveness of an online program to increase emergency responder knowledge about emergency planning and response for individuals with disabilities.

Method

Researchers developed an online course designed to teach public health, emergency planning/management and other first response personnel about appropriate, efficient and equitable emergency planning, response, interaction and communication with children and adults with disabilities before, during and after disasters or emergencies. Course features include an ongoing storyline, exercises embedded in the form of “real life” scenarios, and game-like features such as points and timed segments.

Results

Evaluation measures indicated significant pre- to post-test gains in learner knowledge and simulated applied skills.

Conclusion

An online program using scenarios and simulations is an effective means to make disability-related training available to a wide variety of emergency responders across geographically disparate areas.

Introduction

Since the well-publicized negative and sometimes fatal experiences of individuals with disabilities during the September 11, 2001 terror attacks, Hurricanes Katrina (2005), and Sandy (2012), advocates and authors have highlighted the importance of inclusive, effective emergency preparedness, response and recovery services for and with individuals with disabilities.^1–4 During Hurricane Sandy at least 4,000 New Yorkers with intellectual disabilities were evacuated; 1,300 remained out of their homes for weeks,³ challenging the service provider system. Hospital transportation of patients with physical disabilities was reported to be poorly planned, putting them at risk.⁵ As in prior disasters, some public emergency shelters reportedly turned away people with disabilities,⁶ and those with disabilities in at least one shelter reported being denied freedom of movement and access to their own funds.⁷

These negative effects appear to continue despite legal protection such as the Americans with Disabilities Act (ADA) (1990)^{8, 9} which requires federal, state, local governments, and emergency responders in their employ to provide equal access to emergency services to individuals with disabilities. The U.S. Department of Homeland Security’s mitigation, response and recovery frameworks,^10–12 recent FEMA guidance,¹³ and the U.S. Centers for Disease Control and Prevention public health preparedness capabilities¹⁴ support this obligation.

Several factors may be responsible for these inadequacies. Emergency plans often assume that all Americans can walk, run, talk, hear, drive and quickly follow directions, thereby ignoring the likely adaptations needed for some with disabilities.¹⁵ Emergency responders (such as law enforcement, fire, emergency medical services [EMS] and public health personnel, emergency managers/planners, and Medical Reserve Corps (MRC)/Community Emergency Response Team (CERT) volunteer responders may lack sufficient education about disability-related emergency needs.^16–18 For example, without specialized training, law enforcement personnel may interpret an averted eye gaze as a sign of lying or suspicious behavior,¹⁹ rather than potentially typical behavior of someone with autism. And at least one EMS protocol suggests that an unsteady gait or slurred speech may imply hypoglycemia due to diabetes.²⁰ An equally plausible explanation is that the individual exhibits behaviors associated with Cerebral Palsy. Educating responders regarding disability-related emergency needs would promote equal access to emergency services and remove barriers to response efficiency,^{21, 22} enhancing safety and well-being for those with disabilities

Education and Training Options For First Responders

When responder education is provided about the emergency needs of individuals with disabilities it tends to be delivered via online or classroom-based courses, drills and guides, often focused on recommendations to enhance 1:1 interactions and communication with those with disabilities. The relatively few online responder courses related to disability appear to include mainly pre-recorded, user-paced offerings, often presented in a PowerPoint^(R) format. See the Online Data Supplement for a further review of responder training options.

The Benefits of Asynchronous, Online Educational and Training Materials

Computer-based online education is an increasingly beneficial educational method for teaching emergency responders, enhancing and reinforcing knowledge, satisfaction, and motivating learning.^23–26 Online courses offer the opportunity to educate a potentially large and widely dispersed audience ²⁷ and can provide immersive, participatory, realistic scenarios.²⁴ Appropriate response is greatly influenced by responder familiarity with the response scenario plus skill, knowledge and psychological state.²⁴ Thus, like drills, online courses can offer interactive response scenarios that simulate real life, offering the learner practice before encountering a like situation in the field.^{24, 25} The U.S. Centers for Disease Control and Prevention (CDC) notes that simulations, including those with game-like features, are so advantageous that they should be a key component of public health education.²³ An additional advantage of online courses includes the ability to repeatedly review a course and score it electronically to aid performance assessment and develop corrective action.²⁴ Finally, online training is thought to be cost effective; the U.S. Department of Defense found that computer simulations reduced training costs by ten percent when compared to in-person training.²⁵

Purpose of the Investigation

This study investigated the impact and perception of online instruction, presented via simulation-based scenarios and incorporating game-like features, on emergency responder knowledge and skills regarding the emergency needs of people with disabilities. Specifically, University of Massachusetts Medical School (UMMS) E. K. Shriver Center (Shriver Center) and Praxis, Inc. researchers designed, developed and evaluated an Internet-based distance education course (“Rescue-D”), intended to teach emergency responders to anticipate, plan for and address the needs of children and adults with disabilities before, during and after emergencies. The course consisted of four modules (a portion of a more extensive course), incorporating an ongoing storyline and scenario-based learner interactions. The research aim was to assess learner factual knowledge, applied knowledge (skill) and satisfaction with the course to determine the feasibility of this educational format for future potential development and implementation of a more extensive curriculum.

The effort is unique in that it is a first attempt known to the authors to combine scenario-based learning and game-like features in a self-paced online course to teach emergency responders about people with disabilities. This is important as work force training increasingly includes asynchronous online instruction, yet continues to require evidence-based “real world” training and application.

Methods

Rescue-D Design, Development and Technology

Design and development

Prior to development we consulted with a variety of first responders, geographic information system (GIS) mapping specialists, instructional designers and individuals with disabilities and their families. Rescue-D is a scenario-based online course incorporating a storyline, role-play, and game-like features designed to simulate real emergency situations for the learner. Set in a fictional east coast U.S. county, Rescue-D depicts events before, during and after an explosion, hurricane and flood. The learner assumes the role of “Responder Brown,” a local emergency responder, and begins by learning about the county, its residents, and physical infrastructure. The learner progresses through the course’s storyline, reviewing information, asking and answering questions, and making decisions (e.g. the best way to evacuate a family from home; whether an emergency shelter policy comports with the ADA⁸ and if a dog would be considered a “service animal”). While “interacting” with characters with and without disabilities (e.g. the boss, a colleague or someone who needs rescuing from the “flood zone”) the learner addresses disability demographics, the negative impacts of disasters on people with disabilities, appropriate 1:1 interactions and communication, evacuation, transportation and inclusive emergency planning for and with people with disabilities. The course includes a mapping tool (a map with different layers of information and related editable database) used to locate shelters, evacuation vehicles, the flood zone area, and certain vulnerable populations. The pre-test, post-test, four tutorials, nine knowledge and applied mapping skill-based scenarios were integrated into the storyline; the learner never encounters obvious “tests” or lecture. Table 1 provides the course topic outline.

Table 1.

Rescue-D Topic Outline

Disability Community Demographics and Disaster Experiences Disability-related census data Disaster experiences of individuals with disabilities and negative impacts Effective and Ineffective 1:1 Interaction and Communication with Individuals with Disabilities Observe-Assess-Respond-verify Success (OARS) framework How to use the framework to guide interactions and communication with individuals with autism, people who are blind (and use a service animal) and those who use wheelchairs Inclusive, Whole Community Functional and Access Needs-Based Emergency Planning Brief introduction to legal requirements Inclusive emergency planning meetings How to use the C-MIST (Communication, Medical, Independence, Supervision, Transportation needs) functional and access need framework Mapping Techniques and Technology for Vulnerable Populations Brief introduction to registries (voluntary vulnerable population databases) How to use the mapping tool to: Locate individuals or groups of people with disabilities Add a map layer with location information Combine two map layers to create a third new layer Identify shelter and transportation resources for people with disabilities

Figure 1 depicts a scene from “Rescue From the Flood Zone” (RFFZ), one of nine timed tests of simulated interactions, with differential points awarded based on the quality of decision-making. In this scene the learner is asked to identify the evacuation and transportation needs of elderly Bingo players, some with disabilities (for example, mobility disabilities, oxygen needs). The learner chooses whether to “ask” questions of the scene’s characters with points awarded for questions that elicit more accurate or useful information. Once the learner determines that his/her information is complete, (s)he uses the mapping tool to: (1) identify and assess potential evacuation vehicles for the group; (2) “send” an appropriate vehicle to pick up the group; and (3) identify the closest, most appropriate mass care (general public) shelter. The learner gains more points for faster, better decisions.

Figure 1.

Rescue From the Flood Zone

Game elements incorporated into Rescue-D include a “quest” or central theme, winning and losing points, a scoreboard, and time challenges. Learners also earn points for completion of pre- and post-test segments (unrelated to accuracy) and only receive performance feedback in the tutorials and RFFZ. Correct responses result in points; error responses result in point loss or no point change. Learners receive two scores, one for the course overall and one for RFFZ. Progress is shown by displaying cumulative points and badges that appear upon progression through the course.

Development was guided, in part, by the scientific formative evaluation process outlined by Markle and others^{28, 29} and the Distance Learning Design Protocol³⁰ developed and used at the Shriver Center to create distance-learning courses. After designing learning objectives, the course outline, and nine potential application scenarios, four responders and three individuals with disabilities reviewed the objectives, curriculum content, scope, and storyline. They also reviewed the scenarios, including artwork, for complete and realistic portrayals of responders, the disability community, and disaster experiences. Rescue-D was revised based on their feedback.

Prior to course development, Rescue-D was granted an exemption by the UMMS Institutional Review Board. After design/development, Rescue-D underwent a rigorous internal quality assurance (QA) process by five Praxis/Shriver Center personnel previously unfamiliar with the course. Reviewers included an individual with a disability, an instructional designer, and behavioral research scientists who completed Rescue-D several times, reviewing content, language, vocabulary, layout and usability. Following the QA review the research team revised Rescue-D as necessary, then conducted a formative preliminary beta-test (pre-beta test), described further below. After the pre-beta test Rescue-D was revised again based on learner feedback regarding content, usability, and analysis of learner responses to course interactions, with the navigation interface, knowledge and mapping assessments adjusted until the course functioned as intended.

Technology

Rescue-D was designed as an asynchronous course, available “anywhere anytime” over the Internet, using a modern browser and broadband connection. The course was developed in Unity 3d, a cross-platform system for creating video games and interactive activities. Learners logged in and out of the course as needed, except for RFFZ, which needed to be completed in one sitting for technical reasons. Data from online learner interactions was recorded by Unity 3d and posted in real time to an online UMMS database that in turn analyzed the raw data and produced individual and group scores for research team review.

Participants

Formative pre-beta test participants included a pool of 10 individuals (5 completers) and summative beta test participants included a pool of 50 individuals (18 completers). Recruitment for pre-beta testers and beta testers occurred simultaneously and from the same participant pool. We emailed two invitation letters seeking beta testers to 132 local, county, state and national public health and safety organizations and to individual responders and asked the recipients to share the email with their networks. The invitation described Rescue-D as an “online course/game” for first responders to learn appropriate and efficient interactions with, and emergency response for, children and adults with disabilities and described eligibility criteria and participant selection. Potential participants were also informed that the course would include knowledge, demographics and learning experience questions. Interested individuals completed an online Survey Monkey^(R) survey asking about computer broadband Internet access, online course and game experience, and prior education about, or relationship with, individuals with disabilities.

Inclusion criteria included adults age 18 or older who were “first responders” (defined to include emergency response personnel “such as police, sheriffs, fire fighters, emergency medical services personnel, federal, state, county or local government emergency managers, and/or federal, state, county or local public health officers”), currently working full or part-time in the field in a paid or volunteer capacity. Responders needed to indicate competence reading and understanding written English and access to a computer with broadband Internet access.

One hundred ninety-nine individuals from 34 states completed the eligibility survey. Twenty-five individuals who did not meet the inclusion criteria were excluded. The research team analyzed the results and added voluntary MRC and CERT responders to the profession list, then divided potential respondents (n=174) into six response profession categories (law enforcement, fire, EMS, public health, CERT/MRC, emergency manager/planner). To ensure participants represented a broad array of response professions, we used an online random number generator to select 10 participants from each responder group (n=60), with at least one participant from each of the 10 FEMA regions.³¹

Ten pre-beta testers were randomly selected; each received a “welcome letter” (with information needed to access the course and other instructions) and unique access to Rescue-D. Five participants (50%) completed the formative pre-beta test. After revising the course as described above, we emailed the welcome letter and unique URL to the remaining 50 participants. Of these, 28 started and 18 (36%) completed Rescue-D and the satisfaction/usability survey within the requested two-week window. Tables 2 and 3, in the Online Data Supplement, describe participant characteristics. Pre-beta and beta testers completing Rescue-D and the survey received a $100 gift card as compensation for their time.

Data Collection and Measurement

Prior to the study the researchers performed a statistical power analysis for sample size estimation, based on data from another Shriver Center distance learning project,³² where researchers evaluated pre- and post-test scores of student learning participants (N=26). The effect size (ES) was 1.76, large according to Cohen’s criteria.³³ With a two-tailed alpha = .05 and power = 0.80, the projected sample size for this ES was approximately N = 8. With an N of 10, one would approximate yet a higher power setting (at 0.90). Thus, our actual sample size of 18 beta-testers appeared adequate.

During the formative and summative assessments we collected data on learner demographics, knowledge and skill acquisition via pre- and post-tests. The pre- and post-tests each included 29 questions, consisting of multiple choice, multiple selection, and mapping skill application items, incorporated throughout the ongoing storyline. Pre- and post-tests were not identified as such, thereby controlling post-test contamination. Pre- and post-test items measured identical concepts, with parallel, but not identically worded, questions, to lessen the likelihood of learning via pre-test exposure. Pre- and post-test questions covered: (1) disability community demographics and disaster experiences; (2) successful 1:1 “interaction” and “communication” with people who are blind and have service animals, have autism, or use a wheelchair; (3) “functional and access need”- based inclusive emergency planning per FEMA guidance³⁴ and the ADA⁸; and (4) the use of the mapping tool to identify resources and vulnerable populations. We assessed mapping tool use via simulated activities occurring during scenarios (e.g. identify public housing locations; create a map layer showing two disability provider agency locations). A Survey Monkey^(R) survey available after completion of the last Rescue-D scene collected satisfaction/usability data via Likert-type ratings and open-ended questions. Data were analyzed using SAS statistical software.³⁵

Results

The section below reports data from the summative beta test.

Knowledge and Skill Acquisition Assessment

Data were analyzed based on total responding (corrects and errors). Each question required a response. Data for a single question was not recorded for two of the 18 beta testers (due to errors in electronic capturing), thus data were analyzed using two approaches. In the first, missing data were assumed incorrect, producing a bias toward the null hypothesis. The second approach imputed missing data with the sample mean, which credits the participants with the mean score of the remaining sample when a question is omitted, producing a bias toward intervention success. While the true estimate of intervention success likely lies between these two estimates, Table 4 displays pre-/post-test comparisons using the former, more conservative, statistical analysis and shows an overall mean knowledge and applied skill gain. Beta testers demonstrated highly statistically significant pre- to post-test gains in knowledge (mean=16%, p<.0001), mapping skill (mean=39%, p<.0001) and overall (mean=19%, p<.0001). Mean gain in percent correct for the 25 knowledge questions was 16%, while mean gain for the 4 map questions, where pre-test scores tended to be very low, was 40%. When all 29 questions were considered, the overall gain in percent correct was 19% (i.e., an average increase of five additional questions correct).

Table 4.

Pre-Test and Post-Test Percent Correct

Variable	Mean	Median	Mode	Range	Min	Max	S.D.
Pre-test knowledge items % correct	54.00	56.00	56.00	36.00	36.00	72.00	10.29
Post-test knowledge items % correct	70.22	72.00	68.00	52.00	36.00	88.00	12.36
Knowledge gain in % correct	16.22	18.00	16.00	36.00	−8.00	28.00	9.05
Pre-test mapping items % correct	37.50	50.00	50.00	75.00	0.00	75.00	30.01
Post-test mapping items % correct	76.39	75.00	100.00	75.00	25.00	100.00	26.39
Mapping skill gain in % correct	38.89	37.50	25.00	125.00	−25.00	100.00	28.73
Total gain in % correct	19.35	18.97	13.79	27.59	0.00	27.59	8.37
Wilcoxona knowledge: p<.0001, ESb=1.4
Wilcoxon map: p=.001, ES=1.4
Wilcoxonc total gain: p<.0001, ES=2.3

^aWilcoxon signed-rank test is the nonparametric equivalent of the paired t-test for testing significant pre to post change.

^bEffect size (Cohen’s d) on pre to post difference (ES > 0.8 => large effect).

^cTotal gain in % correct provides equal weight to all 29 questions.

The results indicate (regardless of how missing data were handled) that the intervention had a very large effect for knowledge gain, map skill gain, and overall gain. The effect sizes of the pre- to post-test differences were also large: 1.4 for knowledge gains, 1.4 for mapping skill gains, and 2.3 overall. Cohen³³ identifies rules of thumb for measuring effect sizes for mean change as .2 indicative of a small effect, .5 as a medium effect, and .8 as a large effect; thus these effect sizes in the range of 1.4 to 2.3 indicate a very large effect and highly effective intervention.

Satisfaction and Usability

Upon completing Rescue-D, beta testers completed a survey with 11 satisfaction and usability items. Eighty-nine percent (89%) of beta testers agreed or strongly agreed that RescueD was easy to navigate, played well, and that content and activities were presented at a comfortable pace. All beta testers (N=18) indicated that it was helpful to complete Rescue-D at their own pace and without an instructor. Seventy-eight percent (78%) indicated that Rescue-D was preferable over an in-person course. We determined a “total usability score” from all usability questions. Ninety-three percent (93%) of beta testers rated usability positively (Satisfied or Very Satisfied). Ninety-four percent (94%) indicated that the content and activities were logical and clear, and that they would recommend Rescue-D to a colleague. All beta testers indicated the difficulty level and content depth were appropriate and that Rescue-D increased their awareness of the emergency needs of people with disabilities. A “total satisfaction score” from all satisfaction questions revealed that 97% of beta testers rated total content and activities positively (Satisfied or Very Satisfied).

Half of the beta testers completed Rescue-D in two to three hours, 22% required one to two hours, 17% three to four hours, and 11% completed Rescue-D in four to five hours.

The beta testers rated the game-like elements in the course very highly. All beta testers indicated that the continuing storyline and learning “in a fun way” enhanced their interest in the content and activities. Seventy-eight percent (78%) reported that the role-play and interactive map enhanced interest while 72% indicated that point scoring, and 56% indicated that the timed elements, enhanced interest.

Representative comments to open-ended questions about game features, technology used, and suggestions for improvement are presented in the Online Data Supplement, Table 5.

Discussion

This study demonstrated the feasibility of an online role-play simulation-type course to effectively teach public health and safety responders about the emergency needs of individuals with disabilities. Results indicate significant knowledge and skill acquisition gains from pre- to post-test. Learner usability and satisfaction scores were high. These results provide support for further developing Rescue-D into a complete distance-learning curriculum.

The literature suggests potential reasons for these results. We developed Rescue-D as an immersive educational simulation, with a storyline consisting of rescue and other scenarios. Many emergency responder trainings consist of practical exercises in the form of simulated emergencies²¹ designed to mimic actual potential events. The simulated real life scenarios and interactivity in Rescue-D may have felt more familiar or comfortable to learners used to participating in such drills. Online simulations and games have been found to be particularly motivating to learners;²⁵ satisfaction and motivation impact performance.³⁶ Adults learn best when applying theory to practice and prefer experiential to passive learning.²⁵ During development we learned that our intended audience didn’t want the pressure of “tests” thus embedding pre- and post-tests into the storyline may have also encouraged learner motivation and enthusiasm, leading to knowledge and skill gains. Another factor impacting the results may have been ease of technology use. Learners needed only brief instructions, with no special training or familiarity with Unity 3d, to complete Rescue-D. By contrast, online simulations and games developed in other platforms might require more extensive training in platform use. Authors describe a required one hour tutorial before beginning online trainings developed in the Second Life platform.^{25, 26} Finally, simulated online scenarios are risk free for the learner; in real life disaster victims may become injured or die, and an emergency responder may make potentially harmful or fatal mistakes. Rescue-D’s lack of risk may have contributed to the results.

Future Curriculum Development and Dissemination

We consider Rescue-D as a prototype, and intend to revise and expand the course with additional topics and depiction of a greater number and variety of disabilities, including people with multiple disabilities or disabilities combined with chronic health conditions. Segments could be included to teach learners to incorporate disability community strengths and abilities into local emergency planning and response. Additional features might be developed to enable learners to ask content-related questions (as one could in a classroom-based training) and a follow-up resource guide might be developed for future consultation during actual response. Additional scenarios could be developed based on traditional response protocols to encourage learners to re-consider those conventions (for example, traditional response protocols may call for evacuating victims from a building as fast as possible,²¹ however, following this procedure might endanger the health or well-being of someone with a particular disability.)

Effective dissemination of such a course, aimed at attitude and system change in addition to education, might prove challenging. Providing CEUs or college credits might provide incentives for course completion. Some writers suggest that, in addition to stand alone courses, disability curriculum content should be integrated into existing undergraduate, graduate, and continuing education responder courses.^21,37 Because people with disabilities represent such a large population segment (approximately 19% of the non-institutionalized population³⁸), a disability-related course could be required for responders (at least one state, New Jersey, mandates a responder course on developmental disabilities.³⁹)

Limitations and Future Research

The current study evaluated a subpart of a larger course and that led to some research limitations. Further research will be needed to assess learner knowledge and skill gains across a wider variety of disabilities and emergency-related challenges. This study did not assess the role of individual instructional components to determine which, if any, component may have had a greater impact on knowledge and skill gains. Future research might examine which components were more or less beneficial to the learning process, with suggestions for re-design for maximum learner knowledge gain. We also did not evaluate whether learner knowledge and skill gains persisted over time or translated into actual practice. Additional assessment via post-surveys might shed greater light on knowledge retention and translation into practice. While the benefits of cross-training (training multiple responder audiences together and regarding multiple roles) ²¹ is widely recognized, this study didn’t evaluate whether learners from different response services differed in performance in relation to each other or needed education individualized to their specific service. While it is known how knowledge is disseminated to emergency responders, there is a gap in understanding the knowledge acquisition process and best training practices.²¹ Future study could examine these issues and survey responders about knowledge acquisition needs.

Conclusions

Computer based distance learning is recognized as an acceptable means of teaching emergency responders and a viable alternative to traditional training modes.^{23, 24} The results of the current study indicate that an online course with simulation-based scenarios and game-like features can successfully transfer knowledge and skills about the emergency needs of people with disabilities to emergency responders. Experts posit that prior training influences emergency responder readiness self-perception,²⁴ and past experience improves future response.⁴⁰

Emergency responders play an important part in protecting Americans from harm due to manmade or natural disasters. Responders who are better trained and have practiced response techniques under simulated conditions may be better prepared to address the likely needs of individuals with disabilities during an actual emergency and incorporate the needs and skills of people with disabilities into “whole community” inclusive planning, enhancing community resilience and speeding future disaster recovery.