A Hazardous Materials Educational Curriculum Improves Pediatric Emergency Department Staff Skills

Abstract

Objective

Emergency department (ED) providers require competency in responding to hazardous materials (HAZMAT) events. The optimal strategies to teach HAZMAT response principles to ED providers and to ensure skill retention are not known. Our aim was to design, implement, and evaluate a multifaceted, interprofessional educational curriculum for pediatric ED staff to improve their skills, knowledge, and confidence in responding to a HAZMAT event.

Methods

In this longitudinal cohort study, we created and assessed a 3‐hour educational curriculum comprised of didactics, skills stations, a tabletop exercise, and a simulated multivictim disaster. Learning objectives included critical aspects of pediatric HAZMAT incident response with an emphasis on donning personal protective equipment (PPE). The primary outcome was the number of HAZMAT PPE donning steps correctly completed within 10 minutes at pre‐ and postcurriculum assessments measured using a 32‐item checklist. Secondary outcomes included skill retention at 3 months, change in knowledge assessed using multiple‐choice questions, and change in participant confidence.

Results

Eighty‐one of 84 participants (96%) completed the entire curriculum. Compared to the precurriculum assessment, participants completed more donning steps correctly after the intervention (mean increase = 58%, 95% confidence interval [CI] = 48%–70%). Relative to the baseline, more steps were also correctly completed at 3 months (mean increase = 49%, 95% CI = 38%–61%). Performance on multiple‐choice knowledge questions and confidence in skills also significantly increased from the pre‐ to postcurriculum assessments.

Conclusions

A newly developed HAZMAT educational curriculum improved skills‐based performance, knowledge, and confidence in PPE and decontamination skills. Brief, multifaceted educational interventions for ED staff can effectively develop sustainable skills needed for uncommon emergency events.

Mass casualty incidents in the emergency department (ED) setting are “high‐stakes but low‐frequency” events,1 making training and retention of skills difficult. Response to events involving patient exposure to hazardous materials (HAZMAT) is an important part of ED disaster‐preparedness protocols. Disaster skills for ED health care professionals are likely to become even more important as the threat of terrorist events increases and natural disasters become more frequent. Additionally, regulatory requirements surrounding hospital emergency‐preparedness are increasing, including Joint Commission regulations that require emergency response plans be exercised twice annually.2, 3

Hazardous substances are used and transported daily around the world. There are thousands of unintentional hazardous material exposures each year in the United States.4 Additionally, first responders and hospital personnel become especially vulnerable when terrorist attacks involve hazardous agents. One such example is the Tokyo subway sarin attack in 1995, where limited access to personal protective equipment (PPE) and decontamination facilities resulted in 10% of fire department personnel and 23% of hospital staff becoming symptomatic from secondary chemical exposure.5, 6 Appropriate resources and training are needed to mitigate the risk for health care providers.

Optimal methodologies for teaching HAZMAT event response to health care personnel have not been determined. Core competencies for disaster education include recognizing potential critical events, implementing a response, understanding institutional emergency plans, and demonstrating the skills and knowledge required to perform one's individual task during a disaster.7 A number of approaches have been utilized to teach these necessary skills including didactics, tabletop exercises,8, 9 and simulation sessions.1, 9, 10 However, limited data exist regarding the impact of such educational efforts on skill performance.11 Additionally, there are unique challenges in caring for pediatric patients given their increased physiologic susceptibility to hazardous substances and the inherent difficulty in caring for children while wearing PPE.12 Few protocols for pediatric HAZMAT disaster response exist,13 with no published data on effectiveness.

Our aim was to create a structured, multifaceted HAZMAT curriculum and assess the effect on ED staff's skills, knowledge, and confidence in regard to HAZMAT response and decontamination and to assess whether any demonstrated gains were sustained at 3 months.

Methods

Study Design

This was a longitudinal cohort study enrolling participants from October 2015 to July 2016. The hospital institutional review board approved this study.

Study Setting and Population

The study was performed at an urban pediatric tertiary care hospital. The ED has approximately 60,000 visits per year and is a pediatric level I trauma center. Participation in the curriculum was mandatory for all pediatric emergency medicine–credentialed physicians and encouraged for all nursing staff. Administrative staff were also invited to participate. Inclusion in the evaluative component was voluntary. Written consent was obtained from participants prior to the start of the curriculum.

Study Protocol

Curriculum Design

A hospital‐level needs assessment served as the impetus for creation of this curriculum. A multidisciplinary team consisting of pediatric emergency medicine physicians and nurses, hospital emergency management personnel, and hospital environmental health and safety leadership convened and designed the curriculum over a 3‐month period. Consensus on subject matter was achieved through an iterative process. The curriculum included principles of HAZMAT incident response as well as donning and doffing of PPE. Instruction focused on the use of level C PPE equipment, which includes chemical resistant suits, boots, gloves, and hoods and powered air‐purifying respirators (PAPRs). Activation of hospital emergency protocols, assembly and coordination of decontamination areas, and treatment of HAZMAT patients were also covered. A sample 3‐hour curricular path is shown in Figure 1. Multiple pedagogic approaches were utilized, including didactics, small group skills stations, a tabletop exercise, and a hands‐on simulated multivictim decontamination scenario. The 15‐minute didactic presentation provided an introduction to hospital emergency management structure and protocol activation, as well as an overview of HAZMAT principles. Three skills stations included instruction on the setup and use of PAPRs, principles of pediatric patient decontamination, and a problem‐solving session addressing common pitfalls in donning and doffing PPE. The tabletop exercise included a simulated HAZMAT event during which participants reviewed the steps required to activate and implement hospital protocols. The hands‐on simulation scenario focused on decontamination practices using dedicated fixed decontamination showers and communication while wearing PPE. The curriculum was offered on eight separate dates over the study period. A core group of seven facilitators was instructed on curriculum delivery and participant evaluation.

Figure 1.

Sample curricular path.* *Order may be different for participants. †Skills stations include setup and use of PAPRs, principles of pediatric decontamination, and common pitfalls in donning and doffing of PPE. PAPR = powered air‐purifying respirator; PPE = personal protective equipment.

Evaluation

A standardized 32‐item direct‐observation checklist was used to evaluate donning exercises. The checklist was developed jointly by a multidisciplinary team of local content experts. Available literature and resources on accepted steps in HAZMAT donning were reviewed. An initial checklist was created and then refined using an iterative process until expert consensus was achieved. The donning components included items such as checking all equipment for damage; assembling the PAPR; correctly taping ankles, wrists, and zippers to prevent exposure; and properly applying the PAPR and hood. Each component was evaluated in a binary fashion as being performed correctly or incorrectly. The correctly completed steps were summed to create a total score. Given the need to have at least one assistant to the person donning, participants were randomly assigned to dyads for skills evaluation. Interdisciplinary dyads (i.e., nurse–physician) were created whenever possible. Donning was evaluated at three time points—at the start of the curriculum, at the end of the curriculum, and at least 3 months after participation. At each of these time points, the number of items correctly completed within 10 minutes was recorded for each dyad.

An 11‐question multiple‐choice knowledge test was administered before and after participation in the curriculum. The questions covered general principles of HAZMAT response and key aspects of local institutional disaster plans. Each question on the test had been previously sent to an external group of pediatric disaster experts to achieve consensus on clarity and accuracy.

Participants completed a precurriculum questionnaire addressing demographics, prior HAZMAT and disaster response experiences, and perceived confidence with HAZMAT disaster response preparedness. Confidence was assessed on four items using a 7‐point Likert scale (range from 1 = ”not at all confident” to 7 = ”very confident”). The postcurriculum survey reassessed participant confidence and elicited feedback about the curriculum and interest in participating in other types of emergency response training. The knowledge test and surveys were piloted on a subsample of emergency medicine fellows prior to initiation of the study. Study data were collected and managed using REDCap electronic data capture tools hosted at Boston Children's Hospital.14

Key Outcome Measures

The primary outcome of the study was the difference in the number of PPE donning steps correctly completed within 10 minutes. This was assessed at two different time points, at the start of the curriculum and again after participation. Formal recommendations for appropriate staff preparation time prior to a HAZMAT event do not exist. We chose 10 minutes based on hospital expert consensus and guided by data from time to patient arrival during a recent local mass casualty event.15 Secondary outcomes included measurement of PPE donning skills retention at 3 months, change in percentage of correctly answered knowledge questions before and after participation in the curriculum, and change in confidence in performing HAZMAT related tasks. Additionally, we collected narrative participant feedback about the curriculum for educational improvement efforts. Themes from these responses were reported.

Data Analysis

We determined that 100 providers were required to complete the curriculum to provide 80% power to detect a difference of one correctly completed donning step between the pre‐ and postcurriculum evaluation, using an estimated standard deviation of 2.5 steps. The primary outcome (difference in number of correctly completed steps within 10 minutes) was assessed using a generalized linear model, with clustering by participant to account for repeated measures over time. Changes in scores on the knowledge tests and reported levels of confidence were analyzed using a Wilcoxon signed‐rank test. Participant characteristics were reported using medians and interquartile ranges for continuous variables and frequencies and percentages for binary variables. All statistical analysis was performed using STATA 13.0 (StataCorp).

Sponsorship

The development and administration of this curriculum were sponsored internally by the Division of Emergency Medicine, Emergency Management, and Environmental Health and Safety at Boston Children's Hospital.

Results

Eighty‐four participants were enrolled in the curriculum and 81 (96%) completed all components. Seventy‐eight of these participants were clinical providers; three were administrative. The prior HAZMAT experience of the 82 participants who completed the demographic questionnaire is delineated in Table 1. Overall, a larger proportion of nurses had completed prior HAZMAT response training, but the percentages of physicians and nurses who had experienced a true HAZMAT event were similar. The median number of years in practice was also similar between the physicians and nurses.

Table 1.

Participant Experience

	Physicians (n = 56)	Nurses (n = 23)	Admin (n = 3)	Total (N = 82)
Years in position, median (IQR)	11 (3–15)	13 (6–20)	4 (2–4)	10.5 (3–16)
Prior HAZMAT education	57%	87%	0%	65%
Prior HAZMAT experience	16%	18%	0%	16%

IQR = interquartile range.

Donning of PPE was assessed in dyads. Fifty‐seven percent of dyads were nurse–physician, 38% were physician–physician, 2% were physician–administrator, and 2% were administrator–administrator. Participants demonstrated an improvement in the number of donning steps correctly completed within 10 minutes between assessments before and after participation in the curriculum, with a mean increase in score of 58% (95% confidence interval [CI] = 48%–70%). Sixty‐three percent of eligible dyads were retested at least 3 months after completion of the curriculum (median = 112 days, range = 84 to 220 days). Dyads demonstrated skills retention at 3‐month retesting compared to their precurriculum performance with a mean increase in their score from the baseline assessment of 49% (95% CI = 38%–61%). See Figure 2. There was no difference in the median postcurriculum PPE donning scores of the dyads that returned for follow‐up versus those that did not (31 steps correct vs. 30 steps correct, respectively; p = 0.68).

Figure 2.

Number of correctly completed HAZMAT donning steps. Box‐and‐whisker plot demonstrating donning scores as measured by 32‐item checklist prior to the curriculum, immediately after the curriculum, and at 3‐month reevaluation. Shaded boxes represent interquartile ranges with horizontal line within the box representing the median. Outer horizontal bars represent the range of scores. *p < 0.001 when compared to precurriculum assessment. HAZMAT = hazardous materials.

Participants also significantly improved their knowledge of HAZMAT principles as assessed by multiple‐choice questions. For the 71 participants who completed both tests, the pre‐ and postcurriculum median test scores were 63% (interquartile range = [IQR] = 54%–72%) and 91% (IQR = 82%–100%; p < 0.001), respectively. There was a significant increase across the four items assessing confidence, based on reported Likert values before and after participation (p < 0.001), as shown in Figure 3.

Figure 3.

Distribution of Likert‐level confidence ratings. Pre‐ and post‐curriculum comparison of confidence scores as measured by 7‐level Likert confidence scoring. (A) Confidence to perform assigned role; (B) confidence to don and doff PPE; (C) confidence to perform multicasualty triage; (D) confidence to perform patient decontamination. All changes in reported confidence from pre‐ to post‐ curriculum were statistically significant (p < 0.001).

On the postcurriculum questionnaire, 89% of participants reported that the 3‐hour length was “just right” for this type of educational curriculum, and 74% felt that additional training in this area would be helpful. Narrative responses suggested that participants particularly enjoyed the hands‐on donning and doffing practice, the interactive nature with frequent switching of activities, the ability to work with interdisciplinary teams from within the ED, and the ability to use the in situ facilities where actual patient decontamination would take place. Suggestions for improvement included extending the time allotted for the tabletop exercise, opportunities to practice decontamination on standardized patients rather than mannequins, and review of the integration of HAZMAT response with other hospital emergency plan logistics.

Discussion

Advanced instruction and drilling of scenarios is important to prepare providers for a potential pediatric HAZMAT event. In particular, ED staff need to be able to work within established systems to provide optimal care to patients while protecting themselves from the potential harm of an unintentional secondary exposure. We designed, implemented, and evaluated a 3‐hour educational curriculum to provide pediatric ED providers with the skills and knowledge necessary to respond to a HAZMAT event. Participation in the curriculum improved skills, knowledge, and confidence in HAZMAT principles and practice. Importantly, these educational gains were retained after 3 months.

We chose to focus our curriculum on three of the seven core disaster competencies identified by Hsu et al.,7 specifically recognizing potential critical events and implementing a response, understanding institutional emergency plans, and demonstrating the skills and knowledge required of individual health care providers. The curriculum was designed to meet the needs of adult learners with an emphasis on interactive and hands‐on learning. We deliberately started with a donning drill to prompt recognition of gaps in relevant procedural competence and create a sense of need for the participants. After this initial engagement, we provided a brief introductory lecture. We built on this foundation through hands‐on procedural instruction, case‐based discussions, and simulation exercises.

Models exist to guide evaluation of training programs. Perhaps the most widely accepted was introduced by Kirkpatrick,16 often offered as a pyramidal construct. In this model, reactions to the intervention and demonstration of learning are considered the lower levels of assessment, while changes in behavior are higher in the pyramid. With regard to assessment of educational efforts on disaster‐preparedness for providers, the majority of assessments have focused on lower levels of assessment using Likert scale confidence or knowledge‐based questions.8, 10, 13 Additionally, few studies have combined multiple types of instruction, and those that have often required extensive time commitments for their participants. Collander et al.13 developed a comprehensive 16‐hour all‐hazards course, although such a time commitment may make recruitment and participation challenging, particularly for multidisciplinary groups. In addition, the assessment was limited to demonstrated improvement on a postcourse knowledge test without evaluation of skills. Rumoro et al.17 similarly designed a thoughtful, 1‐year pilot training program for comprehensive disaster response. Again, evaluations were limited to knowledge and perceptions, and only 10 participants were able to be included. Bartley et al.11 described a curriculum that used an audiovisual presentation followed by a simulation exercise and debriefing event to teach hospital disaster‐preparedness to a multidisciplinary group. Assessments were again limited to knowledge and perceptions, with a demonstrated increase in pass rate on the post‐participation knowledge assessment but no significant change in participants’ subjective feeling of preparedness.

Few previous studies have specifically assessed skill performance related to preparation for disaster events. Cicero et al.1 focused on triage skill performance by pediatric residents at different time intervals after instruction although there was no baseline score comparison prior to instruction. Northington et al.18 examined the degradation of paramedic students’ HAZMAT PPE donning skills 6 months after instruction, but all participants had to practice until obtaining a perfect score on the assessment at the time of initial instruction.

We targeted multiple levels of Kirkpatrick's assessment framework, including higher‐level evaluations by utilizing a standardized skill performance checklist. This was performed at three different time points during and months after participation in the curriculum. The top level of the Kirkpatrick pyramid is change in results, i.e., clinical outcomes, which was not feasible in our study design. Evaluation of the impact of our 3‐hour HAZMAT curriculum was able to demonstrate improvements in knowledge and confidence, with an added objective assessment of skills‐based performance. Our study not only demonstrated improvement immediately after instruction but also skill retention at 3 months.

Limitations

Our study has some limitations. First, this is a single‐center study involving clinical staff based in a large ED in an urban pediatric tertiary care center. This may limit its generalizability to other ED settings. Second, the 32‐item checklist used for evaluation of the primary outcome has not been previously validated. To our knowledge, no such validated instrument currently exists. We attempted to address this by including items currently included in accepted disaster protocols and then using an iterative process with multiple disaster experts until consensus on each item was achieved. Third, we did not assess performance during a true HAZMAT event. The infrequency of such events would make determination of improvement in a true HAZMAT setting very difficult. Finally, our retesting rate of 63% of participants was lower than we had planned. However, even with lower retention rates, we were able to demonstrate sustained significant improvement from the baseline. We found no difference in postcurriculum median checklist scores between the group that returned and the group that did not complete the follow‐up, suggesting that our findings were unlikely to have been affected by systematic retention of a higher skilled group. Although information about skill retention at longer term follow‐up would be valuable, for feasibility reasons, we limited our retesting to 3 months after participation.

Conclusions

This multifaceted, interprofessional hazardous materials response curriculum improved pediatric ED staff knowledge and confidence and, most importantly, the skills necessary to respond to a hazardous materials event. The mixture of educational strategies may contribute to the sustained gain in skills over a period of months. This style of instruction may be particularly valuable for hazardous materials and similar type events that are uncommon and occur with unpredictable frequency. The next steps include validation of the donning checklist and dissemination of the curriculum to other health care providers.

AEM Education and Training 2018;2:40–47.