A Comparison of Simulation-Based Education Versus Lecture-Based Instruction for Toxicology Training in Emergency Medicine Residents

Abstract

Simulation-based teaching (SIM) is a common method for medical education. SIM exposes residents to uncommon scenarios that require critical, timely actions. SIM may be a valuable training method for critically ill poisoned patients whose diagnosis and treatment depend on key clinical findings. Our objective was to compare medical simulation (SIM) to traditional lecture-based instruction (LEC) for training emergency medicine (EM) residents in the acute management of critically ill poisoned patients. EM residents completed two pre-intervention questionnaires: (1) a 24-item multiple-choice test of four toxicological emergencies and (2) a questionnaire using a five-point Likert scale to rate the residents’ comfort level in diagnosing and treating patients with specific toxicological emergencies. After completing the pre-intervention questionnaires, residents were randomized to SIM or LEC instruction. Two toxicologists and three EM physicians presented four toxicology topics to both groups in four 20-min sessions. One group was in the simulation center, and the other in a lecture hall. Each group then repeated the multiple-choice test and questionnaire immediately after instruction and again at 3 months after training. Answers were not discussed. The primary outcome was comparison of immediate mean post-intervention test scores and final scores 3 months later between SIM and LEC groups. Test score outcomes between groups were compared at each time point (pre-test, post-instruction, 3-month follow-up) using Wilcoxon rank sum test. Data were summarized by descriptive statistics. Continuous variables were characterized by means (SD) and tested using t tests or Wilcoxon rank sum. Categorical variables were summarized by frequencies (%) and compared between training groups with chi-square or Fisher’s exact test. Thirty-two EM residents completed pre- and post-intervention tests and comfort questionnaires on the study day. Both groups had higher post-intervention mean test scores (p < 0.001), but the LEC group showed a greater improvement compared to the SIM group (5.6 [2.3] points vs. 3.6 [2.4], p = 0.02). At the 3-month follow-up, 24 (75 %) tests and questionnaires were completed. There was no improvement in 3-month mean test scores in either group compared to immediate post-test scores. The SIM group had higher final mean test scores than the LEC group (16.6 [3.1] vs. 13.3 [2.2], p = 0.009). SIM and LEC groups reported similar diagnosis and treatment comfort level scores at baseline and improved equally after instruction. At 3 months, there was no difference between groups in comfort level scores for diagnosis or treatment. Lecture-based teaching was more effective than simulation-based instruction immediately after intervention. At 3 months, the SIM group showed greater retention than the LEC group. Resident comfort levels for diagnosis and treatment were similar regardless of the type of education.

Electronic supplementary material

The online version of this article (doi:10.1007/s13181-014-0401-8) contains supplementary material, which is available to authorized users.

Introduction

Poisoned, critically ill patients often require specific, urgent interventions that must be initiated after a diagnosis based on limited history and physical examination findings. Rapid identification and appropriate treatment of a poisoned patient can have significant clinical impact as evidenced by poisoned patients with life-threatening dysrhythmias having survival rates as high as 50 % with appropriate management [1, 2]. The benefit of identification and treatment of the critically ill poisoned patient highlights the importance of appropriate emergency medicine resident education.

The rarity of life-threatening poisonings leaves emergency physicians in training with little and random clinical experience. Examples of such poisonings include serotonin syndrome, severe salicylism, tricyclic antidepressant-induced ventricular tachycardia, or organophosphorous compound poisoning. Alternative and engaging education approaches have been used for trauma resuscitation, advanced cardiac life support, military combat casualty care, and pilot training [3]. The airline industry has used simulation for decades to train pilots to handle unexpected disaster or events [3, 4]. Simulation training may be an appropriate modality to teach the identification and management of toxicologic emergencies to emergency medicine residents.

The benefits of medical simulation include the ability to allow the learner to manage a medical challenge and learn from mistakes without placing an actual patient at risk [5–7]. Despite the growth of medical simulation used in medical education, research into the appropriate application and its benefits is limited. Most simulation-based medical education (SBME) studies are observational evaluations of the learners’ self-reported satisfaction with the simulation experience with no objective findings [8, 9]. The few randomized controlled studies reported improved performance in the SBME groups compared to the control groups (lecture-based education); however, in most of these studies, a significantly greater amount of time was spent training the subjects in the SBME arm than in the control arm. Thus, it was unclear if the subjects in the simulation arms performed better because simulation was superior to lecture or if because more time was spent training them [8, 10]. One published study evaluated the use of simulation in toxicology education, but this study included exclusively second year medical students [11].

We hypothesized that training emergency medicine residents with SBME would improve their medical knowledge and confidence compared to traditional lecture-based education at 3 months after training.

Methods

Study Design

We conducted a prospective randomized controlled trial of mannequin simulation versus lecture. The San Antonio Military Medical Center Institutional Review Board approved the study.

Study Setting and Population

Residents at a single 3-year emergency medicine residency were the subjects of the study. Simulation training was conducted by the hospital’s graduate medical education simulation center using Laerdal SimMan® 3G patient simulators (product number 212-00050; Wappingers Falls, NY).

Study Protocol

Residents were assigned to either a simulation arm or control arm using block randomization based upon the residents’ current level of training (e.g. first-year, second-year, or third-year resident). Sixteen residents were assigned to the simulation arm, and 17 were assigned to the lecture arm. The simulation arm was then further randomly divided into four near-equal teams (three teams of four subjects and one team of five subjects) with at least one first-, second-, and third-year emergency medicine resident on each team.

Subjects on the four simulation teams were presented in series with each of four high-fidelity mannequin-based medical simulation cases (beta-blocker and calcium channel antagonist toxicity, organophosphorous compound poisoning, salicylate toxicity, and tricyclic antidepressant poisoning). Following the completion of the case, the instructor used the remainder of the 20 min available to review the case and provide additional teaching. The groups then rotated to the next station. Each of the four instructors was assigned to present the same case to each of the four teams assigned to the simulation arm. The instructors were also cautioned to ensure the simulation session progressed rapidly enough to allow sufficient time for debriefing at the end. All four instructors were board-certified emergency physicians, and one was also board certified in medical toxicology. Each instructor was provided a simulation case and trained in a standard format by the research team on how to conduct the simulation training (instructor training documents available for review online).

The 16 subjects assigned to the control arm were given a 20-min lecture on each of the four aforementioned topics (beta-blocker and calcium channel antagonist toxicity, organophosphorous compound poisoning, salicylate toxicity, and tricyclic antidepressant poisoning) with a break for questions between each lecture. A board-certified emergency physician in the second year of medical toxicology fellowship training provided the four lectures. The lecturer was also directed to ensure all pertinent information was covered but to refrain from directing his teaching specifically towards the test questions.

Measurements

The subjects were given an identical pre-intervention and post-intervention questionnaire and test immediately before and immediately after the educational intervention (simulation or lecture). The questionnaire and test were also administered again 3 months following the intervention to assess for knowledge retention. The questionnaire assessed the subject’s comfort level in diagnosing and managing each of the four different toxicities as well as their overall comfort level using a six-point Likert scale. The test consisted of 24 vignette-based questions (6 questions for each of the four topics). The test questions and the simulation instructor teaching documents were developed by the investigators utilizing the American Board of Emergency Medicine (ABEM) core competencies and are available to view. The primary outcome was comparison between the simulation (SIM) and lecture (LEC) groups’ mean test scores at 3 months post-intervention.

Data Analysis

Test score outcomes between groups were compared at each time point (pre-test, post-instruction, 3-month follow-up) using Wilcoxon rank sum test. Data were summarized by descriptive statistics. Continuous variables were characterized by means (SD), and the means were tested using t tests or Wilcoxon rank sum. Categorical variables were summarized by frequencies (%) and compared between training groups with chi-square or Fisher’s exact test. Mean test scores were compared using a two-sample t test.

Results

Thirty-three EM residents completed pre- and post-intervention tests and comfort questionnaires on the study day (Table 1). SIM and LEC groups had similar mean pre-intervention baseline test scores. Both groups had higher post-intervention mean test scores (p < 0.001), and the LEC group showed a greater improvement compared to the SIM group (5.6 [2.3] point increase vs. 3.6 [2.4] point increase, p = 0.02). At the 3-month follow-up, 24 (75 %) tests and questionnaires were completed (11 in the simulation group and 13 in the lecture arm). The SIM group had higher final mean test scores than the LEC group (16.6 [3.1] vs. 13.3 [2.2], p = 0.009, Fig. 1). When comparing the immediate post-intervention test scores of those who did and did not complete the 3-month post-intervention test, there was no significant difference between the simulation (17 [1.9] vs. 18.4[2.1], p = 0.18) and lecture (18.1[2.7] vs. 18.2[1.7], p = 0.95) arms. SIM and LEC groups reported similar diagnosis and treatment comfort level scores at baseline and improved equally after instruction (Table 2). At 3 months, there was no difference between groups in comfort level scores for diagnosis or treatment.

Table 1.

A list of demographic characteristics for the subjects enrolled in each arm of the study

	Training groups
	Simulation	Lecture	Total
Subjects, n (%)	16 (100)	17 (100)	33 (100)
Age (years), n (%)
25 to 30	11 (68.75)	12 (70.59)	23 (69.7)
31 to 35	5 (31.25)	4 (23.53)	9 (27.27)
36 to 40	0 (0)	1 (5.88)	1 (3.03)
Gender, n (%)
Female	2 (12.5)	4 (23.53)	6 (18.18)
Male	14 (87.5)	13 (76.47)	27 (81.82)
Current year of EM residency, n (%)
1st	6 (37.5)	7 (41.18)	13 (39.39)
2nd	6 (37.5)	6 (35.29)	12 (36.36)
3rd	4 (25)	4 (23.53)	8 (24.24)
Years of clinical experience, n (%)
n/a	11 (68.75)	12 (70.59)	23 (69.7)
1	4 (25)	3 (17.65)	7 (21.21)
3	0 (0)	1 (5.88)	1 (3.03)
≥4	1 (6.25)	1 (5.88)	2 (6.06)
Completed tox rotation, n (%)
Yes	2 (12.5)	2 (11.76)	4 (12.12)
No	14 (87.5)	15 (88.24)	29 (87.88)

EM emergency medicine

Fig. 1.

Composite test scores The test scores for each training class (Sim vs Lect) at each testing period (Pre, Post, 3Mo Post) are represented by box-and-whisker plots. For each box-and-whisker plot, the solid dark line represents the median Test score. The bottom and top of the box represent the inter-quartile range (1st and 3rd quartiles). The whiskers represent the minimum and maximum scores excluding outliers. The circles represent outlier values

Table 2.

Comfort composite scores by training group

	Training Groups
	Simulation			Lecture
Composite comfort scorea	Pre (n = 16)	Post (n = 16)	3 months (n = 11)	Pre (n = 17)	Post (n = 17)	3 months (n = 13)
Diagnosingb	12.75 (3.94)	10.13 (3.12)	10.91 (3.05)	12.88 (4.73)	11.53 (4.52)	13 (4.71)
Treatingc	13.31 (4.61)	10.63 (3.54)	10.91 (3.56)	13.12 (4.54)	11.24 (4.63)	12.85 (4.67)
Overall comfortd	26.06 (8.45)	20.75 (6.48)	21.82 (6.49)	26 (9.16)	22.76 (9.07)	25.85 (9.2)

Questionnaires are available online. A lower numerical value indicates a higher level of comfort

^aComposite scores are represented by means (standard deviation) by training group for each testing follow-up period

^bComposite score for diagnosing comfort (questions 8,9,11,13,15)

^cComposite score for treating comfort (questions 7,10,12,14,16)

^dComposite score for overall (questions 7 through 16)

Discussion

Our results demonstrate no difference between the simulation and lecture arms in comfort level or immediate post-intervention testing; however, at the 3-month post-intervention test, the simulation arm had higher scores.

Multiple studies have demonstrated SBME improves subjects’ test scores and performance measures when compared to no intervention, but most of these studies lack a control arm of standard educational approaches (i.e., lecture) [8, 10]. Furthermore, most previous studies evaluated only knowledge immediately following the intervention [8, 10]. We compared the simulation group to a control group who received lecture-based training, and we evaluated knowledge retention 3 months after the intervention. Our findings suggest that both education modalities improve toxicology knowledge; however, while not statistically significant, lecture may be superior to simulation for the acquisition of short-term knowledge of poisoning identification and management. SIM performed better at the 3-month follow-up assessment, and therefore, is superior to lecture for providing residents with retained knowledge of poisoning identification and management.

Previous research in adult education has demonstrated the greatest learning occurs with events that are emotionally charged, challenging, and stressing the learner to the point of causing a significant change in body state [12]. This helps to explain why anecdotally physicians and medical students describe difficulty learning about a disease until they see an actual patient with the disease. A properly developed and implemented simulation case can be emotionally charged, challenging, and stressing the learner [12]. Obtaining a similar response to a toxicology lecture would be difficult.

Throughout the study, the median comfort levels of subjects remained in the “very comfortable” to “neutral” range of responses. The lack of difference between comfort levels in the two arms of the study after intervention was unexpected. The majority of previous randomized simulation studies have found a greater improvement of confidence in the simulation arm when compared to controls [8–11]. We speculate that the subjects remained somewhat comfortable in both groups because of their limited experience with clinical cases in toxicology. Given that simulation is intended to enhance learning by providing a stressful environment, the increased stress may explain why a single session does not significantly enhance confidence. Serial simulation sessions during which a student can demonstrate mastery of a topic may provide increased confidence as demonstrated in other studies.

While we aimed to evaluate the effectiveness of 20 min of lecture versus 20 min of simulation, we were unable to evaluate characteristics that ensure the greatest benefit from simulation. Previous studies have advocated repetitive practice, curriculum integration, and a range of difficulty levels [13]. These were not evaluated in this study. In the future, we intend to integrate simulation as a regular component of our toxicology and emergency medicine education with the hopes of allowing for thorough curriculum integration and repetitive practice to ensure mastery of these important skills.

Limitations

Our study has several limitations. The greatest limitation of our study is the use of a written test and questionnaire to evaluate the quality of the education. While the questions were designed using an ABEM format and based upon the ABEM core competencies, they may not be the best indicator of a physician’s ability to care for a toxicological patient. Ideally, we would evaluate which intervention allowed for the greatest improvement in caring for actual toxicology patients. Unfortunately, clinically significant toxicology exposures are insufficiently common to make this feasible [14]. Furthermore, it would be unethical to allow residents to care for such patients and follow patient outcome without attending intervention [3]. While simulation patients could be used as a surrogate, this would clearly bias the results in favor of the SIM arm due to increased familiarity with SIM and the equipment.

Other threats to the validity include the quality of the lectures and the simulation training. It is difficult to generalize our findings to other emergency medicine residencies where staff may have different levels of experience with using lecture-based and simulation-based instruction. Most of our residents were male and may bias the results based upon gender differences in learning.

The small sample size is a limitation of our study; however, our research still found a statistically significant difference. Future studies with a larger sample size and multiple residencies should be conducted in an effort to validate our findings.

Finally, 25 % of our subjects did not complete the 3-month follow-up due to either being away from the area conducting off-service rotations or being unwilling to volunteer to retake the test and complete the questionnaire. The loss of 25 % of our subjects to follow-up evaluation may bias our results. However, the mean immediate post-intervention test scores were similar in both arms of the study when comparing those who did and did not complete the 3-month post-intervention test. Furthermore, while not statistically significant, those in the simulation arm who did not complete the 3-month post-intervention test had a higher mean immediate post-intervention test score than those who did complete the 3-month post-intervention test. Since those who have the highest immediate post-test are likely to have the highest 3-month post-test, this would bias our study to underestimate the superior performance of simulation.

Conclusions

Lecture-based and simulation-based instruction had similar impacts on toxicology knowledge immediately after intervention. At 3 months after training, the simulation-trained group had greater retention than the lecture-trained group. Resident comfort levels for diagnosis and treatment were similar regardless of the type of education. Future studies should be conducted to determine if simulation-based training is an effective modality to ensure residents meet or exceed the new ACGME milestones for toxicology training.