Mastery Learning of Advanced Cardiac Life Support Skills by Internal Medicine Residents Using Simulation Technology and Deliberate Practice

Abstract

BACKGROUND

Internal medicine residents must be competent in advanced cardiac life support (ACLS) for board certification.

OBJECTIVE

To use a medical simulator to assess postgraduate year 2 (PGY-2) residents' baseline proficiency in ACLS scenarios and evaluate the impact of an educational intervention grounded in deliberate practice on skill development to mastery standards.

DESIGN

Pretest-posttest design without control group. After baseline evaluation, residents received 4, 2-hour ACLS education sessions using a medical simulator. Residents were then retested. Residents who did not achieve a research-derived minimum passing score (MPS) on each ACLS problem had more deliberate practice and were retested until the MPS was reached.

PARTICIPANTS

Forty-one PGY-2 internal medicine residents in a university-affiliated program.

MEASUREMENTS

Observational checklists based on American Heart Association (AHA) guidelines with interrater and internal consistency reliability estimates; deliberate practice time needed for residents to achieve minimum competency standards; demographics; United States Medical Licensing Examination Step 1 and Step 2 scores; and resident ratings of program quality and utility.

RESULTS

Performance improved significantly after simulator training. All residents met or exceeded the mastery competency standard. The amount of practice time needed to reach the MPS was a powerful (negative) predictor of posttest performance. The education program was rated highly.

CONCLUSIONS

A curriculum featuring deliberate practice dramatically increased the skills of residents in ACLS scenarios. Residents needed different amounts of training time to achieve minimum competency standards. Residents enjoy training, evaluation, and feedback in a simulated clinical environment. This mastery learning program and other competency-based efforts illustrate outcome-based medical education that is now prominent in accreditation reform of residency education.

The American Board of Internal Medicine (ABIM) requires candidates for certification to be judged competent by their residency program director in 9 procedures including advanced cardiac life support (ACLS).¹ Internal medicine residents frequently satisfy this requirement by completing an American Heart Association (AHA) ACLS provider course. These courses typically include 1 day of reading, lecture, and practical instruction about the recognition and treatment of ACLS events. Despite recommended ACLS renewal on a 2-year cycle,² it has been shown that physicians, nurses, and laypersons display poor skill retention in shorter time periods.^3–5 Several authors have argued that frequent refresher courses should be added to current training protocols to increase ACLS skill and knowledge retention.^6,7

In addition to concerns about the adequacy of training in ACLS, another barrier to certifying residents as competent in these required procedures is the knowledge that in-hospital events prompting an ACLS response occur rarely. A recent review of 207 academic and community hospitals showed that the average number of annual events requiring an ACLS response was 54.1 per facility.⁸ Furthermore, data on in-hospital cardiac arrests from the University of Chicago demonstrated that the quality of resuscitation efforts varied and often did not meet published guidelines, even when performed by well-trained hospital personnel.⁹ Thus, internal medicine residents are expected to recognize and manage life-threatening events that occur infrequently, in which their performance is often not subject to audit or accountability assessment, and for which they may be poorly prepared and insufficiently practiced.

Medical education at all levels increasingly relies on simulation technology to provide a tool to increase learner knowledge, provide controlled, safe, and forgiving practice opportunities, and shape the acquisition of physicians' clinical skills.^10–12 Simulations vary in fidelity from inert task trainers used to practice endotracheal intubation to standardized patients to sophisticated mannequins linked to computer systems that can mimic complex medical problems, display interacting physiologic and pharmacologic parameters, and present problems in “real time.”¹³ Combined with opportunities for controlled, deliberate practice with specific feedback,^14,15 simulations are highly effective at promoting skill acquisition among medical learners^15–17 and generalizing simulation-based learning into patient care settings.^18,19 Gaining proficiency in clinical skills also gives rise to a sense of self-efficacy²⁰ among medical learners, an affective outcome that accompanies mastery experiences.

Mastery learning,²¹ an especially stringent variety of competency-based education,²² means that learners acquire essential knowledge and skill, measured rigorously against fixed achievement standards, without regard to the time needed to reach the outcome. Mastery indicates a much higher level of performance than competence alone. In mastery learning, educational results are uniform, with little or no variation, while educational time varies among trainees. This approach to education has its origins in theory and data beginning at least 4 decades ago,^23–28 and yet finds contemporary expression in statements about outcome-based medical education²⁹ and calls for accreditation reform in internal medicine residency programs.³⁰ To our knowledge, a genuine mastery learning model has never been used in U.S. medical education. This is the first empirical report of a mastery learning application in clinical medical education.

The study reported in this article amplifies a randomized-controlled trial (RCT) about ACLS skill acquisition among internal medicine residents reported previously by our research group.³¹ In the earlier RCT, we demonstrated a 38% improvement in ACLS skills after an 8-hour simulation-based educational intervention compared with clinical experience alone. This report presents original research data that extend the previous RCT in 2 ways: (a) it uses research-based mastery standards as floors for resident performance of 6 ACLS scenarios; and (b) unlike previous research where learning time was fixed and learning outcomes were varied, in this study each resident met or exceeded the minimum mastery standard for each procedure while learning time varied. The intent was to produce high achievement among internal medicine residents in ACLS procedures required for board certification with little or no outcome variation.

METHODS

Objectives and Design

The study was a pretest-posttest design without a control group³² of a simulation-based, mastery learning educational intervention designed to increase internal medicine residents' clinical skills in ACLS procedures. Primary measurements were obtained at baseline (pretest) and after the educational intervention (posttest).

Participants

Study participants were all 41 second-year residents at Northwestern University's Chicago campus internal medicine residency program from July 2004 to March 2005. The Northwestern University Feinberg School of Medicine Institutional Review Board approved the study. Participants provided informed consent before the baseline assessment.

The residency program is based at Northwestern Memorial Hospital and the Jesse Brown Veteran's Affairs Medical Center. Resident teams respond to all cardiac arrests at both hospitals. Teams are composed of 2 to 3 internal medicine residents and representatives from the anesthesia, surgery, and nursing services. All residents complete an AHA-approved ACLS provider course at the beginning of residency training and again 2 years later. However, only second- and third-year residents are designated as the code leader. First-year residents respond to cardiac arrests but do not serve as code leaders.

Procedure

The study began in July 2004 at the mid-point of the residents' ACLS 2-year renewal cycle. All 41 residents were retained as an intact group throughout the 8-month study. The research procedure had 2 phases. First, all residents underwent baseline pretesting on a random sample of 3 of 6 ACLS scenarios. Second, the residents received a minimum of 4, 2-hour education sessions with deliberate practice of ACLS events and procedures using a medical simulator. After training, the residents were retested and expected to meet or exceed a minimum passing score (MPS) on each of 6 ACLS scenarios. Those who failed to meet the MPS on any ACLS scenario engaged in more deliberate skills practice until the mastery standard was reached. The amount of extra training time that these residents needed to achieve the MPSs was recorded.

Residents were scheduled according to their clinical responsibilities and the capacities of the simulator center. Efforts were made to standardize the schedule so that training sessions occurred weekly and testing occurred no more than 1 to 2 weeks after the last training session.

Educational Intervention

The intervention was designed to help residents acquire, shape, and reinforce the clinical skills needed to respond to ACLS situations. The intent was to engage the residents in deliberate practice involving high-fidelity simulations of clinical events. Deliberate practice involves (a) repetitive performance of psychomotor skills in a focused domain, coupled with (b) rigorous skills assessment that provides learners (c) specific, informative feedback, which results in increasingly (d) better skills performance in a controlled setting.¹⁴ Research on the acquisition of expertise consistently shows the importance of intense, deliberate practice in a focused domain, in contrast with so-called innate abilities (e.g., measured intelligence) for the acquisition, demonstration, and maintenance of mastery.¹⁴ Thus, practice, feedback, and correction in a supportive environment were the operational rules of the educational intervention.

The study was conducted in Northwestern Memorial Hospital's Patient Safety Simulation Center using the life-size Human Patient Simulator (HPS^®) developed by Medical Education Technologies Inc., Sarasota, FL. Using computer software, the mannequin displays multiple physiologic and pharmacologic responses observed in ACLS situations. Features of the mannequin include responses of the respiratory system, pupils, and eyelids as well as heart sounds and peripheral pulses. Monitoring of noninvasive blood pressure, arterial oxygen saturation, electrocardiogram, and arterial blood pressure is available. Simulator personnel can also give the mannequin voice through a speaker in the occipital area by talking into a microphone in an adjacent control room.

Six case scenarios were developed to assess resident proficiency in ACLS techniques. The scenarios are based on case studies described in the ACLS Provider Manual used by the AHA as instructional materials for ACLS provider courses.³³ The 6 scenarios (asystole, ventricular fibrillation, supraventricular tachycardia, ventricular tachycardia, symptomatic bradycardia, and pulseless electrical activity) were selected because they were the ones most commonly encountered by residents in actual ACLS situations during a 15-month monitoring period during 2003 to 2004. Scenarios began with a brief clinical history also based on content found in the AHA text.

Simulator sessions were standardized and labeled as teaching or testing sessions. Teaching sessions gave groups of 2 to 4 residents time to practice protocols and procedures and to receive structured education from simulator faculty. Debriefing allowed the residents to ask questions, review algorithms, and receive feedback. The 4 teaching sessions were presented in uniform order: (a) procedures—intubation, central line placement, pericardiocentesis, and needle decompression of tension pneumothorax; (b) pulseless arrhythmias—asystole, ventricular fibrillation, and pulseless electrical activity; (c) tachycardias—supraventricular and ventricular; and (d) bradycardias—second- and third-degree AV block.

Two residents were present at each baseline pretesting session. One resident and an experienced ACLS instructor were present for posttest measurement. While one resident directed resuscitation efforts, the other individual performed cardiopulmonary resuscitation (CPR) or other tasks but did not make management decisions or lead the arrest scenario. The presentation order of the ACLS scenarios was randomized within each testing session. As described in the ACLS guidelines, residents were expected to obtain a history, perform a physical examination, request noninvasive and invasive monitoring, order medications, procedures, and tests, and direct resuscitative efforts of other participants. Residents did not review the scenarios before the session and were not permitted to use written materials while directing the simulations.

Measurements

A checklist was developed for each of the 6 conditions from the ACLS algorithms using rigorous step-by-step procedures.^34,35 Within the checklists, each patient assessment, clinical examination, medication, or other action was listed in the order recommended by the AHA and given equal weight. None of the checklist items were weighted differentially. A dichotomous scoring scale of 0=not done/done incorrectly and 1=done correctly was imposed for each item. Checklists were reviewed for completeness and accuracy by 3 of the authors (D.B.W., J.B., and V.J.S), all of whom are ACLS providers. One author (V.J.S.) is an ACLS instructor.

The MPS for each ACLS procedure was determined in a previous research study involving 12 clinical experts using the Angoff and Hofstee standard setting methods.³⁶ Each of the 6 MPSs is the average of the Angoff- and Hofstee-derived standards.

Evaluations of each resident's adherence to the 6 ACLS protocols were recorded by 1 of the 2 faculty raters on the checklists during the testing sessions. A 20% random sample of the testing sessions was rescored by the other rater from videotapes to assess interrater reliability. The rescoring was blind to the results of the first checklist recording.

Demographic data were obtained from the participants, including age, gender, ethnicity, medical school, and scores on the United States Medical Licensing Examination (USMLE) Steps 1 and 2. Each resident's experience in managing patients with any of the 6 conditions was collected at each test occasion.

Primary outcome measures were posttest checklist scores. Secondary outcome measures were the total training time needed to reach the MPS for all 6 ACLS problems and a course evaluation questionnaire completed at the end of the study period.

Raw checklist scores ranged from 16 to 31 items for the 6 ACLS simulations.

Data Analysis

Checklist score reliability was estimated in 2 ways: (a) interrater reliability was calculated using the κ coefficient³⁷ adjusted using the formula of Brennan and Prediger,³⁸ while (b) internal consistency reliability was derived using Cronbach's α coefficient.³⁹ Within-group differences from pretest (baseline) to posttest (outcome) were analyzed using paired t tests. The association of ACLS posttest performance with (a) pretest ACLS performance, (b) medical knowledge measured by USMLE Steps 1 and 2, and (c) whether additional training was needed to master the 6 ACLS procedures was assessed using multiple regression analysis.

RESULTS

All residents consented to participate and completed the entire training protocol. The simulator operated without error or breakdown.

Table 1 presents demographic data about the 41 residents who participated in the study. These data provide a descriptive portrait of the physician trainees. A majority of the residents had little or no experience responding to actual ACLS situations during the first residency year.

Table 1.

Baseline Demographic Data

Characteristic	PGY-2 Residents
Mean age (y)	27.39
SD	1.53
Male	24 (58.5%)
Female	17 (41.5%)
White	21 (51.2%)
Asian	17 (41.5%)
Other	3 (7.3%)
U.S. medical school graduate	40 (97.6%)
Foreign medical school graduate	1 (2.4%)
Actual cardiac arrests participated in during first year of training
0 to 5	10 (24.4 %)
5 to 10	21 (51.2%)
10 to 15	7 (17.1%)
>15	3 (7.3%)

PGY-2, postgraduate year 2

Table 2 reports descriptive statistics about resident performance on each of the 6 ACLS scenarios at the baseline pretest and at the posttest expressed in 2 ways. The Matched posttest presents resident performance on the same 3 ACLS problems that each resident received as a random sample at pretest. The Complete posttest displays resident performance on all 6 ACLS problems. The number of items on each checklist, interrater reliability coefficients expressed as the mean κ, Cronbach's α internal consistency reliability coefficients, and the MPS for each ACLS scenario are also given. Without exception, the reliabilities indicate a high degree of interrater agreement about resident scenario performance and good internal consistency.

Table 2.

Checklist Descriptive Statistics and Reliabilities^*

Scenario	Pretest (3 Stations, n=41)	Matched posttest (3 Stations, n=41)	Complete posttest (6 Stations, n=41)
	Mean (SD)	Mean (SD)	Mean (SD)
Asystole (n†=17)
κ¯n=0.82,‡α=0.53,§ MPS=74.3	72.3 (12.6)	91.9 (6.3)	92.4 (5.8)
Ventricular fibrillation (n†=31)
κ¯n=0.86,‡α=0.82,§ MPS=76.4	75.4 (8.2)	87.6 (6.5)	86.0 (10.8)
Supraventricular tachycardia (n†=30)
κ¯n=0.82,‡α=0.68,§ MPS=72.4	72.3 (11.2)	89.0 (5.4)	88.2 (6.0)
Ventricular tachycardia (n†=22)
κ¯n=0.93,‡α=0.50,§ MPS=74.4	83.1 (8.8)	92.9 (7.4)	93.8 (6.7)
Symptomatic bradycardia (n†=19)
κ¯n=0.79,‡α=0.32,§ MPS=71.5	59.7 (8.1)	85.8 (8.6)	84.0 (8.7)
Pulseless electrical activity (n†=16)
κ¯n=0.84,‡α=0.43,§ MPS=76.6	75.6 (10.3)	90.1 (5.9)	89.1 (5.4)

^*Tabular entries=percentage correct

^†n=number of checklist items

^‡κ¯_n=mean kappa inter-rater reliability coefficient

^§α=internal consistency coefficient Cronbach alpha

MPS, minimum passing score

Thirty-three of the 41 medicine residents (80.5%) achieved mastery within the standard 8-hour training and deliberate practice ACLS curriculum. The remaining 8 residents (19.5%) needed extra time to reach mastery ranging from 15 minutes to 1 hour. Only 5 residents needed a full extra hour of deliberate practice to reach all 6 ACLS scenario MPSs.

The aggregate research outcomes expressed as percentages³¹ for all 6 ACLS scenarios are reported as a figure in the Appendix (online). This figure shows that at the baseline pretest the residents' mean percent correct for the ACLS scenarios was 72.2% (95% confidence interval [CI] 71.0% to 73.4%). The Matched and Complete posttest scores were identical statistically (Matched=88.7%; 95% CI, 88.0% to 89.4% vs Complete=87.9%; 95% CI, 87.2% to 88.6%). The pretest to posttests contrasts in overall ACLS performance represents a 24% improvement, a highly significant difference (P≤.0001). Results from the regression analysis indicate that neither pretest ACLS scores nor USMLE Step 1 and 2 scores were significant predictors of posttest performance. However, the need for additional deliberate practice for those who failed to reach the overall mastery standard on the posttest was a powerful negative predictor of posttest performance. For the Matched posttest b=−7.1 (95% CI −11.3 to −2.9; P=.002; r²=.27). For the Complete posttest b=−8.2 (95% CI −11.8 to −4.5; P<.0001; r²=.37). On average, for those who required additional practice to achieve mastery, the predicted posttest scores were 7.1% and 8.2% lower than their peers for the Matched and Complete posttests, respectively. Thus, in both cases the need for extra deliberate practice was associated with relatively lower posttest scores despite the outcome that all residents met or exceeded the rigorous ACLS minimum passing standards.

Follow-up data from 40 of the 41 residents on a course evaluation questionnaire about the ACLS training experience were uniformly positive. Responses were recorded on a Likert scale where 1=strongly disagree, 2=disagree, 3=uncertain, 4=agree, and 5=strongly agree (Table 3). The questionnaire data show that residents strongly agree that practice with the medical simulator boosts their clinical skill and self-confidence, that practice with the simulator should be a required component of residency education, and that the medical simulator prepared them to be a code leader better than the AHA ACLS provider course. The data also reveal resident uncertainty that practice with the medical simulator has more educational value than patient care. This underscores the recognition that deliberate practice with medical simulators complements but does not replace patient care in graduate medical education.^10–19

Table 3.

ACLS Course Evaluation Sample Items

	Mean	SD
All Residents (n=40)
1. Practice with the medical simulator boosts my clinical skill	4.6	0.8
2. It is ok to make clinical mistakes using the medical simulator	4.1	1.3
3. I receive useful educational feedback from the medical simulator	4.4	1.2
4. Practice with the medical simulator boosts my clinical self-confidence	4.4	0.9
5. Practice with the medical simulator has more educational value than patient care	3.1	1.6
6. The simulator center staff is competent	4.5	0.9
7. Practice sessions using the medical simulator should be a required component of residency education	4.4	1.3
8. Practice with the medical simulator has helped prepare me to be a code leader better than clinical experience alone	4.4	0.9
9. The medical simulator has helped prepare me to be a code leader better than the ACLS course I took	4.7	0.6
Extra Practice Residents (n=8)
1. The additional training I had was necessary	3.9	1.1
2. My questions were answered sufficiently	4.5	1.0
3. I was angry that I had to return for more training	1.8	1.0
4. I was embarrassed that I had to return for more training	2.1	1.5
5. I felt the additional training increased my ability to lead a hospital code	4.2	1.3

¹Strongly Disagree

²Disagree

³Uncertain

⁴Agree

⁵Strongly Agree

ACLS,advanced cardiac life support.

The 8 residents who needed extra practice time to reach the mastery standard provided additional positive endorsement about the educational experience. They acknowledged that the additional training time was needed, were not angry or embarrassed about returning for more training, and believed that the extra practice increased their ability to lead a hospital code (Table 3).

COMMENT

This study illustrates the application of a mastery-learning model to skill acquisition in internal medicine residency training. Use of a computer-enhanced mannequin in a structured educational program that features a combination of clear learning objectives, opportunities for deliberate practice with feedback, rigorous outcome measurement, and high achievement standards yielded large and consistent improvements in residents' ACLS skills. As shown in Table 2, despite fulfillment of an AHA provider course and clinical experience, the residents met the MPS for only 1 of the 6 ACLS skills (17%) before the educational intervention. By contrast, after the mastery-learning program, all of the residents met or exceeded the MPS for 100% of the skills. Given recent concerns about the quality and adherence to published guidelines of in-hospital cardiac arrests,⁹ programs such as this could be a useful adjunct to current ACLS certification procedures. As shown by our experience to date with several classes of residents (n=118), this model is effective, feasible, and practical for use in internal medicine residency training. In addition, the residents have consistently enjoyed participating in the mastery educational experience.

Our data also demonstrate that medical knowledge, measured by USMLE Step 1 and 2 scores, had no correlation with ACLS skill acquisition measured by checklists. This reinforces findings from our previous randomized trial of ACLS skill acquisition³¹ and supports the difference between professional and academic achievement described in earlier research.^25,40 Pretest skill performance also had no correlation with posttest results. However, the amount of deliberate practice needed to reach the mastery standard was a powerful negative predictor of posttest scores. Even though all residents met or exceeded the 6 MPSs, the need for more deliberate practice is associated with a relatively lower posttest performance.

A key question about this and other simulation-based studies is whether performance in a highly controlled simulator environment will generalize to variable clinical practice settings. This is especially the case for critical yet infrequent clinical events like in-hospital ACLS events.⁸ Several studies have demonstrated a simulator to clinic relationship^18,19; yet, the generalizability of simulator training to clinical practice warrants further study. Studies that aim to generalize skill acquisition in simulator environments to in vivo clinical practice should at minimum contain large and representative samples of medical trainees. However, such studies must also attend to such basic principles of causal generalization from laboratory to life as surface similarity, ruling out irrelevancies, making discriminations, interpolation and extrapolation, and causal explanation.³²

This project is an operational expression of outcome-based education now at the heart of calls for reform of internal medicine residency program accreditation. It shows that shopworn expressions about medical skill acquisition such as “see one, do one, teach one” are inadequate and obsolete. Goroll et al.³⁰ argue on behalf of the Residency Review Committee for Internal Medicine of the Accreditation Council for Graduate Medical Education for “… a new outcomes-based accreditation … [that] shifts residency program accreditation from external audit of educational process to continuous assessment and improvement of clinical competence” (p. 902). Our study is an objective, data-based demonstration of the proposed accreditation model. Further work at our institution is ongoing to expand the mastery model to include other required procedures, document compliance with published guidelines in actual ACLS events, and assess long-term retention of skill following initial simulator training.

This study has several limitations. It was conducted within one residency program at a single academic medical center. The sample size (n=41) was relatively small. The computer-enhanced simulation mannequin was used for both education and testing, potentially confounding the events. These limitations do not, however, diminish the pronounced impact that the simulation-based training produced among the medical residents.

This article also complies with recommendations for reporting nonrandomized evaluations of behavioral and public health interventions contained in the TREND statement.⁴¹

In conclusion, our study demonstrates the ability of deliberate practice using a medical simulator to produce mastery performance on ACLS scenarios at high achievement standards among internal medicine residents. The project was implemented successfully in a complex residency schedule, received high ratings from learners, and complies with new residency program accreditation requirements because it provides reliable assessments of residents' ACLS competence.

Supplementary Material

The following supplementary material is available for this article online: