Impact of Simulation-based Mastery Learning on Resident Skill Managing Mechanical Ventilators

Abstract

Background: Caring for patients requiring mechanical ventilation is complex, and residents may lack adequate skill for managing these patients. Simulation-based mastery learning (SBML) is an educational model that trains clinicians to a high standard and can reduce complications. The mastery learning model has not been applied to ventilator management.

Objective: The purpose of this study was to determine whether SBML, as compared with traditional training, is an effective strategy for teaching residents the skills necessary to manage patients requiring mechanical ventilation.

Methods: We developed an SBML curriculum and a 47-item skills checklist to test ventilator management for patients with normal, restricted, and obstructed lung physiology. A minimum passing standard (MPS) on the checklist was set using the Mastery Angoff method. Residents rotating through the medical intensive care unit in Academic Year 2017–2018 were assigned to SBML or traditional training based on their medical intensive care unit team. The SBML group was pretested on a ventilator simulator using the skills checklist. They then received a 1.5-hour session (45 min didactic and 45 min deliberate practice on the simulator with feedback). At rotation completion, they were posttested on the simulator using the checklist until the MPS was met. Both SBML-trained and traditionally trained groups received teaching during daily bedside rounds and twice weekly didactic lectures. At rotation completion, traditionally trained residents were tested using the same skills checklist on the simulator. We compared pretest and posttest performance among SBML-trained residents and end of the rotation test performances between the SBML-trained and traditionally trained residents.

Results: The MPS was set at 87% on the checklist. Fifty-seven residents were assigned to the SBML-trained group and 49 were assigned to the traditionally trained group. Mean checklist scores for SBML-trained residents improved from 51.4% (standard deviation [SD] = 17.5%) at pretest to 86.1% (SD = 7.6%) at initial posttest and 92.5% (SD = 3.7%) at final (mastery) posttest (both P < 0.001). Forty-two percent of residents required more than one attempt at the posttest to meet or exceed the MPS. At rotation completion, the traditionally trained residents had a mean test score of 60.9% (SD = 13.3%).

Conclusion: SBML is an effective strategy to train residents on mechanical ventilator management. An SBML curriculum may augment traditional training methods to further equip residents to safely manage ventilated patients.

Approximately 310 per 100,000 adults in the United States require invasive mechanical ventilation (MV) for nonsurgical indications annually (1). MV is frequently a life-saving intervention, yet it requires complex patient management and is associated with significant morbidity and mortality. Overall hospital mortality for patients receiving MV is estimated at 32% (1). Prior research shows that complications associated with MV can be avoided or minimized with close attention to multiple factors including 1) indications for initiation of MV, 2) hemodynamic effects of MV, 3) complications of sedation, 4) oxygen toxicity, 5) effects of MV on respiratory muscles, 6) risk of infection, and 7) incidence of ventilator-induced lung injury (2).

High patient volume and duty-hour requirements for resident physicians leave limited time for dedicated critical care education in the intensive care unit (ICU) (3, 4). Trainees demonstrate a wide range of critical care knowledge and skills, yet many do not have adequate evidence-based knowledge about the management of patients receiving MV (5). In one study, half of senior internal medicine (IM) residents could not identify an appropriate tidal volume for a patient with acute respiratory distress syndrome, 38% could not recognize a patient ready for a weaning trial, and nearly one-third did not recognize indications for noninvasive ventilation. In addition, nearly half of the residents in this study indicated they were dissatisfied with their training in MV and many perceived their knowledge to be only minimally adequate to provide effective patient care. Similarly, emergency medicine (EM) residents in another study frequently reported being uncomfortable caring for patients receiving MV in the emergency department (6).

Simulation-based education (SBE) may provide a solution to gaps in MV training. Multiple studies support simulation as a superior method for educating healthcare providers across a broad range of clinical skills, allowing trainees to master a skill before working with actual patients (7–9). Furthermore, the American Board of Internal Medicine recommends that IM residents receive simulator training as one strategy for determining procedural competence (10). Prior studies of SBE have established its effectiveness to improve skills in areas such as advanced cardiac life support (11, 12), central venous catheter insertion (13, 14), emergency airway management (15), thoracentesis (16), and paracentesis (17).

Traditional SBE tests learners on a simulator before and after a simulation-based skill intervention. However, learners are often not held to a high and uniform skills achievement standard. This approach results in significant variability in the skills achieved by learners after training (18–21). In contrast, simulation-based mastery learning (SBML) reduces or eliminates variability in trainee skills after an educational intervention (14, 22–24). Mastery learning is a rigorous form of competency-based learning and evaluation in which knowledge and skills are measured against a high and uniform achievement standard (25). All learners must reach this predetermined minimum passing standard (MPS) before the completion of training. This is accomplished by allowing practice times for individual learners to vary as needed so that all can achieve the MPS. An expert panel sets the MPS using methods that ensure skills are performed independently and safely on the simulator before trainees perform them on actual patients.

Studies demonstrating the use of SBE for ventilator management demonstrate improved confidence, knowledge, and skill among trainees (26–29). However, none of these studies trained residents to mastery or a predetermined MPS signifying an acceptable level of performance. Two other studies did assess SBML during an intern boot camp where MV simulation was included in a five-part clinical skills exam (30, 31). These studies showed that SBML was effective and boosted MV management skill for at least 4 weeks. However, they lacked a comprehensive assessment of skill managing the varied respiratory pathologies encountered in the ICU. True “mastery” of the skills necessary to manage patients receiving MV takes many years of advanced training. It should be noted that the term “mastery” is used in relation to a teaching methodology described by Benjamin Bloom in 1968 (32) rather than a comprehensive assessment of one’s skill managing all the complexities of MV. Important questions remain regarding whether SBML is a feasible and effective strategy for teaching residents the skills necessary to manage patients with complex respiratory physiology in an ICU setting. Therefore, the current study had the following three aims: first, to design and implement a comprehensive MV SBML curriculum for residents during their medical ICU (MICU) rotation, which includes education on varied respiratory pathologies; second, to determine whether the curriculum improves the knowledge and skills in the simulated environment of MICU residents; and third, to determine whether any learner characteristics predict resident performance when managing simulated patients receiving MV.

This work was previously presented in the form of an abstract at the American Thoracic Society 2018 International Conference (33) and at the Central Group on Educational Affairs 2018 Regional Meeting.

Methods

Study Design and Setting

We performed a prospective cohort study of residents’ MV management skills during their 1-month MICU rotations at a tertiary care academic medical center in Chicago, Illinois, from July 2017 to June 2018. The Northwestern University Institutional Review Board determined this study met the criteria for exemption from further institutional review board review. All participants provided their written informed consent before participation.

The MICU is a closed unit staffed by faculty members and fellows from the Division of Pulmonary and Critical Care Medicine (PCCM) at Northwestern University Feinberg School of Medicine. The residents provide patient care on one of two teaching services, MICU1 or MICU2. MICU1 consists of one PCCM attending; one PCCM fellow; two postgraduate year (PGY)-2 or PGY-3 IM residents; and two to four PGY-1 categorical IM, preliminary IM, or categorical anesthesiology residents. MICU2 consists of one PCCM attending; one PCCM fellow; one PGY-2 or PGY-3 IM resident; one PGY-2 EM resident; one to two PGY-1 categorical IM, preliminary IM, or categorical anesthesiology residents; and one PGY-1 EM resident. The faculty and fellows are randomly assigned to MICU1 or MICU2, and the patients are intermixed within the same physical space. The residents share a common team room but do not share patient care responsibilities.

Participants

All residents rotating through the MICU during the study period were eligible to participate in the study. Based on scheduling difficulties for EM residents, the MICU1 team was assigned to receive the SBML educational MV training intervention, whereas the MICU2 team received traditional training and served as the comparison group.

Intervention

All residents participated in traditional training, which consisted of a minimum of 4 hours of formal didactic education on various key critical care topics (including MV) provided by PCCM fellows during their 4-week rotation. Fellows review a shared slide set to cover core topics in MV during a 30–45 minute session that included selection of mode, measurement of respiratory mechanics, recognition and management of auto–positive end-expiratory pressure (PEEP), readiness to wean, and spontaneous breathing trials. In addition, all IM PGY-1 residents completed critical care skills training, including an introduction to MV, during an intern boot camp. Residents also received informal teaching during daily rounds and at the bedside and additional teaching sessions at the discretion of the PCCM attending, fellow, and residents.

In addition to traditional training, the SBML group participated in MV management SBML, whereas the traditionally trained group did not. During their first week in the MICU, the SBML group underwent a baseline skills assessment (pretest) using a case-based 47-item checklist on mechanical ventilator management (see data supplement for cases and checklist items) using the ASL5000 (IngMar Medical) MV simulator. The ASL5000 is a breathing simulator capable of simulating a full range of patients and can be used with any ventilator and any ventilator mode. Three case scenarios were designed for the pretests to include patients with normal, restricted, and obstructed lung physiology. After baseline testing, residents subsequently received 1.5 hours of education including approximately 45 minutes of didactic material and 45 minutes of deliberate practice with the simulator. Didactic teaching was standardized and provided by the study authors (C.J.S., A.F., and L.S.). Content covered broad topics, such as ventilatory modes, selection of appropriate settings, measurement of ventilatory mechanics, and weaning. During the deliberate practice sessions, the checklist questions were used as a rough template to ensure key concepts were reviewed. However, the actual test items were not provided to the learners. Teaching and deliberate practice sessions were arranged for groups ranging in size from one to four participants and were scheduled at convenient times for full resident participation. Often, these sessions were held in the afternoon of a noncall day. At the completion of the 4-week rotation, residents completed an end of the rotation MV skills posttest using the simulator and checklist. The case scenarios for the posttest were modified slightly from the pretest. The skill assessment checklist items were unchanged except to reflect changes in patient demographics, such as height and sex. All residents in the SBML group were required to meet or exceed a predetermined MPS. Those who were unable to meet or exceed the MPS underwent further deliberate skills practice on the simulator until they were able to achieve the MPS at retesting, which was completed within 1 week of their previous posttest attempt.

Residents in the traditionally trained group also underwent the same simulated MV skills posttest at the end of their MICU rotation. However, residents in the traditionally trained group were not required to meet the MPS at posttest because they did not participate in SBML.

Measurement

Demographic and learner characteristics were collected from all participants using a self-report questionnaire that included age, sex, training program, level of training (PGY), and prior ICU experience at the start of the rotation.

We developed an initial 47-item checklist using relevant literature, best practices, and checklist design strategies described by Stufflebeam and colleagues (34). After the initial checklist was developed, we used the modified Delphi technique with 10 board-certified critical care faculty physicians across three institutions and four critical care specialties (pulmonary, neurology, anesthesiology, and surgery) to finalize the checklist. The checklist addressed learning objectives based on three case scenarios for patients with normal, restricted, and obstructed lung physiology. The checklist scoring was dichotomous and graded as 1 = done correctly or 0 = not done/done incorrectly. The checklist was pilot tested on a group of 10 nonstudy subjects (critical care fellows and chief residents) and modified for clarity. Checklist raters (C.J.S., A.F., and L.S.) completed 6 hours of rater training and calibration to ensure consistent scoring of items, and they created a grading rubric for use during all testing sessions. Subsequently, a different panel of 10 board-certified expert critical care faculty (pulmonary, neurology, anesthesiology, and surgery) set the MPS for the skills using the Mastery Angoff method (35). In brief, this method asks experts to review each checklist item and estimate the percentage of well-prepared trainees who would perform the item correctly at posttest. Well-prepared trainees are those who can perform the skill safely and independently.

All pretests and posttests were graded by one of three unblinded instructors (C.J.S., A.F., and L.S.). To assess interrater reliability (IRR), pretests and posttests that were completed during the first block of the rotation were scored live by two of the three raters, including one who was blinded to the training status of the residents and one who was unblinded. We compared pretest and posttest checklist scores from SBML-trained residents to measure the impact of training sessions. SBML-trained residents completed a nine-question course evaluation questionnaire to rate their satisfaction with the curriculum based on a five-point Likert scale (1 = strongly disagree, 3 = neutral, and 5 = strongly agree) that has been used in multiple other SBML studies (22, 24). In addition, we compared posttest checklist scores from SBML-trained residents with those of traditionally trained residents performed at the end of the rotation.

Statistical Analysis

We compared demographic characteristics between the SBML training and traditional training groups using χ² or independent t tests. We analyzed differences between pretest (baseline) to posttest scores using a paired t test. We compared posttest scores between SBML-trained residents and traditionally trained residents using the independent t test. We also compared the proportion of residents meeting the MPS between training groups at posttest with the χ² test. We used multiple linear regression to evaluate the relationship between posttest scores and training status (SBML-trained vs. traditionally trained); when controlling for demographic variables with differences between groups, P < 0.01. We calculated pretest and posttest checklist score reliability using the κ coefficient. Statistical analyses were performed using IBM SPSS version 25.

Results

A total of 118 residents were eligible to participate in the study during the study period. A total of 106 residents (89.8%) completed the entire study protocol over 13 MICU rotation blocks. Twelve residents did not complete the entire protocol because of illness or scheduling difficulties. Figure 1 summarizes training assignments for the SBML-trained and traditionally trained groups. Fifty-seven residents were assigned to SBML, whereas 49 were assigned to traditional training over the study period. Table 1 displays demographic and learner characteristics for the SBML-trained and traditionally trained groups. There were no statistically significant differences in age, sex, U.S. medical school, previous ICU experience in medical school, or PGY level. There was a difference in training programs between groups, as there were no EM residents in the SBML-trained group, and there were more preliminary IM and anesthesiology residents in the SBML group. There was no difference in previous ICU rotations in residency between groups. However, when EM residents were analyzed independently, they did have more ICU rotations (mean = 2.65; standard deviation [SD] = 1.65) than other learners in the traditionally trained group (mean = 1.51; SD = 1.49) (P = 0.002).

Figure 1.

Flow diagram showing the education occurring in the SBML and the traditional training groups. MICU = medical intensive care unit; MV = mechanical ventilation; SBML = simulation-based mastery learning.

Table 1.

Learner characteristics of SBML-trained and traditionally trained residents

Characteristic	SBML-trained Residents (n = 57)	Traditionally Trained Residents (n = 49)	P Value
Age, yr, mean (SD)	28.34 (2.14)	28.35 (2.16)	0.98
Sex, n (%)			0.99
F	29 (51)	25 (51)
M	28 (49)	24 (49)
Training program, n (%)
Categorical internal medicine	46 (81)	20 (41)	<0.001
Emergency medicine	0	24 (49)
Anesthesiology	6 (11)	4 (8)
Preliminary internal medicine	5 (9)	1 (2)
Attended U.S. medical school, n (%)	57 (100)	49 (100)	—
Intensive care unit rotation in medical school, n (%)			0.28
No	31 (54)	28 (57)
Yes	25 (44)	15 (31)
Missing	1 (2)	6 (12)
Prior intensive care unit rotations as resident, n (%)
0	15 (27)	7 (14)	0.40
1–3	34 (60)	27 (55)
4–7	7 (12)	8 (16)
Missing	1 (2)	7 (14)
Postgraduate year, n (%)
1	36 (63)	28 (57)	0.53
2	10 (18)	13 (27)
3	11 (19)	8 (16)

Definition of abbreviations: SBML = simulation-based mastery learning; SD = standard deviation.

The MPS was set at 87% (41/47) of items correct on the skills checklist. SBML-trained residents significantly improved their skills performance from a mean score of 51.4% (SD = 17.5%) of items correct at pretest to 86.1% (SD = 7.6%) at initial posttest and 92.5% (SD = 3.7%) at final (mastery) posttest (P < 0.001) (Figure 2). Twenty-four (42%) of the 57 SBML-trained residents required more than one attempt at the posttest to meet or exceed the MPS for the skills checklist. Twenty-three residents required less than 30 minutes of additional deliberate practice to meet the MPS within one retesting session, and one resident required two retesting sessions. The most frequently incorrect items among the SBML-trained residents on their first attempt at posttest were checking the plateau pressure after adjusting the PEEP (26% correct) and increasing sedation as a strategy to decrease auto-PEEP in a patient with obstructive lung disease (25% correct).

Figure 2.

SBML-trained resident pretest and posttest and traditionally trained resident posttest scores (percentage correct) on a 47-item mechanical ventilation skills checklist. Each circle represents an individual resident. M = mean; MPS = minimum passing score; SBML = simulation-based mastery learning; SD = standard deviation.

At the initial posttest, SBML-trained residents significantly outperformed the traditionally trained group (86.1%; SD = 7.6% vs. 60.9%; SD = 13.3%) (P < 0.001). However, the traditionally trained group did score significantly better at posttest than the SBML-trained group did at pretest (60.9%; SD = 13.3% vs. 51.5%; SD = 17.5%) (P = 0.003), suggesting the possibility of learning during the MICU rotation. Only one resident (2%) in the traditionally trained group met the MPS at posttest. These results remained significant after controlling for training program, which had no significant impact on posttest scores. Twelve checklists were double scored for IRR calculation, consistent with previous SBML studies (36). IRR was very high for both rater combinations (C.S. and L.S. κ_n = 0.99; C.S. and A.F. κ_n = 0.98) across the 47 checklist items.

Fifty-five of 57 (96%) of SBML-trained residents completed a course evaluation questionnaire. They rated the curriculum highly (Table 2) and believed it increased their self-confidence and ability to manage patients requiring MV.

Table 2.

Course evaluation questionnaire completed by SBML-trained residents (n = 55) based on a five-point Likert scale (1 = strongly disagree; 5 = strongly agree)

Item	Mean (SD)
1. Practice during the ventilation simulation session boosts my skill to manage a ventilator.	4.76 (0.43)
2. It is OK to make clinical mistakes using the ventilator simulation model.	4.60 (0.53)
3. I received useful educational feedback from the training session.	4.78 (0.42)
4. The ventilator model simulates a mechanically ventilated patient realistically.	4.51 (0.60)
5. Practice with the ventilator simulator boosts my clinical self-confidence.	4.64 (0.62)
6. Repetitive practice using simulation is a valuable educational experience.	4.80 (0.40)
7. My instructor was prepared to teach the session.	4.91 (0.29)
8. This session has helped prepare me to better manage ventilated patients in the intensive care unit.	4.71 (0.50)
9. Please rate the overall quality of the session.*	4.89 (0.31)

For definition of abbreviations, see Table 1.

^*Five-point Likert scale (1 = very poor; 5 = very good).

Discussion

Regarding our first study objective, we designed and successfully implemented a comprehensive MV management SBML curriculum for residents during their MICU rotation. Residents rated the curriculum highly and reported improved clinical self-confidence managing mechanical ventilators. Need for this curriculum is demonstrated by poor pretest performance by residents despite prior clinical training in the ICU. This is not surprising, as traditional training models featuring passive and time-limited clinical exposure do not reliably yield adequate clinical skills (37). Even completion of the ICU rotation among traditionally trained residents did not allow for the learning of adequate skills for independent and safe MV patient care.

In terms of our second study objective, we found that SBML resulted in a significant increase in the simulated MV clinical skills of MICU residents at both initial pretest and final mastery test. It remains unknown whether the increased skill could improve the preparedness and patient care of residents who will practice in the ICU without further advanced fellowship training. Multiple other studies show that SBML improves patient care and clinical outcomes for procedures such as advanced cardiac life support (38), central venous catheter insertion (13, 14), paracentesis (17), thoracentesis (39), and laparoscopic common bile duct exploration (40). Furthermore, MV SBML standardized the educational approach to teaching, thus assuring all residents have experience managing patients with normal, restricted, and obstructed lung physiology even if their clinical experience does not offer them the opportunity to care for such patients. Forty-two percent of SBML-trained residents required additional training beyond the 4-week clinical experience and 1.5-hour curriculum to achieve mastery. This is important for two reasons. First, it suggests that mastery of the simulated skills is attainable. Second, it reflects the complexity of the skill being taught. For many, additional time is required to master the skill of MV management so that safe patient care can be delivered. The cycle of testing, feedback, and practice is the fundamental core of the SBML curriculum. Additional practice time or recall related to repeated test attempts does not diminish the learning that has occurred. Our traditional training model in the MICU does have some positive impact on resident skill managing MV. However, a mean posttest score of just 60.9% on a rigorously designed skills assessment does not suggest sufficient skill acquisition.

Our third study objective was to determine whether any learner characteristics were predictive of performance. We found that resident training program had no significant impact on performance. Despite more time spent on previous ICU rotations during residency, the more experienced EM residents, who were all in the traditionally trained group, still performed poorly on the posttest. This suggests that experience alone is a poor predictor of skill. Focused attention with time-variable education is necessary if residents are to master this skill. This is consistent with multiple other studies that show experience alone is not a proxy for skills (36, 41–43).

The Accreditation Council for Graduate Medical Education Program Requirements for Graduate Medical Education (GME) in IM expect residents to demonstrate the ability to manage patients in critical care units, where residents spend 3–6 months during their 36 month of training (44). Appropriate management of respiratory failure, which often includes MV, is imperative for all residents managing critically ill patients. In the August 2019 edition of the Journal of Graduate Medical Education, an authorship group formed through the American Thoracic Society Section on Medical Education conducted a narrative review of the available evidence for GME training in MV (45). The authors found heterogeneity within MV education, learners who were generally dissatisfied with their training in MV, and suboptimal performance on MV knowledge assessment. Furthermore, they suggested that every GME program caring for critically ill patients should adopt MV-specific learning objectives and incorporate simulation into training curricula. Finally, they suggested establishing minimum performance standards and focusing on patient-level outcomes. Our study addresses these needs in a number of ways. First, we used consensus experts to define learning objectives for MV management. Second, we created a case-based checklist and set an MPS using validated methodology. Third, we demonstrated that we could achieve the MPS for every resident who received the intervention. Our study did not address patient-level outcomes.

This study has several important limitations. First, it was conducted at a single institution, and we limited our intervention to the MICU. Therefore, we cannot extrapolate our findings to other settings and disciplines. However, with buy-in from training program leadership, the intervention could likely be expanded to other ICUs in which mechanically ventilated patients are managed. SBML curricula such as this have been successfully disseminated in the past (36, 46). Second, we did not perform a randomized controlled trial because of the possibility of cross contamination between the intervention and control groups. Therefore, there was a significant difference in the baseline characteristics between groups, with all EM residents receiving traditional training. However, we controlled for training program and found that it had no significant impact on posttest scores. Third, our raters were unblinded to the training group of the residents. However, they completed 6 hours of rater training and calibration to ensure consistent scoring of items. By double scoring a proportion of the tests, we were able to demonstrate high interrater reliability without evidence that the unblinded rater scored SBML-trained residents more favorably. Based on the clinical demands of our three raters and the need to share the responsibility for rating based on rater availability, it was impossible to keep raters blinded to the training group of the residents. Fourth, the traditionally trained group did not receive a pretest, so it is possible that the SBML-trained group was “trained to the test.” However, the test was designed based on the information and skills that the residents needed to care for patients receiving MV, and the residents were never shown the items on the skills checklist. The baseline assessment is a critical feature of mastery learning and allows “test enhanced learning” to give learners performance feedback and set them up for learning (47). The objective of this study was to determine the effectiveness of a mastery learning curriculum bundle compared with traditional training in the MICU, in which pretesting is not performed. Administering a baseline assessment of the traditionally trained group may have led to enhanced performance on a posttest. However, administering a baseline assessment without incorporating the other steps of the mastery model creates a watered-down curriculum without the demonstrated rigor and outcomes achieved with mastery (48). Based on our study design, we cannot definitively determine which individual component(s) of the intervention led to improved knowledge and skill acquisition in the SBML group. It remains unclear whether the pretest, didactic teaching, deliberate practice, time, or a combination of factors led to improved outcomes. Importantly, an SBML curriculum is time and resource intensive and may not be feasible for all institutions. Finally, we did not assess downstream clinical care, skill retention, or skills other than MV, which could be addressed in future studies. Without assessing patient-level outcomes, we cannot determine whether the knowledge and skills obtained in the simulated environment translate to the clinical environment.

Conclusions

SBML is an effective strategy to train residents on mechanical ventilator management. Relying on traditional training methods in the MICU may leave residents ill equipped to safely manage patients receiving MV. Future study should address skill retention, training of the interprofessional team, expansion to other trainees in the critical care environment, and assessment of clinical care outcomes.