Simulation-based Mastery Learning Improves Critical Care Skills of Advanced Practice Providers

Abstract

Background

Advanced practice providers (APPs) are essential members of intensive care unit (ICU) interprofessional teams and are expected to be competent in performing procedures. There are no published criteria for establishing when APPs can independently perform procedures. Simulation-based mastery learning (SBML) is an effective strategy for improving critical care skills but has not been applied to practicing ICU APPs.

Objective

The purpose of this study was to evaluate if an SBML curriculum could improve the critical care skills and procedural self-confidence of ICU APPs.

Methods

We performed a pretest–posttest study of central venous catheter (CVC) insertion, thoracentesis, and mechanical ventilation (MV) management skills among ICU APPs who participated in an SBML course at an academic hospital. For each skill, APPs underwent baseline skills assessments (pretests) on a simulator using previously published checklists, followed by didactic sessions and deliberate practice with individualized feedback. Within 2 weeks, participants were required to meet or exceed previously established minimum passing standards (MPS) on simulated skills assessments (posttests) using the same checklists. Further deliberate practice was provided for those unable to meet the MPS until they retested and met this standard. We compared pretest to posttest skills checklist scores and procedural confidence.

Results

All 12 eligible ICU APPs participated in internal jugular CVC, subclavian CVC, and MV training. Five APPs participated in thoracentesis training. At baseline, no APPs met the MPS on all skills. At training completion, all APPs achieved the mastery standard. Internal jugular CVC pretest performance improved from a mean of 67.2% (standard deviation [SD], 28.8%) items correct to 97.1% (SD, 3.8%) at posttest (P = 0.005). Subclavian CVC pretest performance improved from 29.2% (SD, 32.7%) items correct to 93.1% (SD 3.9%) at posttest (P < 0.001). Thoracentesis pretest skill improved from 63.9% (SD, 30.6%) items correct to 99.2% (SD, 1.7%) at posttest (P = 0.054). Pretest MV skills improved from 54.8% (SD, 19.7%) items correct to 92.3% (SD, 5.0%) at posttest (P < 0.001). APP procedural confidence improved for each skill from pre to posttest.

Conclusion

SBML is effective for training APPs to perform ICU skills. Relying on traditional educational methods does not reliably ensure that APPs are adequately prepared to perform skills such as CVC insertion, thoracentesis, and MV management.

Advanced practice providers (APPs), including both nurse practitioners (NPs) and physician assistants (PAs), are essential members of interprofessional patient care teams. Many institutions have employed APPs across a number of inpatient settings, including intensive care units (ICUs), where they have positively impacted patient care by decreasing the length of stay and improving patient outcomes (1–5). In the United States in 2015, approximately 10,000 APPs listed the ICU as their primary clinical site (6). In addition, in recent years, there has been concern about shortages of critical care physicians (7–9) that has only been exacerbated by the coronavirus disease (COVID-19) pandemic. Therefore, the ability to staff ICUs with well-trained APPs has become even more essential.

Although postgraduate training programs exist for APPs, they are not widely available, and most critical care APPs do not receive advanced specialty training beyond what is institution-specific and provided during job orientation (9, 10). Despite the paucity of standardized postgraduate training, ICU APPs are expected to be competent in a broad number of complex cognitive and bedside procedural skills as they enter the workforce. In addition, policies for demonstrating procedural readiness vary by institution, with no published criteria for establishing when APPs are able to independently perform invasive procedures without direct supervision. Furthermore, it is uncertain whether critical care APPs maintain an adequate volume of procedures to maintain their clinical skills over the span of their careers. The traditional training model at our institution requires APPs to attempt 10 central venous catheter (CVC) insertions, 10 thoracenteses, and manage 10 patients receiving mechanical ventilation (MV), under direct observation from a physician, before receiving privileges for independent practice. These direct observations are more subjective and not on the basis of rigorous skills evaluations using standardized assessment tools.

Multiple studies have demonstrated that simulation-based education improves APP confidence and skills (11–14). However, traditional simulation-based skill interventions do not hold participants to high or uniform standards (15–17) and are not often used to train participants to a competency standard. Simulation-based mastery learning (SBML) offers a method to train APPs to a high degree of competency before they perform procedures on patients. SBML is a rigorous form of competency-based education in which all learners are required to meet or exceed a predetermined minimum passing standard (MPS) on a simulated skills assessment before performing the clinical skill in actual patient care (18). To accomplish this goal, SBML requires that all participants undergo a pretest assessment followed by active educational sessions consisting of deliberate practice with feedback. Deliberate practice specifically engages learners in organized and goal-oriented tasks under direct supervision from teachers who readily provide feedback and coaching, allowing learners to make corrections and improve their overall skills. Learners are then retested until they are able to meet or exceed the MPS and move on to the next unit of study (19, 20). In the mastery learning model, training time varies, but skill achievement is uniform (18, 21). Studies demonstrate that SBML is more effective than traditional clinical education or simulation without mastery learning because it decreases or eliminates the variability in learners’ skills after an educational intervention (15–18, 22, 23). More specifically, studies have shown SBML to be effective at improving physicians’ invasive critical care skills, such as CVC insertion (24–26), thoracentesis (27–29), and MV management (30).

Important questions remain as to whether SBML is an effective and feasible educational strategy for teaching practicing APPs important critical care skills. The current study had the following aims: 1) to investigate how many ICU APPs were able to meet or exceed the MPS at baseline for three commonly performed critical care skills (CVC insertion, thoracentesis, and managing patients receiving MV); 2) to implement an SBML curriculum and evaluate its effect on simulated skills; and 3) to determine participants’ confidence in performing these skills after course completion.

Methods

Study Design and Setting

We performed a pretest–posttest study of CVC insertion (internal jugular [IJ] and subclavian [SC]), thoracentesis, and MV management skills among APPs who participated in an SBML Continuing Medical Education course at the Northwestern University Feinberg School of Medicine Simulation Center (Northwestern Simulation) from January to June 2019. These critical care skills were selected because they are common ICU procedures for which robust assessment tools exist (24, 25, 28, 30). Participants could receive up to 14 American Medical Association Physician’s Recognition Award (AMA PRA) Category 1 credits after completing the course. The Northwestern University Institutional Review Board approved this study.

Participants

All full-time APPs whose primary clinical practice sites were the adult ICUs at Northwestern Memorial Hospital, a tertiary care academic medical center in Chicago, were eligible to participate in the study. All participants provided written informed consent before participation.

Intervention

The CVC, thoracentesis, and MV SBML interventions have been previously described (24, 25, 28, 30). In brief, for each skill, the APPs received baseline skills assessments (pretests) on appropriate simulators using previously published skills checklists. Pretests were followed immediately by teaching sessions that included didactic content in which participants viewed a recorded lecture (CVC and thoracentesis) or attended an in-person lecture (MV) followed by demonstrations of all procedures. Subsequently, learners participated in deliberate practice for each skill on the simulators with individualized feedback provided in real-time by educators. Teaching sessions and deliberate practice lasted 4 hours for CVC insertion and 2 hours each for thoracentesis and MV management. Additional details of the SBML interventions are provided in Table 1.

Table 1.

Details on the teaching, content, and assessment for each of the clinical skills taught and assessed during the course

Clinical Skill	Curriculum, h	Simulator or Model	Skills Assessed	Outcome Measurement	MPS (Items Correct, %)
CVC insertion (IJ and SC) (24, 25)	4	CentraLineMan (Simulab)	Obtaining informed consent, performing CVC insertion under ultrasound guidance, maintaining sterile technique, communicating with a nurse, and procedural wrap-up	IJ procedural skills 29-item checklist	88
				SC procedural skills 30-item checklist	87
Thoracentesis (28)	2	Blue Phantom Midscapular Thoracentesis Ultrasound Training Model	Obtaining informed consent, performing thoracentesis under ultrasound guidance, maintaining sterile technique, communicating with a nurse, and procedural wrap-up	Thoracentesis procedural skills 26-item checklist	84
MV management (30)	2	ASL 5000 Breathing Simulator (IngMar Medical)	Assessing the selection of ventilatory modes and settings, measurement of ventilatory mechanics, and ventilator liberation in patients with normal, restrictive, and obstructive physiology	Mechanical ventilator skills 47-item checklist	87

Definition of abbreviations: CVC = central venous catheter; IJ = internal jugular; MPS = minimum passing standard; MV = mechanical ventilation; SC = subclavian.

Within 1–2 weeks of baseline assessment and the deliberate practice session, all participants were tested again to evaluate whether or not they were able to meet or exceed the previously established MPS on a simulated skills assessment (posttest) for each skill, using the same checklists. The checklists were not provided to the participants at any stage of the study. Those unable to meet or exceed the MPS underwent further deliberate skills practice on the simulator until they were able to achieve the MPS at retesting within 1 week after their initial posttest attempt. Didactic teaching, deliberate practice, and skills assessments were performed by one of four evaluators who were experienced in performing critical care procedures and SBML methodology.

CVC and MV SBML were offered to all participants. Thoracentesis was only offered to those APPs who worked in the medical ICU, where the procedure is more commonly performed by APPs at our institution. As such, only five participants were eligible for and participated in thoracentesis assessment and training. The course was held over several half- and full-day sessions, and each learner had an individualized schedule allowing for one-on-one skills assessments, deliberate practice using the simulators, and feedback.

Measurement

Learner characteristics were collected from all participants using a self-report questionnaire that included age, sex, training background (e.g., NP, PA, or other), total years as an APP, and total years as an ICU APP. A clinical experience survey was also collected, which included self-reported experience (e.g., number of times performing each skill) in the three critical skills examined in this study (CVC insertion, thoracentesis, and managing MV). We also surveyed APPs’ self-confidence (0 = not confident to 100 = very confident) performing these skills before and after the curriculum. Finally, participants completed an eight-question course evaluation to rate their satisfaction with the curriculum on the basis of a five-point Likert scale (1 = strongly disagree, 3 = neutral, and 5 = strongly agree) that was modified from a questionnaire used in other SBML studies. All surveys were adapted from previously published work (25, 28–31).

Details on the development and validation of the skills checklists, establishment of MPS for each skill, and creation of SBML curricula have been previously published (24, 25, 28, 30). In brief, checklists were created through the review of relevant literature and best clinical practices and by using design strategies previously described (32). The checklists were subsequently revised and finalized using a modified Delphi approach by a panel of multidisciplinary, board-certified physicians. The MPS was previously set using the modified Angoff and Hofstee methods for CVC and thoracentesis (25, 28, 33) and the Mastery Angoff method for MV management (30). These widely accepted standard-setting approaches ask expert judges to review the individual checklist items and estimate the percentage of trainees who would perform each item correctly at the posttest. Checklists for CVC insertion and thoracentesis included every step of the procedure, starting from obtaining informed consent and ending with informing the nurse that the procedure is complete. The MV checklist assessed ventilator management skills for patients with normal, restricted, and obstructed lung physiology.

Checklist raters for MV management had previously completed rater training and calibration. The interrater reliability (IRR) was high (κ_n = 0.98) (30). Thirty percent of CVC skills assessments (pre- and posttests) were double-scored in real-time by two independent raters (J.H.B. and C.J.S.) to evaluate IRR. Twenty percent of thoracentesis assessments (pre- and posttests) were double-scored in real-time by two independent raters (C.J.S. and J.M.W.). Pre- and posttest checklists were scored by graders using dichotomous grading (1 = done correctly or 0 = not done or done incorrectly). The MPS for each skill can be found in Table 1.

We compared pre- to posttraining skills checklist scores and self-confidence from SBML-trained APPs to measure the impact of training sessions.

Statistical analysis

We used descriptive statistics to report APP demographics and clinical experience. We assessed IRR using the Kappa coefficient for CVC checklists. Given a small participant sample size for thoracentesis, we used percent agreement to measure IRR on thoracentesis checklists. We analyzed differences between pretest (baseline) and posttest scores for each skill using paired t tests. We also compared APP confidence before and after training on each skill using paired t tests. Statistical analyses were performed using IBM SPSS version 25.

Results

All 12 eligible ICU APPs participated in IJ CVC, SC CVC, and MV training. Five learners participated in thoracentesis training. Learner demographics and experience are available in Table 2. The APPs were predominantly women (92%). Sixty-seven percent were NPs and 33% were PAs. Their total years as practicing APPs were variable, ranging from fewer than 2 years to more than 4 years. Similarly, baseline experience with procedures before engaging in this study was variable.

Table 2.

Learner characteristics and clinical experience of simulation-based mastery learning-trained advanced practice providers

APP Characteristic	N = 12
Age (yr), mean (SD)	31.8 (4.5)
Sex, n (%)
Female	11 (92)
Male	1 (8)
Training background, n (%)
Nurse practitioner	8 (67)
Physician assistant	4 (33)
Other	0 (0)
Total years as APP, n (%)
0 to <2	4 (33)
⩾2 to <4	5 (42)
⩾4–7	3 (25)
IJ CVC experience, n (%)
0	3 (25)
1–3	4 (33)
>3 to ⩽10	2 (17)
>10	2 (17)
Missing	1 (8)
SC CVC experience, n (%)
0	8 (67)
1–3	2 (17)
⩾4	1 (8)
Missing	1 (8)
Thoracentesis experience,* n (%)
0	2 (40)
1–3	3 (60)
MV management experience,† n (%)
0	2 (17)
1–15	3 (25)
16–100	3 (25)
>100	3 (25)
Missing	1 (8)

Definition of abbreviations: APP = advanced practice provider; CVC = central venous catheter; IJ = internal jugular; MV = mechanical ventilation; SC = subclavian; SD = standard deviation.

^*Thoracentesis experience was only reported for the five participants involved in thoracentesis training.

^† Numbers listed indicate the number of patients managed who required mechanical ventilation.

The IRR across all IJ and SC CVC items was high, with a mean Kappa of 0.94. Of the double-scored thoracentesis checklists, there was 99% agreement on all checklist items. At baseline, no APPs met the MPS on all three clinical skills. Three participants met the MPS for IJ CVC at baseline, one for SC CVC, two for thoracentesis, and none for ventilator management. After SBML training, the APPs significantly improved their performance, and all met the MPS for each skill (Figure 1). Mean pretest IJ CVC skill improved from 67.2% (standard deviation [SD], 28.8%) checklist items correct to 97.1% (SD, 3.8%) at posttest (P = 0.005). Mean pretest SC CVC skill improved from 29.2% (SD, 32.7%) checklist items correct to 93.1% (SD, 3.9%) at posttest (P < 0.001). For the five learners who participated in thoracentesis training, the mean pretest skill improved from 63.9% (SD, 30.6%) checklist items correct to 99.2% (SD, 1.7%) at posttest (P = 0.054). MV pretest skill improved from a mean of 54.8% (SD 19.7%) checklist items correct to 92.3% (SD, 5.0%) at posttest (P < 0.001). One learner did not meet the MPS at the initial posttest for SC CVC and MV skills. This learner participated in approximately 30 minutes of additional deliberate practice time for each skill and required one additional test to achieve the MPS for SC CVC and MV.

Figure 1.

Simulation-based mastery learning-trained intensive care unit advanced practice provider (APP) pretest and posttest scores (percentage correct), compared using paired t tests, on four critical care skills checklists: (A) CVC IJ, (B) CVC SC, (C) thoracentesis, and (D) mechanical ventilation management. Each circle represents an individual APP. CVC = central venous catheter; IJ = internal jugular; M = mean; MPS = minimum passing standard; SC = subclavian; SD = standard deviation.

Confidence in performing each of the skills increased from before to after the curriculum. IJ CVC insertion mean confidence increased from 35.5 (SD, 27.3) at pretest to 79.1 (SD, 13.8) at posttest (P < 0.001). SC CVC insertion mean confidence levels increased from 12.7 (SD, 14.9) at pretest to 61.8 (SD, 14.7) at posttest (P < 0.001). Thoracentesis mean confidence levels increased from 42.5 (SD, 37.7) at pretest to 75.0 (SD, 10.0) at posttest (P = 0.14). MV management mean confidence levels increased from 62.3 (SD, 22.3) at pretest to 86.0 (SD, 10.8) at posttest (P = 0.002). The participants rated the SBML curriculum highly (Table 3).

Table 3.

Course evaluation questionnaire completed by simulation-based mastery learning-trained advanced practice providers (n = 12) on the basis of a five-point Likert scale

Item	1, n (%)	2, n (%)	3, n (%)	4, n (%)	5, n (%)
Practice during the training sessions boosts my skills to manage patients in the ICU	0	0	0	0	12 (100)
It is OK to make clinical mistakes using the simulation models	0	0	1 (8)	3 (25)	8 (67)
I received useful educational feedback from the training sessions	0	0	0	0	12 (100)
Practice with the simulators boosts my clinical self-confidence	0	0	0	2 (17)	10 (83)
Repetitive practice using simulation is a valuable educational experience	0	0	0	1 (8)	11 (92)
My instructor was prepared to teach the session	0	0	0	0	12 (100)
APP simulation training should be a required component of APP practice at our institution	0	0	0	4 (33)	8 (67)
Please rate the overall quality of the curriculum*	0	0	1 (8)	3 (25)	8 (67)

Definition of abbreviations: APP = advanced practice provider; ICU = intensive care unit.

^*Five-point Likert scale (1 = very poor, 3 = neutral, 5 = very good).

Discussion

The number of APPs working in ICU settings has dramatically risen in the last few years, and it is likely that this number will continue to increase, given the high clinical demands enhanced by the COVID-19 pandemic. Studies have demonstrated the numerous positive impacts APPs have on patient care and cost-effectiveness in the ICU (1–4). ICU APPs at our institution are expected to perform invasive procedures on critically ill patients. Despite varying years of clinical experience and confidence levels, no APPs in this study were able to meet the MPS on all three of the ICU skills evaluated at baseline. This is concerning because procedural errors increase patient morbidity, mortality, and overall healthcare costs (34–36). As a result, it is imperative that all providers who perform ICU skills are able to reach a high clinical skills standard that is rigorously assessed to optimize patient outcomes and minimize risks. SBML is a training methodology that has proven to achieve this goal by training clinicians to a high standard, ultimately improving patient care and outcomes (14, 26, 27, 37, 38).

This study advances what is known about the current training models for APPs in practice and demonstrates an effective strategy for teaching and ensuring a high degree of competency in common critical care skills. We used a rigorous SBML training methodology, including multiple validated checklists and previously established MPSs, to evaluate important critical care skills. After participating in an SBML curriculum, all participating APPs demonstrated very high degrees of skill by meeting or exceeding the MPS and increased their self-confidence in all of the critical care skills assessed (IJ CVC and SC CVC insertion, thoracentesis, and MV management). CVC insertion and MV management skills significantly improved as a result of training, whereas thoracentesis improvement showed a clear effect size without meeting statistical significance because of the small participant sample size.

Multiple studies show that SBML is an important adjunct to traditional clinical training for physicians that improves skills across many domains, including advanced cardiac life support skills, code status discussions, and procedural skills such as temporary hemodialysis catheter insertion, thoracentesis, paracentesis, and lumbar puncture (27, 29, 39, 40). However, to our knowledge, our study is the first to apply the SBML methodology to boost the skills of practicing APPs and specifically focus on frequently performed critical care skills. It is important to apply SBML to APPs and physicians alike because health care is readily evolving toward an interprofessional team-based approach. As defined by the World Health Organization, interprofessional team-based care is “when a group of healthcare workers collectively share responsibility for the care of patients” (41). Although this study focused on APPs, it is also true that skills among other groups of practicing physicians have been found to be lacking. For example, a national study of critical care attending physicians in the Veterans Affairs Healthcare system found that very few possessed the skill to competently insert a simulated CVC (25). As such, it is imperative that all members of an ICU care team have equitable access to education and skills training that allows for consistent, high-quality patient care.

One specific strength of our study is that we identified an opportunity to extend educational interventions that were initially designed for physicians to all members of the critical care team. Another strength of our study is that all eligible APPs at our institution completed the SBML curriculum showing feasibility, and the participants favorably viewed the curriculum and felt strongly that simulation training should be a required component of APP practice. The results of this study have had a lasting impact on the training model used for the large number of ICU APPs hired at our institution during the COVID-19 pandemic. Specifically, all newly hired ICU APPs now participate in simulation training for CVC insertion and MV management during their onboarding process as part of hospital privileging.

Limitations

This study has several limitations. First, it was conducted at a single institution with a limited number of APPs. However, all eligible ICU APPs at our institution participated in the study. Second, we did not evaluate all critical care procedural skills and did not query the participants about any previous formal training they may have experienced before their involvement in this study. Future curricula could include additional skills such as arterial line insertion, chest tube insertion and management, airway management, and communication skills. Third, our study was underpowered to show improvements in thoracentesis skills. We chose to include thoracentesis training on the basis of APP interest and as proof of concept despite our small sample size. This should not take away from the fact that all five APPs met or exceeded the MPS after training. Fourth, in curriculum development, there is always the concern that learners are being “taught to the test.” However, during our SBML intervention, learners were not shown the checklists or any items on the checklists. Rather, they were taught the broad concepts and important clinical skills subsequently evaluated by the checklists. Ultimately, meeting or exceeding the MPS at the posttest suggests that learners were able to demonstrate the learning that occurred during this rigorous educational intervention. Lastly, we do not have data on skill retention after course completion or whether attained skills are linked to improvements in patient care outcomes. Prior studies have shown that learners who undergo SBML training demonstrate long-term retention of those skills (42) and link training to improvements in patient care (14, 43–45). Further study is needed to determine the optimal role of longitudinal assessment and booster training for maintaining clinical skill competency as well as evaluation of translational patient outcomes.

Conclusions

SBML is an effective educational strategy for training APPs to competently perform critical care skills. Educational interventions that were initially designed for physicians should be adapted and applied to all members of the critical care team because traditional educational methods and individual hospital orientation training processes may not reliably ensure that APPs are adequately trained to perform frequently used ICU skills. Future studies should address skill retention, assess patient care and clinical outcomes tied to this educational intervention, and extend this SBML procedural training to newly hired faculty and critical care fellowship graduates.