Mastery learning improves simulated central venous catheter insertion by emergency medicine teaching faculty

Abstract

Objectives

Routine competency assessments of procedure skills, such as central venous catheter (CVC) insertion, do not occur beyond residency training. Evidence suggests variable, suboptimal attending physician procedure skills. Our study aimed to assess CVC insertion skill by academic emergency physicians, determine whether a simulation‐based mastery learning (SBML) intervention improves performance and investigate for variables that predict competence.

Methods

This is a pretest–posttest study that evaluated simulated CVC insertion by emergency medicine (EM) faculty physicians. We assessed 44 volunteer participants at a large academic medical center over a 1‐month period using a published 29‐item checklist. Our primary outcome was the difference in assessment score before and after a SBML intervention. A secondary analysis evaluated predictors of pretest performance.

Results

A total of 44 subjects participated. Only four of 44 (9.1%) of subjects met a predefined minimum passing score on pretest. Mean assessment scores increased by 21.5% following the SBML intervention (95% confidence interval [CI] of the difference = 18.1% to 24.8%, p < 0.001). In a regression model, pretest scores increased by 10.8% (95% CI = 2.9 to 18.7%, p = 0.009) if subjects completed postgraduate training within 5 years. Frequency of CVC insertion did not predict performance, but 25 of 44 (56.8%) faculty members had no documented performance or supervision of a CVC insertion within 1 year of assessment.

Conclusions

SBML is a promising method to assess and improve CVC insertion performance by EM faculty physicians. Recent completion of postgraduate training was a significant predictor of CVC insertion performance. Our results require validation in larger cohorts of EM physicians across other academic institutions.

INTRODUCTION

Emergency physicians must be competent to perform a range of bedside procedures. Yet, there is an historic lack of standardized, objective assessment of procedure competence during residency training and none during the American Board of Emergency Medicine (ABEM) certification process. ¹ , ² The recent update to ABEM’s maintenance of certification requirements does not incorporate psychomotor skill assessment or require simulation‐based skills program participation, ³ the latter of which improves patient care and safety. ⁴ , ⁵ This is problematic, because factors such as variable skill acquisition from the “see one, do one, teach one” era of training and skill decay may result in substandard physician competence. ⁶ , ⁷ A small but important body of evidence suggests procedure skill is variable and substandard among attending physicians across specialties. ⁸ , ⁹ , ¹⁰ , ¹¹ For example, only 18% of attending physicians in a Department of Veterans Affairs system met a minimum passing standard (MPS) when assessed on internal jugular (IJ) central venous catheter (CVC) insertion technique. ¹¹ Sawyer's pedagogical framework for procedure training emphasizes the use of simulation as part of skill maintenance, particularly when physicians have infrequent opportunity to practice in the clinical environment. ¹² Simulation‐based mastery learning (SBML) is well studied to train novice learners CVC insertion technique, and multiple studies show a reduction in complications after implementation of an SBML curriculum. ¹³ , ¹⁴ , ¹⁵ However, no studies have evaluated procedure skill among academic emergency medicine (EM) physicians or the use of simulation to improve performance.

The primary goal of this study was to assess the difference between (1) baseline assessment of IJ CVC insertion by academic EM physicians in a simulated environment and (2) performance of IJ CVC insertion following a SBML skills refresher intervention. We hypothesized that IJ CVC insertion by EM faculty physicians prior to the intervention would be variable and suboptimal. A secondary goal was to evaluate whether the frequency of CVC insertion in a clinical environment or time since completion of residency training would predict baseline performance.

METHODS

Study design, setting, and participants

This was a pretest–posttest study evaluating IJ CVC insertion performance by clinically active EM faculty physicians at a large, academic medical center before and after an SBML intervention. Evaluations occurred between July 20 and August 10, 2020. Participants on professional or personal leave from clinical duties were excluded. Participation was voluntary and subjects provided informed consent. The study was approved by our institutional review board (IRB 57201).

Interventions

Participants were asked to insert an ultrasound‐guided IJ CVC in a task trainer (CentraLineMan, SimuLab) using a CVC kit at our institution (Arrow multilumen central venous catheterization kit CDC‐45703‐XP1A, Teleflex Inc.). We informed faculty members that participation was confidential and assessment would not impact their professional standing to ensure psychological safety. The intervention consisted of a baseline assessment by one of six expert assessors, deliberate practice with up to 1 h of 1:1 individual expert feedback, and repeat assessment on the same day. If the participant met the MPS on their initial assessment, they were not required to undergo further testing or training. Participants who did not meet the MPS on posttest were provided with additional feedback, the opportunity for deliberate practice, and repeat testing with the same expert assessor until the MPS was achieved or the session ended due to time constraints.

Methods of measurement

All assessors underwent training prior to the intervention that consisted of (1) an in‐person evaluation by an expert using the same assessment tool until the MPS was achieved, (2) calibration via video review of four mock participants, and (3) a 1‐h video conference to clarify questions.

Expert assessors used a published checklist ¹³ , ¹⁴ , ¹⁵ with MPS set at 98%. This MPS was taken from a related study that used a Mastery Angoff standard setting approach. ¹⁶ The MPS for procedures taught using SBML is generally high, as learners are expected to complete each step when performing a procedure.

We searched procedure codes in the electronic medical record to determine the number of CVC insertions (IJ, subclavian, and femoral) performed or supervised in clinical practice by each participant within the year prior. Years since completion of postgraduate training per participant were determined using an online institutional directory.

Outcome measures

The primary outcome measure was the difference in mean assessment scores before and after our SBML intervention. Secondary outcome measures included associations between pretest assessment score, CVC insertions performed or supervised in clinical practice, and time since completion of residency training. Using an alpha level of 0.05, a sample size of 34 subjects was required to detect an effect size of 0.5 with a power of 0.8 in a two‐tailed paired t‐test for our primary outcome. ¹⁷

Data analysis

We performed statistical analysis using SPSS Statistics for Macintosh, Version 27, (IBM Corp.). A single reviewer independently assessed 20% of the participants to determine a Cohen's kappa statistic for each of the other reviewers. Differences between mean pretest and posttest scores were analyzed using a two‐tailed paired t‐test. For our secondary outcome measures, we performed a linear regression analysis using two predictor variables (number of CVC insertions performed or supervised within the past year, completion of postgraduate training within 5 years).

RESULTS

Characteristics of study subjects

A total of 44 of 88 (50%) eligible faculty members participated in the study. An equal number of male and female subjects participated. Median time since completion of residency training was 9 years (interquartile range [IQR] = 4–16 years). The median number of CVC insertions per faculty member performed or supervised was 0 per year (IQR = 0–1 insertion per year). Twenty‐five faculty members (56.8%) had no CVC insertions within the preceding year.

Main results

Assessors had substantial interrater agreement (range, Κ, 0.69–0.79). Only four of 44 (9.1%) of participants met the MPS on the pretest. Items commonly missed are in Table S1 (available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10703/full).

Overall, mean participant scores increased by 21.5% following the SBML intervention (95% CI of the difference = 18.1–24.8%; p < 0.001). Of those who did not achieve the MPS on pretest, 33/40 (82.5%) did so after the intervention. The remaining seven participants did not achieve the MPS due to time constraints.

After linear regression, completion of residency training within 5 years significantly predicted pretest assessment score (pretest assessment score = 85.7% +10.8% (completion of residency training within 5 years) +1.9% (number of CVC insertions)); CVC insertions did not (Table 1). There was no association between either predictor variable and ability to achieve the MPS.

TABLE 1.

Results of linear regression analysis of variables predictive of pretest assessment score

Variable	B	β	95% CI	p‐value
Completion of postgraduate training within 5 years	10.8	0.414	2.9 to 18.7	0.009
Number of CVC insertions within the past year	1.9	0.162	–1.6 to 5.4	0.284

Model R ² = 0.207, p = 0.016.

Abbreviation: B, unstandardized coefficient.

DISCUSSION

Our study suggests that EM faculty physicians exhibit variable, suboptimal CVC insertion performance. Importantly, gaps in CVC insertion skills were measurable and correctable through a faculty procedure skills refresher intervention that used SBML pedagogy. A mastery learning approach allowed faculty members to reach a uniform, high level of procedure competency. Completion of training within 5 years, but not frequency of CVC insertion or supervision in the clinical environment, significantly predicted performance.

Several previous studies demonstrated the effectiveness of SBML to reduce CVC insertion complications. Most notably, Barsuk et al. ¹³ found a reduction in central line–associated bloodstream infections in critical care units in which residents were trained using SBML. Additional studies by the Barsuk team and others have since firmly established SBML as a true criterion standard for bedside procedure training. ¹⁸ , ¹⁹ , ²⁰ However, ours is the first study to use SBML for IJ CVC insertion training in a cohort of only EM faculty physicians. It is unclear if academic EM physicians, who primarily supervise procedures, suffer from skill decay more so than those who routinely perform such procedures. Participants who completed residency training within 5 years performed significantly better, which may indicate skill decay during independent practice. More senior faculty members, however, trained to perform CVC insertion without the use of ultrasound, which was included as part of our assessment. Additionally, procedure frequency, a commonly relied on metric to infer competence, did not significantly impact assessment score among our sample. This conflicts with studies related to other critical procedures in EM, such as endotracheal intubation, which imply adequate skill maintenance when meeting a threshold number of procedures performed or supervised. ²¹ A low frequency of CVC insertions across all faculty members may contribute to our finding, although it is unclear if other academic departments face similar challenges.

Our findings have policy and patient safety implications given the lack of required, periodic procedure competency assessments by ABEM. Emergency physicians clearly must be competent to prevent patient harm. Future study of larger cohorts of EM faculty physicians across institutions is required, including if decreased assessment scores in the simulated environment translate to increased adverse patient outcomes. Absent a national assessment mandate, local institutions may find SBML interventions to be effective safeguards of patient safety.

LIMITATIONS

There are several study limitations. A pre–post study design has inherent threats to internal validity ²² and our study was limited to T‐2 level outcomes in a simulated environment. However, the training and assessment methods we used mirror those of robust experimental SBML studies that demonstrated T‐3 patient safety effects. ¹³ , ¹⁴ , ¹⁵ This study also did not assess the impact of our intervention on skill retention over time when performing a same‐day posttest; that is an ongoing study at our institution. Our study was powered only for the primary outcome of change in procedure performance; a larger sample size is needed to confidently determine predictors of performance using our methodology. Volunteer subjects could have introduced sampling bias, such as if less confident participants felt the need for refresher training, resulting in lower baseline assessment scores. Some commonly missed checklist items may be attributed to simulation artifact, which could have affected our assessment metrics. For example, “call time out” may be overlooked in a simulated environment. Finally, time constraints prevented all participants from achieving mastery, which represents a deviation from standard SBML pedagogy.

CONCLUSIONS

Academic emergency physicians had variable and suboptimal internal jugular central venous catheter insertion performance when assessed in a simulated environment, and simulation‐based mastery learning improved performance. Frequency of central venous catheter insertion or supervision in the clinical environment was individually low and did not predict insertion performance. Faculty members who completed training within 5 years performed better than more senior clinicians.

CONFLICT OF INTEREST

The authors report no financial conflicts of interest.

AUTHORS CONTRIBUTIONS

Nicholas Pokrajac and Kimberly Schertzer conceived and designed the study, with Michael A. Gisondi providing subject expertise. All authors except Michael A. Gisondi contributed significant time to assess study participants and collect data. Nicholas Pokrajac recruited participants and oversaw data collection. Nicholas Pokrajac primarily performed data analysis. Nicholas Pokrajac drafted the manuscript, and all authors contributed substantially to its revision. Nicholas Pokrajac takes responsibility for the manuscript as a whole.

Supporting information

Table S1

Pokrajac N, Schertzer K, Roszczynialski KN, et al. Mastery learning improves simulated central venous catheter insertion by emergency medicine teaching faculty. AEM Educ Train. 2021;5:e10703. doi: 10.1002/aet2.10703