Simulation improves procedural protocol adherence during central venous catheter placement: a randomized-controlled trial

Abstract

Background

Simulation training may improve proficiency at and reduces complications from central venous catheter (CVC) placement, but the scope of simulation’s effect remains unclear. This randomized controlled trial evaluated the effects of a pragmatic CVC simulation program on procedural protocol adherence, technical skill, and patient outcomes.

Methods

Internal medicine interns were randomized to standard training for CVC insertion or standard training plus simulation-based mastery training. Standard training involved a lecture, a video-based online module, and instruction by the supervising physician during actual CVC insertions. Intervention-group subjects additionally underwent supervised training on a venous access simulator until they demonstrated procedural competence. Raters evaluated interns’ performance during internal jugular CVC placement on actual patients in the medical intensive care unit. Generalized estimating equations were used to account for outcome clustering within trainees.

Results

We observed 52 interns place 87 CVCs. Simulation-trained interns exhibited better adherence to prescribed procedural technique than interns who received only standard training (p=0.024). There were no significant differences detected in first-attempt or overall cannulation success rates, mean needle passes, global assessment scores or complication rates.

Conclusions

Simulation training added to standard training improved protocol adherence during CVC insertion by novice practitioners. This study may have been too small to detect meaningful differences in venous cannulation proficiency and other clinical outcomes, highlighting the difficulty of patient-centered simulation research in settings where poor outcomes are rare. For high-performing systems, where protocol deviations may provide an important proxy for rare procedural complications, simulation may improve CVC insertion quality and safety.

Introduction

Central venous catheterization is a common inpatient procedure with risks — including arterial puncture, pneumothorax, catheter-associated bloodstream infection, and death — that diminish with operator experience.^1,2 The regular use of ultrasound guidance, careful attention to sterile technique, and thoughtful site selection all reduce these risks.^1–3 Simulation-based training can also improve resident competency in placing central venous catheters (CVCs) and reduce rates of CVC complications including central line-associated blood stream infections (CLABSI) and arterial puncture.^4–8 The American Board of Internal Medicine now recommends that trainees’ instruction in procedural skills begins with simulation.^5,9–12

Simulation enables deliberate practice in a risk-free setting, ostensibly building “effective” experience with a procedure or clinical situation.^4,5,13–15 Prior studies suggested CVC simulation training cements protocol adherence^6,9,11,16,17 and helps novice practitioners achieve the spatial and kinesthetic skills — and the resulting procedural proficiency^5,9,11,12 and complication rates^4,5,15 — of their more experienced colleagues.² To date, however, most studies of technology-enhanced simulation employed non-randomized designs (including 18 of 20 CVC studies in one recent systematic review), were subject to evolving circumstances for clinical care influencing CVC outcomes, evaluated learner performance on simulators, or focused on learner behaviors rather than patient outcomes.^8,14,18,19 Moreover, in previous studies, it has been difficult to distinguish simulation’s impact on “effective experience” from the benefits attributable to increased uptake of other measures that improve CVC placement outcomes. For instance, simulation training has been associated with increased ultrasound use in some studies,⁶ including one study correlating simulation training with a 20-fold increase in the proportion of CVCs inserted with ultrasound guidance.¹⁵ Hands-on training also improves observation of strict sterile technique.^17,20 These data suggest that enhanced protocol adherence may be an important mechanism for simulation’s reported beneficial effect on patient-centered outcomes. Uptake of process-based interventions such as ultrasound guidance, however, may also be advanced by means other than simulation training, including checklists and promotion of institutional standards of care.²¹ When such interventions are thoroughly incorporated into institutional practices, the marginal benefit of simulation for patient-centered outcomes remains unclear. Since protocol adherence is an important quality indicator linked to patient outcomes in the ICU,^21–23 however, deviations from critical procedural protocols — complication “near misses” — may still be useful surrogates to gauge simulation’s impact in already high-performing environments.

We hypothesized that a pragmatic simulation training program would yield improvements in novice practitioners’ patient-centered outcomes and procedural protocol adherence during CVC placement above and beyond that achieved by systems interventions targeting ultrasound use and sterile technique. To test our hypothesis, we conducted a randomized-controlled trial of the effect of simulation-based mastery training²⁴ for internal jugular (IJ) CVC insertion on protocol adherence, technical skill, and patient-centered outcomes in our institution’s medical intensive care unit (MICU), where ultrasound guidance and maximal sterile barrier protection for CVC placement were already standard of care.

Methods

Internal medicine residents beginning their first year of post-graduate training at a large quaternary hospital were recruited during orientation activities in June 2010. Since nearly all CVC placements by medical interns occur in the MICU at our institution, study intervention and data collection took place at the time of each intern’s 2- or 4-week rotation in this unit. Training and data collection were restricted to the internal jugular (IJ) site because the vast majority of CVCs in our MICU are placed in this anatomic location and subclavian CVCs were not placed under ultrasound guidance during the study period. Under preexisting MICU protocol, all IJ CVCs are placed under ultrasound guidance and nurses actively monitor and enforce maximal sterile barrier protection during the procedures performed on their patients.

Using stratified block randomization with stratification by ICU rotation period, participating interns were assigned based on a computer-generated randomization sequence to standard training alone (no intervention) or standard training plus simulation-based training. Standard CVC training involved a 45–60 minute didactic lecture, an interactive online module structured around an 18 minute video,²⁵ familiarization with our hospital’s CVC placement checklist, and instruction by an upper-level resident, fellow or attending during all actual procedures. In addition to standard training, intervention group subjects received 1–2 hours of individualized instruction and supervised practice on a venous access simulator (Blue Phantom; Redmond, WA) with an experienced pulmonary and critical care (PFC) or emergency medicine (TS) attending physician. Instructors reviewed CVC placement indications, complications and relevant anatomy as well as sterile technique and ultrasound use. The instructor then guided subjects through a stepwise simulated IJ CVC placement using the hospital standard CVC protocol. Instructors provided active feedback, with progression to the next procedural step dependent on correct performance of the preceding step.²⁶ Subjects thereafter continued supervised deliberate practice with instructor guidance until the subject was sufficiently prepared to undergo a test of mastery. The subject then performed an unassisted simulated IJ CVC insertion and was evaluated by the attending physician on his or her ability to complete the procedure independently. Based on prior studies and expert opinion, we defined successful independent CVC placement by (1) vein cannulation on the first or second needle pass combined with (2) 100% adherence to seven key indicators of procedural technique (Table 1, see below).²⁷ In accordance with principles of mastery learning,²⁶ intervention subjects who did not meet these criteria during the unassisted simulated CVC placement repeated supervised simulator training and evaluation until they achieved procedural competence based on these criteria. Simulation training occurred during the two weeks prior to their MICU rotation or, when scheduling conflicts arose, as soon as possible following the start of this rotation. Subjects randomized to standard training alone had the option to participate in simulation training after completing their MICU rotation.

Table 1.

Seven key indicators of procedural technique during central venous catheterization

Performs “time out”

Sterile technique properly observed

Ultrasound used to identify jugular vein by compression

Aspirates continuously while advancing needle

Guidewire always kept in hand and under control

Confirms venous cannulation by ultrasound or manometry

Sharps always returned to needle holder or safe position

Data collection

Data were collected from July 2010 until June 2011. During actual IJ CVC insertion by interns in the MICU, the resident, fellow or attending physician who supervised the procedure and the patient’s nurse recorded the following learner behaviors and patient outcomes on a standard instrument:

1. Number of skin punctures required.

2. The intern’s adherence to seven key indicators of procedural technique adapted by the investigators from a previously validated instrument²⁸ and our hospital’s CVC placement checklist (Table 1). A technical proficiency score was calculated as the proportion of eligible procedural indicators performed correctly (range 0–1). Raters were instructed to record “incorrect” performance for a procedural indicator if the supervisor needed to intervene to prevent an error. When there was disagreement between the physician and nurse raters on whether a checklist item was performed correctly, the intern’s procedural technique was judged to have been inadequate for that item.

3. A global performance score (range 1–5) adapted by the investigators from a previously validated instrument^29,30 and referenced to operators’ supervision needs, technical lapses, and functional training level (evaluation matrix, Table S1, Supplemental Digital Content 1). A score of 1 indicated the trainee required step-by-step guidance, performing at the medical student/new intern level. Trainees demonstrating high-quality unassisted CVC insertion without technical lapses (performance expected of an experienced senior resident) scored a 5.

4. Immediate complications (hematoma, arterial puncture).

The patient’s mechanical ventilation status and presence of perceived anatomic obstacles to CVC placement were also recorded. The hospital’s preexisting surveillance system was queried to ascertain CLABSI, defined according to the National Health Safety Network criteria.³¹ After hospital discharge, a blinded investigator (IDP) reviewed the radiologist’s interpretation of post-procedure chest X-rays for pneumothorax and line malposition and abstracted other major complications and patient demographic data by chart review.

MICU fellows, attendings, upper level residents and nurses served as raters during CVC placements they supervised. We provided potential raters information about the study and detailed instructions for completing the CVC insertion evaluation by e-mail followed by in-person training in the use of the data collection instrument. Data collection training was subsequently reviewed by e-mail and/or in person several times per month. Both the supervising physician and the nurse were blinded to the CVC operator’s training assignment. The study protocol was approved by the Massachusetts General Hospital Institutional Review Board.

Statistical analysis

The primary outcome was the proportion of first-attempt venous cannulation success. Analysis was according to randomization group, regardless of training received. We calculated that observation of 49 CVC insertions by each group would provide 80% power to detect a two-fold increase (30% control vs 60% intervention group) in first-pass venous cannulation.^9,11,30 Secondary outcomes included CVC placement success, the number of skin punctures required for venous cannulation, the technical proficiency score, the mean global performance score, and whether procedures had at least one complication. Exploratory analyses of adherence to individual procedural indicators and specific procedural complications were performed without correction for multiple comparisons.

Since our study measured outcomes from individual CVC placements clustered within trainees, we accounted for outcome correlation using generalized estimating equations (GEE) with linear regression (continuous outcomes) or relative risk regression (binary outcomes) with robust standard errors and an exchangeable working correlation matrix.^18,32,33 Results are reported as the relative risks (RR) or mean differences with 95% confidence intervals (CI)^34,35 When binomial-based models could not produce effect estimates because the outcome of the dichotomous dependent variable was uniform among one randomization group (complete separation),³⁶ we performed a sensitivity analysis using a GEE-based normal distribution model and report the risk difference and 95% CI.^37,38 The kappa statistic was calculated for interrater agreement and is reported with its 95% confidence interval. Data were analyzed using Stata version 13.1 (College Station, TX).

Results

Study population

All eligible subjects (N=73) provided informed consent to participate in the study. Fifty-one interns placed at least one observed CVC in an actual patient (subject flow diagram, Figure S1, Supplemental Digital Content 2). There were no significant differences in baseline demographic characteristics between the interns assigned to the standard training only (no intervention) group and the simulation-plus-standard training group or between patients undergoing CVC insertion by the two study groups (Table 2). One intern assigned to simulation training did not receive this training due to scheduling conflicts. During the planned one year study duration, 87 CVC placements on actual patients — 49 by simulation-trained interns, 38 by the standard training only group — were observed by 50 supervising physicians and 45 nurses. The study was not continued beyond the planned one year duration because CVC simulation training was included in orientation activities for the intern class starting July 2011. Interrater agreement was very good for first-pass cannulation success (kappa statistic 0.88 [0.74–1.00]).

Table 2.

Demographic characteristics of participating interns and patients

Interns	Standard training only	Simulation plus standard training
N	37	36
Mean age (SD)	28.6 (0.4)	28.4 0.5)
Female (%)	16 (43.2)	19 (52.8)
Training track (%)
Categorical	20 (54.1)	25 (69.5)
Primary care	5 (13.5)	3 (8.3)
Combined*	3 (8.1)	5 (13.9)
Preliminary	9 (24.3)	3 (8.3)
Degree (%)
MD	24 (64.9)	26 (72.2)
Combined MD/PhD	8 (21.6)	7 (19.5)
Combined MD/other	5 (13.5)	3 (8.3)
Patients
N	38	49
Mean age (SD)	56 (2.9)	59 (2.2)
Female (%)	19 (51.4)	17 (35.4)
Anatomic obstacle (%)	5 (13.2)	4 (8.2)
Intubated (%)	25 (67.6)	35 (71.4)

^*Residency programs combining medicine with pediatrics or with psychiatry

CVC placement proficiency

There was no difference between the standard training alone group and the standard training plus simulation group in the probability of first-attempt or overall cannulation success or the mean needle passes required (Table 3). Subjective global ratings based on the provided five-point scale, with higher numbers indicating greater proficiency, were also similar between the two groups.

Table 3.

Patient-centered outcomes during central venous catheterization after standard training alone versus simulation training added to standard training

Procedural success measures	Standard training only* (N=38)	Simulation plus standard training* (N=49)	Relative risk or mean differcence (95% CI)	p value
First-attempt cannulation	23 (60.5)	29 (59.2)	1.00 (0.75 – 1.34)	0.99
Overall cannulation success	34 (89.5)	45 (91.8)	1.02 (0.88 – 1.18)	0.84
Mean needles passes (SD)	1.64 (1.11)	1.57 (1.02)	−0.13 (−0.54 – 0.28)	0.54
Mean global assessment score (SD)	2.9 (1.1)	3.1 (1.1)	0.20 (−0.29 – 0.69)	0.43
Complications
Arterial puncture	2 (5.4)	2 (4.1)	0.73 (0.11 – 4.79)	0.74
Hematoma	2 (5.3)	5 (10.2)	1.75 (0.41 – 7.44)	0.45
Catheter malposition	2 (6.1)	5 (11.4)	1.80 (0.39 – 8.29)	0.45
Catheter-associated infection	1 (3.2)	1 (2.6)	0.83 (0.05 – 12.8)	0.90
Pneumothorax	0 (0)	0 (0)	—	—
Death	0 (0)	0 (0)	—	—
Total CVC attempts with ≥1 complication†	5 (13.9)	11 (22.9)	1.70 (0.60 – 4.78)	0.32

^*Data are reported as N (%) except as indicated.

^†Total attempts with complication is less than the total number of complication because some individual attempts had more than one complication

Abbreviations: SD, standard deviation; CI, confidence interval; IQR, interquartile range; CVC, central venous catheter

Complications

There were no procedure-associated pneumothoraces or attributable deaths. We observed no significant differences in individual (catheter-associated bloodstream infection, catheter malposition, arterial puncture, or hematoma) or overall complication rates between study groups (Table 3).

Procedural protocol

Simulation training improved performance on the composite index of procedural technique. After adjusting for clustering, the average proportion of key procedural indicators (Table 1) performed correctly by simulation plus standard training group was 93% (90%–97%), 11% (1–20%) higher than the standard training alone group (82% [74%–91%], p=0.024). Among simulation-trained interns, 71% of CVC placements exhibited perfect adherence to procedural indicators compared to 55% among interns who received standard training alone (RR 1.20, 95% CI 0.91–1.59, p=0.19).

We also conducted an exploratory analysis of adherence rates for each of the individual key indicators of prescribed procedural protocol (Table 4). The simulation-trained interns were 32% (3%-69%) more likely to confirm CVC placement in the vein by ultrasound or manometry (p=0.026) and 17% (3%–69%) more likely to maintain control of their sharps (p=0.026). Simulation training’s influence on adherence to sterile technique approached but did not achieve statistical significance (RR 1.13 [1.00–1.28], p=0.055). All simulation-trained subjects visualized the vein with ultrasound prior to all observed CVC placements, preventing application of binomial regression models for statistical analysis. A sensitivity analysis employing a GEE-based normal distribution model estimated that subjects who received simulation training performed this step during 13% (1%–25%) more CVC placements than their colleagues who did not receive simulation training (p=0.032).

Table 4.

Procedural protocol adherence (without prompting from supervisor) after standard training alone versus simulation training added to standard training

Procedural step	Standard training only* (N=38)	Simulation plus standard training* (N=49)	Relative risk (95% CI)	p value
Time out	29 (80.6)	43 (87.8)	1.09 (0.92 – 1.29)	0.33
Sterile technique	32 (86.5)	47 (97.9)	1.13 (1.00 – 1.28)	0.055
Vein visualized with ultrasound†	33 (86.8)	49 (100)	—	—
Aspirates continually	33 (89.2)	46 (93.9)	1.05 (0.91 – 1.23)	0.49
Guidewire controlled	30 (81.1)	42 (87.5)	1.07 (0.87 – 1.32)	0.53
Venipuncture confirmed	24 (70.6)	42 (93.3)	1.32 (1.03 – 1.69)	0.026
Sharps controlled	29 (80.6)	45 (91.8)	1.17 (1.02 – 1.35)	0.026

^*Data are reported as N (%)

^†Due to complete separation of outcomes by intervention group for “vein visualized with ultrasound,” relative risk regression cannot be applied to this outcome.

Discussion

In this single-blind, randomized-controlled trial, adding simulation-based mastery training to standard pre- and intra-procedural instruction for IJ CVC placement improved interns’ adherence to critical procedural protocols during actual patient CVC placement compared to conventional training alone. However, the study was ulimately underpowered to detect the projected differences in the primary outcome, venous cannulation proficiency, limiting the conclusions that may be drawn from simulation’s failure to significantly improve cannulation success or reduce complications of CVC insertion by novice practitioners.

Whereas past studies correlated simulation training with increased uptake of ultrasound guidance, checklist use, and full barrier sterile precautions, the present study took place in an ICU where these quality-improvement interventions were fully in place.³⁹ “Standard” procedural training was also more intense relative to some past studies.^4,6,9 Nevertheless, for every 9 CVCs placed, simulation-trained interns committed one fewer deviation from protocols important to prevent arterial dilation, needle stick injury, and other serious procedural complications. Since protocol adherence is likely a proxy measure for quality and safety in high-performance environments,^19,21,23 simulation therefore remains an important tool to speed adoption and dissemination of new innovations in procedural technique while promoting adherence to protocols demonstrated to improve patient-centered outcomes and likely to reduce the risk of infrequent but critical complications.

Simulation’s impact may vary according to baseline outcomes. Our intervention occurred in an already high-performing ICU characterized by lower complication rates and better cannulation proficiency among controls than in past studies.^5,6,9,11 For instance, ICUs at our institution reported 1.0 CLABSI per 1,000 “line days” during our study compared to a national standard of 2.0 and to prior CVC simulation studies reporting baseline CLABSI rates of 3–8 infections per 1,000 line days.^4,11,40 Improved baseline procedural outcomes entail smaller absolute benefits for simulation, increasing the cost-benefit ratio for simulation training and limiting our ability to detect simulation’s impact on patient-centered CVC outcomes without very large sample sizes.^19,39

Our study has several strengths.^8,14 We minimized confounding using a randomized, parallel-group design and blinding of evaluators. Second, few other studies have evaluated outcomes during CVC placement on patients rather than simulators.^8,18 Third, all subjects were first-year residents engaged in their first ICU rotation, minimizing variability in subjects’ prior experience and maximizing any benefit of effective experience. Fourth, to determine if simulation works in a real-world setting (effectiveness)— as opposed to whether it can work in an ideal setting (efficacy) — we employed a practical, feasible intervention, compared the intervention to usual training practice, and focused on practical outcomes.^41,42

Inadequate power is our study’s most important limitation. Our study had less power than expected because we observed fewer than our goal of 96 CVC placements and overestimated the effect size — a doubling of the first-pass cannulation rates — used for our power analysis. The fact that 30% of enrolled subjects had no observed CVC placements and failure to account for clustering in power calculations further increased the risk of type II error. Based on data now available from other studies,^6,11 a 40–50% increase in first-pass cannulation would have been more realistic. Our pragmatic trial design also impaired our ability to detect an effect of the intervention¹⁹ since the simulation intervention was less intense than in some^12,27 but not all^{15,16,20,30,43} prior studies and guidance by a supervising physician during each procedure may have improved performance and prevented some complications. Overall, insufficient power prevents definitive conclusions regarding simulation’s influence on procedural proficiency and complications. However, the absence of any trend toward improvement with simulation suggests we did not miss the kind of large intervention effect we hypothesized due to insufficient sample size.

Our study has a number of additional limitations. Evaluation of study subjects’ performance by raters of convenience without formal training against model CVC placements raises the possibility of inter-rater variability. Physicians evaluating the same CVC placements they supervised may also have been less likely to report complications or procedural errors. To address these problems, we provided frequent training on our evaluation instrument, asked raters to score steps requiring supervisor intervention as errors, and obtained simultaneous evaluations from an independent, non-physician rater. Although adapted from previously validated instruments, we did not independently validate our procedural assessment checklist or global performance score. Interns in both study groups worked in the ICU at the same time, so benefit spillover from the simulation to the no intervention group could have reduced the intervention’s measured effect. Such spillover effects would be expected to be limited to knowledge and protocol adherence as opposed to effective experience. We were unable to measure all CVC placements in the MICU during the study period and cannot report data collection rates or rule out bias due to systematic differences in unevaluated CVC placements. Similarly, outcomes may have differed for the 30% of interns who did not perform observed CVC placements (although this proportion is smaller than in previous randomized trials⁶). Our data, collected between 2010 and 2011, may not reflect present realities. Simply adding time to standard training — rather than simulation specifically — may have improved protocol adherence. Future studies should compare simulation training methods,^44,45 though our data suggest such studies would need much larger sample sizes to meaningfully assess patient outcomes. Finally, whether our results can be generalized beyond internal medicine interns or the MICU is unclear.

Conclusions

Simulation-trained interns exhibited better adherence to prescribed protocols in our randomized controlled trial, which was ultimately underpowered to demonstrate significant improvement in the primary outcome, venous cannulation proficiency. Overall, our findings suggest that implementation of protocol-directed quality assurance measures, intensification of “conventional” training, and falling complication rates may reduce the marginal effect of simulation training on patient outcomes. This raises important research questions regarding the study of increasingly safer systems of care, settings where trials may require substantial resources to demonstrate incremental but meaningful improvements in patient-level outcomes. Further investigation is needed to confirm whether procedural protocol adherence is a meaningful surrogate for tracking quality and safety in high-performing clinical units.