From the simulation center to the bedside: Validating the efficacy of a dynamic haptic robotic trainer in internal jugular central venous catheter placement

Abstract

Background

The objective of this study was to validate the transfer of ultrasound-guided Internal Jugular Central Venous Catheterization (US-IJCVC) placement skills from training on a Dynamic Haptic Robotic Trainer (DHRT), to placing US-IJCVCs in clinical environments. DHRT training greatly reduces preceptor time by providing automated feedback, standardizes learning experiences, and quantifies skill improvements.

Methods

Expert observers evaluated DHRT-trained (N=21) and manikin-trained (N=36) surgical residents on US-IJCVC placement in the operating suite using a US-IJCVC evaluation form. Performance and errors by DHRT-trained residents were compared to traditional manikin-trained residents.

Results

There were no significant training group differences between unsuccessful insertions (p = 0.404), assistance on procedure (p = 0.102), arterial puncture (p = 0.998), and average number of insertion attempts (p = 0.878). Regardless of training group, previous central line experience significantly predicted whether residents needed assistance on the procedure (p = 0.033).

Conclusion

The results failed to show a statistical difference between DHRT- and manikin-trained residents. This study validates the transfer of skills from training on the DHRT system to performing US-IJCVC in clinical environments.

Introduction

Over the last two decades, surgical education has seen a large shift from the Halstedian model of “See one, do one, teach one” to simulation training, in order to increase the safety and efficiency of the learning process ^1,2. Surgical simulation-based training (SBT) has been shown to effectively improve trainees’ skills and help reduce complication rates when transferring laparoscopic, colonoscopic, and other surgical skills to real patients ^3–6. For ultrasound-guided internal jugular central venous catheterization (US-IJCVC), ultrasound guidance has been shown to significantly reduce the failure rate of catheterization and incidence of mechanical, infectious, and thrombotic complications compared to the traditional landmark method ^7,8. Past research before ultrasound usage was widespread has shown that individuals who have placed less than 50 lines experience twice the number of complications, showing the importance of central line experience ⁹. In recent years, manikin simulators have been used to help learners achieve the skills necessary to place ultrasound-guided central venous catheters ¹⁰. While manikin trainers are anatomically realistic, they only represent a single patient anatomy, provide only procedural success feedback (through withdrawal of fluid), and require a trained preceptor (e.g. faculty) to provide any additional real-time feedback on performance, making them time and resource intensive ^11,12. In addition, the Association for Surgical Education (ASE) Simulation Committee has identified limitations in existing SBT education that need to be addressed before it can be effectively deployed throughout medical education including investigations into: standardization of evidence-based curriculum, optimal performance feedback metrics and assessment, cost effectiveness, and evidence of SBT to improving patient outcomes ¹³. In order to alleviate the need for preceptor feedback and quantify skill improvements, an alternative SBT method for training US-IJCVC needle insertion skills is through the Dynamic Haptic Robotic Trainer (DHRT), which was developed by researchers at the Pennsylvania State University ¹⁴.

The DHRT system provides a variety of patient scenarios by altering the virtual ultrasound images (e.g. size, depth, and location of vessels) and simulating force changes of different types of tissues (e.g. skin, adipose tissue, vessel) through the haptic robotic arm ¹⁵. After trainees complete each patient scenario, the DHRT system provides personalized learning feedback through a graphical user interface (GUI) that includes metrics such as (1) An overall performance score; (2) Case difficulty; (3) Number and success of insertion attempts; (4) Angle of needle insertion; (5) Distance of the needle tip to the center of the vessel; (6) Percent time spent aspirating throughout the procedure; any errors that occurred (e.g. arterial puncture), and a high score board for that specific patient profile, see Figure 1 ¹⁶. Research has shown that providing novices with detailed and appropriate feedback is a critical part of the learning process, and that the opportunity to incorporate that feedback is crucial to the development of expertise in surgery ¹⁷.

Figure 1:

(Left) Manikin simulator training with an expert observer providing feedback. (Middle) Dynamic Haptic Robotic Trainer (DHRT) with simulated ultrasound image. (Right) DHRT feedback screen providing objective feedback on each patient case.

Past research with first-year surgical residents (novices) and expert physicians has shown that novices approach expert performance on the DHRT system ¹⁵. In addition, prior work has also shown that the DHRT system significantly improves residents’ self-efficacy scores from pre- to post-test ¹⁸; Positive self-efficacy, or confidence in one’s own ability, has been linked with increased skill performance ^19–21. Additionally, individuals trained on the DHRT were better at identifying errors and evaluating their needle insertion performance than manikin-trained individuals ¹⁸. This is important because prior research has suggested that providing individuals with specific feedback about their performance, and specific feedback about how to evaluate their performance, can increase the accuracy of their self-evaluations ²², help them allocate more attention to the task, and improve learning and skill retention ^21,23. Residents who can accurately assess and improve their performance in these skills may be more adequate in determining where their needle tip is located and are therefore less likely to cause a pneumothorax or a hematoma caused by arterial puncture, both of which are common complications for IJCVC placement ^24,25.

While these prior studies present the design of the DHRT system and show its promise as an alternative training method to traditional manikin simulators, the impact of robotic training on performance in clinical environments has yet to be investigated. In a recent review of what’s needed next in surgical simulation, Stefanidis et al. stated, “the best validity evidence [to validate surgical simulators] emanates from reports demonstrating transfer of skill to the clinical environment and improvements in patient outcomes.” ¹³. As the first step to addressing this, DHRT-trained and manikin-trained residents were evaluated on their US-IJCVC performance on patients in clinical environments.

Methods and Materials

The goal of this study was to validate the transfer of skills from DHRT training by evaluating how well DHRT-trained residents performed US-IJCVC compared to traditional manikin-trained residents in the clinical environment. In order to answer this, participants were recruited and evaluated at Penn State Milton S. Hershey Medical Center (HMC).

Participants

Surgical residents at HMC were recruited from hospital-based procedural units including the intensive care unit (ICU) and operating rooms (ORs). Specifically, surgical residents were informed of the Institutional Review Board approved study during the weekly surgical education lectures. A total of 10 unique DHRT-trained and 24 unique manikin-trained surgical residents (20 female, 13 male, and 1 unknown) participated. The surgical residents specialized in general surgery (N=24), plastic surgery (N=3), vascular surgery (N=1) radiology (N=1), urology (N=1), neurosurgery (N=1), and unspecified (N=3). These surgical residents had collectively placed 21 (DHRT-trained) and 36 (manikin-trained) US-IJCVCs as PGY1 (N=11), PGY2 (N=9), PGY3 (N=4), PGY5 (N=10), PGY6 (N=1), with one resident participating both as a PGY1 and a PGY2 during the duration of the study.

Prior Training

As the study presented here is evaluating surgical residents at HMC, it is important to understand how surgical interns are trained on US-IJCVC. During intern bootcamp, all first-year surgical residents at HMC are first shown a 15.5-minute video on US-IJCVC placement followed by a live training demonstration of the procedure by a senior surgical resident using a Blue Phantom Gen II Ultrasound Central Line Training Model (Model #BPH660). There had been two distinct training conditions at HMC since 2016 and the 26 residents (2016), 26 residents (2017) are divided evenly among these training conditions: DHRT and manikin trained, see Figure 1. Residents from the 2018 surgical intern cohort (N=27) completed all training on the DHRT system. Therefore, the pool of residents available to participate in this study was 79, but participation was voluntary. During the residency bootcamp each summer, the manikin-trained residents were asked to complete 22 needle insertions on a manikin system over a span of 3 skills training days and were provided with feedback from a preceptor. On the other hand, DHRT-trained residents were asked to watch a 7-minute video introduction to the DHRT system and complete 22 patient scenarios on the DHRT system over the same training days. DHRT-trained residents received performance feedback from the DHRT learning interface, while manikin-trained residents received feedback from a trained preceptor. Twenty-two needle insertion attempts were selected as the end point because prior work on the DHRT system has shown that residents achieve at or near expert proficiency on the DHRT system within this time period ¹⁵. At the completion of training, all residents completed a 23-item verification of proficiency (VOP) in US-IJCVC placement on a manikin. This final assessment is the institutional standard for evaluating resident proficiency in CVC placement, and residents are expected to complete each item on the procedure independently in order to verify their proficiency on the entire procedure. Individuals who fail to pass the VOP by failing to achieve a step on the checklist required additional practice and re-evaluation using the VOP.

Procedure

At the start of the study, the purposes and procedures were explained, and informed consent was obtained. Next, the resident completed a demographic survey that included PGY level, area of specialty, training that they’ve received, and how many IJCVCs they’ve placed, see Appendix I. Residents placing the US-IJCVC were then evaluated by an expert observer on a US-IJCVC evaluation form, which included all 23 procedural skills from the VOP, and also noted areas for improvement such as inadvertent arterial punctures and the need for assistance during the procedure, see Appendix II. Both the resident placing the central line and the expert observer evaluating the resident were compensated with a $5 Starbucks gift card for each observed central line. All participants were tracked using a 6-digit participant code in order to ensure that participant identities were anonymous. The expert observers rating the surgical residents were themselves proficient in the IJCVC skill being performed or previously familiar with the checklists due to their own training experience. However, they were provided with guidance on how to complete the procedure by the senior resident on the research team prior to its deployment. The checklist was an adaptation from the accepted institution training checklist. Completed surveys were deposited into a locked metal box. The collected forms were analyzed for training group comparison in this study, with metrics detailed below.

Metrics

The US-IJCVC evaluation form used for this study includes the 23 procedural skills tested in the VOP, such as maintaining sterile technique or locating the needle position on the ultrasound image. Additionally, the US-IJCVC evaluation form tracks the number of insertion attempts, inadvertent arterial punctures, unsuccessful insertions, and the need for assistance on the procedure. In addition, observers provided comments on specific skills and the overall procedure using the US-IJCVC evaluation form. Deductive content analysis²⁶ was completed on the comments provided by the expert observer and paired with the error checklist (inadvertent arterial puncture, unsuccessful insertion, and assistance on procedure), to analyze specific mechanical and procedural errors that occurred during the placement. The specific metrics were computed as follows:

Pass Rate was calculated as the percent of individuals within each training group who independently completed each of the binary 23 skills on the US-IJCVC evaluation form, see Appendix II. For example, if 27 out of 36 manikin-trained participants successfully obtained a clear ultrasound image, this would translate to a 75% (27/36) pass rate.

% Entire procedure completed was the percentage of the 23 skills on the US-IJCVC evaluation form that the resident completed independently. Although all participating surgical residents passed VOP at the end of intern bootcamp, trainees may be subject to central line skill decay ²⁷, or loss of trained skills or knowledge ²⁸, during extended periods of nonuse due to rotations. The procedural and technical elements of the checklist were initially examined to compare differences in performance between the two training groups. There were no significant differences in procedure vs. technical elements. Thus, the % entire procedure completed was presented.

Number of insertion attempts was noted on the US-IJCVC evaluation form by the observer. The number of insertions is important because the number of unsuccessful needle insertion attempts is one of the greatest predictors of mechanical placement complications in subclavian CVC placement, where the rate of catheter placement failure increases from 1.6% (1 attempt) to 10.2% (2 attempts) to 43.2% (3 or more attempts) ²⁹.

Inadvertent arterial puncture was calculated as the total number of arterial punctures noted by the observers in each condition. Inadvertent arterial puncture was tracked because it can result in serious complications such as hematoma or arterial cannulation ^30,31.

Unsuccessful Insertion was the total number of central lines that were not successfully catheterized by residents in each training group. These were marked when an observer took over and completed the procedure, or when there were patient complications and the procedure was aborted. Failure to place the catheter is one of the most common mechanical complications in central venous catheterization ³². This metric was important in evaluating how frequently residents could successfully catheterize the patient independently.

Assistance on Procedure was computed as the total number of residents in each group who needed assistance on the procedure, whether this was through verbal prompting throughout the procedure, or having the procedure taken over and completed by the observer. This metric was important in evaluating residents’ proficiency in completing the procedure because research has highlighted the importance of simulator training for automaticity ^33,34, or an expert ability to perform tasks with minimal attentional resources ^35–37. However, little research has shown how much central line experience is required in order for trainees to develop automaticity. As the first step to this, assistance on the procedure provides additional insight beyond pass rates and performance metrics in evaluating resident proficiency in US-IJCVC.

Central line experience was a continuous variable based on the # of previous central lines that residents self-reported having completed in clinical environments. This variable was important to consider because previous research has shown that individuals who have placed less than 50 lines experience twice the number of complications, showing the importance of central line experience ⁹. This metric was used as a covariate (ANOVA) and an independent variable (binomial logistic regression) during analyses.

Data Analysis

All data were analyzed using SPSS (v. 25.0) with significance considered at p-value of 0.05. For continuous variables (% procedure completed independently, number of insertion attempts), univariate analysis of variance (ANOVA) was computed to determine if these measures differed between training groups, with central line experience taken as a covariate. In addition, a Wilcoxon signed rank test was conducted to determine whether the median number of insertion attempts was significantly different from the optimal number of 1 for each training group. For each binary 23 skills on the US-IJCVC evaluation form (e.g. locating the needle’s position on the ultrasound image) and error checklist (e.g. unsuccessful insertion, need for assistance on the procedure, arterial puncture), binomial logistic regressions were conducted with training group (independent variable), central line experience (covariate) and each binary item (dependent variable) to determine whether training group could predict the odds of success or failure for each binary variable.

Assumption checking for continuous dependent variables revealed that homogeneity of variances was met for % entire procedure completed (p = 0.980) and number of insertion attempts (p = 0.314). The data were not normally distributed, as assessed by boxplot and the Shapiro-Wilk’s test (p < 0.0005). Because the group data were similarly skewed for each continuous metric, and ANOVA is considered robust to non-normality and Type I errors (rejecting the null hypothesis to find a “false positive” results), ANOVA was conducted. An outlier analysis was conducted on continuous dependent variables before analysis. During this assumption check, two outliers for % entire procedure completed were identified because of having standardized values (Z-scores) greater than 4 standard deviations lower than the mean. The outliers were found to have no significant impact on the significance of the results, and therefore, the full analysis (with outliers) is presented here.

Binomial logistic regressions were used to analyze the impact of training group and central line experience on each binary dependent variable. Linearity of the continuous independent variable (central line experience) with respect to the logit of each dependent variable was assessed via the Box-Tidwell (1962) procedure ³⁸. A Bonferroni correction was applied using all four terms in the model, resulting in statistical significance being accepted when p < 0.0125 ³⁹. Based on these assessments, central line experience was found to be linearly related to the logit of each dependent variable, meeting the assumptions required for binomial logistic regressions.

Results and Discussion

The current study was developed to validate the transfer of US-IJCVC placement skills from DHRT-training into clinical environments. In order to answer this, data was analyzed to determine whether training group (DHRT or manikin) impacts US-IJCVC placement performance, and whether central line experience affects performance. Qualitative and quantitative analyses were conducted on the US-IJCVC evaluation forms. Data are presented as mean ± standard error unless otherwise noted. A summary of resident performance on the % entire procedure completed, successful catheterization, and major errors can be found in Table 1.

Table 1:

Descriptive statistics on pass rates for manikin-trained (N=36) and DHRT-trained (N=21) for US-IJCVC skills, insertion attempts, and error checklist items. Continuous variables are presented as mean ± standard error. Binary variables are shown with count of occurrences.

Pass rates and complications	Manikin (N=36)	DHRT (N=21)
% Entire procedure completed independently	92.6 ± 2.6%	93.6 ± 4.0%
Average # insertion attempts	1.56 ± 0.153 attempts	1.52 ± 0.148 attempts
Unsuccessful catheterization	6/36	2/21
Needed assistance on procedure	13/36	4/21
Inadvertent arterial puncture	3/36	0

A multivariate analysis of variance (MANOVA) with independent variables (training group, central line experience, PGY level, area of specialty) and dependent variable (% entire procedure) showed no interaction or main effect of surgical specialty or PGY level. Thus, a follow-up one-way univariate analysis of variance was conducted.

The one-way univariate analysis of variance (ANOVA) results showed that the training group (DHRT or manikin) did not have a significant impact on the percent of the total US-IJCVC procedure completed by the resident, with prior central line experience taken as a covariate, F(1,54) = 0.062, p = 0.805, partial η²= 0.001, observed power of 0.057. On average, 92.6 ± 2.6% percent of the procedure was completed by manikin-trained participants, and 93.6 ± 4.0% was completed by DHRT-trained residents. Additionally, there were no significant training group differences in the average number of insertion attempts required, F(1,52) = 0.125, p = 0.725, partial η²= 0.125, observed power of 0.064. However, a Wilcoxon signed rank test showed that the median number of insertion attempts required was significantly more than the optimal 1 needle insertion attempt for both DHRT-trained residents, z = −2.810, p = 0.005, and manikin-trained residents, z = −3.134, p = 0.002. Specifically, DHRT-trained residents required 1 attempt (N=12), 2 attempts (N=7), and 3 attempts (N=2), while manikin-trained residents required a total of 1 insertion attempt (N=22), 2 attempts (N=7). 3 attempts (N=3), and 4 attempts (N=2), with 2 participants missing data, see Figure 2. This is important because prior research has shown that numerous needle insertion attempts may increase the risk of infections ⁴⁰ and downstream mechanical complications, such as failure to successfully place and secure the catheter ²⁹.

Figure 2:

Wilcoxon signed rank tests showed that the median difference in the number of insertion attempts was significantly more than 1 needle insertion attempt for both DHRT-trained residents, z = −2.810, p = 0.005, and manikin-trained residents, z = −3.134, p = 0.002.

Next, binomial logistic regressions were calculated to determine the impact of training group and central line experience on the pass rates for each of the 23 skills on the US-IJCVC evaluation form, as well as the unsuccessful insertions, inadvertent arterial puncture, or assistance during the procedure. While DHRT-trained residents successfully completed 90.5% catheterizations compared to 83.7% for manikin-trained residents, the results failed to show a significant effect of training group or central line experience for any of the 23 skills (p = 0.197 to p = 0.998). In addition, while there were no inadvertent arterial punctures by DHRT-trained residents, there were 3 cases of inadvertent arterial punctures by manikin-trained residents; the binomial logistic regression models showed no statistically significant effect of training group and central line experience on unsuccessful insertions (χ²(2) = 1.622, p = 0.444) or inadvertent arterial puncture (χ²(2) = 5.104, p= 0.078). In order to understand what led to the inadvertent arterial puncture, the observer-provided comments were reviewed. Observer 1 noted that the “IJ wall was flaccid. Very hard to cannulate” while Observer 2 stated that the “Patient [was] hypotensive in OR leading to under-filled IJ. Even with ultrasound, carotid was stuck & procedure temporarily aborted.” Finally, Observer 3 stated that the resident “Went through-and-through and hit artery, held pressure, successfully accessed vein after.” It is worth noting that these challenging scenarios can all be simulated by the DHRT system, with its ability to provide haptic feedback, and to vary the size, depth, and position of the vessels.

Finally, the results of the binomial logistic regression model for the impact of training group and central line experience on the need for assistance on the procedure was statistically significant, χ² = 8.634, p = 0.013. Specifically, the model explained 20.0% (Nagelkerke R²) of the variance in assistance on the procedure and correctly classified 71.9% of the cases. However, of the two predictors, only central line experience was statistically significant (p = 0.033), training group was not (p = 0.102). On average, 36.1% manikin-trained residents required some form of assistance (verbal prompting or procedural takeover) from observers, while 19.0% DHRT-trained residents required assistance, see Figure 1. These results indicate that residents with less central line experience had 1.114 higher odds of needing any type of assistance on the procedure (p = 0.033). This shows that as residents complete more central lines, they gain proficiency and can perform more independently in the procedure.

The results of this study fail to show a statistical difference on US-IJCVC between DHRT-trained and manikin-trained residents. While no statistical difference exists, DHRT training has several benefits over manikin-training. First, the system simulates multiple patient profiles, compared to the static manikin model, and reduces preceptor time by providing automated feedback after each patient case. DHRT training standardizes the training process by quantifying skill improvements and providing consistent and objective areas of feedback. Without the need of a preceptor, future trainees have the flexibility of practicing the procedure at any time and tracking their own skill improvements. For this study however, the DHRT system was only available for initial bootcamp training. No residents received additional access or training on the DHRT in order to maintain consistency in structured training provided to the two groups. Finally, the DHRT system has the ability to provide future cost savings as an all-in-one training system that provides training for multiple image-guided procedures and replacing the multiple training manikins needed for various procedures. Additional work is being completed to improve the capabilities and value of the DHRT system. Future work will be conducted to examine the cost analyses on the DHRT.

Limitations

While these results are promising, there were several limitations to the study. First, this study focused on evaluating the initial US-IJCVC placement performance; patients were not tracked or monitored after this initial placement to evaluate whether they experienced any complications related to the procedure. This is important in tracking the downstream effects of US-IJCVC placement. Additionally, completing research surveys was not part of the clinical workflow, thus recruited participants may have been biased towards those more diligent in participating in research studies. Finally, since DHRT-training focused on residents’ ability to place central lines, the US-IJCVC data collection focused on central line placement and occurrence of complications associated with the initial central line placement. Future work will examine the impact of patient clinical specifics (e.g. stability, anesthesia) on central line placement.

Conclusions

The current study showed that DHRT-trained residents performed US-IJCVC placement as well as manikin-trained residents in clinical environments. Central line experience significantly predicted the need for assistance on procedures. DHRT training reduces preceptor time by providing automated feedback, standardizes the learning experience, and quantifies skill improvements in US-IJCVC placement. This study validates the transfer of skills from training on the DHRT simulator into placing US-IJCVCs placement in clinical environments.