Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Sep 16.
Published in final edited form as: J Surg Educ. 2018 Mar 21;75(5):1410–1421. doi: 10.1016/j.jsurg.2018.02.018

Investigating the Effect of Simulator Functional Fidelity and Personalized Feedback on Central Venous Catheterization Training

Mary A Yovanoff *, Hong-En Chen *, David F Pepley †, Katelin A Mirkin ‡, David C Han ‡, Jason Z Moore †, Scarlett R Miller *,§
PMCID: PMC6139271  NIHMSID: NIHMS976064  PMID: 29574019

Abstract

OBJECTIVE

To compare the effect of simulator functional fidelity (manikin vs a Dynamic Haptic Robotic Trainer [DHRT]) and personalized feedback on surgical resident self-efficacy and self-ratings of performance during ultrasound-guided internal jugular central venous catheterization (IJ CVC) training. In addition, we seek to explore how self-ratings of performance compare to objective performance scores generated by the DHRT system.

DESIGN

Participants were randomly assigned to either manikin or DHRT IJ CVC training over a 6-month period. Self-efficacy surveys were distributed before and following training. Training consisted of a pretest, 22 practice IJ CVC needle insertion attempts, 2 full-line practice attempts, and a posttest. Participants provided self-ratings of performance for each needle insertion and were presented with feedback from either an upper level resident (manikin) or a personalized learning system (DHRT).

SETTING

A study was conducted from July 2016 to February 2017 through a surgical skills training program at Hershey Medical Center in Hershey, Pennsylvania.

PARTICIPANTS

Twenty-six first-year surgical residents were recruited for the study. Individuals were informed that IJ CVC training procedures would be consistent regardless of participation in the study and that participation was optional. All recruited residents opted to participate in the study.

RESULTS

Residents in both groups significantly improved their self-efficacy scores from pretest to posttest (p < 0.01). Residents in the manikin group consistently provided higher self-ratings of performance (p < 0.001). Residents in the DHRT group recorded more feedback on errors (228 instances) than the manikin group (144 instances). Self-ratings of performance on the DHRT system were able to significantly predict the objective score of the DHRT system (R2 = 0.223, p < 0.001).

CONCLUSION

Simulation training with the DHRT system and the personalized learning feedback can improve resident self-efficacy with IJ CVC procedures and provide sufficient feedback to allow residents to accurately assess their own performance.

Keywords: central venous catheterization, virtual reality, simulator fidelity, medical training simulation, self-efficacy, residency training

COMPETENCIES: Practice-Based Learning and Improvement

INTRODUCTION

Central venous catheterization (CVC) is a common procedure used to administer medication, nutrition, and obtain measurements to monitor patients.1 Although more than 5 million central lines are placed annually in the US each year,2 up to 39% of patients incur adverse effects3–5 such as hematoma, pneumothorax, or arterial puncture during catheter insertion.3 Importantly, surgeons who have inserted CVCs less than 50 times are twice as likely to incur mechanical complications.6 Although recent calls in medical research have identified a need for improving and standardizing CVC education in order to reduce these complication rates,7 advances in medical simulation research have focused primarily on examining the effect of increases in the structural rather than the functional fidelity of simulators. This is problematic because previous research has shown that increases in structural fidelity, or the realism of the environment, does not correspond with increases in educational effectiveness.8,9 However, increases in functional fidelity, or the match between the system and how a user performs a specific task, has been shown to be a vital component of effective learning and skill transfer.10 In ultrasound-guided CVC, this functional fidelity includes using the ultrasound and haptic feel to accurately identify the needle location relative to anatomical structures and make corrections to the needle and ultrasound position in real-time to accurately place the needle into the vein.

Current CVC simulation training typically includes a functionally static manikin with an arterial pulse (controlled through a hand-pump) and self-sealing veins.11 This type of training is limited by the fact that it does not contain realistic force profiles for different tissues (e.g., skin, adipose tissue, and vessel). This is problematic because the high complication rates in CVC procedures have been attributed, in part, to variations in patient anatomy such as body habitus and coagulopathy.1,12 In addition, this type of manikin training provides only basic feedback on performance, such as blue liquid being aspirated when the introducer needle hits the target vessel. Because of this, manikin training has been criticized as being resource intensive13,14 because it requires a trained preceptor (e.g., faculty) to be present to provide meaningful, real-time feedback on performance. Thus, in order to efficiently reduce the mechanical complication rates associated with CVC procedures, higher functional-fidelity simulators are needed to objectively evaluate and prepare resident surgeons.

The Dynamic Haptic Robotic Trainer (DHRT)15 virtual reality system was developed to respond to this gap by teaching CVC needle insertion skills through variations in patient anatomy and by providing objective feedback on performance. This system includes seventeen unique patient cases that vary in the anatomical configurations of the patient. These variations in patient anatomy are simulated in the system through changes in a visual ultrasound image (e.g., size, location, and pulsatility of the vessel) and through haptic feedback provided through the robotic arm of the DHRT system that simulates the force changes of different types of tissues (e.g., skin, adipose tissue, and vessel), see Pepley et al.15 for details. Validity evidence based on test content was identified by comparing learning and confidence gains in medical students between the DHRT and manikin-based learning systems.17 Some validity evidence was identified with respect to motion tracked variables, as experts performed better than novices in time to complete and standard deviations of deviations of their needle path.18 Finally, prior work by the authors has shown that participants improved their performance throughout the course of training on the DHRT system and they approached the level of an expert at the end of their training.19 An advantage of using functionally high-fidelity simulators like the DHRT for IJ CVC trainings is that they can be designed to automatically present feedback in both textual and graphical forms which can improve information retention.20 For example, the DHRT system provides users with feedback after each insertion trial on the needle angle, final position of the needle tip, number of insertions, and amount of unnecessary movements, see Yovanoff et al.21 for discussion. This type of variation in scenario and objective feedback can increase both skill transferability, skill retention, and self-efficacy.22–25 Self-efficacy is a particularly important construct to explore in surgical residency education because there are links between positive self-efficacy and increased skill performance.26–28 However, no research to date has explored the effect of simulator functional fidelity or personalized feedback on surgical resident confidence and objective performance and thus it is unclear how the factors affect resident skill gains.

MATERIALS AND METHODS

Based on this prior work, the current study was developed to answer the following research questions (RQ):

  • RQ1: (A) Do surgical residents improve their CVC self-efficacy over the course of training and (B) is this improvement in confidence dependent on the training environment? Specifically, this research question sought to understand if resident CVC self-efficacy changed from pre-CVC to post-CVC training and if this difference was owing to variations in the CVC training method (manikin or robotic). It was hypothesized that self-efficacy would increase throughout training because prior work has shown that simulator training can increase resident comfort with CVC procedures.11 However, it was also hypothesized that residents in the robotic group would have larger improvements on CVC self-efficacy because prior research has shown that providing feedback increases self-efficacy22,23 and help users cognitively engage during the learning process.29

  • RQ2: (A) How do perceived ratings of performance change throughout training and (B) is this change in self-rated performance dependent on the functional fidelity of the training environment? Specifically, this question sought to address how participant self-ratings of performance compared across training groups over the duration of training. The manikin-trained individuals received observational feedback from a higher-level resident, whereas robot-trained individuals received specific feedback from the robotic system. It was hypothesized that self-ratings of performance would increase in both groups throughout training because prior studies have indicated that self-perception of motor skills improve with training.22,23 In addition, research has shown that competency-based simulation training in US-guided CVC insertion was more effective than relying solely on the traditional apprenticeship model in improving in-hospital performance and resident skills.30 However, manikin systems only provide very basic feedback on performance (blue liquid is aspirated if the catheter hits a vein), and there is no objective performance criterion. Instead, these systems require a trained preceptor (e.g., faculty) to be present to provide real-time feedback on performance, which also introduces subjectivity to the evaluation process and makes standardization difficult. The DHRT system aims to improve simulation training by providing multiple patient scenarios and objective, standardized feedback. Because of these differences, it was hypothesized that the consistent and objective feedback from the robotic simulator would help residents more accurately assess specific skills and performance throughout the sessions, when compared to the manikin-trained individuals.

  • RQ3: How do perceived ratings of performance on the robotic trainer relate to objectively calculated performance scores? Specifically, this question sought to address how participant self-ratings of performance compared with the objective score generated by the DHRT system. Calculated performance scores are generated based on several variables, including average angle of insertion, distance to center of vein, puncturing the back wall penalties, and more, see Pepley et al. for more details.19 Although previous research has indicated that even experts may have a low ability or no ability to judge their own skill performance,31–33 providing individuals with feedback during training may improve their ability to accurately assess their own performance.34 Thus, it was hypothesized that there would be a weak positive correlation between self-ratings of trial performance and scores generated by the DHRT system.

Participants

In order to address these questions, participants were recruited from the first-year residency program at Hershey Medical Center (HMC). Participants were invited to participate in the study during their “residency boot camp” which occurs during the first week of their surgical residency. In total, there were 26 participants (6 females, 20 males) who were in the following specialties: general surgery (6), preliminary residents (8), orthopedics (5), otolaryngologists (2), urology (2) plastic surgery (2), and vascular surgery (1). It should be noted that 3 participants (all males) were unable to finish the study owing to constraints on their residency training. Of the 3 participants, 1 was in the robotic training group and 2 were in the manikin training group. Additionally, 1 participant from the manikin training group only completed 12 out of the 22 required needle sticks. Their data were included in the analysis when available.

Procedures

The study was conducted in 6 sessions over a period of 6 months (Fig. 2). Specifically, session 1 was conducted during the first day of boot camp where a brief oral summary of the proposed research was provided and consent was attained. Next, each participant individually completed a 14-item, 5-point Likert scale central-line self-efficacy survey (the complete survey can be viewed at http://www.engr.psu.edu/britelab/projects_cvc.html) regarding their confidence in their abilities to perform CVC skills (with 1 being “not at all confident” and 5 being “very confident”). As part of their general training, participants viewed an 18 minute video36 on central line placement. Beyond watching the video, participants were given no instructions on how to perform the procedure. After viewing the video, each participant individually completed a pretest where they inserted a needle for central line placement into a Blue-Phantom Gen II Ultrasound Central Line Training Model (Model #BPH660) while verbalizing what they were doing according to the think-aloud procedure.35 During the pretest, a second-year surgical resident observed each participant and evaluated them using a modified internal jugular catheterization (IJCVC) evaluation form (For full checklist visit http://www.engr.psu.edu/britelab/projects_cvc.html). The standard 23-item IJ CVC evaluation form used at HMC evaluates individuals on the entire CVC procedure, from draping the manikin to inserting the guidewire, securing the catheter, and additional steps. The modified IJ CVC evaluation form is a 10-item checklist focusing exclusively on the needle insertion portion of the procedure and includes items like “continuously aspirating the entire time” and “selecting the appropriate site for venipuncture.” The observer did not provide participants with feedback throughout their pretest, but once the pretest was complete, they did inform the participant if they had successfully placed the needle and what errors occurred during the procedure. There was a 5-minute time limit for the pretest and if the participant did not successfully stick a vessel during the 5-minute time period, the pretest was terminated. After the pretest, each participant was randomly assigned to 1 of 2 training conditions: manikin or robotic.

FIGURE 2.

FIGURE 2

Open in a new tab

Training Session Outline.

Manikin training (N = 13)

Participants completed training on a Blue-Phantom Gen II Ultrasound Central Line Training Model (Model #BPH660) or a similar training manikin.

Robotic training (N = 13)

Participants completed training on the DHRT.

Session 2 was conducted 5 weeks after session 1. During this session, a second-year surgical resident gave a demonstration of central-line placement on a manikin simulator using the same procedures used to train individuals in the surgical residency program at HMC. After viewing the demonstration, participants in the robotic training condition watched an instructional video on how to use the DHRT system. Both groups then individually completed 2 practice insertion attempts on their assigned training devices. Importantly, individuals in the robotic condition only received feedback from the DHRT personalized learning system (Fig. 1), whereas individuals in the manikin condition received their feedback from a second-year surgical resident who observed their performance. It is important to note that after each insertion attempt on the DHRT system and before personalized feedback was provided, robotic participants were asked to rate their performance on the system on a scale of 1 (“very poor”) to 5 (“very good”). After receiving their feedback on each insertion attempt, participants in both groups recorded their performance on a practice insertion performance evaluation form (PIPEF) (Fig. 3) included in a self-evaluation training books in a self-evaluation training book (for full book please visit http://www.engr.psu.edu/britelab/projects_cvc.html).

FIGURE 1.

FIGURE 1

Open in a new tab

(left) DHRT system features and (right) personalized feedback screen that is presented after each insertion trial on the DHRT.

FIGURE 3.

FIGURE 3

Open in a new tab

Practice insertion performance evaluation form included in self-evaluation training books.

Sessions 3 and 4 each included 10 practice needle insertion attempts on the participant assigned training system. After each trial, participants were asked to fill out the practice insertion performance evaluation form. These sessions were approximately 4 weeks apart. Session 5 was the final practice session where all participants, regardless of condition, completed 2 full practice sessions on the entire CVC insertion procedure, which included items such as sterilization and patient consent, on a manikin. Their performance was evaluated by a second-year surgical resident using the IJ CVC checklist. Finally, session 6 was conducted over a period of 2 months based on resident availability. During this session, participants individually performed a posttest needle insertion on a manikin using the same procedures outlined in the pretest including the completion of the same CVC self-efficacy survey given in session 1. Aside from this focused training in central line placement, participants did not have additional CVC insertion practice between sessions. However, it is possible that subjects of either cohort may have witnessed others inserting central lines in the surgical intensive care unit.

RESULTS AND DISCUSSION

In order to answer our RQs, statistical analyses were performed on the central line self-efficacy survey (CLSE), the practice insertion performance evaluation forms (PIPEF), and objective scores generated by the DHRT. All analyses were conducted using SPSS (v. 24.0) with an error rate of 0.05 unless otherwise specified.

RQ1a: Do Surgical Residents Improve Their CVC Self-Efficacy Over the Course of Training?

Statistical analyses were performed on the pretraining and posttraining CLSE (central line self efficacy) surveys. In order to analyze the change between responses on the pre-CLSE and post-CLSE survey, an ordinal logistic regression was conducted with the test order (pretest or posttest) as the independent variables and each survey question as a dependent variable, see Table 1 for median survey response data. These results exclude the questions owing to singularity in the algorithm, or when the software could calculate these values (Table 1). For the manikin training group, the results showed that all survey questions were significantly improved from the pretest to the posttest. For the DHRT training group, all survey questions were significantly improved from the pretest to posttest with the exception of “Locating the needle on the ultrasound image” which had a median value of 4 for both the pretests and posttests. These results are encouraging because both training groups showed improvements in resident self-efficacy, which has been shown to relate to skill performance.26,27

TABLE 1.

Median Values of Responses of the 5-Point Likert Scale Central-Line Self-Efficacy Survey (With 1 Being “Not At All Confident” and 5 Being “Very Confident”)

Manikin Robotic
Median Pretest Median Posttest p Value Median Pretest Median Posttest p Value
Obtaining a clear image of the target vessel using the ultrasound machine 3 4 **0.004 4 5 **0.007
Locating the needle on the ultrasound image 3 4 *0.011 4 4 0.192
Identifying the correct site of insertion on the skin for the introducer needle 3 4 **0.001 3 4 -
Using tactile feedback (sense of feel) during placement of the line 3 4 **0.002 2 4 **0.002
Using the ultrasound image/transducer, identifying the correct vessel for puncture 3 4 *0.021 4 5 0.092
Using tactile feedback, identifying the correct vessel for puncture 3 4 **0.002 2 4 **0.003
Advancing the introducer needle slowly and steadily 3 4 **0.001 3 4 **0.006
Modifying the needle trajectory based on ultrasound feedback 3 4 **0.004 3 4 *0.01
Identifying when the introducer needle is in the correct location 2 4 **0.001 3 4 *0.016
Using tactile feedback to help guide the introducer needle 2.5 4 **0.002 2 4 *0.002
Placing the introducer needle at the center of the vein in one attempt 2 4 - 2 4 -
Placing the introducer needle at the center of the vein in multiple attempts 2.5 4 **0.002 3 4 *0.016
Conducting the entire procedure without any mistakes 2 4 **<0.001 2 4 **0.001
Conducting the entire procedure on a CVC simulator (manikin) 3 4 **0.004 3 4 **0.009
Open in a new tab
*

p < 0.05.

**

p < 0.01, - indicates that a p value could not be calculated owing to a singularity.

RQ1b: Is This Improvement in confidence Dependent on the Training Environment?

Although the analysis of the CLSE survey revealed that training on either the manikin or the robotic system improved self-efficacy for CVC insertion skills, there were differences in confidence gains across training environments for different skills. In order to determine specific differences between the 2 training groups for the preself-efficacy and postself-efficacy, the CLSE surveys were analyzed using a generalized estimating equation, which allows analysis of repeated measures (pretest and posttest) across multiple observations (multiple questions). The between-subject variable was participant ID, the within-subject variable was test order (pretraining or posttraining), the factors were the type of training (DHRT or manikin) and each question on the CLSE survey, with results reported in Table 2.

TABLE 2.

Values of General Estimating Equation for the Central-Line Self-Efficacy Survey

Factor β Std Error 95% CI p Value Odds Ratio
Obtaining a clear image of the target vessel using the ultrasound machine −0.316 0.9428 (−2.164, 1.531) 0.74 0.73
Locating the needle on the ultrasound image −0.672 1.1453 (−2.917, 1.573) 0.56 0.51
Identifying the correct site of insertion on the skin for the introducer needle 1.878 1.5043 (−4.827, 1.070) 0.21 0.15
Using tactile feedback (sense of feel) during placement of the line 2.604 2.3863 (−2.073, 7.281) 0.28 13.51
Using the ultrasound image/transducer, identifying the correct vessel for puncture 0.302 1.3162 (−2.278, 2.882) 0.82 1.35
Using tactile feedback, identifying the correct vessel for puncture 2.514 2.3972 (−2.184, 7.213) 0.29 12.35
Advancing the introducer needle slowly and steadily −1.322 1.228 (−3.729, 1.084) 0.28 0.26
Modifying the needle trajectory based on ultrasound feedback 0.078 1.339 (−2.546, 2.703) 0.95 1.08
Identifying when the introducer needle is in the correct location −0.039 1.321 (−2.628, 2.550) 0.98 0.96
Using tactile feedback to help guide the introducer needle 3.762 1.7723 (0.288, 7.235) **,0.03 43.03
Placing the introducer needle at the center of the vein in 1 attempt −1.99 1.90949 (−4.136, 0.156) *,0.07 0.13
Placing the introducer needle at the center of the vein in multiple attempts −0.649 1.1986 (−2.998, 1.700) 0.59 0.52
Conducting the entire procedure without any mistakes 0.14 1.244 (−2.579, 2.299) 0.91 0.86
Conducting the entire procedure on a CVC simulator (manikin) 0.403 0.9951 (−1.547, .164) 0.69 1.49
Open in a new tab
*

p < 0.10.

**

p < 0.05.

The results showed that for “Using tactile feedback to help guide the introducer needle” there was a statistically significant interaction effect between pretest/posttest and training method (p = 0.034) with a beta value of 3.762 and an odds ratio of 43.03. This indicates that individuals in the robotic group were 43 times more likely than the manikin group to increase their self-efficacy rating for this measure. In addition, although not significant (p = 0.069), the interaction effect between pretest/posttest and training method for “Placing the introducer needle in the center of the vein in one attempt” (beta value of −1.990 and an odds ratio of 0.13) also points to the need for further exploration. Specifically, manikin-trained individuals who did not get objective feedback on how close they were to the center of the vein, rated their performance more highly. These results indicate that there may be an effect of training environments and confidence changes in this skill.

Although the analysis of the CLSE survey revealed that training on either the manikin or the robotic system improved self-efficacy for CVC insertion skills, participants trained on the robotic system showed larger improvements in their confidence in their ability to use tactile feedback to guide the introducer needle. On the other hand, participants trained on the manikin had larger gains in confidence in their ability to place the needle in the center of the vessel. This finding is interesting, because the DHRT system provides specific feedback on the closeness of the needle to the center of the vein after each trial, information that the manikin training system would have difficulty providing feedback on. As such, manikin-trained residents are unlikely to receive feedback on this skill during training and may have a false sense of confidence in this ability. However, future work is needed to explore why this is occurring and how accurate these self-efficacy gains align with actual performance on the manikin trainer. Regardless, the results presented here indicate that the DHRT system is at least as effective as the manikin for increasing resident self-confidence with CVC insertion skills, confirming prior research that found that self-efficacy can be improved through the practice of skills.11

RQ2a: How Do Perceived Ratings of Performance Change Throughout Training?

In order to understand how self-ratings changed within each training group, a Wilcoxon Signed Ranks test was run to compare self-ratings on the first and last needle insertions for both groups with “self-rating” as the dependent variable and “insertion number” as the independent variable. Assumptions were checked and the distributions were approximately symmetrically shaped. The results revealed a statistically significant increase in self-rating for the manikin training group from pretest (Mdn = 3) to posttest (Mdn = 5) z = 2.762, p = 0.006. There was also a statistically significant increase in self-rating for the robotic training group from pretest (Mdn = 3) to posttest (Mdn = 4) z = 2.919, p = 0.004. As hypothesized, this shows that for both groups, participants felt they had improved their CVC insertion performance. More importantly, we wanted to determine if there were specific differences in self-ratings of performance on various skills between the 2 training groups.

RQ2b: Is This Change in Self-Rated Performance Dependent on the Functional Fidelity of the Training Environment?

In order to determine differences between the 2 training groups for increases in their self-ratings of performance, a generalized estimating equation was conducted with all 22 trials. Specifically, the between-subject variable was participant ID, the within-subject variable was the trial number, the factor was type of training (robotic or manikin), and dependent variable was self-rating of performance on scale of 1 (very poor) to 5 (very good). The ordinal logistic regression showed a statistically significant main effect for training method (p < 0.000, χ2 = 20.517) and trial number (p < 0.000, χ2= 1020.276). These results indicated that performance changed over time and that in general the manikin group rated their performance higher than the robotic group did throughout the trials (Fig. 4).

FIGURE 4.

FIGURE 4

Open in a new tab

Mean self-rating of performance on each trial. Trials 1, 2, 3, 12, 13, and 22 use the same baseline patient scenario. * Indicates the mean self-rating of performance for that trial was significantly different than the mean of all trials combined.

In addition, there was a statistically significant interaction effect for the entire model between trial and training group (p < 0.001, χ2 = 196.273). In other words, for each individual trial, there was no significant difference between how the manikin and robotic group rated their performance on each individual trial (e.g., trial 1). However, when all 22 trials were considered together, the way that the ratings changed over time was different for each training group. Specifically, the results revealed that ratings for trials 1, 2, 3, 4, 13, and 16 were statistically significantly different from the mean rating of the rest of the trials. Trials 1 and 2, 3, and 4, and 13 were all the first trials of training sessions 2, 3, and 4, respectively. Prior research indicates that skill retention declines over time for CVC procedures,36 so it is not surprising that the mean ratings of performance would be significantly lower for the first insertion after a period of not practicing.37 Figure 4 shows that the mean rating for performance on trial 16 on the manikin was similar to the surrounding trials, but that self-ratings for the robotic group were visibly lower. This trial presented a patient with very thin skin and shallow vessels that were nearly vertical. Of the 13 robotic participants, 1 participant punctured the artery and 3 participants punctured though the back wall of the vein during this trial meaning that this trial may have been more difficult than some of the other cases presented to the robotic group, see Pepley et al.38 for detailed findings. This suggests that the specific patient case for trial 16 had an effect on performance and self-rating. However, future work is needed to further investigate the effect of the patient case on perceived performance.

In order to understand if the amount and type of feedback impacted self-ratings of performance, content analysis was performed on the responses to “List any errors you made” recorded in the self-evaluation training books after each insertion trial. The responses were transcribed and analyzed by 2 independent raters using combined inductive and deductive content analysis39 in NVivo v11.4.0. Specifically, each feedback item from the learning feedback screen (see Yovanoff et al.21 for a detailed description of learning feedback screen), with the exception of overall score, was made into a node in NVivo. This resulted in the following 6 nodes: angle, aspiration, distance to center of the vein, number of insertions, and arterial puncture. Next, 2 researchers verbally discussed each node until both felt satisfied that they had a mutual understanding of each item. Each rater then, independently, rated 7 of the participant books and added any nodes they felt were frequently mentioned. Together the raters then reevaluated the existing nodes to form a consensus for which nodes were a good representation of the feedback being provided. The following 3 nodes were added: “external landmark,” “left versus right hand,” and “ultrasound monitor.” Once the coding scheme was set and an interrater reliability (weighted κ) of 0.72 was achieved, the remaining 19 self-evaluation participant books were coded, see Table 3 for representative coding schema.

TABLE 3.

Errors from the Self-Evaluation Training Book by a Manikin Participant and a Robotic Participant and the Code of the Item from the Content Analysis

Insertion Number Manikin Group Participant Robotic Group Participant
Response NVIVO Code Response NVIVO Code
1 Probe orientation, aspire as you insert needle – Probe
– Aspiration		 2 insertions – Number of insertions
2 NA – Failed to access, far from center – Distance to center of vein
3 Went past vein – Distance to center of the vein 2 insertions – Number of insertions
4 NA – 2 insertions, pulled out of vein at end – Number of insertions
– Distance to center of vein
5 NA – Pulled out after removing probe and had to re-access – Distance to center of vein
– Number of insertions
6 NA – Still pulling out when removing probe, 2 insertions – Distance to center of vein
– Number of insertions
7 NA – Pulled out when removing probe – Distance to center of vein
8 NA – Pulled out when probe removed – Distance to center of vein
9 NA – Didn’t pull out when removing probe but required 2 attempts to access – Number of insertions
10 NA – 2 attempts to access – Number of insertions
11 NA – 2 attempts to access – Number of insertions
12 NA – Attempts because not close to center and losing access – Number of insertions
– Distance to center of vein
13 Didn’t aspirate initially, went through vein, but was able to troubleshoot – Aspiration
– Distance to the center of the vein		 Angle too steep – Angle
14 NA – Pulled needle out – Distance to center of vein
15 NA – Shallow angle – Angle
16 NA – Multiple attempts – Number of insertions
17 NA – 2 attempts – Number of insertions
18 NA – Hit carotid, multiple passes – Arterial Puncture
– Number of insertions
19 NA – 2 attempts – Number of insertions
20 NA – 3 attempts? fairly certain hit vein and stayed in – Number of insertions
21 NA – 2 attempts – Number of insertions
22 NA – 2 attempts, close to center – Number of insertions
– Distance to center of vein
Open in a new tab

In order to determine whether an equal number of participants from each of the training conditions reported each type of errors during their CVC training, chi-squared goodness of fit tests were conducted. Of the 372 errors recorded in the self-evaluation participant books, 144 were reported from the manikin group and 228 from the robotic group. The results indicated that the number of errors were not equally represented by participants in the study. Specifically, the robotic group reported a significantly higher number of total errors (χ2 (1) = 18.97, p < 0.001), errors in the angle of insertion (χ2 (1) = 14.52, p < 0.001), errors in the needle tip’s final distance to the center of the vessel (χ2 (1) = 19.15, p = 0.002), and errors in having multiple insertion attempts (χ2 (1) = 23.06, p = 0.005). Although not significant (χ2 (1) = 3.45, p < 0.063), the robot-trained individuals reported more errors in aspirating during the insertion attempt. On the other hand, participants in the manikin group reported a significantly higher number of errors in pressure or torque during the insertion attempts (χ2 (1) = 8.00, p = 0.005). The remaining differences were not significant. These detailed error reports relate to prior research which has indicated that, when provided with specific feedback on cues that are indicative of performance, self-evaluation skills will improve in accuracy.34 This suggests that the fidelity of the robotic simulator (e.g., haptic feel and patient scenarios) or objective feedback provided may better prepare participants at evaluating their performance when compared to the manikin group.

These results showed that participants in both the manikin and robotic training groups increased their self-ratings or performance over the course of their training. On average, individuals in the manikin group rated their performance higher than the participants in the robotic group and recorded approximately half the amount of errors as those in the robotic group, despite no differences in actual posttest performance. This suggests that the participants in the robotic group may have been better at evaluating their performance than participants in the manikin group. This is important because prior research has suggested that providing individuals with specific feedback about their performance, and specific feedback about how to evaluate their performance, can increase the accuracy of their self-evaluations.5 Evaluating performance and noticing errors may result in greater attention allocation to the task and therefore, lead to increased learning and skill retention.28,41 Residents who can accurately assess and improve their performance in these skills may be more adequate in determining where their needle tip is located and therefore, less likely to cause a pneumothorax (puncturing a lung), or a hematoma (excessive bleeding) caused by arterial puncture, both of which are common complications for IJ central line placement.42,43

RQ3: How Do Perceived Ratings of Performance on the Robotic Trainer Relate to Objectively Calculated Performance Scores in the DHRT System?

In order to answer this research question, a linear regression analysis was conducted in order to determine if the self-ratings of performance a participant received before receiving from the DHRT system were related to the objective scores generated by the DHRT system. The dependent variable in the model was the objective score generated by the DHRT system and the independent variable was the participant’s self-rating of performance recorded in the DHRT system. The assumption of a monotonic relationship was confirmed through a visual evaluation of a scatter plot. The results showed that self-ratings of performance on the DHRT system could significantly predict the objective score of the DHRT system, R2 = 0.223, F (1,285) = 81.294, p < 0.001; adjusted R2 = 0.220, B = 0.109. These results indicate that participants who used the DHRT system were able to predict how well they performed on the DHRT system.

In summary, both groups felt that their training system was an effective method of teaching the necessary skills for CVC placement. However, the DHRT system helped trainees more accurately assess their own performance by giving feedback on objective measures. The ability to accurately assess performance may have been improved by the multiple patient scenarios and specific feedback on performance cues provided.34 This is supported by the finding that participants’ self-ratings of performance were correlated to the score generated by the DHRT. This is especially interesting when considering the fact that the robotic training group had consistently lower self-ratings of performance, even though the analysis of the CLSE revealed that both groups were equally confident in their ability to perform CVC skills. The findings that robot-trained participants rated their performance more critically and accurately suggest that there may be a discrepancy between how well manikin-trained novices think they are performing and how well they are actually performing. These findings, combined with prior research,44 suggest that individuals trained in the manikin group were over confident in their skills. Although these results indicate that the self-efficacy may not be a good reflection of actual performance, it is still important to evaluate self-efficacy as part of overall skills competency45 because research has shown that self-efficacy can affect skill gains46 and performance.47

CONCLUSION

The goal of this research was to determine how self-efficacy and self-ratings of performance were affected by the functional fidelity and feedback provided of the surgical training system. The main findings of the study were as follows: (1) participants in both the manikin and DHRT condition showed statistically significant improvements in their CVC insertion skill self-efficacy over the course of their training; (2) participants in both training groups increased their self-ratings over the course of their training, but participants in the manikin group rated their self-performance higher than participants in the DHRT system over the course of the training; (3) participants who used the DHRT system reported significantly more errors during their training; and (4) participants who used the DHRT system were able to predict their objective performance score on the DHRT system.

Although these results are promising for the use of the DHRT system, there are several limitations to this study. One limitation of this study is that the patient scenarios and objective personalized feedback were examined as a complete system and not investigated separately. In other words, we holistically compared gains in self-confidence and performance assessment between 2 functionally distinct simulators: manikin, which provides a single anatomical configuration and subjective feedback, and robotic, which provides multiple patient configuration and objective feedback. As such, we can only conclude that both of these items, together, affect performance and cannot speak to the effect of these 2 items individually. Future work should be geared at exploring these factors in greater depth. Additionally, the pretests and posttests were conducted on the same manikin that was used for the manikin training group. This is encouraging for the use of the DHRT as a training method since participants trained using this device were equally confident in their ability to perform the procedure, but it does limit the conclusions that can be drawn about participants in the manikin group. Although this robotic system is not currently available on the market, our team is working to duplicate and commercialize this patent-pending system. The detailed costs of the system can be found in Pepley et al.19 Lastly, this study focused performance on training simulators, but did not examine skill transfer. Future work will focus on the effect of training in these robotic simulators on central line placement in clinical settings through an on-going longitudinal study in the surgical intensive care unit.16,40

Acknowledgments

Research reported in this publication was supported by the National Heart, Lung, And Blood Institute of the National Institutes of Health under Award Number R01HL127316. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health

APPENDIX.: CVC INSERTION CHECKLIST

Please identify how successful the resident was in performing each of the following skills without any assistance or prompting:

Satisfactory Comments
Selecting the appropriate site for venipuncture Y N
Selecting the correct ultrasound probe and using it in the appropriate orientation Y N
Obtaining a clear image of the target vessels using the ultrasound machine Y N
Correctly distinguishing artery and vein: demonstrates compressibility of vein Y N
Inserting the needle at a 35–45° angle from the skin Y N
Locating the needles position on the ultrasound image Y N
Advancing the introducer needle slowly and steadily Y N
Placing the introducer needle at the center of the vein Y N
Number of attempts ______
Confirming vessel entry by aspiration of blood Y N
Conducting the entire procedure without any mistakes Y N
Open in a new tab

References

  • 1.Polderman KH, Girbes AR. Central venous catheter use. Intensive Care Med. 2002;28(1):1–17. doi: 10.1007/s00134-001-1154-9. [DOI] [PubMed] [Google Scholar]
  • 2.Inhoff AW, Wang J. Encoding of text, manual movement planning, and eye-hand coordination during copytyping. J Exp Psychol Hum Percept Perform. 1992;18(2):437–448. doi: 10.1037//0096-1523.18.2.437. [DOI] [PubMed] [Google Scholar]
  • 3.McGee DC, Gould MK. Preventing complications of central venous catheterization. N Engl J Med. 2003;348(12):1123–1133. doi: 10.1056/NEJMra011883. [DOI] [PubMed] [Google Scholar]
  • 4.Merrer J, De Jonghe B, Golliot F, et al. Complications of femoral and subclavian venous catheterization in critically ill patients: a randomized controlled trial. J Am Med Assoc. 2001;286(6):700–707. doi: 10.1001/jama.286.6.700. [DOI] [PubMed] [Google Scholar]
  • 5.Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients: results of the Harvard Medical Practice Study II. N Engl J Med. 1991;324(6):377–384. doi: 10.1056/NEJM199102073240605. [DOI] [PubMed] [Google Scholar]
  • 6.Sznajder JI, Zveibil FR, Bitterman H, Weiner P, Bursztein S. Central vein catheterization: failure and complication rates by three percutaneous approaches. Arch Intern Med. 1986;146(2):259–261. doi: 10.1001/archinte.146.2.259. [DOI] [PubMed] [Google Scholar]
  • 7.Moureau N, Lamperti M, Kelly LJ, et al. Evidence-based consensus on the insertion of central venous access devices: definition of minimal requirements for training. Br J Anaesth. 2013;110(3):347–356. doi: 10.1093/bja/aes499. [DOI] [PubMed] [Google Scholar]
  • 8.Cook DA, Hamstra SJ, Brydges R, et al. Comparative effectiveness of instructional design features in simulation-based education: systematic review and meta-analysis. Med Teach. 2013;35(1):e867–e898. doi: 10.3109/0142159X.2012.714886. [DOI] [PubMed] [Google Scholar]
  • 9.Alessi SM. Fidelity in the design of instructional simulations. J Comput Based instruct. 1988;15(2):40–47. [Google Scholar]
  • 10.Norman G, Dore K, Grierson L. The minimal relationship between simulation fidelity and transfer of learning. Med Educ. 2012;46(7):636–647. doi: 10.1111/j.1365-2923.2012.04243.x. [DOI] [PubMed] [Google Scholar]
  • 11.Barsuk JH, McGaghie WC, Cohen ER, Balachandran JS, Wayne DB. Use of simulation-based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit. J Hosp Med. 2009;4(7):397–403. doi: 10.1002/jhm.468. [DOI] [PubMed] [Google Scholar]
  • 12.Mansfield PF, Hohn DC, Fornage BD, Gregurich MA, Ota DM. Complications and failures of subclavian-vein catheterization. N Engl J Med. 1994;331(26):1735–1738. doi: 10.1056/NEJM199412293312602. [DOI] [PubMed] [Google Scholar]
  • 13.Ogden PE, Cobbs LS, Howell MR, Sibbitt SJ, DiPette DJ. Clinical simulation: importance to the internal medicine educational mission. Am J Med. 2007;120(9):820–824. doi: 10.1016/j.amjmed.2007.06.017. [DOI] [PubMed] [Google Scholar]
  • 14.Sherertz RJ, Ely EW, Westbrook DM, et al. Education of physicians-in-training can decrease the risk for vascular catheter infection. Ann Intern Med. 2000;132(8):641–648. doi: 10.7326/0003-4819-132-8-200004180-00007. [DOI] [PubMed] [Google Scholar]
  • 15.Pepley D, Yovanoff M, Mirkin K, Miller S, Han D, Moore J. A virtual reality haptic robotic simulator for central venous catheterization training. J Med Device. 2016;10(3):030937. doi: 10.1115/1.4033867. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.McDougall EM. Validation of surgical simulators. J Endourol. 2007;21(3):244–247. doi: 10.1089/end.2007.9985. [DOI] [PubMed] [Google Scholar]
  • 17.Yovanoff M, Pepley D, Mirkin K, Moore J, Han D, Miller S. Improving medical education: simulating changes in patient anatomy using dynamic haptic feedback. Proceedings of the Human Factors and Ergonomics Society Annual Meeting; Washington D. C. Los Angeles, CA: SAGE Publications Sage CA; 2016. pp. 603–607. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Chen H-E, Yovanoff M, Pepley D, et al. Can haptic simulators distinguish expert performance? A case study in central venous catheterization in medical education. Simul Healthc. 2018 doi: 10.1097/SIH.0000000000000352. In review. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Pepley DF, Gordon AB, Yovanoff M, et al. Training surgical residents with a haptic robotic central venous catherization simulator. J Surg Educ. 2017;74(6):1066–1073. doi: 10.1016/j.jsurg.2017.06.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Rieber LP, Tzeng S-C, Tribble K. Discovery learning, representation, and explanation within a computer-based simulation: finding the right mix. Learn Instr. 2004;14(3):307–323. [Google Scholar]
  • 21.Yovanoff M, Pepley D, Mirkin K, Moore J, Han D, Miller S. Personalized learning in medical education: designing a user interface for a dynamic haptic robotic trainer for central venous catheterization. Proceedings of the Human Factors and Ergonomics Society Annual Meeting Austin, TX: SAGE Publications Sage CA: Los Angeles, CA. 2017;61(1):615–619. doi: 10.1177/1541931213601639. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Saemi E, Porter JM, Ghotbi-Varzaneh A, Zarghami M, Maleki F. Knowledge of results after relatively good trials enhances self-efficacy and motor learning. Psychol Sport Exerc. 2012;13(4):378–382. [Google Scholar]
  • 23.Badami R, VaezMousavi M, Wulf G, Namazizadeh M. Feedback about more accurate versus less accurate trials: differential effects on self-confidence and activation. Res Q Exerc Sport. 2012;83(2):196–203. doi: 10.1080/02701367.2012.10599850. [DOI] [PubMed] [Google Scholar]
  • 24.Schmidt RA, Bjork RA. New conceptualizations of practice: common principles in three paradigms suggest new concepts for training. Psychol Sci. 1992;3(4):207–218. [Google Scholar]
  • 25.Shafizadeh M, Platt GK, Bahram A. Effects of focus of attention and type of practice on learning and self-efficacy in dart throwing. Percept Mot Skills. 2013;117(1):182–192. doi: 10.2466/25.30.pms.117x12z5. [DOI] [PubMed] [Google Scholar]
  • 26.Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191–215. doi: 10.1037//0033-295x.84.2.191. [DOI] [PubMed] [Google Scholar]
  • 27.Bandura A. Self-efficacy: The Exercise of Control. New York, NY: W.H. Freeman and Company; 1997. [Google Scholar]
  • 28.Themanson JR, Rosen PJ. Examining the relationships between self-efficacy, task-relevant attentional control, and task performance: evidence from event-related brain potentials. Br J Psychol. 2015;106(2):253–271. doi: 10.1111/bjop.12091. [DOI] [PubMed] [Google Scholar]
  • 29.Lee TD, Swinnen SP, Serrien DJ. Cognitive effort and motor learning. Quest. 1994;46(3):328–344. [Google Scholar]
  • 30.Evans LV, Dodge KL, Shah TD, et al. Simulation training in central venous catheter insertion: improved performance in clinical practice. Acad Med. 2010;85(9):1462–1469. doi: 10.1097/ACM.0b013e3181eac9a3. [DOI] [PubMed] [Google Scholar]
  • 31.Moritz SE, Feltz DL, Fahrbach KR, Mack DE. The relation of self-efficacy measures to sport performance: a meta-analytic review. Res Q Exerc Sport. 2000;71(3):280–294. doi: 10.1080/02701367.2000.10608908. [DOI] [PubMed] [Google Scholar]
  • 32.Pena G, Altree M, Field J, et al. Surgeons ‘and trainees’ perceived self-efficacy in operating theatre non-technical skills. Br J Surg. 2015;102(6):708–715. doi: 10.1002/bjs.9787. [DOI] [PubMed] [Google Scholar]
  • 33.Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. J Am Med Assoc. 2006;296(9):1094–1102. doi: 10.1001/jama.296.9.1094. [DOI] [PubMed] [Google Scholar]
  • 34.Butler DL, Winne PH. Feedback and self-regulated learning: a theoretical synthesis. Rev Educ Res. 1995;65(3):245–281. [Google Scholar]
  • 35.Ericsson KA, Simon HA. Verbal reports as data. Psychol Rev. 1980;87(3):215–251. [Google Scholar]
  • 36.Barsuk JH, Cohen ER, Nguyen D, et al. Attending physician adherence to a 29-component central venous catheter bundle checklist during simulated procedures. Crit Care Med. 2016;44(10):1871–1881. doi: 10.1097/CCM.0000000000001831. [DOI] [PubMed] [Google Scholar]
  • 37.Smith CC, Huang GC, Newman LR, et al. Simulation training and its effect on long-term resident performance in central venous catheterization. Simul Healthc. 2010;5(3):146–151. doi: 10.1097/SIH.0b013e3181dd9672. [DOI] [PubMed] [Google Scholar]
  • 38.Pepley D, Yovanoff M, Mirkin K, Han D, Miller S, Moore J. Design of a virtual reality haptic robotic central venous catheterization training simulator. ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference; Charlotte, NC, USA: American Society of Mechanical Engineers; 2016. V05AT7A033-V05AT07A. [Google Scholar]
  • 39.Mayring P. Approaches to Qualitative Research in Mathematics Education. Dordrecht: Springer; 2015. Qualitative content analysis: theoretical background and procedures; pp. 365–380. [Google Scholar]
  • 40.Ericsson KA. The Cambridge Handbook of Expertise and Expert Performance. Cambridge University Press; 2006. The influence of experience and deliberate practice on the development of superior expert performance; pp. 683–703. [Google Scholar]
  • 41.Gardner AK, Abdelfattah K, Wiersch J, Ahmed RA, Willis RE. Embracing errors in simulation-based training: the effect of error training on retention and transfer of central venous catheter skills. J Surg Educ. 2015;72(6):e158–e162. doi: 10.1016/j.jsurg.2015.08.002. [DOI] [PubMed] [Google Scholar]
  • 42.Oliver WC, Nuttall GA, Beynen FM, Raimundo HS, Abenstein JP, Arnold JJ. The incidence of artery puncture with central venous cannulation using a modified technique for detection and prevention of arterial cannulation. J Cardiothorac Vasc Anesth. 1997;11(7):851–855. doi: 10.1016/s1053-0770(97)90119-1. [DOI] [PubMed] [Google Scholar]
  • 43.Bowdle A. Vascular complications of central venous catheter placement: evidence-based methods for prevention and treatment. J Cardiothorac Vasc Anesth. 2014;28(2):358–368. doi: 10.1053/j.jvca.2013.02.027. [DOI] [PubMed] [Google Scholar]
  • 44.Leopold SS, Morgan HD, Kadel NJ, Gardner GC, Schaad DC, Wolf FM. Impact of educational intervention on confidence and competence in the performance of a simple surgical task. J Bone Joint Surg Am. 2005;87(5):1031–1037. doi: 10.2106/JBJS.D.02434. [DOI] [PubMed] [Google Scholar]
  • 45.Dijksterhuis MG, Voorhuis M, Teunissen PW, et al. Assessment of competence and progressive independence in postgraduate clinical training. Med Educ. 2009;43(12):1156–1165. doi: 10.1111/j.1365-2923.2009.03509.x. [DOI] [PubMed] [Google Scholar]
  • 46.Jourden FJ, Bandura A, Banfield JT. The impact of conceptions of ability on self-regulatory factors and motor skill acquisition. J Sport Exerc Psychol. 1991;13(3):213–226. [Google Scholar]
  • 47.Neiss R. Expectancy in motor behavior: a crucial element of the psychobiological states that affect performance. Hum Perform. 1989;2(4):273–300. [Google Scholar]

RESOURCES