Looks can be deceiving: Gaze pattern differences between novices and experts during placement of central lines

Abstract

Background:

The objective of this study was to determine whether gaze patterns could differentiate expertise during simulated ultrasound-guided Internal Jugular Central Venous Catheterization (US-IJCVC) and if expert gazes were different between simulators of varying functional and structural fidelity.

Methods:

A 2017 study compared eye gaze patterns of expert surgeons (n=11), senior residents (n=4), and novices (n=7) during CVC needle insertions using the dynamic haptic robotic trainer (DHRT), a system which simulates US-IJCVC. Expert gaze patterns were also compared between a manikin and the DHRT.

Results:

Expert gaze patterns were consistent between the manikin and DHRT environments (p = 0.401). On the DHRT system, CVC experience significantly impacted the percent of time participants spent gazing at the ultrasound screen (p < 0.0005) and the needle and ultrasound probe (p < 0.0005).

Conclusion:

Gaze patterns differentiate expertise during ultrasound-guided CVC placement and the fidelity of the simulator does not impact gaze patterns.

Introduction

Ultrasound-guided central venous catheterization (CVC) is a procedural skill that has been taught on manikin simulators for over a decade¹. While research has shown these simulators are effective for short term skill gains, these manikins have several known limitations, including a lack of long-term skill retention². This has been attributed, in part, to the fact that these simulators only train surgeons on a single patient anatomy and thus do not represent the range of patient profiles surgeons face in clinical settings³. In addition, these simulators lack objective performance criterion and instead rely on a trained preceptor (e.g. faculty) to be present in order to subjectively evaluate trainee performance using a proficiency checklist^4–6 which includes evaluation on mechanical (e.g. motion and accuracy) and procedural-based skills (e.g. aseptic technique). Finally, current CVC simulators provide only basic summative feedback on performance (blue liquid is aspirated if the catheter hits a vein) and no concurrent or formative feedback. Because of these shortcomings and the overreliance on faculty feedback, simulation-based surgical training has been criticized as being resource intensive^7,8.

In light of these deficiencies, the Dynamic Haptic Robotic Trainer (DHRT) was developed to provide residents with CVC training on a variety of patient anatomies while providing automated feedback during the training process, see Figure 1. While the DHRT system lacks some of the physical realism of manikin simulators, it has increased functional fidelity. Specifically, the DHRT provides training on ultrasound guided CVC needle placement by simulating variations in patient anatomy through changes in a virtual ultrasound image (e.g. size and depth of the vessels) and through haptic feedback provided through a robotic arm that simulates force changes of different types of tissues (e.g. skin, adipose tissue, vessel), see Pepley et al. for full details^9,10. The DHRT also provides learners with automated feedback on their mechanical performance after each needle insertion attempt without the need of a trained preceptor, including feedback on the number of insertion attempts, average angle of insertion, and the final distance of the needle tip from the center of the vein¹¹. Prior research has shown that novice learners approach expert performance when training on the DHRT system¹⁰ and indicated that there were no differences in pre- to post-test learning between manikin- and DHRT-based instruction¹². While this prior work on the DHRT is promising, the feedback provided to learners is based purely on their mechanical skill acquisition (e.g. needle angle) and not on the cognitive skills necessary to complete this surgical procedure. This is important because prior studies have shown that cognitive skills training and mental imagery practice enhance the learning and acquisition of surgical skills^13–15. One method of improving cognitive skills training in the DHRT system is through the use of eye-tracking and gaze-training.

Figure 1:

(Left) Experimental setup with participant wearing Tobii Pro Glasses 2 at the DHRT system. (Right) Two main areas of interest (AOIs). The ultrasound screen and the needle and ultrasound probe.

In medical domains, gaze behavior has been used for skills training¹⁶, expertise assessment^17,18, and feedback¹⁹. Reviews of eye-tracking applications in medical and non-medical domains have also shown the reliability and validity for using eye-tracking as an objective assessment tool, as well as its potential in assessing surgical skills^20,21. For example, one study on virtual laparoscopic training found that experts spend more time gazing at target locations while novices spend more time tracking tools²². When using gaze training as a teaching tool in laparoscopic surgery, researchers have also found that gaze trained groups learned more quickly than self-guided groups²³, and adopted expert-like gaze patterns by fixating more on target locations rather than on tool movement²⁴. Finally, Tien and colleagues²⁵ validated the use of eye tracking for differentiating expert and novice gaze behavior during open hernia surgery.

While this prior work demonstrates the potential benefits of differentiating expert and novice gaze patterns and using gaze training to accelerate learning, this has not yet been explored in CVC training. Thus, as a first step to improving training feedback, the goal of the current study was to assess gaze tracking as it relates to CVC placement and determine whether differences between experts and novices exist. If so, there is potential to use these differences to provide additional formative feedback to aid a learner’s path to competency in CVC training.

Research Questions

The purpose of the current study was to determine whether gaze patterns could differentiate expertise during simulated ultrasound-guided Internal Jugular Central Venous Catheterization (US-IJCVC) and if expert gazes were different between simulators of varying functional and structural fidelity. Thus, the two research questions (RQ) explored in the current study were:

RQ1: Are expert eye gaze patterns consistent during central-line insertions between manikin and DHRT training systems? This research question was developed to validate if eye gaze patterns were consistent between the manikin (higher structural fidelity or physical realism) and the DHRT (higher functional fidelity) systems. It was hypothesized that since the ultrasound-guided procedure for completing the central line was similar between the two systems, gaze patterns would not differ between the environments.

RQ2: Do expert, senior resident, and novice gaze patterns differ when placing central lines on the DHRT? The second research question aimed to compare the percent of time experts, senior residents, and novices spent gazing at the ultrasound screen versus the needle and ultrasound probe during placement of central lines on the DHRT system. It was hypothesized that novices would spend a higher percentage of time tracking the needle and ultrasound probe during these trials, while experts would spend more time focused on the ultrasound screen, because past research has shown that experts spend less time tool-tracking than novices²². Senior residents, representative of a learner group that is often considered competent enough to perform the skill on patients with limited to no supervision²⁶, were hypothesized to perform similarly to experts in this study.

Methods and Materials

In order to answer these research questions, a series of experiments were conducted at Penn State Milton S. Hershey Medical Center (HMC). The details of this experiment, conducted as part of a larger study, are discussed in this section.

Participants

Data for the current study was collected over two time periods. Specifically, the data from summer of 2017 included seven surgical interns (novice) (N_n=7) while the data from fall of 2017 included eleven experts (N_e=11) and four senior residents (N_sr=4). To qualify expertise, the expert participants in this study included four fellows and seven attending surgeons who self-reported to have placed more than 50 central lines in their career, a procedural volume which has been stated to denote expertise in earlier research on central-lines^27,28. On the other hand, the senior residents were all PGY4 or PGY5, and had placed a minimum of 10 central lines on patients in the course of their careers and only one of the novice residents had placed an Internal Jugular CVC prior to the study, reporting they had placed 3 prior lines. Senior residents were included in the current study to identify whether residents perform similarly to experts after several years of residency training. This was particularly interesting since a large portion of central-lines are placed by surgical residents at HMC while under the supervision of attending faculty.

Procedure

At the start of the study, the purposes and procedures were explained to all of the participants and informed consent was obtained according to an Institutional Review Board approved protocol. In the summer session, the novices were first shown an 18-minute video on CVC placement followed by a live training demonstration of the procedure by a fourth-year surgical resident using a Blue Phantom Gen II Ultrasound Central Line Training Model (Model #BPH660). This video was not included in the fall session, given the experience of the expert and senior residents. For the data collection, each participant was fitted and calibrated with the Tobii Pro Glasses 2²⁹, see Figure 1, and asked to complete one baseline needle insertion for central line placement on the Blue Phantom manikin. This was followed by an introduction to the DHRT through a 5-minute video, after which, the participants individually completed two practice insertions on the DHRT in order to familiarize themselves with its functions. During these two practice insertions, the participants were guided through the functions of the DHRT and any questions were answered. Finally, the DHRT insertion data was collected, where each participant completed 10 insertions on the DHRT system. All of the participants received performance feedback from the DHRT learning interface, and no other external feedback was provided. Due to restrictions in the residency boot camp where the resident data was collected, novices had a day between their first two practice insertion and their 10 insertion trials. On the other hand, the 2 practice insertions and 10 insertion trials were conducted on the same day for the experts and senior residents, see Figure 2 for timeline of study.

Figure 2.

Diagram summarizing the procedural flow

Eye Tracking Metrics

Eye gaze point data collected during the study was analyzed using the Tobii Pro Lab real-world automatic mapping functionality³⁰. Once auto-mapped, an independent rater used the attention filter to manually check all auto-mapped data. Specifically, raw data from trial 1 was analyzed by one rater and compared to the automatically mapped and checked data to validate the use of the auto-mapping feature. Importantly, the inter-rater reliability was found to be very good, as classified by Altman (p.404)³¹, Cohen’s Kappa = 0.827, p < 0.0005. Thus, the automatic mapping function with the attention filter in the Tobii Pro Lab software was used to analyze participant eye gaze patterns.

Once the gaze points were properly mapped, data for each participant and each trial was reviewed for its accuracy and gaze sample percentage captured during needle insertions. For display or screen-based tasks, eye-tracking translational researchers recommend excluding participants who spend less than 70% looking at the screen^32,33. With the Tobii Pro Glasses 2, gaze points are only captured when the participant is looking through the glasses range (e.g. not above or below the rim of the glasses). Thus, for this study a lower threshold was set at 70% of each participant’s eye gazes captured during needle insertions. During review of the collected data, one expert and one novice participants’ gaze patterns were discarded due to having less than 70% (in many cases, less than 50%) of their gaze points tracked during the needle insertions, leading to a total participant count of 6 novices, 10 experts, and 4 senior residents, see Figure 2 for summary.

After participant gaze data for all 10 trials were reviewed for eye-tracking quality, the data were categorized into two main areas of interest (AOI): the ultrasound image on the screen, and the needle and ultrasound probe interface (Figure 1). These two areas of interest were chosen because prior work has compared the amount of time spent gazing at the target (e.g. ultrasound or image visualization), compared to the amount of time spent tracking the tools or patient (e.g. needle and ultrasound probe)²². The percent time spent gazing at these two AOIs during each needle insertion was calculated in lieu of other eye-tracking metrics such as fixation duration or count due to the large variability in time to complete needle insertion trials among the participant groups. Specifically, the average (SD) for each group was as follows: experts 22.41 (13.93), senior residents 24.75 (8.05), and novices 29.66 (12.60). Thus, the percent metric was used as a way to normalize gaze patterns among participant groups.

Data Analysis

All metrics were analyzed using SPSS (v. 25.0) with significance considered at p-valueof 0.05. For the first research question, two paired-samples t-tests were computed to determine differences in the percent of time experts gazed at the ultrasound screen and the needle and ultrasound probe between the one insertion trial conducted on the manikin and the average of the ten trials completed on the DHRT system. The data had no significant outliers, as assessed by boxplots, and the assumption of normality was not violated, as assessed by boxplot and the Kolmogorov Smirnov test (p > 0.200).

For the second research question, a one-way multiple analysis of variance (MANOVA) was computed to determine if the percent of time participants spent gazing at the ultrasound screen or the needle and ultrasound probe were different among groups with different levels of central-line insertion experience throughout the 10 trials on the DHRT system. Groups were compared as a whole, with each needle insertion representing one data point. Preliminary assumption checking revealed that data were normally distributed, as assessed by boxplot and the Kolmogorov Smirnov test (p > 0.200); there were no univariate or multivariate outliers, as assessed by boxplot and Mahalanobis distance (p > 0.001), respectively; there were no linear relationships, as assessed by scatterplot; no multicollinearity (r = −0.887, p < 0.001). Follow-up post-hoc analyses were conducted to examine pairwise differences between groups.

Results

Data are presented as mean ± standard error unless otherwise noted. Analysis of the areas of interests (AOI) showed that participants on average spent a total of 96.8 ± 1.44% of their time during the study trials gazing at the ultrasound screen and the needle and ultrasound probe. This indicates the importance of focusing data analyses on these two AOIs. During the study, the participants spent an average of 74.3 ± 0.72% percent of the time looking at the ultrasound screen and an average of 22.5 ± 0.72% of time tracking the needle and ultrasound probe.

The paired-samples t-test showed that there was no significant difference between the percent of time experts spent gazing at the ultrasound screen on the manikin (71.94 ± 2.80) and the DHRT system (74.63 ± 0.99), t(9) = −0.820, p = 0.433. Likewise, there was no significant difference between the percent of time experts spent gazing at the needle and ultrasound probe on the manikin (21.17 ± 3.34) and the DHRT system (21.59 ± 1.05), t(9) = −0.112, p = 0.913.

The one-way MANOVA results showed that the differences among central line insertion experience (novice, senior resident, and expert) on the combined dependent variables (percent gaze time on ultrasound screen and needle and ultrasound probe) was statistically significant, F (4, 392) = 6.254, p < 0.005, Pillai’s Trace = 0.119, partial η²= 0.06, observed power of 0.989. See Figure 3 for example gaze plots for the three groups. Follow-up univariate ANOVAs showed that both the percent time spent gazing at the ultrasound screen (F(2, 1024.7) = 10.976, p < 0.0005, η² = 0.100, observed power of 0.991), and the needle and ultrasound probe (F(2, 862.239) = 8.872, p < 0.0005, η² = 0.083, observed power of 0.971) were statistically different between the participants from the different central-line experience groups, using a Bonferroni adjusted α level of 0.025.

Figure 3:

(Left) Example novice gaze pattern on DHRT. (Middle) Example senior resident gaze pattern. (Right) Example expert gaze pattern.

Because the assumption of homogeneity of variances was violated for both the percent time spent gazing at the ultrasound screen and the needle and ultrasound probe, as assessed by Levene’s Test for Equality of Variance (p = 0.004 and 0.019, respectively), the Games-Howell post-hoc test was used to compare pairwise differences between the group gaze patterns, see Figure 3. The post-hoc results showed that novices (70.32 ± 1.41) spent significantly less time gazing at the ultrasound screen than experts (74.63 ± 0.99) who spent significantly less time than the senior residents (79.52 ± 1.02). In addition, novices (26.68 ± 1.35) spent significantly more time tracking the needle and ultrasound probe than experts (21.59 ± 1.05) who were not significantly different than senior residents (18.63 ± 1.10). Specifically, differences between novices and experts (−4.30, 95% CI [−8.39, −0.21], p = 0.037), novices and residents (−9.20, 95% CI [−13.34, −5.05], p = 0.0005), and experts and residents (−4.89, 95% CI [−8.28, −1.51], p = 0.002) were statistically significant when gazing at the ultrasound screen. On the needle and ultrasound probe, differences between novices and experts (5.08, 95% CI [1.03, 9.12], p = 0.010), as well as novices and residents (8.04, 95% CI [3.90, 12.18], p = 0.0005) were statistically significant. However, there was no significant difference between experts and senior residents (2.97, 95% CI [−0.64, 6.57], p = 0.129).

Discussion

The results for the first research question support our hypothesis that gaze patterns would not differ between the manikin and DHRT environments. This demonstrates that even in the absence of the physical realism, in the DHRT simulator, experts maintained similar gaze patterns. These results provide evidence that the DHRT system may require similar eye gaze patterns even in the absence of these physical features.

The results for the second research question showed that expertise in IJCVC can be differentiated using gaze behavior, supporting related work that differentiated expertise with gaze patterns during open hernia surgery and laparoscopic environments^22,25.

Finally, it was hypothesized that experts would spend more time gazing at the ultrasound screen, while novices would spend more time tracking the needle and ultrasound probe, with senior residents falling along a continuum between the two groups. An interesting discovery was that senior residents focused more attention on the ultrasound screen and less time tracking the needle and ultrasound probe when compared to experts, thus falling farthest on the spectrum from novices. When combined with prior work showing that senior resident performance was comparable with expert performance¹⁰, this additional similarity in gaze patterns indicate that senior residents attain expertise in CVC placement during their post graduate residency career. This may be due in part to the fact that IJCVC are largely performed by residents at HMC under the supervision of experts. This means that while our experts met the criteria of a minimum 50 central lines)²⁷, the senior residents in the study may have had more active line placement in the months leading up to the testing which may have led to higher similarities in the performance among these two groups.

Limitations

This study was performed at a large academic institution. While the participant numbers were limited, the current study provides some of the first evidence for the use of gaze training in IJCVC education. All interns at our institution attend surgical bootcamp, thus protected time opportunity to participate in the study was present. Experts and senior surgical resident participation occurred during clinical work-days. Thus, time was not protected for study participation, potentially leading to a self-selection bias as well for individuals with higher interest or experience with CVC procedure. Finally, data was excluded for two participants (1 novice, 1 expert) due to poor tracking quality. Future work should examine more participants at various stages of expertise.

Conclusions

The current study differentiated expert and novice gaze patterns during CVC placement on the DHRT system and showed that experts performed similarly between DHRT and manikin-based simulation systems. These findings indicate that the DHRT training system effectively captures gaze behaviors of expert surgeons as well as manikin-based systems. Additionally, the results showed that differences in gaze patterns could be discerned between varying levels of expertise providing evidence on the potential utility of gaze training in IJCVC education.

Figure 4:

Average percent of time each group spent gazing at the ultrasound screen and the needle and ultrasound probe. On the ultrasound screen, novices spent significantly less time than experts, who spent significantly less time the senior residents. On the needle and ultrasound probe, novices spent significantly more time than experts, who were not significantly different than senior residents.