Exploring Eye‐tracking Technology as an Assessment Tool for Point‐of‐care Ultrasound Training

Abstract

Objectives

Eye‐tracking technology has emerged as a potentially useful learner assessment tool in several medical specialties. In the fields of general surgery and anesthesiology, it has been shown to reliably differentiate between different levels of expertise in procedural skills. In the field of radiology, it has been shown to be a valid assessment tool for diagnostic test interpretation. Current methods of competency assessment in point‐of‐care ultrasound (POCUS) remain a challenge, because they require significant direct observation time by an instructor. The purpose of this study was to determine if eye‐tracking technology can accurately distinguish between novice and experts in the interpretation of POCUS clips, specifically of the focused assessment using sonography in trauma (FAST) scan.

Methods

A convenience sample of medical students, residents, and emergency physicians from a single academic emergency department were invited to participate. Participants included both novices and experts in POCUS. Each participant completed a baseline questionnaire and viewed 16 video clips of a FAST ultrasound examination while their gaze patterns were recorded by a commercially available eye‐tracking device. The primary outcome was total gaze time on the area of interest (AOI). Secondary outcomes included total time to fixation, mean number of fixations, and mean duration of first fixation on the AOI.

Results

Fifteen novices and 15 experts completed this study. For total gaze time on the AOI, experts fixated their gaze significantly longer than novices (75.8 ± 16.2 seconds vs. 56.6 ± 12.8 seconds, p = 0.001). Similarly, experts were significantly faster to fixate on the AOI and had a higher fixation count on the AOI (8.5 ± 4.0 seconds vs. 15.1 ± 6.8 seconds, p = 0.003; and 170 ± 30 vs. 143 ± 28 seconds, p = 0.016). There were no differences on the mean duration of first fixation on the AOI (0.42 ± 0.12 seconds vs. 0.39 ± 0.09 seconds, p = 0.467).

Conclusion

Eye‐tracking technology shows the potential to differentiate between experts and novices by their gaze patterns on video clips of FAST examinations. The total gaze time on the AOI may be a useful metric to help in the assessment of competency in POCUS image interpretation. In addition, the evaluation of gaze patterns may help educators identify causes of interpretation errors. Future studies are needed to further validate these metrics in a larger cohort.

The use of point‐of‐care ultrasound (POCUS) has expanded beyond its traditional role in radiology, obstetrics, and cardiology. It is now utilized across many medical and surgical specialties ¹ because it can improve physical examination findings and diagnostic accuracy. ² Incorporation of POCUS training into postgraduate medical education has increased and is a core component in most emergency medicine curriculums around the world. ³ , ⁴ , ⁵ , ⁶ Assessment for competency in POCUS is a challenge as most assessment methods are subjective and time‐intensive. Assessing the ability to interpret POCUS images often requires a combination of multiple choice questions, visual examinations, simulator models, videotape reviews, supervised scans, overreading of images by sonologists, and logging a predetermined number of scans for each application. ⁵ , ⁶ , ⁷

Eye‐tracking technology involves measuring an individual’s eye movements over time as they complete a task or procedure. ⁸ It reflects attentional behavior and has proven to be a reliable and accurate way detailing attention allocation. ⁹ Gaze analysis has been shown to provide reliable quantitative data that can be interpreted to give an indication of clinical skill and provide feedback in educational assessment. ¹⁰ It has been shown that proficiency in a task improves when there is an increase in focused attention that can be measured by fixation time in the area of interest (AOI). ¹¹ Previous studies in the fields of general surgery, anesthesiology, and radiology have demonstrated that eye‐tracking can reliably differentiate novices from experts and may provide a useful objective metric of competency. ¹² , ¹³ , ¹⁴

To date, eye‐tracking technology has not been evaluated as an assessment tool for interpreting POCUS video clips. If this technology proves to be a valid tool, it may help provide objective metrics to support a learner’s progression through their training. In addition, the analysis of gaze patterns may be useful to help identify knowledge gaps and facilitate remediation. ¹⁰ The primary objective of this study was to determine whether gaze pattern analysis is a valid method for distinguishing between novices and experts in the domain of interpretation of POCUS video clips.

METHODS

Study Design

We conducted a prospective observational study from February 2020 to April 2020. This study was approved by the research ethics board at Sunnybrook Health Sciences Centre.

Setting and Study Population

The study was conducted at a large academic health sciences center. A convenience sample of emergency physicians, residents, and medical students on rotation at the emergency department were invited to participate. The inclusion criteria for the expert group was previous training and formal certification in POCUS and regular use of POCUS in clinical practice for more than 3 years. This group included attending physicians, ultrasound trained fellows, and residents certified in POCUS by an accrediting body. The inclusion criteria for the novice group composed of participants without any formal POCUS certification from an accreditation body. However, the novice participants had all completed an introductory POCUS workshop. Participants were excluded from the study if they failed the calibration with the eye‐tracking device (for example, due to strabismus). All participants completed a data collection form on basic demographics and experience with POCUS in the focused assessment using sonography for trauma (FAST) application.

Study Protocol

For gaze tracking, we used a commercially available eye‐tracking device, Tobii Pro Nano (Tobii, Karlsrovagen, Sweden). Each participant was seated in front of a 14‐inch monitor with the eye‐tracking device mounted at the bottom of the screen (Figure 1). Lighting in the room was kept constant with the overhead lights turned on during data collection and the monitor screen was set at a constant brightness and contrast.

Figure 1.

Schematic of setup from (A) horizon view and (B) aerial view.

Participants' eyes were calibrated using the eye‐tracking software (Tobii Pro Lab, Version 1.130) by having the participant following on a white dot across different spots on the screen. If the participant was unable to successfully calibrate to the screen, adjustment to the sitting position was made and the calibration process repeated. If the calibration was still unsuccessful after three attempts, the participant was removed from the study.

After calibration, the participant was instructed to view 16 videos that were part of the FAST protocol and was asked a standardized question. Each 4‐second video was looped twice and participants were not allowed to pause or stop the video. Following each video, the participant was asked about the presence of free fluid or pericardial effusion. The participant could also respond with “unsure” if they were unable to determine the answer. The eye movements were recorded continously throughout the session. Participants were blinded to the hypothesis of the study.

We designed our study protocol to gather validity evidence according to Messick’s framework. ¹⁵ The framework identifies five sources of evidence to support construct validity, namely content, response process, internal structure, relations to other variables, and consequences. This provides evidence from different sources to support any given interpretation. To demonstrate that eye tracking has the potential to be used as an instrument to infer a reasonable degree of competency, validity must be established for its intended interpretation.

Content evidence evaluates the “relationship between a test’s content and the construct it is intended to measure.” ¹⁶ To ensure content validity, videos used were selected from a database of ultrasound images to represent a variety of normal and abnormal images from the FAST application. They included videos recorded from the right upper quadrant, left upper quadrant, pelvic, and subxiphoid views (Figure 2). The AOI in the videos was determined by consensus between two faculty members who are POCUS experts and certified as Master Instructors by the Canadian Point‐of‐Care Ultrasound Society (CPoCUS). There was no disagreement on the AOI between the faculty members.

Figure 2.

Sample ultrasound images from (A) right upper quadrant, (B) left upper quadrant, (C) suprapubic, and (D) subxiphoid.

Response process is the evidence of data integrity such that all sources of error associated with the test administration are controlled or eliminated to the maximum extent possible. ¹⁷ It can also be done by reviewing the actions and thought processes of the test takers and the instrument measuring it. ¹⁸ Participants’ eyes were calibrated with the eye‐tracker and the study performed was performed in a controlled environment. Measurements were recorded and calculated using the eye tracker with a sampling frequency of 60Hz, which detects the eyes 60 times per second. It also has an accuracy of 0.3 degrees and precision of 0.1 degrees when determining the gaze on the screen. ¹⁹ Apart from the quantitative data of gaze parameters that were measured, qualitative data of the gaze map and the gaze process can also be used to assess the thought process.

To assess for relation to other variables, we had participants complete a written examination concurrently with the eye tracking assessment. This correlation with scores from the outcome of identifying pathology in the ultrasound videos would support or refute the underlying construct. Test scores is an existing instrument that is currently relied on heavily as a measure of competency in POCUS assessment. ⁶ The resulting convergence or divergence results can provide confirmatory or counter‐confirmatory evidence on validity.

Outcome Measures

The primary outcome was the total gaze time on the AOI (seconds). This was based on a study using eye‐tracking in ultrasound‐guided regional anesthesia ¹² , ²⁰ because there were no similar studies in POCUS. In that study, experts took less time to identify the sonoanatomy of interest and had a more focused gaze pattern when compared to novices. ¹² Secondary outcomes were total time to fixation on the AOI (seconds), mean number of fixations on the AOI, and mean duration of first fixation on the AOI (seconds). The outcome measures were generated using the Tobii Pro Lab software (Version 1.130).

Data Analysis

We calculated the sample size based on pilot study data. A total of 20 participants were required to establish a 40% difference between the groups with a power of 80%. Gaze parameters were automatically recorded by the eye‐tracking software, including total gaze time on the AOI, total time to fixation on the AOI, mean number of fixations on the AOI, and mean duration of first fixation on the AOI. Statistical analysis was performed using SPSS (Version 25). Student’s t‐test was used to calculate the difference between the different gaze parameters comparing novices and experts. Standard deviation was calculated along with the p‐value for the difference in results. p‐values were two‐sided with p < 0.05 considered as statistically significant.

We assessed the reliability of each parameter using Cronbach’s alpha. Cronbach’s alpha is a measure of internal consistency that describes the extent to which all the items in a test measure the same construct. ²¹ The interrelatedness of each gaze parameter within the test was analyzed separately to support its validity. Test scores were also analyzed using Student’s t‐test with its standard deviation and p‐value for the difference in results as a measure for relations to other variables to support validity.

RESULTS

A convenience sample of 32 participants (15 in the novice group and 17 in the expert group) were recruited. Two (6%) participants from the expert group were excluded due to unsuccessful eye‐tracker calibration despite adjusting the seating position. The remaining 30 participants completed all the study procedures and their baseline characteristics are shown in Table 1.

Table 1.

Baseline Characteristics of Participants

	Novice (n = 15)	Expert (n = 15)	p‐value
Sex
Male	6	9	0.289
Female	9	6
Age (years)
30 and below	8	3	0.034
31 to 40	5	7
Above 41	2	5
Mean no. of years of POCUS experience	0	8.3	<0.001

The primary outcome, total gaze time on the AOI, was significantly longer in the expert group compared with the novice group (75.8 ± 16.2 seconds vs. 56.6 ± 12.8 seconds, p = 0.001). Similarly, experts outperformed novices in total time to fixation on the AOI (8.5 ± 4.0 seconds vs. 15.1 ± 6.8 seconds, p = 0.003) with a quicker gaze to the AOI. Experts also demonstrated a higher fixation count on the AOI compared to novices (170 ± 30 seconds vs. 143 ± 28 seconds, p = 0.016).

There were no significant differences between experts and novices for the mean duration of the first fixation on the AOI (0.42 ± 0.12 seconds vs. 0.39 ± 0.09 seconds, p = 0.467). The results are summarized in Table 2.

Table 2.

Results of Eye‐tracking Parameters for Novice and Expert Participants

	No.	Mean	SD	Sig. (2‐tailed)
Total gaze time on AOI
Novice	15	56.6	12.8	0.001
Expert	15	75.8	16.2
Total time to fixation on AOI
Novice	15	15.1	6.8	0.003
Expert	15	8.5	4.0
Mean number of fixations on AOI
Novice	15	143	28	0.016
Expert	15	170	30
Mean duration of first fixation on AOI
Novice	15	0.39	0.09	0.467
Expert	15	0.42	0.12

AOI = area of interest.

Total gaze time on the AOI and mean number of fixations on AOI showed good internal consistency, with α = 0.867 and α = 0.830, respectively. Total time to fixation on AOI and mean duration of first fixation on AOI demonstrated low internal consistency with α < 0.5 (Table 3).

Table 3.

Internal Reliability of Eye‐tracking Parameters.

	Cronbach’s Alpha
Total gaze time on AOI	0.867
Total time to fixation on AOI	0.410
Mean number of fixations on AOI	0.830
Mean duration of first fixation on AOI	0.091

AOI = area of interest.

When comparing the written test, the expert group significantly outperformed the novice group. The novice group interpreted the FAST images less accurately compared to the expert group, with the novice group achieving a mean score of 55.0% ± 17.2% versus 82.0% ± 11.3% in the expert group (p < 0.005).

DISCUSSION

In this study, we found that eye tracking appears to be a valid instrument for differentiating novices from experts for interpreting FAST ultrasound video clips. The primary outcome of total gaze time on the AOI was significantly longer among experts compared with novices. In addition, we found that the total time to fixation and the mean number of fixations on the AOI were significantly different between the groups. The total gaze time and the mean number of fixations on the AOI demonstrated high reliability as demonstrated by Cronbach’s alpha. When comparing eye‐tracking metrics with other variables, we have also demonstrated a convergent measure of competency compared to a written examination.

The results of the study provide evidence that experts in the FAST application spend a longer time fixated on the AOI. This finding may be explained by several factors, including that they spend less time gazing outside the AOI, and spend a longer time in the highest‐yield location confirming their interpretation. Experts are also faster to fixate on the AOI compared to novices, possibly due to an ability to recognize patterns in anatomy. However, this parameter did not obtain an acceptable internal consistency and we postulate that there may be more than one construct being measured affecting the internal consistency. These results suggest that experts are more efficient in their gaze patterns with the ability to ignore visual distractions and irrelevant findings.

Our findings are consistent with other studies evaluating the use of eye‐tracking technology in competency assessment. In a study by Harvey et al., ¹³ eye‐tracking metrics were used to compare three highly experienced and seven less experienced surgeons while dissecting the recurrent laryngeal nerve in cadavers. They found that the more experienced surgeons fixated longer on structures of interest and devoted less attention to irrelevant visual stimuli. In a radiology study conducted by Wood et al., ¹⁴ eye tracking was used to asses 10 experts, 10 intermediates, and 10 novices while interpreting skeletal radiographs. They found that experts were faster to fixate on the site of a fracture compared to novices, underpinned by superior pattern recognition skills. Borg et al. ¹² tested the feasibility of using eye‐tracking to differentiate five novices from five experts in ultrasound‐guided regional anesthesia. They found that experts took less time to identify sonoanatomy and spent less unfocused time away from a target compared to novices. Our study expanded on existing literature, in that we used video clips instead of still images, and consisted of complex patient anatomy that included the liver, spleen, kidneys, and heart. When compared to the task of ultrasound static image interpretation, ultrasound videos require a significantly higher cognitive load.

The use of eye‐tracking technology for learner assessment is appealing for several reasons. First, it does not require an instructor to be present, which reduces the assessment burden on the faculty. Current methods for assessing competency in POCUS are time‐ and resource‐intensive. ²² International guidelines suggest that assessment involve a combination of direct observation, along with visual, written, and practical examinations. ⁶ This high‐resource requirement is currently a major barrier for many institutions for implementing comprehensive POCUS training programs. ²³ The results from this study demonstrate that eye‐tracking metrics may have a role in assessing learner progression and may help to reduce resource needs while at the same time ensuring that high‐quality standards are maintained.

Second, eye‐tracking technology may play a valuable role in providing feedback to learners during their training. In addition to the quantitative data provided by eye‐tracking technology, gaze patterns can also be assessed qualitatively through tools such as gaze mapping (Figure 3) or real‐time video review (Video S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10508/full). These qualitative data can be potentially helpful to an educator by providing insight on where the learner may require assistance. It provides a visualization to where the learner is fixating the vision and inference can be made with the information. For instance, a learner who has a gaze pattern that is fixating at the wrong area or searching a large area on the screen may suggest a weak understanding of sonoanatomy of that region, whereas a learner who is fixating at the correct AOI but provided a wrong response may require more knowledge on image interpretation.

Figure 3.

Visual representation of area of interest (left) and gaze patterns for the novice (middle) and expert (right) groups viewing the (A) right upper quadrant and (B) suprapubic region. Fixations are represented as color circles and duration represented by the size of the represented circle.

Finally, eye‐tracking technology may provide insights into the learning process, which may help to improve curriculum design. The analysis of gaze patterns may provide insight to educators on the cognitive processes of the learners and facilitate the refinement of the POCUS curriculum to enhance learning by adding feedback and reflection. ⁹ Eye tracking can be used as an early warning system to detect novice‐type gaze behavior, which will allow the faculty to take corrective actions earlier.

LIMITATIONS

This study was performed at a single academic health sciences center, and therefore, the results may not be generalizable to other populations. Convenience sampling was utilized and this may have resulted in selection bias due to the practice and behavior of the participants unique to this center. The expert group consisted of participants with varying skill levels ranging from standard ultrasound certification to fellowship training. Despite this, our results demonstrated that this heterogeneous group of experts could be distinguished from novices using eye‐tracking metrics. The sample size of this study was small, and future studies are needed that include a broader spectrum of skill levels and a larger study cohort. In addition, this study only tested image interpretation; therefore, we cannot comment the utility of eye tracking for assessing competency in image generation.

CONCLUSION

In this study, we determined that eye‐tracking technology was able to discriminate between expert and novice point‐of‐care ultrasound users when interpreting focused assessment using sonography for trauma examination videos. Specifically, the total gaze time and the mean number of fixations on the area of interest were significantly different between the two groups with a high level of internal consistency. Gaze pattern analysis may represent a useful tool allows educators to assess learning and determine competency of image interpretation in point‐of‐care ultrasound. It may also be incorporated into the point‐of‐care ultrasound curriculum as a way for learners to receive feedback. Future studies are needed to further validate this technology for competency in a larger cohort and a broader range of point‐of‐care ultrasound applications.

Supporting information

Video S1. Gaze pattern between a novice and an expert.

AEM Education and Training 2021;5:1–7