Image Annotation by Eye Tracking: Accuracy and Precision of Centerlines of Obstructed Small-Bowel Segments Placed Using Eye Trackers

Abstract

Small-bowel obstruction (SBO) is a common and important disease, for which machine learning tools have yet to be developed. Image annotation is a critical first step for development of such tools. This study assesses whether image annotation by eye tracking is sufficiently accurate and precise to serve as a first step in the development of machine learning tools for detection of SBO on CT. Seven subjects diagnosed with SBO by CT were included in the study. For each subject, an obstructed segment of bowel was chosen. Three observers annotated the centerline of the segment by manual fiducial placement and by visual fiducial placement using a Tobii 4c eye tracker. Each annotation was repeated three times. The distance between centerlines was calculated after alignment using dynamic time warping (DTW) and statistically compared to clinical thresholds for diagnosis of SBO. Intra-observer DTW distance between manual and visual centerlines was calculated as a measure of accuracy. These distances were 1.1 ± 0.2, 1.3 ± 0.4, and 1.8 ± 0.2 cm for the three observers and were less than 1.5 cm for two of three observers (P < 0.01). Intra- and inter-observer DTW distances between centerlines placed with each method were calculated as measures of precision. These distances were 0.6 ± 0.1 and 0.8 ± 0.2 cm for manual centerlines, 1.1 ± 0.4 and 1.9 ± 0.6 cm for visual centerlines, and were less than 3.0 cm in all cases (P < 0.01). Results suggest that eye tracking–based annotation is sufficiently accurate and precise for small-bowel centerline annotation for use in machine learning–based applications.

Introduction

The small bowel is the site of a wide range of acute and chronic diseases that are diagnosed by cross-sectional abdominal imaging. Small-bowel obstruction (SBO) is a common cause of acute abdominal pain and is the etiology identified in up to 20% of surgical admissions for that chief complaint [27, 32]. SBO is often evaluated with CT to help triage patients to management with either decompression with a nasogastric tube or surgical intervention. CT findings have been shown to predict the need for surgery more accurately than either clinical or laboratory parameters [31]. Prompt diagnosis of the cause and severity of SBO is essential, as delays in surgical intervention dramatically increase mortality to as high as 25% if ischemia is present [3, 27]. Many other important diseases also affect the small bowel. Crohn’s disease causes lifelong complications for afflicted patients, and along with ulcerative colitis affects approximately 1.3% of the entire US population [9]. MR enterography is a useful complement to ileocolonoscopy in its management [2, 5, 30].

On cross-sectional imaging, these diseases of the small bowel manifest as changes in its caliber, wall thickness, and enhancement characteristics, as well as changes of the adjacent mesentery. The location of findings along the linear course of the small bowel is important in evaluation as well. For instance, a transition point is the location along the linear course of the bowel at which its caliber narrows, and must be identified to determine the cause of an SBO. Two adjacent transition points suggest closed-loop SBO, a severe variant which can lead to ischemia and require surgical management. In the context of inflammatory bowel disease, correlation with endoscopy is only possible if the linear location of abnormalities is determined.

Automated methods to determine the course and characterize the appearance of the small bowel would thus have wide ranging benefits in cross-sectional abdominal imaging of SBO and other diseases. Such methods have been developed to facilitate analysis of other organ systems [7, 10, 12–14, 17–19, 23, 24, 37, 38, 40]. However, the small bowel is unique relative to the systems in which these methods have been successfully applied. Since it is a hollow viscus rather than a solid organ, its contents can vary due to ingested food, bowel preparation, or disease states. Since it is attached via a flexible mesentery, its course can vary dramatically due to positioning, peristaltic motion, congenital variants, and surgical alteration. Thus, at present, few published methods address these challenges in small bowel [33, 34, 42]. New automated methods that address these unique challenges will be required to facilitate prompt and accurate diagnosis of diseases like SBO.

Fortunately, new technologies may make it possible to address these challenges successfully [6, 38]. Machine learning has advanced rapidly in the last decade. In particular, convolutional neural networks (CNNs) running on powerful graphics processing units (GPUs) have proved superior to traditional methods of image analysis in several contexts [15, 16]. CNNs have been applied to both image classification and segmentation [28]. However, the vast amounts of annotated data required to successfully train CNNs poses a great limitation for their usage, due to the time consuming nature of these annotations.

Eye tracking systems provide a potential solution for more facile image annotation, as has been suggested in the context of annotation for object detection within two-dimensional images [26]. During interpretation of cross-sectional imaging, the radiologist uses a mouse to adjust the position within the volume and to adjust the window and level settings of the display. During image annotation, the analyst uses a mouse to contour regions of interest in the volume, but must concurrently make the same adjustments as during image interpretation. Using the same interface device for both tasks may limit the efficiency with which annotation can be performed. Using a separate interface device such as an eye tracker to record locations of interest on the screen may relieve this limitation, if the information it generates is of acceptable accuracy and precision. Regardless of whether eye trackers improve efficiency, the supplemental information which they provide may be of use in addition to manual annotations. For example, eye trackers have previously been found useful in radiology for research regarding pulmonary nodule assessment [8, 29, 35].

In this study, we investigate the use of eye trackers for annotation of centerlines of segments of obstructed small bowel on CT. We assess the accuracy of visually placed centerlines by comparison to manually placed centerlines using a dynamic time warping (DTW) algorithm. We assess the precision of visually placed centerlines on the basis of intra- and inter-observer variability over repeated annotations. As a benchmark for adequacy, we compare measures of accuracy and precision to clinical thresholds for the radius and diameter of obstructed small bowel respectively.

Materials and Methods

Subject Population

In this HIPAA compliant, IRB-approved study, images from seven subjects, 4 males and 3 females, with small-bowel obstruction diagnosed by CT were retrospectively included. Subjects ranged from 31 to 79 years old (mean 54 years). CT images were acquired using a 64 or 320 slice scanner, with 3.75 mm reconstructed slice thickness and fields of view ranging from 35–40 cm. Only axial CT images were used in the subsequent analysis. Each CT image was also modified to include a square region measuring 10 × 10 pixels with attenuation value of 1000 Hounsfield units (HU) centered 15 pixels from each corner as a reference marker.

Eye Tracking

The Tobii 4C Eye Tracker (Tobii Tech, Sweden) was used for this study, with license for analytical use. Eye trackers were mounted on each observer’s personal laptop and desktop computers, using 15.6-, 24-, and 29-in. screens. Each observer sat with their eyes positioned approximately 2 ft from the monitor. Room lights were left on and were not dimmed during annotation. Calibration of the eye trackers was performed with the standard routine available through the Tobii device driver, using one central point, then three peripheral points, and then another set of three peripheral points.

A custom 3D Slicer module was developed to interface with the eye tracker using the Tobii Research Software Development Kit Python API. The module recorded points on the screen at which the observer was looking while scrolling through a cross-sectional imaging volume. Each gaze point recorded by the module was used to instantiate a fiducial point in 3D Slicer, by transforming the screen coordinates of the gaze point to the coordinate system of the CT scan based on the geometry of the computer window and of the imaging slice. Fiducial locations were exported into a text file for further analysis.

The Tobii 4C eye tracker records gaze points at a rate of 90 Hz; however, the version of the module used in this study was only able to record at a rate of 10 Hz due to limitations related to multi-threading in Python. Subsequent versions of the module have relieved these limitations to enable recording at the full 90 Hz, but were not available at the time of this study.

Although accuracy and precision of angular eyetracking measurements is available through the Tobii eye tracking software and Tobii product literature, recapitulating these measurements was not within the scope of this study, since accuracy and precision in the imaging domain are more relevant to utility in the context of image annotation. The exact size of the CT images on the screen during image annotation was variable, since observers were allowed to magnify the images as needed for optimal visualization, and for that reason was not recorded.

Image Annotation

A single continuous segment of obstructed small bowel in each CT was chosen for analysis based on its subjective prominence by a radiologist with 6 years of experience. Using 3D Slicer and a standard computer mouse, this radiologist manually placed fiducial points to designate the beginning and end of each segment and subsequently placed a series of fiducial points along the approximate centerline of the entire segment. Two other observers studied this centerline, and afterwards, manually placed fiducials points along the centerline of the segment as well. These two observers were not blinded to the expert annotation due to their level of training, since the purpose of this study was to assess variance arising from technique rather than from interpretation (Fig. 1).

Fig. 1.

Diagramatic representation of the number of centerline annotations performed

Next, using the custom 3D Slicer module and eye tracker, all three observers were instructed to look at the center of the bowel while scrolling from the beginning to the end of the segment. Fiducial points were placed by the module at locations corresponding to gaze points recorded by the eye tracker during this visual task. In addition, at the beginning and end of the segment, each observer was instructed to look at the reference markers in each of the four corners of the image. Using the same custom 3D Slicer module, gaze points were recorded during this visual task as well and were used to instantiate fiducial points at each of the eight corners of the volume. Each observer repeated manual and visual annotations three times, for a total of 126 annotations across 7 subjects, 3 observers, 2 methods, and 3 repetitions as shown in Fig. 2.

Fig. 2.

Representative image of visually placed centerline in an obstructed small-bowel segment. Different colors represent different observers

The average lengths of all annotated segments were 82.4 ± 22.4 cm. The average distance between adjacent centerline points was 1.43 ± 0.37 cm for manual annotations, and 0.59 ± 0.18 cm for visual annotations. Finally, the average number of placed centerline points was 42 ± 13 for manual annotations and 199 ± 65 for visual annotations (Fig. 3).

Fig. 3.

Representative image of visually placed corner points in the beginning and end slice. Different colors represent different observers. White circles represent a 1.5 cm boundary around the center of the marker

Dynamic Time Warping

Dynamic time warping (DTW) is an algorithm for aligning two sequences with unequal numbers of ordered elements [1, 36], for instance, two trajectories containing different numbers of points along a small-bowel centerline as in this application. Since there are a large number of potential pairings between all points of each trajectory, DTW applies two constraints to determine the alignment [20]. First, the order of points in each pairing is non-decreasing, in keeping with the notion that two trajectories are time series, and that progress along one should reflect progress along another. Second, the total cost of all pairings is minimized. In this application, the cost of each individual pairing is defined to be the Euclidean distance between pairs of points. Minimizing the total cost ensures that the pairings represent the closest possible pairings of points subject to the first constraint. The average cost over all pairs of points can then be interpreted as the distance between trajectories.

The details of our implementation of the DTW algorithm are as follows. In an analogous formulation to Berndt et. al. [4], we define two three-dimensional trajectories S and T with m and n points, as shown in Eq. 1.

$$\begin{array}{rcl}S& =& (x^1,y^1,z^1),(x^2,y^2,z^2)\mathrm{...},(x^m,y^m,z^m),S^i=(x^i,y^i,z^i)\\ T& =& (x^1,y^1,z^1),(x^2,y^2,z^2)\mathrm{...},(x^n,y^n,z^n),T^j=(x^j,y^j,z^j)\end{array}$$ 1

The cost is defined as the three-dimensional Euclidean distances between points, as in Eq. 2.

$$\begin{array}{rcl}\delta (S^i,T^j)& =& \sqrt{{\left(x_S^i-x_T^j\right)}^2+{\left(y_S^i-y_T^j\right)}^2+{\left(z_S^i-z_T^j\right)}^2}\\ \mathit{for}i& =& 1,\mathrm{...},m\mathit{and}j=1,\mathrm{...},n\end{array}$$ 2

The DTW algorithm constructs a m × n matrix DTW through dynamic programming as shown in Algorithm 1. A step between an element of the matrix and an adjacent prior element represents a pairing of points on each trajectory. For each element, steps from three prior adjacent elements must be considered. Of these three possibilities, the step with the minimum sum of the cost of the prior element and cost of the step from that prior element is chosen. This minimum summed cost is recorded as DTW[i,j], and the step is recorded as part of a path through the matrix to that element. Thus, each element DTW[i,j] represents the minimal total cost of the pairings of subsets of each trajectory S_1...i and T_1...j. After the entire matrix is constructed, the value of DTW[m,n] represents the minimum total cost of pairings between the entire trajectories S and T, and the path through the matrix represents the pairings themselves.

Using this algorithm, DTW distances between pairs of centerlines were calculated for each subject as shown in Fig. 4. Average DTW distances were used as a metric of similarity between manually and visually annotated centerlines for assessment of accuracy, and between repetitions of annotations for assessment of precision.

Fig. 4.

Example of a DTW alignment between two visual annotations of a small bowel segment done by separate observers. The black lines match the corresponding points in each trajectory according to the DTW algorithm

Centerline and Corner Point Filtering

Since small-bowel centerlines and corner points were recorded in the same visual annotation session, they were first filtered to allow for separate analysis. Points were considered corner points if they met two criteria. First, they were in the same axial slice as the designated beginning or end of the bowel segment, but were closer to the reference markers within that slice. This criteria corresponds to the instructions given to observers during visual annotation. Second, they were within 3.3 cm of the reference markers. This criteria excludes a small number of stray points were felt to represent unintentionally recorded saccades between corners. The percentage of such points removed from the analysis across all cases and observers was 2.3 ± 3.1%, with a maximum of 6.7%. The distribution of distances between filtered corner points and reference markers is shown in Fig. 7. Since this distribution does not show any abrupt cut-off, we felt these criteria captured most gazes that were intended to be directed towards the reference markers. The remainder of the points were considered small-bowel centerline points.

Fig. 7.

Histogram representation of the accuracy and precision analysis of the visually annotated corner points. Panel a represents the distribution of the radial distances between visually annotated corner points and the corner markers. Dashed red line represents 1.5 cm. Panel b represents the distribution of distances between visually annotated corner markers in the first and last CT slice of the small-bowel segments. Dashed line represents 3.0 cm

Centerline Analysis

To assess the accuracy of the visual annotations, the average DTW distance was computed between visual and manual annotations for all repetitions for a given subject. This generated a total of 9 distances per subject for each observer, and a total of 189 distances across all subjects and observers.

To assess the precision of both manual and visual annotations, intra- and inter-observer variability was assessed. For intra-observer variability, the average DTW distance was calculated between repetitions performed for a given subject by a single observer. This was done separately for annotations from manual and visual annotations. This generated 3 distances per subject for each observer, and a total of 63 distances per method across all subjects and observers. For inter-observer variability, the average DTW distance was calculated between repetitions performed by all observers for a given subject. This generated 18 distances per subject, and a total of 126 distances per method over all subjects.

Corner Point Analysis

Corner points were placed to evaluate visual annotations in the context of a fixed rather than an interpreted structure. To assess accuracy, the two-dimensional Euclidean distance between each corner point and its corresponding reference marker was calculated for all subjects and observers. To assess precision, the two-dimensional distance between each corner point in the beginning slice and the projection of the corresponding corner point from the end slice was calculated for all subjects and observers.

Statistical Analysis

To determine whether visual annotation would be sufficient in the clinical context of small-bowel obstruction, DTW distances were compared to one of two fixed thresholds, 1.5 cm for accuracy and 3.0 cm for precision. These thresholds were chosen based on the caliber at which small bowel is considered dilated. This caliber varies according to clinical practice, but a threshold of 3-cm diameter is commonly used [11]. Thus, a centerline placed within a 1.5 cm radius of the true centerline would be expected to be intraluminal. Likewise, two centerlines placed within 3.0 cm of each other both would also be expected to be intraluminal.

Each of the sets of DTW distances in the above sections were statistically compared to these fixed thresholds for accuracy or precision using a one-sample one-tailed t test, with the null hypothesis that means were above the fixed thresholds. For centerline accuracy analysis, four t tests were performed, one for each of the three observers, and one for all observers in aggregate. For centerline precision analysis, four t tests were performed, for manual and visual methods, and for intra- and inter-observer comparisons. For corner point analysis, two t tests were performed, one for accuracy and one for precision.

Euclidean distances often follow a chi-squared rather than normal distribution. However, the robustness of the t test allows for deviations from normality to be acceptable as long as the number of samples is sufficiently large. This is the case because it is the mean of the random samples of the population that must be normally distributed, not the population itself. For a large number of samples, the central limit theorem ensures this is the case [41].

Results

Centerline Analysis

To assess accuracy, DTW distances between manual and visual annotations of small-bowel centerlines were calculated and are shown in Fig. 5. For each observer individually, DTW distances were 1.1 ± 0.2 cm, 1.3 ± 0.4 cm, and 1.8 ± 0.2 cm and were less than 1.5 cm for two of three observers (P < 0.01). For all observers in aggregate, DTW distances were 1.40 ± 0.43 cm, which was also less than 1.5 cm (P < 0.01).

Fig. 5.

Accuracy as represented by the average DTW distance of the visually annotated centerlines to the manually annotated centerlines. Each boxplot represents all annotations done by a single observer. The dashed line represents the 1.5 cm distance specified as acceptable for accuracy. Mean ± standard deviations as well as P values shown below. P values correspond to statistical difference between data and the 1.5 cm distance

To assess precision, intra- and inter-observer DTW distances among repetitions of manual and visual annotations of small-bowel centerlines were calculated and are shown in Fig. 6. Intra-observer DTW distances for manual and visual annotations were 0.6 ± 0.1 cm and 1.1 ± 0.4 cm, respectively. Inter-observer DTW distances for manual and visual annotations were 0.8 ± 0.2 cm and 1.9 ± 0.9 cm, respectively. All of these distances were significantly lower than 3.0 cm (P < 0.01).

Fig. 6.

Intra- and inter-observer variability for manual and visual centerline annotations as a metric for precision. Each boxplot represents data from all cases and observers for the two annotation types. The dashed line represents the 3.0 cm distance specified as acceptable for precision. Mean ± standard deviations as well as p-values shown below. P values correspond to statistical difference between data and the 3.0 cm distance

Corner Point Analysis

To assess accuracy, 2D Euclidean distances between visually annotated corner points and reference markers were calculated and are shown in Fig. 7a. These distances were 1.51 ± 0.77 cm. Although the peak of this distribution was below 1.5 cm, its mean was not significantly below 1.5 cm (P > 0.05).

To assess precision, 2D Euclidean distances between corresponding projected corner points from the beginning and end slice were calculated and are shown in Fig. 7b. These distances were 1.10 ± 1.00 cm, which were significantly below 3.0 cm (P < 0.01).

Discussion

Machine learning tools require annotated data for training and validation. Manual annotation of medical images can be a time-consuming process. The inefficiency of manual annotation is partly due to the use of the same device for both annotation and for image adjustments that must be done concurrently with annotation. Eye trackers are an alternative to the standard computer mouse. They provide information about the position on the screen at which a user is looking. Since this information is derived from the user’s eye rather than hand, the mouse can be used for other tasks concurrently. In this study, we developed a custom 3D Slicer module that to facilitate this novel division of labor between eye and hand. The module uses an eye tracker to perform visual annotations, while allowing the mouse to be used for manual image adjustments such as repositioning within the imaging volume. We investigated the performance of this system in the clinical context of small-bowel obstruction, a common and important disease.

The results of this study suggest that visual annotation of the small-bowel centerline is sufficiently accurate and precise relative to clinical thresholds for diagnosis of obstruction. Visually placed centerlines were within 1.5 cm of manually placed centerlines for all observers in aggregate, and for two of three observers individually. Since 1.5 cm is the approximate radius of obstructed bowel, we reason that this level of accuracy is sufficient for both centerlines to be intraluminal. Likewise, visually placed centerlines were well within 3.0 cm of each other when repeated placement was performed. Since the variability of visual placement was greater than that of manual placement, the imprecision likely reflects both the performance of the eye tracker as well as differing interpretations of the location of the centerline. Nonetheless, since 3.0 cm is the approximate diameter of obstructed bowel, we reason that this level of precision is still sufficient for all centerlines to be intraluminal.

We also investigated the performance of visual annotation of reference markers placed at the corners of the imaging volume. We found that visual placement of corner points was precise to well within 3.0 cm, but was not accurate to within the specified threshold of 1.5 cm. Since the position of the reference markers was not a matter of interpretation, the inaccuracy likely reflects inadequate calibration of the eye tracker. Accuracy was noted to vary between reference markers in different corners of the volume (results not shown), which also suggests inadequate calibration. Nonetheless, the high degree of precision suggests that improved calibration and thus improved accuracy, may be possible in the future.

Radial distance between centerlines is the quantity that is most relevant to determining whether both are intraluminal. DTW distance incorporates both axial and radial offsets, since it is a sum of Euclidean distances between points without regard to the orientation of the centerline. For that reason, DTW distance is an upper bound on the radial offset, since any axial offset will also be included in the measurement. Thus, if the average DTW distance is less than the specified thresholds of 1.5 cm and 3 cm, the average radial distance would be expected to satisfy those thresholds as well. We considered using other metrics, based on distances between line segments or continuous representations, rather than distances between points, to better capture radial distance only. However, other representations require parameterization or processing that adds complexity, and at some point require discretization even if only in an integral. For this reason, we selected DTW distance as it was the simplest metric that still provided useful information about radial distance.

This study has several limitations. First, many technical factors varied among observers during visual annotations, including the computer monitor used, the screen size of the 3D Slicer application, and the level of magnification of the displayed CT images. We did not investigate the effects of these factors on accuracy and precision, but expect that they could affect both. Second, we did not investigate the observer-specific performance seen for many measurements, which may arise from the observers’ varying level of training in radiology, but also could potentially arise from their use of eyeglasses, facial features, or other factors. Third, we did not record the time required for either manual or visual annotation. Although we suspect visual annotation may prove more efficient, we did not directly evaluate that claim in this study. Fourth, the exclusion of non-dilated segments limits the applicability of the results to normal caliber bowel and populations without bowel obstruction. Finally, the small size of the dataset included in this study limits the applicability of the results to larger studies, in which additional factors such as observer fatigue may also be sources of error.

We plan to pursue several future directions motivated by the findings of this study. First, to address the inaccuracy observed in the results, other calibration techniques will be investigated. One such technique would be to use a grid of points across the screen rather than just the few that are shown during the off-the-shelf calibration routine. Such a technique may leverage the precision to improve the accuracy of the results. Second, to address the question of whether visual annotation is more efficient than manual annotation, we will develop an improved module to record the time taken for each annotation alongside the annotation itself. Third, we will investigate whether eye trackers can be used to annotate other linear or branching structures in the human body, such as blood vessels or bile ducts [21, 22], or other volumetric structures like organs or lesions [25, 39]. Finally, we will investigate how to incorporate these visual annotations into training data for development of machine learning tools. One such approach would be to label the region around each centerline as belonging to bowel, and use those labels to train machine learning tools to perform segmentation of the bowel. The development of such tools relies on large volumes of training data, the creation of which may be accelerated by visual annotation technique described here.

Conclusion

Small-bowel obstruction is a common and important disease. Development of machine learning tools for detection and characterization of small-bowel obstruction will require a large volume of annotated data. Using a Tobii eye tracker and custom 3D Slicer module, we found that visual annotations of the small-bowel centerline were of sufficient accuracy and precision relative to clinically derived thresholds for obstruction. These promising results suggest that visual annotation may help to generate the large volumes of training data needed for machine learning tools in the context of small-bowel obstruction.

Funding information

Dr. Kang Wang was supported in part by the NIH grant T32EB005970.