Visual search in breast imaging

Abstract

Breast cancer is the most common cancer among females worldwide and large volumes of breast images are produced and interpreted annually. As long as radiologists interpret these images, the diagnostic accuracy will be limited by human factors and both false-positive and false-negative errors might occur. By understanding visual search in breast images, we may be able to identify causes of diagnostic errors, find ways to reduce them, and also provide a better education to radiology residents. Many visual search studies in breast radiology have been devoted to mammography. These studies showed that 70% of missed lesions on mammograms attract radiologists’ visual attention and that a plethora of different reasons, such as satisfaction of search, incorrect background sampling, and incorrect first impression can cause diagnostic errors in the interpretation of mammograms. Recently, highly accurate tools, which rely on both eye-tracking data and the content of the mammogram, have been proposed to provide feedback to the radiologists. Improving these tools and determining the optimal pathway to integrate them in the radiology workflow could be a possible line of future research. Moreover, in the past few years deep learning has led to improving diagnostic accuracy of computerized diagnostic tools and visual search studies will be required to understand how radiologists interact with the prompts from these tools, and to identify the best way to utilize them. Visual search in other breast imaging modalities, such as breast ultrasound and digital breast tomosynthesis, have so far received less attention, probably due to associated complexities of eye-tracking monitoring and analysing the data. For example, in digital breast tomosynthesis, scrolling through the image results in longer trials, adds a new factor to the study’s complexity and makes calculation of gaze parameters more difficult. However, considering the wide utilization of three-dimensional imaging modalities, more visual search studies involving reading stack-view examinations are required in the future. To conclude, in the past few decades visual search studies provided extensive understanding about underlying reasons for diagnostic errors in breast radiology and characterized differences between experts’ and novices’ visual search patterns. Further visual search studies are required to investigate radiologists’ interaction with relatively newer imaging modalities and artificial intelligence tools.

Introduction

Visual search plays a crucial role in our daily life and involves easy tasks such as finding our tea cup on a table or difficult ones which require certain level of experience and training. Among the visual search tasks, those conducted by health professionals are among the most complex and differ from other tasks in various aspects.¹ First, the cost of error caused by incorrect medical image interpretation is considerably high compared to our everyday tasks. Secondly, in medical images, the nature of the target and surrounding distractors differ from simple search tasks. When dealing with medical images, the number of targets and target categories might be unknown or unlimited and targets might be hard to find or identify. In some cases, such as screening mammography, low target prevalence might change radiologists’ visual search patterns. Moreover, most of the traditional visual search models, which successfully describe simple visual tasks, do not consider observer’s knowledge or experience, which are essential in medical image interpretation.² Basically, medical image interpretation is comprised of two processes: visual inspection of image (visual perception) and rendering a diagnostic decision. In both processes, efficiency and accuracy highly depend on the observer’s expertise.³

As breast cancer is the most common cancer among females, a large volume of breast images is interpreted each year. This highlights why visual search studies focused on breast imaging have been necessary. Although most of these studies focused on mammographic image interpretation, visual search in other imaging modalities such as digital breast tomosynthesis (DBT) or breast ultrasound have also been investigated. Mammography is the primary imaging modality in the investigation of breast changes.⁴ Breast ultrasound, DBT, and breast MRI are also widely used to detect breast cancers in high risk females and for complete evaluation of breast symptoms.⁴ Other breast imaging modalities include CT, positron emission tomography (PET), electrical impedance-based imaging, scintimammography, and optical imaging.⁴ Although, at a fundamental level, the interpretation of all different image modalities shares similarities in terms of involving perceptual and cognitive processes,³ visual search in each modality has its own unique challenges and properties. This article briefly reviews visual search studies done in different imaging modalities and identifies the questions still to be answered to address gaps in knowledge. To understand medical image interpretation procedures, scientists have widely used eye tracking technology. Therefore, first we briefly describe how eye tracking data is usually analyzed.

Eye tracking technology

Eye tracking technology is an important tool to understand medical image interpretation procedures. Eye trackers are sensors that show exactly where the eyes are focused and measure eye movements. Eye tracking data are often analyzed in terms of fixations and saccades. Fixations represent those times when the observer’s eyes pause over an area and they use the central foveal vision to obtain detailed information about the area. Saccades are rapid, often ballistic movements between fixations, which reposition the eyes. An important step done by most of the eye tracking software applications is converting raw eye position data to a sequence of fixations. Raw eye position data contain gaze points representing the instantaneous spatial locations [an (x,y) coordinate] and a timestamp corresponding to each measurement of location. The sampling frequency of the eye tracking device determines how the raw data are spaced temporally. For example, at 500 Hz, time stamps are spaced 2 ms (1/500 s) apart. A fixation contains multiple gaze points and hence has duration, start and end timestamps in addition to a spatial location. There are different algorithms to translate the raw sequence of gaze points into an associated sequence of fixations, but basically they group spatially and temporally adjacent gaze points into a single a fixation. Figure 1 represents a sample eye tracking data, which indicates fixations over a mammogram (red circles) and eye movement trajectories (red lines). The represented data are simulated data.

Figure 1. .

(a) Example of mammographic scanpath (simulated signal); each circle represents a fixation. (b) corresponding clusters of fixations based on spatial information and discarding temporal information; each colour represents centre of one of the clusters. Clusters are randomly colour-coded in a way to ensure that neighbouring clusters do not have similar colours. The original fixation points were plotted in light grey. (c) Corresponding clusters of fixations based on spatial and temporal information; the centre of clusters are randomly colour-coded. The numbers indicate the total dwell time of each fixation cluster. As indicated, the cluster shown using an arrow had the shortest duration (0.13 s). (d) Corresponding eye-tracking focus map (the intensity of green colour shows the total dwell time on an area).

After collecting eye tracking data, nearby fixations are usually grouped together to generate fixation clusters.⁵ This can be done based on spatial proximity (falling within certain distance of the centre, discarding temporal adjacency) of fixations or both temporal and spatial adjacency.⁵ Commonly extracted gaze parameters from clusters include total dwell time on each cluster, time to hit the cluster (first fixate on the cluster), number of fixations per cluster, maximum number of consecutive fixations on cluster, and saccade amplitude of the first fixation in a cluster (i.e. the angular distance that the eye travels when fixating on a cluster for the first time).^6–9 Total dwell time of each fixation cluster is also shown in Figure 1 (c). Another common way to represent fixation clusters is using eye-tracking focus maps or heatmaps, which indicate aggregated dwell time in each cluster and are usually projected back onto the original image. As shown in Figure 1 (c), the duration of fixations in the far left (showed using an arrow on 1 (c)) was only 130 ms. That one fixation does not form a “cluster”, therefore that little fixation on the left of Figure 1(a) is negligible.

Mammography

Findings from previous studies

Mammography is the most common breast imaging modality and large volumes of mammographic images are produced each year.⁴ Therefore, several visual search studies have been conducted devoted to characterizing radiologists’ mammographic scanpaths in order to provide a deeper insight into an effective search strategy (which will likely lead to correct diagnoses) and to determining the factors that influence the radiological visual search.² Success in characterizing visual search patterns that lead to errors can open opportunities for utilizing eye-tracking data to prevent diagnostic errors⁷ and improve radiology educational programs.

Modelling radiologists’ visual search pattern

At a fundamental level, directed search and free search can be considered as two possible search strategies for interpreting a medical image. Directed search refers to a search strategy that follows a preconceived plan (e.g. the radiologist has a specific pattern for investigating a chest radiograph, starting in the left upper lobe of the lungs), while free search is inspecting the image without a preconceived idea of what it may depict. More than four decades ago, early eye-tracking experiments suggested that the orderly approach for visual search instructed in radiology textbooks is not reflected in the eye-movement behaviours observed in practice, and both experienced and less-experienced radiologists use free search rather than directed search.¹⁰ When an image is abnormal, radiologists fixate promptly on the abnormality and neglect relatively large areas of image. Indeed, radiologists can identify abnormal images based on a momentary glimpse of image before detailed fixations at any location.¹¹ This highlights the importance of the holistic processing of a medical image, which starts immediately after image onset.

Different theoretical frameworks, including the two-state detection model,¹² the global-focal search model,¹³ and the holistic model,¹⁴ have included holistic or global processing as a core element characterizing the underlying mechanisms of medical image perception. According to the global-focal search model (Figure 2), immediately after the image onset, the scene is globally analysed with low resolution peripheral vision (generating a “global impression”) and the perturbations in the image are identified. Then, gaze is directed to the perturbations, and they are analyzed by using the high-resolution central vision (“fixation checking”). After making a covert decision about the perturbation, the gaze is either directed to another suspicious location based on information from the global impression step or the radiologist starts “discovery scanning” of the image. Discovery scanning may be interrupted by checking fixations if the observer identifies perturbations based on peripheral vision. Meanwhile, the observer may revisit the perturbations several times to compare them with normal tissue or to make difficult decisions about ambiguous features in the image. Although the global-focal search model initially described visual search for pulmonary nodules,¹⁵ later studies indicated its success in describing mammographic visual search.^14,16 For example, the distribution of time to hit the lesion (indicator of the initial perception of cancer) was estimated by a mixture of two Gaussian distributions, the first one representing lesion perception happening as a result of the global impression and the second one representing lesion discovery during subsequent fixations.¹⁷ It has been shown that the true-positive fraction of the first group (the lesions hit shortly after image onset) was large for most readers, and concluded that this initial detection occurs as a result of parallel global analysis of the image.^14,17

Figure 2. .

Steps of global-focal search model. The figure is adopted from the diagram provided in Nodine and Kundel.¹³ As shown, visual search is modelled as a serial process, which starts with capturing the global impression which is based on prior knowledge. Once the perturbations in the image are identified (as shown by the rightmost arrows), they can be either verified foveally, or used as input for Discovery or Reflective Search to specify a set of potential areas or features to be fixated. Information collected by the detailed inspection using the foveal vision combined with prior knowledge can lead to the final decision for a case. Discovery Search refers to coarse screening of the image in order to detect potential target features, while Reflective Search aims at building up enough visual evidence to make a decision on an ambiguous target through detailed image screening. In experimental conditions where images are presented for a limited time, radiologists are required to make their final decision when the image is no longer presented based on “Post-Search Recall”. As shown, Global and Focal steps can be done recursively.

The two-stage detection model¹² is very similar to the global-focal search model. According to this model, expertise development in medical image perception involves developing an initial pre-attentive filter that automatically detects features that need further inspection and filtering out normal anatomical structures, in order to rapidly direct the expert’s attention toward regions of the image that contain probable abnormalities. Thus, both global-focal and two-stage detection models suggest that experts can process large areas of an image using their peripheral vision and rapidly identify potential abnormal areas, which should be subsequently inspected using foveal vision. The main difference between the global-focal search model and the two-stage detection model is the assumption that the two stages (i.e. global and local processing) can be done recursively.

As presented in Figure 3, the holistic model is concordant with the two-stage detection and global-focal search models in terms of including holistic and local processing pathways. It suggests that expertise development involves a shift in the perception mechanism from a relatively slow search-to-find mode to a fast holistic mode. Unlike the two earlier models, which assume that global and local processing are two serial stages, the holistic model assumes that the holistic processing can work in parallel with the search-to-find mode. This aligns with several other models, trying to conceptualize the holistic processing in other visual search domains,¹⁸ including the view from the basic visual science, where visual search is explained by a selective and a non-selective pathways. Based on these models, in the non-selective pathway, information can be extracted from global image properties.¹⁹

Figure 3. .

Steps of the holistic processing model. As shown, visual search starts with capturing the global impression and it is followed by two parallel processes.

The holistic model assumes that radiologists’ first global impression, or “gist” signal, arising from the mammogram, has a local source of information within the image, i.e. a perturbation or an abnormality. However, recent findings questioned this assumption. First, Evans et al²⁰ showed that after a half-second presentation of mammograms, radiologists can distinguish normal from abnormal at above-chance levels but they perform at chance-level in the abnormality localization. More recently Evans et al suggested that radiologists can identify abnormal cases from normal ones based on a half-second glimpse of normal breasts contralateral to malignancies.²¹ Moreover, radiologists performed at above-chance level in differentiating normal prior mammograms belonging to females who were diagnosed with breast cancer in a subsequent screening.²² The fact that both prior mammograms and contralateral mammograms contain no apparent visible signs of cancer suggests that radiologists’ first impression might not have a localized source, presenting a perturbation in the image, but depend on global image statistics suggestive of an abnormal mammographic texture.^20–22 The importance of these new findings about the gist signal is two-fold. First, these findings suggest that modifying the holistic processing model might be required, as it currently assumes that holistic processing provides scene structural information that directs eye movement to areas that could contain items of interest. However, recent experiments showed holistic processing might also involve a global process, which roughly classifies images as, e.g. cancer or normal.²³ Secondly, the gist signal can be used to predict a subsequent breast cancer incidence.²⁴

Reader’s characteristics and visual search

Although models for medical image interpretation aim at characterizing mammographic scanpaths, eye tracking of radiologists indicates differences in scanpaths and areas that attracted viewers’ attention. Extensive studies have been undertaken to understand how reader’s characteristics, especially their expertise level, affect mammographic scanpaths.

The holistic model characterizes expertise development as a shift in the recognition mechanism from “scan-look-detect” to “look-detect-scan”.¹⁷ This shift makes experts’ scanpaths more efficient as they are better at using the peripheral vision to process large image areas in parallel. Figure 4 (a) , (b) show two fixation sequences, one corresponding to a highly experienced radiologist and one corresponding to a junior radiologist, who just completed breast imaging fellowship. As shown, the two sequences differ considerably. The holistic model can successfully explain several observed differences between the experts’ mammographic scanpaths and those of less-experienced readers. Based on this model, the experts should fixate on the lesion earlier as they first “detect” the abnormality and then “scan” the remaining of the image. Based on the holistic model, experts developed skills process large areas of a mammogram using their peripheral vision, therefore the model expects that experts are more likely to detect potential abnormalities and first fixate on lesions quicker with less image coverage compared to less-experienced readers.

Figure 4. .

Fixation sequences of (a) a highly experienced radiologist and (b) a junior radiologist, while inspecting a normal case.

Congruent with the holistic model, compared to less-experienced readers, experts are more accurate²⁵ while spending less time on each image before making their final decision,¹⁶ first fixate on the lesion earlier during their search,²⁶ cover relatively smaller areas of the image with foveal fixations,²⁶ have fewer fixations with longer saccades,²⁷ and have a smaller number of fixations per saccades.²⁶ On average, experts have longer dwell times on false-negatives (FNs) while inexperienced observers have longer dwell times on true-positives (TPs), false-positives (FPs), and true-negatives (TNs).²⁸ Analysis of the speed–accuracy relationship showed that for the least experienced group, overall performance decreased below the chance level when total decision-making time was above 60 s.²⁹ On average, experts have fewer comparison scans between left and right mammograms compared to inexperienced readers.³⁰ Experienced radiologists also refixate less often on a previously fixated location and mostly use refixating, laminar (consecutive fixations in a small neighbourhood representing detailed scanning of an area within the image) and deterministic eye movements (repeated short sequences of fixations, typically looking back and forth between two locations for comparison) in the areas containing lesions, thus yielding a more efficient search compared to less experienced readers.³¹ Experts' scanpaths show a trend toward a small number of diagnostically relevant areas while less‐experienced readers’ scanpaths are more random and chaotic.³¹ Readers’ level of experience is also an independent factor in predicting complexity of visual search behaviour as measured by fractal dimension of mammographic scanpaths.³² Experts show that a greater proportion of time is spent fixating on diagnostically relevant areas,³³ and greater consistency in their scanpaths in terms of fixated areas.³⁴ Moreover, a spatial frequency analysis of fixated regions indicated that less experienced readers’ search strategies are more significantly affected by the local saliency of the lesions³⁵ when compared to those of experts.

When compared with radiologists, in addition to experience, a naïve observer lacks training in mammogram interpretation. Nodine et al²⁶ explored whether training alone, without extensive experience, can yield the experts’ efficient visual search strategies. Their results suggested that even after training (i.e. one or more mammography rotations), less experienced readers lacked the ability to efficiently dismiss normal areas with lesion-like features and failed to confirm the presence of a lesion in the second view of the mammogram.²⁶ It was suggested in a later study²⁹ that fine-tuned visual recognition skills of radiology trainees might be developed by perceptual-learning, where mentor-guided immediate feedback during image interpretation is provided to trainees.

Showing responses from computer-aided detection (CAD) tools on mammograms could change radiologists’ visual search strategies, as CAD’s input could direct radiologists’ attention to certain suspicious areas annotated by CAD. Krupinski²⁸ showed that experts and less-experienced readers interact differently with input from CAD tools. For example, the search strategy of less-experienced observers seemed to be different if CAD was available, while the experts searched the image as if there were no CAD prompts. These findings suggested that experts spent more time searching the mammogram before inspecting the area marked by the CAD system. On the other hand, the dwell time on FNs increased after inspecting CAD responses for less experienced observers and became longer than that of experts.²⁸

Further studies of radiologists’ eye tracking data suggested that there are individual aspects associated with mammographic scanpaths that are not explained simply by differences in expertise level. Alamudun et al³² showed that the individual differences among radiologist are independent factors in predicting complexity of visual search behaviour as measured by fractal dimension. Also, in Gandomkar et al,³¹ it was shown that each reader’s scanpath exhibits a unique spatiotemporal pattern that can be successfully differentiated from other readers’ scanpaths by feeding quantitative gaze-related parameters to a machine-learning classifier. When a reader was more experienced, the accuracy of the classifier in differentiating his/her scanpaths from others’ scanpaths was substantially higher compared to when a less-experienced reader was considered. This implies that experienced radiologists more consistently exhibit their unique spatiotemporal dynamics across all images compared to less-experienced radiologists.

Case properties and visual search pattern

Alamudun et al³² showed that breast density and case pathology (normal, benign, or malignant) are independent predictors of visual search complexity. Also, the spatiotemporal dynamics of fixations, as measured by recurrence quantification analysis, depend on the breast density and presence of a cancer.³¹ The observed dependency was stronger for the experienced radiologists who had a more efficient visual search and were more responsive to the presence of the target (i.e. dense tissue and lesion).³¹ This suggests that the extent of the effects of case properties on the visual search pattern of radiologists could depend on readers’ characteristics. For example, lesion conspicuity^36,37 and subtlety³⁸ are associated with significantly different visual search parameters than obvious lesions. Taken together these findings suggest that technological advances to improve lesion visibility and supress dense tissue may have potential in reducing detection errors.

Relationship between visual search and errors in mammogram interpretation

Interpretation of mammograms is a challenging task and subject to both FN and FP errors. Nodine and Kundel¹³ divided FN errors into three sub types: search errors (lesions were not fixated at all during visual search), recognition errors (lesions attracted radiologist’s attention to some extent but failed to exceed the minimum time needed for perception of an abnormality) and decision-making errors (lesions attracted prolonged attention but were finally interpreted as a normal or benign structures).

The distribution of these different types of errors depends on the reader’s experience level. The percentage of search, recognition, and decision-making errors made by experts in the interpretation of mammograms was 24%, 24%, and 52%, respectively, while corresponding percentages were 29%, 42%, and 29% for radiology trainees.²⁸ Recognition error occurs in the perception step and could be associated with many psychophysiological factors such as lesion’s conspicuity.^36,37 Decision-making errors also relate to various factors like lack of expertise,²⁵ bias from the first fixation³⁹ or incorrect background sampling.⁴⁰

Beside the classification of errors, eye tracking technology helped in understanding underlying causes of diagnostic errors and the characteristics of visual search patterns, which relate to a greater likelihood of making errors while reading mammograms.⁴¹ One of these characteristics is the total time to make a decision on a case. Using an eye-tracker, Nodine et al⁴² analyzed the time course of searching mammograms for suspicious lesions. The analysis showed that TPs were detected faster than FPs during the rapid-phase of visual search (the initial 25 s of case reading for breast radiologists and 40 s for trainees) but after that the FPs continue to increase relative to the TPs. The findings suggested that observers should terminate the visual search when they no longer make a high-confidence decision, as prolonging the search at this point has a low probability of yielding a TP and results in FPs.⁴²

The characteristics of background samples, which a reader uses for carrying out comparisons and deciding whether an area is cancer-containing or not, can also predict the diagnostic outcome. Comparing radiologists’ fixations while making correct and incorrect decisions revealed the effect of faulty background sampling on likelihood of making an error.⁴⁰ In Mello-Thoms,⁴⁰ the characteristics of the fixated background areas were extracted using spatial frequency analysis after the observers’ eyes hit for the first time the location of TP, FN, and FP areas. For TP areas, the spatial frequency representation of background areas sampled before and after first fixating on the location of the decision did not differ, while for FN and FP areas, significant differences were reported.⁴⁰ The analysis showed that after fixating on FN areas for the first time, the correlation of the background samples with the lesion increased while the correlation with the average background decreased. This suggested that the observers were actively trying to insert the lesion in the context of the normal background breast tissue and reconcile the perception of the lesion with its background.⁴⁰ For FP areas, the perception of a finding effectively biased further visual search as the observer tried to collect evidence to support the presence of a malignant mass in a lesion-free area of the parenchyma. Therefore, it can be hypothesized that in some of the decision making errors, an incorrect background sampling strategy led to the failure in reporting the lesion.⁴⁰

Studying radiologists’ fixations also showed that the radiologist’s first decision, which mainly occurs as a result of the global impression, affects the rest of performance.³⁹ When an observer makes a FP decision according to his/her first impression from the image, then the probability that the radiologist will make a faulty decision as a subsequent outcome is higher than the probability of reporting a true lesion. Therefore, an incorrect initial decision (or an incorrect initial "holistic" view) makes the observer effectively "blind" to the characteristics of the real lesion present.³⁹

Satisfaction of search (SOS) can also lead to FN errors. SOS happens when an abnormality is missed because the observer has already detected another abnormality in the same image. Most of the previous studies that explored SOS error and its causes focused on chest images. Earlier studies on chest X-ray and trauma radiography showed SOS could lead to a reduction in the number of TPs while search behaviour, in terms of total inspection time and dwell time in the location of reported abnormality remained unchanged.^43,44 Later, Samuel et al,⁴⁵ categorized SOS errors that occurred when finding nodules on chest X-rays into scanning, recognition, and decision-making errors. Their results indicated that SOS was not caused by scanning errors (i.e. inadequate scanning of the image), but rather it mostly occurred because the missed nodules did not received prolonged attention (i.e. they were recognition errors). In mammography, Mello-Thoms et al⁴⁶ showed that detection of abnormalities in a case is greatly associated with their similarities to the first perceived abnormality reported by the radiologist (regardless of being TP or FP). Therefore, like those in chest radiographs and trauma radiography, the study supported the idea that SOS is due to perceptual suppression of recognition instead of faulty scanning or visual neglect of certain regions.⁴⁶

Providing feedback to radiologists based on their search pattern

As stated, studies suggested that more than 70% of missed lesions in mammography attracted the radiologists’ attention to some extent (i.e. they were recognition and decision-making errors)⁴⁷ and time to first hit the location of FP areas differed significantly from the time to hit TP areas.⁴² Therefore, early studies suggested that giving feedback based on the observer’s gaze pattern might reduce errors.⁴⁷ However, in the case of mammograms, there was no significant improvement of reader’s performance after providing feedback based on the total dwell time of fixated areas.⁴⁸ Later, as discussed next, studies combined total dwell time with other gaze and image-related features and applied machine learning techniques to predict different types of errors.

Voisin et al⁹ developed a machine learning-based model to predict diagnostic errors by merging radiologists’ gaze behaviour and image characteristics when investigating a region of interest (ROI) containing either a benign or malignant mass. The main limitation of this model was considering an image ROI instead of a whole mammogram. Tourassi et al⁸ utilized a more realistic experimental setup, where radiologists viewed the whole mammogram, and image features along with gaze parameters were extracted from each fixated location. However, they used analogue (digitized) images and only included a limited number of cases. In a recent study,⁷ an individualized tool for identification of interpretation errors in mammograms, called eye-Computer Assisted Perception, was developed and validated on a relatively large data set containing eye tracking data from eight breast radiologists reading 120 two-view digital mammograms. Eye-Computer Assisted Perception was personalized for each reader as it was shown each radiologist had his/her own unique error-making pattern. Despite of the promising results obtained in the study, the fact that experimental set-up was different from the real clinical practice (e.g. presenting two views of a single breast instead of four views of both breasts and lack of prior images) limits the application of the proposed model in practice.

What’s to come

Since the 1990s, mammography has been the frontline screening tool for early detection of breast cancer. As discussed in the previous section, in the past four decades, visual search studies provided a valuable understanding of the mammographic interpretation process and revealed several underlying causes of diagnostic errors. However, the current literature has a few limitations, which can be addressed in future studies on visual search in mammography. First, the experimental design of the previous studies was not consistent with clinical practice. In most of them, using image viewing features such as zooming, inverting, Tabár’s systematic viewing mask techniques,⁴⁹ and other types of image manipulation were not allowed, as analysis of the eye tracking data can be significantly more difficult when these manipulations are used. However, in the real clinical practice those options are available for radiologists and are frequently used by some individuals.

Moreover, due to difficulty of eye tracking across multiple displays, usually studies are based on a single view mammogram, while in the real practice four views along with prior mammograms (if available) are presented simultaneously. A few previous studies^{6,7,23,24,30,31,50,51} used craniocaudal (CC) and mediolateral (MLO) views, which were presented on two displays. When two views were available, radiologists were able to compare between them to make decisions to report perceived findings, and this could result in different mammographic scanpaths as compared to the single mammogram viewing condition. Study of spatiotemporal dynamics of radiologists when viewing CC and MLO views simultaneously showed that radiologist’s gaze frequently moves from one view to another, and experts are better in fusing image information from different views.³¹ Such a difference would not be observed in a study based on a single view mammogram. Also, based on analysis done,³¹ the temporal dynamics of experts’ fixations when the cancer is present only on a single view differ from cases where the cancer is visible on two views. In addition, experienced radiologists more frequently move back and forth between diagnostically relevant areas on two views.³¹ Although parameters such as time to first hit the lesion or the role of holistic processing will be similar in the two experimental conditions, without recording eye movements on two views, the above-mentioned differences cannot be characterized. Similarly, eye tracking data based on both sides of the breasts will be required to explore how radiologists assess asymmetry-related features and inclusion of prior images will be needed to characterize mammographic scanpaths when a radiologist compares prior and current mammograms. Only in one previous study,³² where fractal dimension was used to analyse the complexity of eye-tracking data, an experimental set-up with simultaneous viewing of four co-ordinated breast views as typically done in clinical practice was used. When different hanging protocol are allowed, analyzing data could be challenging as radiologists might switch between hanging protocols. In Alamudun et al,³² the fixations were aggregated back into the predominant hanging protocols used by the reader for each case. Future studies can use a similar strategy to analyse the fixation data when recording eye tracking across multiple views.

Also, most of the earlier studies focused the association of observers’ characteristics and visual search patterns.^{16,17,25–32} More recent studies showed that radiologists’ visual search patterns relate to breast density and case pathology,^31,32 however, further studies are required to determine how visual search relates to different types of malignancies, lesion difficulty, and other features describing image content. Findings from such studies might help in providing a more effective education to trainees and in determining the optimal practice (search strategy) to mitigate diagnostic errors in detecting more difficult lesions, which are missed more frequently.

In recent years, application of deep learning algorithms led to a new generation of computerized tools for providing a second opinion to radiologists.^52,53 Previously, Krupinski²⁸ studied interaction of both experienced and less-experienced radiologists with CAD prompts. However, previous generations of CAD tools, which relied on classical machine learning framework and hand-crafted features, annotated many FPs on mammograms and were not popular among radiologists.⁵⁴ Given that deep learning provides a new perspective for computerized tools, it is necessary to investigate how displaying decisions made by these new, more-accurate computerized tools, changes the radiologists’ visual search. Understanding the interaction between the radiologists and the machine’s output will ultimately help in optimizing the way that computerized tools should be implemented in the radiological workflow. Beside deep learning-based tools, computerized tools could include computer-aided perception tools,⁷ which combine gaze-related features with image features to predict errors. Current studies^6–9 have assumed that radiologist would accept decisions made by computer-aided perception tools. However, in practice these tools would provide feedback to radiologists, who would still make the final decision. A potential future work is exploring how radiologists interact with the output of computer-aided perception tools.

Moreover, even though radiologists have their own unique error making patterns and a personalized model to predict diagnostic errors outperformed a one-size-fits-all model,^8,9,27,30,31 further studies are required to find out how we can tailor educational materials based on each reader’s unique error making and visual search patterns,⁶ and to investigate any changes to the reader’s visual search pattern after providing such educational materials. Despite advancements in technology, the accuracy of mammographic image interpretation still suffers from intrinsic human errors. Improving radiology education through innovative tools, which audit reader’s visual search and provide immediate feedback on their errors, may potentially improve diagnostic efficacy.

Breast ultrasound

Findings from previous studies

Ultrasound is the most common complement to mammography.⁵⁵ As with mammography, the observer’s capability of recognizing normal tissue appearances and differentiating them from subtle abnormalities is crucial to make an accurate diagnosis.⁴ Unlike mammography, where the entire breast area is visible, ultrasound examination only shows a portion of the breast containing the area of interest. Moreover, sometimes the scans are not performed by radiologists, and sonographers (technologists, non-radiologists) perform the diagnostic scan, select images which capture pathologies and present them to the radiologist for the final reporting.⁵⁶ Carrigan et al⁵⁷ explored the gaze behaviour of breast sonographers and showed that, similar to what was found in studies on mammograms, sonographers committed search, recognition and decision making errors. For example, prolonged fixations were observed on the FN locations.⁵⁷ As sonographers’ decisions are sometimes made on unrecorded image data, the findings of this study suggested that areas that attracted prolonged sonographer’s attention should be presented to radiologists as they might contain a subtle finding, incorrectly recognized as normal tissue by sonographers.

What’s to come

In past two decades, breast ultrasound has been widely used in addition to mammography to better characterize suspicious lesions and determine if an ultrasound-guided interventional procedure should be advised for the confirmation of diagnosis. Interpretation of breast ultrasound is a unique perceptual task from different aspects, especially as it requires hand-eye coordination and real-time decision making.⁵⁸ Therefore, findings from visual search studies on mammography might not be directly translatable to breast ultrasound and further investigations in this field are required. First, visual search behaviour of radiologists in breast ultrasound have not been studied yet, as the study conducted by Carrigan et al⁵⁷ only included breast sonographers. It is unknown whether the visual search patterns of radiologists differ from those of sonographers. Also, the effect of clinical history and presence of abnormality on mammograms may affect the radiologists’ visual search behaviour while reading breast ultrasound, but this effect, if it exists, has not been determined. Mostly, ultrasound is used to evaluate a suspicious area found by mammography and hence during its interpretation, radiologists are informed about the location of the area of concern and its appearance on the mammograms.⁴ However, for the investigation of breast symptoms in high-risk females under 35 years of age, ultrasound may be the primary (and only) imaging modality.⁵⁹

Volumetric breast images

Findings from previous studies

Although it is estimated that 25% of the reduction in breast cancer mortality rate in the past decade is a result of mammography screening,⁶⁰ mammography has been recently criticized for its relatively low sensitivity and specificity.⁶¹ DBT is an emerging imaging modality with potential to improve cancer detection rate in the screening program.⁶² DBT produces a CT-like stack of two-dimensional (2D) images that radiologists can scroll through in depth. When a suspicious area is found, radiologists can examine it in different slices with less masking effect of overlapping fibroglandular tissue.⁶² One of the main obstacles in the wide adoption of DBT is the increased time required to interpret a screening examination, as a DBT exam can contain up to 80 images for the reader to examine, and radiologists are facing challenges to adapt their search strategy to this new imaging modality.^63,64 Better understanding of the optimal search strategy for DBT using eye tracking can help readers to develop more efficient interpretation approaches.

Timberg et al⁶⁵ eye-tracked 4 observers while reading 55 DBT images, in which simulated lesions were inserted. Readers’ performance in free scroll volume browsing was compared to viewing of an initial cine loop, followed by additional free scrolling. The cine loop was produced in fast, medium, and slow frame rates.⁶⁵ Based on this study, cine loop with fast and medium frame rates and horizontal presentation speeded the process of selecting relevant suspicious regions in free scrolling.⁶⁵ However, the scanpaths were not recorded in this study and having only four observers and simulated lesions limit generalizability of their findings. In another study, Jiang et al⁶⁶ eye-tracked three medical physicists while reading DBT, however, the study investigated the correlation between fixation times and sets of morphological feature values extracted from the images. Although the findings have the potential to improve visual search model observers in the detection-localization task of DBT images, the study did not analyse the radiologists’ scanpaths and eye movements.

Two recent eye-tracking studies by Aizenman et al⁶³ and Dong et al⁶⁴ studied the visual search behaviour of radiologists while reading DBT in more detailed. Both studies showed that total search duration was longer for DBT, and dwell time in the fixated areas was longer. Aizenman et al⁶³ indicated that DBT was associated with a saccadic amplitude similar to that of mammography. They also investigated whether DBT observers align with “driller” or “scanner” strategies, previously identified when radiologists searched for lung nodules in volumetric lung CTs.⁶⁷ However, there are two main differences between lung CT and DBT images that make comparing these two modalities challenging. First, the number of slices in CT images is larger than DBT. Secondly, image size is smaller in CT images. Therefore, normalized distances in depth and XY plane were used. The results suggested that the DBT observers cannot be fully categorized as previously identified “drillers” or “scanners”. They drill through the depth of the breast comparatively quickly and cover more depth levels in a 5 s window when compared with “drillers” in the lung CT study. However, unlike the “drillers”, the DBT viewers do not fixate roughly in the same area in XY plane while flying through the depth. Having said this, the observed movement in XY plane was smaller than what observed among “scanners” in Drew et al.⁶⁷ The study concluded that the search strategies of DBT observers were more aligned with drilling, albeit it was not exactly like it. In the other study, Dong et al⁶⁴ eye-tracked experienced radiologists while presenting them a 2D overview and the DBT images. It was found that the radiologists were inclined to do a global-focal scan over the 2D view to detect the abnormalities and then drilled through the DBT slices to verify their detections. No significant difference in viewing time between normal and abnormal cases was found.

What’s to come

Considering the inherent limitations of mammography due to masking effect as a result of projecting a three-dimensional (3D) area into a 2D plane and technological advancements, volumetric breast imaging has become more widely used in recent years. Given that 3D breast tomograms and 2D mammograms are very different, results of visual search studies in one modality are not directly translatable to the other modality. Generally, due to challenges associated with eye tracking methodology in volumetric images, the number of visual search studies that looked at volumetric images lags behind conventional 2D radiography.⁶⁸ For example, in DBT tomograms radiologists can freely scroll through the image. Scrolling back and forth adds an additional factor to the study. To deal with this, some studies presented a video playback of flying through a stack of image slices, rather than individual slices. Furthermore, analyzing the raw eye tracking data, which relies on detecting fixations and saccades, is fundamentally difficult when dealing with the videos or during scrolling because the content of foveated areas may change during the course of one fixation, and the eyes also frequently perform smooth pursuit movements. Several strategies have been proposed to tackle these challenges and adjust the definition of eye tracking parameters for volumetric images.⁶⁸

As mentioned in the previous section, limited visual search studies have been conducted using DBT images, while reading other types of volumetric breast images (e.g. breast MRI or CT or functional imaging) have not been studied yet and future studies in this area are required. These imaging modalities can be used in various situations, e.g. breast MRI might be used for screening of high-risk females,⁶⁹ determining the extent of breast cancer,⁷⁰ or distinguishing fibrosis from tumour recurrence in females who underwent breast-conserving surgeries.⁷¹ Depending on the task at hand, the visual search behaviour of the observers might differ. Better understanding of differences and optimal search behaviour can help in training radiologists and increasing the specificity of breast MRI, which is an ongoing area of investigation.⁷² Moreover, the radiologist’s visual search patterns when using workstations that provide multimodality image viewing has not been studied. Currently, in clinical practice, using multimodality image viewing is gaining popularity and understanding how an experienced radiologist switches between modalities and relates findings in different modalities could help in establishing best practice approaches. Previously, recurrence quantification measures have been used to explain how radiologists switch between views in mammograms and compare two locations in mammographic images.³¹ Similar analysis can be done to quantify how they search two different modalities simultaneously.

How Basic Visual Search Research Aids Visual Search Studies In Breast Imaging

In addition to studies conducted using breast images, findings from ongoing basic vision science research, which utilized simplified visual search experiments (e.g. finding T-shaped targets among L-shaped distractors), are helpful in studying some of the challenges that radiologists face in breast cancer detection. One of the main challenges, particularly in the interpretation of screening mammograms, is the low prevalence of cancer cases, which could reduce the observer’s detection rate. To study the effect of low prevalence target on observer’s performance, Wolfe et al⁷³ designed an experiment where observers searched for a target among items drawn at random from other categories. Three different trials with different target frequency (1%, 10% or 50%) were generated. At prevalence of 50%, observers missed 7% of targets, while error rates increased dramatically with decreased prevalence: a prevalence of 1% produced an error rate of 30%. Interesting, observers who participated in the 50% condition before the 1% condition had an average error rate of 26% in 1% condition, while observers who participated in the 1% condition first averaged 46% of errors. This suggests that a training regimen with high prevalence targets before image interpretation might improve the detection rate, as it might counteract the low prevalence effect.⁷⁴ In the context of screening mammography, cancer prevalence is only about 0.5%,⁷⁵ which is even lower than lowest-prevalence rate in Wolfe et al’s study.⁷³ Certain abnormalities, such as architectural distortions, only appear on less than 0.05% of cases. Mitroff and Biggs⁷⁶ called these targets “ultra-rare” items and showed that the error rate was considerably higher for targets with prevalence below 1% than for targets with prevalence above 1%. They also reported a logarithmic relationship between target detection and target frequency.

Another area where taking cognitive laboratory approaches was helpful was to better understand underlying causes of errors is SOS. Fleck et al⁷⁷ designed experiments with simplified search displays and showed SOS outside the discipline of radiology for the first time. Their experiments demonstrated that SOS occurs as a result of biased expectations about the low likelihood of specific targets or events and external pressure (e.g. time or reward) to do tasks efficiently. Later, Cain et al⁷⁸ used an almost similar experimental protocol and investigated the breakdown of SOS errors. About 24% of SOS errors were due to resource depletion, where searchers refixated on a detected high-salience target. The large number of fixations on this target might disrupt the memory for searched locations⁷⁹ or it could bias visual search toward items that match the content of working memory.⁸⁰ The study also explored the magnitude of “satisfaction” errors, in which searchers terminated their search early. This error was called “strategy error” and only accounted for 6% of the SOS errors. This echoes findings from earlier research done using medical images, where it was reported that SOS errors were not due to insufficient time-on-task. However, an interesting finding about this category of error was that two given participants were mainly committing this type of error. Thus, although overall strategy errors are not the main cause of SOS, the magnitude of strategy errors accounting for SOS in certain individuals might differ considerably from that of the average observer. In a recent study,⁸¹ where individual differences of observers have been taken into account, evidence for the “satisfaction” account of SOS was provided. Specially, a significant and negative correlation between time spent searching after finding a high-salience target and SOS error rates was found. This implies that observers who are faster to terminate search after detecting the first target are more likely to commit SOS errors.

A new challenge in breast radiology is moving from 2D mammography to 3D tomosynthesis. As mentioned in "Findings from previous studies", there are a few studies that compared the search strategies in these two modalities.^63–66 However, the relatively longer interpretation time of tomosynthesis, the difficulty in accessing radiologists as participants in research studies (due to their time constraints), and the lower feasibility of experimental control when using medical images makes studying the cognitive processes underlying tomosynthesis visual search difficult. In a recent study, Adamo et al⁸² proposed an alternative experimental paradigm to explore 3D visual search with both radiologists and non-radiologists, and obtained insight about how radiologists would search tomosynthesis. They used a simplified experiment where observers were asked to find one T-shaped target among L-shaped distractors. The search items were randomly placed within a 15 × 15 × 15 location matrix. The 3D images were either compressed into a single flat plane (analogous to mammograms) or were divided into 30 different slices (analogous to tomosynthesis). Participants were able to scroll through the slices. The performance of professionals (radiology residents and certified radiologists) and non-professionals (undergraduate students) observers were studied in this experiment. The results showed that the performance of the non-professionals matched the performance of professionals and the performance reported in previous radiological studies. Therefore, the study suggests that their experimental paradigm is a cost-effective tool to conduct experiments using non-professional observers for examining relevant visual search performance in this task. Considering the difficulties in recruiting radiologists in visual search studies, such experimental paradigms are important to formulate and test initial hypotheses regarding different aspects of medical image perception.

Beside experimental paradigms in basic visual search, various techniques, which have been proposed in basic vision sciences to analyze eye-tracking data, can be used in medical visual search studies. Among these techniques, clustering the fixations and extracting gaze features from each cluster is probably the most popular technique among medical image perception studies. However, depending on the study’s hypotheses and specific aims, other techniques have been also used in new studies. For example, Anderson et al⁸³ first proposed a framework based on recurrence quantification analysis to study the global and local spatiotemporal characteristics of fixation sequences of observers while looking at different scenes. This framework has been later used in³¹ to study how spatiotemporal characteristics radiologists’ fixations, more specifically, how they refixate on a previously fixated area, how they compare different areas with each other, and how the temporal dynamics of fixations depends on the image content. Also, various quantitative metrics have been developed in vision sciences to quantify differences and similarities between two eye tracking sequences.⁸⁴ Where required, these techniques can assist in understanding differences and similarities between radiological visual search patterns. For example, they can be used to identify pairs of radiologists who searched the same mammograms in the most different fashion, with the aim of finding optimal pairings of radiologists.⁵¹

Conclusion

Given that breast cancer is the most common female cancer, more than 1 billion mammography examinations are annually performed and interpreted worldwide. Each year, radiologists also read a large volume of breast ultrasound, DBT, and breast MRI. Studies show extensive inter radiologist variability in their accuracy while reading breast images, and suggest that readers’ characteristics, especially their experience and annual working load may influence their diagnostic accuracy.⁸⁵ Therefore, extensive visual search studies have been conducted to characterize how cognitive and perceptual processes of interpreting breast images relate to readers’ characteristics. Evidence of these studies can provide guidance for enhancing practice in breast radiology and improving the diagnostic accuracy.

Findings of visual search studies could also assist educators to enhance training in breast radiology. The holistic processing model suggests that educators should present large volumes of normal and abnormal cases to students so that they build a good awareness of normal/abnormal appearances. Moreover, recent machine learning classifiers based on features extracted from fixation data were able to successfully distinguish less-experienced readers from experts. This opens new opportunities for monitoring the development of expertise and developing individualized training programs with different pace for each trainee’s needs. Given the fact that widely accepted models for explaining expertise development have been suggested, designing good theory-driven studies of specific interventions in radiological education could be a possible future line of research.

Another important area addressed in medical image perception studies is mapping out the underlying reasons for errors in breast radiology. Despite the technological advances, which led to producing high quality images, error rates in radiology are still significantly high, and improving diagnostic errors in breast radiology is required. For example, the error rate in detecting cancer on mammograms varies from 10% to 30%, while approximately 80% of females recalled for additional imaging are ultimately diagnosed as normal.⁸⁵ Considering this high error rate, proper understanding of error causes could lead to error mitigation through developing appropriate educational interventions, a strategy for auditing and providing feedback to readers (e.g. computerized feedback system based on their visual search pattern). These solutions will potentially benefit millions of females each year. In summary, studies suggested that radiologists fixated on the majority of missed cancers but did not report them due to factors such as improper background sampling, incorrect first impression about the image, perceptual suppression of recognition, and prolonged visual search before making their decisions. In addition to the characteristics of the fixated area, the pattern of visual search itself relates to the diagnostic errors. Quantitative features extracted from mammographic scanpaths, describing their complexity, timing, and spatiotemporal characteristics were shown to be predictors of diagnostic errors. A few recent studies designed tools based on eye tracking data and image content for detecting diagnostic errors. A potential direction for future research could be further improving these tools to make them compatible with clinically realistic viewing configurations. Current tools rely on features extracted from scanpaths recorded during the assessment of a single mammographic view or CC-MLO views. However, these experimental set-ups do not resemble the real practice, where four breast views, prior examinations, and different options for image manipulation and configurations for image presentation are available to the radiologists. Another important future line of research is exploring optimal ways to integrate the responses from such tools as well as CAD into radiology workflow.

A limited number of studies have been conducted to explore radiologists’ visual search in imaging modalities such as breast ultrasound, DBT, and breast MRI. Considering the unique characteristics of visual search in each search task, future studies focusing on these modalities are highly recommended. For example, radiologists’ scanpaths while reading DBT images cannot be fully categorized as either drillers’ or scanners’ search patterns, which were previously identified while searching for lung nodules in volumetric lung CTs. Moreover, it is unknown whether the holistic processing model proposed to describe radiological eye movements in 2D images can explain the radiologists’ visual search patterns while examining a stack of images. Also, considering new findings emphasizing the importance of global image statistics in establishing the radiologists’ first impression about the image, it is necessary to investigate how the holistic processing search model should be refined. Moreover, determining the intersection between models rooted in basic vision science and in medical image perception should be further investigated. Experimental and cognitive psychology can also offer different techniques to design simplified experiments for understanding the roles of attentional mechanisms and working memory. These techniques may be applied to design simplified experiments with non-radiologist participants which may possibly explain causes of diagnostic errors, role of attentional mechanisms, and better modelling of the medical image interpretation process.