Gist processing in digital breast tomosynthesis

Abstract.

Evans et al. (2016) showed that radiologists can classify the mammograms as normal or abnormal at above-chance levels after a 250-ms exposure. Our study documents a similar gist signal in digital breast tomosynthesis (DBT) images. DBT is a relatively new technology that creates a three-dimensional image set of slices through the volume of the breast. It improves performance over two-dimensional (2-D) mammography but at a cost in reading time. In the experiment presented, radiologists ($N=16$) viewed “movies” of DBT images from single breasts for an average of 1.5 s per case. Observers then marked the most likely lesion position on a blank outline and rated each case on a six-point scale from (1) certainly normal to (6) certainly recall. Results show that radiologists can discriminate normal from abnormal DBT cases at above-chance levels as in 2-D mammography. Ability was correlated with experience reading DBT. Observers performed at above-chance levels, even on those images where they could not localize the target, suggesting that this is a global signal that could prove valuable in the clinic.

1. Introduction

During natural scene viewing, individuals are exposed to much more information than they can process. To cope with this problem, people use attentional processes to select portions of the current scene for analysis. Attention can be guided to potentially important objects or locations by a variety of factors.¹ One of those factors is the global, holistic information the observers can quickly and efficiently extract from the scene.^2,3 This global information is sometimes referred to as the “gist” of the scene.⁴

The role of a similar initial evaluation of an image has been also discussed in medical image perception for many years. The first glance at a lung x-ray might tell a radiologist where it might be profitable to search for signs of lung cancer. Kundel and Nodine⁵ asked whether radiologists could extract any useful information at the beginning of the search. They asked radiologists to evaluate x-ray images after only a 200-ms exposure. When there was unlimited reading time, the performance by their experts was nearly perfect. Interestingly, they found that the accuracy with which experts could classify the images still reached about 70% with just a 200-ms glimpse of the image. Clearly, there was significant information available within the first second of the search. Carmody, Nodine, and Kundel⁶ later varied the image duration to ask how the detection performance changed over the first half-second of exposure. They used exposure durations from 60 to 480 ms and found that detection performance asymptoted after 240 ms. Even for the hardest cases in their set, there was still substantial information available in the first quarter of the second. This global analysis occurs not only in lung screening but also in the screening of mammography.⁷

In a more recent study, Evans et al.⁸ showed radiologists a breast image for a brief duration, between 250 ms and 2 s, and asked them to classify this mammogram as normal or abnormal. After this brief presentation of the stimulus, radiologists were asked to attempt to localize the suspicious lesion on a white outline mask of the breast and then rate their decision confidence on a 0 to 100 scale from abnormal to normal. Even with this single, brief glance of breast image, radiologists could still perform at above-chance levels at all stimulus durations between 250 and 2000 ms. Surprisingly, this awareness of abnormality did not rely on any visible feature of a lesion because radiologists were at chance in their ability to localize the abnormalities in these images when asked to mark the potential lesion location. A subsequent study from Evans et al.⁹ showed that this gist signal was not based on asymmetry between a normal and abnormal breast in a bilateral mammogram, nor was performance a proxy for a quick measure of breast density, which is associated with higher cancer risk. Moreover, radiologists were able to discriminate between normal and abnormal patients when the “abnormal” images were the images of the breast contralateral to the breast with a visible lesion. Since there was no lesion presented in the image, the observed performance cannot be due to a lucky fixation on a mass. Brennan et al.¹⁰ and Schill et al.¹¹ further demonstrated that experts can classify women as normal or abnormal when the images viewed were taken 3 years prior to the onset of visible cancer. These results show that the global gist signal need not arise from a visible local feature in the breast.

In this paper, we ask if a similar signal is present in the images generated by digital breast tomosynthesis (DBT). DBT is a technology that produces a stack of images through the breast, rather than a single x-ray. DBT has been shown to be more accurate than 2-D digital mammography (DM) and is becoming widely adopted in clinical practice. Though DBT improves both sensitivity and specificity, it can take radiologists twice as long to interpret a DBT case compared to the DM version.¹² Therefore, minimizing the additional cost in time without sacrificing the accuracy has become a substantial topic for research.¹³

Given the results described above for briefly presented mammograms, it is possible that a brief exposure to the sequence of DBT slices might provide similar global information and direct attention to suspicious corresponding areas in the full-field digital mammography (FFDM) image. Even if it was not possible to use what we could call “DBT gist” to guide search for a breast lesion, knowing whether there was a signal of abnormality extracted from a brief glimpse of the DBT stack of images could be informative for developing DBT image diagnosis protocols. Demonstration of the existence of such a signal is the purpose of this paper.

2. Method

2.1. Participants

Sixteen radiologists were tested in this study. All participants had experience in breast imaging and had an average of 13.2 years of experience reading FFDM ($\mathrm{SD}=14.02$, range [1 to 44]) and 3.9 years of experience reading DBT ($\mathrm{SD}=2.03$; range [1 to 7]). Nine participants participated during the 2018 annual meeting of the Radiological Society of North America (RSNA) as part of the NCI Medical Image Perception Lab, run at the meeting, and the other radiologists participated at the University of Pittsburgh Medical Center. All radiologists had normal or corrected-to-normal vision and gave informed consent prior to participation. The readers at the RSNA meeting were offered a chance to win a $500 Amazon gift card in exchange for their participation. The experiment had institutional review board approvals from the University of Pittsburgh and Brigham and Women’s Hospital.

2.2. Stimuli and Procedure

Forty-four negative cases along with 44 biopsy-proven cancer cases were selected to create the set of images used in the current experiment. The details of all positive cases are listed in Table 1 of the Appendix. Among these positive cases, 26 of them were deemed to be subtle in nature with the remaining 18 more obvious when given time to review the cases. The majority of the cancer cases selected ($n=23$) had a BI-RADS density classification of 2, BI-RADS 1 density was the next most common ($n=14$) followed by BI-RADS 3 ($n=6$), and 4 was least common with only one case. The distribution of BI-RADS density classification among the negative cases was 1 ($n=17$), 2 ($n=11$), 3 ($n=12$), and 4 ($n=4$). There was a distribution of the cancers in all quadrants of the monitor. The left upper quadrant had the most cancers in it with 15 cases, followed by the left lower quadrant ($n=13$), then the right upper and right lower quadrant, both with eight cases in each. The cases had a mean of 72.7 ($\mathrm{SD}=12.9$) slices with the pathology visible on an average of 28.7 ($\mathrm{SD}=8.7$) slices meaning this was visible to the reader for an average of 598 ($\mathrm{SD}=180.1$) msec. The slice number in the negative cases was very similar to that of the cancer positive cases with a mean slice number of 71.4 ($\mathrm{SD}=12.6$). The appendix shows the details of all positive cases used in this study.

Table 1.

The details of positive cases used in the current study.

Case	Type of lesion	Density	Affected breast and view shown		Number of slices		Time abnormality shown (ms)	Quadrant of lesion	Pathology	Max. lesion dimensions (mm)
					Per case	Lesion visible				Anterior/posterior	Superior/inferior
1	Mass	1	Right	MLO	96	25	521	2	ILC	9.4	6.8
2	Asymmetry	3	Left	CC	67	40	833	3	IDC and DCIS	16.4	10.2
3	Asymmetry	2	Right	CC	73	31	645	4	IDC	11.4	13.9
4	Mass	2	Right	MLO	83	50	1041	4	IDC and DCIS	16.4	12.7
5	Mass + microcalcifications	3	Left	MLO	51	29	604	2	IDC	10.9	12
6	Architectural distortion + calcifications	2	Right	CC	76	29	604	1	Not specified	8	5.9
7	Mass	2	Right	MLO	83	59	1228	1	Not specified	12.1	14.2
8	Mass	2	Left	CC	74	26	541	2	IDC	3.4	3.5
9	Mass	2	Right	MLO	82	29	604	3	IDC and DCIS	10.2	4.9
10	Mass	3	Left	CC	41	12	250	2	ILC	10.7	8.5
11	Asymmetry	2	Left	CC	58	24	500	1	IDC and DCIS	9	4.6
12	Mass	2	Left	MLO	86	23	479	3	IDC and DCIS	13.1	15.8
13	Mass	2	Right	MLO	65	18	375	1	IDC	9.2	8.9
14	Architectural distortion	3	Left	MLO	60	20	416	2	IDC	8.8	7.5
15	Asymmetry	2	Left	MLO	82	28	583	3	IDC	10.4	4.6
16	Mass	1	Left	CC	80	16	333	3	IDC and DCIS	8.3	6.6
17	Mass + architectural distortion	4	Left	MLO	64	16	333	3	Multicentric ILC, IDC, DCIS	17.8	18.9
18	Asymmetry	1	Left	MLO	67	21	437	2	IDC and DCIS	9.4	9.7
19	Architectural distortion	2	Left	CC	63	38	791	3	DCIS and radial scar	16.4	13.4
20	Mass	1	Left	MLO	74	35	729	3	IDC and DCIS	8.2	11.8
21	Asymmetry	2	Right	CC	63	26	541	4	Papillary carcinoma	8.1	8.2
22	Calcification	2	Left	MLO	74	38	791	2	DCIS	10.9	8.6
23	Mass	1	Right	CC	67	28	583	4	ILC	20.1	20.4
24	Mass	1	Left	CC	71	32	666	2	IDC and DCIS	11.3	6.2
25	Architectural distortion	1	Left	MLO	91	34	708	2	IDC	7.2	5.5
26	Asymmetry	1	Left	MLO	71	31	645	1	IDC and DCIS	8.4	5.4
27	Asymmetry	1	Right	MLO	103	28	583	2	ILC	8.4	7
28	Architectural distortion	3	Right	CC	60	29	604	1	IDC	19.9	10.8
29	Mass with architectural distortion	2	Right	CC	67	31	645	3	IDC and DCIS	8.5	5.1
30	Architectural distortion	2	Left	CC	55	29	604	3	DCIS, very limited IDC	12.6	12.3
31	Asymmetry	1	Left	MLO	79	22	458	2	IDC	6.5	3.9
32	Mass	2	Left	MLO	66	32	666	2	Tubular carcinoma	25	27
33	Asymmetry	2	Right	MLO	70	31	645	4	IDC and DCIS	13.6	12.2
34	Asymmetry	1	Left	MLO	72	27	562	1	IDC	12.3	10.8
35	Mass	1	Left	MLO	94	35	729	3	IDC and DCIS	7	4.8
36	Mass	2	Right	CC	71	32	666	4	ILC	29.2	37.8
37	Mass	1	Right	CC	79	27	562	4	IDC	11.7	9.1
38	Architectural distortion	2	Left	MLO	94	26	541	2	ILC	16.8	15.1
39	Asymmetry	3	Left	MLO	55	15	312	2	IDC	14.2	12.5
40	Mass	1	Left	CC	85	22	458	3	IDC and DCIS	12.7	9.8
41	Architectural distortion	2	Left	MLO	55	30	625	3	IDC and DCIS	18.6	15.8
42	Architectural distortion	2	Right	CC	81	34	708	1	IDC	7	6.4
43	Asymmetry + calcifications	2	Right	CC	72	36	750	4	ILC	8.4	7.3
44	Asymmetry	2	Left	CC	79	20	416	2	IDC	6.2	5.2
	Mean	1.86			72.70	28.73	598.07			11.91	10.49
	Standard deviation				12.89	8.65	180.13			5.13	6.44
	Range						250 to 1228			3.4 to 29.2	3.5 to 37.8

Each case was presented on a Barco Coronis 5 megapixel monitor with a resolution of $2048\times 2560$. The monitor was calibrated to the DICOM GSDF and the light was set by a standard dimmer to recreate reading-room conditions. Observers were able to adjust the lighting to their desired level. Before the experiment, observers could self-select viewing distance to a comfortable distance. These ranged from 55 to 65 cm. Each image subtended $\sim 33.8\times 42.2\mathrm{deg}$ of visual angle at a viewing distance of 60 cm.

On each trial, a mask, which was a gray outline of the breast for the current case, was shown for 1 s so observers would know where to fixate when they viewed each case. After the mask, the DBT case was presented at a rate of 20 ms per slice and each slice was only presented once. The presentation rate was chosen based on the limits imposed by hardware, with the intent of creating a brief stimulus presentation, akin to that used in earlier gist studies. The average presentation time per case was 1.5 s. After the stimulus was presented, the same mask reappeared, and observers were asked to make a single click on the most likely lesion location even if they thought the case was normal. Then, observers were asked to use a mouse to indicate how likely they would be to recall the patient on a scale of 1 (definitely not recall) to 6 (definitely recall) as shown in Fig. 1. There were three practice trials before the experiment. All observers viewed the same 88 cases in a randomized order across trials. The experiment took $\sim 30\min$ to complete.

Fig. 1.

Experimental procedure. Observers would click to start each trial when ready. They would preview a blank breast outline for 1 s prior to presentation of the DBT images. After DBT presentation, they used the mouse to mark the most likely lesion position on the blank breast mask. Finally, they rated the case on a six-point to indicate how likely they would be to recall this patient.

2.3. Data Analysis

We were interested to determine if there is a gist-like signal in a single pass through a cine-loop of digital breast tomosynthesis (DBT) images that could be used by experts to distinguish normal from abnormal patients. To test this possibility, we calculated performance in two ways. First, we analyzed observers’ rating data in the same way as Evans et al. (2016). We calculated $d^{\prime }$ by picking a cutoff rating, e.g., 4, and treating all ratings of that value or greater as positive responses and all lower ratings as negative. Thus, if a case was abnormal and observers’ rating was greater than 3, this case would be scored as a hit. If the case was normal but the rating was greater than 3, the case was classified as false positive. Similarly, an abnormal case with a rating less than 4 would be a “miss” or false negative while a normal case with a rating less than 4 would be a “true negative” response. Second, the reader performance was also assessed by calculating an AUC using MRMC (multireader, multicases) analysis.^14–18 These are roughly equivalent methods as noted below.

3. Results

The results show that, when viewing the DBT images for one and a half seconds, expert radiologists can reliably perform at a better than the chance level [$d^{\prime }=0.81$, $t(15)=9.49$, $p<0.001$]. By repeating this analysis at each rating, we can generate a receiver operating characteristic (ROC) curve as shown in Fig. 2 and can calculate the area under the curve (AUC). Figure 2 shows that 15 out of 16 observers (dotted lines) have above-chance performance (chance performance falls on the main diagonal). The average AUC (solid line of ROC curve) calculated by MRMC is about 0.70 ($\mathrm{SD}=0.07$), which outperforms the chance performance AUC of 0.5 [$t(15)=11.84$, $p<0.001$]. This result suggests that even with only a second and a half exposure, observers were able to distinguish the abnormal from normal cases. Table 2 in the Appendix also shows the recall performance across all observers.

Fig. 2.

ROC curves for this study. Solid line shows the average ROC and the light dotted lines represent the individual observers.

Table 2.

Observers’ recall decisions in the current study.

	Normal	Cancer
No recall	0.68 (478/704)	0.39 (272/704)
Recall	0.32 (226/704)	0.61 (432/704)

The ability to distinguish normal from abnormal cases at above-chance levels might be due to the detection of identifiable features in the images rather than the holistic/gist processing, described by Evans et al.⁸ This is more of a concern because of the relatively long exposure to the DBT stimulus in this study. While each frame was presented for only 20 ms, some features persist over many frames, making it possible that they could be attended and fixated. As one test of this possibility, we repeated the analysis after excluding all trials where observers appeared to successfully localize the lesion. The lesion was categorized as successfully localized if the observer’s marker was placed within 3 deg from the center of the lesion for the positive trials. Otherwise, the case was categorized as unsuccessfully localized. Of a total of 704 positive cases, 279 cases were excluded ($\sim 40\%$) by this exclusion criterion. The remaining positive cases and the control cases were used to generate the ROC curves as shown in Fig. 3.

Fig. 3.

ROC curve only for the trials where observers cannot localize the lesion. Solid line shows the average ROC and the light dotted lines represent the individual observers.

When the cases with localized lesions were excluded from the AUC analyses, the average AUC decreased from 0.70 to 0.63 [paired $t$-test: $t(15)=8.26$, $p<0.001$]. However, their performance was still markedly better than chance [$t(15)=6.8$, $p<0.001$]. In addition, $d^{\prime }$ was 0.53, which was also better than chance [$t(15)=5.55$, $p<0.001$]. Thus, while we cannot rule out some contribution from a lucky fixation on the target, it is clear that the performance, shown in Fig. 2, did not depend on the cases where observers had an explicit idea about the location of the lesion.

If $z$ (true positive) is plotted against $z$ (false positive), the resulting zROC function has a slope of almost exactly 1.0 for the average data. This indicates that the underlying signal (abnormal) and noise (normal) distributions have equal variance. This means that the $d^{\prime }$ derived from the Evans et al. method and the AUC analysis will produce essentially the same results.

To better understand whether subjects’ performance was associated with the amount of time that the lesion was visible, we compared the accuracies of correct recall rate and their localization performance based on how long each lesion was actually shown to observers. Figure 4 shows that observers’ recall rate was higher when the lesion was presented longer ($R=0.34$, $p=0.0018$), while the localization performance was not correlated with the lesion presentation time. The scatterplot makes clear that even the recall rate is not strongly associated with presentation time.

Fig. 4.

Performance of recall and localization as a function of lesion presentation time. Blue circles represent the performance of recall (true positive) and red triangles represent the performance of localization.

The Kundel–Nodine group argues that the ability to process the holistic information is a critical part of expertise development and, certainly, nonexperts cannot perform these radiological gist tasks. To assess the role of expertise in more detail, we computed the Spearman correlation between $d^{\prime }$ and radiologists’ self-reported years of experience with DBT and DM. Radiologists with more experience reading DBT are, indeed, better at distinguishing normal from abnormal cases [Fig. 5(a), Spearman $r=0.51$, $p<0.05$]. DM experience is not significantly correlated with observers’ performance on the gist task [Fig. 5(b), spearman $r=0.27$, $p=0.17$].

Fig. 5.

Scatterplots of the relationship between $d^{\prime }$ and year of experience (a) reading DBT or (b) reading DM. Line shows the best-fit linear regression.

4. Discussion

This study adds DBT to the family of medical images that appear to contain a global signal that allows experts to distinguish normal from abnormal cases at above-chance levels after a brief presentation. This has been shown for standard 2-D mammograms and chest radiographs.^5,6,19 It is also true for micrographs of cells used to diagnose cervical cancer (classic Pap smear slides).^20,21 No doubt, other classes of medical images would show similar results if tested.

There are two types of very rapid processing that occur in medical images and in scenes, more generally. If one is looking for a target, be it a lung nodule or a cat, the known features of the target can guide your attention to that target if it is present.^1,22 Thus, attention will be guided to a briefly glimpsed orange blob of the right size in the search for a ginger cat. Kundel and colleagues were investigating this form of guidance when they found that a brief glance could successfully direct an expert’s eye to the location of a lesion. The second form of rapid processing has to do with extraction of the affordances of a scene, regardless of whether the actual target is present. Your first glimpse of a new restaurant may allow you to make an above-chance assessment of whether this will be a good place for you to eat. You will make that assessment without seeing the menu or tasting any food. Your assessment will not be infallible, but it is likely to be above chance. In laboratory studies, extremely brief glimpses of a photograph of a scene are adequate to determine at above-chance levels whether the scene has any of a number of quite complex characteristics, such as whether it is deep or shallow (e.g., field versus wall), open or closed (field versus forest), navigable or not.²³ These assessments are made on the basis of something like the texture of the scene.²⁴ They do not require processing of individual objects or structures in the scene. Thus, it is not entirely surprising that the texture of a medical image would contain information that did not require attentional scrutiny of specific parts of the image.

It is not perfectly clear whether performance with the current set of images depended on the presence of an overt lesion in the image. We found that observers still performed at above-chance levels on the normal/abnormal classification, even when they did not correctly localize the lesion. However, Carrigan et al.²⁵ have argued that it is possible to detect the existence of target but fail to localize it in the DM version of the experiment. For DM, Brennan et al.¹⁰ and Evans et al.²⁶ have found gist information in digital mammograms taken several years before the patient developed overt signs of cancer. A similar study could be done for DBT to confirm that detection of the gist signal does not require an overt lesion.

The set of features that guide attention may well be innate.²⁷ Expert gist assessment of a scene semantics, undoubtedly develops over time. It is possible to imagine that an infant might be sensitive to a property such as the “openness” of a scene as assessed in infancy. However, assessing the suitability of a restaurant or the appeal of a toy store is a self-evidently learned skill. Similarly, medical image gist perception must be learned. The results of the present study give us some insight into the specificity of that learning. In the case of DBT gist, it appears that specific experience with DBT is necessary. The readers with more 2-D DM reading experience did not have an advantage in extracting the DBT signal in our study. This suggests that the signals in 2-D and 3-D mammography are at least somewhat different. It would be interesting (if logistically demanding) to more precisely control the training of experts in order to understand this form of perceptual learning in more detail.²⁸

There are two arenas in which the existence of DBT gist might prove useful. First, it could contribute to a radiologist’s assessment of a case. No one would ever suggest a diagnosis based on a 1-s exposure to a DBT movie. However, the gist might be harnessed as one more piece of information in the effort to evaluate a case. In 2-D mammography, it has been shown that a gist signal is present in breast tissue where overt signs of cancer will not appear for several years.^10,26 If the gist signal can be “read” years before other signals, it could serve as a risk factor that could shape how screening was managed, with more frequent screening for those with the risk factor. That is, the gist signal in no way says that the patient has cancer but might suggest greater vigilance.

The second potential use for gist signals is related to gist’s possible role as a detectable risk factor. The gist signal seems like an obvious candidate for detection by a deep learning network. Indeed, efforts to train deep nets to read breast “texture” are steps in this direction.^29,30 Research with human observers might guide the development of new deep nets that could better exploit the global signal in the images.

In summary, this study demonstrates that DBT images, like other medical images, contain a global signal that can be used to classify cases as normal or abnormal at above-chance levels. The ability to detect this signal is acquired with DBT experience though without any explicit training in detecting the signal. More general experience with breast imaging does not appear to correlate with the ability to detect this signal. It remains for future work to determine if this or other global signals in medical images can have clinical utility.

5. Appendix

The details of all positive cases are listed in Table 1 of the Appendix. Table 2 in the Appendix also shows the recall performance across all observers.

Biographies

Chia-Chien Wu is a postdoctoral research fellow at Brigham and Women’s Hospital and Harvard Medical School. His research interests include medical image perception, visual search, and eye tracking.

Biographies of the other authors are not available.

Disclosures

No competing conflicts of interest, financial or otherwise, are declared by the authors.