An investigation into the mammographic appearances of missed breast cancers when recall rates are reduced

Abstract

Objective:

This study investigated whether certain mammographic appearances of breast cancer are missed when radiologists read at lower recall rates.

Methods:

5 radiologists read 1 identical test set of 200 mammographic (180 normal cases and 20 abnormal cases) 3 times and were requested to adhere to 3 different recall rate conditions: free recall, 15% and 10%. The radiologists were asked to mark the locations of suspicious lesions and provide a confidence rating for each decision. An independent expert radiologist identified the various types of cancers in the test set, including the presence of calcifications and the lesion location, including specific mammographic density.

Results:

Radiologists demonstrated lower sensitivity and receiver operating characteristic area under the curve for non-specific density/asymmetric density (H = 6.27, p = 0.04 and H = 7.35, p = 0.03, respectively) and mixed features (H = 9.97, p = 0.01 and H = 6.50, p = 0.04, respectively) when reading at 15% and 10% recall rates. No significant change was observed on cancer characterized with stellate masses (H = 3.43, p = 0.18 and H = 1.23, p = 0.54, respectively) and architectural distortion (H = 0.00, p = 1.00 and H = 2.00, p = 0.37, respectively). Across all recall conditions, stellate masses were likely to be recalled (90.0%), whereas non-specific densities were likely to be missed (45.6%).

Conclusion:

Cancers with a stellate mass were more easily detected and were more likely to continue to be recalled, even at lower recall rates. Cancers with non-specific density and mixed features were most likely to be missed at reduced recall rates.

Advances in knowledge:

Internationally, recall rates vary within screening mammography programs considerably, with a range between 1% and 15%, and very little is known about the type of breast cancer appearances found when radiologists interpret screening mammograms at these various recall rates. Therefore, understanding the lesion types and the mammographic appearances of breast cancers that are affected by readers' recall decisions should be investigated.

INTRODUCTION

Several studies have demonstrated substantial variability in recall rates among radiologists reporting in breast screening programs, with large international variations ranging from 1% to 15.1%.^1–3 Although many countries use the breast imaging reporting and data system (BI-RADS) as a standardized method of reporting mammograms, a considerable variability in assessment is still seen, even when reporting the same mammographic images by different readers.^1,4–6

Many factors influence the difficulty of reaching a correct diagnosis for normal and abnormal cases. Specific mammographic lesion features have been found to significantly contribute to cancer detection, especially lesion conspicuity.⁷ Ikeda et al⁸ found that 22% of missed cancers in the Malmo Screening Trial showed subtle mammographic signs of malignancy, with lesions that present with architectural distortion (AD) being the most challenging malignancy feature for readers to detect. Furthermore, dense breast tissue has been found to be a strong confounder for lesion detection and cases with high mammographic breast density were more likely to be recalled.^9–11

With a large variation in the recommended target recall rates within screening mammography programs internationally,^12–14 very little is known about the mammographic features or breast cancer appearances which are affected by recall decisions at reduced rates. A recent study by Onega et al¹⁵ with 119 radiologists reading 109 screening mammograms found low recall agreement for lesions with ADs and asymmetric density features. Identifying cancer appearances that are more likely to be missed when recall rates are reduced is clinically important because it can inform readers' decisions in recalling females for further assessment. Therefore, the purpose of this study was to investigate which types of mammographic appearances of breast cancer are most likely to be missed when radiologists read at lower recall rates.

METHODS AND MATERIALS

Sample

Institutional ethics approval was granted for this study (project number: 2014/484). Five breast radiologists who reported for BreastScreen New South Wales with 15–26 years of experience participated in this study. The radiologists read between 2000 and 30,000 (median 8000) mammograms each year and spent a median of 10 hours a week reading mammography cases.

Cases

A test set of screening mammograms, comprising of 200 cases, was obtained from the BreastScreen New South Wales digital imaging library. An enriched test set containing 180 normal cases and 20 abnormal cases, with each abnormal case containing a single biopsy-proved malignancy, was obtained. Each mammographic examination consisted of a two-view digital mammogram, a craniocaudal view and a mediolateral oblique of both breasts with a range of lesion conspicuity, from subtle to obvious cancer presentations and a variety of normal mammographic appearances. In 16 cases, the cancer was visible on both mammographic views (craniocaudal view and mediolateral oblique) of a given breast, whereas 4 cases had a visible lesion on either 1 of the mammographic views, which resulted in a total of 36 malignant lesions available for localization. The “truth” locations and mammographic appearances of all malignant cases were identified by an expert radiologist (MR), who is involved in training, quality assessment and clinical policies of BreastScreen NSW and also is responsible for the clinical management of a screening centre. This expert radiologist had access to the biopsy reports and prior images, which assisted in determining the location and the mammographic appearances of the abnormalities based on the Australian Synoptic Breast Imaging Report of the National Breast Cancer Centre that is endorsed by the Royal Australian and New Zealand College of Radiologists.^16,17 Normal cases were validated after 2 years' normal screening follow-up. Lesion descriptors used in this study have been defined in Table 1.

Table 1.

Definition of lesion terms used for classification¹⁷

Lesion abnormalities	Definition
Calcification	Deposition or collections of calcium compounds in breast tissue of sufficient size to be seen on mammograms and malignancy is characterised by size (0.05–0.5 mm), distribution (cluster, multiple cluster or sometimes scattered), pleomorphism and variation of density
Stellate lesion	Spiculations of variable length radiating from a central point or mass. When a central mass is present, it may be small or large and of low, mixed or high density compared with surrounding breast parenchyma
Architectural distortion	Abnormal configuration of the ductal and ligamentous structures of breast parenchyma compared with the remainder of the breast tissue markings; often appears with spiculation, focal retraction, distortion of the parenchymal edge and disorganisation of markings
Non-specific density	Asymmetry of breast tissue seen in one of the breasts, on either one or two mammographic views with poorly defined characteristics of breast density

The categories of mammographic cancers' appearances were: stellate, AD, non-specific density (NSD), mixed appearance of calcification and AD, and stellate and NSD (Figure 1). The mammographic appearances of the cancer lesions were classified into three categories: lesion type, breast density and location of the lesions in the images. Breast density was graded according to the BI-RADS criteria, fourth edition: 1—the breast is almost entirely fat (<25% glandular), 2—there are scattered fibroglandular densities (approximately 25–50% glandular), 3—the breast is heterogeneously dense (approximately 51–75% glandular) or 4—the breast tissue is extremely dense (>75% glandular).

Figure 1.

Examples of lesion features present in this study: (a) stellate mass; (b) mixed features of calcification and architectural distortion; and (c) non-specific density.

Reading sessions

This study was conducted in a laboratory reading environment designed to closely resemble the clinical environment, with all images displayed on the same calibrated, high-specification workstation with 5-MP EIZO Radioforce GS510 medical-grade monitors (Ishikawa, Japan). Readers were required to read all the 200 mammographic cases in three separate reading sessions using different recall rates. At the first reading condition, no numerical percentage recall rate was imposed and the readers were tasked with a “free recall” when interpreting the cases; that is, they could recall as many cases as they believed necessary. In the second condition, the number of mammographic cases that readers could recall was restricted to 30 cases (15%) based on international recall rates, to reflect 15.1% in the USA.³ For the third session, readers were restricted to recall only 20 cases (10%) to align with the first screening recall required by BreastScreen Australia. To reduce any memory effect, each reading session was separated by a minimum of 2 months, and the reading order of images was randomized for each reading session and each reader.

Reading task

During the three reading sessions, readers indicated whether they would recall or not recall the mammographic cases as per their usual clinical practice for BreastScreen Australia. For each recalled case, readers were required to mark the location of the detected lesions on both mammographic views and score them on the scale of one to five (with five being the highest confidence of malignancy) using a custom made recording software. This scoring system is aligned to BreastScreen Australia practice for classifying mammographic lesions; a score of one and two indicated a normal and benign decision, respectively, and a score of three, four or five would be considered as a recall for assessment. Readers were not permitted to exceed their target recall rates at the end of each recall condition; however, if they exceeded during the reading session, the readers were able to scroll back through the cases and alter their decision to ensure that the number of cases recalled aligned with the prescribed target recall rate condition. Readers were also able to digitally manipulate the images including windowing, zooming and panning as in the actual clinical setting. For the purpose of simulating the first screening read, no prior images or clinical history were provided during the three reading sessions. The readers were not aware of the prevalence of abnormal cases in the test set.

Data analysis

Reader performance was assessed by sensitivity and receiver operating characteristic (ROC) area under the curve (AUC). Sensitivity was defined by the proportion of cancers correctly marked by readers. Even though the readers were encouraged to mark a perceived lesion on both views, for analysis purposes, a true-positive was assigned when the reader correctly marked it in one view of the positive breast only. All marked lesions were compared with those contained in the truth table. A significant difference of both metrics was compared across the three recall conditions using a non-parametric Kruskal–Wallis test. The false-positive rate was calculated by the number of false-positive decisions made on normal cases divided by the total number of normal cases.

Further analysis for this study focused on determining whether the detection of any specific cancer types was altered when the recall rates were reduced (Condition 15% and 10%), which narrowed the analysis to the 20 abnormal cases only, and the analysis was performed at a case-based level. Cancer difficulty was scored out of 15, which was the total number of readers (n = 5) multiplied by the number of reading conditions (n = 3). Cancer difficulty was then classified as the sum of lesions that were correctly marked throughout all recall conditions resulting in three difficulty levels as follows;

• (1) lower difficulty: lesion in the case was correctly marked by readers at least 12 times across the 3 reading conditions

• (2) medium difficulty: lesion in the case was correctly marked by readers between 5 and 11 times across the 3 reading conditions

• (3) higher difficulty: lesion in the case that was marked by readers less than five times across the three conditions

RESULTS

For each cancer type, our results demonstrated that readers have higher sensitivity (0.80) and ROC AUC (0.84) when reading at the free recall (mean recall rate of 25.6%) condition as compared with 15% (0.65 and 0.79, respectively) and 10% (0.55 and 0.75, respectively).

Changes in sensitivity at 15% and 10% recall rates were compared against the baseline free recall using ROC AUC. There was a significant decrease in sensitivity at 15% and 10% for NSD (H = 6.27, p = 0.04 and H = 7.35, p = 0.03, respectively) and for mixed features (H = 9.97, p = 0.01 and H = 6.50, p = 0.04, respectively). There was no significant difference in sensitivity at 15% and 10% for stellate lesions (H = 3.43, p = 0.18 and H = 1.23, p = 0.54, respectively) and AD (H = 0.00, p = 1.00 and H = 1.23, p = 0.37, respectively) (Table 2). An average false-positive rate of 0.17 (range 0.12–0.21) was observed for free recall, 0.08 (range 0.08–0.09) for 15% and 0.05 (range 0.05–0.06) for 10%.

Table 2.

Mean values of sensitivity and receiver operating characteristic (ROC) area under the curve (AUC) of each mammographic feature at free recall, 15% and 10% recall rates

Lesion type	Free recall	15% recall	10% recall	p-value
Mean sensitivity
Stellate	0.95	0.90	0.83	0.180
AD	0.80	0.80	0.80	1.000
NSD	0.67	0.47	0.20	0.043a
Mixed features	0.80	0.45	0.30	0.007a
Mean ROC AUC
Stellate	0.93	0.93	0.89	0.541
AD	0.81	0.86	0.88	0.368
NSD	0.75	0.70	0.58	0.025a
Mixed features	0.79	0.68	0.62	0.039a

AD, architectural distortion; NSD, non-specific density.

^aSignificant differences (p < 0.05).

Table 3 shows an analysis of mammographic appearances of cancer cases in relation to case difficulty at free recall, 15% and 10% recall rates. Ten cancer cases were grouped as “lower difficulty” cases with six of the cancers characterized with stellate masses. Cancers with NSD were the most common cancer features found among eight cases in the “medium difficulty” group. The “higher difficulty” cases were characterized with NSD and mixed features of calcification + AD.

Table 3.

Distribution of detection and cancer appearances (lesion type, breast density and lesion location) for each cancer in relation to case difficulty at free recall, 15% and 10% recall rates

Case ID	Number of readers detected cancer for each reading session			Total	Lesion type	Breast density (BI-RADS)
	Free recall	15%	10%
Lower difficulty
MJBL	5	5	5	15	Stellate	>75%
MJCX	5	5	5	15	Stellate	<25%
MJDA	5	5	5	15	Stellate	51–75%
MJEA	5	5	5	15	Stellate	25–50%
MJGR	5	5	5	15	Stellate	25–50%
MJDH	5	5	5	15	Stellate	51–75%
MJCQ	5	5	4	14	NSD	51–75%
MJEG	5	4	3	12	Calcifications + AD	25–50%
MJBJ	5	3	4	12	AD	25–50%
MJHD	3	5	4	12	AD	<25%
Medium difficulty
MJHJ	4	3	3	10	Stellate	51–75%
MJAS	5	1	3	9	Calcifications + AD	>75%
MJBK	3	3	2	8	NSD	51–75%
MJCR	4	3	1	8	Stellate	25–50%
MJDU	3	3	0	6	Stellate + NSD	51–75%
MJHH	4	2	0	6	NSD	25–50%
MJHK	3	3	0	6	NSD	25–50%
MJEB	4	1	0	5	NSD	51–75%
High difficulty
MJBG	3	1	0	4	Calcifications + AD	51–75%
MJCF	1	1	0	2	NSD	>75%

AD, architectural distortion; BI-RADS, breast imaging reporting and data system; NSD, non-specific density.

In this study, cancers characterized with stellate mass features were most likely to be recalled by all five readers regardless of any recall conditions. At the 15% recall condition, cancer with mixed features of calcification + AD (e.g. MJAS) showed the highest reduction in recall decisions (from five to one), followed by cancers with NSD (from four to one). When the recall rate was further reduced to 10%, six cancers were less likely to be recalled, with all of these cases having been recalled at “free recall” and 15% recall, but missed by all readers at the 10% recall condition. NSD was found to be the most common feature that was missed by all readers at the 10% recall, followed by cancers with mixed features of calcifications + AD and stellate + NSD. It is noted that two cancers with AD features showed an increase in detection at 15% (from three to five) and 10% recalls (from three to four).

When considering the lesion type and mammographic breast density together, the analysis revealed most of the cancers in the lower difficulty group were in cases with low mammographic density (≤50% glandular, BI-RADS 1 or 2) (Table 3). Conversely, a greater number of cancers in the medium difficulty and higher difficulty group were located in cases with high mammographic density (≥51% glandular, BI-RADS 3–4).

DISCUSSION

This study provides a unique perspective on the variability in mammographic interpretation by evaluating the mammographic appearances/features that were more likely to be recalled when recall rates are reduced. In agreement with our findings, a recent study by Onega et al¹⁵ has shown that asymmetric densities contributed to low agreement for recall cases.

In this study, readers were able to recall cancers with stellate masses regardless of any recall condition. Retrospective analysis of the 20 cancer cases has demonstrated that cancers characterized with NSD were less likely to be recalled as soon as the readers were asked to reduce their recall rate to 15% and 10% (Figure 2). NSD was then followed by cancers with mixed features, which in this study were calcifications + AD and stellate + NSD. This is likely due to their subtle and indirect signs of malignancy. These subtle features of malignancy have been recognized in previous studies as being frequently missed by readers.^8,18–20 A study by Duncan et al,²⁰ when reviewing the mammographic features of 112 incidental screen-detected cancers, found greater asymmetric density and parenchymal deformity in the missed cancers than in those detected. Readers may have also interpreted irregular opacities in NSD as benign in breasts composed of tissue with irregular densities which do not warrant recall.^21,22

Figure 2.

All three cancers above characterized with non-specific density but with variability in lesion detectability and level of difficulty at reduced recall rates; (a) MJCQ: lower difficulty; (b) MJBK: medium difficulty; and (c) MJCF: higher difficulty.

Considering that the features of cancer lesions are varied, past research has shown that cancers that present with mixed appearances (more than one mammographic feature) are more likely to be missed and readers are more susceptible to omission error when cancers display mixed mammographic appearances.²³ Our study supports this, whereby the mammographic features of stellate lesions were associated with spiked linear extensions radiating outwards and ill-defined spicules from the central lesion which indicate the demosplastic reaction of breast cancer into surrounding tissue.²⁴ The distinctive features of stellate lesions have a positive-predictive value of 84–91% and are most common mammographic features of invasive breast cancer.²⁵ These features were easily recognized by our readers and were recalled for further assessment. However, when stellate features were associated with other mammographic features such as NSD with ill-defined borders or AD, these lesions became less suspicious and hence were not recalled at lower rates. Similarly, cancers characterized by combination features of calcifications and AD are less likely to be recalled than cancers with AD alone, although in a study by Craft et al,²⁶ up to 48% of cancers with calcifications were found associated with malignancy. Calcification has also been found to have uncertain malignant potential when associated with atypical breast lesions.²⁷ It is interesting to note that we also found inconsistency in readers' decisions when recalling cancers characterized with AD. Unlike other cancer features, the fine linear structures of AD normally seen on mammograms can resemble superimposed normal breast tissue, but it also can appear as a stellate shape and an accompanying feature of other abnormalities, which turn out to be a breast cancer. A recent study on the ability of readers to detect AD by Suleiman et al²⁸ has shown that readers had greater difficulty detecting cancers with AD than other cancer features, with significantly lower sensitivity and ROC AUC results. With the limited number of cases that were allowed to be recalled in the conditions of 15% and 10% recall rates, several features of AD such as trabecular thickening, which disrupt the normal breast tissue pattern, may also lead to uncertainty in readers' decision making, resulting in recalling for further assessment.^15,28

Other perceptual factors such as mammographic density may have an attractor and distractor effect that also influenced recall decisions.^10,29–32 Our high difficulty cases occurred in conjunction with high mammographic density. In addition, higher mammographic density may also increase the likelihood of cancers being missed in mammography.^10,29–32 In this study, cancers within cases of low mammographic density (BI-RADS 1 and 2) were more likely to be recalled and cancers present in cases with higher mammographic density (BI-RADS 3 and 4) were likely to be missed. With some limitations specific to two-dimensional mammography, it is possible that lesions with subtle malignancy signs such as NSD might be obscured by dense parenchyma. This finding is concurrent with earlier findings by Bird et al¹⁹ who reported that 24% of missed cancers in screening mammography were due to higher mammographic density. Additional views such as coned compression or magnifications, which give better contrast and spatial detail on the targeted area, were not available to our readers and may have improved recognition of the lesions; however, these views are not part of a standard screening protocol. Further research in this area employing eye-tracking analysis may aid in understanding the visual search patterns of readers making their decisions under strict recall conditions. An additional area for further research may include the role of training to improve the identification of more difficult lesions, including the effect of experience upon consistent recalling of certain lesion types.

Reflecting on the clinical significance of the results of this study, the fact that the readers continued to recall stellate lesions even at reduced recall rates may be due to the biological significance of these findings. This is because stellate lesions are often recognized as highly likely to be malignant³³ but not often associated with a high histological grade.³⁴ The correlation between mammographic features and histological grade was evident in a study by De Nunzio et al³⁴ when investigating 212 patients with invasive cancer which found that lesions presenting as stellate had significant correlation with low histological grades, which suggests stellate as a good prognostic feature and as associated with reduced breast cancer mortality. This was supported later by findings from Alexander et al,³³ where patients with this type of lesion had better survival rate (>95%) than other mammographic features. Unlike stellate, cancer characterized with NSD was associated with high histological grades and larger size (up to 90 mm). As high-grade cancer is faster growing compared with low-grade cancer, this may give a shorter window for this type of cancer to be detected. Such factors may have been taken into account by the radiologists in their decisions at strict recall conditions, as they may improve the survival rate of the screened females. Conversely, other lesions such as calcifications, mixed features and NSD have high likelihood of being benign lesions,³⁵ thus perhaps justifying the reduced need to recall these lesions.

This study was conducted in a laboratory environment rather than in a clinical setting which may have affected the readers' reporting pattern. A previous comparison study by Gur et al³⁶ with nine experienced radiologists demonstrated higher recall rates when the reading took place in the laboratory as compared with in-clinic reading. On the other hand, in our study, radiologists were “forced” to recall the most significant cases that required further assessments and were not allowed to exceed a prescribed recall rate. In some cases, although the radiologists found more cases might need to be recalled, they had to in effect “let go” some of the cases due to the strict recall rule. In addition, it may be argued that a relatively low number of each cancer type was presented in this study. However, we believe that it has given important insights of the type of breast cancer that affect upon readers' recall decisions. A study with a greater number of cancer types and readers will minimize biasing in results.

CONCLUSION

This study provides important insights into the types of cancer cases that contribute to the greatest uncertainty or are missed at low recall rates. Cancers with a stellate mass were more easily detected and were likely to continue to be recalled, even at lower recall rates. Lesions with NSD and mixed features were most likely to be recalled at reduced recall rates. By understanding which cancer features are likely to be missed, a dedicated training intervention can be developed to improve readers' performance when considering an optimal recall rate.