Pursuing optimal thresholds to recommend breast biopsy by quantifying the value of tomosynthesis

Abstract

A 2% threshold has been traditionally used to recommend breast biopsy in mammography. We aim to characterize how the biopsy threshold varies to achieve the maximum expected utility (MEU) of tomosynthesis for breast cancer diagnosis. A cohort of 312 patients, imaged with standard full field digital mammography (FFDM) and digital breast tomosynthesis (DBT), was selected for a reader study. Fifteen readers interpreted each patient's images and estimated the probability of malignancy using two modes: FFDM versus FFDM + DBT. We generated receiver operator characteristic (ROC) curves with the probabilities for all readers combined. We found that FFDM+DBT provided improved accuracy and MEU compared with FFDM alone. When DBT was included in the diagnosis along with FFDM, the optimal biopsy threshold increased to 2.7% as compared with the 2% threshold for FFDM alone. While understanding the optimal threshold from a decision analytic standpoint will not help physicians improve their performance without additional guidance (e.g. decision support to reinforce this threshold), the discovery of this level does demonstrate the potential clinical improvements attainable with DBT. Specifically, DBT has the potential to lead to substantial improvements in breast cancer diagnosis since it could reduce the number of patients recommended for biopsy while preserving the maximal expected utility.

1. INTRODUCTION

Image-guided core-needle breast biopsy has become an integral part of diagnostic workup for a suspicious mammographic finding. In the United States over 700,000 women undergo breast biopsies per year^{1, 2}. However, most women who have a breast biopsy do not have breast cancer—in fact, approximately 80% of image-guided breast biopsies are benign. Reducing unnecessary biopsies is important for several reasons. Breast biopsy is the most costly per capita component of a breast cancer screening program³. Moreover, breast biopsy can cause side effects such as bleeding, bruising, and infection^{4, 5}. Furthermore, for a patient, waiting days for the results of a breast biopsy appears to affect stress hormone levels just as much as the news of a breast cancer diagnosis⁶.

One strategy of reducing unnecessary biopsies is to improve breast cancer risk estimation. Mammography is currently the standard of care for breast cancer early diagnosis. However, conventional mammography has nontrivial false-positive (recall for additional images or biopsy) and false-negative (missed cancer) rates⁷. Thus, substantial effort is being invested to improve breast cancer risk estimation through a variety of new imaging technologies and practice techniques. Recently, digital breast tomosynthesis, also referred to as “3-D mammography”, has been developed to reduce false-positive and false-negative findings. The feasibility of using tomosynthesis in breast imaging have been demonstrated^{8, 9}, and the U.S. Food and Drug Administration recently approved the first commercial system in clinical use¹⁰ but many questions remain regarding the optimal use.

Attempts to improve breast cancer risk estimation to more accurately target individuals who are most likely to benefit from breast cancer early diagnosis and least likely to experience false positives are being actively pursued¹¹. Nevertheless, another promising strategy to accomplish this goal is to pursue the optimal breast cancer risk threshold that radiologists should use to recommend biopsy. There have been several methods to determine the optimal thresholds for medical diagnosis^12–15. Currently, clinically radiologists typically use a 2% threshold as a probability level of breast cancer above which a breast biopsy is recommended in mammography¹⁶. The American College of Radiology (ACR) developed the Breast Imaging Reporting and Data System (BI-RADS) lexicon which standardizes mammography reporting and provides a guide on mammography audits and performance measures¹⁷. BI-RADS reinforces this 2% threshold level below which biopsy need not be recommended. Furthermore, a recent Markov decision model validates that the 2% threshold currently used for biopsy is reasonable by considering clinical relevant variables².

This study evaluates the performance of digital breast tomosynthesis for breast cancer risk estimation. This study aims to reveal how the biopsy threshold varies to achieve the maximization of the utility of the diagnosis of breast cancer when digital breast tomosynthesis is available in addition to mammography.

2. MATERIALS AND METHODS

2.1 Subjects

A cohort of 312 patients, imaged with standard full field digital mammography (FFDM) and digital breast tomosynthesis (DBT), was selected from a screening group and a biopsy group for a reader study¹⁸. These cases included 51 malignant and 261 benign/negative subjects. Fifteen readers interpreted each patient's images and provided the probability of malignancy in two modes: FFDM versus FFDM + DBT. The reference standard was based on biopsy results or the radiologist's final interpretation if no biopsy was performed.

2.2 Utility analysis in the ROC domain

Receiver Operating Characteristic (ROC) curve analysis is one of the most widely used statistical approaches to characterize the predictive ability of technologies and methods^19–21. It illustrates the performance of a binary classifier system by plotting true positive rate (TPR) vs. false positive rate (FPR), at various threshold settings.

In the ROC domain, after utility is assigned for each category of outcomes (True Negative (TN), False Positive (FP), False Negative (FN) and True Positive (TP)), expected utility of a diagnostic technology or method f is defined as follows.

$$E\left[U\right(f\left)\right]=p\times [U_{\mathit{TP}}\times \mathit{TPR}+U_{\mathit{FN}}\times (1-\mathit{TPR}\left)\right]+(1-p)\times [U_{\mathit{FT}}\times \mathit{FPR}+U_{\mathit{TN}}\times (1-\mathit{FPR}\left)\right]$$

where E[U()] represents expected utility and p is the prevalence of breast cancer. The maximum expected utility (MEU) is defined as expected utility at the operating point where the line with the slope S is tangent to the ROC curve (Figure 1). The MEU occurs at the operating point where a rational radiologist should make clinical decision^{14, 22}. The slope S is determined by utility values of four outcomes and the prevalence of breast cancer, and given by:

$$S=\frac{U_{\mathit{TN}}-U_{\mathit{FP}}}{U_{\mathit{TP}}-U_{\mathit{FN}}}\times \frac{1-p}{p}$$

Figure 1.

Relationship between expected utility (EU) curve and ROC curve.

2.3 Study design

We first generated receiver operating characteristic (ROC) curves for all readers combined using ROCKIT software^{20, 23} based on the probabilities of malignancy in two modes (FFDM, FFDM+DBT). We compared accuracy in terms of the area under ROC curve (AUC) of FFDM+DBT versus FFDM alone.

Then, we assigned utility for each category of outcomes in breast cancer diagnosis as follows.

• We chose TN outcomes as our baseline and assigned a utility of zero.

• We assigned a loss of 4.7 days to the utility of FP based on the literature^{24, 25}.

• We used the University of Wisconsin Breast Cancer Simulation (UWBCS) model²⁶ to estimate the utility of FN as a loss of 2.52 years. The UWBCS model developed as part of the Cancer Intervention and Surveillance Modeling Network (CISNET), is a discrete-event, stochastic simulation model designed to replicate breast cancer epidemiology in the U.S. population. The UWBCS includes a detailed and flexible natural history model that explicitly models breast cancer subtypes. It has been cross-validated against Wisconsin state registry and Surveillance Epidemiology and End Results (SEER) data²⁷. The UWBCS has been previously utilized to provide guidance about the risks and benefits of alternative approaches to breast cancer diagnosis.

• For TP, we assumed that its utility was U_FN × (1-α), 0≤ α ≤1, where α is an unknown parameter representing the overall effectiveness of breast cancer treatment.

We compared MEU of FFDM+DBT with that of FFDM alone by using assigned utilities. We used 0.03 as the prevalence in MEU calculation, which is comparable to the incidence of breast cancer for the diagnostic population²⁸.

Finally, using the baseline 2% biopsy threshold established in the literature for FFDM, we evaluated the threshold for FFDM+DBT such that its expected utility attained the maximum. That is to say, we looked for the optimal threshold for FFDM+DBT such that its related slope on the ROC space was the same as that of FFDM with 2% threshold. We calculated biopsy rate and positive predictive value (PPV) when different thresholds were used. Biopsy rate is the ratio of the number of recommended biopsies to the number of findings. PPV is percentage of all findings that result in a tissue diagnosis of cancer. We also estimated the effects of each threshold on the number of false-negative results, the number of false-positive results, and the number of biopsies avoided.

In addition, we obtained sensitivity and specificity of FFDM at 2% threshold. We also found a threshold for FFDM+DBT when the same sensitivity or specificity was maintained.

3. RESULTS

FFDM+DBT improved accuracy significantly compared with FFDM alone in terms of AUC (0.879 vs. 0.802, p-Value<0.001) (Figure 2). Overall, FFDM +DBT provided higher MEU compared with FFDM alone (Figure 3).

Figure 2.

ROC curves comparing FFDM+DBT (dashed curve) with FFDM (solid curve).

Figure 3.

Difference in MEU between FFDM+DBT and FFDM alone.

When a 2% threshold was used for FFDM, the sensitivity of FFDM was 0.906, the specificity was 0.360, biopsy rate was 0.684, and PPV was 0.218. The slope of the line that is tangent to the ROC curve at the 2% threshold was 0.324. When DBT was involved in the diagnosis in addition to FFDM, the sensitivity was 0.924, the specificity was 0.474, biopsy rate was 0.591, and PPV was 0.256 if the 2.0% was chosen as a biopsy threshold. To achieve the maximum expect utility for FFDM+DBT, the threshold was increased to 2.7%, with which the sensitivity of FFDM+DBT was 0.890 and the specificity was 0.610 (Figure 4). Biopsy rate was 0.414 and PPV was 0.347 when 2.7 % was used as a threshold for FFDM+DBT. FFDM+DBT with the 2.7% threshold avoided 1,263 biopsies but missed 25 more cancers as compared to FFDM with the 2% threshold.

Figure 4.

MEU of FFDM with 2% threshold (solid curve) and FFDM+DBT with 2.7% threshold (dashed curve).

To maintain the same sensitivity as when the 2% threshold was used for FFDM, the threshold was increased to 2.4% for FFDM+DBT and the corresponding specificity was 0.554. To maintain the same specificity, the threshold was reduced to 1.7% for FFDM+DBT and the corresponding sensitivity was 0.949 (Figure 5).

Figure 5.

Sensitivity and specificity for different biopsy thresholds.

4. DISCUSSION

We investigate and describe optimal thresholds for recommending biopsies based on digital breast tomosynthesis for the first time. We find an optimal threshold of 2.7% in FFDM+DBT for recommending biopsy as compared with 2% in FFDM alone in order to achieve maximum expected utility. For FFDM+DBT, a biopsy threshold of 2.7% threshold results in substantially fewer biopsies. Understanding the optimal threshold from a decision analytic standpoint creates the opportunity for development of decision support tools to support and reinforce this threshold in clinical practice to ultimately improve care.

The value of decreasing biopsy rate for FFDM+DBT is immense. Decreased biopsy rate can be directly translated into the benefits of cost reduction in breast cancer diagnosis programs. It can avoid the anxiety and side effects caused by unnecessary biopsies. This study reinforces two strategies for decreasing biopsy rate. One is to improve breast risk estimation, which is demonstrated by reduced biopsy rate from 0.684 to 0.591 when DBT was involved in the diagnosis of breast cancer in addition to FFDM. The other is to pursue the optimal breast cancer risk threshold, which is demonstrated by further reducing biopsy rate to 0.414 when the 2.7% threshold was utilized for recommending biopsy instead of routinely used 2% threshold. Although the benefits of decreasing biopsy rate are significant, sufficient attention has to be paid to the potential harms^{7, 29, 30}. This study found that FFDM+DBT with the 2.7% threshold avoided 1,263 biopsies but missed 25 more cancers as compared to FFDM with the 2% threshold. These results indicate that our decision model based on maximum expected utility is an effective tool to pursue the optimal breast cancer risk threshold by balancing potential benefits and harms in breast cancer diagnosis.

In this study, PPV at 2.7% threshold for FFDM+DBT is superior to that at 2% threshold for FFDM. PPV is a critical measure of the performance of a diagnostic method, as it reflects the proportion of positive test results that are malignant cancers. A high PPV indicates an effective threshold for reducing the ratio of the number of false positive results to the number of true positive results, which may suggest a low biopsy rate for avoiding unnecessary biopsies, medical costs, and patients' emotional stress^{31, 32}.

The recent emphasis on cost-effective medical practice has strengthened the need to seek optimal threshold in ROC curve analysis for diagnostic tests. In this study, we align optimal threshold suggested from clinical community with that induced from utility analysis, and find that the slope for optimal threshold is 0.324. This slope is less than the data from earlier studies^{13, 33, 34}. A large-scale cohort study would be necessary to investigate the cause of the difference since this study uses the outcomes from a small subset of screening population to derive ROC curve, which would significantly affect the characterization of the slope at the optimal threshold. Verifying our methodology in a large cohort study is the subject of ongoing investigation.

There are several limitations in this study. First, we have made an important assumption that for mammography an optimal threshold derived from maximum expected utility accords the clinically used 2% threshold. We believe this assumption is sound since a rational radiologist would seek to maximize the expected utility for recommending breast biopsy¹⁴. However, theoretic derivation and experimental demonstration are limited to support this assumption. In the future, we plan to derive the optimal threshold from utility analysis after quantifying utilities for four outcomes on a large cohort study. We will compare the optimal threshold with the clinically used threshold for validating this assumption. Second, we have combined all outputs of breast cancer probabilities from a multi-reader multi-case reader study to derive ROC curve. We did not consider the complicated correlation among outputs^{35, 36}. Third, age is important in determining an optimal biopsy threshold². In the future, we will use our methodology to determine optimal biopsy thresholds for different age groups.

5. CONCLUSIONS

We find an optimal threshold of 2.7% in FFDM+DBT for recommending biopsy as compared with 2% in FFDM alone. Tomosynthesis has the potential to lead to substantial improvements in breast cancer diagnosis since it could reduce the number of patients recommended for biopsy while preserving the maximal expected utility. In addition, our proposed expected utility model could be used as a general technique to characterize optimal thresholds and to quantify the potential diagnostic value of different modalities.