Auditing a Breast MRI Practice: Performance Measures for Screening and Diagnostic Breast MRI

Abstract

Objective

Breast magnetic resonance imaging (MRI) is increasingly being used for both screening and diagnostic purposes. While performance benchmarks for screening and diagnostic mammography have been published, performance benchmarks for breast MRI have yet to be established. The purpose of our study was to comprehensively evaluate breast MRI performance measures, stratified by screening and diagnostic indications, from a single academic institution.

Subjects and Methods

Institutional review board approval was acquired for this HIPAA compliant study. Informed consent was not required. Retrospective review of our institutional database identified all breast MRI examinations performed from 4/1/07 to 3/31/08. After application of exclusion criteria, the following performance measures for screening and diagnostic indications were calculated: cancer detection rate, positive predictive values (PPV), and abnormal interpretation rates.

Results

The study included 2444 examinations, 1313 for screening and 1131 for diagnostic indications. The cancer detection rates were 14 per 1000 screening breast MRI examinations and 47 per 1000 diagnostic examinations (p-value < 0.00001). The abnormal interpretation rate was 12% (152/1313) for screening and 17% (194/1131) for diagnostic indications (p-value = 0.00008). The positive predictive values of MRI were lower for screening (PPV1 = 12%, PPV2 = 24%, PPV3 = 27%) compared to diagnostic indications (PPV1 = 28%, PPV2 = 36%, PPV3 = 38%).

Conclusion

Breast MRI performance measures differ significantly between screening and diagnostic MRI indications. Medical audits for breast MRI should calculate performance measures for screening and diagnostic breast MRI separately, as recommended for mammography.

Introduction

Since 1992, the Mammography Quality Standards Act (MQSA) has required a mandatory audit of mammography practices. Published in 2003, the fourth edition of the Breast Imaging Reporting and Data System (BI-RADS) atlas suggests additional data to collect and calculate for clinically meaningful mammography audits [1]. More recently, Congress commissioned a study from the Institute of Medicine (IOM) to assess whether additional steps could be taken to further improve breast imaging quality standards. The Institute of Medicine's “Improving Breast Imaging Quality Standards” suggests revision of the MQSA audit to require calculation of abnormal interpretation rate, cancer detection rate, and positive predictive value 2 (PPV2) with each calculation stratified by diagnostic or screening mammography [2]. The IOM report also recommended increased standardization and accreditation of breast magnetic resonance imaging (MRI) practices [2]. To standardize appropriate utilization of breast MRI, the American Cancer Society (ACS) published guidelines for breast MRI screening indications in 2007, and the American College of Radiology (ACR) has outlined recommendations for the use of breast MRI in both screening and diagnostic settings [3, 4]. In addition, accreditation of breast MRI facilities by the ACR began in 2010 [4]. The ACR breast MRI accreditation program requires each facility to establish and maintain a medical outcomes audit program; however, the audit requirements for breast MRI have not yet been formally outlined, as has been done for mammography [1, 4-8].

Although no formal audit requirements for screening or diagnostic breast MRI yet exist, some reports of various breast MRI performance benchmarks, including abnormal interpretation rate, cancer detection rate, and positive predictive values, have been published [9-18]. However, these performance measures are often reported individually, for only screening examinations, or for an aggregate cohort of screening and diagnostic examinations. Because performance measures are affected by the prevalence of cancer, calculations from an aggregate screening and diagnostic cohort should be interpreted with caution [19]. The purpose of our study was to comprehensively evaluate breast MRI performance measures, stratified by screening and diagnostic indications, from a single academic institution.

Subjects and Methods

Study Population

This study was approved by the institutional review board, and its methodology is in compliance with federal HIPAA regulations. A retrospective search of our institution's prospectively populated breast imaging database identified 2,596 breast MRI examinations performed between April 1, 2007 and March 31, 2008. From the preliminary list of 2,596 studies, 152 (5.9%) examinations were excluded for the following reasons: 92 breast MRI deemed non-diagnostic for technical reasons (inadequate fat saturation, contrast extravasation, etc.), 23 non-gadolinium enhanced MRI performed for silicone implant integrity assessment, 4 breast MRI aborted at patient's request, 23 studies with missing dictations in the electronic medical record, and 10 studies performed in patients who were lost to follow up. For each of the 2,444 breast MRI included in the study, the BI-RADS assessment was obtained from the imaging report in the electronic medical record [1]. An examination was assigned an overall BIRADS assessment according to the most actionable BI-RADS lesion category (BI-RADS 5 > 4 > 0 > 3 > 6 > 2 > 1).

Breast MRI Technique and Interpretation

Breast MRI technique was refined over the study period, but adherence to the basic imaging principles detailed in the following sequences was maintained. At our institution, breast MRI examinations were performed with the patient prone in a 1.5 T magnet with a dedicated breast coil (GE HD 8-channel Breast Array Coil, GE Healthcare). The standard bilateral acquisition protocol included a multi-planar localizing sequence, axial T1-weighted, axial fat-suppressed T2-weighted, and axial and sagittal T1 weighted fat suppressed sequences performed before and after injection of 20 mL gadopentetate dimeglumine (Magnevist, Bayer HealthCare Pharmaceuticals, Wayne, New Jersey). Bilateral axial post gadolinium acquisitions were centered at 1.5 minutes, 3 minutes, and 4.5 minutes, with the bilateral sagittal post gadolinium acquisition centered at 7 minutes. All MRI examinations were post-processed to generate subtraction images and lesion kinetic analysis using commercially available software (CADstream, Confirma).

Each breast MRI examination was interpreted by one of ten board-certified radiologists specializing in breast imaging. Each radiologist had greater than 3 years of breast MRI experience at the time of interpretation.

Data Extraction

For each breast MRI examination, the following data were extracted from the electronic database or electronic medical record: patient age, clinical indication for breast MRI, number and laterality of findings, available clinical follow-up, results of any additional imaging workup at our institution, and biopsy type and pathology, if applicable.

The authors coded each breast MRI study as screening or diagnostic based on the information available from the electronic database and the patient's electronic medical record. Screening examinations were defined as those ordered for the following indications: known BRCA mutation carrier, prior history of breast cancer after completion of treatment, family history of breast cancer, prior breast biopsy with pathology demonstrating atypia or lobular carcinoma in situ, or prior mediastinal radiation. Additional indications assigned to the screening category included patients without a current breast symptom whose medical records indicated that breast MRI was performed secondary to dense breast tissue, fibrocystic breast tissue, a prior history of a papilloma, or a prior history of ovarian cancer. Diagnostic examinations were defined as those performed to evaluate extent of disease in the setting of newly diagnosed breast cancer, examinations performed for additional evaluation of a clinical or imaging finding, or examinations for short interval follow up of probably benign findings on a prior breast MRI. When the breast MRI indication was uncertain, the case was discussed among the authors, and a consensus indication was determined.

Benign versus malignant outcome was determined for each patient with a breast MRI examination performed during the study period. Outcomes were assigned based upon biopsy results, imaging follow up, or clinical surveillance, performed within one year of the breast MRI examination. If a biopsy was performed at our institution or recorded in the electronic medical record, the method of biopsy, percutaneous and/or excisional, was extracted in addition to the pathology results. During the study period, excisional biopsy was routinely recommended for high risk lesions identified on percutaneous biopsy. High risk lesions included flat epithelial atypia, atypical ductal hyperplasia, lobular neoplasia, atypia of any other type, phyllodes tumor, radial scar, or papilloma. Final histopathologic outcomes were assigned as benign (no upgrade to malignancy) or malignant (upgrade to malignancy) based upon the excisional biopsy pathology. Ductal carcinoma in situ (DCIS), invasive lobular carcinoma, and invasive ductal carcinoma were categorized as malignant.

Statistical Analyses

Breast MRI assigned BI-RADS categories 0, 4, or 5 were defined as positive examinations. Breast MRI with suspicious findings were defined as those with a final BI-RADS 4 or 5 after additional imaging evaluation or work-up. A true positive case was defined as one with a diagnosis of malignancy within 1 year after a positive exam. Positive predictive values were calculated based upon BI-RADS recommendations for mammography, using the following definitions as recommended by the American College of Radiology: PPV 1 = true positives divided by positive exams; PPV 2 = true positives divided by exams with suspicious findings for which biopsy was recommended; and PPV 3 = true positives divided by biopsies performed [1]. PPV3 calculations were performed at the examination level, rather than the lesion level. Cancer detection rate was calculated as the number of cancers correctly detected at breast MRI per 1,000 examinations [1]. Abnormal interpretation rate was calculated as the number of positive examinations divided by the total number of examinations [1]. Each calculation was performed for the entire patient cohort and also stratified by screening or diagnostic indication.

Differences in cancer detection rate and abnormal interpretation rate were compared for screening and diagnostic indications using a chi square test. A p-value of less than 0.05 was considered statistically significant. 95% confidence intervals were calculated for cancer detection rates and abnormal interpretation rates. Because positive predictive value depends upon the prior probability of cancer, p values were not calculated to compare the differences in PPVs for screening and diagnostic indications [19].

Results

Of 2,444 breast MRI included in the study, the 1,313 screening examinations comprised 54% of the breast MRI performed. The most common reasons for screening MRI included a prior personal history of breast cancer and a family history of breast cancer (Figure 1). The 1,131 (46%) diagnostic breast MRI were most commonly performed for short term follow up of a previously described breast MRI lesion, problem solving, or evaluation of extent of disease in a new breast cancer diagnosis (Figure 2).

Fig 1.

Indications for screening breast MRI

Fig 2.

Indications for diagnostic breast MRI

Abnormal interpretation rate was 12% (95% CI, 10%-13%) for screening and 17% (95% CI, 15%-19%) for diagnostic indications (Table 1), a statistically significant difference (p value =0.00008). Following inter-modality correlation (final BI-RADS 4 or 5 assessment after inter-modality correlation), the biopsy recommendation rate was 6% (75/1313, 95% CI, 4%-7%) for screening and 13% (149/1131, 95% CI, 11%-15%) for diagnostic indications (p-value < 0.00001), with an overall rate of 9% (224/2444, 95% CI, 8%-10%). Following inter-modality correlation, 277 (21%) screening and 314 (28%) diagnostic breast MRI examinations were assigned a final BI-RADS 3 assessment.

Table 1.

Performance measures for screening and diagnostic breast MRI

	Screening	Diagnostic	Overall
Patient Age Mean (standard deviation)	51.1 yrs (10.3)	50.6 yrs (10.6)	50.9 yrs (10.5)
Total # of MRIs performed	1,313	1,131	2,444
# Positive MRIs (BI-RADS 0,4,5)	152	194	346
Final BI-RADS 4 or 5	75	149	224
Biopsies (lesion level)	77	172	249
Biopsies (exam level)	67	139	206
Cancer diagnoses (lesion)	18	61	79
Cancer diagnoses (exam)	18	54	72
Cancer detection rate (per 1000 exams)	14	47*	29*
Abnormal interpretation rate	152/1313 (12%)	194/1131 (17%)	346/2444 (14%)
PPV1	18/152 (12%)	54/194 (28%)	72/346 (21%)
PPV2	18/75 (24%)	53/149 (36%)*	71/224 (32%)*
PPV3 (exam level)	18/67 (27%)	53/138 (38%)*	71/205 (35%)*

^*One patient underwent a diagnostic breast MRI due to her current history of breast cancer. Her study demonstrated non mass-like enhancement in the contralateral breast, for which correlative mammography was recommended. Inter-modality correlation was negative, so her final BI-RADS category was probably benign (BI-RADS 3). The patient elected to undergo a prophylactic contralateral mastectomy, with pathology demonstrating an invasive carcinoma in the same quadrant as the non mass-like enhancement. Therefore, this patient is excluded from the cancer detection rate, PPV2, and PPV3 calculations.

Table 2 details the pathologic diagnoses in the 249 MR-detected lesions for which biopsies were recommended. Breast MRI performed for diagnostic indications accounted for 69% of the lesions recommended for biopsy, including 77% (61/79) of the biopsies resulting in a diagnosis of malignancy (Table 2). The majority of MR-detected malignancies were invasive carcinomas with no statistically significant difference in invasive cancer size for screening versus diagnostic indications (p-value = 0.62).

Table 2.

Pathologic diagnoses in 249 suspicious breast MRI lesions

	Screening	Diagnostic	Overall
Total Lesions (%)	77	172
Malignant	18	61	79
Invasive ductal carcinoma	11/18 (61%)	27/61 (44%)	38/79 (48%)
Ductal carcinoma in situ	7/18 (39%)	20/61 (33%)	27/79 (34%)
Invasive lobular carcinoma	0/18 (0%)	3/61 (5%)	3/79 (4%)
Other*	0/18 (0%)	11/61 (18%)	11/79 (14%)
Invasive Cancer Size (mm)†
Mean (Standard Deviation)	15 mm (13)	15 mm (15)	15 mm (13)
Benign	59	111	170
Fibrocystic changes, stromal fibrosis, or sclerosing adenosis	18/59 (31%)	25/111 (23%)	43/170 (25%)
Fibroadenoma	21/59 (36%)	33/111 (30%)	54/170 (32%)
Pseudoangiomatous stromal hyperplasia	3/59 (5%)	15/111 (14%)	18/170 (11%)
Atypia including lobular neoplasia	3/59 (5%)	8/111 (7%)	11/170 (6%)
Hamartoma	5/59 (8%)	9/111 (8%)	14/170 (8%)
Radial scar	3/59 (5%)	3/111 (3%)	6/170 (4%)
Papilloma	0/59 (0%)	3/111 (3%)	3/170 (2%)
Phyllodes tumor	0/59 (0%)	0/111 (0%)	0/170 (0%)
Post radiation changes	2/59 (3%)	4/111 (4%)	6/170 (4%)
Other‡	4/59 (7%)	11/111 (11%)	15/170 (9%)

^*Included phyllodes with sarcoma, lymph node with metastatic carcinoma, mixed invasive ductal and lobular carcinoma, and lesions missing data on the specific type of malignant diagnosis

^†Data only available for 75 lesions

^‡Included other benign pathology results and lesions missing data on the specific type of benign diagnosis

The cancer detection rate was significantly lower for screening (14 per 1000 breast MRI, 95% CI 7 -20 per 1000 breast MRI) compared to diagnostic indications (47 per 1000 breast MRI, 95% CI 35 – 59 per 1000 breast MRI) (p-value <0.00001). Positive predictive values were also lower for screening than diagnostic indications (Table 1). PPV1 (cancers detected after positive MRI) was 12% for screening, 28% for diagnostic indications, and 21% overall. PPV2 (cancers detected after biopsy recommended) was 24% for screening, 36% for diagnostic indications, and 32% overall. PPV3 (cancers detected after biopsy performed) was 27% for screening, 38% for diagnostic indications, and 35% overall (Table 1).

Discussion

In this study, we adapted calculations from the MQSA mammography audit to comprehensively analyze performance measures for all breast MRI examinations performed during a single calendar year at a large academic center. This study is one of the largest studies of MRI performance in clinical practice rather than in a clinical trial setting. Because cancer detection rate and positive predictive values are inextricably linked, a single performance measure viewed in isolation may not be meaningful. Another strength of our study is the joint reporting of multiple breast MRI performance measures. Furthermore, our study demonstrates that breast MRI performance measures vary significantly when stratified by screening versus diagnostic indication. Published studies of mammography performance have reported significantly different performance benchmarks for screening and diagnostic mammography, and our study extends those findings to non-mammographic breast imaging [20-23]. Our results of significantly different MRI performance for clinical and diagnostic indications suggests that stratified, rather than aggregate, analyses of performance measures will provide more accurate performance estimates to guide the application of MRI in clinical practice.

In our study, abnormal interpretation rates of 12% and 17% for screening and diagnostic indications, respectively, are consistent with the published literature, which report abnormal interpretation rates for breast MRI ranging from 8-17% for screening MRI and up to 22% for an aggregate screening and diagnostic population [9, 12, 13, 15]. For mammography, abnormal interpretation rate calculations differ for screening and diagnostic indications. Positive screening mammograms include those assigned to BIRADS categories 0, 4, and 5, whereas positive diagnostic mammograms include only BIRADS 4 and 5 because the use of BI-RADS 0 is discouraged in diagnostic mammography [1]. However, utilization of the BI-RADS 0 category differs in mammography and breast MRI, and this difference affects the abnormal interpretation rate calculations. For breast MRI performed at our institution during the study period, the BI-RADS 0 category was used for both screening and diagnostic MRI examinations when the interpreting radiologist recommended inter-modality correlation for a finding identified on MRI. Our usage of BI-RADS 0 during the study period reflected recommendations in the first edition of the BI-RADS MRI atlas published in 2003, which defined BI-RADS 0 as “A recommendation for additional imaging evaluation includes repeating MRI with satisfactory technique, obtaining information from other imaging modalities (mammographic views, ultrasound, etc.), or correlation with prior breast history.” [1]. Therefore, our abnormal interpretation rate calculations presented in Table 1 include BI-RADS 0, 4, and 5 examinations for both screening and diagnostic breast MRI indications. However, the use of BI-RADS 0 is now discouraged for screening or diagnostic breast MRI when inter-modality correlation is recommended [24]. As a result, a contemporary definition of abnormal interpretation rate for screening and diagnostic breast MRI would describe a “positive exam” as one categorized as a BI-RADS 4 or 5. Therefore, in our study, we also calculated abnormal interpretation rates for breast MRI including only examinations with a final BI-RADS 4 or 5 assessment after inter-modality correlation. Using this contemporary definition, the abnormal interpretation rate was 6% for screening and 13% for diagnostic indications. There is discussion that the new edition of the BI-RADS atlas, which remains unpublished at this time, may include BI-RADS 3 examinations as a “positive exam”, in addition to BI-RADS 0, 4, and 5. However, the published version of the BI-RADS atlas, in widespread clinical use at the time of exam interpretation and statistical analyses for this study, currently categorizes a BI-RADS 3 examination as negative [1]. Therefore, in our study analyses and in our clinical practice, BI-RADS 3 breast MRI examinations are currently categorized as negative for auditing purposes. All published studies to date regarding individual performance measures for breast MRI have similarly categorized “positive” and “negative” examinations per the published BI-RADS atlas, permitting the reader to compare our results with the prior literature. Because BI-RADS 3 examinations may be included as “positive” examinations in the upcoming BI-RADS 5 atlas, we included our data regarding the number of BIRADS 3 assessments for both screening and diagnostic breast MRI following inter-modality correlation. The percentage of final BI-RADS 3 assessments was high in our data set, although our BI-RADS 3 assessments over the preceding two years have been low (2.6 – 4.7%, unpublished data). This is consistent with published literature demonstrating that the frequency of BI-RADS 3 assessments decreases with prevalent breast MRI examinations, advances in image quality and interpretation, and radiologist experience [25-29].

Compared to mammography alone, screening breast MRI in high risk women demonstrates an incremental cancer detection rate of 8 to 67 cancers per 1000 patients [12, 13, 30]. Our cancer detection rate for diagnostic indications (48 per 1000) was significantly higher than for screening indications (14 per 1000), as expected due to the higher prior probability of breast cancer in the diagnostic group. Our results correlate with the higher cancer detection rates in diagnostic versus screening mammography [20, 23]. Our positive predictive value calculations for screening, diagnostic, and combined indications are also consistent with previously published results. Breast MRI PPV1 has been reported as high as 50% for combined screening and diagnostic breast MRI and 24% for screening alone [9, 13, 15, 18]. Previously published breast MRI PPV2 (biopsy recommended) is 42% overall and 22% in a screening population [9, 13]. Prior studies of PPV3 (biopsy performed) for breast MRI report ranges of 20-61% overall, 10-25% for screening indications, and 28-42% for diagnostic indications [9-17].

Different patient populations likely result in different distributions of screening and diagnostic breast MRI indications. In screening and diagnostic mammography, performance measures vary substantially by exam indication due to different prior probabilities of breast cancer in these patient groups. For example, diagnostic mammograms performed to evaluate areas of clinical concern have higher PPVs than examinations performed for short interval follow-up or to evaluate screen-detected abnormalities [20]. Our study's inclusion of overall and stratified cancer detection rates, as well as the distributions of screening and diagnostic exam indications, should enable the reader to evaluate the degree to which our patient population differs from his or her own patient population. The relatively smaller sample size of our single institution study precluded analyses further stratified by individual screening or diagnostic exam indications (e.g. family history of breast cancer, prior history of breast cancer, or short interval follow up examination).

Limitations of this study include the reporting of breast MRI performance measures from a single tertiary care center, which may limit generalizability of the observed findings. Compared to the remainder of the United States, patients in Massachusetts are more likely to be white and have health insurance [31]. Another limitation of our study is our inability to confirm tissue diagnoses of cancer in patients who may have been diagnosed outside of our institution because our patient population is not tracked within a cancer registry. However, greater than 90% of women with a recommendation for biopsy returned to our institution to undergo the biopsy, suggesting that any effect on our analyses from missing pathology results is likely to be minor. An additional limitation of our study is the lack of stratified analyses for incident versus prevalent examinations. Prevalent examinations have higher abnormal interpretation rates compared to incident examinations in previously published reports of screening breast MRI examinations [12].

In summary, we comprehensively analyzed performance measures for breast MRI and identified significant differences in cancer detection rate, abnormal interpretation rate, and PPVs, when stratified by screening and diagnostic indications. We suggest that audits of breast MRI examinations incorporate separate performance benchmarks for screening and diagnostic breast MRI, as recommended for mammography.

Take-Home Points.

1. Performance measures, including positive predictive value, abnormal interpretation rate, and cancer detection rate, vary significantly for screening and diagnostic breast MRI examinations.

2. Comprehensive audits of breast MRI examinations should incorporate separate performance benchmarks for screening and diagnostic breast MRI, as recommended for mammography.