Test-Retest Repeatability and Reproducibility of ADC Measures by Breast DWI: Results from the ACRIN 6698 Trial

Abstract

Background:

Quantitative diffusion-weighted imaging (DWI) MRI is a promising technique for cancer characterization and treatment monitoring. Knowledge of the reproducibility of DWI metrics in breast tumors is necessary to apply DWI as a clinical biomarker.

Purpose:

To evaluate the repeatability and reproducibility of breast tumor apparent diffusion coefficient (ADC) in a multi-institution clinical trial setting, using standardized DWI protocols and quality assurance (QA) procedures.

Study Type:

Prospective.

Subjects:

In all, 89 women from nine institutions undergoing neoadjuvant chemotherapy for invasive breast cancer.

Field Strength/Sequence:

DWI was acquired before and after patient repositioning using a four b-value, single-shot echo-planar sequence at 1.5T or 3.0T.

Assessment:

A QA procedure by trained operators assessed artifacts, fat suppression, and signal-to-noise ratio, and determine study analyzability. Mean tumor ADC was measured via manual segmentation of the multislice tumor region referencing DWI and contrast-enhanced images. Twenty cases were evaluated multiple times to assess intraand interoperator variability. Segmentation similarity was assessed via the Sørenson-Dice similarity coefficient.

Statistical Tests:

Repeatability and reproducibility were evaluated using within-subject coefficient of variation (wCV), intraclass correlation coefficient (ICC), agreement index (AI), and repeatability coefficient (RC). Correlations were measured by Pearson’s correlation coefficients.

Results:

In all, 71 cases (80%) passed QA evaluation: 44 at 1.5T, 27 at 3.0T; 60 pretreatment, 11 after 3 weeks of taxanebased treatment. ADC repeatability was excellent: wCV = 4.8% (95% confidence interval [CI] 4.0, 5.7%), ICC = 0.97 (95% CI 0.95, 0.98), AI = 0.83 (95% CI 0.76, 0.87), and RC = 0.16 * 10⁻³ mm²/sec (95% CI 0.13, 0.19). The results were similar across field strengths and timepoint subgroups. Reproducibility was excellent: interreader ICC = 0.92 (95% CI 0.80, 0.97) and intrareader ICC = 0.91 (95% CI 0.78, 0.96).

Data Conclusion:

Breast tumor ADC can be measured with excellent repeatability and reproducibility in a multi-institution setting using a standardized protocol and QA procedure. Improvements to DWI image quality could reduce loss of data in clinical trials.

Level of Evidence:

2

Technical Efficacy:

Stage 1

MAGNETIC RESONANCE IMAGING (MRI) has become a standard method to assess breast cancer in diagnostic, high-risk screening and treatment response scenarios. The adoption of neoadjuvant, or preoperative, chemotherapy (NAC) as standard treatment for locally advanced invasive breast cancer has provided an opportunity to use serial MRI studies for in vivo observation of changes in the tumor to assess treatment response. Results from the ACRIN 6657 clinical trial^1,2 demonstrated the value of MRI-derived metrics for prediction of both pathological and survival outcomes. The primary MRI technique for assessing breast cancer is dynamic contrast-enhanced (DCE) imaging, which characterizes tissue vascularity. However, growing evidence supports diffusion-weighted imaging (DWI), a functional imaging technique reflecting water diffusion properties in tissue, as a supplemental and/or alternative technique to DCE.^3–12 Specifically, the apparent diffusion coefficient (ADC) measured by DWI reflects water mobility impeded by cellular constituents and interstitial tortuosity, and it has shown promise as an imaging biomarker to measure early tumor response to cytotoxic treatment.¹³

In order to utilize ADC as a reliable biomarker for monitoring therapeutic effects, the underlying variability of the measurement must be understood. In addition to patient physiologic factors (eg, menstrual phase, breast density, and menopausal status), technical sources of variability include both those more accessible to the investigator’s control, such as the MRI protocol and analysis methods employed, and those more difficult to control such as variations in patient positioning, patient motion, and intra- and interoperator variability for nonautomatic analysis schemes. Previous single-site studies performed in relatively small numbers of subjects have investigated the repeatability and reproducibility of breast ADC measures in normal^14–18 and cancerous^{15,16,19–21} tissue. Within-subject coefficients of variation ranged from 5–11%, and reproducibility interoperator intraclass correlation coefficients (ICC) varied widely from 0.47 to >0.9. These studies leave open the question of the reproducibility of breast tumor ADC measurements, and hence their overall value in the assessment of treatment response, when applied in multicenter clinical trials. The lack of appropriate reproducibility data was cited by QIBA (The Radiology Society of North America’s Quantitative Imaging Biomarkers Alliance) at the time of the initial draft as the primary reason for excluding breast from the 2017 QIBA Profile for DWI,²² stating that several organs including the breast: “…have been excluded for the time being due to lack of sufficient literature (test-retest data from a total of ~35 subjects, either from a single publication or in total from multiple manuscripts) support.”

The American College of Radiology Imaging Network ACRIN 6698 Trial^3,23 is a substudy of the multi-institution I-SPY 2 TRIAL (investigation of serial studies to predict your therapeutic response with imaging and molecular analysis 2),²⁴ a phase II treatment trial using response-adaptive randomization within breast cancer subtypes to evaluate investigational agents for women with high-risk stage II/III breast cancer. The primary aim of the ACRIN 6698 imaging trial was to evaluate whether changes in whole-tumor mean ADC measured early in the course of NAC treatment are predictive of pathologic complete response (pCR).

The purpose of this test-retest substudy was to prospectively evaluate the repeatability and reproducibility of breast tumor ADC measures in a multi-institution, multi-MRIplatform clinical trial setting, using a standardized MR-DWI protocol and quality assurance (QA) procedures.

Materials and Methods

Site Qualification

All actively enrolling or newly qualified I-SPY 2 imaging sites were eligible to participate in the ACRIN 6698 trial. Sites qualified for participation via a process consisting of evaluation of phantom and human exams as prescribed in the ACRIN 6698 study materials.²³ Scanners could be of any manufacturer, with field strength of 1.5 or 3.0T, and had to employ a dedicated bilateral breast RF coil. Qualification was specific to the particular scanner and coil combination. While qualification of multiple scanners by a site was allowed, all MRI studies for any given patient were required to be performed using the same scanner configuration (manufacturer, field strength, model, and breast coil model). Qualification scans using a bilateral ice/water mixture phantom with a known ADC value (1.1 * 10⁻³ mm²/sec)²⁵ were performed using the ACRIN 6698 study DWI protocol (Table 1) and submitted to a core lab at the University of Michigan for quality control analysis. Scanner qualification required a maximum ADC bias in standardized breast imaging regions of less than 10%. In addition, two clinical DWI test scans using the ACRIN 6698 study protocol on volunteers or currently enrolled ISPY 2 patients were required prior to study imaging. These in vivo scans were evaluated for protocol compliance and adequate DWI image quality for ADC measurement in breast tissue.

TABLE 1.

ACRIN 6698 DW-MRI Protocol Parameters and Protocol Adherence for N = 89 Test/Retest Cohort

Parameter	DWI protocol specification	Test/retest cohort values (N)	Nacq out of protocola
Acquisition order	DWI precontrast injection	Compliant (88), mixed (1)b	1
Sequence type	SS-EPI	All compliant
Frequency direction	R/Lc	R/L (60), A/P (10), mixed (1)d
FOV (mm)	260–360	280–390	4
Matrix-frequency	128–192	128–196	2
Matrix-phase	128–192	128–194	2
Fat-suppression	Active fat-sat	All compliant
TR (msec)	≥4,000	4800–9400
TE (msec)	Minimum TE (50–100)	59.2–114.0	5
Flip Angle	90 degrees	90
B values	0, 100, 600, 800 s/mm2	All compliant
Slice thickness (mm)	4–5	4 (44), 5 (27)
Number of slices	Complete bilateral coverage	24–50
Slice gap (mm)	≤ 1.0	0
No. averages	≥2	2–5
Parallel imaging factor	≥2	N/C
Total acquisition time	≤ 5 minutes	N/C

SS-EPI: Single-shot echo-planar imaging, N/C: data not collected.

^aNumber of acquisitions (test or retest) with out-of-protocol values.

^bOne study was done with the retest DWI acquisition following the contrast injection.

^cProtocol requirement of R/L frequency encoding was relaxed after investigation indicated better image quality with A/P frequency on some scanners.

^dOne study was done with reversed phase/frequency encoding directions between test and retest scans.

Patient Population

Patients were enrolled into the ACRIN 6698 trial at qualified study centers following I-SPY 2 trial procedures and inclusion/exclusion criteria.²⁶ Both studies were HIPAA-compliant and performed under individual site Institutional Review Board (IRB) approval, and all patients gave informed consent prior to enrolling. Women aged 18 and over were eligible if they had biopsy-confirmed diagnosis of stage II–III disease, and clinically or radiologically measurable disease in the breast with a tumor longest diameter (LD) >2.5 cm. Patients were classified by biomarker assessments of hormone receptor (HR), human epidermal growth factor receptor-2 (HER2), and Mamma-Print (MP) status performed at pretreatment; patients with low-risk disease (HR+/HER2-/MP-low) were excluded. For patients enrolled into ACRIN 6698, a separate IRB-approved consent was used for enrollment into the test/retest arm of the trial. Details on the population subgroup for the test/retest arm are given in the Results section, below.

MRI Protocol

The MRI component of the ACRIN 6698 trial consisted of four sequential studies: pretreatment (T0), early-treatment (T1), midtreatment (T2), and posttreatment (T3) (Fig. 1). T₂-weighted (T₂W), multi-b-value DWI, and DCE acquisitions were taken at each study timepoint. Imaging was done in the axial plane with full bilateral coverage of the breasts. The DWI protocol was standardized to the greatest extent possible given constraints of the multiple scanner platforms, and required acquisition using a fat-suppressed SS-EPI sequence with four b values (0, 100, 600, 800 sec/mm²). Specified parameter ranges for the DWI protocol are given in Table 1. DWI was implemented with a single series multi-b DWI acquisition or with three consecutive two b-value acquisitions (0/100, 0/600, and 0/800 sec/mm²). Test and retest DWI measurements for a given patient were performed on the same day in a single imaging session. The patient was positioned normally (prone) and scanned with initial localization, T₂W, and DWI acquisitions. They were then removed from the scanner and taken off the scanner bed, then repositioned as before. The full ACRIN 6698 protocol was then performed, consisting of localization, T₂W, DWI, and DCE acquisitions. A single test/retest study was conducted for each consented subject at either T0 or T1, with T0 specified as the preferred timepoint.

FIGURE 1:

I-SPY 2 TRIAL study schema. Patients were randomized onto the control arm (paclitaxel treatment) or onto one of the multiple experimental drug treatment arms. DCE MRI was conducted at four timepoints to monitor treatment response via changes in MRI functional tumor volume. For the ACRIN 6698 Trial, an advanced four b-value DWI sequence was added at each of the MRI timepoints T0–T3, and test/retest repeatability scans were done on a subset of patients at T0 or T1. (We note that the T0–T3 nomenclature is used in this article for consistency with I-SPY 2, and differs from that defined in the ACRIN 6698 protocol.)

DWI Analysis

DICOM images from all acquisitions were deidentified and transmitted using TRIAD (Transfer of Images and Data, ECOGACRIN, Philadelphia, PA) to the University of California, San Francisco, (UCSF) imaging core lab for centralized analysis. DWI images were assessed with a standardized QA protocol²⁷ for the three quality categories of artifacts, fat suppression, and signal-to-noise ratio (SNR), and given an overall quality rating of poor, moderate, or good. Poor-quality studies were judged not analyzable and were excluded from the study. Moderate- and good-quality studies were then evaluated as analyzable or not analyzable based on the degree to which any negative quality issues were found to prevent confident definition of a region-of-interest (ROI) contoured to the whole primary tumor region. QA evaluation was done by trained operators, blinded to pathologic outcomes, with a minimum of 5 years experience evaluating breast MR images.

ADC parametric maps were created based on the classic monoexponential decay model²⁸:

$$\text{S}(\text{b})={\text{S}}_0*{\text{e}}^{-{\text{b}}^{*}\text{ADC}}$$

where S(b) is the signal intensity with diffusion weighting b. ADC was calculated using a linear least-squares fit of the log of the signal intensities at all four b-values. For studies using three consecutive two b-value acquisitions (in contrast to the standard four b-value acquisition), an automatic protocol check was used to ensure that no MR parameters other than b-value were varied and then the three acquisitions were combined into a single four b-value series for analysis. All centralized DWI image processing was done using UCSFdeveloped software written in IDL (Exelis Visual Information Solutions, Boulder, CO).

Tumors were identified on postcontrast DCE subtraction images and then localized on the corresponding DWI images. Multislice, whole-tumor ROIs were manually defined by selecting regions with hyperintensity on high b-value DWI (b = 600 or 800 s/mm²) and relatively low ADC, while avoiding adjacent adipose and fibroglandular tissue, biopsy clip artifacts, and regions of high T₂ signal (e.g. seroma and necrosis). For all cases, including large and multicentric/multifocal disease, all apparent disease regions were included in the ROI by using multiple distinct contours per slice and multiple slices as required to cover the entire tumor region without including intervening or adjacent tissues. All voxels from the individual contours were combined into a single composite ROI for statistical analysis. The test and retest ROIs for a given patient were defined separately and independently with no cross-referencing between the two DWI scans, and were performed by the same operator to minimize operator variability. All ROI definitions were reviewed and adjusted, if necessary, by the senior operator (final review performed by J.E.G., with over 10 years quantitative breast MR analysis experience). Mean and median ADC values were calculated for each composite whole-tumor ROI.

Whole-tumor ROIs for the evaluation of intra- and interoperator variability were defined independently of those for test/retest repeatability, but using the same procedures as described above. However, unlike the ROIs for the repeatability measurement, no final review of the ROIs by a senior operator was done for the reader study. For this reader reproducibility study only the second “retest” acquisition on each subject was analyzed. Reader one (J.E.G., senior study operator) analyzed each case twice for intraoperator assessment (RD1, RD1B), while reader 2 (W.L.) did a single analysis (RD2) that was compared with RD1 for interoperator assessment. RD1 and RD1B analyses for intraoperator variability were conducted 5–6 weeks apart. In addition to comparisons of the mean tumor ADC values, ROI reproducibility was directly evaluated using the Sørenson-Dice similarity coefficient (DSC),²⁹ given for two ROIs “A” and “B” by:

$$\text{DSC}=2\times {\text{V}}_{\text{AB}}/\left({\text{V}}_{\text{A}}+{\text{V}}_{\text{B}}\right)$$

where V_AB is the volume common to both ROIs and V_A and V_B are the individual ROI volumes. Thus, DSC varies from 0 for no overlap between the regions, to 1 for complete overlap (identical ROIs). For every case in the reader study DSC was evaluated between each pair of ROIs: [RD1 vs. RD1B], [RD1 vs. RD2], and [RD1B vs. RD2]; and the mean of the three values was used to estimate the ROI variability of each case:

$${\text{DSC}}_{\text{Mn}}=\left({\text{DSC}}_{[\text{RD1},\text{RD}2]}+{\text{DSC}}_{[\text{RD1B},\text{RD}2]}+{\text{DSC}}_{[\text{RD1},\text{RD1B}]}\right)/3$$

Scatterplots and Pearson’s correlation were used to estimate the dependence of the ROI variability with ADC and DCE-derived tumor characteristics.

Statistical Analysis

The reproducibility of each marker was assessed using four different indices: within-subject coefficient of variation (wCV),³⁰ intraclass correlation coefficient (ICC),³¹ agreement index (AI),³² and repeatability coefficient (RC).³³ Both RC and wCV are based on withinsubject standard deviation (wSD), where RC = 2.77*wSD and wCV = 100%*wSD/mean. RC has the same units as the marker, while wCV is unit-less. Both estimates are unbounded in the upper range and sensitive to extreme outliers. ICC is derived from the analysis of variance (ANOVA) model estimates, while AI is based on the data’s overall ranking. Both ICC and AI are bounded (−1 to 1 for ICC and 0.5 to 1 for AI) and unit-less. The confidence intervals of AI were done with a bootstrap method.

Results

Enrollment

Between August 2012 and January 2015, 406 patients were enrolled into the ACRIN 6698 trial at 10 US institutions. In all, 388 patients aged 23–77 years (median age 48 years) were found eligible under the I-SPY 2 inclusion criteria. Of these, 89 patients aged 27–73 years (median age 47 years) from nine institutions were consented to the test/retest substudy. Individual sites were limited to 14 test/retest patients to better balance the accrual across different MRI scanner manufacturers and field strengths. Eleven patients from one site were imaged using three consecutive two b-value acquisitions, while the rest were done with single four b-value scans. Table 2 shows accrual numbers and QA results for the test/retest study, including a breakdown by site, visit, field strength, and MRI scanner vendor. Three patients (3.4%) were excluded for major test/retest protocol deviations: change in slice thickness, retest done after contrast injection, and changes between the successive two b-value acquisition parameters. Of the remaining 86 patients, both the test and retest scans for 71 patients (median age 46, range 27–71 years) from eight institutions were assessed as analyzable for tumor ADC through the image QA process. Fifteen subjects were rejected for image quality issues that prevented analysis of one (n = 7) or both (n = 8) of the DWI acquisitions, giving an overall rejection rate for image quality issues for DWI acquisitions of 13.4% (23 of 172 acquisitions). Image quality issues in the rejected acquisitions included excessive artifacts (n = 15), fat suppression failure (n = 12), and poor SNR (n = 10).

TABLE 2.

Accrual and Analyzability Results for Test/Retest Studies

		N accrued	N analyzable	% Lost	N reader study	Field strength	Vendor
	Alla	89	71	20.2	20
Site	Site A	5	3	40.0	0	1.5	Siemens
	Site B	11	9	18.2	3	3	GEHC
	Site C	15	14	6.7	4	3	Philips
	Site D1	7	5	28.6	2	1.5	Siemens
	Site D2	5	4	20.0	1	3	Siemens
	Site Eb	11	7	36.4	2	1.5	GEHC
	Site F	12	11	8.3	3	1.5	Philips
	Site G	8	6	25.0	2	1.5	Philips
	Site H	14	12	14.3	3	1.5	GEHC
	Site I	1	0	100.0	0	3	GEHC
Study visitc	T0	75	60	20.0	17
	T1	14	11	21.4	3
Field strength	1.5T	57	44	22.8	12
	3.0T	32	27	15.6	8
Vendor	GEHC	37	28	24.3	8
	Philips	35	31	11.4	9
	Siemens	17	12	29.4	3

^aAll scans were done on qualified scanners with dedicated bilateral breast coils

^bSite E studies were done with three separate two b-value acquisitions (0/100; 0/600; 0/800 s/mm²). All others sites used a single four b-value acquisition.

^cT0 (baseline, pretreatment) was specified as the preferred visit for test/retest studies

ADC Repeatability

Figure 2 shows typical images and ADC maps for two example cases that illustrate poor (Subject 1, |ADC_difference|/ADC_mean = 0.117) or good (|ADC_difference|/ADC_mean =0.025) repeatability. The left panels show the DCE subtraction images (nominal 2.5 min postinjection - preinjection) that highlight the enhancing regions of the breast; these images were used for tumor localization. The center (test) and right (retest) panels show the corresponding slices of the ADC maps for the two DWI acquisitions. The differences in the ADC maps illustrate the effects of patient repositioning combined with the relatively thick slices in the DWI acquisitions. Repositioning makes perfect matching of ADC slices very unlikely, yielding different partial-volume effects across the thick slices, thus contributing to the observed differences between the independently defined test and retest wholetumor ROIs.

FIGURE 2:

Sample images of representative slices for two subjects. The left panels show DCE subtraction images used for tumor localization. The corresponding ADC map images and ROIs are shown for the test DWI (center panels) and retest DWI (right panels). Subject 1 had relatively poor repeatability (|ADC_difference|/ADC_mean = 0.117), while Subject 2 had good repeatability (|ADC_difference|/ADC_mean = 0.025). Each inset shows the mean whole-tumor ADC, the ROI volume segmented on the DWI, and the extent in the slice direction of the multislice ROI.

Comparisons of test and retest ADC measurements are shown in Bland-Altman plots in Fig. 3. Cases measured at T0 (pretreatment) are represented as blue diamonds and T1 (early-treatment) cases as red circles. The test and retest ADC values were similar for the 71 subject analyzable cohort (Fig. 3a) with minimal bias between test and retest (mean [SD] ADC: 1.16 [0.32] and 1.17 [0.31] × 10⁻³ mm²/sec, respectively) over tumor ADC values ranging from 0.80–2.62 × 10⁻³ mm²/sec. Absolute ADC values trended higher for the T1 cases, as expected due to treatment effects, but repeatability appeared similar. Figure 3b,c shows Bland-Altman plots for subgroups defined by field strength. We observed minimal bias and no trend in ADC difference with mean ADC for these subgroups.

FIGURE 3:

Bland-Altman plots for the difference in whole-tumor ADC measurements between test and retest for: (a) entire analyzable cohort, (b) 1.5T cases, and (c) 3T cases. Blue diamonds indicate the 60 measurements taken at timepoint T0 (pretreatment) and red circles indicate the 11 measurements taken at timepoint T1 (early treatment). Mean difference and 95% CI (1.96 * SD) are shown as horizontal solid red and dashed black lines, respectively. No trend was seen in the ADC difference with ADC magnitude.

Repeatability of tumor ADC values was excellent by all measures for the whole cohort and within subgroups (Fig. 4a, b, Table 3). For the whole cohort, RC = 0.16 × 10⁻³ mm²/s (95% confidence interval [CI] 0.13–0.19), wCV = 4.8% (95% CI 4.0, 5.7%), ICC = 0.97 (95% CI 0.95, 0.98), and AI = 0.83 (95% CI 0.76, 0.87). For the subgroups defined by the field strength or the treatment points, ICC values ranged from 0.91–0.99, AI from 0.81–0.83, and wCV from 3.6–5.2%—with generally tight 95% CIs, as shown by the error bars in Fig. 4. The AI results for the 11 patients done at timepoint T1 show a wider range in the 95% CI, due to the small sample size and the use of a bootstrap technique for this calculation.

FIGURE 4:

Repeatability measures intraclass correlation coefficient (ICC, blue) and agreement index (AI, red) for full cohort and subgroups defined by field strength (1.5 and 3.0T) and study visit (T0: pretreatment and T1: early-treatment). Error bars indicate 95% CIs.

TABLE 3.

Summary of Agreement Values and Inter- and Intraoperator Variability for All Cohorts for Mean Tumor ADC

		RC (10−3 mm2/sec)			wCV			ICC			AI
Cohort	n	Value	95% CI		Value	95% CI		Value	95% CI		Value	95% CI
All	71	0.16	0.13	0.19	4.8%	4.0%	5.7%	0.97	0.95	0.98	0.83	0.76	0.87
1.5T	44	0.15	0.13	0.20	4.7%	3.6%	5.8%	0.98	0.96	0.99	0.83	0.74	0.87
3.0T	27	0.16	0.12	0.21	5.1%	3.7%	6.5%	0.95	0.89	0.98	0.81	0.68	0.86
T0	60	0.16	0.13	0.19	5.2%	4.2%	6.1%	0.91	0.85	0.94	0.81	0.73	0.85
T1	11	0.16	0.12	0.28	3.6%	2.0%	5.3%	0.99	0.96	1.00	0.83	0.60	0.84
inter	20	0.18	0.14	0.26	5.4%	3.7%	7.1%	0.92	0.80	0.97	0.67	0.44	0.78
intra	20	0.17	0.13	0.25	5.6%	3.8%	7.4%	0.91	0.78	0.96	0.78	0.58	0.84

RC: repeatability coefficient; wCV: within-subject coefficient of variation; ICC: intraclass correlation coefficient; AI: agreement index; inter: interperator variability; intra: intraoperator variability.

ADC and ROI Reproducibility

For the reader variability study, a random sample of 20 cases (median age 43, range 27–69 years) were selected in proportion to the vendor and field strength representations in the main analysis set (see Table 2, “N Reader Study” column). Table 3 (bottom two rows) and Fig. 5 show results for interand intraoperator reproducibility. All measures showed good reproducibility, e.g. ICC was 0.92 (95% CI 0.80, 0.97) for interoperator and 0.91 (0.78, 0.96) for intraoperator variability. Bland-Altman plots (Fig. 5) indicated minimal bias between the measurements and no trends in ADC difference with increasing mean ADC.

FIGURE 5:

Bland-Altman plots showing interreader (a) and intrareader (b) variability in whole-tumor ADC measures. Plotted is the difference in the two independent reader ADC evaluations vs. the mean ADC. Mean difference and 95% CI (1.96 * SD) are shown as horizontal solid red and dashed black lines, respectively. Black diamonds indicate 1.5T scans and blue squares indicate 3T scans. Plots show minimal bias and no trends in ADC difference with increasing ADC values.

The only source of operator variability in our ADC analysis is in the definition of the whole-tumor ROIs. To quantify these variations we compared the ROIs for inter- and intraobserver tests using the Sørenson-Dice similarity coefficient (DSC) as a measure of ROI similarity. Median DSC across the 20 patient cohort was 0.69 (range 0.33–0.79) for interoperator ROI pairs (DSC_[RD1,RD2]) and 0.70 (range 0.40–0.84) for intraoperator pairs (DSC_[RD1,RD1B]). The interoperator and intraoperator similarity values (DSC_[RD1,RD2] and DSC_[RD1,RD1B]) were strongly correlated (Fig. 6a, Pearson’s r = 0.84, P < 0.0001, 95% CI 0.63, 0.94). To evaluate possible factors affecting ROI variability, we calculated the Pearson’s correlation between the mean DSC for each case (DSC_Mn) and tumor characteristics, including the mean ADC, and the MRI DCE-derived functional tumor volume³⁴ and tumor surface area. A trend of decreasing DSC_Mn with increased ADC was observed (r = −0.49, P = 0.03, 95% CI −0.77, −0.06; Fig. 6b), consistent with the readers having increased difficulty in defining ROIs later in treatment when tumor ADC values are closer to those of normal breast tissue. No significant correlation was found between DSC_Mn and ADC variability (r = −0.37, P = 0.11, Fig. 6c), nor between DSC_Mn and the DCE-based tumor size measures (r = 0.34, P = 0.14 for volume and r = 0.32, P = 0.17 for surface area).

FIGURE 6:

ROI reproducibility results. (a) Sørenson-Dice similarity coefficient (DSC) for the interreader (DSC_[RD1,RD2], X-axis) and intrareader (DSC_[RD1,RD1B], Y-axis) ROI pairs for each case. Inter- and intravalues were highly correlated: Pearson’s r = 0.84 (P <0.0001, 95% CI 0.63, 0.94). (b) Mean DSC (DSC_Mn = (DSC_[RD1,RD2] + DSC_[RD1B,RD2] + DSC_[RD1,RD1B])/3) vs. mean whole-tumor ADC for each study. A trend of decreasing DSC_Mn with increasing ADC was observed (Pearson’s r = −0.48 [P = 0.03, 95% CI −0.77, −0.06]). Correlations were somewhat driven by the high-ADC outlier. Analysis without the outlier resulted in r = 0.89 (P < 10⁻⁶) for inter-DSC vs. intra-DSC and r = −0.59 (P = 0.008) for DSC_Mn vs. mean ADC. (c) DSC_Mn vs. ADC variability (SD of the mean ADC values for three reader study observations). No significant correlation was observed (r = −0.37, P = 0.14).

Protocol Compliance

A post-hoc detailed protocol compliance assessment was done both with respect to the ACRIN 6698 DWI acquisition specifications and for consistency between the test and retest scans on each subject. Overall compliance within the 89 patient test/retest cohort was reasonably high, with 166 (93%) of the 178 DWI acquisitions within protocol specifications (Table 1). Within the n = 71 analyzable cohort, three subjects had protocol deviations that were considered major, as they resulted in sizable differences in TE (78 msec test vs. 107–114 msec retest) and TR (7.1 sec test vs. 8.9–9.4 sec retest). Nine other subjects, while having DWI parameters within study specifications, had differences between test and retest protocols. Only one of these deviations—a swapping of frequency/phase encoding directions—was considered major. The whole cohort wCV was essentially unchanged when the four subjects with major deviations were excluded from the analyzable cohort, dropping from 4.8% to 4.7%.

Discussion

There is notable interest in using quantitative diffusion metrics as biomarkers for breast cancer treatment response, as recently summarized in the review article by Partridge et al.³⁵ However, given the challenges inherent in quantitative breast DWI due to motion, off-isocenter effects, and SNR considerations, as well as the analysis challenge of consistent and optimized ROI definition, the variability in breast ADC measures is a major concern. These challenges are magnified in a typical multisite clinical trial setting due to the wide variety of scanner platforms in use and the large number of sequence variants optimized for the different platforms. In this study, we evaluated the repeatability and reproducibility of breast tumor ADC in a multicenter setting, under the conditions of a standardized protocol and centralized quality-control and analysis. Excellent agreement in repeat measures of tumor ADC was found. Repeatability did not appear to be significantly influenced by field strength or treatment timepoint. Inter- and intraoperator reproducibility were also good, even though whole-tumor ROI reproducibility varied between studies. Therefore, our findings support the use of ADC values as a reproducible, functional metric for tumor response in this setting.

Multiple studies have been conducted on breast DWI reproducibility, including studies on phantoms, normal volunteers, and patients. While not directly comparable to in vivo studies, phantoms with known ADC values can help establish minimum variability limits for DWI techniques. For example, using ice/water phantoms (ADC = 1.1 10⁻³ mm²/s), Malyarenko et al³⁶ demonstrated short-term test/retest repeatability wCV <0.5% and day-to-day wCV <2.2%, across a range of vendors and field strengths. With a similar phantom, Sorace et al³⁷ reported <0.01% average difference in ADC between repeated scans with intrascan repositioning of the phantom for three instances of a single scanner platform. These and other phantom results generally indicate that the scannerdependent variability in ADC measurement of uniform media is small compared with in vivo effects and ROI delineation within nonuniform tissues.

For single-site test/retest repeatability in normal volunteers, Aliu et al¹⁴ found a wCV of 11% (n = 9); Partridge et al¹⁸ showed a 4.5% wCV (n = 9) for ADC derived from diffusion tensor imaging; and Sorace et al³⁷ found a wCV =3.72% (n = 10). Despite the differences in methodology, these are all comparable to our findings. The higher wCV in Aliu et al is perhaps attributable to increased time between repeat scans, which were done either within 72 hours or at the same menstrual phase in successive periods. Sorace et al’s study also included multisite/single-platform reproducibility on three subjects, each scanned at three different sites. The wCV for these repeat measures was 6.3%.

Patient studies of reproducibility of breast tumor ADC^15,16,19,20 have shown a range of intra- and interobserver variability values, with wCV ranging from 3.2–8.3% and ICC values >0.9 in most cases. Our results are of comparable magnitude to observer variability in these reader studies. In a retrospective single-site study of 54 patients Giannotti et al¹⁵ reported excellent ICC values for intraobserver variations for clinical and technical readers ranging from 0.92–0.99, with some dependence on ROI techniques, tumor environment, and tumor size. Interobserver ICC values were lower, however: 0.86 for a large single-slice ROI and 0.68 for a small ROI measuring minimum ADC. In a diagnostic study measuring ADC values of suspicious breast lesions (Breast Imaging Reporting and Data System [BIRADS] category 4 or 5) in 40 patients with repeat scans 1–11 days apart, Spick et al²⁰ found wCV of 6% and 6.6% for two different readers, with ICC values >0.9. We note that all these previous studies were single-platform with, except for Giannotti et al’s work, relatively small numbers of patients. The ACRIN 6698 study expanded this to a true clinical trial setting. Our overall repeatability values wCV = 4.8%, ICC = 0.97, and RC =0.16 × 10⁻³ mm²/s indicate that performance similar to single-site studies is attainable in the multisite trial setting, at least under the conditions of a standardized protocol and centralized QA and analysis. We found this true across multiple field strengths and despite the challenges of protocol standardization across multiple vendor platforms.

Previous studies in patients undergoing NAC have reported significant changes in tumor ADC with treatment response, with effect sizes generally large compared with our repeatability value of wCV = 4.8%. Responders exhibited tumor ADC increases ranging from 9% to over 60% in the prior studies, depending on treatment timepoint.^3–6,38

Perhaps the most significant factor currently limiting reproducibility of breast tumor ADC measurements is operator variability in the ROI definition. For this study, we evaluated whole-tumor mean ADC as specified in the study protocol, which required multicontour/multislice 3D ROIs. While offering a potential advantage of sampling the entire diseased region, these ROIs are also the most difficult and time-consuming to define. The high ICC values we found for intra- and interreader reproducibility and the similarity between these metrics indicate that the analysis techniques are both reproducible and transferable with suitable training. However, the wide range of the Dice similarity between ROIs across the patient cohort may indicate that the excellent reproducibility is a reflection of the robustness of our whole-tumor ADC analysis protocol in the presence of ROI variations. The lack of significant correlation between ROI similarity and ADC variability in our data further indicates that tumor characteristics (eg, homogeneity, morphology) play a more important role in determining ADC reproducibility on an individual case than do the details of the ROI definition.

There is not currently evidence to our knowledge that the whole-tumor ADC is optimum for treatment monitoring. In particular, in a single site study of 150 patients in a diagnostic setting, Bickel et al³⁹ found that small, single-slice, 2D ROIs identifying a region of minimal ADC gave better performance than either larger single-slice 2D ROIs or multislice 3D ROIs. Further work is clearly needed in evaluating both different ROI techniques and automatic or semiautomatic segmentation techniques for optimizing any DWI treatment response metrics.

Acquisition of consistently high-quality breast DWI continues to be a challenge. The overall rejection rate of DWI acquisitions in this study due to image quality issues was 13%, representing cases where the operator felt it was not possible to confidently define an ROI for the whole tumor volume. Failed or inadequate fat suppression and excessive artifacts were the primary causes for many of these rejections. In their larger single-site diagnostic study, Bickel et al³⁹ reported a loss of only 3 out of 150 studies (2%) due to technical failures of the DWI; however, they excluded cases with no MR visible lesion. Our higher rejection rate may be in part due to the compromises inherent in standardizing a protocol across multiple scanner platforms. It is possible that scanner-optimized protocols may yield better results for treatment response monitoring in multisite clinical trials, assuming all longitudinal scans on a given patient are restricted to a single platform. Ongoing investigations in the broader I-SPY 2 cohort may shed some light on these QA issues. While our prospective study was not powered to detect differences between scanner configurations, we did observe a somewhat lower rejection rate for 3T scanners compared with 1.5T, indicating a possible quality advantage for the higher field strength. Qualitycontrolled studies with larger cohorts are needed to establish whether this is a valid finding.

This study was limited to exploring the variability in whole-tumor ADC due to patient repositioning, fundamental variabilities in the DWI acquisition and analysis, and reader variability. Further investigation is required to assess variability due to other factors, most particularly ROI definition protocols, and of other DWI metrics. We were also limited by the predominance of pretreatment studies. This prohibited any substantive determination of differences in reproducibility as treatment progresses, which may be critical for treatment response determination. This may be of particular concern for measurements at timepoints late in the course of treatment, since experience in the ACRIN 6698 trial showed that it becomes increasingly difficult to confidently define a whole-tumor ROI after significant response to NAC. Finally, we were limited in this study to the case of centralized analysis with single-platform generated ADC maps. On-site analysis and the use of ADC maps from multiple platforms (eg, on-scanner parametric map generation) may further affect ADC repeatability.^21,40

In conclusion, within the prospective multicenter ACRIN 6698 trial, we found excellent reproducibility of whole-tumor ADC—with test/retest repeatability and operator variability both small compared with typical treatment induced changes and consistent across different field strengths. To facilitate the use of DWI for monitoring therapy, further work is needed to improve MR-DWI image quality in the breast to reduce losses, particularly with respect to fat suppression and image artifacts.