Accuracy, repeatability, and reproducibility of water-fat magnetic resonance imaging in a phantom and healthy volunteer

Highlights

• •Water-fat magnetic resonance imaging can evaluate bone marrow fat changes.
• •Water-fat magnetic resonance imaging should be validated for use in clinical studies.
• •Our study serves as an example of a robust validation methodology.
• •Phantom measurements showed an excellent accuracy, repeatability, and reproducibility.
• •Volunteer measurements showed an acceptable repeatability and reproducibility.

Abstract

Bone marrow (BM) damage due to chemoradiotherapy can increase BM fat in cervical cancer patients. Water-fat magnetic resonance (MR) scans were performed on a phantom and a healthy female volunteer to validate proton density fat fraction accuracy, reproducibility, and repeatability across different vendors, field strengths, and protocols. Phantom measurements showed a high accuracy, high repeatability, and excellent reproducibility. Volunteer measurements had an excellent intra- and interreader reliability, good repeatability, and moderate to good reproducibility. Water-fat MRI show potential for quantification of longitudinal vertebral BM fat changes. Further studies are needed to validate and extend these findings for broader clinical applicability.

1. Introduction

Bone marrow (BM) is composed of red BM, which is actively involved in generating blood cells, and inactive yellow BM, which stores fat and has reduced hematopoietic activity [1]. Chemoradiation of pelvic cancers depletes the BM population and allows mesenchymal stem cells to differentiate towards adipocytes, leading to an increase in the fatty yellow marrow [2], [3]. Chemoradiation-induced BM damage causes hematologic toxicity (HT), defined as a reduced number of circulating blood cells.

Primary chemoradiation is the standard treatment for women with locally advanced cervical cancer [4], [5]. HT, especially lymphopenia, due to this treatment can lead to the discontinuation of chemotherapy, the need for blood transfusions, and decreased overall survival [6], [7], [8]. Water-fat MRI can generate proton density fat fraction (PDFF) maps. Multiple studies used these PDFF maps to detect BM fat changes in the lumbar vertebras of gynecological cancer patients treated with chemoradiotherapy, which was related to the number of circulating blood cells [9], [10], [11]. Water-fat MRI could identify locations of active and radiation-induced damaged BM and facilitate the development of BM sparing radiotherapy treatment planning techniques and the evaluation of treatment.

However, the studies evaluating BM PDFF in patients with gynecologic cancer have several limitations [9], [10], [11]. Studies were not performed in a multicenter or multivendor setting, which decreases the external validity of the outcomes, or did not compare PDFF of vertebras inside and outside the radiation field, which limits the quantification of radiotherapy impact specifically. Therefore, a prospective, multicenter study was designed to longitudinally assess PDFF changes in BM of the whole vertebral column in women with locally advanced cervical cancer treated with BM sparing chemoradiotherapy [12].

The Radiological Society of North America Quantitative Imaging Biomarkers Alliance (QIBA) recommends to validate protocols under study conditions to determine their inherent reliability to measure the endpoint [13]. The aim of this study is to measure the accuracy, repeatability, and reproducibility of water-fat MRI in a multivendor setting for the assessment of PDFF to use in a longitudinal, prospective study.

2. Materials and Methods

2.1. Data acquisition

The Calimetrix PDFF phantom (Fat Fraction Phantom, Model 300, Calimetrix, USA) consists of a spherical acrylic housing containing twelve gel-based vials with known reference PDFF values in a doped water bath. The nominal values of the vials are 0, 2.7, 5.3, 7.8, 10.1, 15.5, 20.4, 23.6, 30.2, 39.9, 50.1, and 100 %, as indicated in Figure S1. More details on the PDFF phantom can be found in the study by Hu et al (2021) [14]. The volunteer was a healthy, 52-year old woman. The study was approved by the Medical Ethics Review Committee Erasmus Medical Center and Medical Ethics Review Committee Leiden The Hague Delft. Written informed consent was obtained prior to examination.

Clinical 1.5 T and 3 T MR scanners by the vendors Philips (Philips Healthcare, the Netherlands) (P1.5T and P3.0T) and General Electric (GE Healthcare, USA) (G1.5T and G3.0T) were used. Scanner parameters and settings are listed in Supplementary Table S1. As the vendor-specific quantitative 3D chemical shift-encoding based scanning technologies were developed for hepatic fat quantification, the resolution of the protocols was optimized for the evaluation of the (smaller) vertebras [15], [16]. For P1.5T and P3.0T, both vendor-specific protocols (v1) and reoptimized protocols with improved resolution (v2) were assessed. For G1.5T and G3.0T, only the optimized protocols were assessed. Vendors’ commercial software for online PDFF map reconstruction was used to obtain the PDFF maps. This software includes multipeak fat spectral modeling, T2* confounding effect correction, and T1 bias reduction.

The phantom acclimated at least one hour to the room temperature before scanning, according to the vendor’s instructions. The volunteer was scanned in supine position with a field of view extending from the third cervical vertebra to the lesser trochanter of the femur using three to four overlapped axial stacks with three to four table positions. The coils used are specified in the Supplementary Material. The stacks were combined with MR MobiView (Philips Healthcare, The Netherlands) for the Philips system and with MR Autobind (GE Healthcare, USA) for the GE system. Repositioning took place between test and retest scans to imitate interscan position differences. All scans took place within a time window of 30 days and were acquired at a temperature of 19–20 °C.

2.2. Data analysis

The methods for placing regions-of-interest (ROI) in the vertebras, muscle, and adipose tissue, the latter two of which served as additional datapoints, on the PDFF map are detailed in the Supplementary Material. The mean PDFF, including standard deviation, minimum, and maximum value, was calculated over all voxels per ROI and expressed as a percentage [%].

2.3. Statistical analysis

Intrareader variation was assessed using two-way mixed-effects intraclass correlation coefficients (ICCs) [17]. Additionally, interreader agreement was assessed with the two-way random-effects ICCs. The level of reliability was classified according to Koo et al (2016) [17].

Pearson correlation (r) and Bland-Altman analysis were used to determine the accuracy of the first (test) scanning session of the phantom [18]. Bias and limits of agreement (LoA) were computed with the Bland-Altman analysis according to the difference between measured and reference PDFF as a function of the mean of these two values and have the same unit as the absolute PDFF [PDFF%].

Paired t-tests were performed to compare test and retest volunteer scans per vertebra and reference tissue. The repeatability coefficient (RC) represents the least significant difference between two repeated measurements taken under identical conditions in absolute PDFF [PDFF%] (see Supplementary Material) and was determined per protocol [19].

Reproducibility across all protocols was evaluated using Lin’s concordance correlation coefficient (CCC) (see Supplementary Material) and classified according to Carbonell et al. [20]. Furthermore, Bland-Altman analysis was performed to assess bias and LoA across protocol versions, field strengths, and manufacturers for both phantom and volunteer measurements.

Statistical analysis was performed using R for Statistical Computing (version 4.3.1). Statistical significance level was defined as p < 0.05.

3. Results

3.1. Intra- and interreader variability

The intrareader ICC for the PDFF measurements across the six protocols was 0.97 [95 % CI, 0.96 – 0.98]. The interreader ICC for the PDFF measurements across the six protocols was 0.99 [95 % CI, 0.98 – 0.99]. Both the intra- and interreader ICC were > 0.9 and indicated an excellent reliability of the PDFF measurements in the volunteer scans.

3.2. Accuracy

Figure S2 shows that PDFF measurements were highly correlated to the reference values of the phantom. Absolute differences between measured PDFF and reference values were between −2.81 PDFF% and 6.98 PDFF%. Pearson correlation coefficients of the PDFF measurements were between 0.9967 and 0.9998. Pooled PDFF measurements from the Philips MR systems, GE MR systems, 1.5 T systems, 3 T systems, and all protocols combined likewise showed high correlation with the reference values, all having r = 1.00 [95 % CI, 1.00 – 1.00]. The mean bias of the phantom measurements ranged between −0.57 PDFF% and 1.09 PDFF% [95 % LoA, −4.6, 5.78 PDFF%], showing a slight underestimation of the true PDFF with P3.0T-v1, P3.0T-v2, and G3.0T, and a slight overestimation with P1.5T-v1, P1.5T-v2, and G1.5T.

3.3. Repeatability

Fig. 1A. and B. demonstrate the within-examination (test–retest) PDFF maps for the healthy volunteer with the G3.0T protocol. Qualitative inspection of the PDFF maps identified some breathing artefacts mainly around the organs such as the heart and the liver that did not affect the vertebral column. Fig. 1C. visualizes the PDFF and corresponding standard deviations per vertebra on the within-examination scans. The adipose tissue and muscle PDFF ranged from respectively 78.0 % to 93.7 % and 2.30 % to 6.12 % (Figure S3). Paired t-tests in Table S2 showed no significant differences between test and retest scans, except for the first sacral vertebra (p = 0.03).

Fig. 1.

Repeated PDFF maps of the healthy volunteer acquired with the 3 T General Electric Signa Premier (General Electric, USA) in test (A) and retest (B) setting. In this image, yellow (test) or purple (retest) box-shaped regions-of-interest (ROIs) are positioned in thoracic vertebras six to twelve and lumbar vertebras one to five. Note that for the measurements the ROIs were placed in all vertebras between the fifth cervical vertebra and the second sacral vertebra, but this is not visualized in this figure. The ROIs did not include the bone cortex and, if present, physiological or pathological deviations. The mean PDFF and corresponding standard deviation per vertebra for repeatability measurements per protocol are visualized in C. The images in A. and B. correspond to G3.0T in Figure C. PDFF = proton density fat fraction, P1.5T = Philips 1.5 T, P3.0T = Philips 3 T, G1.5T = General Electric 1.5 T, G3.0T = General Electric 3 T, v1 = version 1, v2 = version 2, C = cervical, T = thoracic, L = lumbar, S = sacral. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

The RC ranged from 0.61 PDFF% to 1.34 PDFF% for the phantom measurements, from 4.14 PDFF% to 6.04 PDFF% for the volunteer BM measurements, which means that 95 % of the PDFF measurements repeated under the same conditions were below these values.

3.4. Reproducibility

The Bland-Altman plots in Fig. 2 show good agreement of PDFF measurements across protocol versions, field strengths, and manufactures for both phantom and volunteer measurements. Lower BM PDFF values were measured with protocol versions 2 compared to versions 1, 3 T scanners compared to 1.5 T scanners, and with GE scanners compared to Philips scanners. The bias for the muscle ranged from −1.20 PDFF% [LoA −3.36, 0.95 PDFF%] to 2.13 PDFF% [LoA 0.95, 3.32 PDFF%] and for adipose tissue from −9.43 PDFF% [LoA −26.66, 7.8 PDFF%] to 2.76 PDFF% [LoA −1.60, 7.10 PDFF%] (results not shown in Fig. 2). Furthermore, the CCC for the phantom measurements were excellent, with the highest CCC for P1.5T-v1 v.s. P1.5T-v2 (0.9997) and the lowest CCC for Philips v.s. GE (0.9982). For the volunteer the CCC was moderate to good with the highest CCC for P3.0T-v1 v.s. P3.0T-v2 (0.8463) and the lowest CCC for G1.5T v.s. G3.0T (0.6701).

Fig. 2.

Reproducibility of phantom (A.) and volunteer (B.) fat fraction measurements expressed with the concordance correlation coefficient (CCC) and Bland-Altman analysis (bias and limits of agreement (LoA) [PDFF%]). PDFF = proton density fat fraction, P1.5T = Philips 1.5 T, P3.0T = Philips 3 T, G1.5T = General Electric 1.5 T, G3.0T = General Electric 3 T, v1 = version 1, v2 = version 2.

4. Discussion

In this study, the use of quantitative water-fat MRI in a multivendor setting was validated with measurements with a PDFF phantom and a healthy, female volunteer. The phantom measurements showed a high correlation with the reference values, were highly repeatable, and had an excellent reproducibility when comparing vendors, field strengths, and scan protocols. Water-fat scans of the volunteer had limited artefacts in vertebral BM, an excellent intra- and interreader reliability for vertebral BM, and a good repeatability for both vertebral BM and reference tissues. The reproducibility was lower than that of the phantom measurements, but nevertheless showed moderate to good agreement across manufacturers, field strengths, and protocols.

The phantom measurements showed excellent results, whereas the volunteer measurements had a lower repeatability, a lower CCC, and a higher mean bias for reproducibility. Various biological factors, such as regional inhomogeneities in the vertebras, and MR scanning characteristics could contribute to these differences. It is important to inspect the scans on potential artefacts to identify factors that could influence the measured PDFF. The detected biases are, however, clinically less relevant as the fat fraction of BM within the irradiation field, including the sacral and lumbar vertebras, could increase from 43 – 50 % towards 63 – 74 % [9], [10], [11].

Our study serves as an example of a robust methodology to assess the accuracy, repeatability, and reproducibility of PDFF measurements. This method can be applied before the implementation of quantitative MRI techniques in the clinic or for longitudinal clinical studies and should be repeated after software or hardware upgrades of the MR scanner.

Our study had several limitations. The commercially available phantom used had a limited PDFF range, so PDFF between 50 % and 100 % could not be validated optimally. The accuracy could also not be evaluated for the volunteer, since there was no reference standard used such as ¹H-MR spectroscopy [21]. However, the lumbar vertebral PDFF of the female volunteer in our study was comparable with the lumbar vertebral PDFF in 50-year old healthy females in other water-fat MR studies [22], [23], [24] and measurements of the adipose tissue served as an additional PDFF datapoint between 50 % and 100 %. Several factors could introduce inaccuracies in the measurement of BM PDFF using water-fat MR, including the effects of T1-related bias [25], [26], [27], T2* decay [27], [28], and spectral complexity of fat [29], [30]. Small flip angles ranging from 3 to 6° [25], the acquisition of six echoes [31], incorporation of a T2* map into the reconstruction algorithm, and the use of a multi-peak fat spectrum [32] accommodates for these factors. Another limitation of this study is that only two measurements were performed per scanner. Nevertheless, the validation outcomes are in line with earlier studies [21], [33]. Lastly, the most significant limitation is that only one female volunteer was scanned, which diminishes the robustness and generalizability of the findings. Caution should be exercised when extrapolating our results to different cohorts.

In conclusion, the water-fat MR scans used in this study have an excellent accuracy, repeatability, and reproducibility for PDFF phantom measurements, as well as an acceptable repeatability and reproducibility in vertebral BM fat quantification in a healthy 52-year old female volunteer. Since the detected biases in BM PDFF may have limited clinical relevance compared to the substantial increase in BM PDFF during chemoradiotherapy, our water-fat MR protocols show potential for quantification of BM fat changes due to chemoradiotherapy in a prospective, longitudinal, multicenter study. Further studies with larger datasets are needed to validate and extend these findings for broader clinical applicability.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: The study presented in this manuscript is partly funded by Varian, a Siemens Healthineers Company.

Appendix A. Supplementary data

The following are the Supplementary data to this article: