Precision and Test-Retest Repeatability of Stiffness Measurement with MR Elastography: A Multicenter Phantom Study

Abstract

Background

MR elastography (MRE) has been shown to have excellent performance for noninvasive liver fibrosis staging. However, there is limited knowledge regarding the precision and test-retest repeatability of stiffness measurement with MRE in the multicenter setting.

Purpose

To determine the precision and test-retest repeatability of stiffness measurement with MRE across multiple centers using the same phantoms.

Materials and Methods

In this study, three cylindrical phantoms made of polyvinyl chloride gel mimicking different degrees of liver stiffness in humans (phantoms 1–3: soft, medium, and hard stiffness, respectively) were evaluated. Between January 2021 and January 2022, phantoms were circulated between five different centers and scanned with 10 MRE-equipped clinical 1.5-T and 3-T systems from three major vendors, using two-dimensional (2D) gradient-recalled echo (GRE) imaging and/or 2D spin-echo (SE) echo-planar imaging (EPI). Similar MRE acquisition parameters, hardware, and reconstruction algorithms were used at each center. Mean stiffness was measured by a single observer for each phantom and acquisition on a single section. Stiffness measurement precision and same-session test-retest repeatability were assessed using the coefficient of variation (CV) and the repeatability coefficient (RC), respectively.

Results

The mean precision represented by the CV was 5.8% (95% CI: 3.8, 7.7) for all phantoms and both sequences combined. For all phantoms, 2D GRE achieved a CV of 4.5% (95% CI: 3.3, 5.7) whereas 2D SE EPI achieved a CV of 7.8% (95% CI: 3.1, 12.6). The mean RC of stiffness measurement was 5.8% (95% CI: 3.7, 7.8) for all phantoms and both sequences combined, 4.9% (95% CI: 2.7, 7.0) for 2D GRE, and 7.0% (95% CI: 2.9, 11.2) for 2D SE EPI (all phantoms).

Conclusion

MRE had excellent in vitro precision and same-session test-retest repeatability in the multicenter setting when similar imaging protocols, hardware, and reconstruction algorithms were used.

© RSNA, 2024

Supplemental material is available for this article.

See also the editorial by Tang in this issue.

Summary

MR elastography demonstrated excellent stiffness measurement precision and test-retest repeatability across five study centers, 10 scanners, three manufacturers, two field strengths, two MR elastography sequences, and three phantoms mimicking different degrees of liver stiffness.

Key Results

• ■ In this multicenter study using three phantoms mimicking different degrees of liver stiffness in humans, MR elastography demonstrated excellent precision for stiffness measurements, with a mean coefficient of variation of 5.8% for all phantoms and sequences combined.

• ■ Excellent same-session test-retest repeatability of stiffness measurement was also observed in vitro, with a mean repeatability coefficient of 5.8% for all phantoms and sequences combined.

Introduction

Patients with chronic liver disease are at high risk of liver fibrosis and cirrhosis and experience complications such as portal hypertension and hepatocellular carcinoma (1). Although liver biopsy can help provide a comprehensive evaluation of liver fibrosis staging and necroinflammatory changes, it has been largely replaced in the clinic by noninvasive methods including blood tests and US or MR elastography (MRE) measurements of liver stiffness (2). Whereas MRE generally has high diagnostic performance in various etiologies of chronic liver disease (3–9), it is important to understand the potential variability in stiffness measurement across various sequences, field strengths, and vendors when planning multicenter studies or drug trials. The Quantitative Imaging Biomarkers Alliance MRE profile aims to standardize MRE procedures, highlighting the need for reproducible measurements, and focuses on achieving sufficient accuracy while avoiding unnecessary variability of the measurement of liver stiffness (10). Most studies of liver stiffness measured at MRE have been performed at a single center and only a few have assessed interplatform reproducibility and/or repeatability of stiffness measurements in vivo or in vitro (11–24). Of note, one of these studies (25) collected multicenter data aiming to establish the normal range of liver stiffness in children, without evaluation of precision. Although single-center studies have shown that MRE has good to excellent interplatform reproducibility and test-retest repeatability (11–14,16–23,25–28), there is lack of knowledge regarding measurement precision in multiple systems and centers.

Phantoms are reliable surrogates for evaluating consistency among measurements for a wide range of frequencies and stiffnesses, as confirmed with MRE and dynamic mechanical analysis (29). Thus, this study addresses a knowledge gap with a multicenter MRE evaluation of three phantoms mimicking different degrees of liver stiffness in humans, which is more feasible than involving volunteers or patients.

The objective of this study was to determine the precision and test-retest repeatability of stiffness measurement with MRE across multiple centers using the same phantoms.

Materials and Methods

This multicenter phantom study was conducted by a subgroup of investigators from the Society of Abdominal Radiology Liver Fibrosis Disease Focused Panel and involved five centers in North America (center 1: Mayo Clinic [Rochester, Minn], which provided the phantoms; center 2: Icahn School of Medicine at Mount Sinai [New York, NY], which coordinated the study; center 3: University of Cincinnati College of Medicine [Cincinnati, Ohio]; center 4: University of Toronto [Toronto, Canada]; and center 5: University of California San Francisco [San Francisco, Calif]). Because the study did not involve human patients or animals, institutional review board approval was not necessary. Throughout the study, the same MRE phantoms were passed among the centers, starting from the coordinating center.

Phantom Design

In this study, three polyvinyl chloride homogeneous gel phantoms (Lure-Craft Industries) were constructed to mimic the stiffness range of human liver fibrosis in vivo (30–32). The mix ratios of plastic liquid to softener agent were chosen as a 45:55 ratio for the soft phantom, a 55:45 ratio for the medium stiff phantom, and a 65:35 ratio for the hard phantom. The phantoms were enclosed in nonferrous, nonparamagnetic, nonporous, and nonconducting cylindrical containers with an inside diameter of 15 cm and a lid that could be sealed airtight. Additionally, three layers of vinyl electrical tape (or equivalent) were applied around the seam to ensure a secure seal. The manufacturing ensured that phantom material had the necessary longevity for minimal loss of MRI signal and changes in measured stiffness for 2 years. Detailed instructions regarding phantom positioning for MRE acquisition were provided to each site (Appendix S1).

MRE Acquisition

MRE measurements were obtained using 10 MRI systems with two field strengths (1.5 T, five systems; 3 T, five systems) from January 2021 to January 2022. All systems used two-dimensional (2D) gradient-recalled echo (GRE) MRE pulse sequences, and six systems also used a 2D spin-echo (SE) echo-planar imaging (EPI) MRE sequence. Three centers used both 1.5-T and 3-T systems, one center used a 1.5-T system only, and one center used a 3-T system only. Moreover, four participating centers had single-vendor systems; one center did not. MRE data were collected using the same mechanical driver hardware (Resoundant) and a head coil, whereas a coronal acquisition below the midpoint of the phantom was performed with a shear-wave frequency of 60 Hz across all platforms.

To assess measurement precision, the three phantoms were scanned in a random order with different systems by using a 2D GRE sequence and 2D SE EPI where applicable and two different field strengths (1.5 T and 3.0 T where applicable) from three major vendors (Siemens Healthineers, GE HealthCare, and Philips Healthcare, assigned numbers 1–3, respectively) (Tables 1, 2). The sequence parameters were consistent across centers. The MRE examinations produced elastograms, phase and magnitude images, and confidence maps that showed the regions with high and low signal-to-noise ratio. The reconstruction algorithms used for elastograms were standard and provided by the scanner vendors; no custom algorithms were used. A second measurement for evaluating test-retest repeatability was performed in the same session with identical conditions. This involved temporarily removing the passive driver, coil, and phantom from the MRI table and then reinserting them.

Table 1:

Sequence Parameters for 2D GRE and 2D SE EPI MR Elastography Sequences

Table 2:

MRI Systems and MRE Sequences Used at Participating Centers

Image Analysis

The MRE data were uploaded in Digital Imaging and Communications in Medicine format for centralized analysis at the coordinating center (center 2). A single observer (E.O., an MRI physicist with 1 year of postdoctoral experience) analyzed the data using software (Horos version 3.3.6; Horos Project, https://horosproject.org). The observer manually placed a circular region of interest on each acquired single-section elastogram to measure mean shear stiffness of each phantom in kilopascals. The size of the region of interest was kept constant across acquisitions (10.8-cm diameter). The circular region of interest covered 50% of the total cross-sectional area inside the phantom to minimize the impact of artifacts and inaccuracies near the edges of the phantom (Fig 1).

Figure 1:

Example coronal stiffness maps of (A) soft, (B) medium stiff, and (C) hard phantoms obtained with a two-dimensional gradient-echo sequence. (D) Coronal gray-scale elastogram shows a 10.8-cm-diameter region of interest used for stiffness measurements.

Statistical Analysis

The precision of stiffness measurements was assessed through the calculation of the coefficient of variation (CV, in percentage) across systems from all sites, as well as on a per-sequence and field strength basis, as follows:

where

and i is the i-th measurement; j is the type of phantom used, with 1 denoting a soft phantom, 2 denoting a medium stiff phantom, and 3 denoting a hard phantom; n is the total number of stiffness measurements conducted for the specific phantom type, denoted as j; and Σ is the sum of all stiffness measurements from the i-th to the n-th measurement.

Test-retest repeatability was measured using the repeatability coefficient (RC) according to the Quantitative Imaging Biomarkers Alliance guidance (10):

where

where Y_i1 is the stiffness measurement for the i-th measurement from the first scan, Y_i2 is the stiffness measurement from the second scan, and wSD is within SD.

All statistical analyses were conducted (E.O.) using software (Python version 3.9.7; Python Software Foundation) using a Jupyter notebook (Project Jupyter). The Student t test was used to compare measured mean stiffness between sequences (GRE vs SE EPI) and field strengths (1.5 T vs 3 T) and 95% CIs were determined. P < .05 indicated statistical significance.

Results

Stiffness measurements for each phantom, categorized by sequence, field strength, and vendor, are in Table 3. The calculated mean stiffness values (including all measured data for both sequences) were 2.66 kPa ± 0.26 for the soft phantom, 4.62 kPa ± 0.40 for the medium stiff phantom, and 8.02 kPa ± 0.59 for the hard phantom, with a few outliers (Fig 2).

Table 3:

Stiffness Measurements for Each Phantom

Figure 2:

Dot plots show stiffness measurements (in kilopascals) of three phantoms from five participating centers. The solid black lines show the mean measurement value for each phantom. Mean stiffness was 2.66 kPa for phantom 1 (soft), 4.62 kPa for phantom 2 (medium stiffness), and 8.02 kPa for phantom 3 (hard). Superimposed box and whisker plots show medians and IQRs and outliers. Center 1: Mayo Clinic (Rochester, Minn), which provided the phantoms; center 2: Icahn School of Medicine at Mount Sinai (New York, NY), which coordinated the study; center 3: University of Cincinnati College of Medicine (Cincinnati, Ohio); center 4: University of Toronto (Toronto, Canada); and center 5: University of California San Francisco (San Francisco, Calif).

Precision of Stiffness Measurements

Precision of stiffness measurement was based on 47 measurements gathered for each phantom across all centers. The CV was 5.8% (95% CI: 3.8, 7.7) for all systems and both sequences combined. The CV for each individual phantom (for both sequences) was 6.3% (95% CI: 2.2, 10.4) for the soft phantom, 6.3% (95% CI: 2.8%, 9.8%) for the medium stiff phantom, and 4.7% (95% CI: 1.7, 7.8) for the hard phantom (Fig 3). For 2D SE EPI measurements, the mean CV was 7.8% (95% CI: 3.1, 12.6), whereas the 2D GRE sequence yielded a mean CV of 4.5% (95% CI: 3.3, 5.7) for all phantoms. There was no evidence of a difference in the mean CV of all phantoms combined between SE EPI versus GRE (P = .09). For the two SE EPI measurements, the mean CV was 8.4% (95% CI: 0, 21.3) for the soft phantom, 8.2% (95% CI: 0, 17.7) for the medium stiff phantom, and 6.9% (95% CI: 0, 16.1) for the hard phantom. Similarly, there was no evidence of a difference in the mean CV for measurements of the soft phantom when grouped as SE EPI versus GRE (P = .41), for measurements of the medium stiff phantom when grouped as SE EPI versus GRE (P = .35), and for measurements of the hard phantom when grouped as SE EPI versus GRE (P = .27) (Fig 3). For 3-T systems, the CV was 4.8% (95% CI: 2.6, 7.0) for all phantoms, 4.4% (95% CI: 2.3, 6.5) for the soft phantom, 5.4% (95% CI: 0.8, 10.0) for the medium stiff phantom, and 4.6% (95% CI: 0, 10.6) for the hard phantom. For 1.5-T systems, the CV was 6.7% (95% CI: 3.5, 10.0) for all phantoms, 8.2% (95% CI: 0, 17.0) for the soft phantom, 7.1% (95% CI: 0.8, 13.4) for the medium stiff phantom, and 4.9% (95% CI: 1.2, 8.6) for the hard phantom (Fig 3). There was no evidence of a difference in the mean CV of all phantoms combined between the two field strengths (P = .31). Similarly, there was no evidence of a difference in the mean CV for measurements of the soft phantom when grouped as 1.5 T versus 3 T (P = .34), for measurements of the medium stiff phantom when grouped as 1.5 T versus 3 T (P = .63), and for measurements of the hard phantom when grouped as 1.5 T versus 3 T (P = .91) (Fig 3). Mean CVs for all phantoms and sequences combined were 5.0% (95% CI: 2.4, 7.5) for vendor 1, 9.5% (95% CI: 5.3, 13.7) for vendor 2, and 2.8% (95% CI: 1.4, 4.2) for vendor 3.

Figure 3:

Box and whisker plots show distribution of coefficients of variation (CV) of stiffness measurements of three phantoms: means (▲), medians and IQRs, and outliers (○) for each phantom (A) and for phantoms stratified by sequence (B), field strength (C), and vendor (D). Lower CV indicates better precision. Phantom 1, soft; phantom 2, medium stiffness; and phantom 3, hard. EPI = echo-planar imaging, GRE = gradient-recalled echo, MRE = MR elastography, SE = spin echo.

Test-Retest Repeatability of Stiffness Measurement

Test-retest repeatability was assessed on 86 measurements collected from all centers. The overall RC across all phantoms and both sequences was 5.8% (95% CI: 3.7, 7.8). Based on both sequences, the RC was 6.0% (95% CI: 0.6, 11.4) for the soft phantom, 3.7% (95% CI: 1.5, 6.0) for the medium stiff phantom, and 7.3% (95% CI: 4.2%, 10.4%) for the hard phantom (Fig 4). For all phantoms combined, the RC was 7.0% (95% CI: 2.9, 11.2) for 2D SE EPI and 4.9% (95% CI: 2.7, 7.0) for 2D GRE, and no evidence of a difference in RC between the two sequences was observed (P = .29). For the 2D SE EPI measurements, the RC was 9.2% (95% CI: 0, 22.5) for the soft phantom, 4.0% (95% CI: 0.3, 7.8) for the medium stiff phantom, and 7.9% (95% CI: 1.0, 14.8) for the hard phantom. Similarly, there was no evidence of a difference in RC for measurements of the soft phantom when grouped as SE EPI versus GRE (P = .28), for measurements of the medium stiff phantom when grouped as SE EPI versus GRE (P = .81), and for measurements of the hard phantom when grouped as SE EPI versus GRE (P = .75) (Fig 4). The RC at 3 T (4.3%; 95% CI: 2.6, 6.0) showed no evidence of a difference to that at 1.5 T (6.9%; 95% CI: 3.4, 10.5) (P = .2). Similarly, there was no evidence of a difference in RC for measurements of the soft phantom when grouped as 1.5 T versus 3 T (P = .54), for measurements of the medium stiff phantom when grouped as 1.5 T versus 3 T (P = .27), and for measurements of the hard phantom when grouped as 1.5 T versus 3 T (P = .36) (Fig 4). Among the 10 systems used for the study, one system at center 4 (system 4.1, a 1.5-T system manufactured by vendor 1) had an RC greater than 10% for all phantoms combined (Fig 5).

Figure 4:

Box and whisker plots distribution of repeatability coefficients (RC) of stiffness measurements of three phantoms: mean (▲), medians and IQRs, and outliers (○) for each phantom (A) and for phantoms stratified by sequence (B), field strength (C), and vendor (D). Lower RC indicates better repeatability. Phantom 1, soft; phantom 2, medium stiffness; and phantom 3, hard. EPI = echo-planar imaging, GRE = gradient-recalled echo, MRE = MR elastography, SE = spin echo.

Figure 5:

Box and whisker plot distributions show repeatability coefficient (RC) of stiffness measurements of three phantoms provided for individual MRI systems (medians and IQRs of RCs). Scanner ID: first number is the center number, second number is the MRI unit number (some centers have only one unit). Lower RC indicates better repeatability. Of note, scanner 4.1 exhibited RC greater than 20% during the imaging of the soft phantom (phantom 1). Phantom 1, soft; phantom 2, medium stiffness; and phantom 3, hard.

Discussion

For depicting and staging liver fibrosis, MR elastography (MRE) is often accurate as a noninvasive imaging technique. However, there is limited understanding regarding the precision and repeatability of stiffness measurements in the multicenter or multivendor setting. The goal of our multicenter study was to evaluate the precision and same-session test-retest repeatability of two-dimensional (2D) spin-echo echo-planar imaging and 2D gradient-recalled echo MRE sequences in multiple systems from the three major vendors for the range of stiffness values typically observed in human liver fibrosis. Using three phantoms with different stiffness levels, identical acquisition parameters, and the same hardware and software, we found excellent precision (5.8%; 95% CI: 3.8, 7.7) and repeatability coefficient (5.8%; 95% CI: 3.7, 7.8) for stiffness measurements for all phantoms and both sequences combined. No difference was observed when phantoms were grouped per sequence and field strength, both for precision and repeatability coefficient (RC). Notably, the hard phantom exhibited the lowest coefficient of variation (CV) among the three phantoms (4.7%) and the highest RC (7.3%). Our findings are consistent with the MRE Quantitative Imaging Biomarkers Alliance profile requirement of a CV less than 7% and met the criterion of an RC less than 19%. These criteria are based on analysis from 10 single-center studies, with two studies not reporting CV, for which MRE was performed in patients at 60 Hz using 1.5-T or 3-T systems (10).

Previous single-center studies have shown that MRE has excellent interplatform reproducibility across vendors (11,12), field strengths, pulse sequences (13,16,26,27), and drivers (14). For example, Yasar et al (12) evaluated interplatform reproducibility of liver and spleen stiffness using a 2D GRE sequence in five volunteers and seven patients with two systems (3 T, GE HealthCare; and 1.5 T, Siemens). They found excellent interplatform reproducibility (intraclass correlation coefficient, >0.88), intraobserver reproducibility (intraclass correlation coefficient, >0.99), and interobserver reproducibility (intraclass correlation coefficient, >0.97) for both liver and spleen stiffness, suggesting that different platforms and observers do not substantially impact measurements (12). Trout et al (13) assessed the repeatability and agreement of liver stiffness measurement in volunteers across different MRI systems (GE HealthCare and Philips Healthcare) and pulse sequences (2D GRE, 2D SE EPI, and three-dimensional SE EPI). Their findings showed strong agreement between various pulse sequences, irrespective of the instrument manufacturer or field strength. The pairwise agreement between sequences from different manufacturers and field strengths was good to excellent, with intraclass correlation coefficient values ranging from 0.62 to 0.83, and a CV of 10.7%. Hines et al (15) and Shire et al (17) assessed liver stiffness test-retest repeatability in healthy volunteers and patients. Hines et al (15) found that the SD of the difference between two measurements taken within the same day and 2 weeks apart was 11.9% and 17.4%, respectively, whereas Shire et al (17) reported a test-retest intraclass correlation coefficient of 0.88 for same-day examinations. In patients with metabolic dysfunction–associated fatty liver disease, MRE showed reduced reproducibility with longer intervals (2 weeks) (16). Compared with these studies, our reported CV was the smallest, likely because of the homogenous nature of our phantoms.

Our study had limitations. First, minimal changes in phantom stiffness may have occurred during the study period. If any changes in phantom stiffness occurred, this did not affect measurement precision, acknowledging outlier values observed at some centers (eg, center 4), which could have been unrelated to phantom degradation. Second, we used homogeneous phantoms, which may not represent the heterogeneous distribution of liver fibrosis in vivo. Future MRE studies could use phantoms with a wider variety of stiffness levels to more closely simulate the conditions found in human liver fibrosis. Third, our study may not accurately reflect the precision of in vivo acquisitions because it did not account for factors such as breathing, motion, blood flow, variations in positioning, and susceptibility artifacts of various causes. These factors can be better addressed by using more sophisticated phantoms that mimic human physiology (33). Finally, because the phantoms were sealed airtight, the internal temperature could not be monitored during transportation between study centers or MRE acquisition. However, we do not expect this to affect the outcome of our study because the service temperature range for polyvinyl chloride gels is designed to produce durable, long-lasting products.

Our study supports the use of MRE in clinical trials and multicenter studies; precise and reproducible stiffness measurements are essential for diagnosis and follow-up in patients with chronic liver disease. This is timely because an approved treatment for metabolic dysfunction–associated steatohepatitis has been recently approved, and this may require precise posttreatment measurement of changes in liver stiffness (34). Furthermore, our study underscores the importance of maintaining a consistent protocol and mechanical driver hardware at the single or multicenter level to avoid measurement variability. Although our study does not directly address hardware and software variations, it provides the basis for developing standardized calibration methods for MRE.

In conclusion, our multicenter phantom study shows that MR elastography provides precise and repeatable measurements of stiffness in vitro when using similar protocols, mechanical driver hardware, and reconstruction algorithms, with an overall mean precision and test-retest repeatability of less than 6%.