Reproducibility and Post-Acquisition Correction Methods for Quantitative Magnetic Resonance Imaging of the Anterior Cruciate Ligament (ACL)

Abstract

Quantitative MRI (qMRI) has been used to evaluate the structural integrity of knee joint structures. However, variations in acquisition parameters between scanners pose significant challenges. Understanding the effect of small differences in acquisition parameters for quantitative sequences is vital to the validity of cross-institutional studies, and for the harmonization of large, heterogeneous datasets to train machine learning models. The study objective was to assess the reproducibility of T₂* relaxometry and the constructive interference in steady state sequence (CISS) across scanners, with minimal hardware-necessitated changes to acquisition parameters. It was hypothesized that there would be no significant differences between scanners in ACL T₂* relaxation times and CISS signal intensities (SI). Secondarily, it was hypothesized that differences could be corrected by rescaling the SI distribution to harmonize between scanners. Seven volunteers were scanned on 3T Prisma and Tim Trio scanners (Siemens). Three correction methods were evaluated for T₂*: inverse echo time scaling, z-scoring, and Nyúl histogram matching. For CISS, scans were normalized to cortical bone, scaled by the background noise ratio, and log-transformed. Prior to correction, significant mean differences of 6.0±3.2ms (71.8%;p=.02) and 0.49±0.15 units (40.7%;p=.02) for T₂* and CISS across scanners were observed, respectively. After rescaling, T₂* differences decreased to 2.6±2.7ms (23.9%;p=.03), 1.3±2.5ms (10.9%;p=.13), and 1.27±3.0ms (19.6%;p=.40) for inverse echo time, z-scoring, and Nyúl, respectively, while CISS decreased to 0.01±0.11units (4.0%; p=.87). These findings suggest that small acquisition parameter differences may lead to large changes in T₂* and SI values that must be reconciled to compare data across magnets.

Introduction

With the increasing prevalence of quantitative magnetic resonance imaging (qMRI) to perform noninvasive and objective analyses of the knee soft tissues,^1–10 it is critical to understand the robustness and reproducibility of qMRI sequences frequently used across MR scanners. The scalability of qMRI methods, and the comparison of qMRI results across studies, depend on the inter-scanner reproducibility of a given sequence. Should differences across scanners be observed, it would be important to understand the source of variability, and how to correct for it.

T₂* relaxometry and its variants are commonly used to evaluate the integrity of the anterior cruciate ligament (ACL) or ACL graft post-surgery.^{1; 3–5; 11–13} The T₂* relaxation times of the ligament or graft have been shown to reflect its structural properties,^{1; 3; 4} and longitudinal changes in T₂* relaxation time have been associated with tissue remodeling.^{11; 13} The advantage of T₂* relaxometry and the reason behind its recent popularity is that it is directly dependent on the collagen organization of the soft tissue of interest and is considered robust to inter-scanner bias.¹⁴

However, due to the lack of standardization of T₂* sequences, acquisition parameters may vary between studies and even within a study, for reasons such as hardware differences and limitations.¹⁵ Beyond meeting minimum echo time thresholds to detect the short T₂* decay of the highly organized ACL tissue, there is considerable variation in parameters across studies, such as in the number of echoes used, echo times, and the repetition time. In recent studies, for example, the flip angle ranged from 9–30°, repetition time ranged from 13–86.2ms, and first echo time ranged from 0.032–2.8ms.^{1; 6; 11–13}

Similarly, T₂*-weighted constructive interference in steady state (CISS) scans have also been associated with ACL structural properties.^{3; 10} While typical T₂-weighted sequences are thought to have increased inter-scanner variability in the signal intensity (SI) distribution due to increased sensitivity to hardware related effects,¹⁶ this has not been empirically demonstrated for T₂*-weighted CISS. In fact, inter-scan variability in CISS may be diminished by normalizing the ACL SI to a low signal reference tissue, such as cortical bone.^{2; 10} However, like T₂* relaxometry, the sensitivity of CISS to variations in its acquisition parameters is unknown.

A number of approaches have been applied for post-acquisition harmonization of MRI scans. These range from post-processing methods that act by shifting and scaling the SI distribution to match a reference (e.g. Nyúl, z-scoring)^{17; 18} to machine learning-based interpolation methods.¹⁹ These approaches have shown promising results for both diffusion MRI^{19; 20} and radiomics,¹⁷ though MRI sequences for evaluation of the ACL, including T₂* relaxometry and the CISS sequence, have not been evaluated using harmonization methods.

Therefore, the goal of this study was to assess the reproducibility of T₂* relaxometry and the SI values of the CISS sequence across scanners with hardware-necessitated differences in acquisition parameters, and to evaluate harmonization methods to address these potential differences for the native ACL. The null hypothesis was that there would not be significant differences in T₂* relaxation time (for T₂* relaxometry) or signal intensity (for the CISS sequence) between scanners with minor differences in acquisition parameters. Secondarily, it was hypothesized that differences in T₂* relaxation time and CISS SI between scanners may be corrected by harmonizing the SI distribution.

Methods

Imaging

A total of 7 healthy volunteers were recruited after the study received IRB approval (BCH: IRB-P00033650 and RIH CMTT#: 006520). All volunteers granted their informed consent. The sample size was sufficient to detect a 10% difference in T₂* relaxation time or CISS SI with 80% power. The subjects were scanned on two magnets: a 3T Prisma (Siemens, Erlangen, Germany) at Brown University and a 3T Tim Trio (Siemens) at Boston Children’s Hospital, Waltham. A 15-channel transmit/receive coil (Siemens) was used with both scanners. A summary of acquisition parameters for both magnets is shown in Table 1.

Table 1.

MRI acquisition parameters for T2* relaxometry and CISS sequences on Prisma and Tim Trio scanners.

	T2*		CISS
MR scanner	Prisma	Tim Trio	Prisma	Tim Trio
Flip Angle [°]	12	12	35	35
Repetition Time [ms]	29	29	12.92	12.78
Echo Time(s) [ms]	[2.5, 6.9, 11.2, 15.6, 20, 24.4]	[3.4, 6.9, 11.2, 15.6, 20, 24.4]	6.46	6.39
Field of View [mm]	160	160	130 × 160	140
Acquisition Matrix [-]	384 × 384	384 × 384	512 × 270	384 × 384
Voxel Size [mm]	0.42 × 0.42 × 0.80	0.42 × 0.42 × 0.80	0.31 × 0.31 × 1.00	0.36 × 0.36 × 1.50

Quantitative Measurements

The ACL was manually segmented from the MR image stack using commercial software (Mimics Research 19.0, Materialise, Leuven, Belgium). The CISS scans were segmented by one segmenter with more than 5 years of experience. The T₂* scans were segmented by a separate segmenter with 4 years of experience. It was previously shown that there was no significant difference between the two segmenters and that inter-segmenter (ICC=.97) and intra-segmenter (ICC≥.97) reliability was high.²¹

From the CISS images, mean SI and ligament volume were extracted from the segmentation mask.¹ For the T₂* images, the T₂* relaxation time was calculated by using a custom software package in Matlab (Mathworks, Natick, MA, USA) to fit an exponential decay curve to the per echo SI on a voxel-wise basis.¹

Post-Acquisition Harmonization

For the T₂* correction, three post-processing correction methods were investigated. First, the ACL SI at each echo was scaled using the inverse echo time ratio of that echo. Second, the ACL SI was mean centered and scaled by the standard deviation of the reference sequence SI. Third, the SI histogram was scaled relative to the reference sequence standard histogram using the Nyúl algorithm.¹⁸ The Nyúl algorithm performs segmental linear histogram matching using landmarks from the reference standard histogram.^{17; 18} For all scaling methods, the results on the Prisma were treated as the reference, and images on the Tim Trio treated as the target to be scaled. Following SI scaling with each of these methodologies, the T₂* relaxation time was recalculated in the same manner as previously described.

For the CISS scan correction, one method was assessed, which was an extension of a previously described method.^{2; 10} Because of the previous success of the earlier iteration of this approach, other post-processing methods were not considered for CISS.^{2; 10} In the original method, ACL SI was normalized to the anterior cortex of the femur, which is consistently near the noise floor. Next, the normalized SI was log-base 2 transformed because the ACL SI values were not normally distributed. To this, a background noise scaling term was added to account for differences in noise texture between steady state acquisitions. To do so, the noise was calculated from segmenting the background in the same central slice used to segment the cortical bone. Prisma scans were scaled to Tim Trio scans, which had the shorter echo time.

$${SI}_{Scaled}={log}_2\left(\frac{{SI}_{ACL}+\mathrm{\Delta }Noise}{{SI}_{Bone}+\mathrm{\Delta }Noise}\right)$$ Equation. 1

Leave-One-Out Analysis

Because the z-score and Nyúl scaling methods used references that included the subjects being scaled, an additional leave-one-out analysis was performed to validate the generalizability of these scaling methods. Each subject was successively withheld from the dataset to be used as the scaling reference. The remaining 6 subjects were then scaled with each method using the withheld subject reference, and the T₂* relaxation time was recalculated.

Statistical Analysis

The differences in T₂* relaxation times and CISS signal intensity values between scanners and before and after correction between scanners were assessed with Wilcoxon signed-rank tests. Significant differences in T₂* relaxation time or SI were noted when p<.05. All statistical analyses were conducted in Python using SciPy.²²

Results

The results of the T₂* scaling methods are shown in Figure 1. A mean per subject difference of 6.0±3.2ms (71.8%) in the T₂* relaxation time was observed between the Tim Trio versus the Prisma scanner (p=.02). When scaling by the inverse echo time ratio, a mean per subject difference of 2.6±2.7ms (23.9%) was observed, which was significant (p=.03). When scaling by the z-score method, there was a mean per subject difference of 1.3±2.5ms (10.9%), which was not significant (p=.13). Scaling by the Nyúl algorithm resulted in a mean per subject difference of 1.27±3.0ms (19.6%), which was not significant (p=.40).

Figure 1.

T₂* relaxation times from the Tim Trio before and after scaling, relative to the Prisma (bars indicate the mean value; p<.05=*; p<.005=**; p<.001=***).

The results of CISS image SI scaling are shown in Figure 2. Prior to scaling, there was a mean per subject difference of 0.49±0.15 units (40.7%) in the CISS SI between the Tim Trio and Prisma (p=.02). After scaling, a mean per subject difference of 0.01±0.11 units (4.0%) was observed, which was not significant (p=.87).

Figure 2.

CISS signal intensity before and after normalization (bars indicate mean value; p<.05=*).

For the leave-one-out analysis of the T₂* correction methods, the unscaled results were significantly different between scanners regardless of the subject withheld (Table 2). There were no significant differences between scanners when scaling with either the z-score or Nyúl method, regardless of the subject withheld as the reference.

Table 2.

Leave-one-out scaling results (p<.05 indicated in bold).

Scaling Factor	Scaling Subject	Test Statistic	P-Value
Original	1	−4.332	.007
	2	−5.066	.004
	3	−6.944	.001
	4	−4.173	.009
	5	−4.493	.006
	6	−4.189	.009
	7	−4.469	.007
Z-Score	1	0.849	.435
	2	1.045	.344
	3	1.393	.222
	4	1.532	.186
	5	1.159	.299
	6	1.258	.264
	7	1.769	.137
Nyúl	1	−1.311	.247
	2	−0.747	.489
	3	−1.824	.128
	4	−0.696	.517
	5	−1.180	.291
	6	−0.858	.430
	7	−0.838	.440

Discussion

Significant differences were observed between scanners for both the unadjusted T₂* relaxation times and CISS SI values. Increasing the first echo time by 0.9ms while holding other acquisition parameters constant between scanners resulted in increased T₂* relaxation times. This increase is both statistically significant and clinically relevant. For example, in the ongoing BEAR III clinical trial (NCT03348995), the standard deviation of the mean T₂* relaxation time for the repaired ACLs was 3.22ms (unpublished data). Therefore, the observed mean difference between scanners of 6.0ms represents 1.86 standard deviations in an ACL-injured cohort. This finding emphasizes the need to reconcile qMRI data across magnets, sites, and/or studies using appropriate harmonization methods.

In this study, we determined that two scaling methods for T₂* relaxometry resulted in no significant differences between scanners with minor changes in acquisition parameters: z-score scaling and Nyúl scaling. Scaling by the inverse echo time ratio still resulted in significant differences in T₂* relaxation time. Of the successful scaling methods, z-score scaling resulted in the smallest difference in T₂* relaxation time, followed by Nyúl scaling.

Nyúl scaling presented a limitation in both the primary analysis and the leave-one-out analysis, where it compressed the range of inter-subject T₂* values (Figure 1). This could pose a problem for downstream prediction models of the ligament or graft structural properties and/or surgical outcomes by decreasing sensitivity. By reducing the inter-subject range of T₂* relaxation times, it may become more difficult to differentiate between outcomes using this metric. This limitation was not observed with the z-score scaling method.

The CISS sequence also demonstrated a statistically significant and clinically relevant difference between scanners/acquisition parameters. The mean difference of 0.49 units corresponds to 1.62 standard deviations in the same ACL-injured cohort. No significant difference was observed after correction. Furthermore, the post-correction difference was smaller than that for T₂* (4% vs. 11% respectively). This finding highlights the potential for the CISS sequence to be used as a method of assessing the ACL in multi-scanner and multi-site settings, especially given that it has the advantages of higher spatial resolution and contrast relative to T₂* relaxometry within a clinically useful acquisition time.

There are a few study limitations to consider. First, only two different Siemens scanners were evaluated. While both are commonly used, there are other manufacturers and scanner models that were not evaluated due to availability at our institutions. Second, the parameter space that qualifies as either a T₂* relaxometry or CISS scan is vast relative to the differences in acquisition parameters examined here. In the present study, we sought to make acquisition parameters as similar as possible within the hardware constraints of the scanners used at the two sites. Different formulations within this parameter space may result in quantitative measures that are more, or less sensitive to the differences in acquisition parameters. It is possible that larger differences in acquisition parameters may result in measurements that are not reconcilable by harmonization. Future studies could be designed to address this issue. Other acquisition parameter modifications, such as increasing the number of echoes in the case of T₂* relaxometry, may also reduce sensitivity to differences in acquisition parameters. However, this approach would only be applicable to a short T₂* tissue such as the ACL if additional echoes are added within the upper echo time limit. In the present study, the T₂* sequence used was already implemented with the minimum echo time interval. For the CISS images, normalizing to the femoral cortical bone may show decreased efficacy in elderly patients with osteoporosis, diabetes, HIV, or other conditions resulting in degenerative changes to cortical bone architecture and composition.^23–25 These bone alterations result in changes to MR properties of the bone, such as an increase in magnetic susceptibility.²³ However, ACL injuries are most common in adolescent patients and active young adults who typically have good bone quality.^26–29 Lastly, some of the variation observed may be due to segmenter reproducibility. To address this potential issue, only experienced segmenters with validated reliability were used for this study. On a relatively large tissue such as the ACL (~10,000 voxels), median T₂* relaxation time and median CISS SI are insensitive to segmentation variability.²¹ Furthermore, the observed inter-scanner effect sizes for both T₂* relaxometry and CISS sequence far exceeded previous measures of inter- and intra-segmenter reliability.²¹

These study results emphasize the need for post-acquisition harmonization of qMRI measures of the ACL when performed on different scanners and across institutions, as commonly is the case in multicenter studies. In a systematic review, Van Dyck et al. showed a wide range of MRI sequences and acquisition parameters used for evaluation of the ACL.¹⁵ A more cursory review of the literature specific to T₂* relaxometry (Supplementary Table 1) also indicates a wide variation in T₂* acquisition parameters for quantitative measurements of the ACL. A minimum requirement for measurement of T₂* decay in the ACL is a first echo time less than 10ms,¹³ but besides this consideration there is no standardization in echo times, repetition time, flip angle, voxel size, etc. This is due in part to a combination of hardware limitations requiring sequence modifications across scanners,¹⁶ and the balancing of unavoidable tradeoffs between signal-to-noise ratio, resolution, and acquisition time, among other challenges.¹⁴

Differences in measures of both T₂* relaxation time and CISS SI values due to different hardware and acquisition parameters may propagate into large errors in downstream prediction models of ligament or graft structural properties and/or surgical outcomes.^1–5 In addition, due to the lack of standardization of qMRI sequences, comparisons of results between studies utilizing different acquisition parameters or hardware without harmonization would be limited. Furthermore, to leverage large, heterogeneous qMRI datasets for training machine learning models (e.g., for predicting ACL failure load), harmonization will likely be necessary. In the present study, when acquisition parameters were set to the minimum hardware-necessitated differences, z-score scaling was an effective method for harmonizing differences in T₂* relaxation time. Scaling CISS SI by the difference in background noise effectively harmonized CISS. In the future, harmonization methods could be added to standard post-processing software for both clinical and research applications. Emphasis should also be placed on the standardization of qMRI acquisition parameters should be sought to preemptively alleviate this reproducibility challenge.