A Transfer Learning Approach for Automatic Segmentation of the Surgically Treated Anterior Cruciate Ligament

Abstract

Quantitative magnetic resonance imaging enables quantitative assessment of the healing anterior cruciate ligament or graft post-surgery, but its use is constrained by the need for time consuming manual image segmentation. The goal of this study was to validate a deep learning model for automatic segmentation of repaired and reconstructed anterior cruciate ligaments. We hypothesized that 1) a deep learning model would segment repaired ligaments and grafts with comparable anatomical similarity to intact ligaments, and 2) automatically derived quantitative features (i.e., signal intensity and volume) would not be significantly different from those obtained by manual segmentation. Constructive Interference in Steady State sequences were acquired of ACL repairs (n=238) and grafts (n=120). A previously validated model for intact ACLs was retrained on both surgical groups using transfer learning. Anatomical performance was measured with Dice coefficient, sensitivity, and precision. Quantitative features were compared to ground truth manual segmentation. Automatic segmentation of both surgical groups resulted in decreased anatomical performance compared to intact ACL automatic segmentation (repairs/grafts: Dice coefficient=.80/.78, precision=.79/.78, sensitivity=.82/.80), but neither decrease was statistically significant (Kruskal-Wallis: Dice coefficient p=.02, precision p=.09, sensitivity p=.17; Dunn post-hoc test for Dice coefficient: repairs/grafts p=.054/.051). There were no significant differences in quantitative features between the ground truth and automatic segmentation of repairs/grafts (0.82/2.7% signal intensity difference, p=.57/.26; 1.7/2.7% volume difference, p=.68/.72). The anatomical similarity performance and statistical similarities of quantitative features supports the use of this automated segmentation model in quantitative MRI pipelines, which will accelerate research and provide a step towards clinical applicability.

INTRODUCTION

Quantitative magnetic resonance image measurements to evaluate soft tissue organization enable noninvasive assessments of healing in the anterior cruciate ligament (ACL) or graft post-surgery. Several quantitative magnetic resonance imaging (qMRI) methods are increasingly utilized in orthopaedic research to evaluate soft tissue structures of the joint.^1–3 T₂* relaxometry (and its ultra-short echo time variant^4–9) is a quantitative method that has been recently implemented to non-invasively determine the structural properties,^10–12 degeneration of soft tissues,^{4; 5; 7; 9} and to identify risk factors for graft rupture^{10; 12} and/or posttraumatic osteoarthritis.^{4; 5; 13; 14} Constructive Interference in Steady State (CISS), a T₂-weighted sequence, has been shown to be associated with the biomechanical properties of the healing ACL and ACL graft in a porcine model.¹⁵

To facilitate further adoption of qMRI sequences for the evaluation of the ACL treatments, it is necessary to address the image segmentation bottleneck. Currently, segmentation of the ACL is performed manually with an experienced individual labeling the voxels that belong to the ACL or ACL graft on a slice-by-slice basis. This process is user intensive, time consuming and requires an extended training period to achieve adequate inter- and intra-segmenter reliability. These limitations may potentially be addressed by automating the image segmentation process through deep learning.

Deep learning has been successfully applied to MRI segmentation of the brain,^16–19 and is gaining traction for knee structures such as articular cartilage and menisci.^20–22 However, to the best of our knowledge, automated segmentation of the ACL has yet to be accomplished. This unaddressed need is likely due to several challenges presented by the ACL, including poor contrast and indistinct boundaries. Previous semi-automated ACL segmentation methods struggled with the boundaries at the origin and insertion of the ligament.^{23; 24} These challenges are magnified in qMRI sequences, which often sacrifice spatial resolution for increased signal-to-noise ratio.²⁵ In surgically repaired or reconstructed ACLs, additional challenges are introduced. Residual sutures and microscopic metal particles from surgical instruments, such as drills and reamers, can create susceptibility artifacts in the MR image that obstruct the ACL. Additionally, the tissue boundaries of repaired ACLs are often less distinct than those of intact ACLs. In the case of ACL reconstructions (ACLR), the segmentation of the intra-articular portion of the graft while excluding the portions of the graft in the bone tunnels adds an additional challenge.

To address the contrast and resolution hurdles for segmenting ACLs and ACL grafts, we previously trained and validated a 2D U-Net²⁶ convolutional neural network on CISS scans of intact ACLs.²⁷ The model was capable of segmenting intact ACLs as reliably as manual segmentation in terms of anatomical similarity and quantitative features (i.e. normalized signal intensity and volume), but in a fraction of the time. Furthermore, model derived quantitative features did not differ from those derived from manual segmentation. In the present study, the pretrained model was retooled to segment surgical ACLs in a process known as transfer learning. The advantage of this approach is that it often achieves better performance than training a model from scratch on the target population (i.e. surgical ACLs), especially when the available training data are limited.²⁸

The study objective was to apply transfer learning to train this idealized model to automatically segment ACL grafts and ACL repairs. We hypothesized that the model would attain similar anatomical similarity performance to intact ACL segmentation for the healing ACL repair or ACL graft. We also hypothesized that quantitative features derived from the automated segmentation of these structures would not be significantly different from those obtained during manual segmentation. Signal intensity and volume were specifically chosen as quantitative features because they have been used as inputs to ACL structural property prediction models.^{15; 29; 30} In order for automatic segmentation to be a viable alternative to manual segmentation in our prediction model pipeline, these quantitative features must not be significantly different from manual segmentation.

METHODS

Imaging Data

CISS scans (FA=35°; TR=12.78ms; TE=6.39ms; FOV=140mm; 384x384 acquisition matrix with voxel size 0.365mm x 0.365mm x 1.5mm) were acquired from the ongoing clinical trials of the “Bridge Enhanced ACL Repair” (BEAR) procedure (BEAR I Trial: NCT02292004, IRB-P00012985; BEAR II Trial, NCT02664545, IRB-P00021470).^{29; 31; 32} All MRI scans were performed on a 3T TIM TRIO (Siemens; Erlangen, Germany) with a 15-channel transmit/receive knee coil (Siemens). For the BEAR I Trial, MR images of the surgical limb were acquired at 3, 6, 12, and 24 months post-surgery. For the BEAR II Trial, the MR images were acquired at 6, 12, and 24 months post-surgery. From this dataset, MRI scans were available at multiple timepoints for 76 BEAR subjects (238 MRI scans) and 45 ACLR subjects (120 MRI scans). These MRI scans were randomly divided by subject into 70% training (BEAR n=174, ACLR n=86), 20% validation (BEAR n=45, ACLR n=24), and 10% test sets (BEAR n=19, ACLR n=10).

Image Preprocessing

To reduce computation time, a 256x256x20 region was automatically extracted from the center of each MR image volume to decrease the volume of each scan by 89%. The dimensions of the extracted region were conservatively selected to ensure that the ACL or ACL graft would occupy the extracted region across all subjects to ensure that the structure of interest was not accidentally cropped. For ACL repairs, this procedure resulted in 3480 sagittal slices for training, 900 sagittal slices for validation, and 380 sagittal slices for testing. For ACL grafts, this procedure leaves 1720 sagittal slices for training, 480 sagittal slices for validation, and 200 sagittal slices for testing. The signal intensity of each MR image volume was standardized via z-scoring. The manual segmentations of the repaired ACL or ACL graft, which served as the ground truth, were performed using commercial software (Mimics Research 19.0, Materialise, Leuven, Belgium) by one segmenter with more than 5 years of experience segmenting ACLs and ACL grafts.

Base Model

The previously trained model²⁷ was a modified 2D U-Net²⁶ that learned on sagittal slices of intact ACL CISS scans from the contralateral knees from the same clinical trials.^{31; 32} This model architecture was selected because it is fast, flexible, and can be effective even with low to moderate sample sizes.²⁶ The optimal model for segmenting the intact ACL had a depth of 5 and filter start size of 64, with contracting path blocks consisting of two 5x5 convolutional layers with batch normalization and rectified linear unit (ReLU) activation function, a 2x2 max pooling layer with stride of 2, and a dropout layer with dropout probability of 0.5. Expanding path blocks contained a 2x2 up sampling layer with stride of 2, skip connection to the symmetric contracting path block, and two 5x5 convolutional layers with batch normalization and ReLU activation function. The Dice coefficient served as the loss function (Eq. 1) to prevent the class imbalance between ACL and non-ACL voxels from introducing bias into the model. The Dice coefficient measures the overlap between two sets of data, in this case the predicted ACL segmentation and the ground truth manual segmentation. Laplacian smoothing was added to ensure that the loss function would be continuously differentiable, and a negative sign was also added so that Dice coefficient loss was minimized.

$$Dice{coefficient}_{LOSS}=-\frac{2|y_{true}\cap y_{pred}|+1}{|y_{true}|+|y_{pred}|+1}$$ [1]

The model was trained and validated on intact ligaments from a total of 246 scans (4920 sagittal slices) that were available (scans/slices: training n=171/3420, validation n=46/920, testing n=29/580). On the test set of intact ligaments, the model achieved median Dice coefficient=.84, precision=.82, sensitivity=.85 relative to the experienced “ground truth” manual segmenter. There were no significant differences in median signal intensity or ligament volume relative to manual segmentation of the intact ACL.²⁷

Transfer Learning

Transfer learning was performed on the Google Cloud Computing platform (Alphabet Inc., Mountain View, CA, USA) with an n1-standard-8 CPU configuration and an NVIDIA Tesla K80 GPU (NVIDIA Corporation, Santa Clara, CA, USA). TensorFlow 2.1.0³³ was used as the transfer learning software platform.

Model transfer was conducted in two phases. In phase one, the pretrained model was cloned and the weights of early layers that learn low-level structures were frozen to maintain stability of the model during the first five epochs.²⁸ The number of layers frozen during phase one was optimized as a hyperparameter. A learning rate of 1E-2 was used for these first five epochs. In phase two, the layers were unfrozen, the learning rate was decreased to 1E-3, and further decreased logarithmically until 1E-5, with patience of 5 epochs, conditioned on validation loss. Model training was halted when validation loss plateaued for 10 epochs. Transfer learning was performed separately on the BEAR and ACLR datasets.

Performance

Anatomical similarity of the predicted segmentation was compared to the ground truth manual segmentation on the basis of the unmodified Dice coefficient (Eq. 2), sensitivity (Eq. 3), and precision (Eq. 4). As previously noted, the Dice coefficient is a measure of similarity between two sets of data. Sensitivity measures the proportion of ACL voxels that are correctly classified as such. Precision measures the proportion of true ACL voxels out of all voxels classified as ACL by the automated segmentation model. On these measures, a score (x) of 0 ≤ x < 0.7 would be considered “poor” performance, 0.7 ≤ x < 0.8 “fair” performance, 0.8 ≤ x < 0.9 “good” performance, and x ≥ 0.9 “excellent” performance.

$$Dicecoefficient=\frac{2TP}{FP+2TP+FN}$$ [2]

$$Sensitivity=\frac{TP}{TP+FN}$$ [3]

$$Precision=\frac{TP}{TP+FP}$$ [4]

Statistical Analysis

Statistical analyses were conducted in Python using the SciPy³⁴ and statsmodels³⁵ packages. The performance of the transferred models on the anatomical similarity measures was compared to automatic segmentation of the normal ACL using Kruskal-Wallis tests and Dunn post-hoc tests (α=.05). These non-parametric tests were selected because the anatomical similarity data were non-normally distributed (Shapiro-Wilk test: Dice coefficient p=5.7e-07, precision p=4.7e-3, sensitivity p=1.1e-7). For the available images of ACL repairs and grafts, the study was powered to detect a 5% difference in anatomical performance metrics with power=.91 and .75, respectively.

For quantitative feature comparison, the median signal intensity and volume of predicted ligament segmentations in the test set were calculated. Quantitative features data were normally distributed (Shapiro-Wilk test for repairs/grafts: signal intensity p=.70/.61, volume p=.21/.57). Therefore, statistical differences relative to ground truth manual segmentation were assessed using paired t-tests (α=.05).

RESULTS

Anatomical Performance

Optimal transfer learning on the BEAR dataset was achieved with four frozen layers in phase one. One subject was removed from the test set due to severe susceptibility artifact that occluded more than 75% of the ligament at all timepoints (Supplementary Figure 1). Metal-induced susceptibility artifacts in the intra-articular space are common post-ACL surgery, and are typically induced by microscopic remnants of the metal drill bit used to create the bone tunnels.³⁶ However, the severity in this case made accurate ground truth segmentation or prediction impossible.

The final BEAR automatic segmentation model achieved slightly lower performance on anatomic performance metrics relative to intact ACL segmentation (Figure 1; median Dice coefficient=.80, precision=.79, sensitivity=.82). This small decrease in anatomical performance was driven primarily by the presence of artifact in surgically repaired ligaments which was determined by analyzing MR images in the test set without artifact. In this cohort, anatomic performance increased to median Dice coefficient=.84, precision=.83, and sensitivity=.84. Example segmentations may be found in Figure 2.

Figure 1.

Anatomic performance metrics for automatic segmentation model transfer learning to BEAR ligaments (n=19; median Dice coefficient=.80, precision=.79, sensitivity=.82).

Figure 2.

Example segmentations of repaired ligaments without artifact (A) and with artifact (B).

The intact ligament segmentation model was transferred to ACLR grafts with seven frozen layers in phase one. The ACLR segmentation model achieved slightly lower performance than the BEAR segmentation model (Figure 3; median Dice coefficient=.78, precision=.78, sensitivity=.80). While susceptibility artifact was present in some ACLR MR images, the artifacts did not affect the anatomical similarity performance of automatic segmentation relative to ground truth to the same degree as in BEAR ligaments. Example segmentations of ACL grafts with and without susceptibility artifact may be found in Figure 4.

Figure 3.

Anatomic performance metrics for automatic segmentation model transfer learning to ACLR grafts (n=10; median Dice coefficient=.78, precision=.78, sensitivity=.80).

Figure 4.

Example segmentations of ACL grafts without susceptibility artifact (A) and with susceptibility artifact (B).

In Kruskal-Wallis tests, only the Dice coefficient demonstrated a significantly different decrease (Dice coefficient p=.02, precision p=.09, sensitivity p=.17). However, in post hoc Dunn tests, the pairwise comparisons only approached the threshold of significance (BEAR vs. Intact p=.054, ACLR vs. Intact p=.051).

Quantitative Feature Performance

There were no significant differences in quantitative features between the ground truth manual segmentation and the automatic segmentation of BEAR ligaments, even with the presence of minor susceptibility artifact (Figure 5; 0.82% median signal intensity difference, p=.57; 1.7% volume difference, p=.68).

Figure 5.

Paired comparisons of quantitative features A) signal intensity and B) volume between manual segmentation and automatic segmentation of BEAR ligaments (n=19).

For ACL grafts, there were also no significant differences in median signal intensity or volume between manual segmentation and automatic segmentation (Figure 6; 2.7% median signal intensity difference, p=.26; 2.7% volume difference, p=.72). However, large disagreements in ligament volume between automatic segmentation and ground truth were observed for three of the scans, due to disagreement around the femoral tunnel.

Figure 6.

Paired comparisons of quantitative features A) signal intensity and B) volume between manual segmentation and automatic segmentation of ACLR grafts (n=10).

DISCUSSION

Anatomical similarity performance decreased mildly in the case of both ACL repairs and grafts relative to intact ACL segmentation, but neither of these decreases were statistically significant. Crucially, the differences in anatomic similarity performance did not have any significant bearing on the signal intensity and volume of the automatic segmentations. This is a prerequisite for the use of automatic segmentation in qMRI-based prediction models of ACL structural properties.

Transferring the automated segmentation algorithm to BEAR achieved good anatomic performance in terms of Dice coefficient and sensitivity, and fair precision. The decrease in anatomical similarity performance for ACL repair segmentation was driven largely by the presence of susceptibility artifacts due to metal particles related to tunnel drilling.³⁶ These fragments sometimes distort or occlude portions of the ligament. This was demonstrated by the fact that an analysis of the test set BEAR scans without metal artifact exhibited a 2% greater sensitivity and 4% greater Dice coefficient and precision. When comparing segmentation on this cohort of MR images without artifact to intact ligament segmentation, the Dice coefficient was equivalent, precision was 1% greater, and sensitivity 1% lower. However, artifact occluded part of the ligament in 47% of test set scans. Due to the prevalence of artifact in the intra-articular space post-surgery,³⁶ it is necessary to include MR images with artifact in the training, validation and testing of the automatic segmentation algorithm to ensure its real-world usefulness.

However, as previously noted, there were no significant differences in median signal intensity or ligament volume between automated and manual segmentation, even with the inclusion of scans with minor susceptibility artifacts. This suggests that the presence of minor artifact is not an impediment to quantitative feature extraction from automatic segmentations of repaired ligaments. The exception to this was one subject in the test set in which artifact was severe enough to obscure most of the ligament (Supplementary Figure 1). Under these conditions, obtaining accurate manual or automatic segmentations of the ligament would not be possible.

Transfer learning to ACL grafts resulted in good sensitivity performance, with fair performance on Dice coefficient and precision. Unlike repaired ACLs, although artifact was present in 40% of test set MR scans, there was no difference in Dice coefficient, precision, or sensitivity when separately evaluating MR images without artifact. It is possible that this and the lower performance on reconstructed ACLs is due to the smaller sample size of the ACLR dataset relative to the repaired ACL dataset in this study. As noted in the statistical methods, the power to detect a statistically significant difference in anatomical performance for the ACL graft test set (power=.75) was lower than that of the BEAR test set (power=.91), given the smaller sample size available.

Despite the small decrease in anatomical performance on ACL grafts, there were no significant differences in median signal intensity or graft volume between automated and manual segmentation. It should be noted, however, that three MR scans from two subjects demonstrated large disparities in segmentation volume. In all three of these scans, the disagreement occurred near the aperture of the femoral bone tunnel. During manual segmentation, the ground truth segmenter only included the intra-articular portion of the ACL graft at this location. The disagreement between the algorithm and ground truth in these scans stemmed from differences in delineating the boundary between the intra-articular graft and intra-tunnel graft. This was not previously an issue in the repaired ACLs, because the native origin and insertion are intact, thus providing concrete endpoints for segmentation. In one scan, there was also some disagreement due to the presence of susceptibility artifact around the mid-substance of the graft.

The strength of the transfer learning approach used in this study is that it capitalizes on a base model initially trained on a comparatively large dataset of intact ACLs to serve as a starting point for training on relatively smaller datasets of repaired and reconstructed ACLs. This allows the automated segmentation model to become proficient at segmentation of repaired ACLs or grafts with less BEAR or ACLR data, because the model has already learned the low-level requirements for ACL segmentation from training on intact ACLs. An added benefit of this approach is that it often results in better performance when sample sizes are constrained.

There are some study limitations to consider. It should be noted that, even with the transfer learning approach, further performance gains could likely be achieved for both ACL repair and graft segmentation the number of training images were increased. Increasing the sample size of training data would likely improve performance around susceptibility artifacts, and at the bone tunnel aperture for ACL grafts, which were key areas of variability. In addition, the model was developed using a specific MR sequence (CISS) acquired on a single make and model of MRI scanner. It is likely that the transfer learning process would have to be repeated to work on other sequences and MRI scanners. Future work will focus on validating ACL mechanical property predictions¹⁵ made from automatically extracted quantitative features, transferring of the automated segmentation model to other MRI sequences, including T₂* and PD-SPACE, as well as transferring to additional MRI scanners.

In conclusion, the automated segmentation model achieved comparable anatomical segmentation performance to the base intact ACL segmentation model on repaired ACLs when no artifact was present, and lower but not significantly different performance on repaired ACLs and ACL grafts with the presence of artifact. Regardless of the presence of artifact, median signal intensity and volume of the automatically segmented repaired ACL or graft were not significantly different from manual segmentation. This finding supports this automated ACL segmentation model as a viable solution to the manual segmentation bottleneck in CISS qMRI pipelines, which will accelerate research and improve the clinical practicality of qMRI.