Multi-label ensemble model for knee joint anatomy and lesion segmentation: Segmentation of Clinical images with Ensembling, Morphology, and Attention (SCEMA)

Abstract

Purpose:

Quantitative MRI analysis holds significant promise for improving the early diagnosis and prognosis of osteoarthritis (OA), where accurate tissue and lesion segmentation is a critical step for reliable quantification. In particular, bone marrow edema-like lesions (BMELs) are associated with OA, but accurate segmentation methods remain elusive.

Methods:

A fully automated pipeline – Segmentation of Clinical images with Ensembling, Morphology, and Attention (SCEMA) – was developed for segmenting bone, cartilage, and BMEL from standard clinical knee MRI. SCEMA leveraged a UNet-based network enhanced with an added attention mechanism, model ensembling, and morphological post-processing. A range of network architectures, model configurations, and post-processing pipelines were evaluated to find the optimal model. Statistical analyses on the segmentation annotations were also conducted, the size and quality of BMEL lesions.

Results:

The model achieved testing Dice similarity coefficients (DSC) of 0.684, 0.966, 0.830, 0.956, and 0.770 for BMEL, femur bone, femoral cartilage, tibia bone, and tibial cartilage respectively. For BMEL, the model performance was found to be more strongly correlated with BMEL quality than size.

Conclusion:

Fully automatic, high-quality anatomical and lesion segmentation was attained from standard clinical knee MRI. For reproducibility, BMEL quality labels and segmentation annotations were shared to the public.

1 |. INTRODUCTION

Osteoarthritis is the most common form of arthritis and a leading cause of disability,¹ with the knee being the most frequently affected joint.² Although there is currently no effective cure for knee osteoarthritis (KOA), early diagnosis is critical for slowing disease progression and managing symptoms.³

Magnetic resonance imaging (MRI) provides a powerful tool for early detection, enabling visualization of multiple joint tissues including cartilage, meniscus, bone, ligament, and synovium.⁴ MRI also can detect osteophytes, joint space narrowing, bone marrow edema-like lesions (BMELs), and other structural abnormalities before radiographs or symptomatic testing can.⁵ Identification and measurement of these structures serve as imaging biomarkers, enabling quantitative studies on KOA progression.

BMELs are of particular interest due to their strong association with KOA progression and pain,^6,7 including in post-traumatic cases following injuries like anterior cruciate ligament and meniscus tears.^8,9,10 BMELs appear as high-intensity regions in fluid-sensitive MR sequences, surrounded by low-intensity, fat-rich marrow.

Annotation of OA imaging biomarkers by expert radiologists is an expensive and time-consuming process. Reliable measurements may be unlikely when combined with high inter-reader variability, as is the case with BMELs.¹¹ BMELs vary widely in terms of size, shape, and brightness, and may be difficult to identify and delineate in some cases. The MRI Osteoarthritis Knee Score (MOAKS) system¹² addresses the low reliability by presenting a low-resolution, semi-quantitative grading scheme: femur, tibia, and patella bone are divided into 14 subregions, each of which is graded for the number and relative volume ([0 ~ 3]/3 of the subregion) of BMEL.

Deep learning has emerged as a promising strategy for automatically segmenting OA-related structures, especially bone, cartilage, and BMELs.

Deep learning has emerged as a successful strategy for automatically segmenting OA-related anatomical structures, including bone and cartilage. Studies in BMEL segmentation are promising, but remain a challenge. Prior approaches for BMEL have included supervised learning,¹³ generative modeling,^14,15 and patch-based detection methods,¹⁶ where a model can be trained on a small set of human annotations and then deployed for automatic detection and measurement. However, these methods rely on manually pre-selected regions or volumes with large, bright BMELs^17,18,19 and thus suffer from limited generalizability.

Prior approaches for segmenting anatomical structures such as bone and cartilage often depend on high-resolution, non-clinical MR sequences with long acquisition times.²⁰ For example, the 3D double echo steady state (3D-DESS) sequence is ideal for bone and cartilage segmentation due to high contrast boundaries and high spatial resolution. Although these methods yield accurate results, they are impractical for clinical use due to limited accessibility and poor visualization of BMELs. The sagittal, fat-suppressed turbo spin echo (TSE) sequence is ideal for detecting and measuring BMELs, since it maximizes the contrast between the fluid edema and fatty, healthy bone marrow.

In this study, we propose Segmentation of Clinical images with Ensembling, Morphology, and Attention (SCEMA). It is a fully automated method that segments BMELs, femur bone, femoral cartilage, tibia bone, and tibial cartilage using only intermediate-weighted TSE (IW-TSE) images (Figure 1), a sequence widely available in clinical knee MRI protocols. SCEMA trains an ensemble of five-class, multi-label, fully convolutional neural network with attention layers using a total of 495 images and matching annotations. For better generalizability on BMELs, we also introduce a qualitative, four-level ordinal labeling system for BMEL annotation: no BMEL, and low, mid, and high quality BMEL. By training the model on a balanced dataset of image with various combinations of BMEL presence, size, and density, we ensure strong performance on the detection and segmentation of BMEL of all sizes and qualities while simultaneously producing high quality anatomical structure segmentations. And finally, we release our BMEL quality labels and segmentation annotations to the public to support consensus-building around its annotation standards and segmentation methodology.

FIGURE 1.

A clinical sequence (IW-TSE) MRI slice of the knee joint, overlaid with the segmentation model prediction of the five target classes: bone marrow edema-like lesion (pink), femur bone (blue), femoral cartilage (orange), tibia bone (red), and tibial cartilage (green).

2 |. METHODS

In this section we detail the steps taken to create SCEMA. Each step in the data pipeline is presented in sequential order, as outlined in Figure 2. Findings drawn directly from the subjects and their MRI are also presented in this section.

FIGURE 2.

Overview of the study design of SCEMA, including mask registration, 4-fold cross-validation, ensembling, and post-processing. Training (solid-line box) occurs in TSE space (green), but requires DESS images and masks (red) for annotation data. Inference (dotted-line box) occurs exclusively in TSE space.

2.1 |. MRI Acquisition and Demographics

The Osteoarthritis Initiative Zuse Institute Berlin (OAI-ZIB) public knee MRI dataset^21,20 was used in this study, with approval granted by the local institutional review board. Pairs of sagittal, fat-suppressed, IW-TSE and 3D-DESS images were collected. Acquisition protocols are given in Table 1. Inclusion criteria for participation were adults aged 45–79 years who either had symptomatic KOA or were considered at elevated risk based on factors such as obesity, history of knee injury or surgery, knee pain within the past year, or a family history of total knee replacement. Exclusion criteria included the presence of rheumatoid or other inflammatory arthritis, bilateral end-stage knee OA or planned bilateral knee replacement, contraindications to MRI, use of ambulatory aids other than a cane, and serious comorbidities. The subjects had moderate radiographic OA, with an average Kellgren-Lawrence Grade of 2.53±1.37 and had an age of 61.87±9.33 years. The subjects exhibited slightly elevated BMI at 29.27±4.52 kg/m² and were 51.68% male.

TABLE 1.

MRI acquisition parameters for the intermediate-weighted turbo spin echo (IW-TSE) and 3D double echo steady state (3D-DESS) sequences.

Parameter	IW-TSE	3D-DESS
Vendor	Siemens
Model	Trio 3T
Orientation	Sagittal
Fat Suppression	Yes
Repetition Time (ms)	3200	16.32
Echo Time (ms)	30	4.71
Image Resolution (mm)	0.357×0.357	0.366×0.366
Slice Thickness (mm)	3.0	0.7
Matrix Size	444×448	384×384
Number of Slices	32 – 44	160

2.2 |. Mask Annotations

The mask annotations were derived from two sources: OAI-ZIB anatomy masks and in-house BMEL masks. OAI-ZIB provides femur and tibia bone and cartilage segmentation annotations in DESS, produced using convolutional neural network model outputs and subsequent manual corrections.²⁰

For the in-house BMEL annotations, each TSE image was reviewed by a trained expert for BMEL quality: No BMEL, Low, Mid, and High. We considered various qualities of BMELs to produce ordinal labels indicating the discernibility of the BMELs for manual detection and segmentation. First, a 1cm diameter threshold was used to distinguish low quality BMELs. Larger BMELs were considered mid or high quality. Then, edema density (relative signal intensity), definition (clarity of boundaries), and distance from the tibiofemoral and patellofemoral joint articular surfaces were used to separate mid and high quality BMELs. These qualities were then considered to override the size-based labeling. For example, a very large but diffuse BMEL could be labeled low quality. BMELs were then segmented semi-automatically using 3D Slicer,²² first by setting a user-defined thresholding value for each slice, then drawing a rough region-of-interest (ROI) to isolate the high intensity BMEL from the surrounding low intensity fat-suppressed bone marrow. These steps ensure that the appropriate region and intensity values are selected. We also performed an analysis on the BMEL annotations, noting the distribution of BMEL qualities and sizes for whole images as well as individual sagittal slices.

2.3 |. Pre-Processing and Augmentation

BMEL annotation was produced in TSE space and no registration was required. However, we noted some undesirable characteristics in the annotations: small islands of high-intensity, non-BMEL voxels inside the ROI that are erroneously labeled as BMEL. To overcome this issue, a variety of optional BMEL mask processing steps were considered: connected components with a connectivity of one and a minimum size threshold of 30 voxels, and eliminating BMELs which spanned only one slice. These steps were applied before model training as a pre-processing step, and/or after model training on both the annotations and model predictions as a post-processing step.

At the end, we applied the following basic transformations to optimize the data for model training. Each TSE image was scaled and center-cropped to uniform dimensions. The peripheral voxels were safely cropped out since the patellofemoral and tibiofemoral joints are placed at the center of the MRI field-of-view. This fact was confirmed using the provided bone and cartilage masks. We also applied spatial and intensity-based data augmentations (affine transforms, horizontal flips, intensity and contrast shifts, Gaussian and Rician noise, and bias field) to aid model robustness. The implementation of these augmentations was provided by Medical Open Network for AI (MONAI).²³

2.4 |. Network Architecture and Design

SCEMA used multiple instances of 2D AttentionUNet, a UNet-based network architecture with attention layers.²⁴ Starting with 64 initial filters, the network used 3 layers of convolution-normalization-activation blocks, each containing group normalization and ReLU, followed by a dropout rate of 0.2. In the downsampling path, more than one block per inner layer were used: one block at level 0 (full resolution), two blocks at level 1 and 2, and four blocks at level 3. The model had a single-channel input layer for the TSE slice and a five-channel output layer for each of the target components, followed by a sigmoid activation function.

2.5 |. Model Training

The dataset of 495 images and annotations were split randomly into five subsets, stratified by BMEL quality. We employed 4-fold cross-validation to monitor data variance and enable model ensembling. The fifth hold-out set was used for testing. To ensure no data leakage between the cross-validation folds, we employed the data splitting at the patient level and isolated all slices of one patient to one fold only.

The training objective was multi-label segmentation with a five-channel output of BMEL, femur bone, femoral cartilage, tibia bone, and tibial cartilage masks. The input was a single-channel TSE image. The model used a sigmoid final activation head and a combination of soft Dice Similarity Coefficient (DSC), binary cross entropy (BCE), and focal losses.²⁵ Each loss function ${ℒ}_f$ were weighted with tunable hyperparameters w_f to scale the initial loss values to equivalent ranges, and the losses for each target component ${ℒ}_{f\in F}\left(y_c,{\hat{y}}_c\right)$ were similarly weighted (w_c) emphasize BMEL segmentation in early training epochs. The combined loss term is defined in Equation 1. For validation and testing, a thresholding value of 0.5 was used to produce a binary output.

$$\begin{gathered} ℒ=\sum _{c\in C}\sum _{f\in F}{w}_c{w}_f{ℒ}_f\left(y_c,{\hat{y}}_c\right), \\ \text{where}C\in \{\text{Bone},\text{Cartilage},\text{BMEL}\}, \\ F\in \{\text{DSC},\text{BCE},\text{Focal}\} \end{gathered}$$ (1)

The weight hyperparameters with the best results were as follows: w_Bone = 1, w_Cartilage = 2, w_BMEL = 5, w_DSC = 1, w_BCE = 5, and w_Focal = 5. Other hyperparameters included batch size of 32, learning rate of 4e-5 without weight decay, and the Adam optimizer was used with β₁ = 0.9 and β₂ = 0.999. The maximum number of training epochs was set to 500 and validation BMEL DSC was monitored once every epoch and used as the early stopping criterion.

2.6 |. Mask Post-Processing

Once the training was complete, four models from each cross-validation fold were deployed on the hold-out test set. The model outputs were post-processed as described in Section 2.3 then ensembled using a voxel-wise majority vote to eliminate low-consensus predictions. An important consideration for post-processing was the order of operations. We focused on the ordering of the two most significant steps: connected components and ensembling. Our final post-processing pipeline was ensembling, connected components, and removing single-slice BMELs, in order.

2.7 |. BMEL Quality Analysis

This study introduced the BMEL quality annotation system, which is a qualitative measurement. We sought to validate these annotations and conducted statistical analyses on the correlation between BMEL quality, annotated segmentation volume, and predicted segmentation accuracy. We used the Spearman correlation coefficient (ρ) for all analyses since BMEL quality is ordinal and annotated segmentation volume was not normally distributed. For comparison with prior work, we also computed the coefficient of determination (R²) between annotated segmentation volume and predicted segmentation accuracy using simple linear regression.

3 |. RESULTS

Since supervised learning is a function of the quality of the input data, we first note some characteristics and outcomes of the data pre-processing steps and BMEL annotation. Then we evaluate the performance of the segmentation model, in context of the input data.

3.1 |. BMEL Quality Annotations

We found a balance of BMEL qualities in the OAI-ZIB dataset, including No BMEL, Low, Mid, and High. Details are shown in Table 2 and Figure 4. There was a strong correlation between quality and annotated volume (Spearman’s ρ = 0.849, p < 0.001), with a heavily right-skewed distribution for each and extreme outliers on the large-volume end. When considering individual slices, the skew was even more stark. This is attributable to the fact that most BMEL present as small regions of high relative intensity (~ 2.52cm³), but few are extremely large, diffuse, and low-intensity, covering nearly the entire bone in some slices (up to ~ 16.70cm²). As a binary annotation mask, there is no distinction between bright and distinct BMEL and dim and diffuse ones. In addition, BMEL occupied less than 0.90% of all dataset voxels.

TABLE 2.

Summary statistics of BMEL annotations across different qualities.

3D Images
BMEL Quality	No BMEL	Low	Mid	High	Has BMEL	All
Count	101	114	163	117	394	495
% with BMEL	0.00	100.00	100.00	100.00	100.00	79.60
Median Volume (mm3)	0.00	188.84	1251.28	4177.81	1345.79	677.30
Mean Volume (mm3)	0.00	646.37	1952.51	7330.72	3171.68	2524.53
Volume Skew	N/A	4.22	1.69	3.28	5.16	5.61
2D Slices
BMEL Quality	No BMEL	Low	Mid	High	Has BMEL	All
Count	3772	4249	6100	4355	14704	18476
% with BMEL	0.00	16.31	28.08	37.36	27.43	21.83
Median Area (mm2)	0.00	0.00	0.00	0.00	0.00	0.00
Median Nonzero Area* (mm2)	0.00	17.47	38.39	105.61	46.30	46.30
Mean Area (mm2)	0.00	5.78	17.39	65.65	28.33	22.55
Mean Nonzero Area* (mm2)	0.00	35.44	61.93	175.72	103.28	103.28
Area Skew	N/A	9.16	4.25	4.23	6.88	7.71

^*Nonzero area per slice is computed for slices with BMEL only.

FIGURE 4.

Relationship between BMEL annotation and predicted volume (left), and DSC (right). Colored by BMEL quality. Note: volume axes are in symmetric logarithmic scale, with a linear threshold of 0.1.

3.2 |. Segmentation Model Performance

After training, SCEMA produced segmentations that were largely in agreement with the annotations. Sample results are shown in Figure 3. The best performing version was an ensemble of four 2D AttentionUNet from the four-fold cross-validation with a majority vote, followed by post-processing of connected components with a minimum component size of 30 voxels for BMELs. It had a DSC of 0.6839, 0.9660, 0.8295, 0.9555, and 0.7697 for BMEL, femur bone, femoral cartilage, tibia bone, and tibial cartilage respectively, on the hold-out test set of 114 images, demonstrating strong improvements over baseline models (Table 3). These findings were corroborated by statistical analyses, where each target component (except for low quality BMELs) exhibited significant improvement over each baseline according to paired Wilcoxon signed rank tests. Since data splitting includes stratification by BMEL quality, BMEL found in the test set is representative of the entire OAI-ZIB dataset. Each fold in isolation had an average DSC for BMEL of 0.656±0.018, highlighting the balance of the data splitting.

FIGURE 3.

Four sample TSE slices with No BMEL, Low, Mid, and High Quality (top to bottom) BMELs, with annotations and predictions (left to right).

TABLE 3.

Testing Dice Similarity Coefficient (DSC) metrics per target class for each model.

Model	BMEL					Bone		Cartilage
	NL	LQ	MQ	HQ	All	Femur	Tibia	Femur	Tibia
UNet+ResNet5013	0.2000	0.5631	0.6511	0.6990	0.5438	-	-	-	-
UNet26	0.4000	0.5511	0.5948	0.6842	0.5753	0.9565	0.9472	0.8055	0.7542
DynUNet27	0.2500	0.4787	0.6331	0.6958	0.5512	0.9564	0.9481	0.7973	0.7195
Swin-UNETR28	0.3000	0.5461	0.6249	0.6835	0.5674	0.9555	0.9425	0.8082	0.7490
SegResNet29	0.3500	0.5579	0.6038	0.6801	0.5701	0.9554	0.9474	0.8006	0.7483
AttentionUNet24	0.4000	0.5934	0.6279	0.6801	0.5947	0.9576	0.9465	0.8144	0.7580
SCEMA-3D (ours)	0.4000	0.5882	0.6597	0.7332	0.6190	0.9571	0.9486	0.8202	0.7653
SCEMA (ours)	0.7000	0.5174	0.6942	0.7865	0.6839	0.9660	0.9555	0.8295	0.7697

For BMEL, per-quality metrics are reported: No Lesion (NL), Low Quality (LQ), Mid Quality (MQ), High Quality (HQ), and All. Underlined values denote statistically significant DSC improvements of SCEMA relative to baseline models, using Wilcoxon signed-rank tests. The significance threshold is adjusted via Bonferroni correction for the five target classes (p < 0.05 corrected to p < 0.01); BMEL subsets based on quality are treated as a single family. For the UNet+ResNet50¹³ model, which segments only BMEL, no correction is applied.

As shown in Table 4, there was a strongly positive and significant correlation between DSC and annotated BMEL quality, compared to DSC and annotated BMEL volume. This is consistent with our labeling BMEL quality based on criteria more than just volume. There was no linear correlation (R²) between DSC and BMEL volume, and even the non-linear correlations (ρ) were rendered nonexistent when non-BMEL volumes were included. This was likely due to noise in DSC for non-BMEL images (discussed in Section 4.4). For the other components, bone volume was weakly correlated with model performance and much more strongly for cartilage. In both cases, model performance on tibia was more sensitive to volume differences.

TABLE 4.

Correlation metrics Spearman’s correlation coefficient ρ and coefficient of determination R² for model segmentation performance (DSC) with target annotation volume and BMEL quality.

	BMEL		Bone		Cartilage
Correlate	All	Has BMEL	Femur	Tibia	Femur	Tibia
Volume (R2)	0.002	0.005	0.013	0.045	0.068	0.143
Volume (ρ)	−0.073	*0.181	0.094	*0.191	**0.305	***0.548
BMEL Quality (ρ)	0.045	***0.416	-	-	-	-

For ρ, significance is denoted by *p < 0.05, **p < 0.01, ***p < 0.001.

4 |. DISCUSSION

In this study, we developed SCEMA, a fully automated deep learning pipeline for the segmentation of five components: BMEL, femur bone, femoral cartilage, tibia bone, and tibial cartilage. It is a model that requires only a single, fast, and clinical imaging sequence: IW-TSE. We prepared the data on 495 publicly available images and generated annotations, including BMEL quality labels and segmentations. We focused our work on the segmentation performance on all qualities of BMEL. In addition to comparing our work with existing literature, we ablated our training configurations to find the best model.

4.1 |. Annotation Data

With BMEL annotation, we made a concerted effort to identify and segment lesions of all qualities. We defined criteria and tools to maximize consistency across the dataset, but we noted some instances, especially with low quality BMELs and lesion-free images, in which experts may reasonably disagree. Even high quality BMELs span slices with very small ROIs, contributing to the extremely right-skewed distribution of BMEL areas.

4.2 |. Clinical MRI Anatomy Segmentation

Since clinical MR images like TSE are more readily accessible and available than non-clinical ones, there has been an increased demand to analyze clinical images directly. The primary challenge lies with the large slice thickness. With this motivating factor in mind, we achieved bone and cartilage segmentation results on TSE that are comparable to the state of the art. Zhou et al.³⁰ used a UNet-based convolutional neural network and conditional random fields on a dataset of 20 fast spin echo images. Using leave-one-out cross-validation, the authors reported validation DSC of 0.970, 0.806, 0.962, and 0.801 on femur bone, femoral cartilage, tibia bone, and tibial cartilage, respectively. Yang et al.³¹ used a conditional generative adversarial network with a UNet generator and fully convolutional discriminator to produce cartilage segmentation on heterogeneous clinical images, reporting testing DSC of 0.812, 0.823, and 0.750 on femoral cartilage, lateral tibial cartilage, and medial tibial cartilage, respectively. Stirling et al.³² used UNet and post-processing to achieve a testing DSC of 0.93 on femur and tibia bone.

4.3 |. BMEL Segmentation

Many efforts have been made to automatically identify, locate, and segment BMEL in recent years, as well as semi-automatically for decades. As with our work, they were motivated to solidify BMEL’s association with OA and reduce the high-cost manual annotation methods, which is the current gold standard. Since it is difficult to manually segment BMELs (intra- and inter-reader reliability of 0.7123 and 0.6423 DSC, respectively¹⁴) and because there are no shared datasets, it is difficult to compare the various methods directly. Improving reproducibility was a motivating factor in publicizing our entire BMEL annotation dataset.

Early attempts to quantify and segment BMEL included contrast-enhanced imaging, landmarks, and thresholding;³³ k-nearest neighbors algorithm was used in later attempts segmenting with for non-contrast imaging,³⁴ but still required human intervention at inference time.

Preiswerk et al.¹⁶ used an alternative method for BMEL data annotation, wherein a square patch centered around each BMEL was used as input to the model instead of full 2D slices. The main benefit of this method was to guarantee that each sample has at least some BMEL, making the segmentation task easier for the model to learn. With this method the authors achieved a testing DSC of 0.70. However, this method was not fully automatic because it required manual prompting during inference time. Notably, some meta-analysis of the data and model performance showed a negative correlation between annotation volume and the number of patches.

Stirling et al.³² presented work on BMEL segmentation that was notable in its fully 3D data pipeline, including the network architecture and evaluation metrics. Femur and tibia bone was segmented alongside the BMEL as well, though not cartilage. The authors also included images without BMEL in their training data, an encouraging sign for proper identification of non-BMEL images during deployment. They reported a testing DSC of 0.68.

Ponnusamy et al.¹⁸ reported results from training a variety of different network architectures, including U-Net, U-Net++, Sharp U-Net, U-Net+ResNet34, ultimately proposing U-Net structure where the encoding stage consisted of InceptionResNetV2 blocks. The authors reported a testing DSC of 0.60.

Yu et al.¹⁴ and Qin et al.¹⁵ attempted a novel approach to BMEL segmentation: semi- and unsupervised learning using generative models. The generative models were trained to synthesize healthy images, then compared against the real, potentially unhealthy images. The difference would then be considered the anomalous BMEL regions. To align the fake and real images, the model was conditioned on a variety of input data, including the same image from a different sequence, the same image with the bone regions removed, and the bone region of the same image from a different sequence. These works were inspired by recent progress in generative image tasks such as inpainting, colorization, and uncropping, using conditional generative adversarial networks³⁵ and diffusion models.³⁶ While eliminating the need for annotations, these methods were overly reliant on the intensity value differences between anomalies and the surrounding bone marrow and did not specify BMELs over other high-intensity structures and artifacts such as blood vessels and partial voluming. As such, their reported testing DSC were low, at 0.23 for Yu et al.¹⁴ and 0.42 for Qin et al.¹⁵

Moradi et al.¹³ presented an end-to-end method for connecting knee MRI to clinical OA outcomes by using a deep learning model to segment BMEL as well as the MOAKS subregions, using existing MOAKS readings to match the segmentation results, and then classifying the outcomes based on BMEL and OA progression over time. As an intermediate step, BMEL segmentation methods and results were reported, showing a very strong validation DSC of 0.88. The authors offered implementation details sufficient for reproduction: a UNet architecture with a ResNet50³⁷ backbone, pretrained on ImageNet and finetuned using DSC loss only. Though we could not reproduce these results directly without the annotation data, we used the provided codebase and modified it as little as possible to fit our in-house BMEL data pipeline into the given network construction and model training framework. We trained the model to convergence with the same stratified splitting and hold-out set testing as with all other experiments in this study. With our dataset, the testing DSC dropped to 0.5438 (Table 3). Nevertheless, the authors made a significant contribution in demonstrating the relationship between BMEL and OA progression in a self-contained pipeline whose intermediate outputs were measurable and whose final output was clinically relevant and actionable.

As we did with the work of Moradi et al., we tested other proposed methods by implementing their network architectures while keeping the training and testing configurations the same, as shown in Table 3. These results show conclusively that SCEMA outperforms all previous methods, thereby affirming its ability to learn BMEL segmentation without suboptimal convergence on any of the other components.

4.4 |. Performance Correlates

With our focus on segmenting images of all BMEL qualities, SCEMA achieved balance and high performance across all qualities except low. These results are contrary to previous experiments, which reported very strong correlations between BMEL volume and segmentation performance (R²=0.94¹⁶ and R²=0.84³²). Our stratified training dataset, especially the inclusion of images with no BMEL, is likely an explanation for this difference. We tuned our post-processing to favor conservative predictions and true negative rate, and so it is not surprising to find low DSC on low quality BMEL. Furthermore, images with low BMEL quality or no BMEL are prone to noisy DSC when the number of true voxels is zero or very small. In particular, when the model correctly predicts no BMEL, the resulting DSC, given by Equation 2 with smoothing factor ϵ, is ϵ/ϵ = 1, while a single-voxel mis-prediction would result in ϵ/(1 + ϵ) ≈ 0.

$$\text{DSC}=\frac{2\text{TP}+ϵ}{2\text{TP}+\text{FP}+\text{FN}+ϵ},ϵ>0$$ (2)

4.5 |. Model Dimensionality

Regarding dimensionality, we had hypothesized that 3D models would perform better on bone and cartilage while 2D models would perform better on BMEL, based on the dimensionality of the annotations and the size and shape of the objects of interest. We found that neither 3D nor context slices (2.5D) improved performance on any components, when controlled for model size. Due to compute limitations, we did not test models larger than 1.4 GB. However, we do not believe that parameter count was a limiting factor in this setting. More details and results can be found in the supplementary documents.

4.6 |. Post-Processing

The impact of various post-processing steps were analyzed, including ensembling and connected components. We observed that they improved performance significantly for BMEL, but had diminishing returns when combined. Ensembling, connected components, and removing single-slice BMELs had the same general effect: removing weak, likely false positive predictions. Thus we conclude that connected components is the computationally cheapest and effective post-processing method. Pre-processing, on the other hand, had an unexpected result. We had hypothesized that a deterministic algorithm like connected components for adjusting the BMEL annotations before training would make them more consistent and easier to learn, but we found that the opposite was true, even when the same algorithm was applied during post-processing. More details and results can be found in the supplementary documents.

4.7 |. Study Limitations

Some aspects of this study were limited by the lack of resources, including data and compute. Our choice of image dataset was limited to the public availability of the bone and cartilage annotations. Because we intentionally focused on a single clinical sequence, other variables in image acquisition were homogeneous, with the same vendor, model, and protocol. Thus, it is difficult to know whether the model we propose is generalizable to other images and robust to artifacts and other suboptimal conditions. Similarly, all of our in-house BMEL segmentation annotations were produced by one expert, without an inter- or intra-reader variability study. A lack of consensus on the annotations puts our model at risk of learning the expert’s biases and errors. By making our BMEL annotations publicly available, we hope it can encourage consensus-building and to help other studies with inter-reader reliability.

Due to the large size of the dataset, image dimensions, and GPU limitations, we did not run a comprehensive hyperparameter search or ablation study. Additionally, some configurations, such as larger batch sizes in 3D models, were not possible without access to more advanced compute.

5 |. CONCLUSIONS

We present a deep learning-based, five-class segmentation model trained on a single IW-TSE MR sequence that achieves accurate segmentation of BMELs, femur bone, femoral cartilage, tibia bone, and tibial cartilage, and BMELs. Our approach bridges the gap between high-quality anatomical and lesion segmentation and accessible clinical imaging. With a thorough exploration of pre- and post-processing methods and model configurations, we show that a single model can be trained to segment all five components with state-of-the-art performance. We introduce and statistically validate a system for labeling the quality of BMEL in each image and stratify the training dataset accordingly, ensuring strong performance on a wide range of BMELs. By publicly releasing our annotation dataset along with our insights, we aim to support future research and facilitate comparative studies in this domain.

Supplementary Material

Supplementary Material

Figure S1. Four sample TSE slices with false positive and false negative model predictions, with annotations and predictions (left to right).

Table S1. Effects of 2D and 3D models and training BMEL quality on BMEL segmentation performance.

Table S2. Effects of the number of context slices in 2.5D models on segmentation performance.

Table S3. Effects of 2D and 3D models and model size on segmentation performance. The number of initial filters (Init. Filters) is adjusted to make the model sizes as similar as possible across 2D and 3D.

Table S4. Effects of loss functions and target classes on BMEL segmentation performance.

Table S5. Effects of BMEL annotation pre-processing on BMEL segmentation performance. Pre-processing includes connected components with a minimum size of 30 voxels, followed by removing all annotations that only span one slice.

Additional supporting information may be found in the online version of the article at the publisher’s website.

Data availability statement

The data that support the findings of this study are available upon request at https://github.com/asy51/OAI-ZIB-BMEL.