Diffusion model-empowered patella shape analysis predicts knee osteoarthritis outcomes

Abstract

Objective

We developed and validated an artificial intelligence pipeline that leverages diffusion models to enhance prognostic assessment of knee osteoarthritis (OA) by analyzing longitudinal changes in patella shape on lateral knee radiographs.

Method

In this retrospective study of 2,913 participants from the Multicenter Osteoarthritis Study, left-knee weight-bearing lateral radiographs obtained at baseline and 60 months were analyzed. Our pipeline commences with an automatic segmentation for patella shapes, followed by a diffusion model to predict patella shape trajectories over 60 months. We developed the Synthetic Patella Shape Incorporated Convolutional Neural Network (SynPatNet), a specialized 2-channel 1-dimensional convolutional neural network (CNN), to incorporate both baseline and synthetic follow-up patella shapes for predicting key outcomes of disease onset and end-stage.

Results

The diffusion model generates plausible synthetic patella shapes that predict deformations and osteophyte developments at the 60-month follow-up. Incorporating synthetic follow-up shapes with baseline patella shapes significantly improved OA outcome prediction: for patellofemoral OA onset, SynPatNet achieved an area under receiver operating characteristic curve (AUC) of 0.909 (vs. 0.830 for baseline model); for knee replacement, an AUC of 0.823 (vs. 0.773 for baseline). Augmenting Kellgren-Lawrence (KL) grade with SynPatNet further improved knee replacement prediction (AUC 0.838) over KL grade alone (AUC 0.785). Noteworthily, our knee replacement risk prediction score showed significant correlations with MRI-based (osteophytes/cartilage morphology/bone attrition) gradings, with Spearman's rho up to (0.51/0.33/0.31, p ​< ​0.001).

Conclusion

Generative diffusion modelling of patellar morphology on lateral knee radiographs provides complementary information to conventional radiographic and clinical metrics that substantially improves prognostication of knee OA.

1. Introduction

There is a pressing need to develop more effective tools for precise diagnosis and early prognosis of knee osteoarthritis (OA). Imaging techniques such as Magnetic Resonance Imaging (MRI) and X-ray have been adopted to predict knee replacement risk [[1], [2], [3], [4]]. Recent benchmarking efforts, such as the KNee OsteoArthritis Prediction (KNOAP2020) challenge, have aimed to objectively compare different MRI- and X-ray-analysis methods for predicting incident symptomatic radiographic knee OA [5]. Although MRI is known for providing excellent details for soft tissues in the knee joint [6,7], its high cost and limited accessibility make it less competent for routine clinical diagnostics [8] compared to X-ray in the primary care setting.

Despite continuous efforts to develop more advanced algorithms across various imaging modalities, exclusive focus on the tibiofemoral joint has yielded only incremental gains in predictive performance. Recent findings on the patellofemoral joint, however, have demonstrated promising clinical potential for further advancements in knee OA prognosis. This often overlooked compartment of the knee joint is also afflicted in knee OA progression [9,10]. Intriguingly, the process of knee joint degeneration often initiates from the patellofemoral joint [11]. Patellofemoral OA not only has a higher prevalence than tibiofemoral OA [12], but also possesses a stronger association with knee OA symptoms and functional impairments [13,14]. Evidence also links radiographic patellofemoral OA with increased knee replacement risk in 10 years [15]. The structural integrity of the patellofemoral joint could be one of the most significant precursors for knee OA prognosis.

In the absence of standardized assessment criteria for the patellofemoral joint, lateral radiographs offer a cost-effective and easily integrated option for early diagnosis and risk prediction, compared to MRI [8] and skyline view radiographs [16]. For example, a very recent study successfully detected patellofemoral OA from lateral view radiographs, achieving the area under receiver operating characteristic curve (AUC) of 0.958 [17]. Similarly, in our previous work, a radiomics score calculated from lateral radiographs of the patellofemoral joint was developed to accurately predict knee replacement risk at 60 months with an AUC of 0.87 when combined with the Kellgren-Lawrence (KL) grade, compared to an AUC of 0.83 when using only KL grade [18]. Lateral radiographs of the patellofemoral joint could provide additional pathological insights into knee OA development that are independent of the tibiofemoral joint.

Meanwhile, bone shapes of the knee joint compartments have been recognized as key biomechanical factors contributing to the deterioration of knee OA, as they could reflect articular cartilage loss and osteophyte development. A study utilizing MRI data from the Osteoarthritis Initiative dataset proposed B-score as a measure of femur bone shape, demonstrating excellent performance in knee replacement prediction [1]. At present, substantial efforts have been made to consolidate the understanding of the association between patella shapes and the development and progression of knee OA. For plain radiographs, the Wiberg classification of patella shape types from the skyline view has demonstrated associations with the severity of patellofemoral OA [19] and knee replacement outcomes [20]. For MRI, the technique of Statistical Shape Modelling (SSM) has been widely adopted to identify significant shape features that indicate the presence [21,22], progression [23], and symptomatic manifestations [22] of patellofemoral OA. Associations were also identified between 3D bone shapes, including the patella, and the future onset of radiographic OA [24] as well as total knee replacement [25]. Nevertheless, the prognostic utility of patella bone shape for knee OA, particularly on plain radiographs, remains largely underexplored and warrants further investigation.

Precise forecasting risk of knee replacement in the future from baseline radiographs directly still poses huge technical challenges to previous approaches. To tackle this challenge, a novel methodology was proposed to incorporate a deep generative model for longitudinal prediction of medical images as an intermediate step [26]. Their generative adversarial network (GAN) is reported to synthesize plausible radiographs that correctly predict radiological signs of OA, most notably joint space narrowing. By incorporating synthetic future radiographs as additional inputs into the predictive model, the performance for predicting knee OA deterioration on the Multicenter Osteoarthritis Study (MOST) dataset significantly improved, the AUC increased from 0.52 to 0.69. Adopting generative models trained for image prediction could open an avenue for the advancement in disease progression prediction.

We hereby deployed cutting-edge deep generative diffusion models to synthesize lateral view patella shapes in 60-month follow-up, thereby advancing the ability to model disease progression and enhance prognostic performance in predicting multiple knee OA outcomes. By incorporating synthetic future patella images, we aimed to detect subtle morphological changes critical for early knee OA diagnosis and risk assessment. This approach will clarify how patellar shape influences OA development and progression, opening new possibilities for early intervention and better patient outcomes.

2. Method

2.1. Study design

Fig. 1 illustrates the workflow of this study, which includes image segmentation, synthesis, and analysis. Firstly, patella shapes were extracted automatically from lateral knee radiographs (technical details in Sections 2.4-2.6). They were then used to train a diffusion model capable of learning the trajectory of patella shape changes between the initial and follow-up visits (technical details in Section 2.7). Lastly, the trained diffusion model generated synthetic follow-up patella shapes based on baseline data to assist the prediction on knee OA outcomes at 60 months (technical details in Section 2.8).

Fig. 1.

The workflow for the diffusion model aided patella shape analysis pipeline constructed for the study, including (a) a two-step patella segmentation pipeline, (b) a diffusion model for predicting patella shape trajectory, and (c) convolution neural network (CNN)-based models predicting the risk of patellofemoral OA onset, tibiofemoral OA onset, and receiving knee replacement (KR) within future 60 months.

Abbreviations: Att ​– ​Attention; 60m – ​60 months; 0m – ​0 month; Syn ​– ​Synthetic; 1D – ​One-Dimensional; CNN ​– ​Convolutional Neural Network; KR ​– ​Knee Replacement.

2.2. Data acquisition, inclusion and exclusion criteria

A cohort of 3,026 patients was retrospectively identified from the MOST dataset [27]. After applying the exclusion criteria, 2,913 patients remained and were randomly split (8:2) at the patient level into training (n ​= ​2,330) and testing (n ​= ​583) cohorts (Fig. 2). Left knee weight-bearing lateral radiographs were collected at baseline and 60 months, alongside KL grade, patellofemoral OA, knee replacement events, and baseline Whole-Organ Magnetic Resonance Imaging Score (WORMS) [28].

Fig. 2.

Cohort inclusion and exclusion criteria for the patella shape trajectory prediction (Fig. 1b) illustrated in green and that for the patella shape analysis (Fig. 1c) in blue. The training and testing cohorts were kept consistent throughout their development.

Abbreviations: 60m – ​60 months.

All 2,913 patients with baseline radiographs were included in the patella shape analysis (Fig. 1c); among these patients, 1,953 (1,565 in the training and 388 in the testing cohort) had both baseline and 60-month radiographs available, which were used for patella shape trajectory prediction (Fig. 1b), as shown in Fig. 2. Cohort divisions were applied consistently to prevent data leakage, ensuring the integrity and validity of the model performance evaluation.

The MOST study received approval from local institutional review boards, and written informed consent was collected from all participants at their initial visit.

2.3. Covariates and outcome measures

For baseline cohort characterization, demographic and clinical variables were collected for all participants, including age, sex, body mass index (BMI), baseline KL grade, and baseline patellofemoral OA status. To assess group comparability between the training and testing cohorts, two-tailed Student's t-test was applied for continuous variables, and the Chi-square test was used for categorical variables. Statistical significance was set at p ​< ​0.050.

The primary outcomes reflecting OA pathogenesis and deterioration were: 1. patellofemoral OA onset, 2. tibiofemoral OA onset, and 3. knee replacement within 60 months, all derived from clinical data. Patellofemoral OA onset was defined using MOST criteria as either an osteophyte score ≥2, or a joint space narrowing score ≥1 plus any osteophyte, sclerosis, or cyst score ≥1 on follow-up lateral radiographs within 60 months, among patients without patellofemoral OA at baseline. Tibiofemoral OA onset was defined as progression from baseline KL grade <2 to KL grade ≥2 by 60 months. Knee replacement was defined as any knee arthroplasty performed during the follow-up period.

2.4. Two-step patella segmentation pipeline

To provide reliable ground truth for developing an automated patella segmentation tool, professional radiologists manually annotated patella contours on 1,460 randomly selected left knee weight-bearing lateral radiographs from the cohort using the open-source software Computer Vision Annotation Tool (CVAT). Annotated knees were randomly separated (8:2) into 1,168 training and 292 testing samples.

The patella segmentation pipeline was devised into two steps – a coarse bone segmentation followed by a fine delineation of the patella contour (Fig. 1a). This ensures high-quality segmentation which reveals detailed bone structures for subsequent diagnostic and predictive model development. In the two-step patella segmentation pipeline, an attention U-Net architecture [29] with ResNeSt-101 backbone was first applied to perform a coarse semantic segmentation on the patella bone. The coarse patella masks then guide a cropping operation on full-sized radiographs to generate resized images centered at the patella region, which could finally be fed to an attention U-Net model with ResNet-18 backbone trained to generate fine segmentation masks of the patella. Flipping, rotation (±0°–30°), Gaussian noise, and brightness adjustment were randomly applied during the training of each model. To test the segmentation performance of the pipeline, Intersection over Union (IoU) and Dice coefficient (Dice score) were employed as evaluation metrics.

2.5. Segmentation mask generation

Inferencing all available lateral knee radiographs collected at the baseline visit and 60-month follow-up using the two-step patella segmentation pipeline (Fig. 1a), and excluding 31 output masks failing segmentation quality control, we yielded 2,946 baseline masks for the development of risk prediction models (Fig. 1c), while 1,953 pairs of baseline and corresponding 60-month masks were available for the development of the diffusion model (Fig. 1b). Output segmentation masks were then allocated according to the patient selection and cohort split described in Section 2.2 and illustrated in Fig. 2.

2.6. Image preprocessing

To mitigate the impact of leg positioning and radiographic magnification [30] on the morphology of the patella, all patella masks were rescaled to a unified size (by segmented pixels) to facilitate direct comparison focusing mask shapes. To standardize alignment, each mask was rotated to maximize the ratio of vertical-to-horizontal dimensions, ensuring the patella appeared as upright as possible, compensating for potential differences in knee flexion angle during image acquisition [30].

2.7. Patella shape trajectory prediction with diffusion model

The objective of a diffusion model is to learn the latent structure of a dataset by modelling the reconstruction of data points through repetitively adding and removing noise. Common diffusion models for generating data involve two stages: a forward diffusion stage and a reverse diffusion stage. In the forward stage, input data is gradually perturbed by adding Gaussian noise over a Markov chain of small diffusion steps. In the reverse stage, a deep learning model that learns to gradually reverse this addition of noise by predicting the noise at each step, with the goal of recovering the diffused data [31].

To predict the future trajectory of patellar morphological changes, we adopted the architectural design of MedSegDiff [32], a denoising diffusion probabilistic model (DDPM) originally designed for medical image segmentation. Our work introduced a novel adaptation of this architecture for the distinct task of image prediction. Specifically, our modified model concatenates encoded baseline patella data input as conditional embeddings into the generative model encoder, the U-Net convolutional neural network (CNN) decoder then performs the denoising process to reconstruct follow-up patella data specific to the inputted baseline from noise.

Our DDPM learns trajectories of the progressive transformation process between baseline patella shapes and their 60-month follow-up pairs. By reversing this learned process, the model can generate synthetic 60-month follow-up masks from new baseline data, effectively predicting morphological changes in the patella bone (Fig. 1b). The model is trained with a standard DDPM loss function [31]. The diffusion model is trained utilizing a Nvidia GeForce RTX 4090 GPU. Model performance is evaluated in pixel-wise accuracy: Mean Squared Error (MSE) & Peak Signal-to-Noise Ratio (PSNR), perceptual quality: Structural Similarity Index Measure (SSIM) & Perceptual Image Patch Similarity (LPIPS), and spatial alignment: IoU & Dice metrics. The detailed descriptions of these metrics are available in Supplementary Table 1.

2.8. Convolutional neural network for patella shape analysis

We specifically developed the Synthetic Patella Shape Incorporated CNN (SynPatNet) (Fig. 3), a CNN-based model designed to predict the three primary outcomes of knee osteoarthritis progression within the future 60-month period.

Fig. 3.

Architecture of SynPatNet - A 1D-CNN with circular padding designed for patella shape analysis. The model integrates baseline patella shape and synthetic follow-up shape as dual inputs to enhance prognostic performance for knee osteoarthritis outcomes.

Abbreviations: 1D – ​One-Dimensional; ReLU – ​Rectified Linear Unit.

To predict risk scores of OA outcomes, SynPatNet uniquely incorporates each patient's baseline lateral patella shape and their synthetic 60-month projection generated by our diffusion model as inputs (Fig. 1c). Real 60-month follow-up images were not required. For each patella mask (baseline and synthetic follow-up), an algorithm sampled 320 equidistant two-dimensional coordinates along the contour. These two sets of coordinates were then each stacked into a 2×320 (channel × length) matrix, forming complementary inputs for the model. The Circular 1D Convolutional (CirConv1d) module enhances shape representation by employing circular padding to maintain contour continuity while offering computational efficiency and shape-specific feature extraction through 1D convolutions. To capture region-specific relevances of the patella [18,33], we integrated a Self-Attention module [34]. A fully connected layer combines information from both baseline and synthetic follow-up patella shapes for classification. Binary cross-entropy was employed as the loss function for training. Model hyperparameters, including batch size, loss function, optimizer, and learning rate, were optimized with a grid search.

We evaluated our methodology by comparing three distinct models: the proposed SynPatNet (utilizing baseline and synthetic 60-month follow-up shapes), a baseline CNN model (modified SynPatNet using only baseline shape), and a traditional SSM-based logistic regression approach (see Supplementary Fig. 1 for details). To further evaluate the clinical utility of our proposed approach, for predicting knee replacement within 60 months, we extended the comparison to include an enhanced SynPatNet incorporating KL grade and conventional clinical logistic regression models based on KL grade and patellofemoral OA status. Model performance was assessed using AUC scores, with 95% confidence intervals obtained from 5,000-iteration bootstrapping, confusion matrices, and DeLong tests [35] for statistical significance of the differences between AUCs (p ​< ​0.050). Due to class imbalance in the dataset, area under the precision–recall curve (PR-AUC) was also reported to provide additional parameter for model performance [36].

2.9. Statistical analysis

We also conducted Spearman correlation analyses between SynPatNet's predictions on knee replacement risk and WORMS scores from baseline MRI examinations to assess our model's biological relevance and understand its focus. MRI is widely regarded as the gold standard for joint assessment in osteoarthritis research due to its superior ability to detect early signs of disease progression, and to visualize a broad spectrum of pathological changes, including cartilage morphology, osteophytes, bone attrition, bone marrow lesions, and subarticular cysts, that are not fully captured by conventional radiography [6,7]. By leveraging these aforementioned MRI-based OA markers, we aimed to explore how patella radiographic shape features might hint deeper tissue alterations and complex disease manifestations that are otherwise detectable only through advanced imaging. This approach potentially offers insights into the progression pathways leading to end-stage knee osteoarthritis from the patellofemoral perspective.

Because the objective of this analysis was to investigate biological associations, not model performance, we included risk scores from the combined training and testing cohorts to maximize statistical power.

3. Results

3.1. Patient characteristics

Table 1 presents the distributions of patient demographics and baseline clinical status of the knees in our study cohorts. Statistical analyses revealed no significant differences between the training and testing cohorts in the distributions of key variables. A detailed breakdown of the extracted outcome variables across the training and testing cohorts is available in Supplementary Table 2.

Table 1.

Summary statistics for demographic and clinical variables of training and testing cohort patients.

Parameter	Training	Testing	p-value
Patient No.	2,330	583
Age			0.232
Mean ​± ​SD	62.43 ​± ​8.10	61.99 ​± ​8.01
Range	(50, 79)	(50, 79)
Sex			0.091
Male	947	369
Female	1,383	214
BMI			0.750
Mean ​± ​SD	30.66 ​± ​5.86	30.74 ​± ​6.15
Range	(17.86, 71.91)	(16.72, 57.83)
Baseline KLG			0.206
0	1,021	265
1	408	93
2	358	75
3	364	89
4	172	60
Missing	7	1
Baseline PFOA			0.869
No	1,831	466
Yes	351	83
Missing	148	34

Note: p-values were calculated using Student's t-test for continuous variables, and Chi-square test for categorical variables. Abbreviations: SD – standard deviation; BMI – body mass index; KLG – Kellgren-Lawrence grade; PFOA – patellofemoral osteoarthritis.

3.2. Automated, accurate patella segmentation with deep learning

The coarse segmentation model can locate the correct patella area from full-sized lateral knee radiographs with a success rate of 0.986 (288/292) among testing samples. For the fine segmentation of the patella shape, the U-Net with self-attention modules and a ResNet-18 backbone was found to have the best performance among tested architectures, achieving a Dice score of 0.973 and an IoU of 0.947 (Supplementary Table 3). It demonstrated the capability of capturing accurate shape of the patella, except some unique structures or poorly imaged parts might not be fully captured (Supplementary Fig. 2).

3.3. Diffusion model predicts longitudinal patella shape changes

Our diffusion model generated plausible synthetic 60-month follow-up patella shapes using only baseline patella shapes from the test set as input. As we pointed out in Fig. 4, the model also successfully predicts several morphological changes consistent with real 60-month follow-up masks from the unseen test cohort. These predictions include accurate representations of patella deformations and bone spur development. Although the development of enthesophytes was also captured, they might not significantly impact the development of symptoms or structural changes in the knee [37]. The accuracy of the model's predictions varies, with some cases showing close alignment in both the location and extent of morphological changes, while others exhibit discrepancies in the magnitude or precise location of deformations. Conversely, patellae initially exhibiting minimal signs of morphological change maintain relatively stable shapes in the model's output, aligning with expected clinical observations.

Fig. 4.

Six exemplary sets of patella shapes are shown, with each set consisting of the real shape at baseline (top row), the real shape at 60-month follow-up (middle row), and the diffusion model synthesized 60-month follow-up output (bottom row). Red arrows indicate successfully predicted patellar morphological changes, while green boxes indicate the prediction correctly demonstrated no significant changes.

Abbreviations: 60m – ​60 months.

As for quantitative evaluations, the diffusion model's synthetic output on the test set achieved the following metrics: MSE of 0.028, SSIM of 0.915, PSNR of 16.21 ​dB, LPIPS of 0.079, Dice score of 0.927, and IoU of 0.867. These results significantly outperform a standard U-Net model (Supplementary Table 1) trained as a baseline comparison across all metrics, demonstrating the superiority of our adapted diffusion model approach for predicting long-term patella shape changes.

3.4. Shape analysis improves prediction of patellofemoral OA and knee replacement

Table 2 presents a comprehensive comparison of the performance of various risk prediction models for key knee OA outcomes, capturing both compartment-specific disease pathogenesis and overall progression towards end-stage [9,10]. Specifically, our analysis evaluated the efficacy of different modelling approaches in predicting patellofemoral OA onset, tibiofemoral OA onset, and knee replacement surgery within a 60-month timeframe. Our shape analysis approach could predict patellofemoral OA onset and knee replacement, except for tibiofemoral OA onset. Leveraging both deep learning models and synthetic data led to improved prediction across these two clinically relevant tasks.

Table 2.

Performance comparison of risk prediction models for knee osteoarthritis outcomes.

Outcome and Model	AUC (95% CI)	p-valuea	PR-AUC (95% CI)
PFOA onset within 60 months			Baseline: 0.090
Baseline shape only (SSM)	0.732 (0.651, 0.803)	0.009	0.212 (0.136, 0.342)
Baseline shape only (CNN)	0.830 (0.750, 0.889)	ref	0.400 (0.246, 0.607)
Baseline ​+ ​synthetic shapes (SynPatNet)	0.909 (0.853, 0.945)	0.012	0.567 (0.388, 0.750)
TFOA onset within 60 months			Baseline: 0.183
Baseline shape only (SSM)	0.545 (0.468, 0.624)	0.054	0.232 (0.161, 0.336)
Baseline shape only (CNN)	0.589 (0.511, 0.664)	ref	0.232 (0.173, 0.320)
Baseline ​+ ​synthetic shapes (SynPatNet)	0.597 (0.519, 0.670)	0.754	0.237 (0.176, 0.331)
KR within 60 months			Baseline: 0.086
Baseline shape only (SSM)	0.679 (0.526, 0.795)	< 0.001	0.188 (0.116, 0.327)
Baseline shape only (CNN)	0.773 (0.657, 0.847)	ref	0.365 (0.172, 0.817)
Baseline ​+ ​synthetic shapes (SynPatNet)	0.823 (0.728, 0.896)	0.046	0.517 (0.256, 0.858)
Clinical variables			Baseline: 0.086
KLG	0.785 (0.663, 0.858)	ref	0.287 (0.149, 0.508)
PFOA	0.605 (0.532, 0.693)	< 0.001	0.126 (0.0770, 0.200)
KLG ​+ ​PFOA	0.787 (0.664, 0.877)	0.861	0.397 (0.200, 0.689)
KLG & baseline ​+ ​Synthetic shapes	0.838 (0.744, 0.908)	0.002	0.503 (0.261, 0.847)

ref Indicates the model used as the comparison reference with other models within each section.

Note: Baseline of PR-AUC corresponds to the proportion of positive samples in the dataset and represents the expected performance of a random classifier. Abbreviations: AUC – area under receiver operating characteristic curve; PR-AUC – precision-recall area under the curve; CI – confidence interval; CNN – convolutional neural network; SSM – statistical shape model; KLG – Kellgren-Lawrence grade; TFOA – tibiofemoral osteoarthritis; PFOA – patellofemoral osteoarthritis; KR – knee replacement surgery.

^ap-values were calculated using the DeLong test for AUC comparisons. p-values less than 0.050 (bolded) indicate statistical significance.

Specifically, for patellofemoral OA onset prediction (Fig. 5a), we observed clear performance improvements from SSM (AUC 0.732) to baseline CNN (AUC 0.830, p ​= ​0.009) to SynPatNet (AUC 0.909, p ​= ​0.012), with each improvement being statistically significant. In contrast, tibiofemoral OA onset prediction showed only modest gains (Fig. 5b), with all models performing marginally above random chance (SSM: AUC 0.545, CNN: AUC 0.589, SynPatNet: AUC 0.597) without statistically significant differences. Knee replacement surgery prediction showed intermediate success (Fig. 5c), with significant improvements from SSM (AUC 0.679) to baseline CNN (AUC 0.773, p ​< ​0.001), and further enhancement with SynPatNet (AUC 0.823, p ​= ​0.046), demonstrating the value of incorporating synthetic shape information for this outcome.

Fig. 5.

Comparison between receiver operating characteristic curves of the SSM model (baseline shape only), CNN model (baseline shape only) and SynPatNet (baseline ​+ ​synthetic shapes) in predicting (a) the onset of patellofemoral OA (PFOA), (b) the onset of tibiofemoral OA (TFOA), and (c) the risk of knee replacement surgery (KR) within 60 months. The confusion matrices of SynPatNet corresponding to predicting (a) (b) (c) are plotted in (d) (e) (f) respectively, those of SSM and CNN models are available in Supplemental Fig. 3.

Abbreviations: SSM – statistical shape modeling; AUC – area under the receiver operating characteristic curve; CNN – convolutional neural network; NEG – negative; POS – positive; PFOA – patellofemoral osteoarthritis; TFOA – tibiofemoral osteoarthritis; KR – knee replacement.

In evaluating knee replacement surgery prediction using clinical variables (Supplementary Fig. 4), baseline KL grade alone performed well (AUC 0.785), while baseline patellofemoral OA alone showed deficient performance (AUC 0.605). Combining patellofemoral OA with KL grade offered minimal improvement (AUC 0.787, p ​= ​0.861), but the integration of SynPatNet's patella shape analysis with KL grade achieved the best performance (AUC 0.838, p ​= ​0.002), demonstrating that shape-based features provide complementary prognostic information beyond traditional clinical measures.

The trends in PR-AUC across different models and inputs were largely consistent with those observed for AUC, further validating our AUC-based observations in the context of a highly imbalanced dataset.

3.5. Association between MRI-based anatomic features and knee replacement risk score

The SynPatNet model was evaluated by correlating its patella shape–based knee replacement risk predictions with baseline WORMS measures in patellofemoral subregions (Fig. 6). Across subregions, the model demonstrated significant positive correlations with cartilage damage, osteophyte formation, and bone attrition, especially in the lateral facet of the patella (PL) and lateral anterior femur (FLA). Osteophyte scores showed the strongest overall correlations, particularly in the femur medial anterior (FMA: rho ​= ​0.51, p ​< ​0.001) and patella inferior (PI: rho ​= ​0.51, p ​< ​0.001), while bone marrow lesions presented more variable patterns, with only moderate positive associations in the lateral patellofemoral areas (PL: rho ​= ​0.14, p ​< ​0.001; FLA: rho ​= ​0.20, p ​< ​0.001). Subarticular cysts exhibited the weakest correlations to the knee replacement risk predictions, highlighting the model's heightened sensitivity to degenerative changes such as osteophytes and cartilage alterations in lateral subregions.

Fig. 6.

Association analysis of the knee replacement risk scores (predicted by SynPatNet with baseline and synthetic patella shapes) and WORMS scores at various subregions of the patellofemoral joint. Spearman's correlation coefficients were used to quantify the strength of these relationships. The results are visualized as heatmaps, where colour intensity indicates correlation strength and bubble size reflects statistical significance. Significance levels are denoted by ∗p ​< ​0.05, ∗∗p ​< ​0.01, and ∗∗∗p ​< ​0.001. Abbreviations: WORMS – Whole-Organ Magnetic Resonance Imaging Score; PM – patella medial; PL – patella lateral; PS – patella superior; PI – patella inferior; FMA – femur medial anterior; FLA – femur lateral anterior.

4. Discussion

This study presents and validates a novel deep learning–based pipeline for the longitudinal analysis of patella shapes from lateral knee radiographs, aiming to refine the prognostication of knee OA. By leveraging diffusion model–derived synthetic follow-up images and integrating them into a specialized CNN, our approach demonstrated significant improvements in predicting key OA outcomes, including the onset of patellofemoral OA and the risk of knee replacement.

Lateral patella shape holds valuable prognostic information beyond what is traditionally assessed through tibiofemoral joint analysis. In particular, the patellofemoral joint has been shown to exhibit osteophyte formation, cartilage attrition, and bony outgrowths that precede or coincide with knee OA [12]—a process that our shape analysis system captured quantitatively. While tibiofemoral joint radiographic assessments remain standard, the high accuracy we achieved in predicting patellofemoral OA onset strongly showcases the value of considering the lesser-studied patellofemoral compartment for early warning signs of OA disease progression.

The enhanced performance achieved by incorporating synthetic follow-up shapes underscores the importance of modelling longitudinal data over relying solely on baseline, static images. By synthesizing a future representation of the patella, the model effectively reduces complex disease progression patterns to more tractable features, essentially giving the network a “peek” at potential future outcomes. This approach parallels other emerging trends in medical imaging, where temporal modelling is increasingly recognized as essential to capturing the dynamic progression of degenerative diseases [38,39]. Our findings indicate that generative AI can amplify pathological representations through projecting disease trajectories, enabling the system to detect subtle morphological and biomechanical changes that may herald more advanced disease.

Interestingly, although lateral patella shapes yielded strong predictive performance regarding early patellofemoral changes and the clinical endpoint of knee OA, it was less informative for predicting tibiofemoral OA onset. This finding suggests that patellofemoral OA-related morphological changes do not necessarily correlate with tibiofemoral changes or that the pathophysiologic pathways in these compartments may differ. This compartmental distinction is in line with current understanding, as patellofemoral OA has long been recognized as a separate entity that can independently drive both symptoms and structural progression [13]. These results align with observations that patellofemoral signs can appear early and often have unique or more localized biomechanical drivers. Nonetheless, patella shape did show a robust association with knee replacement, a procedure predominantly guided by overall knee symptom burden, radiographic severity, and functional limitations—most often attributed to tibiofemoral degeneration [40,41]. Consequently, our findings may reflect a summation of global knee health, wherein subtle patellofemoral changes identified on radiographs could signal the onset of advanced disease in the entire joint [15].

Correlation analysis showed that SynPatNet's radiograph-based knee replacement risk predictions were significantly associated with several MRI-based WORMS features, most notably osteophyte scores, particularly in the PI and FMA regions. The strongest association was consistent with prior studies highlighting osteophyte formation as an early indicator of osteoarthritis [42]. Cartilage morphology and bone attrition scores also demonstrated significant, albeit weaker, associations—especially in PL and FLA subregions, aligning with the known lateral predominance of patellofemoral OA [14]. These results suggest that radiographic patella shape changes, including localized bone attrition [43] and osteophyte formation [42], may reflect broader osteoarthritic processes within clinically relevant compartments. These statistical associations support the prognostic value of radiographic shape analysis, yet do not establish causation; future studies are required to clarify the underlying biological mechanisms.

Despite these promising findings, there are a few limitations in the present study. First, our diffusion model trained on follow-up pairs likely underrepresents rapid-progression cases because it excludes patients undergoing knee replacement before 60 months, biasing estimates toward slower disease trajectories. Future work should develop multi-timepoint training using radiographs from all follow-ups to better capture the non-linear, episodic nature of OA progression [44,45]. Second, using knee replacement as the sole end-stage outcome confounds clinical need with socioeconomic and patient-preference factors [46,47], possibly diluting associations with true osteoarthritic severity. Subsequent studies should adopt validated composite endpoints [48] that integrate pain, function, and radiographic progression to provide a more holistic measure of advanced disease. Third, confining development to the U.S.-based MOST cohort [27] means our findings may not translate to populations that differ in genetic, lifestyle, or cultural factors influencing knee OA [49]. Moreover, MOST's strictly standardized imaging protocol and our own preprocessing steps produced a level of image uniformity seldom seen in routine practice, masking the typical variability in positioning, technique, and equipment that could distort patellar shape representation on lateral radiographs [30]. Rigorous external validation, coupled with transfer learning or domain adaptation strategies using diverse independent datasets, is therefore essential to ensure the robustness and generalizability of our approach.

In summary, this study highlights that lateral radiographs, a widely available and routinely used imaging modality, offer underutilized prognostic information for knee osteoarthritis. By enabling interpretable, synthetic projections of patella shape changes, our approach supports more systematic evaluation of patellofemoral OA and improved risk stratification for patellofemoral OA onset and knee replacement, challenging current clinical routines and offering new opportunities for earlier intervention. The improvements demonstrated in risk stratification underscore the potential of these methods to reveal predictive signals not captured by traditional approaches. Future work should focus on adopting multi-timepoint modelling and validating these findings in larger, more diverse cohorts to fully realize the potential of AI-driven shape analysis for personalized OA management.

Author contributions

Drafting of the article was carried out by Sing-Hin Lau. The conception and design of the study were led by Sing-Hin Lau, Lok-Chun Chan, Tianshu Jiang, and Chunyi Wen. Sing-Hin Lau, Lok-Chun Chan, Jiang Zhang, and Xiangqiao Meng performed the analysis and interpretation of the data. Chunyi Wen obtained the funding for this project. Administrative, technical, or logistic support were provided by Ping-Keung Chan, Jing Cai, Ping Li, and Chunyi Wen. Data collection and assembly were undertaken by Lok-Chun Chan and Wei Wang. All authors contributed to the critical revision of the article and gave final approval of the version to be submitted.

Declaration of Generative AI and AI-assisted technologies in the writing process

During the preparation of this work, the authors used GPT-4o by OpenAI and Claude 3.5 Sonnet by Anthropic to improve grammar, sentence structure, and enhance the readability of the manuscript. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

Role of the funding source

This study is supported by the RISA seed fund (P0043002, P0051049, P0050709) and RIAM seed fund (P0050824), Mainland/GBA Research Funding Scheme (P0049195), Innovation & Technology Fund for Better Living (FBL/B046/22/S).

Declaration of competing interest

The authors have no conflicts of interest to declare.

Handling Editor: Professor H Madry

Appendix A. Supplementary data

The following is the Supplementary data to this article: