Artificial Intelligence in the Diagnosis and Prognostication of the Musculoskeletal Patient

Abstract

As artificial intelligence (AI) advances in healthcare, encompassing robust applications for the diagnosis and prognostication of musculoskeletal diseases, clinicians must increasingly understand the implications of machine learning and deep learning in their practice. This review article explores computer vision algorithms and patient-specific, multimodal prediction models; provides a simple framework to guide discussion on the limitations of AI model development; and introduces the field of generative AI.

Introduction

Vast amounts of data that can be accessed for clinical research were generated by the migration from physical patient charts to electronic health records, but much of that data, such as free-text clinical notes and medical imaging, are “unstructured,” lacking a predefined format. Most traditional statistical methods require unstructured data to be organized into a “structured” format, such as a tabular database (ie, traditional rows and columns) [28]. As a result, medical research faces 2 significant challenges: first, the rapid growth of large datasets, which cannot be efficiently processed by humans due to time and resource constraints, and second, the challenge of working with data that lack a clear structure [68]. To overcome these barriers and inform patient care, clinicians and researchers are increasingly using artificial intelligence (AI), which can uncover hidden patterns and generate new insights from large, complex, and unstructured data sources [21,23,29,42,57,62].

The purpose of this review is to present current AI applications for the diagnosis and prognostication of musculoskeletal (MSK) diseases. Furthermore, it will describe challenges and limitations of current AI models and provide a glimpse into innovations such as generative AI. But first, a brief conceptual primer on AI, machine learning (ML), and deep learning (DL).

Artificial Intelligence and Traditional Statistics

AI consists of mathematical algorithms that give computers the ability to perform tasks that have historically required human intelligence, such as problem-solving, learning, pattern recognition, and understanding language [20,42,46]. A subfield of AI, ML focuses on developing algorithms that have the capacity for iterative learning, the ability to continuously improve performance on a task by processing data and adjusting internal model parameters (sometimes known as weights) in reference to ground truth labels (supervised learning) or internal heuristics (unsupervised learning) [15]. That these algorithms can “self-teach” explains in part why they are being employed to answer research questions pertaining to MSK diseases and to test hypotheses regarding complex, multidimensional datasets.

However, it is valid to ask: “If AI is equipped to analyze large datasets and discover associations between multiple variables in an automated fashion, will it replace traditional statistical approaches to hypothesis testing?” The answer lies in understanding the different approaches employed by ML and statistics to answer a research question. For example, in predicting the likelihood of failure in 1-stage versus 2-stage exchange for treating a periprosthetic joint infection, traditional statistics and ML differ methodologically.

Traditional statistical methods focus on identifying relationships between variables by applying a predefined set of rules. Statistical analyses might reveal which patient characteristics—such as age, comorbidities, or lab results—are associated with the risk of failure for each procedure. These methods rely on assumptions and statistical tests, which makes them useful for drawing inferences from data, but limit their ability to adapt and discover new emerging patterns.

In comparison, an ML model is trained on a large dataset, progressively refining its predictions through repeated loops through the data, generating new “rules” to guide by itself (Fig. 1). The model can learn simple, linear relationships but its real value is in detecting complex, nonlinear relationships in the data that may not be immediately apparent through traditional statistical methods. Over time, as more patient data become available, the ML model can enhance performance through additional training.

Fig. 1.

Conceptual difference between machine learning and traditional statistics.

To summarize, traditional statistics are great for understanding the association between a variable and an outcome, while ML is better suited for making precise, data-driven predictions on unseen data.

DL and Artificial Neural Networks

DL is a subfield of ML that consists of models sharing an architecture known as artificial neural networks (ANNs). These ANNs consist of layers of nodes (aka neurons): an input layer, one or more hidden layers, and an output layer. Layers are interconnected via weights (Fig. 2). During training, the model produces predictions on data pulled from the training set. These predictions are compared against a ground truth value by a metric (eg, mean squared error, mean absolute error) that compares how far the predicted output is from the true value. Based on the calculated metric, a cost/loss function updates the weights of the model, with the end-goal of closing the gap between the predicted and true values (ie, optimizing the metric and minimizing the cost function). An optimizer, which is an additional algorithm, is tasked with overseeing how the ANN updates its weights and guides the loss function toward its minimum value [8]. The Journal of Arthroplasty published a 4-part series on AI applications in orthopedics, including metrics and cost functions, and is a recommended resource for additional information regarding AI [28,44,46,70].

Fig. 2.

Example of input (blue), hidden (yellow), and output (red) layers in an artificial neural network. The metric quantifies the difference between the model’s prediction and the ground truth and informs the loss function. The loss function then controls the process of backpropagation. Backpropagation results in an update of the model’s weights, which leads to more accurate predictions. The cycle is repeated until the metric is optimized (ie, when the difference between predictions and ground truth labels is closest to zero) and the loss function is at its minimal value.

AI in the Diagnosis of the MSK Patient

Neural networks have proven to be particularly useful for accomplishing computer vision tasks, such as face recognition, autonomous navigation, image-to-image translation, and medical imaging analysis [5,6,8,14,34,69]. As many MSK diagnoses rely primarily on radiographic imaging to inform classification of disease and treatment decision-making, our discussion will focus on the following computer vision applications: image classification, object detection, and image segmentation (Table 1).

Table 1.

Summary of AI computer vision applications for the diagnosis of MSK diseases.

Group (year)	Inputs	Outputs	Performance
Image classification
Tiulpin et al (2018) a [67]	AP knee radiographs	KL classification	Externally validated on 3000 patients from the OAI dataset and achieved a multi-class accuracy of 66.7% and AUROC of 0.93 for diagnosis of osteoarthritis (KL ≥2). Saliency maps allow users to visualize which pixels guided the model’s prediction
Shah et al (2020) [56]	Preoperative AP and lateral hip and knee radiographs	Prosthesis loosening at the time of surgery	Accuracy of 70%. When adding clinical data, the accuracy of the model increased to 88.3%
Folle et al (2022) [12]	Hand MRI data including T1 and T2 coronal, T1 coronal and axial fat-suppressed contrast-enhanced, and T2 fat-suppressed axial sequences	Classifying MRIs into seropositive RA, seronegative RA, or psoriatic arthritis	AUROC was 75% for seropositive RA vs PsA, 74% for seronegative RA versus PsA, and 67% for seropositive versus seronegative RA. Psoriatic patients with subclinical inflammation were mostly given the PsA label, hinting at a possible PsA-like MRI pattern during the subclinical phase of the disease
Cheng et al (2021) [7]	Pelvic radiographs	Presence of trauma-related injuries on pelvic radiographs, including hip fractures and dislocations, pelvic fractures, periprosthetic fractures, and femoral shaft fractures	AUROC of 0.972 and sensitivity of 0.908 on the internal test set when classifying between normal and abnormal radiographs. Model performed similarly to orthopedic surgeons and radiologists in terms of accuracy (97% vs 95.7%, P > .05)
Khosravi et al (2024) [24]	AP pelvic radiographs	Presence or absence of ischial spine signs, dysplasia, cam deformity, other morphological abnormalities, and patient sex	AUROC values were 0.89 for ischial spine sign, 0.80 for both cam deformity and dysplasia, and 0.81 for all abnormalities, demonstrating the model’s ability to accurately label radiographs in a binary fashion for these radiographic characteristics. AUROC for patient sex was 1.00. Grad-CAM visualizations were employed to allow users to visualize where the model focuses when making predictions
Object detection
Li et al (2021) a [35]	Lateral spine radiographs	Presence and location of vertebral fractures plus Genant classification	The model achieved 89% on the external test set, and interobserver reliability between AI and human observers was κ = 0.77 for lumbar vertebral fractures on the internal test set
Zech et al (2023) a [74]	Upper extremity radiographs (finger/hand, wrist/forearm, elbow, humerus, shoulder/clavicle) of pediatric patients	Presence and location of pediatric upper extremity fractures	AI accuracy was higher than on-call residents for the internal test set (89.4% vs 85.1%, P = .01)
Von Schacky et al (2021) a [71]	Patient radiographs	Benign or malignant bone tumor	This model simultaneously segmented, placed bounding boxes, and classified primary bone tumors. The model was nearly as accurate at classifying the tumor as benign or malignant as an MSK fellowship-trained radiologist (80.2% vs 83.4%, P > .05)
Shinohara et al (2024) [60]	AP pelvis and hip radiographs	sMRI generated from the pelvic radiographs, detection of osteonecrosis of the femoral head	Mean average precision of 0.951 for object detection of lesion and AUROC of 0.99. Interrater reliability on osteonecrosis grading of sMRI was 0.98
Segmentation
Rouzrokh et al (2021) [53]	Post-THA AP pelvis and cross-table lateral hip radiographs	Measurement of postoperative acetabular inclination and anteversion angles	The mean difference between human and machine measurements was 1.350 and 1.390 for inclination and anteversion, respectively
Faghani et al (2023) [9]	Whole-body low-dose CT	Detection of lytic lesions of multiple myeloma	The model achieved a sensitivity of 91.6% and a specificity of 84.6%. Segmentation model facilitated the object detection model’s task
Vukicevic et al (2021) [72]	Salivary gland ultrasonography scans	Segmentation masks of pSS-affected salivary glands	Interobserver reliability of DL model outshined human annotators (IoU0.85 vs 0.76) and performed at a rate of 24.5 frames/s. The model could eventually aid in the automated diagnosis of pSS
Peeken et al (2024) [48]	T1-contrast-enhanced MRI sequences	Segmentation masks of soft tissue sarcomas	Dice similarity coefficient of 0.88 against ground truth

AI artificial intelligence, AP anterior–posterior, AUROC area under the receiver operating characteristic curve, DL deep learning, KL Kellgren–Lawrence, CT computed tomography, MSK musculoskeletal, OAI osteoarthritis initiative, PsA psoriatic arthritis, pSS primary Sjogren syndrome, RA rheumatoid arthritis, sMRI synthetic magnetic resonance imaging, THA total hip arthroplasty, Grad-CAM Gradient-weighted Class Activation Mapping, IoU Intersection Over Union.

^aExternally validated.

Image classification models predict the correct label for a given image and are often used in screening and diagnostic contexts [22]. Classification models can be trained to label a medical image in a binary (benign vs malignant tumor), multi-class (Kellgren–Lawrence grade in osteoarthritis), or multi-label (presence of cam deformity, dysplasia, and ischial spine sign) manner [13,14,24,67]. Convolutional neural networks (CNNs) are the most common DL classifiers. Popular CNN architectures include ResNet, DenseNet, GoogLeNet, and EfficientNet [63].

Classification models can also be built to assist in data processing, registry creation, and extracting information from unstructured data [39,51]. These models can be used to create imaging datasets for future research by sampling large imaging registries for radiographic studies that match inclusion criteria based on image content [41,52]. A common drawback of DL classification models is the “black box”phenomenon. Model transparency is limited because DL models perform their feature selection by encoding the input image into an abstract representation. As a result, it is difficult to both control how the model learns from the data and understand what it learned. Some features may be inappropriately associated with a label; for example, a DL model may learn to predict the presence of a hemothoraxon a chest radiograph based on the presence of a chest tube rather than on radiographic characteristics of the disease (eg, blunting of the costophrenic angle).

Object detection models are designed to localize an object of interest within a given image. This is commonly accomplished by creating a boundary, known as a bounding box, around the object. Common DL architectures for object detection include You Only Look Once, Region-based Convolutional Neural Networks (R-CNN), and Single Shot MultiBox Detector; each has a unique speed and accuracy profile and is chosen by the developer based on the task at hand [4].

Object detection models are used in orthopedics research for fracture identification [16,35,74]. Zech et al trained a model using R-CNN to detect fractures on pediatric upper extremity radiographs; it demonstrated higher accuracy than residents on preliminarily reviewed radiographs of on-call cases [74]. Similar models have been developed for the detection of other bone lesions, such as tumors and osteonecrosis of the femoral head [60,71].

Image segmentation models generate segmentation masks by assigning a label to each pixel in an image to delineate a region or, in the case of medical images, an anatomic structure. For example, a segmentation model could learn to map the pixels associated with the patella on a knee radiograph. The resulting segmentation mask can subsequently be used to perform various computations, such as measuring area/volume or, if multiple objects are segmented, computing distances and angles between structures. Popular architectures include encoder–decoders (U-Net, SegNet, 3D U-Net) and transformer-based (Swin-Unet) [38].

Researchers use masks as a foundation for other DL algorithms. For example, Vukicevic et al trained a model to segment salivary glands on ultrasound scans to automate the diagnosis of Sjogren’s syndrome [72]. This model has the potential to help overcome limitations associated with ultrasound screening, such as poor inter- and intra-observer reliability and overdependence on technical skill. Other groups have created segmentation models to detect lytic lesions in multiple myeloma and soft tissue sarcomas [9,48]. Segmentation masks can also be useful in automating quantitative measures of radiographic features used to diagnose the severity of disease, such as minimum joint space width in knee osteoarthritis or Cobb angle in scoliosis [40,64].

AI in the Prognosis of the MSK Patient

Surgical planning and decision-making require physicians to consider the benefits and risks of each treatment option, which may be influenced by variables such as patient demographics, comorbidities, prior diagnoses, imaging studies, lab work, and past medical history. These factors and their associated uncertainties make accurate and quantitative prognostication challenging. While statistical models can be used to identify associations and measure clinical significance, they may not capture the complexity of the relationships between variables. Through iterative learning, ML models can be developed to predict patient-specific clinical outcomes and have the potential to guide a new era of precision medicine in the field of MSK health [31]. The following sections will review studies using AI to model disease progression and to predict both postoperative complications and response to treatment (Table 2).

Table 2.

Summary of AI studies relating to the prognostication of musculoskeletal diseases.

Group (year)	Procedure and complication	Inputs	Outputs	Notes
Postoperative complication prediction models
Khosravi et al (2022) [26]	Dislocation post-primary THA	Preoperative AP pelvis radiographs and clinical data (demographics, comorbidities, surgical characteristics)	Patient-specific risk of dislocation within 5 years of THA	Model returns patient-specific risk for 18 combinations of modifiable surgical variables (femoral head size, surgical approach, and type of acetabular liner)
Labott et al (2023) [33]	Hospitalization post-ambulatory UKA procedures	Demographics, social determinants of health, anesthesia type, comorbidities, and concurrent procedures	Risk of unplanned overnight admission (eg, length of stay ≥1 day)	Serves as an aid to identify the most appropriate candidates for outpatient UKA procedures
Kunze et al (2022) [32]	Hyponatremia following TJA	19 variables, including demographics, intraoperative parameters, and clinical history	Patient-specific probability of hyponatremia post-TJA	The most important factors for predicting hyponatremia in this study were preoperative serum sodium concentration, age, intraoperative blood loss, procedure time, BMI, and ASA score
Machine Learning Consortium (2021) [36]	Infection after ORIF of tibial shaft fractures	13 variables including demographics, mechanism of injury, comorbidities, fracture classification, and location	Probability of infection following operative treatment of tibial shaft fracture	Gustilo–Anderson types IIIA and IIIB, age, AO/OTA type 42C3, a crush injury, and a fall were the strongest predictors of infection
Modeling the progression of disease
Group (year)	Disease	Inputs	Outputs	Notes
Schiratti et al (2021) [55]	Osteoarthritis	Knee MRIs and clinical variables	Risk of losing at least 0.5 mm of knee cartilage in 12 months	The DL model achieved an AUROC score of 65%, outperforming the 58.7% by 2 board-certified radiologists
Yahara et al (2022) [73]	Adolescent idiopathic scoliosis	Total spine radiographs (frontal views)	Risk of curve progression	Prediction performance of the model for AIS curve progression was better than spine surgeons (69% vs 47%)
Norgeot et al (2019) a [43]	Rheumatoid arthritis	Medications, demographics, laboratory values, and prior measures of disease activity	RA disease activity at next clinical visit (CDAI scores)	AUROC of 0.91 and 0.74 on internal and external test sets, respectively. This model has the potential to inform personalized treatment strategies based on the predicted progression of disease
Thio et al (2019) [66]	Long-bone metastatic disease	Demographics, comorbidities, laboratory values, primary tumor type, presence of pathologic fracture, metastasis location, previous treatment	Overall survival (90-day mortality, 1-year mortality)	Tested 5 different machine learning algorithms with 10-fold cross-validation. Stochastic gradient boosting showed the greatest performance with AUROC of 0.87 for 90-day mortality
Predicting response to treatment
Group (year)	Treatment intervention	Inputs	Outputs	Notes
Anastasio et al (2022) [2]	Orthobiologics	17 orthobiologics, 26 outcome metrics for bone healing	Predicted efficacy of orthobiologic combinations for bone healing	The most effective combinations of orthobiologics involved high concentrations of bone-morphogenic proteins, such as BMP2 and BMP7, and osteogenin
Nwachukwu et al (2020) [45]	Hip arthroscopy for femoroacetabular impingement	Clinical factors (symptom duration, preoperative outcome scores, intraarticular injection, comorbid depression or anxiety)	Predicted activities of daily living and sport-specific hip outcome scores after arthroscopy	Patient factors that were predictors of not achieving the MCID were comorbid anxiety/depression, symptom duration >2 years, preoperative intraarticular injections, and high preoperative outcome scores
Kumar et al (2020) [30]	Anatomic (aTSA) or reverse (rTSA) total shoulder arthroplasty	Preoperative data (demographics, diagnoses, comorbidities, implant type, ROM, imaging, and patient-reported outcomes)	Various pain scores and measures of active ROM at multiple postoperative timepoints	Accuracy at predicting which patients would experience clinical improvement greater than the MCID at 2- to 3-year follow-up was on average 92.9% across all 5 pain scores and 3 ROM measures
Shohat et al (2020) [61]	Failed DAIR procedure for PJI treatment	52 variables (demographics, comorbidities, clinical and laboratory findings)	Patient-specific probability for failing DAIR	Serum CRP levels and positive blood cultures were the 2 variables most associated with treatment failure

AIS adolescent idiopathic scoliosis, AP anterior–posterior, ASA American Society of Anesthesiology, AUROC area under the receiver operating characteristic curve, BMI body mass index, DAIR debridement antibiotics and implant retention, DL deep learning, MCID minimal clinically important difference, MRI magnetic resonance imaging, ORIF open reduction internal fixation, RA rheumatoid arthritis, ROM range of motion, THA total hip arthroplasty, TJA total joint arthroplasty, UKA unicompartmental knee arthroplasty, CDAI Clinical Disease Activity Index, PJI periprosthetic joint infection, CRP C-reactive protein.

^aExternally validated.

Modeling Disease Progression

At their core, prognostic ML models predict a likelihood of disease progression, usually through a measure of disease severity or overall survival at a future time point. Prediction can be conditioned on clinical variables, such as demographics, past medical history, and current responses to therapies. ML models can make patient-specific prognoses for conditions such as osteoarthritis, adolescent idiopathic scoliosis, rheumatoid arthritis (RA), and long-bone metastatic disease [43,55,66,73]. By combining imaging, patient history, and clinical data, these models can predict time-bound outcomes, such as a patient’s specific risk of losing at least 0.5 mm of knee cartilage in 1 year, RA disease activity at a patient’s next clinical visit, and mortality at 1 year in the setting of metastatic disease [43,55,66]. However, not all models need to be multimodal. For example, Yahara et al developed a model to predict the risk of curve progression and achieved strong predictive performance based solely on prior spine radiographs [73].

Predicting Postoperative Risks and Treatment Response

ML algorithms can be used to estimate future risk based on patient-specific factors from multimodal inputs. Examples of leveraging ML for the prediction of complications following surgical procedures include predicting the risk of hospitalization following ambulatory unicompartmental total knee arthroplasty, hyponatremia following total joint arthroplasty, and infection following internal fixation of tibial shaft fractures [26,32,33,36,61].

ML models can also be built to predict treatment success while giving insight into which variables are most impactful. For example, Nwachukwu et al evaluated 5 ML models to determine preoperative predictors of positive clinical outcomes after arthroscopic intervention for femoroacetabular impingement syndrome [45]. Among other factors, anxiety, depression, and a symptom duration of >2 years before surgery were shown to be significant in predicting clinically meaningful improvements in functional scores after surgery.

Finally, ML algorithms can also help pool and analyze data from current literature to provide comparisons among treatment options. Anastasio et al developed a neural-network-based model to rank the efficacy of various combinations of orthobiologics for bone healing using existing data [2]. In this way, AI can be leveraged for more efficient meta-analysis of treatment efficacy and further hypothesis generation.

Evaluating AI Models

An August 2023 American Medical Association survey of more than 1000 practicing physicians found that 70% reported concern regarding the increased use of AI in healthcare [1]. To increase trust in AI, an understanding of its limitations is critical. Before blindly accepting any AI prediction, one should critically judge how the model was trained and how its metrics are reported [46]. Other more technical issues—such as batch sizes, training length, and model architectures—are outside the scope of this review.

What Data Was the Model Trained on?

It is essential to consider an AI model’s training set, as poor, biased data can only generate a poor and biased model. For example, a model trained on a predominantly white and elderly population will notgeneralize its predictions toinput data from a minority or younger population. Furthermore, as with any research study, sample size matters. For example, if reviewing a classification model for tibial plateau fractures, assessing whether the training data included a proportionate number of each type of fracture is important in order to avoid a phenomenon known as “class imbalance.”

Who Determined the “Ground Truth” of the Data?

Most of the models we reviewed underwent supervised learning, which means that the input data are paired with the correct output, a “ground truth label,” before feeding it to the model. Therefore, it is important to judge who assigned the ground truth. For example, were Kellgren–Lawrence grades assigned to radiographs by a trained medical student or a board-certified MSK radiologist? Also, some classification systemshave poor baseline interrater reliability, even among fellowship-trained physicians. It is important to understand that errors in the ground truth will be propagated through a model. In simple terms,the model’s accuracy is as good as the accuracy of its training data.

Is the Model “Overfitting?”

When ML models learn too much about the distribution of training data, it is known as “overfitting.” In other words, overfitting occurs when the model learns to mimic the distribution of data in the training set, which biases its outputs. DL models are particularly prone to overfitting because of the inherent flexibility of ANNs. Generating a loss function curve during training and utilizing some form of model validation are ways to examine and reduce the risk of overfitting, respectively [37].

Was the Model Externally Validated?

To assess both the generalizability and risk of model overfitting, all published models are encouraged to undergo external validation. This is when a model is deployed on a different and unseen dataset, preferably from a different institution. However, 1 study reported that of 92 papers on AI applications and arthroplasty, only 3 (3.3%) were externally validated [49]. This is largely due to challenges with data sharing agreements and a lack of publicly shared MSK datasets.

Other limitations, such as a lack of model transparency, data privacy concerns, hesitancy from patients or clinicians, and ethical considerations, remain barriers to AI implementation into clinical workflows [3,17,42,49,65].

Future Directions in Generative AI

Generative AI is the term for models that produce synthetic text, images, audio, and other data types. Well-known examples include ChatGPT, DALL-E, and Style-GAN [19]. Current applicatios of generative AI to orthopedics include generating orthogonal radiographs (eg, a lateral knee X-ray from an anteroposterior), rotating hip radiographs by 15° in 3-dimensional space, and creating synthetic computed tomography-like images from magnetic resonance imaging (MRI) studies [11,47,50,54,58]. These innovations have the potential to decrease the number of imaging studies needed to diagnose patients, thus lowering cumulative radiation exposure and associated costs. Furthermore, these models could allow assessment of bone loss and quality from an MRI (traditionally performed on CT) and augment the diagnostic yield of planar radiographs, potentially changing clinical practice. Lastly, by conditioning the generation of synthetic images on demographic data, clinicians have begun to use generative AI to study radiographic differences of MSK diseases based on race, which could improve MSK health equity [25].

Furthermore, researchers have begun to employ generative AI to overcome the underrepresentation of groups in training data, enriching datasets with synthetic images that have been shown to boost model performance on external validation sets [29]. This use of generative AI has direct applications in studying rare diseases, for which acquiring large sample sizes can be difficult, and in decreasing concerns of data privacy, since the synthetic images could be used to train future models [10,18,27,59].