Developing a machine learning algorithm to predict the probability of aseptic loosening of the glenoid component after anatomical total shoulder arthroplasty: protocol for a retrospective, multicentre study

Abstract

Introduction

Despite technological advancements in recent years, glenoid component loosening remains a common complication after anatomical total shoulder arthroplasty (ATSA) and is one of the main causes of revision surgery. Increasing emphasis is placed on the prevention of glenoid component failure. Previous studies have successfully predicted range of motion, patient-reported outcomes and short-term complications after ATSA using machine learning methods, but an accurate predictive model for (glenoid component) revision is currently lacking. This study aims to use a large international database to accurately predict aseptic loosening of the glenoid component after ATSA using machine learning algorithms.

Methods and analysis

For this multicentre, retrospective study, individual patient data will be compiled from previously published studies reporting revision of ATSA. A systematic literature search will be performed in Medline (PubMed) identifying all studies reporting outcomes of ATSA. Authors will be contacted and invited to participate in the Machine Learning Consortium by sharing their anonymised databases. All databases reporting revisions after ATSA will be included, and individual patients with a follow-up less than 2 years or a fracture as the indication for ATSA will be excluded. First, features (predictive variables) will be identified using a random forest feature selection. The resulting features from the compiled database will be used to train various machine learning algorithms (stochastic gradient boosting, random forest, support vector machine, neural network and elastic-net penalised logistic regression). The developed and validated algorithms will be evaluated across discrimination (c-statistic), calibration, the Brier score and the decision curve analysis. The best-performing algorithm will be used to create an open-access online prediction tool.

Ethics and dissemination

Data will be collected adhering to the WHO regulation on data sharing. An Institutional Review Board review is not applicable. The study results will be published in a peer-reviewed journal.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• A large international database will be collected, which increases accuracy, validity and external applicability.

• A clinical prediction model using machine learning algorithms will be developed to estimate the probability of aseptic loosening of the glenoid after anatomical total shoulder arthroplasty.

• An open-access prediction tool based on the best-performing algorithm will be made available online that can guide medical professionals in personalised treatment decision-making.

• The study is dependent on data provided by third parties, which is a potential source of bias.

• Input variables will be selected and categorised based on completeness and uniformity across data sources, potentially decreasing the amount of detail in the study.

Introduction

Anatomical total shoulder arthroplasty (ATSA) is used for glenohumeral arthropathy causing pain and/or a reduction in range of motion. Despite technological advancements in recent years, glenoid component loosening remains a common complication after ATSA and is one of the main causes of revision surgery. Glenoid loosening can be a trying complication to manage, and the optimal course of treatment remains unclear.¹

Consequently, increasing emphasis is placed on the prevention of loosening. The predicted chance of glenoid component failure plays an important role in clinical decision-making such as patient selection for ATSA or which implants and techniques to use. Several previous studies assessing risk factors of glenoid component loosening identified patient, treatment and prosthesis characteristics related to glenoid component loosening. For example, male sex and a higher critical shoulder angle have been associated with higher rates of loosening and revisions.^{2 3} Glenoid retroversion did not impact implant survivorship in one study of ATSA with minimal, non-corrective reaming.⁴ However, a larger degree of retroversion may have more impact. Several previous studies have also identified aspects of the glenoid component design that correlated with the rate of loosening, such as whether the polyethylene is cross-linked, whether the component is pegged or keeled, or the usage of cement.^5–9 In spite of these studies identifying influential factors, accurate prediction of aseptic loosening of the glenoid component remains a challenge with conventional methods.

In recent years, machine learning or artificial intelligence has been used with increasing precision to predict outcomes after ATSA. A previous study using machine learning was able to accurately predict range of motion and patient-reported outcomes after ATSA. The most influential factors they reported were follow-up time, preoperative range of motion and patient-reported outcome measures, patients’ sex and surgery on the dominant upper limb.¹⁰ Another study was able to accurately predict the improvement in American Shoulder and Elbow Surgeons (ASES) scores after shoulder arthroplasty using machine learning.¹¹ The most relevant predictive factors were preoperative ASES scores, preoperative pain scores, Walch classification, fatty infiltration in the supraspinatus and infraspinatus, and age. A previous study using artificial intelligence to predict patient satisfaction 2 years after shoulder arthroplasty found baseline Single Assessment Numeric Evaluation score, exercise and activity, workers’ compensation status, diagnosis, symptom duration prior to surgery, body mass index (BMI), age, smoking status, anatomical versus reverse total shoulder arthroplasty and diabetes to be predicting factors.¹² Two studies report predictive models on short-term complications after ATSA. One study using machine learning to predict complications and 30-day unplanned readmissions found that a history of implant complication, severe chronic kidney disease, teaching hospital status, coronary artery disease and male sex were the most important features.¹³ The machine learning model found teaching hospital status and male sex as a markedly more important predictor compared with a logistic regression analysis of the same data. Another study on short-term complications after total shoulder arthroplasty found percentage haematocrit, BMI and operative time were of highest importance in outcome prediction.¹⁴ These studies demonstrate that machine learning may provide accurate predictions for the outcomes after ATSA. Machine learning is most effective with large amounts of data and is very dependent on the amount of detail. Furthermore, the algorithm needs to be widely applicable; a varied and international database provides the highest external validity.

To our knowledge, there are no studies predicting the long-term complications such as aseptic loosening of the glenoid component using advanced machine learning techniques. Furthermore, previous machine learning studies are limited in accuracy and validity due to the sample size and homogeneity. Therefore, this study aims to develop a clinical prediction model for aseptic loosening of the glenoid component using machine learning algorithms trained on a large international database using clustered data. The large combined dataset is less prone to overfitting, and allows direct validation of models across a range of populations and settings, thereby increasing generalisability.¹⁵ The predictive algorithm will be made available for clinical use through a publicly available online prediction tool.

Methods and analysis

Data collection

For this multicentre, retrospective study, individual patient data (IPD) will be collected from previously published studies reporting failure and revision of ATSA. A systematic literature search will be performed in Medline (PubMed) identifying all studies that report a cohort of ATSA including revision as an outcome, published between January 2000 and June 2023. The limit was set at January 2000 to increase the likelihood of the dataset that was used for the study still being available. The minimum required data retention period varies between countries but is generally 20 years or less. The full search strategy is available in online supplemental appendix 1. All original studies reporting revision or failure rates after primary ATSA will be included. Reviews and letters to the editor will be excluded, as well as studies published in languages other than English, Italian, Dutch and French. Authors will be requested to share the anonymised databases used for the identified studies. Only de-identified databases used for previous studies are included; authors are not required to gather additional data or access patient files. After sharing their data, the authors will be included in the Machine Learning Consortium. Inclusion criteria for individual patients within the provided databases are a minimum age of 18 years and a minimum follow-up of 2 years. Patients who underwent ATSA with a fracture as the indication or patients who underwent concomitant procedures such as a cuff repair, tendon transfer or bone graft will be excluded. The aim is to combine the IPD from previously published studies to create a large international cohort which can be used to train a machine learning algorithm to predict aseptic loosening of the glenoid component after ATSA. Based on previous studies, we estimate a glenoid revision rate of approximately 2%.^{6 16} The minimum number of events per variable to achieve sufficient accuracy differs per model and is not clearly defined for each technique.^{17 18} We aim to include at least 30 events per variable, resulting in a sample size of 7500 patients for a model with up to five predictive variables.

Data curation and missing data

Completeness across data sources will be assessed for each variable in the compiled multicentre database, and variables with sufficient completeness (>70% complete) will be selected as input for the machine learning algorithms. Variables with >30% missing data will be excluded. For the remaining variables, missing data will be completed by imputation using multivariate imputation by chained equations.¹⁹ Uniformity in reporting will be assessed for each variable. If possible, variables will be adjusted or categorised to ensure uniform reporting. In case uniformity of the reported variable across data sources cannot be achieved without guaranteeing correctness, the variable will be excluded. Each dataset will be split into training (80%) and test (20%) subsets, stratified by outcome. Fivefold cross-validation of the training set will be used to develop the machine learning models.²⁰ Data curation and imputation will be performed using R (R foundation for statistical computing, Vienna, Austria).

Variable selection

The primary outcome is a revision of the glenoid component for aseptic glenoid loosening. The input variables for both methods are dependent on the uniformity and completeness of the gathered data but will include demographics (eg, age, sex and ethnicity), patient-specific factors (eg, preoperative BMI, comorbidity, smoking, dominance, previous surgery), disease-specific factors (eg, affected side, indication, Walch classification, fatty infiltration of cuff muscles) and surgical characteristics (eg, corrective reaming, component design and type, component materials, cementing and sizes). Before training the machine learning models, relevant variables will be selected using random forest algorithms with recursive selection.²¹ At least 10 events for each predictor variable will be included in the model, adhering to the rule of thumb in predictive models of binary variables.²²

Development of prediction models

Different machine learning models result in varying performance metrics based on the type of input data (continuous, categorical, dichotomous). Due to the variation in type of input variables in the dataset, several different machine learning techniques will be used and compared based on model performance. The following machine learning algorithms were chosen for modelling based on prior research^23–27: stochastic gradient boosting, random forest, support vector machine, neural network and elastic-net penalised logistic regression. The algorithms will be trained on the training dataset with 10-fold cross-validation repeated three times. Cross-validation means dividing data into a selected number of groups, also called folds. First, the data will be divided into 10 equally sized folds. Then, the algorithms will be trained on 9 of the 10 folds (90% of the training data) and tested on the remaining fold (10% of the training data). Results will be averaged across all repetitions of this sequence. Machine learning algorithms will be developed using Python (The Python Software Foundation, Fredericksburg, USA). Hyperparameter tuning will be performed as recommended in the Python libraries. The statistician who performs the machine learning analysis will be blinded to the origin of the data, but the anonymised data source will be available to be included as a potential confounding factor.

Model performance

After training all models, the model performance will be analysed according to a proposed framework by Steyerberg et al including discrimination with the c-statistic, positive predictive value (PPV), true positive rate (TPR), precision-recall curve, calibration slope and intercept and the overall performance with the Brier score.²⁸

The c-statistic (area under the curve of a receiver operating characteristic curve) is a score ranging from 0.50 to 1.0 with 1.0 indicating the highest discrimination score and 0.50 indicating the lowest. The higher the discrimination score, the better the model’s ability to distinguish patients with and without the outcome of interest.^{20 23} The PPV is the proportion of true positive outcomes over the number of predicted positive outcomes. The TPR is the proportion of true positive outcomes over the number of observed positive outcomes. The precision recall curve is a plot of the PPV versus the TPR. A calibration plot plots the estimated versus the observed probabilities for the primary outcome. A perfect calibration plot has an intercept of 0 (<0 reflects overestimation and >0 reflects underestimation of the probability of the outcome) and a slope of 1 (model is performing similarly in training and test datasets).^{20 23 28} The null-model Brier score, which equals the probability of glenoid revision in the dataset, will be used to benchmark the algorithm’s Brier score. A Brier score lower than the null-model Brier score indicates superior performance of the prediction model to this null benchmark. Perfect prediction would have a Brier score of 0, whereas a Brier score of 1 would indicate the poorest possible prediction.²⁸

In addition, the decision curve analysis will be performed and visualised to investigate the net benefit (weighted average of true positives and false positives) of the conducted algorithms over the range of risk thresholds for clinical decision-making.^{23 28 29} The net benefit is a weighted average of true positives and false positives, formula=sensitivity×prevalence–(1–specificity)×(1–prevalence)×(odds at the threshold probability). The decision curve of the model will be compared with decision curves of treating everyone as being at risk and treating no one as being at risk.²³

Due to the large heterogeneity of the compiled dataset from different international sources and the internal validation of the prediction models, the generalisability of the model can be intrinsically confirmed using the above-mentioned performance tests. Therefore, it is not strictly necessary to externally validate the final algorithm. However, this study’s primary aim is model development. External validation in a specific setting is advised before applying the algorithm to clinical practice.

Open-access clinical prediction tool

The best-performing prediction algorithm will be used to create an open-access clinical prediction tool, in the form of a publicly available web application accessible on desktops, tablets and smartphones.

Statistical analysis

Categorical variables will be described as absolute numbers with frequencies, and continuous variables as medians with IQRs. The model performance metrics will be calculated with 95% CI. Given the retrospective study design, post hoc power analyses will be conducted to evaluate the sample size of the study with an alpha value of 0.05.

Guidelines

The study set-up will be performed following the Transparent Reporting of Multivariable Prediction Models for Individual Prognosis or Diagnosis Guideline for Clustered data.¹⁵

Patient and public involvement

None.

Ethics and dissemination

For safe multicentre data exchange and analysis, our Machine Learning Consortium will adhere to the WHO regulation ‘Policy on Use and Sharing of Data Collected by WHO in Member States Outside the Context of Public Health Emergencies’.³⁰ An Institutional Review Board (IRB) approval has been obtained for each of the included studies and the provided data are anonymised and de-identified; no additional prospective data are collected and contributing authors are not required to access any patient files. No IRB review is required for this study. Patient consent for publication is not applicable to this study.

The study results will be disseminated through publication in a peer-reviewed journal. To facilitate reproduction of the results and external validation of the algorithm, the (anonymous) code of the developed predictive algorithms will be made available upon request with the authors.

Data collection for this project is currently ongoing. The analysis will start in December 2023. The expected time of completion for the project is July 2024.

Discussion

For an informed decision when considering ATSA, it is important to be able to make an accurate prediction of arthroplasty failure. Previous studies have demonstrated several factors that affect complications and revision after ATSA, including male sex, comorbidities such as chronic kidney disease or coronary artery disease, percentage haematocrit, a higher critical shoulder angle, teaching hospital status, operative time, and the material and design of the prosthesis.2 3 5–9 13 14 Psychological studies have shown that in human judgement, only a limited amount of variables can be taken into account, and that prediction models are generally more accurate and less subject to bias.³¹ Machine learning algorithms have been shown to be an effective method in developing patient-specific prediction tools, which may complement human judgement when counselling patients in clinic.³² Creating an online tool for aseptic loosening of the glenoid component after ATSA can help guide surgeons in selecting patients who will most benefit from this treatment and considering alternatives in cases of high-risk estimates.

The strength of this project is the large amount of data that will be gathered from authors participating in the Machine Learning Consortium, aiming to include a minimum of 7500 patients in total. Using a large, heterogeneous international database for development of the algorithm and prediction tool will result in high external validity and may improve applicability worldwide.¹⁵ However, most machine learning techniques require a larger sample size to achieve an accurate prediction compared with traditional regression models. The minimum events per variable are not clearly defined and differ per technique. Furthermore, in gathering data retrospectively from various sources, the study is subject to variances in the included variables. Low completeness and large variability of reporting may introduce bias. However, only variables that are consistently reported in multiple data sources will be included in the final analysis; variables will be categorised to increase uniformity and missing data will be imputed where possible. The exclusion and categorisation of variables will have to be balanced with the amount of detail in the final analysis. Furthermore, the accuracy of data collection is dependent on third parties providing the data; the method of data collection cannot be verified for all sources. However, the data source will be considered as a confounder. Furthermore, the variety in data sources will increase the external applicability of the algorithm. Last, as machine learning prediction models for a dichotomous outcome are limited to risk classification, the individual risk must be interpreted in the clinical context when used for medical decision-making.

Ethics statements

Patient consent for publication

Not required.