A Novel Machine Learning Model to Predict Revision ACL Reconstruction Failure in the MARS Cohort

Abstract

Background:

As machine learning becomes increasingly utilized in orthopaedic clinical research, the application of machine learning methodology to cohort data from the Multicenter ACL Revision Study (MARS) presents a valuable opportunity to translate data into patient-specific insights.

Purpose:

To apply novel machine learning methodology to MARS cohort data to determine a predictive model of revision anterior cruciate ligament reconstruction (rACLR) graft failure and features most predictive of failure.

Study Design:

Cohort study; Level of evidence, 3.

Methods:

The authors prospectively recruited patients undergoing rACLR from the MARS cohort and obtained preoperative radiographs, surgeon-reported intraoperative findings, and 2- and 6-year follow-up data on patient-reported outcomes, additional surgeries, and graft failure. Machine learning models including logistic regression (LR), XGBoost, gradient boosting (GB), random forest (RF), and a validated ensemble algorithm (AutoPrognosis) were built to predict graft failure by 6 years postoperatively. Validated performance metrics and feature importance measures were used to evaluate model performance.

Results:

The cohort included 960 patients who completed 6-year follow-up, with 5.7% (n = 55) experiencing graft failure. AutoPrognosis demonstrated the highest discriminative power (model area under the receiver operating characteristic curve: AutoPrognosis, 0.703; RF, 0.618; GB, 0.660; XGBoost, 0.680; LR, 0.592), with well-calibrated scores (model Brier score: AutoPrognosis, 0.053; RF, 0.054; GB, 0.057; XGBoost, 0.058; LR, 0.111). The most important features for AutoPrognosis model performance were prior compromised femoral and tibial tunnels (placement and size) and allograft graft type used in current rACLR.

Conclusion:

The present study demonstrated the ability of the novel AutoPrognosis machine learning model to best predict the risk of graft failure in patients undergoing rACLR at 6 years postoperatively with moderate predictive ability. Femoral and tibial tunnel size and position in prior ACLR and allograft use in current rACLR were all risk factors for rACLR failure in the context of the AutoPrognosis model. This study describes a unique model that can be externally validated with larger data sets and contribute toward the creation of a robust rACLR bedside risk calculator in future studies.

Registration:

NCT00625885 (ClinicalTrials.gov identifier).

Revision anterior cruciate ligament reconstruction (rACLR) continues to remain a challenge. ⁴² In patients undergoing rACLR, the rate of objective graft failure has been reported to be as high as 13.7%.^7,50 Compared with primary ACLR, rACLR is a more technically demanding procedure with a 3- to 4-fold increased risk of graft failure and poorer functional outcomes.^49-51

The Multicenter ACL Revision Study (MARS) Group, created in 2005, ²⁹ as well as other studies, has implicated several predictive factors for outcomes after rACLR. These include technical factors such as graft type, as well as patient factors like preoperative knee hyperextension, younger age, and higher activity level. ⁹ Other investigations have implicated technical errors in graft preparation, placement, tensioning, and fixation, as well as anatomic factors like posterior tibial slope, anterolateral rotary instability, and meniscal deficiency.^14,41 Although these findings have guided clinical decision-making, rACLR outcomes appear to be dependent on multiple factors. ²⁹ More accurate predictors of graft failure may improve preoperative counseling, operative management decisions, and cost of care.

Machine learning is increasingly utilized in health care to harness insights from clinical data sets (big data) to provide patient-specific predictions of an outcome of interest. ³ In the orthopaedic literature, a number of studies on machine learning have built risk calculators with the potential for use in a variety of settings from preoperative counseling to advocacy for appropriate reimbursement based on patient risk.^{22,23,25,32-34} Yet the lack of standardization in machine learning methodology, awareness of machine learning techniques in the orthopaedic community, and data quality limit the clinical utility of these results.^21,24,27

AutoPrognosis is a novel, validated ensemble algorithm that combines the strongest features of traditional statistical approaches and popular algorithms like random forest (RF) and XGBoost to create a single, well-calibrated predictive “supermodel.” AutoPrognosis has been validated in various orthopaedic subspecialty studies, predicting postoperative complications after total hip arthroplasty, total knee arthroplasty, and posterior cervical spinal fusion.^10,39,40 The application of validated machine learning methodology to MARS cohort data presents a valuable opportunity to translate data into patient-specific insights.

In this study, we sought to apply novel machine learning methodology to the MARS cohort data to determine (1) an optimal predictive model of rACLR graft failure and (2) features that are important for the model to be able to accurately predict rACLR graft failure. We hypothesized that the AutoPrognosis model would have the most robust predictive and discriminative ability for the outcome of interest, and important features would span surgical, clinical, radiographic, and patient-reported outcome (PRO) variables.

Methods

Study Participants

MARS Cohort

The MARS Group consists of 83 sports medicine fellowship–trained surgeons at 52 academic and private practice sites (ClinicalTrials.gov: NCT00625885). Institutional review board approval was obtained at all investigating sites. Between 2006 and 2011, 1233 patients undergoing rACLR were prospectively enrolled. Eligible patients underwent rACLR, including a second or greater rACLR; multiligament reconstructions were excluded. Detailed descriptions of the MARS cohort have been previously published. ²⁹

All patients within the MARS cohort completed an informed consent form and 13-page questionnaire of baseline patient characteristics, injury and sports participation history, knee surgical history, medical comorbidities, and baseline PROs including the Marx activity rating scale, Western Ontario and McMaster Universities Osteoarthritis Index, Knee injury and Osteoarthritis Outcome Score (KOOS), 36-Item Short-Form Health Survey (SF-36), and International Knee Documentation Committee Subjective Form. Patients were followed up at 2 and 6 years postoperatively regarding outcome measures, additional surgeries, and graft failure. Additionally, the following knee radiographs were obtained in 630 patients from a previous MARS Group investigation ³¹ : bilateral knee standing anteroposterior, full-extension lateral, bilateral 45° bent-knee posterior-anterior, long-leg alignment, and bilateral patellofemoral views. The radiographs were used to measure femoral and tibial tunnel positions, sagittal view physiological tibial plateau slope angle, leg alignment, and joint space narrowing. Tunnel position and size were deemed accurate or compromised by the attending surgeons who completed the surgeon questionnaires. These determinations were not held to specific numeric measurements.

Surgeons were free to perform and use the surgery and graft of their preferred choice. If an allograft was used, it was supplied by MTF Biologics to ensure processing consistency. Surgeons completed a 49-page questionnaire of intraoperative findings including examination under anesthesia results; surgical technique; concomitant cartilage injury based on modified Outerbridge classification; meniscal injuries classified by size, location, degree of tear, and treatment; and rehabilitation guidelines. These data were entered into a master database and underwent quality control checks before data analysis.

Patient Selection

MARS cohort patients who completed both the follow-up questionnaire and telephone follow-up for subsequent surgery at 6 years postoperatively were considered for inclusion in the current study. Patients with missing data for the outcome variable were excluded from analysis.

Outcomes

The primary outcome of the present study was graft failure after rACLR, which was ascertained by querying patients. All patients who reported experiencing graft failure also reported receiving professional confirmation with operative report, clinic report, or magnetic resonance imaging report.

Data Preprocessing and Feature Engineering

We conducted data preprocessing, model creation, evaluation, and interpretation using Python (Version 3.10; Python Software Foundation), ⁴⁷ the Anaconda distribution platform, ² and the Jupyterlab interactive development environment. ²² Standard libraries including pandas, ³⁷ NumPy, ¹⁶ scikit-learn (sklearn), ³⁸ and Matplotlib were used for the data analysis and visualization. Descriptive statistics and table creation were performed using R Version 4.1.0 (R Foundation for Statistical Computing).

All data collected for the MARS cohort were considered for inclusion in the analysis. Significant data preprocessing including feature engineering and imputation of missing values was conducted in order to yield a usable set of features for model creation. Nominal (or categorical) features were encoded using OneHotEncoder in sklearn, whereby features are split into their category levels where categories are unrelated. An example of this is the feature “current femoral fixation,” in which the options are “combination,”“cross pin,”“interference screw,”“suture + button/Endobutton,” and “other.” Ordinal features like PROs on a Likert scale, whose categories have a clear order but the size of differences between categories is not necessarily quantifiable, were encoded using sklearn category encoder's OrdinalEncoder, whereby categories were mapped to numeric values that preserved the relationships between categories. ⁴⁶ For all features, category levels were required to meet a percentage threshold of ≥0.04 in order to be encoded and were incorporated into other categories until the threshold was met. Conditional variables were excluded from analysis because of low frequencies that could not be managed without compromising data quality.

Per validated methodology, features with >60% missing values were excluded. For the remaining features, missing values were imputed using the validated SimpleImputer in sklearn. Means were used for continuous and ordinal features, and modes used for binary and nominal features.

Model Development and Evaluation

Five machine learning models were developed to predict graft failure at 6 years postoperatively using preprocessed patient, clinical, and surgical features. These models included traditional binary classifiers (logistic regression [LR], XGBoost, gradient boosting [GB], and RF) and a validated ensemble algorithm (AutoPrognosis).

AutoPrognosis utilizes pipelines to build an ensemble “supermodel,” as depicted in Figure 1. Each pipeline contains an imputation, feature processing, and classification algorithm, as well as a calibration method. AutoPrognosis automatically conducts and optimizes hyperparameter searches for each classification method and selects and ranks top-performing pipelines based on suspected contribution to overall model performance. AutoPrognosis was implemented using code first provided by Alaa and van der Schaar ¹ ; LR, RF, and GB were implemented using the sklearn library; and XGBoost was built using the XGBoost library.

Figure 1.

AutoPrognosis design diagram.

The study cohort was split 80:20 into a training and held-out test cohort. Fivefold stratified cross-validation with the training cohort was used to determine each model's discriminative power and calibration. The held-out test cohort was used to evaluate model performance, reported as mean and standard error.

Validated performance metrics were used to measure discrimination and calibration. The area under the receiver operating characteristic curve (AUROC) is the probability the predictive model correctly assigns a patient who experienced the outcome of interest a higher risk than a patient who did not. AUROC values <0.7 indicate low discriminative power, values from 0.7 to 0.9 indicate moderate discriminative power, and values >0.9 indicate high discriminative power. ¹² The area under the precision-recall curve (AUPRC) is the probability that a model correctly predicts which patients experience the outcome of interest while minimizing false-positive results. AUPRC is useful in the setting of an imbalanced data set like MARS. The closer the AUPRC is to 1 and the higher it is compared with the baseline AUPRC (a result of random prediction), the closer the model is to an ideal classifier.

Calibration is a measure of how closely the model's predictions match the outcomes seen in the study cohort. The Brier score, a measure of both model calibration and discriminative power, is measured by taking the mean square error between values observed in the data and probabilities predicted by the model. Brier scores range between 0 and 1, and a lower score indicates superior performance.

Feature Importance

The Friedman partial dependence function and derived partial dependence plot (PDP)–based feature importance were used to determine the most important individual features based on their contribution to a model's predictive ability by assessing the average effect in predicted risk when a feature's value is altered. ¹³ The caveat with PDP feature importance as a measure is that an important feature can have a large contribution to a model's predictive ability without necessarily being important in a model's discriminative ability. Therefore, feature importance was also calculated using perturbation-based feature importance, which ranks features in terms of AUROC and AUPRC loss with modification of an individual feature. Features identified using perturbation-based importance were key to model discriminative ability.

We reported the most important features that appear in the top 10 in PDPs and both perturbation-based feature importance measures, as these are key to both model predictive and discriminative ability. For each of these features, individual feature behavior in model prediction was further elucidated using PDPs. Furthermore, the use of multiple feature importance measures serves as an additional check to prevent inappropriate importance designations, because the black-box nature of algorithms such as AutoPrognosis can give importance to nonphysiological patient features that affect a small portion of patients in the cohort (eg, surgeon years of experience).

Results

Study Population

Of the complete MARS cohort of 1233 patients, 273 (22.1%) patients were excluded from the study (82 had incomplete data and 191 had inadequate follow-up). The final study cohort used for analysis was 960 (77.9%) patients.

The baseline cohort characteristics and key variables are summarized in Appendix Tables A1 to A4. Variables in the tables represent only a fraction of the total number of variables in the data set that underwent preprocessing and were eventually included in the model. Of the 960 patients, 530 (55.2%) were male. The median age of the cohort was 26 years, and the median body mass index was 25.1. Most often, patients were White, did not smoke, and lived with 2 other people. Patients had a median of 15 years of education and worked 40 h/wk. The reported mechanism of injury was most often nontraumatic with gradual onset (n = 512; 53.3%). At the time of initial injury, 706 (74.3%) patients were playing sports, 230 (24.3%) had contact with another player, 254 (26.9%) were jumping, and 728 (77.7%) felt or heard a pop. Most patients (88%) were undergoing their first rACLR, at a median of 3.7 years from their previous ACLR, and 283 (29.5%) patients underwent revision with the surgeon who had performed their previous ACLR. In rank order, autograft (n = 474; 49.4%), followed by allograft (n = 456; 47.5%) and combination graft (n = 29; 3.0%), was the most frequently used graft type for the current revision surgery.

Model Performance

Using the MARS cohort subset of 960 patients who completed 6-year follow-up, 5.7% (n = 55) of whom experienced graft failure, we built 5 algorithms predicting the risk of graft failure after rACLR. Pipelines were used to build the weighted ensemble AutoPrognosis model (Table 1).

Table 1.

List of 3 Pipelines Fitted to MARS Cohort ^a

Pipeline	Dimensionality Reduction; Scaler; Classifier	Hyperparameters	Weight
1	Variance threshold; MaxAbs scaler; random forest	({threshold=0.001}, None, {n_estimators=50, max_depth=4})	0.333
2	FastICA; MaxAbs scaler; Catboost	({n_components=3}, None, {n_estimators=100, depth=5, grow_policy=“SymmetricTree”})	0.417
3	None; none; Catboost	(None, None, {n_estimators=100, depth=3, grow_policy=“Depthwise”})	0.250

^aMARS, Multicenter ACL Revision Study.

While all models had moderate to good discriminative power, AutoPrognosis demonstrated the highest discriminative power compared with other models (AUROC, 0.703 ± 0.036) (Table 2). The AutoPrognosis model was well calibrated, with calibration scores similar to those of the other studied models (Brier score, 0.053 ± 0.001). The AutoPrognosis model's AUPRC (0.152 ± 0.043) outperformed most models and had scores similar to those of the XGBoost and GB models. A baseline AUPRC of 0.057 would be demonstrated if a classifier were to generate random predictions. A confusion matrix for AutoPrognosis that was created using the held-out test cohort, with a threshold of 0.05729, is illustrated in Table 3. The receiver operating characteristic curves and precision-recall curves for the AutoPrognosis and LR models are illustrated in Figure 2.

Table 2.

Discrimination and Calibration Statistics ^a

Method	AUROC	AUPRC	Brier Score
AutoPrognosis	0.703 ± 0.036	0.152 ± 0.043	0.053 ± 0.001
LR	0.592 ± 0.018	0.116 ± 0.036	0.111 ± 0.021
XGBoost	0.680 ± 0.021	0.150 ± 0.072	0.058 ± 0.006
GB	0.660 ± 0.020	0.151 ± 0.063	0.057 ± 0.006
RF	0.618 ± 0.062	0.129 ± 0.051	0.054 ± 0.001

^aData are presented as mean ± SD. AUPRC, area under the precision-recall curve; AUROC, area under the receiver operating characteristic curve; GB, gradient boosting; LR, logistic regression; RF, random forest.

Table 3.

AutoPrognosis Model Confusion Matrix ^a

	Predicted Positive	Predicted Negative
Observed positive	6	5
Observed negative	18	163

^aData are presented as No. of observations. Accuracy = 0.880; F1 score = 0.343.

Figure 2.

(A) Receiver operating characteristic curves and (B) precision-recall curves for the AutoPrognosis and logistic regression models.

Feature Importance

Features deemed important for AutoPrognosis differed from those for LR model performance. For AutoPrognosis, the 10 most important features based on PDP and perturbation-based feature importance measures are listed in Table 4. Although important features varied by ranking methodology, compromised prior ACLR femoral tunnel position and size, compromised prior ACLR tibial tunnel position and size, and graft type used in current rACLR ranked in the top 10 regardless of methodology. Therefore, these features were deemed most important for both AutoPrognosis model predictive and discriminative ability.

Table 4.

AutoPrognosis Feature Ranking Based on PDP and Perturbation-Based (AUROC Loss and AUPRC Loss) Feature Importance Measures ^a

Feature	PDP Importance Rank	Perturbation Importance Rank (AUROC Loss)	Perturbation Importance Rank (AUPRC Loss)
Surgeon years of experience	1	277	304
Contralateral hamstring tendon graft use in previous ACLR	2	182	160
Compromised prior ACLR femoral tunnel position and size	3 b	1	2
Prior ACLR femoral tunnel position in preop sagittal view radiographs (measurement technique 1) c	4	35	303
Prior ACLR femoral tunnel position in preop sagittal view radiographs (measurement technique 2) d	5	15	52
Baseline age	6	27	307
Compromised prior ACLR tibial tunnel position and size	7 b	2	3
SF-36 PCS subscore	8	118	281
KOOS-ADL subscore	9	278	26
Graft type used in current rACLR	10 b	3	1

^aBolded features are those that rank within the top 10 in all feature importance measures for the AutoPrognosis model. ACLR, anterior cruciate ligament reconstruction; ADL, Activities of Daily Living; AUPRC, area under the precision-recall curve; AUROC, area under the receiver operating characteristic curve; KOOS, Knee injury and Osteoarthritis Outcome Score; LR, logistic regression; PCS, physical component summary; PDP, partial dependence plot; preop, preoperative; SF-36, 36-Item Short-Form Health Survey.

^bFeatures that were ranked within the top 10 for LR based on PDP feature importance measurements.

^cRecorded as a percentage of the distance from the femoral tunnel location to the Blumensaat line.

^dMeasured as a percentage of the distance from the femoral tunnel position to the cortex width.

PDP-based importance calculations for AutoPrognosis demonstrated the following top 5 contributors to model predictive ability: surgeon years of experience, contralateral hamstring tendon graft use in prior ACLR, compromised prior ACLR femoral tunnel position and size, and prior ACLR femoral tunnel position measured via preoperative radiographs (with techniques 1 and 2). AutoPrognosis perturbation-based calculations showed the following top 5 contributors to AUROC performance: prior ACLR femoral tunnel position and size, prior ACLR tibial tunnel position and size, current rACLR graft type, prior ACLR graft choice, and current rACLR femoral tunnel position and size. Perturbation-based calculations showed the following top 5 contributors to AUPRC performance: current rACLR graft type, prior ACLR femoral tunnel position and size, prior ACLR tibial tunnel position and size, prior ACLR tibial fixation method, and patellar tendon autograft use in prior ACLR. Other important features included baseline PRO scores (SF-36 subscale scores for physical function and KOOS Activities of Daily Living), baseline age, sex, body mass index, and sports activity at time of injury (Table 5).

Table 5.

Top 10 Important Features by Feature Importance Measure ^a

Rank	PDP Importance	Perturbation Importance (AUROC Loss)	Perturbation Importance (AUPRC Loss)
1	Surgeon years of experience	Prior ACLR femoral tunnel position and size	Current rACLR graft type
2	Contralateral hamstring tendon graft use in prior ACLR	Prior ACLR tibial tunnel position and size	Prior ACLR femoral tunnel position and size
3	Compromised prior ACLR femoral tunnel position and size	Current rACLR graft type	Prior ACLR tibial tunnel position and size
4	Prior ACLR femoral tunnel position in preop sagittal view radiographs (measurement technique 1)	Prior ACLR graft choice	Prior ACLR tibial fixation method
5	Prior ACLR femoral tunnel position in preop sagittal view radiographs (measurement technique 2)	Current rACLR femoral tunnel position and size	Patellar tendon autograft use in prior ACLR
6	Baseline age	Surgeon’s opinion on cause of prior ACLR failure	Patient sex
7	Compromised prior ACLR tibial tunnel position and size	Patellar tendon autograft use in prior ACLR	Baseline work status
8	SF-36 PCS subscore	Ipsilateral patellar tendon graft use in prior ACLR	Sports activity involving the ipsilateral knee at the time of injury
9	KOOS-ADL subscore	Technical cause of prior ACLR failure (in the surgeon's opinion)	Baseline BMI
10	Graft type used in current rACLR	Prior ACLR tibial fixation method	BPTB graft source for prior ACLR graft

^aACLR, anterior cruciate ligament reconstruction; ADL, Activities of Daily Living; AUPRC, area under the precision-recall curve; AUROC, area under the receiver operating characteristic curve; BMI, body mass index; BPTB, bone–patellar tendon–bone; KOOS, Knee injury and Osteoarthritis Outcome Score; PCS, physical component summary; PDP, partial dependence plot; preop, preoperative; SF-36, 36-Item Short-Form Health Survey.

For the top 3 AutoPrognosis features across all importance measures, PDPs illustrate the feature's relative contribution and behavior in model prediction (Figure 3). Within current graft type, allograft use as a category had the largest relative contribution to predicted risk of graft failure (Figure 3A). Regarding prior ACLR, compromised femoral position and tunnel size (Figure 3B) and compromised tibial position and tunnel size (Figure 3C) were the largest contributors to predicted risk of current rACLR graft failure.

Figure 3.

Partial dependence plots of predicted risk of graft failure after revision anterior cruciate ligament reconstruction (ACLR) based on (A) graft type used in current revision ACLR, (B) femoral tunnel placement in prior ACLR, and (C) tibial tunnel placement in prior ACLR.

Discussion

As graft failure continues to be a devastating complication of rACLR, it remains a challenge to predict failure on a patient-specific level. The most important finding of this study was that of all the described models in this machine learning analysis, AutoPrognosis, appeared to best predict rACLR graft failure with moderate discriminative power. The most important features for AutoPrognosis model performance and discriminative power were both the femoral and tibial tunnel placements and sizes during the prior ACLR and the graft type that was used in current rACLR. Specifically, compromised ACL femoral and tibial tunnel positions and sizes, and allograft use were associated with an increase in contribution to the model's predicted risk of graft failure. The AutoPrognosis algorithm and aforementioned findings build on previous studies that have reported on risk factors for rACLR graft failure.

Of the identified key features, several have been previously implicated as drivers of graft failure risk after rACLR. Conversely, several features previously demonstrated as important, like age and sex, were not found to be important features in the current analysis.^5,20 When considering the implications of these findings, it is important to note that important features must be seen within the context of the predictive model. Important features (or lack thereof) should not be interpreted as having causal relationships or lack thereof with the outcome of interest, and predictive models should not be interpreted as explanatory models.

The present study is the first to conduct a machine learning analysis of the MARS cohort in order to deliver patient-specific risk predictions of rACLR outcomes. Our findings present an opportunity to influence rACLR risk stratification from a subjective consideration based on surgeon knowledge and experience to an evidence-based quantitative measure. A risk calculator built with our model has the potential to guide preoperative counseling, shared decision-making, and intentional risk mitigation. Furthermore, in a health care environment where cost containment has increased in importance^21,36 and cost estimates for a single rACLR (in 2008 US dollars) were reportedly a median of $20,501.93 (range, $14,420.72-$51, 211.13), more accurate risk stratification may allow for fairer risk-adjusted reimbursement. ⁶

Compromised femoral and tibial tunnel positions and sizes in prior ACLR procedures were found to be important contributors in the risk of current rACLR graft failure in the present study. In previous studies, femoral tunnel malpositioning has been extensively studied and widely accepted as a risk factor for graft failure,^{8,15,35,43-45} and it is one of the most commonly cited reasons for primary ACLR graft failure by surgeons, including those in the MARS Group. Although prior femoral tunnel position has not necessarily been implicated as a predictor for current rACLR graft failure in a previous MARS Group investigation, ⁸ prior tunnel position and size are important considerations in rACLR planning.

Poorly positioned and sized prior tunnels may lead to ongoing challenges in the current rACLR with respect to tunnel preparation potentially requiring grafting and identification of osseous landmarks to establish the appropriate footprint. ¹¹ While degree of femoral or tibial tunnel malpositioning and enlargement were not quantitatively measured in the MARS data set, the presence of a type 2 tunnel with slight malpositioning may compromise ideal placement of the revision tunnel, especially in a single-stage procedure where the graft is most at risk of sliding anteriorly. ²⁶

Additionally, while bone graft procedure data and physiological tibial plateau slope angle measurements were included in model creation, they were not deemed important features. Of note, an ideally positioned prior tunnel had a higher contribution to risk of graft failure in the current model compared with an isolated malpositioned tunnel. It is important to note that the combination of tunnel position and size was highly predictive, but compromised tunnel position alone or size alone was less predictive than ideal tunnels. While the mechanism of this remains to be investigated, the extraction of hardware and reuse of the existing ideally placed tunnel may present unique challenges including iatrogenic loss of cortical containment or tunnel interference due to retained implants, sclerotic bone, or residual tissue. ⁴⁸ Although there is a dearth of literature investigating the role of the tibial tunnel in rACLR graft failure, studies have demonstrated the importance of tibial tunnel positioning in restoring knee stability and minimizing graft impingement in extension.^18,19 While optimal femoral positioning may restore rotational control and improve coronal graft obliquity, in the setting of vertical graft position in the sagittal plane, tibial tunnel placement remains important in preventing instability. ⁴ Similar to the femoral tunnel, prior tibial tunnel positioning and sizing may impact the ability to optimize tibial tunnel placement in the current rACLR, rendering it an important independent predictor of graft failure.

Allograft use is a known risk factor for rACLR graft failure compared with autograft use, as demonstrated by previous MARS investigations of graft failure by the 2- and 6-year postoperative time points.^28,30 Our results echo these findings and similarly implicate current graft type as an important feature in the present model.

Limitations

There were several limitations to the present study. First, because of the relative rarity of graft failure as an outcome, our data set was imbalanced. However, to address potential concern of overfitting and overprediction of negative outcomes, all models were evaluated with the AUPRC, wherein AutoPrognosis demonstrated superior performance. Second, we must acknowledge that any biases represented in the data may be amplified by machine learning methods, potentially furthering biases against underrepresented patient populations including women, ethnic minorities, and patients of lower socioeconomic status. ¹⁷ Finally, while graft failure is used as a quantitative proxy for patient outcomes, it is only one of several determinants of patient success or failure. Further studies analyzing PROs, reoperations, risk for posttraumatic osteoarthritis, and graft survival could provide additional insight and support the development of a bedside risk calculator to aid in risk stratification and counseling of patients undergoing rACLR.

Conclusion

The present study demonstrated the ability of the novel AutoPrognosis machine learning model to best predict the risk of graft failure in patients undergoing rACLR at 6 years postoperatively with moderate predictive ability. Femoral and tibial tunnel sizes and positions in prior ACLR and allograft use in current rACLR were all risk factors for rACLR failure in the context of the AutoPrognosis model. This study describes a unique model that can be externally validated with larger data sets and contribute toward the creation of a robust rACLR bedside risk calculator in future studies.

Appendix

Appendix Table A1.

Characteristics and PRO Scores of the MARS Cohort Included in the Machine Learning Analysis (N = 960 Patients) ^a

Variable	Value	Variable	Value
Age, y	26.00 (19.00-35.00)	Giving-way episodes since injury	5.00 (2.00-12.00)
Missing data	0 (0)	Missing data	59 (6.1)
Sex		Reinjuries	1.00 (1.00-2.00)
Male	530 (55.2)	Missing data	34 (3.5)
Female	430 (44.8)	Previous ACLR on contralateral knee
Missing data	0 (0)	No	862 (89.8)
BMI, kg/m2	25.10 (22.40-28.20)	Yes	98 (10.2)
Missing data	5 (0.5)	Missing data	0 (0)
Smoking status		KOOS
Nonsmoker	743 (78.4)	Symptoms	67.86 (53.57-82.14)
Quit	121 (12.8)	Missing data	2 (0.2)
Current	84 (8.9)	Pain	75.00 (61.11-86.11)
Missing data	12 (1.3)	Missing data	2 (0.2)
Ethnicity		ADL	86.76 (69.12-95.59)
White	817 (85.6)	Missing data	1 (0.1)
Other	43 (4.5)	Sports/Rec	45.00 (25.00-65.84)
Black	36 (3.8)	Missing data	5 (0.5)
Hispanic	31 (3.2)	Knee-related QoL	31.25 (18.75-43.75)
Asian	27 (2.8)	Missing data	0 (0)
Missing data	6 (0.6)	WOMAC
Marital status		Stiffness	75.00 (50.00-87.50)
Single	589 (61.7)	Missing data	2 (0.2)
Married	323 (33.8)	Pain	85.00 (70.00-95.00)
Separated	43 (4.5)	Missing data	3 (0.3)
Missing data	5 (0.5)	ADL	86.76 (69.12-95.59)
Members in household, n	2.00 (1.00-3.00)	Missing data	2 (0.2)
Missing data	66 (6.9)	SF-36
Education level, y	15.0 (12-17)	Physical function	44.88 (38.09-52.82)
Missing data	5 (0.5)	Missing data	0 (0)
Work status		Role physical	42.16 (32.36-54.40)
Working full time	448 (47.3)	Missing data	0 (0)
Student	319 (33.7)	Body pain	46.06 (37.18-51.13)
Other	119 (12.6)	Missing data	4 (0.4)
Working part time	61 (6.4)	General health	55.32 (50.55-61.51)
Missing data	13 (1.4)	Missing data	2 (0.2)
Workload, h/wk	40.00 (20.00-50.00)	Vitality	52.09 (45.85-58.33)
Missing data	180 (18.8)	Missing data	2 (0.2)
Playing sports at time of injury		Social function	45.94 (40.49-56.85)
No	244 (25.7)	Missing data	2 (0.2)
Yes	706 (74.3)	Role emotional	51.99 (40.33-55.88)
Missing data	10 (1.0)	Missing data	1 (0.1)
Contact with another player		Mental health	50.01 (41.56-55.64)
No	717 (75.7)	Missing data	2 (0.2)
Yes	230 (24.3)	PCS	45.94 (39.10-52.82)
Missing data	13 (1.4)	Missing data	5 (0.5)
Jumping at time of injury		MCS	52.70 (43.95-57.94)
No	691 (73.1)	Missing data	5 (0.5)
Yes	254 (26.9)	Marx	11.00 (4.00-16.00)
Missing data	15 (1.6)	Missing data	5 (0.5)
Felt/heard a pop at time of injury		IKDC	51.72 (39.08-64.37)
No	209 (22.3)	Missing data	2 (0.2)
Yes	728 (77.7)
Missing data	23 (2.4)
Time to swelling, h	4.00 (2.00-10.00)
Missing data	67 (7.0)

^aData are originally from the MARS Group (2010). ²⁷ Continuous variables are presented as median (interquartile range); categorical variables and missing data are presented as n (%). ACLR, anterior cruciate ligament reconstruction; ADL, Activities of Daily Living; BMI, body mass index; IKDC, International Knee Documentation Committee; KOOS, Knee injury and Osteoarthritis Outcome Score; MARS, Multicenter ACL Revision Study; MCS, mental component summary; PCS, physical component summary; PRO, patient-reported outcome; Rec, Recreation; QoL, Quality of Life; SF-36, 36-Item Short-Form Health Survey; WOMAC, Western Ontario and McMaster Universities Osteoarthritis Index.

Appendix Table A2.

Previous Procedures on Affected Knee (N = 960 Patients) ^a

Variable	Value	Variable	Value
Medial meniscus		Previous medial meniscal surgery
Debridement		No	604 (62.9)
No	661 (68.9)	Yes, excision	279 (29.1)
Yes	299 (31.1)	Yes, repair not healed/unstable	51 (5.3)
Missing data	0 (0)	Yes, repair healed/stable	26 (2.7)
Repair		Missing data	0 (0)
No	869 (90.5)	Previous lateral meniscal surgery
Yes	91 (9.5)	No	769 (80.4)
Missing data	0 (0)	Yes, excision	148 (15.5)
Transplant		Yes, repair	39 (4.1)
No	958 (99.8)	Missing data	4 (0.4)
Yes	2 (0.2)	Previous graft type (most recent only)
Missing data	0 (0)	Autograft	659 (68.6)
Lateral meniscus		Allograft	269 (28.0)
Debridement		Both autograft and allograft	23 (2.4)
No	794 (82.7)	Missing data	9 (0.9)
Yes	166 (17.3)	Previous graft source
Missing data	0 (0)	BPTB	509 (53.0)
Repair		Soft tissue	363 (37.8)
No	912 (95.0)	Both BPTB and soft tissue	8 (0.8)
Yes	48 (5.0)	Other	80 (8.3)
Missing data	0 (0)	Missing data	0 (0)
Transplant	—	Prior femoral fixation
Previous articular cartilage surgeries		Interference screw	583 (60.9)
No	847 (88.2)	Suture and Endobutton	164 (17.1)
Yes	113 (11.8)	Cross pin	115 (12.0)
Missing data	0 (0)	Other	77 (8.0)
ACL repair		Combination	18 (1.9)
No	915 (95.3)	Missing data	3 (0.3)
Yes	45 (4.7)	Prior femoral tunnel position
Missing data	0 (0)	Compromised (position)	546 (58.0)
ACL intra-articular reconstruction		Ideal	313 (33.2)
No	47 (4.9)	Compromised (position and size)	47 (5.0)
Yes	913 (95.1)	Compromised (size)	36 (3.8)
Missing data	0 (0)	Missing data	18 (1.9)
ACL extra-articular reconstruction		Prior tibial fixation
No	950 (99)	Interference screw	674 (70.6)
Yes	10 (1.0)	Other	106 (11.1)
Missing data	0 (0)	Combination	94 (9.8)
PCL intra-articular reconstruction		Suture and post	54 (5.7)
No	955 (99.5)	Intrafix	27 (2.8)
Yes	5 (0.5)	Missing data	5 (0.5)
Missing data	0 (0)	Prior tibial tunnel position
MCL repair/reconstruction		Ideal	581 (61.2)
No	944 (98.3)	Compromised (position)	258 (27.2)
Yes	16 (1.7)	Compromised (size)	86 (9.1)
Missing data	0 (0)	Compromised (position and size)	20 (2.1)
LCL repair/reconstruction	—	Missing data	11 (1.1)
Posterolateral corner reconstruction	—

^aData are originally from the MARS Group (2010). ²⁷ Variables and missing data are presented as n (%). Dashes indicate no procedures were performed. ACL, anterior cruciate ligament; BPTB, bone–patellar tendon–bone; LCL, lateral collateral ligament; MARS, Multicenter ACL Revision Study; MCL, medial collateral ligament; PCL, posterior cruciate ligament.

Appendix Table A3.

Preoperative Radiographic Measurements (N = 960 Patients) ^a

Preoperative Radiographic Measurement	Value
Sagittal view femoral tunnel position
Technique 1 b	0.39 (0.32-0.46)
Missing data	502 (52.3)
Technique 2 c	0.33 (0.27-0.39)
Missing data	502 (52.3)
Sagittal view tibial tunnel position d	0.33 (0.27-0.39)
Missing data	492 (51.3)
Sagittal view physiological tibial plateau slope angle	6.30 (4.30-8.80)
Missing data	483 (50.3)
Coronal AP view tibial tunnel position e	0.45 (0.43-0.48)
Missing data	484 (50.4)
Coronal AP view physiological femoral-tibial tunnel angle	15.80 (11.50-20.40)
Missing data	495 (51.6)

^aData are originally from the MARS Group (2013). ²⁹ Continuous variables are presented as median (IQR); missing data are presented as n (%). AP, anteroposterior; MARS, Multicenter ACL Revision Study.

^bRecorded as a percentage of the distance from the femoral tunnel location to the Blumensaat line.

^cMeasured similarly as above, as a percentage, from the femoral tunnel position to the cortex width.

^dExpressed as a percentage of the tibial tunnel (center to anterior) to the tibial plateau width.

^eExpressed as a percentage of the tibial tunnel (center to medial) to the tibial plateau width.

Appendix Table A4.

Surgical Information at Time of MARS Study Enrollment (N = 960 Patients) ^a

Variable	Value	Variable	Value
Time since last ACLR, y	3.70 (1.50-8.80)	Tibial fixation
Missing data	10 (1)	Interference screw	565 (59.0)
Surgeon years of experience	16.00 (8.00-20.00)	Combination	199 (20.8)
Missing data	0 (0)	Intrafix	90 (9.4)
No. of revisions		Suture and post or button	50 (5.2)
1	845 (88.0)	Other	54 (5.6)
2	99 (10.3)	Missing data	2 (0.2)
3	16 (1.7)	Medial meniscal pathology/treatment
Missing data	0 (0)	Normal	537 (55.9)
Surgeon’s opinion of failure		Excision	262 (27.3)
Traumatic	336 (35.3)	Repair	126 (13.1)
Combination	335 (35.2)	Other	35 (3.6)
Technical	206 (21.7)	Missing data	0 (0)
Biological	74 (7.8)	Lateral meniscal pathology/treatment
Missing data	9 (0.9)	Normal	628 (65.4)
Surgeon’s revision of their own failure		Excision	234 (24.4)
No	672 (70.4)	Other	48 (5.0)
Yes	283 (29.5)	Repair	50 (5.2)
Missing data	5 (0.5)	Missing data	0 (0)
Injury mechanism		LFC articular cartilage pathology
Nontraumatic, gradual onset	512 (53.3)	Grade 1 (normal)	692 (72.1)
Nontraumatic, sudden onset	257 (26.8)	Grade 2	141 (14.7)
Traumatic, noncontact	123 (12.8)	Grade 3	75 (7.8)
Traumatic, contact	66 (6.9)	Grade 3/4 or 4	52 (5.4)
Missing data	2 (0.2)	Missing data	0 (0)
Surgical exposure technique		MFC articular cartilage pathology
1-incision (AM portal)	430 (45.1)	Grade 1 (normal)	548 (57.1)
1-incision (transtibial)	339 (35.5)	Grade 2	223 (23.3)
2-incision	179 (18.8)	Grade 3	125 (13.0)
Arthrotomy/other	6 (0.6)	Grade 3/4 or 4	63 (6.6)
Missing data	6 (0.6)	Missing data	1 (0.1)
Graft type		LTP articular cartilage pathology
Autograft	474 (49.4)	Grade 1 (normal)	794 (82.7)
Allograft	456 (47.5)	Grade 2	128 (13.3)
Both autograft and allograft	29 (3.0)	Grade 3	33 (3.4)
Missing data	1 (0.1)	Grade 3/4 or 4	5 (0.5)
Graft source		Missing data	0 (0)
BPTB	485 (50.6)	MTP articular cartilage pathology
Soft tissue	444 (46.3)	Grade 1 (normal)	859 (89.9)
Other	29 (3.0)	Grade 2	67 (7.0)
Missing data	2 (0.2)	Grade 3	15 (1.6)
Femoral tunnel position		Grade 3/4 or 4	14 (1.5)
Entirely new tunnel	477 (49.8)	Missing data	5 (0.5)
Optimum position	259 (27.0)	Patellar articular cartilage pathology
Blended new tunnel	173 (18.1)	Grade 1 (normal)	682 (71.0)
Added second tunnel	29 (3.0)	Grade 2	176 (18.3)
Other	20 (2.1)	Grade 3	94 (9.8)
Missing data	2 (0.2)	Grade 3/4 or 4	8 (0.8)
Femoral fixation		Missing data	0 (0)
Interference screw	549 (57.4)	Trochlear articular cartilage pathology
Suture and Endobutton	204 (21.3)	Grade 1 (normal)	771 (80.3)
Cross pin	114 (11.9)	Grade 2	77 (8.0)
Combination	51 (5.3)	Grade 3	73 (7.6)
Other	39 (4.1)	Grade 3/4 or 4	39 (4.1)
Missing data	3 (0.3)	Missing data	0 (0)
Tibial tunnel position		Biological enhancement used
Optimum position	569 (59.4)	No	879 (91.9)
Blended new tunnel	180 (18.8)	Yes	77 (8.1)
Entirely new tunnel	162 (16.9)	Missing data	4 (0.4)
Added second tunnel	28 (2.9)
Same tunnel aperture but compromised position	19 (2.0)
Missing data	2 (0.2)

^aData are originally from the MARS Group (2010). ²⁷ Continuous variables are presented as median (interquartile range); categorical variables and missing data are presented as n (%). ACLR, anterior cruciate ligament reconstruction; AM, anteromedial; BPTB, bone–patellar tendon–bone; LFC, lateral femoral condyle; LTP, lateral tibial plateau; MARS, Multicenter ACL Revision Study; MFC, medial femoral condyle; MTP, medial tibial plateau.