Applying Machine Learning in Predicting Medication Treatment Outcomes for Opioid Use Disorder

Abstract

Background

While medication for opioid use disorder (MOUD) is effective for a significant proportion of patients, many return to using opioids during treatment. Understanding which factors lead to successful treatment informs the development of implementation approaches that can improve outcomes. This manuscript and its accompanying website provide an applied introduction to interpretable machine learning for clinical investigators interested in predicting treatment response for people using MOUD.

Methods

This study applied machine learning (ML) algorithms (K-Nearest Neighbors (KNN), logistic regression with and without regularization, Multivariate Additive Regression Splines (MARS), Support Vector Machines, Classification and Regression Trees (CART), Random Forest, Bayesian Additive Regression Trees (BART), Boosted Trees, Neural Networks) to predict failure of treatment in a collection of 2,478 individuals who had participated in the three largest pragmatic, clinical trials of MOUD.

Results

All models produced Receiver Operating Characteristic Area Under the Curve (ROC AUC) estimates in the range of 0.62 to 0.67 using cross-validation data and the optimal model, random forest, achieved 0.65 using testing data. The algorithms nearly universally identified predictive features such as age, intravenous drug use days, study medication, and study site. Most algorithms also identified various aspects of smoking. Only the algorithms that detect complex non-linear trends identified details from timeline follow-back. One algorithm, BART, performed well while devaluing all treatment-specific details.

Conclusion

After explaining how to apply, compare, and contrast various ML workflows, the results show that while overall modeling performance is similar across the models developed, the use of different algorithms identifies different sets of predictive features. Previous research has not recognized some features as important for predicting treatment outcomes. A companion website introduces clinical investigators to the concepts and implementations this study presents. That site also provides a detailed annotated blueprint to fully replicate, or even expand, this work.

1. Introduction

Opioid use disorder (OUD) contributes to significant morbidity and mortality (Centers for Disease Control and Prevention, 2024; Hser et al., 2017; Taylor & Samet, 2022). For myriad reasons, many individuals underutilize effective treatments. (Cantor et al., 2024). Further, while medication for opioid use disorder (MOUD) is effective for a significant proportion of patients, many return to using opioids during treatment. Understanding which factors lead to successful treatment, which is defined here by abstinence from nonprescribed opioids during a treatment or observation period, and beyond, informs the development of implementation approaches that can improve outcomes (Bell & Strang, 2020; Gustin et al., 2015; Hser et al., 1997; National Academies of Sciences, Engineering, and Medicine, 2019; Wakeman et al., 2020). A 2024 review by Mattos and colleagues describes the application of state-of-the-art predictive, machine learning (ML) models to quantify factors associated with MOUD adherence and return to opioid use (Mattos et al., 2024). While many of the papers which they review suffer from methodological flaws or use devices that are not feasible during routine care (i.e., fMRI and EEG analyses) the remaining demonstrate impressive predictive performance (Baucum et al., 2023; Burgess-Hull et al., 2022; Eddie et al., 2024). More recently, investigators presented one ML algorithm, logistic LASSO, as a possible system for predicting treatment success for individuals initiating MOUD (Luo et al., 2023). ML models can not only predict who will respond better to MOUD, but also uncover important and interpretable reasons for success or failure of treatment through the applications of a body of state-of-the-art techniques in explainable artificial intelligence (Biecek & Burzykowski, 2021; Molnar, 2022). While the recently published model by Luo et al. (2023) is conceptually simple and emphasizes ease of clinical application, a strong rationale supports describing more than a “best” possible predictive model. Computer scientists have shown that there is no best/optimal ML method. Furthermore, it may be that more complex models can clarify which predictors are the most important. Various ML algorithms highlight different patterns within patient behavior and treatment. Thus, examining multiple predictive models affords clinical insights beyond optimal prediction (Biecek & Burzykowski, 2021; Wolpert & Macready, 1997). That is, modeling methods use different kinds of information (i.e., linear, nonlinear, nonparametric trends) so they can utilize different kinds of complexity (i.e., traditional statistical interactions vs. the importance of different variables in subgroups). This paper expands on the ML method in Luo et al. (2023), explores the performance of nine algorithms useful to predict treatment success or failure for OUD with MOUD, and assesses data features for variable importance with the aim of detecting cross-methodological consistencies.

Traditional statistical modeling approaches to predicting treatment success or failure have focused on finding statistically significant variables that demonstrate associations with treatment outcomes. Researchers have built models that optimize the best possible estimates of the influence of predictors on the full set of data collected in a study sample. Typically, these effects are reported as beta parameter estimates or as odds or risk ratios with confidence limits. While ML practitioners care about these estimates for the predictor variables, they typically optimize their results based on the ability to make predictions in a new sample. While traditional statistical workflows involve variable transformations, ML workflows expand and systematize this variable “feature engineering” process. Data scientists build and optimize models on a subset of the data, called the training data, to give the best predictions on new data after the variables undergo extensive processing and conversion into “features.” That new data, called the testing subset, is a random sample of the original data, which the research team set aside at the start of the modeling process. The final review of model performance on testing data is critical because ML methods can “overfit” the data. That is, if researchers apply the predictions of a ML model to the original training data, these models will likely give overly optimistic estimates of performance because they will detect and use subtle patterns in the training data that will not replicate in a new sample. Many different metrics can define optimal performance, but popular options include accuracy of prediction or a combination of the true positive and false positive rates (i.e., area under a Receiver Operating Characteristic (ROC) curve, which is a plot that shows the true positive rate against the false positive rate across all classification thresholds) in a new sample (Bradley, 1997; Fawcett, 2006; James et al., 2021). To achieve optimal predictive performance, researchers try many different ML algorithms and select one as the winner. Then the modeler applies that “winning” algorithm to the “testing” results. This study does that, but also for illustration, it provides information showing how the models performed on the training data and demonstrates how the other models would have fared if they had been selected instead as the “winner”.

Beyond making predictions, many ML methods afford the opportunity to identify important variables that are missed using traditional statistical methods. Unlike traditional statistical methods that focus on model method-specific metrics to determine whether a variable is important (e.g., the size of odds or risk ratios and confidence intervals in logistic regression), the ML framework allows assessment of variable importance differently. Model-agnostic methods help determine whether a variable is important. Most ML methods allow calculation of permutation-based variable importance, which examines the degradation in model performance when the modeler uses an algorithm to shuffle the values of a variable and randomly assigns them to new study participants. If the accuracy or Receiver Operating Characteristic Area Under the Curve (ROC AUC) does not change when the modeler shuffles the values of a variable, then that variable was not important to the model. However, if the ROC AUC decreases, then the amount of change in the ROC AUC is indicative of the variable’s importance in the model. This paper provides information on the relative importance of the variables selected by most of the different ML engines and comments on the insights that can be gained by assessing the similarities and differences across the various machines.

This manuscript illustrates important ML tasks for clinical investigators working with MOUD. Given that most clinical investigators receive no formal academic training in the application of ML methods, there is also an extensive companion website (Balise, Grealis, Jean-Baptiste, et al., 2025) which provides a gentle introduction to the methods used, as well as the model tuning details. This supplement allows clinical investigators to i) explore the relative value of different techniques when predicting return to use of opioids (and potentially any number of other treatment outcomes of interest), and ii) uncover predictors identified consistently by different algorithms, as well as important predictors that can only be found using specific algorithms. Further, this manuscript includes a brief technical reference (Balise, Grealis, & Odom, 2025) that provides modeling details for ML practitioners. These tools build a computational framework for addiction researchers to examine and explain future predictive models in substance use disorder research beyond MOUD.

2. Methods

2.1. Subjects

The analyses included a total of 2,478 individuals, all who participated in one of three studies supported by the National Drug Abuse Treatment Clinical Trials Network (CTN) and who had any self-reported drug use information before and after randomization (Balise et al., 2023; Lee et al., 2018; Saxon et al., 2013; R. D. Weiss et al., 2011). These studies represent the three largest pragmatic, clinical trials to date of MOUD: Starting Treatment with Agonist Replacement Therapies (START, CTN-0027), Prescription Opioid Addiction Treatment Study (POATS, CTN-0030), and Extended-Release Naltrexone (XR-NTX) vs. Buprenorphine (BUP-NX) for Opioid Treatment (X:BOT, CTN-0051). These trials included treatments with methadone, buprenorphine, or extended-release injection naltrexone (XR-NTX). The technical reference (Balise, Grealis, & Odom, 2025), web supplement (Balise, Grealis, Jean-Baptiste, et al., 2025), and published materials (Balise et al., 2024) provide details on the study samples and the harmonization process for the data.

2.2. Predictors of Failure of Treatment

Previous research has suggested that Social Determinants of Health (SDOH), criminal involvement, reliance on public assistance, limited education, unemployment and homelessness, as well as clinical characteristics including baseline pain, comorbid psychiatric issues and the use of alcohol and other drugs influence the probability of treatment retention and—ultimately—success (Biondi et al., 2022; Compton III et al., 2003; Dole & Joseph, 1981; Hser et al., 1997; McLellan, 1983). Many of these features, along with information on treatment, exist in data sets from the CTN0094 data harmonization project. A software package for the R programming language, public.ctn0094data (Balise et al., 2023), provides this data. Research has identified many key features that predict success of treatment and public.ctn0094data includes them. Because study protocols did not gather all of these details, the software does not include a few important features, such as criminal involvement, previous treatment attempts (number, type, duration and outcomes), and total years of drug use, as well as details about access to non-study services and needs, particularly transportation, child care, vocational services, and housing services.

The analysis included a total of 46 variables, from domains including demographics, comorbid psychiatric/medical conditions, smoking history, lifetime drug use history, drug use as both yes/no indicators and amount in the month before treatment, and treatment details (inpatient vs. outpatient and treatment medication) as predictors (see technical reference Tables 1 and 2 and the web supplement Table 1 and Table 2) (Balise, Grealis, & Odom, 2025; Balise, Grealis, Jean-Baptiste, et al., 2025). The analysis evaluated several preprocessing recipes, as ML workflows customarily do (Kuhn & Silge, 2022). These preprocessing recipes were built to cover all the necessary steps (e.g., normalization, dummy coding etc.), for all the ML methods the study used (Kuhn & Silge, 2023). The analysis workflow also included other techniques to reduce the complexity of the predictors for comparison (i.e., including principal components analysis as a preprocessing step). The analysis applied various combinations of these steps before testing the ML methods. The web supplement shows that the various recipes resulted in models generated using between 15 and 103 individual features (engineered from the 46 variables). The analysis reported here used the recipe that produced the best ROC AUC on the training data. In order, those steps were: remove near-zero variance variables, convert text strings to categorical factors, impute missing values using a k-nearest neighbors algorithm, dummy-code categorical variables, collapse rare categories, remove highly correlated predictors and then normalize numeric variables.

Table 1:

Overall model performance measured using area under the ROC curve, when predicting failure of treatment in a training data set of 1,858 and a testing dataset of 620 people seeking care for Opioid Use Disorder.

Model	Cross-Validation	Full Training Dataset	Testing Dataset
Random Forest	0.6736	0.889	0.647
Boosted Trees	0.6720	0.839	0.645
BART	0.6673	0.748	0.623
LASSO	0.6658	0.720	0.617
Neural Net	0.6583	0.712	0.636
KNN	0.6478	0.931	0.630
Logistic LASSO	0.6476	0.742	0.618
Support Vector	0.6471	0.710	0.597
MARS	0.6409	0.715	0.615
CART	0.6227	0.670	0.574
Logistic	NA	0.744	0.626

Table 2:

Top 10 variables identified by permutation testing for predicting failure of treatment for 2,478 people seeking care for Opioid Use Disorder.

Importance	BART	Boosted Trees		CART	LASSO
1	Brain Damage DX	Age		Detox Days	Study Site
2	Marital Status	Study Medication		Study Medication	Age
3	Used Stimulants	IV Drug Days		Age	Study Medication
4	In/Outpatient	Study Site		IV Drug Days	IV Drug Days
5	Cocaine Days	Smoking Packs		Job	Study Number
6	Homeless	Detox Days		Study Site	Detox Days
7	Depression	Opioids Days		Cocaine Days	Fagerstrom
8	Schizophrenia	TLFB Days of Use		Smoking Packs	Alcohol DX
9	Used Cocaine	Fagerstrom		Alcohol DX	Used Opioids
10	Alcohol DX	TLFB Number of Drugs		Person Who Smokes	Job
x
Importance	Logistic	Logistic via LASSO	MARS	Neural Net	Random Forest
1	Study Site	Study Site	Detox Days	Study Site	Age
2	In/Outpatient	Age	Study Number	Fagerstrom	Smoking Packs
3	Age	In/Outpatient	Study Site	Study Medication	Study Site
4	Smoking Packs	Study Medication	Age	Age	Study Medication
5	Study Medication	IV Drug Days	Opioids Days	Used Heroin	Detox Days
6	Study Number	Fagerstrom	Study Medication	Job	Fagerstrom
7	Person Who Smokes	Job	IV Drug Days	Smoking Packs	IV Drug Days
8	IV Drug Days	Detox Days	Fagerstrom	IV Drug Days	TLFB Days of Use
9	Fagerstrom	Stimulant Days	Alcohol DX	Anxiety	TLFB Number of Drugs
10	Detox Days	Alcohol DX	Race	Withdrawal	Withdrawal

2.3. Outcome

Researchers have proposed many methods to evaluate the success or failure of treatment for OUD. A recent article by Brandt et al. demonstrated that the treatment evaluation metrics cluster into three different groups based on how they handle missing data and the length of the time frame used to judge success (Brandt et al., 2024). Here, the study uses one of the metrics from their so-called ‘Goldilocks’ cluster where the outcomes avoid being too lenient, like the metrics that ignore missing urine drug screening, and avoid being too strict, like the metrics that look at only a single or a couple of UDS values. Specifically, the analysis uses the definition of treatment failure that Lee et al. reported, which looks for four consecutive weeks with a UDS positive for opioids starting 21 days post-randomization (Lee et al., 2018). See the web supplement (Balise, Grealis, Jean-Baptiste, et al., 2025) for an example using this code.

While the outcome here followed an abstinence-based definition, the authors recognize the growing emphasis on alternative endpoints — such as reductions in use, craving, anxiety, and quality of life — that people with substance use disorders identify as meaningful. To support these perspectives, the research team has developed open-source tools (e.g., public.ctn0094extra (Odom & Balise, 2023) and CTNote (Odom et al., 2022)) that compute over 50 published endpoints, laying the groundwork for future analyses that move beyond abstinence to capture broader dimensions of treatment success.

2.4. Modeling

The research team conducted analyses with R version 4.5.2 using the tidyverse, rUM, table1 and flextable packages to preprocess and summarize data (Balise, Odom, et al., 2025; R Core Team, 2025; Rich, 2025; Wickham et al., 2019; Wickham, 2023). The analysis used the tidymodels ecosystem for modeling and DALEX and DALEXtra packages for interpretable ML processing (Biecek, 2018; Kuhn & Wickham, 2025; Maksymiuk et al., 2020). The study employed techniques that include regression-based (i.e., LASSO, logistic, MARS), tree-based (i.e., BART, Boosted Trees, CART, Random Forest) and neural network methods. Tree-based and neural network methods are particularly adept at detecting complex treatment and medication-specific effects that traditional regression approaches could miss. See the web supplement (Balise, Grealis, Jean-Baptiste, et al., 2025) for details.

Analysis code split the full analysis dataset using a stratification algorithm that ensured the training data (3/4 of the data, N = 1,858) and the testing data (1/4 the data, N = 620) maintained the percentage of people for whom treatment failed. Algorithms used five-fold cross-validation for model tuning. See the web supplement (Balise, Grealis, Jean-Baptiste, et al., 2025) for more details.

Given the historical popularity of ROC AUC, its relatively easy interpretation, and the fact that there was not a major imbalance in the proportion of people for whom treatment was unsuccessful as per the metric introduced above (72% with unsuccessful vs. 28% with successful treatment outcomes), the modelers tuned all models to find the optimal ROC AUC (G. M. Weiss, 2013). If you are not familiar with how to interpret a ROC AUC value, assume you have two people, for one of whom treatment will work and for one of whom it will not. The ROC AUC value gives the probability of correctly guessing for which one of whom the treatment will fail. While the definition of a clinically meaningful improvement in a model is largely arbitrary, a priori we defined a meaningful difference as a 5% improvement (e.g., 65% versus 70%), when comparing any two tuned models, in the probability of correctly guessing for whom treatment would fail. The researchers examined results from all models using permutation-based Variable Importance Plot (VIP) methods and confusion matrices (see Molnar (2022) and the web supplement (Balise, Grealis, Jean-Baptiste, et al., 2025)). The web supplement (Balise, Grealis, Jean-Baptiste, et al., 2025) also contains the details on model tuning, including a definition of the terms for those unfamiliar.

2.5. Evaluation Metrics

Consider that there are many possible algorithms that can be used to make predictions and only a single test dataset that can be used to estimate a model’s performance on new data. Evaluating the performance of every algorithm on the test data would likely yield an overly optimistic estimate of model performance. This would happen because with many different models, an analyst would eventually get “lucky” and obtain particularly good performance simply by chance. Because of this, the modeling community finds it customary and appropriate to optimize the results for each algorithm and then select a “winner” based on a metric like accuracy or ROC AUC using the training data alone. The analysis applies the algorithm that performs best on the preselected metric (e.g., ROC AUC) to the test data and reports the ROC AUC from the “testing” data as the estimate of future performance on unseen new data.

3. Results

Because the goal is to explain the ML workflow, this analysis includes a few extra steps. Table 1 presents the model performance, as assessed by ROC AUC, for each algorithm. Modelers typically use the “Cross-Validation” results to select the “winning” algorithm from the training data. It is an estimate based on repeatedly subsampling the training data. The web supplement (Balise, Grealis, Jean-Baptiste, et al., 2025) explains this process in detail. The top row of Table 1 shows that the best performing algorithm was the random forest algorithm. The “Full Training Dataset” column shows the result when the model is applied to the full training dataset to estimate future performance. Historically, researchers have refit models on the training data when using logistic models. Unsurprisingly, because the model is predicting the data it was built on, the Full Training Dataset results are optimistic. When the random forest model or KNN is applied to the full training data, the results show that the model performance is extremely good and overly optimistic because the model uses the idiosyncratic features in the data, as well as the true signal.

The random forest algorithm has the largest cross-validation ROC AUC (i.e., 0.67), so it could be called the “winner.” In typical real work applications, this would be the only model applied to the testing data. Here, the estimated cross-validation performance of 0.67 drops to 0.65 when that model is applied to the test data. The results from such an ML model would be that: using the “winning” random forest model and the data the study gathered before treatment, from a pair of people, for one of whom treatment will succeed while it fails for the other, there is about a 65% chance of correctly guessing for whom it will fail. While this is the best model, that value is not “good.” Many experts, including Hosmer et al. (2013), who say “there is no ‘magic’ number, only general guidelines”, consider a value of 0.7 “acceptable discrimination.” This suboptimal performance recapitulates what clinical investigators know: predicting failure of treatment, even when given all traditionally gathered variables, is extraordinarily difficult and the predictions should be treated with great caution.

The importance of using metrics such as ROC AUC to assess predictive performance is particularly notable when the researcher contrasts them with the insights provided by a traditional, p-value-based interpretation of a logistic regression model. The traditional logistic regression model identifies five features with p-value < .001 and twenty-two with p-values < .05. Clearly, there are many important features that drive the odds of failure of treatment, but the overall impact of these features is not large because that logistic model has a ROC AUC of “only” 0.63 when applied to the testing data.

While in real world applications the test data is only used once, here (again for illustrative purposes), this study applied each of the methods, with their optimal tuning, to the testing data. The comparison shows that the cross-validation estimates of future performance match nicely with the behavior on the testing data. In other words, the cross-validation methods allowed us to pick up on the true signal without “overfitting” the models (incorrectly focusing on random noise in the training data).

Figure 1 shows the true vs. predicted results for the people in the test data as a confusion matrix, and the predicted probability of failure for those who did and did not return to opioid use. Both the density and the confusion matrix show the relatively high chance of predicting return to use. Ideally, the probability plot would demonstrate a bathtub shape, where the people who experienced failure of treatments clustered close to the right side of the plot and the successes clustered close to the left. Notice that the model is rarely assigning a low probability of failure of treatment.

Figure 1:

Categorical predictions (using a threshold of 0.5) and predicted probabilities of failure of treatment for individuals in the testing dataset (N = 620) when applying the random forest model.

The ML methods above selected different numbers of features as being important for predicting failure of treatment. While the KNN algorithm uses, by design, every feature, and technical reasons make determining the number of features that other algorithms like neural networks and support vector machines use extremely difficult, there are noteworthy differences in the number of features the different algorithms use (MARS N = 15, CART N = 28, boosted trees N = 99, LASSO N = 96, logistic regression N = 99, random forest N = 102). At the extremes, the MARS model identified 15 distinct variables as important, and the random forest chose 102 different binary splits from 46 variables. For example, the random forest model made eight splits based on the scores of 1 through 8 on the Fagerstrom Test for Nicotine Dependence. Interestingly, across all the methods, the models identified a total of 105 different features as being important predictors at least once.

As the web supplement discusses (Balise, Grealis, Jean-Baptiste, et al., 2025), permutation-based VIP measures show the relative importance of different predictors. Figure 2 shows an estimate of how much excluding different variables would hurt our best performing model, a random forest. For example, the overall ROC AUC score would decrease by somewhere between 0.75% and 1% if the age variable was not included.

Figure 2:

Variable Importance Plot (VIP) for Random Forest Model.

Table 2 shows that the algorithms nearly universally selected predictors such as Age, IV drug use days, Study Medication and Study Site. Seven algorithms identified Detox Days and Fagerstom score as recurring predictors. While many of the most important predictors are categorical, the logistic regression, LASSO and MARS methods better detected linear trends for continuous variables. That is, they captured trends in age and the number of days using drugs in the 28 days before treatment as well as IV drug use days. Nearly every method identified smoking (i.e., person who smokes, number of packs smoked or the Fagerstrom score) as being related to failure of treatment. Interestingly, BART, which used many different predictors compared to the other methods, achieved similar model performance using many different variables. The interpretation of the impact of each predictor requires finesse beyond a simple statement about overall increased or decreased risk of failure (Biecek & Burzykowski, 2021).

4. Discussion

The goal of this manuscript, with its web supplement (Balise, Grealis, Jean-Baptiste, et al., 2025), was to provide a self-contained tutorial on modern ML methods and an example of how these methods can predict treatment response in people with OUD. Despite using these state-of-the-art methods, our results show that our predictions for treatment success or failure are far from ideal. Using these models and the data the study gathered before treatment, for a pair of people, for one of whom treatment will succeed while it will fail for the other, a clinician has about a 65% chance of guessing for which one it will fail. While the models afford similar overall performance, exploring and contrasting their findings yields insights.

Different modeling methods detected some common threads. It is unsurprising to see that all but one method selected IV drug use, a risk factor that research has tied to return to opioid use, as an important predictor (Chalana et al., 2016). The methods nearly universally identified polysubstance drug use, including the use of cigarettes, as predictors of failure. Several tree-based methods, which are ideal for detecting non-linear patterns, stressed the importance of length-of-stay in a detoxification unit before study drug administration. To better understand this impact, consider that our models also included details on the amount of illicit drug use before the intervention. The relatively complex and best performing models (i.e., random forests and boosted trees) identified days of drug use before care, as well as days in detox. These findings compliment other analyses that use latent class methods (a technique that does not integrate well with a ML framework), which have identified drug use trajectories in the month before treatment that are associated with success of treatment (Pan et al., 2022). Perhaps, drug use patterns before the start of the trial and time in detox are reflective of people’s ability to reduce drug use before MOUD initiation.

It is interesting that one of the models, BART, identified many variables as predictors that other methods did not identify. This phenomenon, where multiple models using different predictors perform comparably, is an example of the ML “Rashomon Effect” and it suggests the need for further work to explore whether modelers can stack several equally viable explanations of treatment outcomes using ensemble ML methods (Breiman, 1996; Paes et al., 2023; Rudin et al., 2024; Wolpert, 1992). While it will come at a cost of interpretability, the evidence strongly suggests that using ML techniques that combine algorithms will improve predictive power.

The near-universal identification of study site by these methods is noteworthy. As discussed elsewhere, the sites were aggregated to ensure anonymity (Balise et al., 2024). Thus, it is possible that the site effects are more important than this paper documents. While all subjects were participating in highly structured clinical trials, the study site was an extremely important prognosticator, indicating that the local context is important in predicting individual outcomes. However, it is unknown if this feature is capturing details within the clinic, such as the similarity in age, sex, race, and lived experience between the patients and care providers or general positive or negative attitudes/interactions or issues related to ease-of-access or the broader social milieu. In theory, researchers could add details on care providers to future models. While it is technically simple to integrate data from the census to describe the neighborhoods where clinics provided care, there are thorny privacy and ethical issues. One could use sensitive details to reverse engineer the individual clinics which, in turn, could lead to an increased risk of identifying individuals. Methods for dealing with these site-specific effects (which traditional statistical models would consider random effects) are only now becoming more relevant as interpretable ML methods like the VIP values become more popular. As ML methods move beyond their infancy, the authors expect developers will create new techniques to assess or remove the impact of clinic sites on predictions. However, rather than controlling for or removing those effects, we should learn what drives them.

While this study evaluated all methods using ROC AUC, it is important to remember that there are many metrics that researchers could use to evaluate a model (Opitz, 2024; Sokolova & Lapalme, 2009). The choice of metrics is an active area of research. Different metrics emphasize different aspects of model performance: Matthews Correlation Coefficient balances true/false positive and negative rates while accounting for class sizes, Log Loss penalizes incorrect predictions more heavily, Precision-Recall AUC focuses on the positive class, and Cohen’s Kappa emphasizes performance on difficult cases where chance guessing performs well. The web supplement describes and calculates several such metrics (Balise, Grealis, Jean-Baptiste, et al., 2025).

Depending on research and clinical goals, optimizing for sensitivity, specificity, or other metrics may be more valuable than ROC AUC. Beyond classification accuracy, it is important to examine the underlying probabilities that drive predictions. Rather than focusing solely on whether the model correctly predicted treatment failure, researchers should examine the predicted probabilities for individuals, as Figure 1B shows. These probabilities reveal whether a model is highly confident or uncertain in its estimates for specific individuals. In this example, the model was highly confident in predicting treatment failure for many people who actually succeeded in treatment (i.e., did not return to opioid use during treatment).

Furthermore, data which are not “missing at random” cause issues for all analysis methods. CTN-0027 and CTN-0030 measured the number of sexual partners to assess HIV risk, but CTN-0051 did not. CTN-0051 measured days in detox, but CTN-0027 and CTN-0030 did not. Consequently, these variables pose particular challenges for modelling. For example, the presence of a value on a variable could act as a simple proxy for membership in a particular trial. Unfortunately, the sexual partners variable caused the permutation-based VIP methods to fail, so this paper could not assess its relative importance. Because of these concerns, the authors conducted the ML method comparison using different subsets and imputation/simulation methods. These techniques, which adjust for structurally missing data, did diminish the relative importance of these predictors, but they remained influential.

Our models would certainly benefit from information that other studies have identified as relevant to the prediction of treatment success or failure (Biondi et al., 2022; Compton III et al., 2003; Dole & Joseph, 1981; Hser et al., 1997; McLellan et al., 1983; McLellan, 1983). This paper used all data available on tobacco and alcohol use but because the trials did not uniformly assess the data, it could not include as much detail on consumption as desired. Similarly, detailed information on the route of drug use (e.g., oral vs. nasal use of opioid pills) could not be used. The authors posit that contextual factors such as transportation barriers, access to childcare, etc. impact the probability of attending and ultimately succeeding in care. Further, information on previous treatment attempts would be valuable, and genetic, genomic, and other -omic data will likely prove valuable in predicting treatment response and matching people to the best possible medications.

In conclusion, ML methods are useful for making predictions of how individual patients will respond to treatment; however, current data provide only modest veracity in these predictions. The quality, breadth, and depth of the data collected in MOUD clinical trials limits the utility of current ML models. The authors hope that future studies will systematically and richly assess features related to the provider-patient relationship, provider beliefs about MOUD, patient attitudes towards MOUD, prior treatment attempts, socioeconomic and demographic characteristics, polysubstance use, risky sexual behaviors, and the many other potentially important features that the discussion and limitations section highlights. To this end, the code provided with the web supplement (Balise, Grealis, Jean-Baptiste, et al., 2025) allows anyone who is familiar with the R programming language to extend our work and explore other theories and predictive features. Such work will likely allow researchers to improve the quality of their predictions and identify missing key features for predicting treatment success.

Highlights.

• Overall model performance was similar for ML and traditional modeling methods

• Multiple algorithms identified consistent features predicting failure of treatment

• Behavioral features like smoking and IV drug use were important predictors

• Care features like study site and days in detox were important predictors

• Data and replicable analysis workflows with explanations are provided