Toward addiction prediction: An overview of cross-validated predictive modeling findings and considerations for future neuroimaging research

Abstract

Substance-use is a leading cause of disability and death worldwide. Despite the existence of evidence-based treatments, clinical outcomes are highly variable across individuals and relapse rates following treatment remain high. Within this context, methods to identify individuals at particular risk for unsuccessful treatment (i.e., limited within-treatment abstinence), or for relapse following treatment, are needed to improve outcomes. Cumulatively, the literature generally supports the hypothesis that individual differences in brain function and structure are linked to differences in treatment outcomes, although anatomical loci and directions of associations have differed across studies. However, this work has almost entirely used methods that may overfit the data leading to inflated effect size estimates and reduced likelihood of reproducibility in novel clinical sample. In contrast, cross-validated predictive modeling (i.e., machine learning) approaches are designed to overcome limitations of traditional approaches by focusing on individual differences and generalization to novel subjects (i.e., cross-validation), thereby increasing the likelihood of replication and potential translation to novel clinical settings. Here, we review recent studies using these approaches to generate brain-behavior models of treatment outcomes in addictions and provide recommendations for further work using these methods.

Introduction

Substance-use is a leading cause of disability and death (1). Within the United States, annual opioid-associated fatalities recently exceed those caused by firearms and motor vehicles, and those caused by HIV at the height of the AIDS Epidemic (2). Although less publicized, there has been a concurrent rise in cocaine- and stimulant-associated fatalities (3). Thus, improved strategies to combat the current substance-use epidemic are urgently needed (3, 4). While evidence-based treatments exist (5, 6), outcomes are variable across individuals and the majority of individuals experience multiple, unsuccessful treatment attempts. Relapse rates following treatment also remain high and, for some substances, this is a critical vulnerability period for overdose-associated death (7). Thus, methods to identify individuals at particular risk for unsuccessful treatment are needed to improve outcomes.

Despite a large degree of between-patient heterogeneity, individual differences in ‘traditional’ variables (e.g., severity) are typically not sufficient to account for differences in addiction outcomes (8, 9). Thus, more recent translational work has sought to identify neurobiological features that may be used to predict treatment responses (10). Cumulatively, this literature generally supports the hypothesis that individual differences in brain function and structure are linked to differences in clinical outcomes, although anatomical loci and directions of associations have differed somewhat across studies (8, 11, 12). However, this work has not typically used recommended strategies to minimize the risk of overfitting (e.g., cross-validation), leading to inflated effect size estimates and reduced likelihood of reproducibility in novel clinical samples (13–16). Here, we discuss how alternative, machine learning based approaches may generate more robust predictions, review prior findings using these approaches in addictions, and discuss addiction-specific considerations for adoption of these methods. As an exhaustive methodological tutorial is not possible here, popular approaches and key terms are defined in Table 1. For a practical guide on generating brain-behavior models, see (17) (also see (18) for sample size considerations).

Table 1 –

Definitions of popular algorithms, validation and accuracy terms

Classification approaches
Support Vector Classification	Discriminative classifier formally defined by a separating hyperplane
Logistic Regression	Classifier that transforms the output of regression using the logistic sigmoid function to return a probability value which can then be mapped to two or more discrete classes
Linear Discriminant Analysis	Classifier that uses a linear combination of features (reduced dimensions) which best separate different groups
Neighbors-based Classification	Type of instance-based learning that assigns a query point to the data class which has the most representatives within the nearest neighbors of the point
Decision Trees	Classifier in the form of a tree structure
Random Forest Classification	Ensemble learning classifier that constructs a multitude of decision trees
Naive Bayes Classifier	Generative learning model based on Bayes’ Theorem with an assumption of independence among predictors; may also be used for regression
Regression approaches
Ordinary Least Square Regression	Linear estimator that minimizes the sum of squares of the difference between the observed and predicted dependent variable
Support Vector Regression	Regression method formally defined by a separating hyperplane
Ridge Regression	Penalized regression method that uses l2-norm penalty to shrink coefficients towards zero
Lasso Regression	Penalized regression method that uses l1-norm penalty to shrink subset of coefficients to exactly zero
Elastic Net Regression	Penalized regression method that combines both l1 and l2 norm penalty
Random Forest Regression	Ensemble learning approach that constructs a multitude of decision trees
Principle Component Regression	Regression method that first uses PCA on the covariates and uses a subset of PCA components as regressors
Partial Least Squares	Regression method that behaves like principle component regression but also considers projected independent variables correlation with the dependent variable
Regression Trees	Regression method that uses decision trees to predict variables
Validation terms
External validation	Model generated in one sample is tested in a completely separate sample. ‘Gold standard’ for but often not practical in clinical research (e.g., due to need for multiple studies).
Internal validation	Approach in which a single sample is split into testing and training data e.g., K-fold CV.
Bias-variance tradeoff	Tradeoff between having sufficient to minimize bias in model generation (i.e., large K) and retaining sufficient data for testing to minimize variance in test data (i.e., small K).
Leave-one-out CV	Form of K-fold CV in which K=sample size. Useful for small samples but may overfit.
Split-half CV	Form of K-fold CV in which K=2. May be overly conservative.
5-fold CV	Form of K-fold CV in which K=5. Five or 10-fold CV is recommended for larger samples (e.g., n>200) to minimize bias-variance tradeoff.
Nested CV	Special case of internal CV in which data are divided in training, validation and test. Useful for model tuning (e.g., selection of free parameters).
Performance terms
Overall accuracy	Proportion of true positives and true negatives relative to actual cases and non-cases.
Sensitivity (also referred to as recall)	Proportion of true positives relative to cases; e.g. the number of tx completers correctly classified as completers divided by the number of total actual completers
Specificity	Proportion of true negatives relative to non-cases; e.g. number of non-completers correctly classified as non-completers divided by the number of total actual non-completers
Mean squared error (MSE)	Average squared difference between actual and predicted values; used for dimensional (continuous) prediction. Other metrics include r2 and q2 see (17) for more information

Prediction versus explanation

Machine learning—or cross-validated, predictive modeling—approaches are ideally suited for dealing with heterogeneous data and are designed to protect against overfitting via generation of a model in a training dataset and application of the model to a novel, unseen dataset (17, 19). By focusing on both out-of-sample generalizability and individual difference factors, machine learning approaches therefore offer a promising, pragmatic alternative to traditional statistical approaches—which generally aim to explain the variance between two or more variables, rather than to generate predictions in novel data (14).

Many different algorithms can be used to build predictive models, and these include approaches for modeling categorical data—i.e., classification—and for modeling continuous (dimensional) data—i.e., regression. While all approaches involve roughly the same steps, the ‘best’ algorithm depends on the clinical question and performance of different algorithms may vary significantly across and within datasets (18). In the context of addiction treatment, the goal of predictive modeling is typically to estimate (i.e., predict) an individual’s clinical outcome (e.g., abstinence during treatment) using data acquired at the start of treatment. To achieve this, a predictive model is generated using a training dataset and applied to an independent testing dataset (Figure 1) (17). While total independence of training and testing data (e.g., data from two separate clinical trials) is optimal, the practical limitations of clinical research often preclude the acquisition of multiple independent datasets. In the absence of multiple datasets, K-fold cross-validation is used to separate a single dataset into training and testing data: The dataset is randomly divided into K equally sized, non-overlapping subsets. K–1 subsets are iteratively assigned as training data, with the remaining subset reserved as testing data, such that each of the K subsets is used once as the testing data. Choice of K affects performance and depends on the amount of available data (schematic example in Figure 1; choices of K in Table 1).Model performance involves measuring the differences between observed (actual) and predicted (model generated) values and may be measured in several ways. Common metrics are defined in Table 1 and considerations for using these in studies of addictions are discussed in ‘Model assessment’. Below, we first review recent findings.

Figure 1 –

Example workflow for clinical predictive modeling in addiction

Summary of basic steps of predictive modeling (17), modified to emphasize specific aspects of predictive modeling in addiction: Neuroimaging data and clinical data are separated into independent training and testing datasets. Training data are then submitted to a predictive modeling algorithm to identify the most relevant features from the data (e.g., brain regions associated with the clinical outcome). Using the identified features, a mathematical model is generated to map the (typically high dimensional) neuroimaging features onto the (typically low dimensional) clinical outcome. The model is then applied to previously unseen data from the testing dataset to generate individual-level clinical predictions. Finally, the model’s performance is evaluated by comparing predicted and actual clinical outcomes in the testing dataset. Results are used to update current neurobiological models and to inform development of novel treatments.

Tx=treatment; ROI=region of interest; NOI-network of interest; LOOCV=leave-one-out cross validation; CV=cross validation

Prior work

Eight prior studies have combined cross-validated predictive modeling and neuroimaging methods to predict treatment outcomes in addictions (Table 2; see Supplementary Materials for manuscript identification).

Table 2 –

Summary of neuroimaging studies employing cross-validation to predict treatment response in addictions

Ref.	Drug	N	Mode	Input	Outcome	Type	K	Nested	Ext val.	Results
(20)	Poly	89	EEG	ERP PCA	TxC	C	LOO	Y	N	Sensory gating (P2) & post-error response strategy modulation (Pe) ERPs during go/no-go task performance predicted treatment completion. Model including only NI data outperformed models with clinical data.
(22)	Poly	123	EEG	ERP PCA	TxC	C	LOO	Y	N	N200 and P3a ERPs during visual distractor, visual oddball, and go/no-go tasks predicted treatment completion. For the oddball task only, models including only NI data outperformed models with clinical data.
(21)	Poly	139	fMRI	ICA FNC	TxC	C	10	Y	N	Corticolimbic between-network connectivity during go/no-go task performance predicted treatment completion. Model including only NI data outperformed model with only clinical data.
(25)	Coc	24	PET	ROI ΔBPND	TxR	C	10	N	N	ΔBPND [11C]raclopride in ventral striatum predicted treatment response (yes/no responder) with comparable accuracy to clinical data.
(9)	Coc	118	fMRI	WB FC (CPM)	TxR	R	LOO	N	Y	Cocaine abstinence predicted by increased FC between frontoparietal and medial frontal networks, by increased FC between salience, motor/sensory and subcortical networks, and by decreased FC between these two systems. Findings replicated an in external sample. Network strength assessed post-treatment also predicted cocaine use over 6-month FU. Model including only NI data outperformed model with only clinical data.
(28)	Stim	45	fMRI	ROI BOLD	Rel	C	10	N	N	Lifetime mania status + BOLD in rPCC, insula during selective attention task performance predicted relapse status at 6-month FU.
(30)	Stim	92	fMRI	ROI BOLD	Rel	C	n/a	n/a	Y	Neither binary logistic regression or random forest performed better than chance in external sample.
(29)	Alc	46	sMRI, fMRI	ROI BOLD & GMV	Rel	C	LOO	N	N	BOLD in VTA & VS to alcohol cues + volume in OFC & mPFC identified as most robust predictors of relapse status at 3-month FU. Multiple algorithms (Naïve Bayes, SVM) performed similarly. Models including only NI data outperformed model with only clinical data.

Ref=reference; Mode=modality; Ext val=external validation; Poly=polysubstance-use (i.e., mixed clinical sample); Coc=primary drug cocaine; Stim=primary drug stimulants; Alc=alcohol; EEG=electroencephalography; fMRI=functional magnetic resonance imaging; PET=positron emission tomography; sMRI=structural magnetic resonance imaging; ERP=event related potential; PCA=principal component analysis; ICA=independent component analysis; FNC=functional network connectivity; ROI=region of interest; ΔBP_ND=change in binding potential; WB FC=whole-brain functional connectivity; CPM=connectome-based predictive modeling; BOLD=blood-oxygen-level-dependent; GMV=grey matter volume; TxC=treatment completion; TxR=treatment response; Rel=relapse; C=classification; R=regression; LOO=leave-one-out cross validation; NI=neuroimaging; FU=follow-up; rPCC=right posterior cingulate cortex; VTA=ventral tegmental area; VS=ventral striatum; SVM=support vector machine.

Completion

Three studies have used a combined approach involving an initial principal or independent component analysis (P/ICA) of neuroimaging data followed by SVM classification to predict treatment completion (20–22). PCA and ICA are common data reduction methods for high dimensionality data that may be useful as an initial feature selection stage in predictive modeling. However, as inclusion of testing data in the data reduction stage may introduce bias (i.e., non-independence of testing and training data), P/ICA should be run only using the training data with the solution being applied to the testing data (for additional discussion of this, see (17)). Steele and colleagues used SVM with nested leave-one-out-cross-validation (LOOCV, definition in Table 1) to predict completion among incarcerated individuals (n=89) using PCA results of pretreatment electroencephalogram (EEG) data acquired during go/no-go task performance (20). Using a combination of P₂ (an evoked response potential (ERP) component associated with stimulus identification and sensory gating previously linked to abstinence (23)) and Pe (an error processing component associated with anterior cingulate activity (24)), SVM classified treatment completers versus non-completers with 79.6% overall accuracy (20). Models including only ERP data outperformed models including clinical data. In a follow-up study in a similar sample (n=123), and also using PCA results of EEG data, SVM with nested LOOCV of three separate tasks involving distractor stimuli achieved accuracies of 67-71% in predicting completion status (22). More recently, Steele and colleagues used SVM with nested k-fold to predict treatment completion (again, in an incarcerated sample of individuals with stimulant or opioid dependence; n=139) using a combination of ICA results of fMRI data acquired during go/no-go task performance (i.e., network connectivity values) and clinical data (21). SVMs of ICA alone and combined with clinical data had comparable accuracies (~81%) and outperformed an SVM of clinical data alone. Between-group comparisons (non-machine learning analysis) indicated primarily increased connectivity between corticolimbic networks among completers versus non-completers.

Response

Luo and colleagues tested the ability of amphetamine-induced change in D_2/3 receptor binding potential (ΔBP_ND) within the striatum, as assessed via positron emission tomography (PET) imaging with [¹¹C]raclopride, to predict contingency management treatment response for cocaine use (n=24) (25). Multiple SVMs were conducted to test the predictive ability of baseline (pre-amphetamine) BP_ND and ΔBP_ND across striatal regions-of-interest (ROIs) alone and in combination with clinical variables. The SVM including only ventral striatal ΔBP_ND had the highest predictive accuracy (82%) and this was not improved by incorporation of baseline clinical variables. Incorporation of within-treatment behavioral data (clinic attendance) was also a significant predictor of treatment response, both alone and in conjunction with ventral striatal ΔBP_ND (25).

Yip and colleagues used connectome-based predictive modeling (CPM)—a data-driven, whole-brain regression approach (26, 27)—to identify neural networks associated with treatment response (9). CPM with LOOCV (n=53) identified a distributed network that predicted within-treatment abstinence (percent of cocaine-free urines during treatment), as indicated be a significant correspondence between actual and predicted values (9). Clinical variables alone were not sufficient to predict treatment response in this sample. Connectivity within the identified network also significantly predicted within-treatment cocaine abstinence in an independent test sample (i.e., external validation) in both a continuous and categorical (yes/no treatment responder) manner (n=45). Predictive accuracy of cocaine network connectivity alone was 64% in the independent sample and this increased to 71% when combined with past-month cocaine use. Network connectivity assessed at post-treatment in the original sample also significantly predicted cocaine use following treatment. The identified network was complex and incorporated multiple canonical networks, including those previously implicated in addiction response (details in Table 2).

Relapse

Clark and colleagues used several different algorithms to predict relapse status among recently abstinent individuals with stimulan dependence recruited from a range of treatment settings (n=45) (28). Between-group differences (relapsers versus non-relapsers) were first identified using whole-brain comparisons and the predictive ability of blood-oxygen-level-dependent (BOLD) response within identified regions was then assessed using models with 10-fold CV. Across a variety of modeling approaches, BOLD response accurately predicted relapse status with approximately 80-84% accuracy. However, as identification of features (brain regions showing between-group differences) in this case was not done independently for training and testing data, which may result in over-fitting (17), further work using an independent testing data set is needed for cross-validation and model evaluation.

Seo and colleagues tested the ability of several algorithms to predict relapse status among recently detoxified individuals with alcohol use disorder (n=46) (29). Using a combination of structural and functional (BOLD response during an alcohol cue task) data, Naïve Bayes, SVM and robust soft learning vector quantization approaches all predicted relapse at greater than chance levels (72-79% accuracies) and with higher accuracies than models including only clinical data. Follow-up analyses to determine the relative contributions of each ROI and modality (structural vs. functional) identified BOLD response in the ventral striatum, and ventral tegmental area and orbitofrontal cortex and medial PFC volumes as the most robust predictors.

Gowin and colleagues generated binary logistic regression and random forest classification models using a combination of clinical and functional (BOLD response during a reward task) data in a training set of 63 individuals with methamphetamine dependence and applied this to an external testing set of 29 individuals with cocaine dependence (30). The logistic regression model had higher accuracy for predicting relapse status than the random forest model in the training dataset, but neither model performed above chance in the testing dataset.

Summary

In most cases, brain-based variables had comparable or higher predictive accuracies relative to traditional clinical variables, indicating that inter-individual heterogeneity in brain-based features meaningfully contributes to outcomes in addiction. However, all studies to date have had relatively modest sample sizes (18) and only two studies have included external validation (9, 30). Thus, significant additional work is needed prior to clinical adoption of these methods. In addition, most prior studies have used methods requiring a priori specification of regions or networks, limiting neurobiological discovery. Thus, future studies should consider adoption of more wholly data-driven methods of prediction, which, in addition to generating behavioral predictions, are also methods of identifying novel biological targets.. Such systems-level analyses are particularly well-suited for assessment of complex clinical phenomena (e.g., abstinence, relapse), which likely involve spatially distinct—yet functionally coherent—brain regions (9, 27, 31). Using this approach, Yip and colleagues identified a distributed neural network that predicted cocaine abstinence, such that abstinence was predicted by (i) increased connectivity between frontoparietal and medial frontal networks, (ii) increased connectivity between salience, motor/sensory and subcortical networks, and (iii) decreased connectivity between these two systems (9). These findings are consistent with those from ROI-based studies implicating cognitive control related neural circuitry (e.g., anterior cingulate) in treatment completion (20–22), and with ligand-based data indicating dopamine involvement in predicting treatment response (25), but also indicate involvement of more distributed connections than previously hypothesized (32, 33).

Limitations

Several limitations should be noted. Even cross-validated predictive models will likely involve some degree of over-fitting, resulting in erroneous selection of a least some predictors and correspondingly unstable results (34). In addition, even highly reliable predictive models are often nonetheless difficult to interpret. Practically, this means that the model selection process may require a trade-off between accuracy and interpretability (i.e., prediction versus explanation) (19). Further, predicted values are not always readily interpretable within—or easily transferable to—a real-world setting. In other words, in the absence of real-world knowledge about the target variable, a ‘good’ model may still provide an impossible prediction (e.g., greater than 100% abstinence). Below, we suggest methods for dealing with these limitations and review additional neurobiological, clinical and practical considerations to guide future research.

Neurobiological considerations

Timing matters

The time course of recovery from addiction can take months to years and is most often highly non-linear (35, 36). Thus, the timing of neuroimaging should be considered carefully. For studies predicting treatment response, scanning prior to treatment initiation is often conducted. However, scanning during this time may introduce additional confounds (e.g., acute intoxication or withdrawal effects) and may not be feasible unless treatment initiation is delayed (which may place unnecessary burden on the patient). Thus, an alternative approach is to conduct scanning early on in the course of treatment (e.g., following detox but prior to sustained intervention). While this approach necessarily precludes identification of true ‘pre-treatment’ factors, it may nonetheless be useful in identifying early markers of clinical response.

In many instances, brain regions or networks identified as predictive of treatment response may be distinct from those that typically distinguish patients from controls (9, 37). Although perhaps counter-intuitive, it is important to consider that, clinically, factors that predict treatment response (e.g., willingness to change) may be distinct from those that systematically differ between patients and controls (e.g., impulsivity). Similarly, predictors of treatment response may also be distinct from those that change with treatment (e.g., acquisition of new skills), or that predict relapse following treatment (e.g., non-clinical environment) (9). Therefore, longitudinal scanning over multiple time points will likely be needed to elucidate the full time course of addiction recovery.

Elucidation as a goal of clinical prediction

Recovery from addiction involves complex interactions across psychosocial and biobehavioral domains (38). Within this context, elucidation of biological mechanisms of successful behavior change may be used to improve current treatments and/or develop novel interventions (10). However, interpretability is one of the primary challenges of predictive modeling in general and of brain-based clinical models in particular. Many approaches that yield robust predictions may nonetheless do little to advance our understanding of the underlying neurobiology of addictions. Thus, in addition to generating predictions in novel data, an important aim of clinical predictive modeling studies should be to advance neurobiological understanding. As different modeling approaches have different interpretability challenges, we here provide recommendations for enhancing interpretability of findings from predictive models generated using ROI- and whole-brain approaches (see Figure 2 for schematic examples), in order to maximize neurobiological insight.

Figure 2 –

Steps to maximize neurobiological interpretation of clinical neuroimaging models

To maximize interpretability, model features should be considered across multiple levels.

Left: Model features are summarized across descending levels of dimensionality. Individual connections (edge level) are summarized by: (i) overlap with macroscale brain regions (node level), (ii) overlap with canonical neural networks (network level) and (iii) a simplified network model of core systems contributing to cocaine abstinence (theory level).

Right: Recommended steps for maximizing interpretation of neurobiological models for region-of-interest (ROI, top) and whole-brain (connectome-based, bottom) approaches are summarized. In both cases, post-hoc analyses involving ‘virtual lesioning’ of selected features is strongly recommended to guide interpretation of model elements.

Elements of this figure adapted from (9) and are reprinted with permission from the American Journal of Psychiatry, (Copyright ©2019). American Psychiatric Association. All Rights Reserved.

Many popular predictive modeling studies use some degree of a priori specification of ROIs or networks. While this approach can be a powerful tool for testing region- or network-specific hypotheses, it should be noted that the interpretability of these models becomes very difficult when multiple predictors are included in the model. For most algorithms, simply assuming that the largest weightings are the most significant features can lead to misinterpretations as weights need to be standardized from comparison (like any regression coefficient) (39). As standardization can be difficult with cross-validation, follow-up testing to determine the significance of different features is strongly recommended when using these approaches. One simple method for determining significance of different features (e.g., individual ROIs or networks) is to rerun the model excluding specific features (‘virtual lesion’ approach) in order to determine which features are ‘necessary’ for optimal model performance (e.g., does amygdala volume contribute to overall model performance?). Similarly, rerunning the model only including selected features will enable determination of the relative weight of specific features (e.g., what is the predictive ability of amygdala volume alone?).

In contrast, data-driven, whole-brain predictive modeling studies do not require a priori specification of seed regions or ROIs and may allow for one-to-one mapping of predictors back to brain anatomy (which significantly aids interpretability). However, networks identified using these approaches are typically complex (i.e., spanning multiple adjacent and non adjacent brain regions) and may not be readily interpretable. Thus, descriptive data reduction techniques (e.g., summary of networks via overlap with ‘canonical’ networks) are recommended when using these approaches (26, 40). As above, virtual lesion approaches (e.g., exclusion of connections based on overlap with specific brain regions or networks) may be used to characterize contributions of specific features.

Clinical considerations

Defining ‘treatment response’

Despite evidence-based treatments (41, 42), there remains a lack of consensus regarding the optimal measure to define treatment efficacy in addictions (43). Thus, an additional primary consideration is the selection, definition, and measurement of ‘response’. While abstinence is generally accepted by clinical researchers and practitioners as a meaningful, and preferred, outcome, significant variability in its definition and assessment exists. For example, differences in the timing (e.g., first versus last week of treatment) and duration of abstinence can yield different results (i.e., identify different individuals as ‘treatment responders’) (44). This not only makes cross-study comparisons of treatment efficacy difficult, but also presents challenges for the generalizability of predictive models, as there may be study-specific measurement of abstinence. As with all other aspects of predictive modeling, the method of defining abstinence or ‘treatment response’ should be defined ahead of time to prevent ‘p-hacking’. Multiple comparison correction must be used when more than one outcome is tested (17). Below we discuss additional considerations for measurement of treatment outcomes (see Table 3 for a summary of primary outcomes).

Table 3 –

Overview of primary abstinence and non-abstinence based treatment outcomes

	Approach	Measurement considerations	Pros	Cons
Biological	Utox	Detection times vary; Quantitative testing needed to reduce carry-over effects	primary method of biological verification; accurate and reliable; low-cost; on-site testing	poor temporal specificity; potential for adulteration
	Blood	typically used for verification rather than screening; relatively short detection window (~24 hours)	highly accurate; reduced risk of adulteration	invasive; high cost; does not provide immediate results; requires medically trained collectors
	Saliva (oral fluid)	short detection time; indicative of more recent drug use/abstinence	non-invasive, rapid, easily-observed collection; on-site collection and screening; lower biohazard risk; ability to collect multiple samples	difficult/unpleasant to obtain sufficient saliva; sensitivity and specificity mixed; drug concentration may be lower than urine
	Breathalyzer	verification of alcohol abstinence in short-term (past 6-12 hours)	non-invasive, rapid, easily-observed collection; on-site collection and screening; lower biohazard risk; ability to collect multiple samples	limited to alcohol testing; may be challenging for those with asthma or lung disease
	Carbon monoxide (CO)	cutoff may vary for distinguishing smokers from non-smokers depending on whether sensitivity or specificity is prioritized	immediate, non-invasive, and portable assessment of smoking status	may be affected by exposure to environmental tobacco smoke or pollutants; limited sensitivity to detect brief smoking lapses
Self-report	Timeline follow-back (TFLB)	calendar-based method; more reliable when biological specimens also collected	low-cost; ability to calculate multiple outcome measures for flexible intervals (7-day, past-month); retrospective reporting minimizes missing data	potential for under-reporting substance-use; reliability of retrospective self-report has been questioned
	EMA / Daily diary	recording at specified time intervals, signal-contingent, or event-contingent	high ecological validity; reduces reliance on memory; may be more sensitive to change	participant burden; potentially high rates of missing data
Non-abstinence based outcomes	Days in tx	can be defined in multiple ways depending on the type of tx	can be verified through medical records; tx retention has been linked to better outcomes;	indicator of tx acceptability rather than tx response; challenging to determine when tx drop out occurred
	Medication adherence	includes strategies for verification (e.g., tracer, MEMS caps)	increases internal validity of ‘tx response’ outcome	no standards for defining compliance cutoff
	Reduction in frequency/severity	measure of reduction dependent on baseline timeframe	practical and consistent with chronic nature of addiction; may be more sensitive than abstinence	clinical significance of reduction-based measures not established
	Improvement in functioning/quality of life	based on self-report; consideration of whether functioning/quality of life is direct result of drug use	well-established assessment measures for quality of life; clinically meaningful	may not be sensitive to change in drug use; dependent on baseline timeframe
	Diagnostic Threshold	may be measured through interview-based assessment or self-report; DSM-5 provides severity indicators	direct measure of substance-use disorder criteria	may not be sensitive to change in short-term

Utox=urine toxicology; Tx=treatment; EMA=ecological momentary assessment; MEMS=medication event monitoring system; based on (43, 45, 46, 49, 76–79).

Biological versus self-report measurement

The existence of reliable, biological indicators for detecting substance-use is a considerable advantage compared to other psychiatric disorders; yet significant challenges and limitations exist (43, 45, 46). For instance, urine testing—the most common biological method for monitoring drug use/abstinence—has a range of detection times for the presence of different substances and overall relatively low temporal resolution. For most drugs of abuse, the urine detection window is 2-5 days (47–50), which can result in positive tests for an extended period of time depending on the chronicity of use (51). Thus, infrequent urine specimen collection (e.g., once per week) may fail to detect substance-use leading to inflated measures of abstinence, whereas overly frequent collection (e.g., thrice per week) makes detection of abstinence challenging due to carry-over effects/residual positive tests (52, 53). Despite these limitations, Utox (or other biological) testing remains the sole method of empirical assessment of abstinence.

Despite several advantages of self-report methods for measuring abstinence (e.g., flexibility in collection, minimization of missing data), the accuracy of self-reported indices of use in clinical trials remains controversial. While some studies indicate excellent reliability, validity, and sensitivity of self-report methods (54–57), others indicate substantial under-reporting of substance-use through self-report relative to urine testing (58, 59). Collection of both self-report and toxicology data are recommended for clinical trials to provide the most accurate measures of substance-use (43). Yet, strategies for combining these two sources of data, are complex and not always practical (60).

Beyond abstinence

Given the chronic relapsing nature of addiction (35, 36), defining the efficacy of a given treatment solely on the achievement of a sustained period of abstinence may be overly restrictive. Thus, other outcome measures beyond complete abstinence have received attention as potential indicators of treatment response. A recent review of candidate drug use outcome indicators recommended commonly reported measures, such as the percentage of days abstinent and the percentage of negative urine specimens, as well as dichotomous indicators of abstinence at the end of treatment; see (46). In the alcohol field, the absence of ‘heavy drinking days’ is considered a clinically meaningful outcome measure and is accepted by regulatory agencies for demonstration of pharmacotherapy efficacy in Phase 3 trials (61, 62). Similarly, measures of reduction in alcohol consumption, such as reductions in drinking risk level, based on the World Health Organization risk level (expressed as grams of pure ethanol per day) (63), have been accepted by regulatory agencies (64). Unfortunately there is no such acceptable reduction-based equivalent for other drugs of abuse (e.g., cocaine, opioids, cannabis), in part due to the lack of standard units for measuring size and purity of illicit drugs. However, measures based solely on drug consumption may not represent all the significant physical and psychosocial consequences that characterize the disorder (65). Thus, there is potential relevance in evaluating new endpoints that address the fuller range of the symptoms of addiction to better define treatment response (66, 67). While clinically meaningful, the biological substrates of non-abstinence-based outcomes may be partly or wholly distinct from those of abstinence, per se. Thus, different predictive models will likely be needed to characterize different clinical outcomes.

Practical considerations

Model assessment

Performance of categorical models is most commonly quantified by overall accuracy, however more nuanced measures, such as sensitivity and specificity, may also be needed for evaluation of clinical models (68). The ‘best’ metric for characterizing model performance depends on the question (and modeling approach) (17, 68). For example, as illustrated in Figure 3, assuming that the overall goal of addiction treatment is (at a minimum) harm reduction, a model with high sensitivity but low specificity might be acceptable for prediction of medication treatment initiation but might not be optimal for predicting termination of treatment. Similarly, a model with low sensitivity but high specificity might be acceptable for prediction of treatment cessation but not for prediction of treatment initiation. Other measures, such as balanced accuracy (which captures both sensitivity and specificity and is distinct from overall accuracy), should also be considered for categorical models. For dimensional models, other metrics including rank-based measures and mean-squared error should be considered. See (68) and (17) for consideration of additional metrics.

Figure 3 –

Sensitivity vs. specificity within the context of clinical addiction prediction

The relative importance of sensitivity versus specificity may depend on the clinical outcome. Confusion matrices showing rates of true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN) are shown for models (1) with high sensitivity and low specificity (left) and; (2) with low sensitivity and high specificity (right) are illustrated for two different clinical outcomes: (1) assignment to active treatment (top) and; (2) termination of active treatment (bottom). Cells in red bold font correspond to individuals at increased risk for overdose. For outcome 1, overdose risk is minimized when sensitivity is maximized. For outcome 2, overdose risk is minimized when specificity is maximized.

Assessment of model performance should also be considered within the clinical context. If no reliable predictor of treatment outcome exists, then a model performing only marginally better than chance nonetheless represents an improvement. However, note that a statistical improvement over chance does not automatically constitute clinical utility; in all cases the relative clinical utility of a model should be weighed within the context of the added expense and potential cost-benefit to the patient (also see ‘Financial Considerations’). Regardless of the specific metric, model performance should be quantified statistically and this should be done using permutation testing when using cross-validation (17, 69). Note that an iterative cross-validation approach—in which the data is repeatedly randomly split into testing and training data and model performance is averaged across iterations—is recommended to reduce over-fitting (17, 34) and may also be used to assess reliability of specific features (i.e., identify unstable predictors) (70).

Financial considerations

In order for brain-based predictive modeling to reach its full potential, it needs to be adopted in clinical practice and used to improve clinical care. However, the cost of a single fMRI scan is often thought of as prohibitive in routine clinical practice. Thus, it may be useful to weigh this against the cost of an ‘unsuccessful’ addiction treatment episode. Total spending on substance-use disorder treatment in the United States is predicted to grow from $24 billion in 2009 to $42.1 billion in 2020, with the vast majority financed by public sources, including Medicare/Medicaid, federal, state, and local governments (71). Based on information from 110 substance abuse treatment programs in the US (72), estimates of “cost bands” (IQR, 25^th–75^th percentile) indicate the following average per treatment episode costs per patient: non-methadone outpatient=$1,132-$2,099; methadone=$4,277-$13,395; intensive outpatient=$1,384-$5,780. Critical determinants in the variation of costs are the type of modality and the mean length of stay in treatment (72), such that more intensive programs with longer treatment duration are more costly. This suggests that strategies to increase the likelihood of a positive treatment response (e.g., a priori identification of the optimal treatment modality to achieve the fastest response) could have a substantial cost saving effect. However, despite potential financial feasibility, brain-based predictive modeling of addiction outcomes remains a nascent area of investigation and significantly more work is needed prior to clinical translation of existing models, including rigorous comparisons of imaging versus non-imaging based predictive models.

Conclusions

Recent work demonstrate the ability of cross-validated predictive modeling approaches to generate individual-level predictions of complex addiction outcomes (e.g., abstinence, relapse) and to provide novel neurobiological insight into the brain basis of such behaviors. However, significant limitations remain, including limited external validation and lack of consideration of treatment-specific factors. In addition, work to determine who will respond preferentially to a given treatment (e.g., methadone versus buprenorphine) is urgently needed, as are studies to compare predictive markers across addiction subtypes. As both neurobiological differences (73) and differences in behavioral and psychological precipitants of different substance-use behaviors have been noted (74, 75), underlying predictive features may differ across substances, thus yielding different models.