Identifying optimal level-of-care placement decisions for adolescent substance use treatment

Abstract

Background:

Adolescents respond differentially to substance use treatment based on their individual needs and goals. Providers may benefit from guidance (via decision rules) for personalizing aspects of treatment, such as level-of-care (LOC) placements, like choosing between outpatient or inpatient care. The field lacks an empirically-supported foundation to inform the development of an adaptive LOC-placement protocol. This work begins to build the evidence base for adaptive protocols by estimating them from a large observational dataset.

Methods:

We estimated two-stage LOC-placement protocols adapted to individual adolescent characteristics collected from the Global Appraisal of Individual Needs assessment tool (n=10,131 adolescents). We used a modified version of Q-learning, a regression-based method for estimating personalized treatment rules over time, to estimate four protocols, each targeting a potentially distinct treatment goal: one primary outcome (a composite of ten positive treatment outcomes) and three secondary (substance frequency, substance problems, and emotional problems). We compared the adaptive protocols to non-adaptive protocols using an independent dataset.

Results:

Intensive outpatient was recommended for all adolescents at intake for the primary outcome, while low-risk adolescents were recommended for no further treatment at followup while higher-risk patients were recommended to inpatient. Our adaptive protocols outperformed static protocols by an average of 0.4 standard deviations (95% confidence interval 0.2-0.6) of the primary outcome.

Conclusions:

Adaptive protocols provide a simple one-to-one guide between adolescents’ needs and recommended treatment which can be used as decision support for clinicians making LOC-placement decisions.

1. Introduction

Adolescent substance use continues to be a major public health concern in the United States (Johnston et al., 2019), as nearly one million adolescents in the country meet criteria for a substance use disorder (SUD) (Substance Abuse and Mental Health Services Administration, 2018). SUDs during adolescence can have immediate and long-term effects, such as poor school performance, school drop-out and delinquency, health problems, and future substance use problems (D’Amico et al., 2005; Kann et al., 2018; Meier et al., 2016; Tucker et al., 2005). SUD also contributes to the leading causes of adolescent morbidity and mortality, such as motor vehicle accidents, unintentional injuries, and suicide (Das et al., 2016; Nora and Volkow, 2014; Tanner-Smith et al., 2013). Nonetheless, recent reviews suggest that standard outpatient, intensive outpatient, and residential care treatment services for adolescents can decrease substance use and improve quality of life (Das et al., 2016; National Institute on Drug Abuse, 2014; Tanner-Smith et al., 2013). Thus, there are a number of effective treatment options potentially available to meet each adolescent’s SUD treatment needs and goals (Winters et al., 2014b). Moreover, numerous measurement instruments exist to assess SUD, consequences of use, important co-occurring issues, and overall treatment needs (Winters et al., 2014a). These measurement instruments have been combined into multi-dimensional, comprehensive assessment procedures to match each adolescent with the treatment approach, modality, and level-of-care (LOC) most appropriate for addressing their individual needs and achieving their treatment goals (Fishman, 2014).

A major challenge to treatment planning and patient placement is using empirical information at treatment intake—and throughout treatment as an individual adolescent’s needs change—to guide initial and ongoing decisions about how best to intervene, with what modalities, and at what LOC. The development of individually-tailored treatment decisions over time—also known as adaptive interventions (AI)—can help address the challenge of using evidence-based, sequential decision-making (Wallace and Moodie, 2014). An AI is a preplanned sequence of decision rules that guides whether, how, and when to make critical treatment decisions, and (importantly) on which measures to base those decisions (Almirall and Chronis-Tuscano, 2016). Als have the potential to move from “one-size-fits-all” treatment strategies to approaches that are responsive to adolescents’ diverse needs at treatment onset and heterogeneity in response/adherence to different treatments (National Institute on Drug Abuse, 2014). In addition, developing evidence-based Als also may potentially resolve observed patterns of effective short-term interventions with dissipating long-term effects and ineffective short-term interventions laying a foundation for longer-term treatment success (Dennis et al., 2004; Hser et al., 2007; Williams and Chang, 2000; Winters et al., 2009). However, research on AIs for SUDs has focused on adult populations (Nahum Shani et al., 2017), and the current research foundations for leading placement criteria are based on expert opinion and data from adult SUDs (Baker and Gastfriend, 2004; Gastfriend and Mee-Lee, 2004; Levine et al., 2004; Staines et al., 2004). To further improve the utility and effectiveness of AIs for adolescent SUD treatment, additional empirical research on adolescent populations is needed.

This work begins to build the evidence base for developing effective two-stage AIs or adaptive LOC-placement protocols for recommending adolescent SUD treatment at intake and at 3 months post-intake, a meaningful progression point. We estimated adaptive protocols from a large observational dataset for a composite measure of positive treatment outcomes at 12 months post-intake. To demonstrate that treatment protocols might vary when targeting different goals for treatment (e.g., reducing substance frequency versus reducing emotional problems), we also estimated protocols for three stakeholder-identified outcomes: substance use frequency, substance use problems, and emotional problems. We aimed to: (a) better understand which patient characteristics are most useful; (b) obtain an estimate of the most effective adaptive LOC-placement protocol (for each outcome); and (c) compare the relative effectiveness of the adaptive protocols versus static protocols not tailored to adolescents’ needs (e.g., provide all adolescents outpatient for 6 months). This study provides the first evidence of whether adaptive LOC-placement protocols for adolescent SUD treatment can be feasibly estimated from observational data and whether adaptive protocols can lead to better outcomes than static protocols.

2. Material and methods

2.1. Data

This study utilized longitudinal observational data from adolescents receiving SUD treatment, who were administered the Global Appraisal of Individual Needs biopsychosocial assessment instrument (GAIN) (Dennis et al., 2003). The GAIN was routinely collected by 178 adolescent SUD treatment sites funded by the Center for Substance Abuse Treatment between 1997 and 2012. Adolescents receiving services were interviewed up to four times over the course of 12 months using the GAIN: namely, at intake and at 3-, 6-, and 12-months post-intake. Background and demographic characteristics were collected from each youth. The GAIN includes items along the following seven dimensions/problem areas: substance use, physical health, mental health, risk behaviors, environmental, legal, and educational/vocational. Within each problem area, items assessed problem characteristics, including recency, severity, and service utilization. The LOC received since the last assessment was also ascertained.

2.2. Inclusion and exclusion criteria

Our sample included a total of 14,174 youth aged 12-25. We retained 10,131 youth who reported receiving some form of treatment between intake and the 3-month assessment because we presume that, by design, LOC-placement protocols should recommend some intake LOC. We use “adolescents” throughout, though our sample included some transitional-age patients (aged 19-25).

2.3. Measures

Four categories of variables were relevant to our LOC-placement-protocol development: LOCs, treatment outcomes, tailoring variables (characteristics used to personalize treatment), and control variables.

2.3.1. LOCs

The LOCs included in the data are, from least to most intensive: (1) outpatient (OP), defined as having been admitted to a regular (1-8 hours per week) outpatient program for alcohol or other drug use problems (2) intensive outpatient (IOP), defined as having been admitted to more than 8 hours of outpatient programs for alcohol or other drug use problems; and (3) inpatient/residential (IP), defined as having been admitted for at least one night to a residential, inpatient, or hospital program for alcohol or other drug use problems. At each timepoint (i.e., intake, 3-, 6-, and 12-months post-intake), youth reported whether they had received each of the LOCs in the previous 3 months. Youth who reported an LOC did not necessarily receive the LOC for the entire 3-month period. Because some youth may have received multiple LOCs during each 3-month period, for analysis, we coded their LOC as the most intensive LOC received, regardless of frequency. If none of the three LOCs were reported, the youth were coded as having received no active treatment (NAT).

2.3.2. Outcomes of interest

The primary outcome of interest was a composite measure of positive treatment outcomes, defined as the total count of the National Outcomes Measures (NOMs), measured at 12 months post-intake. The NOMs and the use of a composite measure were developed by SAMHSA (Garnick et al., 2009). Each NOM represents a positive binary treatment outcome: 1) abstinence; 2) lack of SUD symptoms; 3) lack of physical health problems; 4) lack of mental health problems; 5) no illegal activity; 6) no justice system involvement; 7) stably housed in the community; 8) lack of family problems; 9) vocational engagement; and 10) evidence of social support. Scores on our composite outcome, which we will denote NOMs count, range from 0 (denoting absence of all 10 NOMs) to 10 (achievement of all 10).

While the NOMs captures information on a wide range of treatment outcomes and weights them all equally, some adolescents or clinicians may be primarily concerned with only certain types of treatment success. To demonstrate that different protocols are necessary to target different treatment outcomes, we additionally examined three secondary 12-month outcomes selected based on stakeholder input from an online modified-Delphi process (Dalal et al., 2011; Hasson et al., 2000), described in detail elsewhere (Grant et al., 2017): the Substance Frequency Scale (SFS), Substance Problems Scale (SPS), and Emotional Problems Scale (EPS). SFS is an eight-item scale that assesses the average proportion of alcohol- and other-drug-using days in the past 90 days, taking into account heavy use and problem days (Dennis et al., 2010). SPS is a 16-item scale of past month symptoms of substance abuse, dependence, and substance-induced health and psychological disorders based on the Diagnostic and Statistical Manual of Mental Disorders-IV-TR (Association, 2013). EPS measures recency and days (during the past 90) affected by emotional problems, disturbing memories, or self-control issues. These three measures corresponded to the most important treatment outcomes identified by all stakeholder groups (SUD treatment service providers, policymakers, researchers, and parents). See (Lennox et al., 2006) for psychometric information on these measures, and full norms and psychometrics for all scales in the GAIN are available at http://gaincc.org/data-statistics/encyclopedia-norms-psychometrics/.

2.3.3. Tailoring variables

Tailoring variables are the variables capturing current and past symptomatology and level of functioning that are used to personalize LOC to each adolescent. Table 1 shows the tailoring variables and which models in which each was included. Six variables were considered for the intake-to-3-month LOC-placement decision; and twelve measures were considered for the followup 3-to-6-month LOC-placement decision. Intake and 3-month versions of the outcomes were included in all models, as was age.

Table 1.

Tailoring variables per model

	NOMs model		SFS model		SPS model		EPS model
Measure	Intake LOC	3-mo. LOC	Intake LOC	3-mo. LOC	Intake LOC	3-mo. LOC	Intake LOC	3-mo. LOC
Age	x	x	x	x	x	x	x	x
Intake LOC		x		x		x		x
Intake SFS	x	x	x	x	x	x	x	x
3-mo. SFS		x		x		x		x
Intake CWS	x	x	x	x	x	x	x	x
3-mo. CWS		x		x		x		x
Intake STS	x	x	x	x	x	x	x	x
3-mo. STS		x		x		x		x
Intake SUDS (year)	x	x	x	x	x	x	x	x
3-mo. SUDS (month)		x		x		x		x
Intake NOMs	x	x
3-mo. NOMs		x
Intake SPS					x	x
3-mo. SPS						x
Intake EPS							x	x
3-mo. EPS								x

Four variables, collected at each timepoint, were also selected based on stakeholder input from the modified-Delphi process because they were considered by all stakeholder groups as the most important tailoring variables when deciding the LOC for an adolescent entering SUD treatment. These were: suicidality, behavioral withdrawal symptoms, substance use frequency, and SUD severity.

Presence of suicidality was measured using the Suicidal Thoughts Scale (STS) which sums across yes/no indicators whether the youth had “thought about ending your life or committing suicide”, “a plan to commit suicide”, “gotten a gun, pills or other things to carry out your plan”, or “attempted to commit suicide”. Substance use frequency was measured by the SFS described above. Presence of behavioral withdrawal symptoms was measured using the Current Withdrawal Scale (CWS), which is the sum of 21 past-week yes/no items related to psychological and physiological withdrawal symptoms based on the DSM-IV. SUD severity was measured by the Substance Use Disorder Scale (SUDS), which is a count of 11 items related to symptoms of either substance abuse or dependence that were endorsed by the youth. At intake, we used SUDS measuring symptoms in the past year, while at 3 months we used SUDS measuring symptoms in the past month. Tailoring variables were centered and scaled to promote interpretability of regression coefficients.

2.3.4. Control variables

We also included gender and race/ethnicity as control variables, but we do not propose to tailor treatments by them. Their inclusion can help account for any confounding effect race/ethnicity or gender may have on the relationship between LOC and the outcomes.

2.4. Data analysis

As described in the study’s protocol paper (Grant et al., 2017), we randomly partitioned the study sample into two datasets of approximately equal size. One dataset (n=5,066, the training dataset) was used for estimating adaptive LOC-placement protocols, and the second dataset (n=5,065, the evaluation dataset) was used for evaluating the relative effectiveness of the protocols. Data partitioning helps prevent “overfitting” which may occur when evaluating the protocol on the same data used to develop it, possibly overstating its usefulness (Friedman et al., 2001).

2.4.1. Estimating adaptive LOC-placement protocols

We employed Q-learning regression (Murphy, 2005; Nahum-Shani et al., 2017); Schulte et al. (2014), which resembles standard moderated regression analyses. Q-learning can be used to (i) estimate the adaptive LOC-placement protocol and (ii) understand whether and how candidate tailoring variables are useful in the adaptive protocol. Specifically, two regressions were fit in sequence, one at each timepoint, to investigate how the LOC effect varies as a function of the tailoring variables. The first regression investigated the optimal followup LOC-placement decision based on the tailoring variables in, e.g., Column 3 of Table 1. The second regression investigated the optimal intake LOC-placement decision based on the tailoring variables in, e.g., Column 2 of Table 1, accounting for the optimal followup LOC-placement decision based on the first regression (Nahum-Shani et al., 2017; Nahum-Shani et al., 2012).

Because decision rules based on Q-learning protocols can be quite complex, we further employed decision lists (Zhang et al., 2018; Zhang et al., 2015) to simplify the Q-learning-based protocol. The simplified protocols were selected to be a series of no more than five if-then statements (e.g., if intake SFS < 10, recommend outpatient). We did not use intake LOC to tailor followup treatments in the simplified protocols to avoid inconsistency between the recommended intake LOC (which was uniformly IOP for the primary protocol – see Table 4) and a later condition for recommending LOC-placement.

Table 4.

Simplified adaptive protocols

Outcome	Timepoint	Condition	LOC Recommendation
NOMs	Intake	Always	IOP
	Followup	If Followup STS = 0 and Followup SUDS = 0	NAT
		Else if Followup CWS > 1	IP
		Else if Intake SUDS < 6	IOP
		Else if Intake CWS > 5	IP
		Else	IOP
SFS	Intake	If Intake SFS < 10.8	OP
		Else if Intake SFS > 25.2	IP
		Else if Age < 16	IP
		Else	OP
	Followup	If Intake SFS < 19.1 and	OP
		Else	IP
SPS	Intake	If Intake SFS < 18.8 and Age > 14	IOP
		Else if Age < 17	IP
		Else if Intake SFS < 33.1	IOP
		Else	IP
	Followup	If Intake SUDS < 5 and Age < 17	IP
		Else if Intake SPS < 5 and Intake STS = 0	OP
		Else if Age < 15	IP
		Else if Age > 21	IOP
		Else	OP
EPS	Intake	If Intake CWS < 6	IOP
		Else if Intake SFS < 16.9	OP
		Else if Intake CWS < 9	IOP
		Else if Intake SFS < 32.9	OP
		Else	IOP
	Followup	If Followup STS = 0 and Followup SUDS < 6	NAT
		Else if Followup CWS < 4	IOP
		Else	IP

2.4.2. Evaluating the relative effectiveness of the LOC-placement protocols

We used marginal mean models (Murphy et al., 2001) on the evaluation dataset to estimate the average outcome one would expect if all adolescents followed each LOC-placement protocol. As per the study’s protocol paper (Grant et al., 2017), relative effectiveness was assessed by comparing the average outcome for each adaptive protocol versus four static protocols not tailored to adolescents’ characteristics: (i) always recommend IP, (ii) always recommend IOP, (iii) always recommend outpatient, and (iv) recommend outpatient at intake and NAT at 3 months.

We also assessed the average outcome for the simplified adaptive protocol compared to two other adaptive protocols: a protocol based solely on Q-learning with no simplification, and a simplified adaptive protocol that also uses intake LOC as a tailoring variable.

Bootstrapping (Efron and Tibshirani, 1986) was used to estimate standard errors for all estimates. Effects are reported on the scale of the outcome as well as in terms of outcome SDs or the “effect size”. See Appendix A for more computational details.

2.4.3. Missing data

Missing data in this study were primarily due to loss-to-followup. Followup rates at 3-, 6-, and 12-months were 89%, 84% and 72%, respectively. The Amelia package in R was used to multiply impute missing data based on a multivariate normal distribution, which has been shown to perform sensibly even when data are nominal or ordinal (Honaker et al., 2011). Because increased numbers of imputations are needed for more complex procedures (He et al., 2010), we imputed 100 datasets. All 100 imputed datasets were analyzed identically and standard rules were used for summarizing the results across the imputed datasets (Rubin, 2004).

3. Results

Table 2 shows characteristics of youth in the entire dataset. Most adolescents (90%) fell between the ages of 14 and 18. At intake the average NOMs count in the sample was 6.1, going up to 6.6 at 3 months. Adolescents in the sample reported alcohol or other drug use in 12.5% of the previous 90 days and 4.6 symptoms of SUD at intake.

Table 2.

Sample characteristics

Variable	Intake	3-month
Age	16,(1.7)	-
Female	28%	-
Race/ethnicity
White	39%	-
Hispanic	29%	-
African American	14%	-
Mixed	15%	-
Other	3%	-
NOMs Count	3.92, (1.7)	3.39, (1.8)
STS	0.24, (0.8)	0.06, (0.3)
SFS	12.46, (14.1)	6.81, (10.8)
CWS	1.26, (3.1)	0.66, (2.3)
SUDS*	4.57, (3.6)	0.98, (2.1)

Table Notes: Averaged over 100 imputed datasets; NOM = National Outcome Measure; STS = Suicidal Thoughts Scale; SFS = Substance Frequency Scale; CWS = Current Withdrawal Symptoms; SUDS = Substance Use Disorder Scale;

^*At intake, SUDS measures past-year symptoms; at 3 months, SUDS measures past-month symptoms.

Table 3 presents the distribution of the observed treatment sequences in the evaluation dataset. The most common treatment sequences were: 1) outpatient at both timepoints (n=1,807) and 2) outpatient followed by NAT (n=1,511). Adolescents followed both step-down treatment trajectories (IP then outpatient or IOP, n=192) and step-up trajectories (outpatient then IP, n=172). We also observed 389 who received IP for all six months.

Table 3.

Observed treatment sequences in evaluation data

Treatment reported at 3 months	Treatment reported at 6 months	n*
OP	OP	1807
OP	NAT	1511
IP	IP	389
IP	NAT	292
IOP	NAT	241
IOP	IOP	234
OP	IP	172
IP	OP	118
IOP	OP	90
OP	IOP	89
IP	IOP	74
IOP	IP	49

Table Notes:

^*Averaged over 100 imputed datasets;

OP = Outpatient treatment; NAT = Not receiving active treatment; IP = inpatient/residential treatment; IOP = Intensive outpatient treatment

3.1. Adaptive LOC-placement protocols

Table 4 outlines the simplified adaptive protocols as a series of if-then statements. The estimated protocol for the primary outcome recommended all patients to IOP at intake. The followup treatment decision was made by recommending no further treatment (NAT) for adolescents with no suicidal thoughts (Followup STS = 0) and no evidence of substance use disorder (Followup SUDS = 0). Among the patients with either Followup STS > 0 or Followup SUDS > 0, those who additionally had CWS > 1 at followup were recommended to step up to inpatient/residential care. The remaining adolescents were recommended to either inpatient or IOP based on their Intake SUDS and Intake CWS. While SFS does not enter into the protocol for 12-month NOMs count, it is the only factor consulted (along with age) to determine LOC for minimizing 12-month SFS. At intake, those with low SFS (<10.8) or moderate SFS (10.8-25.2) and older age (>15) are recommended to OP, while those with high SFS (>25.2) or moderate SFS and young age (<16) are recommended to IP. At followup, IP is recommended for those who have high SFS at either intake or followup.

Table 5 shows how often each treatment sequence was recommended by these simplified adaptive protocols, again summarizing across imputed datasets. In contrast to the observed sequences shown in Table 3, the adaptive LOC-placement protocols tended to recommend the majority of youth to IOP at intake followed by a lower LOC at followup. As expected, the recommendations differed depending on which outcome was being targeted. However, the placement protocols were relatively similar for two of the secondary outcomes, SPS and EPS, where the majority of youth were recommended to IOP at intake (66% for SPS, 95% for EPS), and the most recommended LOC sequence was IOP followed by a lower LOC (IOP followed by NAT for NOMs and EPS, IOP then OP for SPS). In contrast, when targeting SFS, the majority of youth (73%) were recommended to outpatient at intake (that number was 5%, 0%, 0% for EPS, NOMs, and SPS, respectively), and the most recommended sequences was outpatient at both timepoints. This result is consistent with the hypothesis that SUD treatment providers are currently optimizing their LOC-placement decisions to minimize substance frequency and not the other outcomes, as the adaptive LOC-placement protocol targeted for SFS matches most closely the observed LOCs in Table 3.

Table 5.

Recommended treatment sequences in evaluation data using simplified adaptive protocol

	Intake	Followup	n* (NOMs)	n* (SFS)	n* (SPS)	n* (EPS)
IOP		NAT	3610 (71%)	-	-	4459 (88%)
OP		OP	-	3000 (59%)	-	-
IOP		OP	-	-	2227 (44%)	-
IP		IP	-	1191 (24%)	910 (18%)	-
IOP		IP	476 (9%)	-	1078 (21%)	101 (2%)
IOP		IOP	979 (19%)	-	16 (<1%)	256 (5%)
IP		OP	-	198 (4%)	819 (16%)	-
OP		IP	-	676 (13%)	-	21 (<1%)
OP		NAT	-	-	-	207 (4%)
OP		IOP	-	-	-	21 (<1%)
IP		IOP			15 (<1%)

Table Notes:

^*Averaged over 100 imputed datasets;

OP = Outpatient treatment; NAT = Not receiving active treatment; IP = inpatient/residential treatment; IOP = Intensive outpatient treatment

The LOC sequence where IP was received for all 6 months was also common in the observed data as well as relatively so in the adaptive LOC-placement protocols for both SFS and SPS (24% SFS, 18% SPS), but it was never recommended by the NOMs and EPS adaptive protocols. All protocols recommended at least 2% of the highest-risk youth to inpatient at followup.

3.2. Relative effectiveness of adaptive LOC-placement protocols versus static protocols

The simplified adaptive LOC-placement protocol was estimated to produce a 0.75-unit (CI 0.4-1.1) or 0.4-SD (CI 0.2-0.6) higher 12-month NOMs counts than the static protocols, averaging across static protocols. The mean estimated 12-month NOMs count for each placement protocol with CI is given in Figure 1. The estimates for the secondary outcomes are also given there. If all adolescents had followed the simplified adaptive LOC-placement protocol, we estimate that their 12-month NOMs would have been 7.7 (CI [7.4,8.1]), much higher than the population mean of 7.05 (dotted line in the figure) and 0.7 (0.4 SDs) higher than the always-outpatient protocol (CI [0.3,1.1]), 0.8 (0.4 SDs) higher than the step-down protocol (CI [0.4,1.2]), 1.0 (0.5 SDs) higher than the always-IP protocol (CI [0.5,1.5]), and 0.5 (0.3 SDs) higher than the always-IOP protocol (CI [0.2,0.8]).

Figure 1.

Mean estimated outcome for each LOC-placement protocol, primary and secondary outcomes.

Results for SFS and EPS were similar but weaker, where the adaptive protocol outperformed the static protocols by only 0.1-0.2 SDs on average (SFS CI [−0.1,0.3], EPS CI [0.0,0.4]). In individual comparisons, the adaptive protocol was estimated to be either much better or quite similar to the static protocols. For example, we estimated the 12-month SFS would have been 3.9 points (0.3 SDs) lower if all adolescents had followed the adaptive protocol rather than always-IOP (CI [1.3-6.6]). Similarly, the adaptive protocol outperformed the always-outpatient (1.3 points/0.1 SDs better, CI [−1.4,3.9]) and step-down protocols (1.0 better, CI [0.9,2.8]), though the evidence was weaker with CIs that included strong effects in the other direction.

On the other hand, there was very little evidence in favor of using an adaptive rule for SPS, with all static protocols estimated to be slightly (on average 0.04 SDs, CI [−0.2,0.3]) better. CIs for all comparisons were wide, however, compared to the point estimate.

3.3. Comparison of simplified adaptive protocols to alternative adaptive protocols

We compared the simplified protocols which approximate the Q-learning-based protocols with a series of if-then statements to the full Q-learning-based protocols. The simplified protocols actually outperformed the full Q-learning-based protocols on the evaluation data. They were about 0.1 SDs better for the primary outome and EPS (CI [−0.1,0.3] for both outcomes) and very similar for both SPS (0.03 SDs better) and SFS (0.02 SDs worse).

We also compared simplified protocols that did not tailor followup LOC-placement on intake LOC to similar simplified protocols that did use intake LOC as a tailoring variable. The effects of not using intake LOC as a tailoring variable were quite small (<0.03 SDs for all outcomes) in the evaluation data.

4. Discussion

This work is a first step toward building an evidence base for optimizing SUD treatment placement for individual adolescents. We have demonstrated the feasibility of estimating, interpreting, and evaluating LOC-placement protocols for these youth from a large observational dataset. The protocols are proof-of-concept for a simple, empirically-based guide for tailoring LOCs to each adolescent based on their current and past symptomatology, level of functioning, and other relevant characteristics. The protocols lended empirical support to the clinical intuition that higher LOCs ought to be recommended to those with more severe substance use (in line with the American Society of Addiction Medicine’s Criteria) and give easy-to-understand if-then statements to formalize these intuitions for this population.

Finally, we have evaluated our estimated protocol on an independent dataset and shown that an adaptive approach to adolescents’ needs generally outperforms static protocols, where one would expect to see adolescents report three quarters of an additional NOMs on average if all adolescents followed the protocol. This effect denotes a potentially meaningful clinical effect as such an effect would correspond to 75% of all adolescents accomplishing one more NOMs treatment goal (such as abstinence from substance use or lack of mental health problems) than they would have under the static protocols, or one quarter of all adolescents accomplishing three more treatment goals. Another way to contextualize such an effect is in terms of effect size, or standard deviations of the outcome. On average, the adaptive protocol outperformed the static protocols by about 0.4 SDs, which corresponds to a medium-sized effect according to standard rules of thumb (Cohen, 1992). This has the potential to be clinically meaningful in a space where treatments are often known to have only small relative effects to one another.

These results are suggestive that adaptive protocols can be developed to aid clinicians in making LOC-placement decisions for adolescents. However, translating these protocols into clinical practice requires external validation and further study. In this mixed-methods study, we identified key variables and developed protocols using widely available scales on observational data. In future studies, researchers could measure how recommending LOCs based on adaptive protocols compare to the standard-of-care by prospectively recommending treatments to adolescents in a randomized trial. Research designs, such as sequential multiple-assignment randomized trials (SMART), are available to develop and analyze these types of adaptive protocols and could in principle provide more accurate accounting of LOCs and improve estimation of protocol effects. Furthermore, given the unique patterns of clinical problems among youth with SUD, there is ample opportunity to develop more detailed protocols that identify more granular treatment strategies than our two-stage LOC-level protocols.

The development of LOC-placement protocols depends ultimately on the desired goal of treatment. While it may be difficult to prioritize one outcome over others, the meaning and usefulness of the adaptive protocols depends on the clinical meaning of the outcome. We selected the NOMs count, a composite outcome, for our primary analysis because we wanted the primary LOC-placement protocol to apply broadly to a range of treatment goals. More specifically, the NOMs gives equal weight to each of a set of 10 treatment outcomes. If one takes the view that a large set of treatment outcomes should be targeted without specifically prioritizing one, then one might naturally select the NOMs count protocol (or one built on a similar composite). However, individual adolescents or clinicians may have more specific treatment goals, such as limiting substance frequency or minimizing emotional problems. Our work using secondary outcomes demonstrates that LOC-placement protocols vary when targeting these different treatment goals. We note that in many applications, interest might lie in optimizing treatment for multiple outcomes in a way that conjointly optimizes all outcomes (possibly including other considerations such as cost), thereby requiring a multivariate analysis approach. Unfortunately, no statistical methods are currently available to perform such an analysis. This is an active area of research where more work is desperately needed.

We note several limitations of our study. First, our data were collected observationally and thus may not include all relevant confounding variables. Second, it is important to note that this dataset is not necessarily representative of all adolescent SUD treatment facilities—we suspect these are better performing facilities (Hunter et al., 2013) – and thus generalizability to the entire population in SUD treatment may be limited. However, in the absence of such a large nationally-representative dataset, ours is uniquely poised to begin to lay a foundation for evidence-based treatment personalization.

Furthermore, while treatment decisions for adolescents may occur at more intensive time scales (e.g., weekly or monthly), we are limited to evaluating treatment decisions at intake and 3-months only for the duration of 90 days. This limitation is both practical – youth only contribute survey responses at 90-day intervals – and computational – the statistical methods require a fixed set of decision points. However, we note that for decisions about adolescent SUD treatment settings, the 3-month time interval is highly relevant, as participation in treatment services for at least 90 days is generally regarded as a best practice(Fletcher and Chandler, 2006; Hser et al., 2001; Nora and Volkow, 2014) and many evidence-based adolescent programs for SUDs are an average of 3 months long (Tanner-Smith et al., 2013; Williams and Chang, 2000; Winters et al., 2011).

Finally, there are many potential sources for error which may have attenuated the benefits of the adaptive placement protocol. First, limitations in the granularity of the data required that we specify a single LOC for each adolescent at each time period, even though some youth may have experienced multiple LOCs during a given 90-day period. We chose to classify each individual according to their maximum-intensity LOC, which may introduce measurement error if it did not fully capture their experience, though the vast majority (85% at intake, 92% at followup) of the adolescents only listed one LOC. Second, the presence of missing data, which we accounted for via multiple imputation, further may add variability to our adaptive protocol and our evaluation of its usefulness. And finally some adolescents’ survey responses may not be entirely accurate.

5. Conclusions

This study provides the first step towards the development of LOC-placement protocols for use in adolescent SUD treatment settings. Based on these findings, further research is warranted in developing and validating LOC-placement protocols, which provide a one-to-one map from individual adolescent needs and LOC recommendations. Sufficiently validated placement protocols could be used as decision support tools in clinical settings leading to improved outcomes for adolescents receiving SUD treatment.

Highlights.

• One size does not fit all for adolescent substance use disorder treatment

• No empirical basis exists to place adolescents in optimal levels of care over time

• We estimate a personalized placement protocol from observational data

• We demonstrate the protocol’s good performance on independent data

List of Abbreviations

LOC

level-of-care

SUD

substance use disorder

GAIN

Global Appraisal of Individual Needs

OP

Outpatient

IOP

Intensive outpatient

IP

Inpatient/residential

NAT

No active treatment

NOM

National Outcome Measure

SFS

Substance Frequency Scale

SPS

Substance Problems Scale

EPS

Emotional Problems Scale

STS

Suicidal Thoughts Scale

CWS

Current Withdrawal Scale

SUDS

Substance Use Disorder Scale

CI

Confidence interval