Operationalizing a Behavioral Health Services Cascade of Care Model: Lessons Learned from a 33-Site Implementation in Juvenile Justice Community Supervision

ADOLESCENTS (AGE 12–17) in the community who self-report contact with the juvenile justice system (JJS) through probation, parole, or detention during the past year are significantly (p<.05) more likely than those who were non-JJS involved to report meeting criteria for substance use disorder (SUD) (APA, 2013) across substances (21.4 percent vs. 3.7 percent, odds ratio [OR]=7.1), as well as specifically related to cannabis (12.1 percent vs. 2.0 percent, OR=6.7), alcohol (9.4 percent vs. 1.7 percent, OR=6.2), heroin or other opioids (3.5 percent vs. 0.4 percent, OR=9.7), and cocaine or other stimulants (1.3 percent vs. 0.3 percent. OR=4.6; SAMHSA, 2018). In the subset of adolescents with SUD, those with JJS contact are significantly more likely than those without contact to receive substance use (SU) treatment (25.2 percent to 6.3 percent, OR = 5.0); however, 3 out of 4 of them still did not access any kind of SU treatment.

Consistent with the numbers above, prior research suggests that 27 percent to 65 percent of the youth formally in parts of the JJS meet criteria for some kind of SUD—going up with the intensity of involvement. Unfortunately, the research also suggests that only 5 to 35 percent of the JJS youth “in need” receive any kind of SU treatment (Baumer et al., 2018; Becan et al., 2019; Dennis et al., 2009; McReynolds, Wasserman, & Ozbardakci, 2017; Shufelt & Cocozza, 2006; Teplin et al., 2002; Timmons-Mitchell et al., 1997; Wasserman et al., 2002; 2010). Although community supervision (CS) is one of the most common dispositional alternatives in the JJS (Kaeble & Glaze, 2018), it is also one of the least studied in terms of SUD prevalence and treatment (Wilson et al., 2009). A just-released survey from a national probability sample of counties (Scott et al., 2019) estimates that there are 770,323 youth under CS by 3143 CS agencies in the U.S. The subset (45 percent) of CS agencies that collected and had data available on substance use problems reported that 51 percent of their youth had a substance use problem, including cannabis (49 percent), alcohol (25 percent), prescription drug misuse (19 percent), and/or any other drug (18 percent). Most (91 percent) of these CS agencies referred all or most of these youth to SU treatment providers in their community. On the bright side, most of these programs used a range of evidence-based practices and also provided programs for youth with co-occurring mental health needs. Unfortunately, this survey and other research reviews (Belenko et al., 2017; Knight et al., 2016) also document widespread challenges related to the processes of identification, referral, and treatment initiation and retention between systems. This is important because continued SU and SUD are among the key risk factors of long-term delinquent behavior and recidivism; conversely, initiation, engagement, and continuing care in SU treatment is associated with reduced SU, SUD, and recidivism (Clark, 2004; D’Amico, Edelen, Miles, & Morral, 2008; Evans-Cuellar et al., 2006; Hicks et al., 2010; Hoeve et al., 2013; 2014; Kandel & Davies, 1992; Kandel & Yamaguchi, 2002; McReynolds et al., 2010; NIDA, 2006; 2014; Pankow et al., 2019).

The Behavioral Health (BH) Services Cascade

There is a well-established history of tracking the process of SUD identification, referral, initiation, and retention in treatment across complex systems of care in general (Chandler et al., 2015; DiPrete et al., 2019; Dennis et al., 2003; 2012; Morgan et al., 2016; Montgomery et al., 2019; Scott & Dennis, 2009; Scott et al., 2018; Williams et al., 2018, 2019a&b), and specifically in the movement from the justice system to substance use treatment (Scott & Dennis, 2012; Scott et al., 2017). The Behavioral Health (BH) Service Cascade (Belenko et al., 2017; Williams et al., 2018) was specifically developed as a way to quantify and track movement along the following desired pathway: screening/clinical assessment, identification of need, referral to SU Treatment, and actual SU Treatment initiation, engagement, and continuing care. We use the term behavioral health (BH) because two-thirds of the primary SU treatment providers used by CS agencies are also their primary provider of mental health (MH) treatment services and operate co-occurring SU/MH programs (Scott et al. in press and in the programs participating here). The BH Service Cascade is also being used: a) as a framework for improving and harmonizing measurement across CS and SU Treatment agencies within and across communities; b) to identify which stages of the process need improvement; and c) to evaluate if “attempted fixes” improve these rates in actual practice (Becan et al., 2018; Leukefeld et al., 2017).

Although the BH Service Cascade provides a useful heuristic tool for conceptualizing the SU treatment services process in justice settings and helps define the types of data needed to analyze this process, there has been little research to date on a) how the BH Service Cascade framework can be operationalized in multi-system settings, b) the availability and quality of the data needed to analyze Cascade outcomes, or c) how well the Cascade framework captures actual screening, referral, and treatment processes in real-world practice settings.

This paper provides one of the first examinations of a large multi-site study of JJS CS agencies and their SU Treatment provider records to address three aims: (1) provide an empirical test of how well the BH Service Cascade framework works in practice; (2) identify methodological challenges in implementing the framework; and (3) provide recommendations for the next generation of juvenile justice and behavioral health data systems.

Method

Overview of the JJ-TRIALS Data Source

Data are from the Juvenile Justice-Translational Research on Interventions for Adolescents in the Legal System (JJ-TRIALS) cooperative agreement funded in 2013 by the National Institute on Drug Abuse (NIDA; Knight et al., 2016; Leukefeld et al., 2017). The cooperative includes six research centers (RCs: Columbia University, Emory University, Mississippi State University, Temple University, Texas Christian University, University of Kentucky), each working closely with a JJS partner in seven states (Florida, Georgia, Kentucky, Mississippi, New York, Pennsylvania, Texas), a coordinating center (CC; Chestnut Health Systems), and the scientific officer from NIDA. Its purpose is to improve identification and receipt of substance use-related services for treatment services youth under CS, and facilitate more effective collaborations between the JJS and BH systems. The cooperative includes multiple studies: 3 national surveys (Scott et al., 2016; 2019), a substance use prevention pilot study, a HIV prevention pilot study, and a multisite trial to use the BH service cascade to identify gaps in the systems of care and evaluate attempts to address them in a multisite cluster randomized trial (Knight et al., 2016). This paper uses data from the latter study.

Records data are from JJS CS/SU treatment records collected in 33 counties in 7 states on 31,308 youth under CS entering JJS between March 2014 and November 2017. While there were originally 36 county sites, one site with-drew from the study prior to randomization to study condition and two other sites were dropped from this analysis because they were almost entirely limited to post release from detention and had very different trajectories that will be looked at separately. Because the CS agencies varied in whether they had early diversion and the type of early diversion programs, records were excluded if contact was limited to early diversion programs. Thus, this article focuses on youth whose justice contact has been directly and primarily overseen by juvenile community supervision (CS).

Records and Participants

We include 31,308 JJS CS agency/SU Treatment provider youth records as our primary units of analysis. These records come from 24,490 unique youth on CS, with 17 percent of the youth having 2 to 9 records where the previous episode of community supervision ended more than 30 days earlier and a referral for a new offense was made to the juvenile justice system. Other than excluding the withdrawn/detention reentry sites and the youth only on early diversion, there were no other exclusion criteria. Thus, these represent a census of all the remaining youth on CS in the 33 counties. All record abstraction and recoding was done under the supervision of the respective institutional review boards (IRB) of the JJS agencies, RC and CC.

Measures

The cooperative developed a list of common record measures related to demographics, biological drug testing results, clinical problems, charges, adjudication, recidivism risk, justice system status, and BH service cascade outcomes. The specifications included a description of each item’s variable name, definitions, common coding of response set, and coding of missing data. Each RC evaluated the available records data sources to determine which items they were able to collect within current systems, and, if not currently collected, if there was a feasible way to add items to site data collection procedures. Each RC also assessed local item definitions to determine if there would be any issues recoding to the common item definitions, and reviewed records to determine any issues with significant missing data. Any issues with availability or coding were brought to the Cooperative’s Measurement and Data Management working group to evaluate and reach a group consensus on final inclusion or definition. In the end, the collaborative requested sites to compile data on 72 variables that all JJS partners agreed “ideally should be” in CS JJS records (see appendix—more detailed specifications also available from the first author).

Table 1 (adapted from Knight et al., 2016) provides the final operational definition and shows how the relative rates were calculated for each step of the BH service cascade in JJ-TRIALS. Three major changes were made to address limits in the data: 1) we looked at screening results even if the date was missing, 2) we dropped a clinical assessment step that previously happened between screening and referral, and 3) due to increasing missing data in later steps of the cascade, the number of sites considered at each step was limited to those with data. In the results, retention to each step is considered in two ways: 1) simple rate: the n retained to the step as a percentage of the total n of youth referred to CS; and 2) relative rate: the n retained in each step relative to the denominator for youth entering the location. The latter is defined in the last column of Table 1 as: screening & need relative to the total n; referral and initiation relative to the n who were “in need” of SU treatment; and engagement & continuing care relative to the n who initiated SU treatment.

TABLE 1.

Behavioral Health Services Cascade Definitions

Step	Operational Definition	Relative Rate
a. JJ Referrals	Total number of referrals to juvenile justice in time period with a disposition starts date, less any youth already in treatment at that time.	—
b. Screened	Subset of JJ referrals (a) with a screening record.	b/a
c. Need Identified	Subset of JJ referrals (a) with a need for substance use treatment based on screener, urinalysis, clinical assessment, or other sources of information.	c/a
d. JJ Referrals to Treatment	Subset of those in need (d), referred by the juvenile justice system to substance use treatment.	d/c
e. Initiated Treatment	Subset of those referred to treatment (e) who have treatment start date.	e/c
f. Engaged In Treatment	Subset of those who initiate treatment (f) who stay in treatment for at least 6 weeks (based on treatment discharge minus treatment start date).	f/e
g. Continuing Care	Subset of those engaged in treatment (g) that are still getting treatment after 90 days (whether via retention, transfer, or booster).	g/e

Data Abstraction and Coding Procedures

Data were abstracted from a mix of state and local electronic databases, as well as from either scanned or physical paper records. Data from SU treatment providers often had to be added to JJS records or separately obtained. Data were then compiled and cleaned in an iterative process: first by several of the larger JJS local or state agencies, then by the 6 RCs, and finally by the CC. In the final step, records were combined and rechecked for consistency of coding across the cooperative; feedback on any anomalies was generated and returned to each RC for further investigation. All data included in this paper were collected and submitted by September 2018.

Records were collapsed within youth when there were multiple juvenile justice referrals that were collapsed by the court. After submission, the CC began with 33,508 records. Within each record, service records were excluded (screening, clinical assessment, referral to treatment, treatment intake, and treatment discharge) when the date of service preceded the referral to the juvenile justice system due to services occurring across multiple referral episodes. A small number of records (n=12) were removed during the data cleaning process due mostly to missing data.

Because of variation in CS agency record practices (e.g., one had a separate referral for each charge and another just added to existing referral), the CC further collapsed all JJS referral episodes within 0 to 30 days for a unique youth into the first record (reduction of n=1,603 records). Records were further excluded by the CC for youth with less than 90 days between referral to juvenile justice and the final record abstraction for the site (n=585), as they did not have the opportunity to move through the entire service cascade. Due to some overlap, this resulted in the final dataset of 31,308.

Within each record, the date or flag for a given type of BH service cascade was considered sufficient to code as indicating it happened. For need, any indicator (screening, clinical assessment, urine drug testing, referral, charges) was considered sufficient and included 555/14,906 (4 percent) where the specific “basis or source” of the need for SU treatment was not documented in the record. Since the majority of records followed the cascade (discussed further below), records missing documentation on early steps were recoded if they had any of the later steps (e.g., referral without “need” documented, initiating treatment without “referral” documented). The exception was that “need” was not considered sufficient evidence to recode the flag for “screening.” Otherwise records were recoded as the event “not happening.”

Analytic Methods

Missing data was a significant obstacle to the analysis and happened in many forms. Some JJS agencies did not collect a given variable, had a field with open text or scanned documents that could not be easily coded, had partial information (event but not date or vice versa), or had staff that inconsistently filled in the field or did not fill it in at all. Also, it was clear that documentation was much more likely to exist in the records when an event happened (e.g., a screening or positive drug test) than when an event did not happen (e.g., an expected screening not done or a negative test result for a given drug).

To include the maximum number of records in the analysis, each of the BH service cascade flags was interpreted as “yes” (they were retained to this step of the cascade) vs. “other.” The “other” includes all answers of “no,” legitimate skips due to missing prior steps, and other “missing data” in record. Allowing all missing data to be treated in this way provided us with a lower bound and conservative estimate of the rates of retention at each step of the cascade.

As a sensitivity analysis, we also examined the impact using weighted hot deck imputation (Little & Rubin, 2002) to replace some “missing” data with “yes” values in order to make an upper bound and more liberal estimate of the rates of retention at each step. Specifically, data were sorted by site, maximum charge level (felony, misdemeanor, citation/violation, status), gender, supervision type (probation, parole, or juvenile drug treatment court vs. other CS/diversion), and cascade step. The cascade variables were imputed in order of cascade step, with the imputed version of each preceding step being the final sorting criteria for the next step. This means that a record with missing data on an item was surrounded in the list by records that were both mostly similar and that had reached the same point of the cascade. Missing data was then replaced with the median of the nearest 20 valid (non-missing) values. The median was used instead of the mean because cascade variables are all yes/no and vary dramatically in the percent of yes. Using the median within 20 values produces unbiased estimates of the mean and standard error at the group level. To judge the appropriateness of this kind of imputation, we assessed the extent to which the data were “missing at random (MAR)” by comparing the inter-item correlation of the lower bound (without imputation) and the upper bound (with imputation) estimates across maximum charge level, gender, supervision type, and the 6 BH service cascade steps sorting variables. Across the 81 comparisons, the inter-item correlations between the two methods above differed by an average of only r=0.05, with 5 of the 6 cascade steps averaging a difference of r=.1 or less, and the referral step having an average difference between methods of r=.13. This is relatively good evidence of meeting the assumptions of MAR and suggests that this is a reasonable approach.

There was one CS agency that did not have referral data documented and another 10–14 sites that were not able to obtain access to reliable data (less than 5 percent of expected records with yes or no) on SU treatment initiation, discharge, engagement, or continuing care. For these steps we dropped the sites without (reliable) data.

Results

Youth Record Availability and Prevalence of Characteristics

No CS agency record had all variables, and the median number of variables available per record was only 49 out of 72 (68 percent) in these actual CS agency records. The appendix gives the percentage of data available for each variable, the prevalence where it was available, and the site variation in terms of minimum and maximum prevalence. The prevalence of characteristics below is based on the subset of records with data available across sites and columns from column 2 of the appendix.

The youth records were related to juveniles who were primarily aged 11–17 (99 percent; 63 percent 15–17) and male (73 percent). They were primarily white (49 percent) or black (47 percent), with an overlapping 21 percent having Hispanic ethnicity. Clinically, the records showed that 56 percent had some kind of a substance problem, with those substances including cannabis (28 percent), alcohol (9 percent), prescription drugs (1 percent), and any other drugs (7 percent). Multiple other co-occurring problems were documented in the records, including serious family problems (43 percent), violence towards others (39 percent), internalizing (17 percent) or externalizing (19 percent) mental health problems, victimization (13 percent), suicide risk (11 percent), and learning or developmental disabilities (9 percent). Their most common charges were related to property (30 percent), violence (25 percent), substance use (14 percent), probation/parole violations (12 percent), and/or status offenses (11 percent); with their maximum severity being a felony (33 percent), misdemeanor (56 percent), summary/citation (3 percent), status (3 percent), or other (5 percent). Their risk of recidivism was rated as low (31 percent), medium (32 percent), high (16 percent) or very high (3 percent). At various times their justice status included probation (49 percent), detention (30 percent), child in need of supervision (11 percent), diversion (5 percent), juvenile drug treatment court (1 percent), parole (0.2 percent), other community supervision (67 percent), and other justice status (7 percent).

In terms of the BH service cascade variables, 81 percent were screened with one or more standardized tools with evidence bases, including the MAYSI-2 (19 percent), YASI (12 percent), PACT (9 percent), SASSI (3 percent), CRAFT (2 percent), and GAIN-SS (1 percent), as well as other state (31 percent) measures or a local measure (2 percent) with unknown psychometrics. Of the youth records screened, 28 percent indicated a positive need for SU treatment. In addition, 23 percent of the youth were clinically assessed, with 5 percent indicating a positive need for treatment. Other sources for the identified need for SU treatment variable included JJS staff recommendation (12 percent); clinical assessments (5 percent); youth, family, or other referral sources (3 percent); judicial mandate (0.2 percent); and undocumented reasons (0.2 percent). Across all of these sources, 54 percent of the youth records had one or more indicators of need, and 24 percent were referred to SU treatment. One CS agency did not document referrals at all and was dropped from this step.

Only half the records had any information on SU treatment initiation (53 percent) or discharge date (51 percent). Of those that did, only 15 percent indicated the date of SU treatment initiation and 11 percent the date of discharge (both necessary for directly calculating engagement for at least 6 weeks and continuing care for 90 days or more). The primary level of care was outpatient (10 percent), followed by no documented level of care (4 percent) and all other higher levels of care combined (1 percent). Sites that did not systematically document treatment initiation (10), engagement for at least 6 weeks (11), or continuing care for 90 or more days (12) were dropped for these respective steps.

The BH Services Cascade Across Counties

The columns of Table 2 show the location, steps of the BH service cascade, and the two methods for estimating the rates for each step of the cascade. The rows show the method, number of sites with data that could be used, the n of yes or imputed yes, total, and the three rows for the percentage of simple rate (i.e., youth records with yes/total records); youth records with yes/those records indicating “ in need”; youth records with yes/those records indicating that SU treatment was initiated; and the relative rate (repeating lowest row for each column). While the number of sites and total are the same across methods, imputation increases the number of records with yes and consequently the percentage of the total for every pair of columns. Given the changing number of sites and denominator, the last two rows are calculated by dividing the total percentage for the column by the column percentage for the reference listed (e.g., 15 percent referred/48 percent in need = 31 percent in the “Percentage In Need” row). Because imputation increases both numerator and denominator, the imputation method produces relative rates that can be higher (e.g., first 3 of 4 pairs for percentage in need) or lower (fourth comparison for percentage in need row, both comparisons for percentage initiating treatment row; and all 4 comparison in the final row that is used below).

TABLE 2.

BH Services Cascade Number of Sites and Records by Step

Location	Juvenile Justice System				Transition				Substance Use Treatment
BH Cascade Step	Screened		In Need		Referred		Initiated Treatment		Treatment Engagement		Continuing Care
Method	Yes/Total	Imputed Yes/Total	Yes/Total	Imputed Yes/Total	Yes/Total	Imputed Yes/Total	Yes/Total	Imputed Yes/Total	Yes/Total	Imputed Yes/Total	Yes/Total	Imputed Yes/Total
N of Sites	33		33		32		23		20		19
N =Yes	21,382	22,298	14,906	18,220	4,711	8,298	2,613	8,009	1,070	1,626	597	623
Total n of records	31,308		31.308		30,692		26,371		22,994		21,959
Simple Rate
% Total	68%	71%	48%	58%	15%	27%	10%	30%	4.7%	7.1%	2.7%	2.8%
% of In Need					31%	47%	21%	52%	10%	12%	6%	5%
% of Initiation									47%	24%	27%	9%
Relative Rate (lowest row above)	68%	71%	48%	58%	31%	47%	21%	52%	47%	24%	27%	9%

Figure 1 graphs the simple rate or “Percentage of the Total n” of records for each step by location, step, and method within step. At each step, the rates that included only the original “yes” answers (solid color on left) are always lower; the rates that include the original and “imputed yes” answers (slash marks on right) are always higher. In theory the rates should be the same or lower at each subsequent step—and this is the case for the original yes answers. The imputed yes rates, however, go up from referral (27 percent of total) to treatment initiation (30 percent of total). As shown in Table 2, this is because the number of sites and total used in the denominator is going down in the last 4 steps. Substantively the figure shows that the participating CS agencies were screening the majority (68–71 percent) of the youth and found that about half (48–58 percent) were in need of SU Treatment. However, it also shows that only a fraction of these were referred to (15–27 percent of total) or initiated (10–30 percent of total) SU treatment. Moreover, half or less of those who initiated treatment stayed engaged for at least 6 weeks (5–7 percent of total) or received continuing care 90 days or more later (2.7 to 2.8 percent of total).

FIGURE 1.

Behavioral Health Services Cascade from Juvenile Community Supervision to Substance Use Treatment: Simple Rate (% of Total) of Surviving Step by Method

Figure 2 provides an alternative perspective of the BH service cascade using the “relative rate,” this time graphing the last row in each column of Table 2, for each of the two methods. As noted earlier, because imputation is increasing both the numerator and the denominator, these rates can be higher or lower than the rates based on only the original yes answers. While the imputed rates are higher for the first two steps, they are actually lower than the original rates in the last 4 steps. Substantively, this more clearly illustrates the first common problem in CS setting—that during the transition between systems less than a third of the youth on CS identified as “in need” of SU treatment get referred to (26–31 percent of “in need”) or initiate (17–21 percent) SU treatment. It also illustrates the second common problem—that even among those youth who initiated treatment, fewer than half engaged in treatment for 6 weeks (16–47 percent) or continuing care for at least 90 days (9 percent-26 percent). While there was significant variation in these rates by sites, this pattern of problems is consistent—with most youth being lost in the transition between systems, followed by low retention once they initiate treatment.

FIGURE 2.

Behavioral Health Services Cascade from Juvenile Community Supervision to Substance Use Treatment: Relative Rate (% of N Entering Location) of Surviving Step by Method

Validations of BH Service Cascade Estimates

Given the high rates of missing data and site to site variation, it is important to also validate the BH service cascade estimates. The first way we did this was to verify that screening, need, and referral were in fact the most common pathway to treatment initiation. Of the 2,613 youth records where SU treatment was initiated—75 percent followed all three steps along this expected pathway, 22 percent followed two steps in order with the third missing information, and 2.5 percent had only 1 of 3 steps. For the latter two patterns, data on the other steps was largely missing. Only 0.5 percent of the youth records documented SU treatment initiated without the CS agencies taking any of the first three steps in the cascade.

Second, we examined the predictors of each step both by univariate and multivariate analyses considering all potential sources of need and prior steps. Formal “Screening” was the strongest predictor of documenting “Need” in the record in both analyses (OR=114.6 & 932.7 respectively). “Need from any source” was the strongest univariate predictor of “referral” in the univariate analysis and the second strongest in the multivariate analysis (OR=9.0 & 2.7 respectively); “need from screening” was the second strongest in the univariate analysis and the strongest in the multivariate analysis (OR=4.6 & 2.9 respectively). A formal “Referral” by CS agency in turn was the strongest predictor of which youth actually initiated treatment (OR=195.1 & 141.6 respectively). These are all very large odds ratios. Also note that this was all prior to recoding approximately 1 percent of the cases where a latter step occurred with the prior step missing.

The only originally proposed BH service cascade step that did NOT fit was “Clinical Assessment.” This activity was recorded less often than expected, often only on those who had screened positive, and in several agencies only after or as part of initiating treatment. Collapsing it into screening or dropping it (as we have done here) did not impact any of the rates by even 0.1 percent and would not change any of the reported results.

Third, we compared the relative rates reported in Table 2 with available national data (Figure 3). The first two columns on the left side are the relative rates of 2017 National Household Survey on Drug Use and Health (NSDUH; SAMHSA, 2018) and its broader definition of need for SU treatment (already in treatment, weekly use, or SUD) for youth (ages 12–17) without JJS contact (white with gray dots) and with JJS contact (gray with white dots). The next two columns show the relative rates from youth on community supervision as reported above in Table 2 and Figure 2. Relative to youth with any contact, youth on CS here had higher rates of need and similar rates of initiation. The second two columns on the right side show the relative rates from the 2013 Treatment Episodes Data Set-Discharge (TEDS-D; SAMHSA, 2015). Youth initiating treatment are again divided into those without JJS contact (white with gray dots) and with JJS contact (gray with white dots). The solid and hatched light grey, medium gray, and black bars are still from Table 2 and Figure 2. Youth without and with any JJS contact have relatively similar rates of SU treatment engagement for 6 weeks and continuing care for 90 days or more. Youth on CS in this study were found to have lower rates when based on the first method of original yes answers and much lower rates when using the second method with imputation (impacting both denominator and numerator).

FIGURE 3.

Comparison to Relative Rates from National Data

Key Lessons about Current Practice

The study produced several key lessons about current practice listed below:

• Data existed across multiple locations/systems: Where it existed, the 72 variables we sought were often collected through electronic and/or paper records and were not always readily accessible to the CS staff.

• Terms and Definitions varied by CS agency: A stay of probation contingent on good behavior was called different things in different sites (e.g., delayed probation, informal probation, diversion) and basic definitions and data capture procedures varied by agency (e.g., one state added multiple and subsequent charges to an existing record; another created separate records for each charge; a third allowed multiple charges in a record but started a new record if the youth had been released).

• Data that are “Electronic” were not necessarily coded or available: Instead it was often free text or a scanned document; there was also inconsistency in whether and how staff used these fields.

• Dates were often missing: Although date fields were common and/or expected in notes, they were frequently missing. When they existed, some dates referred to earlier encounters with CS (e.g., an earlier charge/CS episode).

• Some juvenile justice agencies did not allow the RC to have direct access to some information on justice records: This meant that their attempts to combine records and data from a relational data set were often error prone, had to be indirectly “discovered,” and had to have the dataset recreated (in one case multiple times) to ensure the most accurate data.

• Staff turnover at the CS agency compounded problems: Failure to keep sufficient documentation and/or having more than one person cross-trained on data tasks led to several short-term setbacks; this was potentially exacerbated if data tasks were added to the role of a CS person who was already overloaded.

• Treatment data were not always readily accessible: In spite of their best efforts, about a third of the CS agencies could not obtain reliable data on SU treatment initiation, discharge, engagement, or continuing care.

Thus, although the justice partners working with each RC thought that all of the 72 variables proposed would be readily available; they initially overestimated the degree of access to and amount and quality of the available data within their respective systems.

Discussion

Reprise of Findings

In spite of a wide array of issues and high rates of missing data, this article demonstrates the feasibility of using the BH Service Cascade framework in practice. With 1 in 8 adolescents with SUD passing through the JJS each year, the JJS is second only to schools as one of the best places to identify and intervene with youth that have SUD (Dennis, Clark, & Haung, 2014). The results here show that the JJS participating here were doing well at screening and identifying youth in need of SU treatment relative to national CS agency data (Scott et al., in press). However, the cascade results also show that two-thirds of the youth on CS with need were dropping out in the transition between systems of care (e.g., SU Treatment referral and initiation). Moreover, among those who initiated SU treatment, over half were out before six weeks of engagement. The latter is important, because six weeks is the threshold as a minimum amount of care as defined by the Healthcare Effectiveness Data and information Set (HEDIS) used by the National Commission on Quality Assurance (NCQA), Medicaid, and the Office of the National Coordinator (ONC) of the Affordable Health Care Act (https://www.ncqa.org/hedis/measures/initiation-and-engagement-of-alcohol-and-other-drug-abuse-or-dependence-treatment/). Even fewer were retained, stepped down, or had any kind of continuing care for the 90 days after SU treatment initiation recommended by researchers as more effective (NIDA, 2006; 2014). While there was CS Agency to Agency variation in rates, these were consistent empirical findings. These findings also rang true for the participating JJS partners.

Confidence in these findings were further strengthened by several types of validation. Among the youth who eventually initiated SU treatment, movement through the cascade in the order shown was the most common pattern observed, followed by moving through 2 steps in order with the other missing; only 0.5 percent initiated treatment without going through at least one of these steps in the CS agency. This suggests that indeed the BH Service Cascade is currently the main route for these youth on CS to enter treatment. This is very important to the JJS because, as noted in the literature review, continued substance use is a risk factor for recidivism, and access to treatment is associated with reducing both substance use and recidivism. Relative to national data, the rates were very similar for the first method of using only documented yes answers. For the second method of changing some missing to imputed yes answers, the rates were similar for need and initiation, but much lower for engagement and continuing care. While the population estimate increased, the reason for the latter is that the denominator population estimates were increasing even more.

Limitations and Methodological Challenges

It is important to acknowledge that the kinds of real data from actual CS agencies have several limitations and present methodological challenges to use. For most of the CS agencies, data were missing for many different reasons, including that it was not collected, not in the right form or not collected consistently, in a difficult or unusable format, missing from the field, or simply not available (e.g., when a state or SU treatment provider would not send it). The RC and JJS partners were able to work through many but not all of these issues with the collaborating CS agencies and states. Although this was one of the largest multisite studies of CS agencies to date, JJ-TRIALS did not use a representative sample. However, the characteristics of the youth and agencies were diverse and similar to what was reported in a survey of CS agencies from a nationally representative sample of counties (Scott et al., in press). The fact that the national survey using only agency-level reports and this study using only CJS/SU Treatment records reach similar conclusions further strengthens confidence in the findings in spite of these limitations.

Recommendations for CS Agencies Going Forward

1. Review the 72 variables and their response sets identified by the JJ-TRIALS cooperative agreement and either adopt them or have clear rationales for why some of them may not be needed or a more detailed or different response set may be needed for your agency (note other/existing variables can be kept for other reasons if needed).

2. Set up a quality assurance protocol to review data completion and consistency in use.

3. Use electronic systems with automating checks (e.g., dates preceding current intake or after current date), simple recodes (e.g., skip outs), and data sharing with other systems (e.g., state, SU treatment providers) where possible.

4. Set up documentation on key terms, definitions, data management protocols/syntax, including how to generate and interpret reports for consistency over time and to address staff turnover/training; updating when changes are necessary or agreed upon).

5. Dedicate staff or set up a memo of understanding (MOU) to work with a local university or consultant to provide support to manage the above, generate reports and/or help CS agencies interpret reports/suggest alternatives, and evaluate them.

6. Encourage CS staff to gather and record treatment referral, initiation, and progress data on their clients in a consistent and systematic way (e.g., through training, incentives, and increasing the “user-friendliness” of data systems).

7. Have the youth on CS and their parent/guardian complete/sign a “limited release” or “disclosure of information” request at the time of referral that is good for at least 12 months.

The last step follows the regular system by which SU treatment agencies share data with each other and other health care providers whether by paper, fax, or electronically—which is important, since JJS is less likely to be the major funder of SU treatment than the Medicaid, state block grants, managed care, or private insurance providers.

Finally, there is also the question of whether or not to impute yes answers from missing data. If the goal is to examine and track improvements on these rates, the first method of actual yes answers vs. other is simpler, easier to explain, closer to rates from other sources, and likely the best to use. If the goal is to estimate the population size of youth in need or what will happen to it if practice is changed, then using both methods (as we have here) may give a better lower and upper bound. But as shown here, one must be aware that the imputed version may actually have lower rates in the later steps of the cascade due to larger denominators.

Conclusion

The hypothesized BH Service Cascade (Belenko et al., 2017) works in actual juvenile CS agencies as expected with the exception of the clinical assessment step—which can be collapsed into screening or dropped (as we have) with minimal to no consequence. When used across sites as done here, it is important to recognize and address the variations in data availability by site. In this example we can evaluate intervention designed to changes in the first 3 steps of the cascade (i.e., JJS CS screening, need, referral) with data from all sites. But to evaluate interventions designed to change the last 3 steps of the cascade (i.e., SU treatment initiation, engagement, and continuing care), the analyses have to be limited to the 19–21 sites with SU treatment data in their records. Future state/regional systems or research studies with multiple sites will need to similarly take into account data availability when evaluating the impact of other interventions.

APPENDIX.

Data Availability, Prevalence, and Site variation in the JJS Youth Records

	% Available	% Prevalence	% Site Min Prev.	% Site Max Prev.
Record Information
Episode Close Flag	77%	70%	7%	100%
Episode Close Date	53%	70%	7%	100%
Adjudication Flag	97%	35%	4%	100%
Adjudication Date	58%	62%	0%	100%
Record Update Date	100%	100%	100%	100%
Biological Testing
Biological Testing Flag	83%	45%	0%	86%
Alcohol Results	77%	0.2%	0%	5%
Amphetamine Results	83%	3%	0%	15%
Cannabis Results	83%	19%	0%	49%
Cocaine Results	83%	1%	0%	4%
Opioid Results	83%	1%	0%	3%
Other Drug Results	83%	3%	0%	13%
Substance Use Screening
Substance Use Screen Flag	85%	81%	24%	100%
SU Screen First Date	84%	81%	22%	100%
SU Screen Last Date	84%	80%	22%	100%
SU Screen Type	86%	—	—	—
Local measure	—	2%	0%	76%
CRAFFT	—	2%	0%	35%
GAIN-SS	—	1%	0%	51%
MAYSI-2	—	19%	0%	97%
SASSI	—	3%	0%	38%
YASI (from notes)	—	12%	0%	100%
PACT (from notes)	—	9%	0%	100%
Other instruments	—	31%	0%	98%
SU Screen Positive	83%	28%	2%	87%
Clinical Assessment
Clinical Assessment Flag	53%	45%	0%	100%
CA First Date	38%	23%	0%	100%
CA Last Date	32%	9%	0%	100%
CA Type	44%	—	—	—
Local measure	—	0.3%	0%	100%
ADI	—	0.01%	0%	0.1%
Child and Adol. Funct. Assess. Scale	—	0.1%	0%	95%
CRAFFT	—	0.1%	0%	0.7%
DISC - Other scales	—	0.02%	0%	0.2%
GAIN-Q3	—	0.3%	0%	27%
MAYSI-2	—	2%	0%	7%
SCID	—	12%	0%	70%
SASSI	—	1%	0%	13%
Other instruments or combinations	—	32%	0%	71%
CA Independent Flag	47%	20%	0%	100%
CA SU Positive	49%	5%	0%	99%
Other Source of Information on Needs
Other SU Positive	66%	—	—	—
No details given	—	64%	0%	100%
Judicial Mandate	—	0.2%	0%	6%
Other staff recommendations	—	12%	0%	55%
Undocumented need	—	0.2%	0%	3%
Other (describe in notes)	—	3%	0%	100%
Need from All Sources
Need Tx or CA Flag	88%	54%	13%	99%
Need Tx Flag	58%	11%	0%	40%
Referral to Substance Use Treatment
Referral To Tx or CA Flag	62%	24%	3%	100%
Referral to Tx Flag	54%	6%	0%	100%
Referral to Treatment Date	59%	20%	0%	100%
Substance Use Treatment
Treatment Flag	53%	16%	0%	100%
Treatment Intake Date	53%	15%	0%	100%
Treatment Discharge Date	51%	11%	0%	100%
Tx Level of Care	51%	—	—	—
Outpatient	—	10%	0%	58%
Intensive outpatient/day program	—	0.1%	0%	0.6%
Group home	—	0.0%	0%	0.2%
Residential/inpatient	—	0.8%	0%	15%
Other	—	0.1%	0%	2%
Treatment Type	50%	—	—	—
Local treatment program(s)	—	0.1%	0%	3%
MET/CBT	—	1.3%	0%	10%
MI	—	0.01%	0%	0.1%
Other SU Tx	—	9.0%	0%	51%
Demographics
Age	100%	—	—	—
0–10	—	1%	0%	4%
11–14	—	36%	18%	58%
15–17	—	63%	40%	81%
18+	—	1%	0%	8%
Date of Birth	90%	100%	100%	100%
Gender	100%	—	—	—
Female	—	27%	8%	39%
Male	—	73%	61%	92%
Hispanic	85%	21%	0%	100%
Race	98%	—	—	—
Asian/Hawaiian/Pacific Islander	—	1%	0%	6%
Black/African-American	—	47%	5%	96%
White/Caucasian	—	49%	4%	91%
Native American/Alaskan Native	—	0.2%	0%	2%
Other Race	—	2%	0%	15%
Mixed or Multiple Races	—	1%	0%	13%
Clinical Problems
Any SU Problem	56%	56%	14%	100%
Alcohol Problem	56%	9%	0%	45%
Cannabis Problem	47%	28%	0%	59%
Rx Drug Misuse Problem	47%	1%	0%	7%
Other Drug Problems	47%	7%	0%	25%
Tobacco Problems	35%	1%	0%	5%
Risky Sexual Activity	0.2%	2%	0%	100%
Risky Needle Activity	0.2%	0%	0%	0%
Victimization	40%	13%	0%	100%
Violence	32%	39%	0%	100%
Externalizing MH Problems	47%	19%	1%	86%
Internalizing MH Problems	51%	17%	1%	100%
Suicide Risk	49%	11%	1%	74%
Physical Health Problems	45%	2%	0%	14%
Serious Family Problems	38%	43%	0%	100%
Learning or Develop Disabilities	35%	9%	0%	48%
Charges
Violent Charge	99%	25%	2%	39%
Property Charge	99%	30%	6%	65%
AOD Related Charge	99%	14%	0%	31%
Probation/ Parole Violation	67%	12%	0%	31%
Weapons Offense	99%	7%	0.2%	23%
Other Status Offense	99%	11%	0%	51%
Other Charges	99%	32%	1%	64%
Charge Severity	93%	—	—	—
Felony	—	33%	12%	86%
Misdemeanor	—	56%	14%	87%
3 Summary/citation	—	3%	0%	22%
Status	—	3%	0%	44%
Other	—	5%	0%	60%
Risk of Recidivism
Recidivism Assessment Type	74%	—	—	—
Staff rating	—	0.1%	0%	2%
Local measure	—	8%	0%	100%
PACT	—	26%	0%	100%
YASI	—	15%	0%	100%
YLS/CMI	—	13%	0%	100%
Other measure	—	10%	0%	100%
Recidivism Risk Level	75%	—	—	—
Low	—	31%	0%	57%
Medium	—	32%	0%	60%
High	—	16%	0%	55%
Very high	—	3%	0%	29%
Justice System Status
Child In need of Supervision	78%	11%	0%	63%
Diversion	85%	5%	0%	37%
Probation	100%	49%	13%	100%
Parole	88%	0.2%	0%	1%
Juvenile Drug Court	80%	1%	0%	9%
Other Community Supervision	100%	67%	0.1%	95%
Detention	85%	30%	0%	83%
Other justice status	83%	7%	0%	62%

\a “Not Applicable/Skip” coding is treated as “available,” but not broken out as a % in prevalence.