Capacity of Juvenile Probation Officers in Low-Resourced, Rural Settings to Deliver an Evidence-Based Substance Use Intervention to Adolescents

Abstract

Substance use is a major public health problem with a host of negative outcomes. Justice- involved youth have even higher risks and lack access to evidence-based interventions, particularly in rural communities. Task-shifting, or redistribution of tasks downstream to an existing workforce with less training, may be an innovative strategy to increase access to evidence-based interventions. Initial findings are presented from a services research trial conducted primarily in rural communities in which an existing workforce, juvenile probation/parole officers (JPOs), were randomized either to learn and deliver contingency management (CM) or to continue delivering probation services as usual (PAU). This study used the prevailing version of CM for adolescents, i.e., family-based with behavior modification and cognitive behavioral components. Data included JPOs’ self-reports, as well as audio-recorded youth/family sessions with JPOs rated by expert and trained observational coders. Data also included ratings from a comparison study in which therapists were trained and supervised by experts to deliver CM to justice-involved youth/families. Results showed JPOs can feasibly incorporate CM into their services. When adherence of CM JPOs was compared against CM therapists, JPOs delivered significantly more cognitive behavioral components of CM and similar levels of behavior modification components of CM. These findings suggest that JPOs can be leveraged to provide evidence-based substance use interventions like CM in similar, or even greater, capacities to clinically trained therapists. This task-shifting approach could dramatically expand service access for these high-risk youth, particularly in rural areas where substance use services are limited or nonexistent.

Introduction

Substance use is one of the worst public health problems, costing the U.S. more than $700 billion each year (National Institute on Drug Abuse, 2017). In particular, youth substance use produces a range of deleterious short- and long-term outcomes for youth, their families, and communities (e.g., Gustavson et al., 2007; Malow et al., 2007; Ringel, Ellickson, & Collins, 2007); for instance, youth who use drugs have a four-fold greater probability of incarceration in young adulthood compared to youth without drug use (Slade et al., 2008). In the U.S., approximately two million youth report illicit drug use in the past month, and over one million annually are in need of substance use treatment (Substance Abuse and Mental Health Services Administration [SAMHSA], 2018). Tens of thousands of youth receive some form of treatment each year, although a staggering number in need of treatment never receive it (SAMHSA, 2018).

The juvenile justice system is particularly impacted. Justice-involved youth are at even greater risk for developing substance use problems compared to their non-justice-involved peers (Wasserman, McReynolds, Schwalbe, Keating, & Jones, 2010; Zajac, Drazdowski, & Sheidow, in press). Juvenile offenders who abuse drugs have more severe delinquent behavior, are more likely to recidivate, and are more likely to continue offending into adulthood (Schubert, Mulvey, & Glasheen, 2011; Wibbelink, Hoeve, Stams, & Oort, 2017). Additionally, substance use in juvenile offenders is linked to worse educational, vocational, and health outcomes (Chassin, Mansion, Nichter, & Losoya, 2014; Schubert et al., 2011; Tolou-Shams, Harrison, Hirschtritt, Dauria, & Barr-Walker, 2019; Zajac et al., in press). Up to three-quarters of detained youth meet criteria for at least one behavioral health disorder, the most common being substance use disorders (Teplin et al., 2013; Schubert et al., 2011). These disorders are persistent (Teplin et al., 2015), and rates continue to climb the further youth are processed in the juvenile justice system (Wasserman et al., 2010). Therefore, it is unsurprising that the juvenile justice system nationally represents the largest referral source for youth substance use treatment (SAMHSA, 2015).

In light of the high prevalence rates, devastating outcomes, and extraordinary costs associated with substance use among justice-involved youth, effective delivery of treatments to this group is imperative. Unfortunately, significant challenges remain in meeting treatment needs of justice-involved youth. A recent meta-analysis found that justice-involved adolescents have low prevalence rates of behavioral health service utilization overall, with less than a third of youth receiving substance-use related services at any point from pre-detention/incarceration through community reentry (White et al., 2019). The vast majority of these youth (85%) report at least one perceived barrier to accessing services (Teplin et al., 2013). Barriers include factors at individual (e.g., youth does not see treatment need), family (e.g., inadequate transportation or insurance), community (e.g., few services in proximity), and structural (e.g., justice agencies not prioritizing treatment or lack of funding) levels (Teplin et al., 2013; Zajac et al., in press). Beyond such barriers, most families are unlikely to have access to evidence-based interventions (National Center on Addiction and Substance Abuse [CASA], 2012).

Rural communities, in particular, are in dire need (Centers for Disease Control and Prevention, 2017). Rural Americans are more likely to experience a host of health problems compared to individuals in urban areas (U.S. Census Bureau, 2016; Henning-Smith, Hernandez, Ramirez, Hardeman, & Kozhimannil, 2019). The current opioid epidemic has highlighted the difficulties. Individuals living in rural areas begin misusing opioids at earlier ages (Monnat & Rigg, 2016) and they experience higher rates of opioid-related deaths (Mack, Jones, & Ballesteros, 2017). One explanation for these differences is the limited access to evidence-based treatment and prevention, including youth drug use services, in rural communities (Click, Basden, Bohannon, Anderson, & Tudiver, 2018). Indeed, rural areas consistently lack the finances and infrastructure to support delivery of evidence-based services. Often local hospitals are the only source of health care, and a 2018 report explained that 95 rural hospitals have closed since 2010 due to factors such as financial struggles and decreased demand for inpatient services (American Hospital Association, 2019). In Idaho (a primary site in the current study), counties commonly have few or no youth substance use treatment providers (M. Ingram, Idaho Statewide Juvenile Justice Judge, personal communication, August 24, 2017); thus, it would be difficult if not impossible for those counties to adopt evidence-based treatments. Compounding this, a 2018 budget shortfall forced Idaho’s state health agency to dramatically cut substance use treatment funding, making evidence-based treatment adoption even farther out of reach (Dutton, 2018).

Juvenile probation/parole officers (JPOs) across the nation, and particularly in rural areas, are on the front line of this crisis. As a large workforce indigenous to every community in the U.S., JPOs attempt to achieve positive outcomes with nearly one million new cases each year (Hockenberry & Puzzanchera, 2018). However, rates of repeat substance use and recidivism among the juvenile cases remain high (Sickmund & Puzzanchera, 2014). At the same time, JPOs have limited options for treatment referrals and little to no access to evidence-based services. One innovative strategy for low-resourced settings (i.e., areas where practices cannot be deployed by traditional clinicians), called “task-shifting,” involves redistribution of tasks downstream to an existing workforce that has less training compared to traditional specialized providers (Kakuma et al., 2011; World Health Organization [WHO], 2008). Task-shifting first emerged in response to the HIV/AIDS crisis in Africa, whereby midwives and community residents were effectively trained to provide basic interventions (e.g., take vital signs) due to a shortage of medical professionals (WHO, 2008). Since then, task-shifting has been successfully applied to target other health conditions (e.g., Institute of Medicine, 2010) and adult mental health problems (Patel et al., 2010). Across studies, findings indicate paraprofessionals can deliver interventions effectively without sacrificing quality or desired clinical outcomes.

Given JPOs’ intensive involvement and frequent contact with youth offenders and their parents, JPOs may be ideally positioned for task-shifting the delivery of an evidence-based substance use intervention, and particularly in rural settings. The adult justice system has attempted to have probation/parole officers facilitate treatment access and retention, resulting in reduced re-arrests and technical violations and increased treatment access (e.g., Bonta et al., 2011; Taxman, 2008). However, this approach limits officers to facilitating linkage to treatment rather than delivering an intervention themselves.

Currently, JPOs are left to target substance use with a few youth-focused strategies (e.g., community service, detention; Juvenile Justice Consortium, 2008). While JPOs’ job structure and level of training do not make it feasible for them to deliver a comprehensive, family-based treatment for justice-involved youth (e.g., Multisystemic Therapy [MST], Functional Family Therapy [FFT]; McCart & Sheidow, 2016), JPOs may be capable of delivering a highly specified, low complexity (relative to MST and FFT) intervention, such as Contingency Management (CM). As noted by Higgins, Silverman, and Heil (2008), CM is one of the most extensively validated substance use interventions for a range of drug types and populations. According to the major innovation diffusion models (e.g., Aarons, Hurlburt, & Horwitz, 2011; Lehman, Simpson, Knight, & Flynn, 2011), CM may be optimal for delivery by JPOs because of its relative low cost, feasibility of delivery during existing probation/parole sessions, compatibility with existing JPO practices (e.g., regular drug screens and imposing consequences, requiring parental participation), and amenability to gradual implementation (trialability).

In sum, JPOs effectively delivering an evidence-based intervention for substance use could dramatically expand service access for youth in the juvenile justice system, particularly in rural areas. We hypothesize that JPOs can deliver intervention advances such as CM, using task- shifting with an existing non-clinical workforce to fill a needed clinical services gap in low- resourced environments. Data from a large-scale randomized trial in primarily rural communities were used to determine (1) the capacity of JPOs to deliver CM with substance using adolescent probationers/parolees and (2) the level of CM adherence for JPO providers relative to that of therapist providers. Of note, the CM protocol used is based on the Community Reinforcement Approach (CRA; Budney & Higgins, 1998), with a strong emphasis on parent involvement and incorporation of both behavior modification and cognitive behavioral components. Given that JPOs work in every jurisdiction in the U.S., the success of this task-shifting approach could have widespread impact in tackling the drug abuse epidemic.

Method

Design Overview

The ongoing parent project is using a randomized design to determine if JPOs can deliver CM, as well as if JPO-delivered CM improves substance use and recidivism. JPO and youth participants are being recruited from seven primarily rural counties in Idaho and Nevada. The current paper focuses on the first aim of the parent study. Specifically, JPOs were randomized either to deliver CM or to continue delivering probation services as usual (PAU). Since some aspects of PAU were thought to overlap with CM (drug screening, rewards, parent involvement), CM delivery was assessed in both conditions. The present paper compares levels of CM delivery in these conditions. Then, JPOs’ CM delivery is compared to CM therapists from a prior study.

Participants and Procedures

All research procedures were approved by the Institutional Review Board at the first author’s institution. Figure 1 depicts the flow from participant enrollment through data analysis. Researchers visited each participating county to recruit the JPO participants. The research team described all aspects of the study, emphasizing its voluntary nature and that participation would have no impact on JPOs’ job performance evaluations. Of the 35 JPOs approached for recruitment, all (100%) consented and completed a baseline demographics survey. Following recruitment, JPOs were randomly assigned to the CM (n = 17) or PAU (n = 18) conditions.

Figure 1.

CONSORT flow diagram. JPO = Juvenile Probation/Parole Officer. CM = Contingency Management. PAU = Probation as Usual.

Univariate analyses indicated that JPOs in the two conditions did not differ on demographic or professional characteristics (all p-values > .05), so characteristics are reported for the sample as a whole. Participating JPOs averaged 41.6 years of age (SD = 10.5, range = 26–61), and 54% were male. They were 91% White and 9% American Indian; 23% were Hispanic (of any race). Regarding highest degree earned, 9% had a master’s degree, 49% a bachelor’s degree, 17% an associate’s degree, and 26% had only a high school diploma. JPOs reported working in the juvenile justice field for an average of 12.6 years (SD = 7.6; range 1–26), and their mean caseload size was 26.1 youth. JPOs were asked about prior training experiences. The vast majority (74%) reported having been trained in motivational interviewing. The next most commonly reported training was case management (60%), followed by restorative justice (49%), mindfulness (23%), behavior modification (20%), and cognitive-behavioral interventions (9%). Pertinent to the aims of this study, none of the participating JPOs reported prior training in CM.

JPOs assigned to the CM condition engaged in initial CM training (described later). Once that was completed, youth recruitment began at each site. To be included, youth needed to meet inclusion criteria: (1) 12–18 years of age, (2) newly opened probation case (could have previous cases), and (3) presence of DSM-5 substance use disorder. To enhance generalizability, exclusion criteria were limited to presence of pervasive developmental disorder, active psychotic disorder, or intellectual disability. Designated site liaisons identified youth who appeared to meet inclusion criteria and referred them to research staff for a more thorough eligibility assessment (e.g., a diagnostic interview was used to confirm presence of a substance use disorder).

Once eligibility was established, research staff solicited informed consent from the youth and his/her legal guardian. Specifically, families consented to: (1) random assignment of the youth to the caseload of a JPO in either the CM or PAU condition, (2) JPO submission of audio recorded family/youth sessions for coding by the research team, (3) JPO submission of a CM Adherence Checklist following each session, (4) youth and parent completion of quarterly research assessments through 9 months to provide confidential information on the youth’s substance use and other behaviors (families received $20 for completing each of these assessments), and (5) research staff collection of archival arrest and detention records for 18 months pre- and post-baseline. To examine JPOs’ CM delivery, data from coded sessions and CM Adherence Checklists (completed by JPOs) are utilized. This study is ongoing and a separate, future report will measure the impact of CM delivery on youth outcomes.

Thus far, 14 JPOs in the CM condition and 13 JPOs in the PAU condition have had the opportunity to deliver their respective interventions to participating families (see Figure 1). Some JPOs in each condition have not yet had the opportunity to serve a study family due to the ongoing nature of family recruitment. Characteristics of the participating study families are summarized next. The target youth in these families averaged 15.5 years of age (SD = 1.29; range = 13–18) and 70% were male. Racial breakdown of youth was 90% White, 2% African American, 2% American Indian, and 6% Biracial; 47% were Hispanic (of any race). Approximately 30% lived with both biological parents, 30% lived with a biological parent and another adult, and 40% lived with other relatives. Socioeconomically, median annual household income was in the $30,000-$40,000 range, 41% of families were receiving public financial assistance, and the median educational attainment of the primary parent was a high school diploma. Of the 14 JPOs in the CM condition who delivered services to study families, 13 provided adherence data (i.e., coded sessions, CM Adherence Checklists). Likewise, the 13 JPOs in the PAU condition who intervened with study families provided adherence data. All available adherence data was included in the statistical analyses, described subsequently.

Intervention Conditions

Family-Based CM.

The family-based CM model used in this study has been refined through several iterations (e.g., Azrin, Donohue, Besalel, Kogan, & Acierno, 1994; Henggeler et al., 2006; Henggeler, Chapman, Rowland, Sheidow, & Cunningham, 2013) and is based on a published intervention manual (Henggeler et al., 2012). A variation of CRA (Budney & Higgins, 1998), this protocol has the greatest evidence supporting use with adolescents (summarized in Henggeler et al., 2012), and active parent involvement is further supported by a meta-analytic review of child psychotherapy studies (Dowell & Ogles, 2010), as well as substance use treatment research (e.g., Hogue, Henderson, Becker, & Knight, 2018). This CM model uses both behavior modification and cognitive behavioral strategies, which have garnered most of the support in the substance use treatment field when delivered within family-based approaches (Hogue et al., 2018). However, the strategies are well-specified and multiple tools (session checklists, handouts, and worksheets) facilitate intervention delivery.

The CM intervention components, all of which involve parental involvement, include the following: (1) CM is introduced to the youth and parent(s), and attempts are made to engage them in the intervention. (2) Functional analyses identify the youth’s triggers. (3) Individualized triggers identified in the functional analyses are targeted via self-management planning and drug refusal skills training, conducted in collaboration with the youth and parent(s). (4) Concurrently, parent(s) are taught to conduct random urine drug screens in the home. Screens also are conducted at every scheduled office visit. Finally, youth are called into the office randomly to complete screens. (5) A contingency contract, described below, is developed that provides youth with rewards for negative drug screens and disincentives for positive screens. (6) Plans are made with the youth and parent(s) for sustaining youth abstinence after CM ends.

The contingency contract follows a well-specified protocol. First, the CM provider, youth, and parent(s) generate a menu of rewards that can effectively compete with the youth’s substance use. The CM provider ensures there is a balance between natural incentives parents can provide (e.g., cell phone access, later curfew, “vacation” from chores, special privileges or activities) and tangible items (JPOs in the CM condition had access to up to $425/youth to help families with purchasing drug screens and rewards). To assist in identifying incentives with the best odds of competing with substance use, rewards are rank ordered by the youth’s preference. From this reward menu, the youth’s “most valued privilege” (MVP) is chosen, which is a natural incentive that can be provided by the parent(s) regularly. Remaining menu items are assigned point values by the CM provider and parent(s), usually 1 point ~ $1. Once the menu is finalized, a point-and-level system is implemented, and the youth receives a 50-point starting balance. The youth earns or loses MVP access from the parent(s) depending on each drug screen result. During the first month (Level 1a of the contract), youth keep their points if they have negative screens but lose 12 points for each week they test positive. Regardless of screen results, youth cannot redeem points in this period. From the fifth week on, negative screens result in youth being able to earn additional points and also to use points to “purchase” items on their reward menu. To provide greater incentive (i.e., escalating reinforcer), the number of points that youth can earn each week starts at 12 (Level 1b), increases to 24 after 8 consecutive weeks of negative screens (Level 2), and then increases to 36 after 10 consecutive weeks of negative screens (Level 3; rapid reset also is used). When making a purchase, youth may use as many points as they like from their balance. Rewards are provided as immediately as possible, consistent with behavioral principles. If a youth has a positive screen from the fifth week on, the youth does not earn points and cannot make a “purchase” until the next negative screen. As treatment progresses, emphasis shifts to incentives provided exclusively by the parent(s) to sustain abstinence and also expansion of the contract to include any other behaviors the parent(s) want to target.

Probation as usual.

The JPOs in Idaho and Nevada use a “Balanced Approach” philosophy to juvenile justice services. This includes equal emphasis on holding youth accountable through community restoration while ensuring public safety and developing competencies of the youth to help them become a contributing member of the community; and thus reduce the likelihood of recidivism (Office of Juvenile Justice and Delinquency Prevention [OJJDP], 1998; Wilson, Olaghere, & Kimbrell, 2017). For example, during sessions JPOs review probation conditions (e.g., payment of fees, following school and home rules) and implement graduated sanctions (e.g., community service, house arrest) when conditions are not being met. Additionally, it is common for JPOs to discuss attendance in substance use or mental health treatment if required/recommended and sign the youth up for educational seminars on effects of drugs (e.g., driving under the influence courses). JPOs also assess different domains in the youths’ lives including home and peer relations and educational/vocational goals. Parents are occasionally present, but may not be routinely engaged by the JPOs in these discussions. Additionally, JPOs check on results of drug screens conducted by treatment agencies, but do not routinely do their own testing, though this varies by agency. The “Balanced Approach” is quite common in juvenile justice systems across the U.S. (Wilson et al., 2017). Throughout Idaho and Nevada, JPOs carry a range of caseloads, but all fall under 30 cases, consistent with national numbers (Torbet, 1996).

CM Training and Quality Assurance

For JPOs in the CM condition, a multifaceted approach was used to train and support their delivery of CM. First, JPOs completed a web-based CM computer assisted training (CM- CAT), developed by the first author and used successfully in previous CM training studies (e.g., Henggeler et al., 2013). It includes modules corresponding to each CM component. Within each module, sections described intervention steps, give troubleshooting tips, and provide sample scripts and video examples. Each module can be completed in one to two online sessions, depending on activities involved (e.g., some sessions include homework), and it takes the average user under eight hours to complete all modules. The website’s navigation guides users through the modules, and a passing score of 80% is required on a multiple-choice test to access the next module. Trainees failing to pass a test are not redirected to specific content for missed items. However, they are able to re-review the module and/or access support from a CM expert, and can take the post-tests as many times as needed to achieve a passing score. Among the JPOs, the average number of testing attempts before passing was 1.5 for the modules corresponding to cognitive behavioral strategies (functional analyses and self-management planning) and 3.6 for the modules corresponding to behavior modification strategies (drug testing and contingency contracting). To reduce burden, JPOs in the CM condition were given one month to finish the CM-CAT, and all completed successfully in that timeframe. However, JPOs had access to the CM-CAT for the full duration of the study and could revisit module content at any time.

Second, JPOs in the CM condition were provided a hard-copy manual that included extra CM handouts and forms organized for ease of use during CM sessions. Third, JPOs were provided access to a web-based, searchable portal with printable tools and materials related to CM and for frequently asked questions to be posted. Fourth, JPOs began implementing CM with study families while receiving ongoing monitoring and consultation from CM experts (the first, second, and fourth authors). Specifically, the experts conducted 30–60 minute monthly group support meetings via webinar with CM JPOs at each site. Experts also conducted 1:1 phone calls with individual JPOs, as needed. The purpose of these webinars/calls was to provide modeling of CM, planning for subsequent sessions, and feedback to JPOs on their CM delivery (experts reviewed a random selection of JPOs’ audio-recorded CM sessions prior to the webinars/calls).

Instruments

CM Adherence Checklist.

For the purpose of this study, a checklist was developed for JPOs to provide ratings on their delivery of CM components to families. This form is part of a larger family of CM adherence monitoring tools created by the research team. The original 34- item CM Adherence Checklist was developed using Rasch-based methods and validated across several studies (Chapman, Sheidow, Henggeler, Halliday-Boykins, & Cunningham, 2008). Items measure the cognitive behavioral (functional analyses, self-management planning) and monitoring (drug screening, consequences) components of CM, based on provider, youth, and parent reports. However, to reduce burden on the JPOs, the tool was shortened to include only the 16 items that exhibited the best psychometric properties in the Rasch-based analysis (Chapman et al., 2008). JPOs completed this checklist following each family/youth session.

Checklists were provided by 25 JPOs for 57 families, with an average of 2.28 families (SD = 1.34) per JPO. Across families, the average number of CM Adherence Checklists was 8.82 (SD = 6.12), and a total of 503 Checklists were provided (all were analyzed). The CM and PAU groups did not differ significantly on the number of families per JPO, β = 0.58, SE = 0.53, t (23) = 1.09, p = .286, 95% CI [−0.46, 1.63], but the CM group submitted a significantly larger number of Checklists per family, β = 6.31, SE = 1.72, t (23) = 3.68,p = .001, 95% CI [2.95, 9.68].

JPO-CM Observational Coding System.

Three trained coders, as well as two CM experts, independently listened to audio recordings of the JPO sessions and provided ratings on a CM Adherence Coding Form. This form was derived from the original 34-item CM Adherence Checklist, described previously. For the CM Adherence Coding Form, items are used to obtain observational ratings on whether or not the different CM components were delivered.

The two CM experts were the first and second authors. Training for the three observational coders matched procedures used successfully in previous CM coding studies (e.g., Chapman, McCart, Letourneau, & Sheidow, 2013). The three observational coders completed the CM-CAT, followed by 2 hours of in-person training on observational coding. The in-person training included detailed review of the CM Adherence Coding Form and group practice ratings using taped examples. Next, coders independently rated session tapes previously rated by the experts. Coders were “certified” after ratings achieved acceptable consistency (≥80% agreement) with experts’ ratings. Quarterly booster trainings were held to monitor and address coder drift.

A subset of tapes was selected for coding, using a carefully devised tape selection protocol to ensure balanced assignment across condition, JPOs, youth, and coders. Sessions were coded for 25 JPOs with 49 families, with an average of 1.96 families (SD = 1.21) per JPO. Across families, the average number of coded sessions was 2.94 (SD = 1.91), and a total of 144 unique sessions were coded. Additionally, 10 double-coded sessions were retained for scoring. The CM and PAU groups did not differ significantly on the number of families per JPO, β = 0.72, SE = 0.47, t (23) = 1.53, p = .140, 95% CI [−0.20, 1.64], or the number of sessions coded per family, β = 0.23, SE = 0.56, t (23) = 0.41,p = .689, 95% CI [−0.87, 1.32].

Comparison data for CM Adherence Checklist and Observational Coding System.

A prior study was used to compare CM adherence for JPOs to that of therapists. The comparison study (Letourneau, McCart, Sheidow, & Mauro, 2017) included therapists conducting CM with youth and families involved in juvenile drug courts located in suburban and urban areas. For that study, CM was combined with a sexual risk reduction intervention to reduce HIV risk (the JDC- HIV trial). The therapists had master’s degrees in clinical social work and possessed three or more years of direct clinical experience. They were trained via the CM-CAT, followed by a 12- hour in-person workshop conducted by CM experts (the first and second authors). Further, these experts provided therapists with regular weekly supervision to promote high levels of protocol adherence. Regarding CM-CAT post-test performance, therapists averaged 1.0 attempts before passing the modules corresponding to cognitive behavioral strategies and 1.2 attempts before passing the modules corresponding to behavior modification strategies. Data from this clinical trial was parallel to the present study: provider self-reports on the CM Adherence Checklist, as well as ratings from trained observational coders and CM experts (Chapman et al., 2013). Thus, the JDC-HIV trial provided an optimal comparison for comparing different types of providers (i.e., non-clinically trained JPOs to clinically trained therapists).

In the JDC-HIV study, the 34-item CM Adherence Checklist was completed by CM therapists (N = 3), trained observational coders (N = 5), and CM expert coders (N = 2). Sessions were only rated for the CM condition in the JDC-HIV study (i.e., not the control condition). The CM therapists treated a total of 27 families (14, 8, and 6 each). All sessions included a CM Adherence Checklist. Of these sessions, 61% had 3, 4, or 5 sessions rated by coders, and the remaining 39% had between 6 and 9 sessions rated. Across families, this resulted in 148 sessions coded. One session was rated by one coder, 66% were rated by 2 to 4 coders, with the remaining 33% rated by 4 to 7 coders, which resulted in a total of 581 session ratings.

Data Analysis Strategy

Evaluating dimensionality.

Prior to the primary analyses, and informed by previous findings (Chapman et al., 2008; Henggeler, Sheidow, Cunningham, Donohue, & Ford, 2008), dimensionality was evaluated to determine the most appropriate scoring procedure for CM adherence. The analyses used data from the combined JDC-HIV and JPO-CM samples. The goal was to determine whether there was a single dimension or multiple distinct dimensions, and this was accomplished using two models. First, a dichotomous Rasch measurement model was performed, implemented in WINSTEPS (Linacre, 2019), focusing specifically on the principal component analysis of standardized Rasch item-person residuals (Smith, 2002). In the residuals—which should reflect random noise—the analysis attempts to identify meaningful structure, as this could reflect additional dimensions. The primary indicator was the eigenvalue for the first contrast, with values greater than 2.0 suggesting non-trivial dimensionality. Item content was also considered. Second, the suspected dimensions were tested using an IRT-based item bifactor measurement model (Gibbons et al., 2007), implemented in IRTPRO (Cai, Thissen, & du Toit, 2015). With this model, all items load on a general factor and one of multiple specific factors. Item loadings indicate whether the suspected dimensions primarily reflect a general dimension or multiple specific dimensions (Reise, Moore, & Haviland, 2010).

The principal component analysis of standardized Rasch item-person residuals indicated that dimensionality was non-trivial, with an eigenvalue of 4.7 for the first contrast. Item loadings broadly identified two potential dimensions: one with items related to behavior modification and the other with items related to cognitive behavioral aspects of the family-based CM model used in this study. This was consistent with prior studies, and the item bifactor model was specified with a general factor and two specific factors (i.e., for the suspected dimensions). The results supported distinct dimensions, with loadings on the specific dimensions that, particularly for behavior modification, were generally stronger than loadings on the general dimension. Thus, both analyses supported two dimensions: Behavior Modification (BM) and Cognitive Behavioral (CB). For analyses specific to the JPO-CM sample, raw score averages were computed for BM and CB, with scores ranging 0 to 1 and reflecting the proportion of CM components delivered. As detailed below, analyses for the combined samples required a different scoring procedure.

Evaluating CM adherence for JPOs.

The capacity of JPOs to deliver CM was evaluated using BM and CB scores from the CM Adherence Checklist and Observational Coding System. The number of observations varied by source, but the data structure was consistent, with repeated measurements (level-1) within families (level-2) within JPOs (level-3). Nesting was addressed using mixed-effects regression models (Raudenbush & Bryk, 2002) implemented in HLM (Raudenbush, Bryk, & Congdon, 2013). The outcomes were raw average scores for the BM and CB sub-scales. An initial unconditional model (i.e., no predictors) estimated variance components and ICCs. To compare CM and PAU, an important consideration was the approach to modeling repeated measurements as there were two potentially relevant timelines: the family’s timeline and the JPO’s timeline. With the focus on evaluating JPO performance, the latter was used. That is, as JPOs gain experience with delivering CM, they may develop higher adherence. However, across outcomes, a linear polynomial indicated that scores did not change significantly. As such, this term was removed. To test for differences between conditions, a dummy-coded indicator (0 = PAU, 1 = CM) was entered at JPO level. This formulation tested for differences between CM and PAU in the overall average level of adherence.

Scoring CM adherence across studies.

To benchmark the performance of JPOs against therapists, it was important to score the different instrument versions on the same scale of measurement. To achieve this, an IRT-based common item equating procedure was used (Wolfe, 2000). With this approach, all data sources were combined in a single file, with cases “linked” by the items common across sources. The items unique to each source were also retained and treated as missing when not applicable. The resulting file included three instrument versions from two studies (i.e., JPO-CM CM Adherence Checklist, Observational Coding System, and JDC-HIV CM Adherence Checklist). JPO-CM provided JPO self-reports, ratings from trained observational coders, and ratings from CM experts across both the PAU and CM conditions. As the benchmark, JDC-HIV included therapist self-reports, ratings from trained observational coders, and ratings from CM experts, all of which pertained to the CM condition. The data were calibrated using a Rasch measurement model implemented in WINSTEPS (Linacre, 2019). The resulting logit-based person measures (i.e., “scores”) reflect the level of adherence for each session on a common scale of measurement across data sources. To facilitate interpretation, a score conversion table was used to translate logit-based measures into raw score units.

Comparing CM adherence for JPO and therapist providers.

CM adherence for JPOs and therapists was compared using the logit-based Rasch measures just described. The data were structured with repeated measurements (level-1) within families (level-2) within providers (level- 3). The combined data had 1238 observations across 83 families and 29 providers. The models included indicators to differentiate the type of rater (two indicators: 0 = trained observational coder, 1 = provider self-report; 0 = trained observational coder, 1 = CM expert), study (0 = JDC- HIV, 1 = JPO-CM), and study condition (0 = PAU, 1 = CM). Of note, because the JDC-HIV study did not provide usual services data, the condition indicator was entered only as an interaction with study. This formulation compares CM and PAU for the JPO-CM study. Controlling for the type of rater, this formulation tested for differences in BM and CB adherence across JPOs delivering CM, JPOs delivering PAU, and therapists delivering CM.

Results

CM Adherence for CM JPOs Relative to Probation as Usual

CM Adherence Checklist.

JPO self-reports of BM and CB adherence were compared for CM and PAU using three-level mixed-effects regression models with repeated measurements (level-1) within families (level-2) within JPOs (level-3). The unconditional model indicated that, of the total variance in BM scores, the percentage attributable to sessions, families, and JPOs was 40%, 3%, and 57%. The intercept indicated that, in an average session, 30% of BM components were delivered, β = 0.300, SE = 0.046, t (24) = 6.50,p < .001, 95% CI [0.209, 0.389]. For CB adherence, the variance attributable to sessions, families, and JPOs was 67%, 9%, and 24%, and on average, 12% of the CB components were delivered, β = 0.118, SE = 0.023, t (24) = 5.14, p < .001, 95% CI [0.073, 0.163]. The condition indicator was then added to test for differences in the overall average level of adherence for CM versus PAU. Adherence was significantly higher for the CM group, both for BM, β = 0.320, SE = 0.068, t (23) = 4.69, p < .001, 95% CI [0.186, 0.453], and CB, β = 0.155, SE = 0.035, t (23) = 4.366, p < .001, 95% CI [0.085, 0.224]. For BM, the PAU group delivered 14% of components, whereas the CM group delivered 46%; and for CB, the PAU group delivered 4% versus 19% for the CM group (Figure 2).

Figure 2.

Percentage of Contingency Management (CM) components delivered by Juvenile Probation/Parole Officers (JPOs) in the CM and Probation as Usual (PAU) conditions based on each instrument.

Observational coding.

The same series of models was performed for BM and CB adherence based on ratings provided by trained observational coders and CM experts. The unconditional model for BM indicated that 31%, 13%, and 56% of the variance was attributable to sessions, families, and JPOs. In an average session, 25% of the BM components were delivered, β = 0.246, SE = 0.051, t (24) = 4.83, p < .001, 95% CI [0.146, 0.346]. For CB, 87%, <1%, and 12% of the variance was attributable to each nesting level, and on average, 8% of the CB components were delivered, β = 0.084, SE = 0.014, t (24) = 5.98, p < .001, 95% CI [0.057, 0.112]. With the condition indicator added, adherence scores were significantly higher for the CM group, both for BM, β = 0.430, SE = 0.058, t (23) = 7.40, p < .001, 95% CI [0.316, 0.544], and for CB, β = 0.099, SE = 0.021, t (23) = 4.77, p < .001, 95% CI [0.058, 0.140]. For BM, the PAU group delivered 3% of components, whereas the CM group delivered 46%; and for CB, PAU delivered 3% versus 13% for the CM group (Figure 2).

JPO CM Adherence Relative to Therapist CM Adherence

The next models compared BM and CB adherence for JPOs and therapists. The data were combined across the JDC-HIV and JPO-CM studies, and because of the equating procedure, the resulting BM and CB outcomes are scaled in logits. Based on the unconditional model for BM adherence, the percentage of variance attributable to sessions, families, and providers was 49%, 5%, and 46%, respectively; and for CB, the percentages were 68%, 14%, and 18%. For an average session, the level of BM adherence was −1.81, which corresponds to ~20% of the BM components being delivered, p = −1.810, SE = 0.291, t (28) = −6.22,p < .001, 95% CI [−2.380, − 1.239]. For CB, the average of −3.59 corresponds to ~6% of the components being delivered, β = −3.590, SE = 0.182, t (28) = −19.68, p < .001, 95% CI [−3.948, −3.233].

To compare BM and CB adherence for CM JPOs, PAU JPOs, and CM therapists, indicators were included to differentiate the studies, conditions, and types of raters. The results are reported in Table 1. For BM, there were overall differences between the three types of raters. Providers (i.e., JPOs and therapists) self-reported significantly higher levels of adherence relative to CM experts’ ratings of audio recordings (95% CI [0.008, 0.602]) and trained observational coders (95% CI [0.532, 0.945], [0.152, 0.716]). Controlling for the type of rater, CM JPOs had significantly higher levels of BM adherence compared to PAU JPOs (95% CI [1.599, 3.256]) but not CM therapists (95% CI [−0.649, 1.764]), and PAU JPOs had lower levels of BM adherence compared to CM therapists (95% CI [−3.083, −0.656]). The predicted percentage of BM components delivered by CM therapists, depending on the type of rater, was approximately 22% to 33%. For PAU JPOs, it was 6% to 11%, and for CM JPOs, it was 33% to 44%.

Table 1.

Estimates from Mixed-Effects Regression Model Comparing CM Adherence in JPO-CM and JDC-HIV Studies

	Behavior Modification				Cognitive Behavioral
	β	SE	t	p	β	SE	t	p
Intercepta,b	−1.728	0.540	−3.20	.004	−3.987	0.232	−17.20	<.001
JPO-CM, PAUa	−1.870	0.619	−3.02	.006	−0.510	0.308	−1.66	.110
JPO-CM, CM vs. PAUa	2.427	0.423	5.74	<.001	1.556	0.268	5.82	<.001
Providerc	0.739	0.105	7.01	<.001	0.210	0.107	1.97	.049
Treatment Expertc	0.434	0.144	3.01	.003	0.287	0.146	1.96	.050
Planned Contrasts	Est.	SE	χ2	p	Est.	SE	χ2	p
Provider vs. Expertd	0.305	0.152	4.04	.042	−0.077	0.154	0.25	>.500
JPO-CM vs. JDC-CMd	0.558	0.616	0.82	>.500	1.046	0.292	12.80	.001
Variance Components	Est.	SD	DF	p	Est.	SD	DF	p
Session	2.155	1.468			2.213	1.488
Family	0.251	0.501	57	<.001	0.469	0.685	57	<.001
Therapist	0.826	0.909	26	<.001	0.090	0.299	26	.089

Note. JPO = Juvenile Probation/Parole Officer. CM = Contingency Management. PAU = Probation as Usual. The three-level mixed-effects model was structured with repeated sessions nested within families nested within therapists. The model includes a main effect for the JPO- CM study and interaction between this term and CM condition indicator, with the resulting estimate reflecting the difference between the CM and PAU conditions in the JPO-CM study. Model estimates are scaled in logits, and 95% CIs for fixed effect estimates are provided in text.

^aReference group is JDC-HIV study (CM condition) with ratings provided by trained observational coders.

^bDF = 26.

^cDF = 1121.

^dDF = 1.

CB adherence, compared to observational coders, was significantly higher based on selfreports from providers (95% CI [0.001, 0.419]) and ratings from CM experts (95% CI [0.001, 0.574]). Providers did not differ significantly from CM experts (95% CI [−0.379, 0.225]). Controlling for the type of respondent, CM JPOs had significantly higher levels of CB adherence relative to PAU JPOs (95% CI [1.031, 2.080]) and CM therapists (95% CI [0.473, 1.619]). PAU JPOs did not differ significantly from CM therapists (95% CI [−1.113, 0.094]). The predicted percentage of CB components delivered by CM therapists, across raters, was approximately 6%. For PAU JPOs, it was also 6% across raters, and for CM JPOs, it ranged from 9% to 12%.

Discussion

Despite clear evidence of effectiveness, very few youth and families across the U.S. have access to CM (Hartzler, Lash, & Roll, 2012). To reduce the impact of drug use problems in communities, access to evidence-based services for youth must increase. Justice-involved youth have greater risk for substance use problems and higher need for effective interventions to prevent deleterious consequences, but they also have greater barriers to accessing treatment. Rural communities are particularly devastated by the drug use epidemic, but also lack resources for effective interventions to be delivered. So, while there are proven methods for increasing the uptake of CM among community-based therapists (e.g., Henggeler et al., 2013), many communities will not be able to benefit from these advances. Therefore, we aimed to use task-shifting as an innovative strategy for low-resourced environments to increase access. Specifically, we used data from an ongoing services research study to examine the feasibility of an indigenous non-clinical workforce, JPOs, to deliver an evidence-based clinical intervention, CM, for youth substance use. So far as we are aware, this is the first study to attempt task- shifting of a full clinical intervention for substance use (or other clinical disorders) to JPOs.

Although some aspects of JPOs’ routine work involve elements of CM, results across all measurement methods show it is feasible for JPOs to incorporate greater CM into their services. That is, JPOs randomly selected to conduct CM showed significantly higher CM levels versus JPOs randomly assigned to continue providing usual probation services. In observational coding of actual youth/family sessions, CM JPOs delivered 4 times more cognitive behavioral components of CM versus PAU JPOs, while they delivered 15 times more behavior modification components of CM. This latter difference is especially remarkable given JPOs’ work focuses on behavior and many JPOs in the study reported past training in behavior modification.

While JPOs trained and supported to conduct CM were able to achieve higher CM adherence compared to JPOs continuing to conduct usual probation services, a further examination of task-shifting was conducted: the level of CM adherence for JPO providers was compared to that of therapist providers. Surprisingly, CM JPOs demonstrated significantly higher levels of adherence compared to therapists when it came to the cognitive behavioral components of CM. Although the predicted percentages for the behavior modification components followed the same general pattern, the difference was not statistically significant. The youth in these separate studies were remarkably similar in terms of justice-involvement and distributions for age, sex, Hispanic ethnicity, family income, and living with parents. The current study was rural and had more white youth, while the study with therapists was suburban and urban and had more African American youth. On the face of it, these factors do not seem like they would drive differences in the providers’ ability to use CM strategies. This will continue to be examined as the sample size increases and more data are collected.

The fact that JPOs may be able to achieve higher CM adherence relative to highly trained and monitored clinicians is unexpected. However, the authors’ experience is that the JPOs have eagerly agreed to participate (which includes a 50/50 chance they will be randomized into the CM condition), and JPOs assigned to CM have been excited to learn tools for helping youth and families with substance use. Indeed, almost one-third of justice-involved youth in Idaho are rearrested for a new crime within 12 months of completing their court sentence (Idaho Department of Juvenile Corrections, 2018), signifying a critical need; JPOs may view targeting substance use as an essential step for reducing recidivism in youth. While the therapists also certainly had a desire to address youth substance use, a few hypotheses are presented for the group differences. First, as master’s level clinicians, the therapists already possessed a range of clinical tools (e.g., family therapy techniques, communication strategies) prior to their training in CM. JPOs, on the other hand, possessed few if any clinical tools prior to learning CM. Thus, when working with families, perhaps therapists were more likely to pull from their broad toolkit, whereas JPOs were limited to the single CM tool and thus used it more frequently. Second, and as noted previously, the JDC-HIV trial had a dual focus on both youth substance use and HIV-related sexual risk behaviors. As a result, the therapists were incorporating other study interventions (e.g., condom use, skill building, education about STDs) into their ongoing CM delivery, whereas the JPOs were focused exclusively on CM delivery within their existing probation services.

The overall levels of CM components were low (well below 100%), but it is worth noting that one would not expect 100% of these to be completed in a single session. This is a known challenge with measuring treatment fidelity (Schoenwald et al., 2011). For example, a CM provider may focus on self-management planning for most of a session, resulting in no instances of other cognitive behavioral or behavior modification components. In particular, cognitive behavioral components cover strategies that are introduced and focused on at varied points in CM (e.g., functional analyses, self-management planning, drug refusal skills training). It also is worth noting that the behavior modification strategies (e.g., reward menu development, drug testing, provision of points/rewards) are started early to improve a youth’s motivation to engage. Thus, it is reasonable to see higher overall levels of behavior modification compared to cognitive behavioral components. Regardless, it appears that CM JPOs were consistently able to cover more components in single sessions compared to CM therapists. What is unknown, however, is what level of CM components is meaningful and generates youth outcomes, as well as whether a more in depth focus on fewer components in a single session is better. These are important questions that can be examined once the present study has accrued enough youth outcome data.

The rural areas where this study has been conducted are particularly interesting given their lack of resources; it also may be a reason for such success in this study. Prior to conducting the trial, the authors met with dozens of JPOs in primarily rural areas to learn about their needs. JPOs noted that the vast majority of their youth needed substance use treatment. However, JPOs reported significant frustrations with community substance use services, primarily due to access (e.g., too few providers including one therapist covering five counties; family transportation issues and too many other appointments) and policies (e.g., youth in mental health care may not simultaneously be able to obtain substance use care; missing a session may result in immediate discharge given long waitlists). In light of these barriers to outpatient treatment, JPOs said they regularly refer youth to residential treatment facilities. However, they also described extreme dissatisfaction with this approach, noting that youth languish in short-term detention or remain in the community without treatment until a bed becomes available. These were the types of barriers we aimed to target with the present study.

Of note, training and supporting providers in rural, distal communities is made feasible in the present study by a web-based training support system developed as part of other research (e.g., R44DA033745). Critically, it is not a one-time training workshop or training without ongoing monitoring and support, both of which are known barriers to effective uptake of evidence-based substance use interventions (e.g., Beidas & Kendall, 2010; Petry, DePhilippis, Rash, Drapkin, & McKay, 2014). Added benefits of this training and support system are that it is more cost effective for reaching rural areas and improves adoption potential if this task-shifting approach is proven effective. Further, JPOs trained with this web-based training and support system achieved higher adherence than therapists, even though therapists received the “gold standard” of in-person training and supervision directly from CM experts.

A fundamental question the study was investigating was whether JPOs would even try doing CM; the authors’ anecdotal experience is that JPOs showed a keen interest in engaging parents into the work they were doing, so the family-based CM was particularly well-suited to this interest. JPOs place a high value on parents’ involvement in the probation/parole process (Haqanee, Peterson-Badali, & Skilling, 2015), and parent involvement has been found to be a critical aspect for success in this process, with linkages to reduced recidivism (Schwalbe, 2012; Robertson, Baird-Thomas, & Stein, 2008). Beyond juvenile justice, parental involvement has been a critical element to facilitate positive outcomes across child mental health, child welfare, youth substance use, and education services (e.g., Atkinson & Butler, 1996; Dowell & Ogles, 2010; Tanner-Smith, Wilson, & Lipsey, 2013) and this has been a central change mechanism across numerous evidence-based interventions for justice-involved youth (McCart & Sheidow, 2016). JPOs’ concern about better engaging parents is a nationwide issue; in a national survey, juvenile justice probation leaders cited improved parent engagement as a top priority, but also one of the most operationally and financially challenging problems facing their system (Center for Juvenile Justice Reform, 2008). Research also documents extremely low levels of parental involvement in the probation/parole process for youth (e.g., Broeking & Peterson-Badali, 2010).

An essential advance of the present study is that it goes beyond potential response bias of provider self-report by including observational coding of real JPO sessions with youth, something that plagues prior studies (e.g., Schwartz, Alexander, Lau, Holloway, & Aalsma, 2017). Notably, providers consistently overestimate their use of evidence-based treatment elements (e.g., Beidas & Kendall, 2010; Carroll, Martino, & Rounsaville, 2010). While using self-report data provides a larger sampling (i.e., every session can receive a rating), such data is less reliable than performance observation; and, observation of actual sessions is a better indicator of “real world” capacity to conduct an intervention than role play performance.

Limitations and Cautions

While the adherence level of CM JPOs was similar to or higher than CM therapists, we cannot yet determine if this will result in improved youth outcomes. To assess the true impact of JPOs delivering CM, both a direct outcome (drug use) and an indirect outcome (criminal activity) will be investigated with future data. Further, sustainment, routinization, satisfaction, and other implementation science factors and processes will need to be examined.

There are some generalization cautions. Results may not generalize to urban settings. Participating JPOs and counties also were volunteers, and it is unknown whether results would generalize to random or population samples of JPOs or counties. Support for task-shifting found in the current study also may not generalize beyond CM. In addition, the version of CM used in this study, while based on a variation of the CRA model and validated in numerous studies of adolescents, differs from some other studies; thus, results may not generalize to all CM versions.

Conclusions and Next Steps

A task-shifting paradigm was successful for showing that JPOs can feasibly deliver an evidence-based intervention, in some cases surpassing the adherence of therapists delivering the same intervention, to juvenile offenders. If this task-shifting of CM can be shown to produce positive youth outcomes, findings could generate a major impact since JPOs work across every region of the U.S., including in rural communities that lack substance use services.

Beyond the parent study aims, risk factors beyond substance use may be important to consider for a task-shifting paradigm. That is, substance use is a primary risk factor predicting recidivism (Andrews & Bonta, 2010), but JPOs may not need a substance use intervention for every youth on their caseload. Thus, for instance, JPOs engaging parents to conduct behavior modification for behaviors beyond substance use may be applicable to all youth on probation. Overall, however, the present study provides evidence indicating that an innovative task-shifting approach can be used to deliver intervention advances to justice-involved youth.