Organizational Context and Individual Adaptability in Promoting Perceived Importance and Use of Best Practices for Substance Use

Abstract

This study examines associations among organizational context, staff attributes, perceived importance, and use of best practices among staff in community-based, juvenile justice (JJ) agencies. As part of the National Institute on Drug Abuse’s Juvenile Justice – Translational Research on Interventions for Adolescents in the Legal System (JJ-TRIALS) study, 492 staff from 36 JJ agencies were surveyed about the perceived importance and use of best practices within their organization in five substance use practice domains: screening, assessment, standard referral, active referral, and treatment support. Structural equation models indicated that supervisory encouragement and organizational innovation/flexibility were associated with greater individual adaptability. Adaptability (willingness to try new ideas, use new procedures, adjust quickly to change), was positively correlated with importance ratings. Importance ratings were positively associated with reported use of best practices. Organizational climates that support innovation likely affect use of practices through staff attributes and perceptions of the importance of such services.

Introduction

Youth who come into contact with the juvenile justice (JJ) system in the United States have disproportionately high rates of substance use (SU);^1–3 which is associated with higher rates of reoffending,⁴ increased risky sexual behaviors,^5,6 reductions in academic achievement,⁷ and heightened risk of suicidal behavior.⁸ Although SU trajectories among youth are not uniform,^9,10 studies have documented how adolescent SU raises the probability that youth will develop a substance use disorder (SUD) during adolescence^11,12 or later in adulthood.^13–17 These risks of SUD appear to be particularly high among justice-involved youth, with research indicating that at least one-third meet criteria for SUD^18,19 (a rate nearly six times greater than the rate for the general adolescent population).

Given the elevated prevalence of SU and SUD among JJ-involved youth, JJ settings represent important sites for identifying youth with substance-related needs and linking them to services.²⁰ Probation departments are particularly important settings for such services because they serve as the initial point of contact for youth entering the juvenile justice system after arrest. Furthermore, probation departments are charged with reducing recidivism via community supervision and coordinating services for all court-referred youth. Similar to the care cascade developed for HIV,²¹ a continuum of care for JJ-involved youth necessitates the adoption of universal screening to identify those with potential substance-related service needs, evidence-based assessment for diagnosis, and then linkage to care.^22,23

Developed as part of NIDA’s JJ-TRIALS Cooperative, the Behavioral Health Services Cascade (hereafter referred to as “Cascade”) is a framework representing one best practice approach to addressing SU needs within JJ settings.²² The Cascade illustrates four primary activities needed to support identification and receipt of appropriate SU services: (1) screening, (2) assessment, (3) referral to services, and (4) treatment (including initiation, engagement, and continuing care). Although these discrete activities are often performed by different individuals at multiple locations, they are conceptualized as a cascade because services following each activity should occur with only youth identified as having a need through the preceding activity. For instance, screening would ideally be universal (conducted with the entire JJ population); however, only youth identified by specific screening criteria as having a potential SU problem would receive comprehensive clinical assessment (the subsequent activity). For in-depth assessment, youth are typically referred to an external provider; occasionally they may be conducted by JJ staff. Because the activities represented in the Cascade would ideally occur within and across both probation and community behavioral health (BH) provider settings, JJ-TRIALS utilizes the Cascade model to facilitate cross-system linkage, coordination, and track changes in service receipt over time.²³

Although strides have been made in screening and service provision for the nearly 50,000 youth in secure facilities nationally,²⁴ practices are more varied (or rare) for the nearly 500,000 youth who are supervised by probation while residing in their home communities.²⁵ Indeed, national data on JJ organizations have shown that screening and SU counseling are routinized in only about half of these organizations,²⁶ For youth on community supervision, their substance use needs are often screened in community probation settings, but they typically receive other services (clinical assessment and treatment) in community BH settings.²² Probation officers serve as gatekeepers in that they are responsible for identifying youth service needs and facilitating contact with other service sectors.^27–29 Typically, 50-80% of JJ-involved, substance-using youth do not receive SU services.^30–33

SU service implementation within JJ settings

Complementary to the Cascade and key to facilitating a shift toward implementing “what works” to improve outcomes, is the Risk Needs Responsivity (RNR) principle.³⁴ RNR asserts that youth supervision and services should be delivered based on his or her risk of re-offense and that any treatment provided should address those criminogenic needs most closely associated with reoffending in a manner appropriate to the youth’s abilities and motivation. The RNR principle clearly emphasizes that, in order to effectively supervise, justice agencies need to identify youth BH treatment needs and, where appropriate, refer youth into services. Actual adoption by the field, however, has focused on the assessment of risk of re-offense rather than the needs of youth.³⁵

Research on the provision and/or availability of screening, assessment, referral, and treatment services for JJ-involved youth has largely focused on agency-level service offerings.^26,36–39 Yet in many JJ contexts, individual staff members, including supervising officers (e.g., court officers, probation officers) and case managers, are responsible for activities pertinent to implementing the Cascade. In some jurisdictions, juvenile probation officers (JPOs) have autonomy in administering and interpreting screening results (e.g., checklists, urinalysis) and referring youth to assessment and treatment services. While recent JJ reform efforts emphasize the importance of identifying and addressing youth needs including SU,^40,41 little is known about the degree to which individuals responsible for activities along the Cascade value and utilize these best practices for identifying SU problems and linking youth to services.

According to the Theory of Planned Behavior, positive attitudes toward a given practice should influence intention to perform and engage in the behavior.^42,43 Research demonstrates that attitudes shape both intentions and behavior,⁴⁴ providing a strong rationale for examining both during efforts to promote behavior change. Studies of probation officers’ attitudes regarding use of best practices for SU treatment among adult probationers demonstrate agreement with the value of such practices.^45,46 Individual characteristics such as job responsibility or role within the organization may also shape these attitudes, as evidenced by less favorable ratings of treatment practices among line staff who work directly with clients when compared to staff in administrative roles.⁴⁶ Regarding use of best practices, Farrell and colleagues⁴⁷ found that few juvenile probation officers report using them. Furthermore, individuals who rated their supervisors’ leadership favorably were more likely to report use of evidence-based practice. However, those with higher cynicism toward change reported less use.⁴⁷ More research is needed to better understand (a) the relationship between attitudes toward and use of best practices for addressing substance use and (b) ways in which organizational factors shape attitudes and practice use along the Cascade.

Organizational factors influencing change

The field of implementation science has accumulated evidence indicating that organizational factors can both promote or impede change efforts.^48–50 An organizational climate in which staff report low cohesion and communication and high stress can interfere with implementation of new practices,⁵¹ whereas perceived leader support for innovation can encourage acceptance and adoption.⁵² Supervisors contribute to both a work climate that is receptive to change and a workforce that values adaptability to new innovative practices.⁴⁸ In fact, individual staff attributes such as adaptability to work demands can promote support of new innovations.⁵³ Thus, leadership that encourages innovation also promotes “evidence based thinking” among staff.⁵⁴ Additionally, staff characteristics, including age and gender,⁵² as well as perceived relevance of new practices for current job responsibilities⁵⁵ can also contribute to increased buy-in and less push-back in implementation efforts. Although associations among leadership, individual adaptability, and organizational climate have been examined in relation to innovation adoption in treatment provider settings,⁴⁸ these factors have not been examined collectively within JJ settings which typically have a hierarchical, top-down structure and experience rapid policy changes resulting from local or state mandates. It is possible that in these types of settings, organizational factors such as flexibility and supervisor encouragement may not promote best practices in the same way as in other service settings. Furthermore, juvenile probation officers experience high levels of stress⁵⁶ due to large caseloads⁵⁷ and challenges reconciling law enforcement and rehabilitation goals.⁵⁸ This study helps address this research gap by examining the roles of both organizational context and staff attributes on JJ staff perceptions of the importance of best practices and their use of those practices in addressing SU among the youth they serve.

Current study

The purpose of the current study is twofold: (1) to document the perceived importance (attitudes) and reported use (behavior) of best practices for SU service support among JJ staff across four Cascade service domains: screening, assessment, referral, and treatment support and (2) to examine the role of selected organization context and individual factors on staff attitudes and reported best practice usage. The four Cascade domains are analyzed in separate models for three reasons. First, these linked activities are often performed by different individuals working within and across institutions. Second, preliminary analyses of JJ youth records data suggests that while most youth progress through the Cascade as expected, variation in the order of receipt does occur.⁵⁹ Third, separate models enable examination of whether organizational and individual factors are consistently associated with these domains; consistent associations would provide stronger evidence of the importance of a given organizational factor. It is expected that perceived organizational context and individual attributes will relate systematically to staff attitudes regarding importance and use across all four Cascade service domains (see Figure 1). Specific hypotheses are as follows. H1: Use of SU practices for screening, assessment, referral, and treatment support will be associated with greater perceived Importance of those practices; H2: Individual Adaptability will be associated with Importance of SU practices; H3: Supervisory Encouragement (as perceived by staff) will be associated with Organizational Innovation and Flexibility, and H4: Organizational Innovation and Flexibility will be associated with Use of SU practices.

Figure 1.

Proposed Relationships among Perceived Organizational Context, Staff Attributes, and Importance and Use of Best Practices for Substance Use (SU)

Method

Data are from the National Institute on Drug Abuse (NIDA)-funded Juvenile Justice-Translational Research on Interventions for Adolescents in the Legal System (JJ-TRIALS) cooperative research initiative, which includes six research centers (RCs, located at Columbia University, Emory University, Mississippi State University, Temple University, Texas Christian University, and University of Kentucky) and a coordinating center (CC: Chestnut Health Systems). Launched in 2013, JJ-TRIALS seeks to improve the delivery of evidence-based SU and HIV services for JJ-involved youth by working with JJ agencies and their local behavioral health (BH) partners to implement customized, organizational-level changes. Each RC engaged six JJ agencies (e.g., county youth court or probation department) and at least one community-based BH service provider working with each JJ agency, resulting in 36 “sites” (a paired JJ and BH agency) in seven states (Florida, Georgia, Kentucky, Mississippi, New York, Pennsylvania, and Texas). The present study focuses on staff working in the 36 JJ sites.

Procedures

Beginning in August 2015, staff at all levels of participating JJ organizations were surveyed on their perceptions of their organization’s climate regarding change and innovation, the importance of interagency collaboration, and their beliefs about and use of practices regarding substance use screening, assessment, referral, and support for treatment. While the protocol includes repeated survey administrations at four time points corresponding to Aarons’ and colleagues Exploration, Preparation, Implementation, Sustainment (EPIS) model,⁶⁰ only survey data collected at baseline are included here (JJ-TRIALS data collection is on-going).

Staff members were informed about the study either through an onsite orientation meeting (where each RC Principal Investigator described the study) or via email/phone by an RC staff member (when the orientation was missed or new staff were hired). Consent was obtained from all interested staff (78% of individuals invited). Participants completed the baseline survey either via web-based or paper format. If online using a secure service (i.e., Qualtrics), participants provided email addresses when consenting and were sent individualized internet links to ensure confidentiality. Paper surveys were distributed at staff orientations or via email. The overall participation rate (% of those who consented) was 85%. Study procedures were approved by the Institutional Review Boards associated with each RC/CC.

Sample

Staff surveys were provided by 492 individuals working in the 36 JJ agencies. Of these 492 individuals, 415 (84%) worked in a JJ unit (e.g., juvenile probation office), 59 (12%) worked in a behavioral unit operated by the JJ agency, and 18 (4%) worked in an undefined unit within the JJ agency. Complete data were available for 90% of respondents (441-450 respondents, depending on the Cascade service domain). The number of individuals with complete data per site ranged from 2 to 46 (M = 12.60; SD = 10.91).

Measures

Organizational context

The current study utilized two measures of organizational context: Organizational Innovation and Flexibility (from the Organizational Climate Measure)⁶¹ and Supervisor Encourages Innovation (from the TCU Survey of Organizational Functioning and Leadership; SOFL).⁶² Organizational Innovation and Flexibility is comprised of six items and is rated on a 4-point Likert scale ranging from (1) “definitely false” to (4) “definitely true” (α = .86). All scale scores are multiplied by 10. Sample items include “new ideas are readily accepted here” and “this organization is quick to respond when changes need to be made.” Supervisor Encourages Innovation⁶³ includes four items where respondents rate their supervisor’s behavior on a 5-point Likert scale ranging from (1) “disagree strongly” to (5) “agree strongly” (α = .87).⁵¹ Sample items include “encourages others’ ideas” and “encourages staff to try new ways to accomplish their work.”

Individual attributes

Individual attribute measures were rated on a 5-point Likert scale ranging from (1) “disagree strongly” to (5) “agree strongly.” Constructs included Adaptability, Job Responsibilities, and demographic variables. Individual Adaptability (from the TCU SOFL),⁶² assesses staff members’ proclivity to change practices in an evolving workplace (α = .64).⁶⁴ Sample items include “you are willing to try new ideas even if some staff members are reluctant.” To assess Job Responsibilities (JR), respondents were asked to select from a list of 18 activities that are typically conducted in JJ or BH settings (e.g., comprehensive assessment, case management). Five separate job responsibility measures were constructed to indicate whether the individual currently performed duties related to the four practice domains. JR—Screening included “screening for substance use;” JR—Assessment included “comprehensive assessment” and “clinical diagnosis;” JR—Referral included “case management” and “supervision of youth;” JR—Treatment Support included “SU education,” “outpatient SU treatment,” “residential SU treatment,” and “aftercare or supportive services;” JR—Community Communication included “Communication with Community Agencies.” Demographics include Gender, Race (White, non-White), and Age.

Importance and use of SU service support practices

Staff were asked to rate the importance of items representing best practices along four Cascade service domains (screening, assessment, referral, and treatment support). Screening was defined in the survey as “a relatively brief standard set of questions designed to identify youth who may be at high risk of having disorders that warrant brief intervention, more comprehensive assessment, or immediate referral for treatment.” Survey instructions noted that screening does not require professional staff to administer, but may need professional staff to interpret the results. Formal clinical Assessment was defined as “more comprehensive than screening;” survey instructions noted that assessment “is designed to support diagnosis, placement, and treatment planning related to substance use disorders and mental health disorders, and typically is conducted by trained professionals”. Survey instructions defined Referral as directing a youth and/or family to a resource or provider for needed substance use treatment services. Examples of referral activities include providing a card with contact information or making the first appointment with a provider. Treatment Support was defined as a set of activities that encourage or support youths’ engagement in treatment services, including encouraging the youth to initiate/engage in treatment and communicating with treatment personnel about attendance and progress. For clarity, respondents were given the following definition: “Treatment services include therapeutic interventions intended to help youth with substance use problems overcome physiological and psychological dependencies, and to identify and alter behavior patterns that contribute to substance use and potential relapse. Treatment may involve a range of settings, therapeutic methods, and levels of intensity.”

Participants were asked to consider JJ-involved youth under community supervision (e.g., on probation) on their caseload when answering all questions. Staff without caseloads were instructed to answer for youth on their units’ caseload. Instructions also stated that they were to consider “all SU services youth might receive within the JJ agency and/or outside the JJ agency.” For each item measuring perceived importance, response options ranged from “not important” to “very important” on a 5-point scale. Perceived importance of screening (7 items), assessment (9 items), referral (13 items), and treatment support (9 items; see Tables 1–4). The same items were used to assess Use of best practices with youth specifically in the respondent’s caseload (5-point scale from “Not Used” to “All the Time”). It is important to note that respondents were asked to rate whether youth on their caseloads received services, not whether they personally performed the service. Thus, Use measures are computed on the whole sample, not just those who indicate the activities as part of their job description.

Table 1.

Screening Items and Exploratory/Confirmatory Item-factor Relationships

Item	EFA Importance	EFA Use	CFA Importance	CFA Use
G1a. Screening youth for substance problems	.77	.03	.72
G1b. Using a scored, standardized evidence based instrument to screen for substance problems	.65	−.09	.62
G1c. Using biological testing to screen for substance problems	.55	.09	.48
G1d. Completing substance use screening within 30 days after the initial offense	.55	.14	.61
G1e. Supplementing the use of at least 1 validated screening tool with another source of information	.55	.12	.69
G1f. Using screening results to make substance use treatment referrals	.84	−.03	.73
G1g. Using results from screening instruments to recommend more comprehensive assessment	.92	−.10	.79
H2a. Screening youth for substance problems	.05	.70		.62
H2b. Using a scored, standardized evidence based instrument to screen for substance problems	.00	.58		.54
H2c. Using biological testing	.04	.54		.45
H2d. Completing substance use screening within 30 days after initial offense	.06	.61		.54
H2e. Supplementing the use of at least 1 validated screening tool with another source of information	.00	.67		.71
H2f. Using screening results to make substance use treatment referrals	.00	.84		.74
H2g. Using results from screening instruments to recommend more comprehensive assessment	−.09	.80		.74

Notes: Items with “G” labels referred to Importance Scale; items with “H” labels were the corresponding Use Scale items

(How often screening used in practice).

For EFA (Exploratory factor analysis) results, entries represent values from joint (two-factor) factor pattern matrix. For CFA (Confirmatory factor analysis), entries are estimated relationships for proposed two-factor model. Entries in bold represent values for the intended factor. EFA correlation between factors = .44; CFA correlation = .39.

Analytic plan

Exploratory and confirmatory factor analyses were used to determine which item groupings best captured perceived Importance and Use of best practices for each Cascade service domain. The data set was randomly split, with one half used for exploratory factor analysis (EFA; N = 225-243) using principal axis extraction with oblique rotation. Confirmatory factor analyses (CFA) were then performed on the remaining half of the data set (using listwise deletion, N = 226-247), using AMOS to estimate loadings, inter-factor correlations, and fit indices. Classical (true-score) reliability estimates were computed using the combined data set. For each service domain and each portion of the staff questionnaire (Importance; Use), EFAs were run in two steps. First, with just one portion of the service domain (e.g., screening importance), the EFA (principal axis factoring) was run with a goal of identifying a single factor solution. Then, items for the two portions (use and importance) were jointly factored, with a goal of identifying a two factor solution. When we were satisfied that the item sets were consistent in this way, CFAs (using maximum likelihood estimation) were run on the other half of the data set, to confirm: (a) each service domain/portion behaved as a single factor; and (b) each service domain with both portions behaved well as a two-factor solution.

Structural equation modeling was used to estimate the paths in the model using SAS Proc CALIS.⁶⁵ Because of the large number of programs with small sample sizes, the robust standard errors recommendation was used rather than a multiple groups model or a model based on a within groups covariance matrix in testing the hypotheses denoted in the models. The use of robust standard errors was done using the two-stage robust estimation procedure⁶⁶ provided in SAS Proc CALIS (Robust = SAT option). Separate models were tested, corresponding to each of the Cascade service domains. The general model guiding the analyses is presented in Figure 1. It identifies the critical pathways among organizational climate and staff attributes, with staff attributes and organizational climate affecting the Cascade domains. Additionally, relationships were addressed within the context of selected staff background characteristics (demographics and job responsibilities corresponding to the Cascade domain being examined). The general model was estimated for each Cascade domain, and adjustments were made to that result. The model adjustments included: (a) omitting paths with non-significant coefficients; and (b) adding a path, where suggested by modification indices and that was justifiable, to yield the final model for each Cascade domain.

Results

The sample was primarily female (58%) and White (72%), with a mean age of 41 (SD = 9.65). Half (50%) were responsible for SU screening, 27% were responsible for assessment, 77% were responsible for referral, and 31% were responsible for providing treatment or treatment support. Means, with a possible range from 10 to 50, were 27.9 (SD = 6.0) for Organizational Innovation and Flexibility, 39.0 (SD = 8.3) for Supervisor Encourages Innovation, and 38.6 (SD = 6.0) for Adaptability.

Scale development and psychometrics

EFA results

From the original development of the Cascade service domain scales, the expectation for each was that: (a) the lone scale would be unidimensional; and (b) the importance and use scales for a specific domain (e.g., screening), would fit to a correlated, two-factor structure. For all scales, the data sets were found to be suitable for factoring (e.g., Kaiser-Meyer-Olkin measure of sampling adequacy ≥ .81, statistically significant Bartlett sphericity test results, p < .001). One-factor solutions appeared plausible, as backed by the scree test plots, eigenvalues, parallel analysis,⁶⁷ and magnitude of item-factor loadings, for screening and assessment scales. For referral (both Importance and Use), two factors were identified; nine items represented a more standard set of referral practices (more “passive” approaches to linking youth to services, such as providing the name and contact info of a provider, using a referral form or selecting a treatment option that addresses the youth’s specific need) while three items represented more active or direct linkage practices (e.g., making the initial contact/appointment, providing transportation). One item (“basing referral to SU treatment on type of offense”) showed poor affiliation and was removed. For treatment support (both Importance and Use), one item was removed (“encouraging every youth with a substance problem to initiate treatment services”), due to poor affiliation with the respective factor. Subsequently, all service domain scales showed promise as single-factor scales and the combined importance-use scales yielded two correlated but distinguishable factors (importance and use). For all individual scales, item-factor loadings exceeded .50, with values typically in the .60s-.70s. Adequacy of the EFA solutions was gauged by the fraction of reproduced correlations in the joint (two-factor) solutions that differed by .05 or more from observed correlations. These were 26%, 32%, 39%, and 58% for screening, assessment, referral, and treatment, respectively. A typical threshold for good data-structure fit is that fewer than 50% of residuals exceed .05,⁶⁸ meaning that only the treatment support scales failed to meet this criterion.

CFA results

Models for both individual and joint importance-use scales were tested (for fit to one-factor and two-factor models, respectively), one for each Cascade service domain, using the final item sets from the EFA analyses. We considered whether the combined (joint importance-use) item set fit to a two-factor model as a more rigorous appraisal of the factor structure than the individual scale analyses. Four aspects were considered: (a) strength of item-factor associations; (b) standardized root mean square residual (SRMR) values; (c) comparative fit index (CFI) values; and (d) root mean square error of approximation (RMSEA) values. With the exception of the treatment scales, model-data fit was acceptable, though not stellar (for screening, assessment, and referral combining importance/use scales: SRMR < .08, CFI > .80, RMSEA < .13; for treatment importance-use scales, SRMR = .10, CFI = .67, RMSEA = .18). For all Cascade service domains, the item-factor loading affiliations were strong, and were consistent with the EFA values (see Tables 1–4). For all scales, model-data fit could have been improved to more traditional thresholds (e.g., SRMR < .05, CFI > .90, RMSEA < .06) by allowing selected item error variances to covary. We chose not to do so, since the specific item pairings were idiosyncratic to the service domain scale; as well as, this could invite concern for additional latent variables that would not be robust.

Reliability

Unstandardized Cronbach alpha estimates of internal consistency reliability were good for all eight scales (≥ .83). Screening scales yielded the lowest estimates (.84 and .83 for importance and use, respectively). For Assessment, the values were .92 and .89; for Standard Referral, .85 and .89; for Active Referral, .81 and .80; and for Treatment Support, .89 and .88 for importance and use, respectively. Thus, the total scores for the respective scales should be more than adequate for research purposes or for making decisions about groups.⁶⁹

Means and SDs

Means and standard deviations for each of the cascade service domains are presented in Table 4. Compared to Use ratings, Importance ratings were higher across all cascade domains, with the largest differences in assessment and active referral and the smallest differences in Treatment Support, Standard Referral, and Screening. Regarding Importance, Standard Referral practices were rated highest, followed by Treatment Support, Screening, Assessment, and Active Referral. The identical order was seen among Use ratings.

Relationship between organizational factors and SU services practice ratings

Screening

The estimation of the proposed initial model for the screening domain resulted in a “fair” fit to the data [χ²(12) = 37.42, p < .0002; CFI = .90; RMSEA = .069; SRMR = .04]. In the revised model, deleted paths were from two demographic variables that were not significantly related to Adaptability (male, age) and the non-significant paths from Job Responsibility and Male to Screening Importance. A path was added from the demographic variable, White, to Screening Use scores. The resulting fitted model was judged “satisfactory” [χ²(10) = 19.78, p = .032; CFI= .96; RMSEA = .05; SRMR = .03], and is shown in Figure 2. The final model suggests that perceptions of Importance are associated with reported Use of corresponding service practices (H1) and Individual Adaptability is associated with Importance (H2). Reported Supervisory Encouragement of Innovation is directly related to perceptions of Individual Adaptability and Organizational Innovation/Flexibility (H3), and these in turn are indirectly related to Importance of screening through Individual Adaptability (H4). Furthermore, staff whose responsibilities included screening were more likely to report this activity with youth in their caseload, although the magnitude of the relationship suggests that other staff also participate in screening even though they did not report screening among their job responsibilities. With respect to demographic effects, White respondents were more likely to report lower Individual Adaptability, to view Screening as less important, and were less likely to report Use of screening practices. Older staff were more likely to report higher Screening Importance.

Figure 2.

SU Screening: Relationships among Perceived Organizational Context, Staff Attributes, and Best Practice Measures⁺

Assessment

The estimation of the initial model of the Assessment domain also yielded a “fair” fit to the data [χ²(12) = 28.86, p = .004; CFI = .92; RMSEA = .056; SRMR = .038]. As with the model for screening, several paths from demographic variables were not significant (White and Assessment Importance; Age and Gender with Individual Adaptability). As well, the path from Job Responsibility to Assessment Importance was not significant. Elimination of these paths and the addition of paths suggested by the modification indices (Age and Race to Assessment Use) resulted in a final model having “good” fit to the data [χ²(13) = 16.70, p = .213; CFI = .98; RMSEA and SRMR = .03], depicted in Figure 3. The model was identical to that for Screening, with some differences in paths for demographics: males were more likely to have lower ratings for Assessment Importance and older staff were less likely to report Assessment Use.

Figure 3.

SU Assessment: Relationships among Perceived Organizational Context, Staff Attributes, and Best Practice Measures⁺

Standard referral

The initial estimation of the Standard Referral model indicated a “good” fit, based on the RMSEA and other fit criteria [χ²(12) = 29.77, p = .003; CFI = .94; RMSEA = .058; SRMR = .034]. As with Screening and Assessment, there were some hypothesized demographic relationships to the referral cascade variables that were unsupported (Gender and Age were unrelated to Individual Adaptability, and the measure of Importance of Standard Referral was unrelated to Age, Race, and Job Responsibility). These paths were removed, and a path from White to Referral Use was added as indicated by the modification indices. This new model yielded a good fit to the data [χ²(12) = 16.96, p = .151; CFI = .98; RMSEA and SRMR = .03], which is depicted in Figure 4. The proposed relationships among organizational climate, individual adaptability, and cascade domains were supported, and Referral Job Responsibilities were significantly related to Referral Use. The direct path between Organizational Innovation/Flexibility and Standard Referral Importance, however, was not significant. With regard to demographics, being White was negatively related to Individual Adaptability and Use of Standard Referral practices, whereas being male was negatively related to Importance of Standard Referral. Age was unrelated to these variables in the Standard Referral cascade model.

Figure 4.

Standard SU Referral Practices: Relationships among Perceived Organizational Context, Staff Attributes, and Best Practice Measures⁺

Active referral

The estimation of the model employing the Active Referral scales yielded a “poor” fit for the initial model [χ²(14) = 40.99, p < .0002; CFI = .95; RMSEA = .066; SRMR = .035]. This initial model differed from the other initial models for Cascade domains in that it included a second Job Responsibility measure that addressed Community Communication, which was believed to be relevant to the Active Referral domain. One reason for the poor fit was that 8 of the 16 estimated paths were non-significant. After these non-significant paths were removed, the re-estimated model was found to have “good” fit indices [χ²(9) = 20.10, p = .017; CFI = .966; RMSEA = .05; SRMR = .038], as depicted in Figure 5. This revised model differs from the other models for the cascade domains in that there was no statistical support for paths from Organizational Innovation and Flexibility to Active Referral Importance or to Active Referral Use. Also, neither of the two job responsibility measures was related to Active Referral Use. White respondents were less likely to report Individual Adaptability and also both Active Referral measures (Importance and Use), and older staff were less likely to report Active Referral Use.

Figure 5.

Active SU Referral Practices: Relationships among Perceived Organizational Context, Staff Attributes, and Best Practice Measures⁺

Treatment support

For the Treatment Support domain of the cascade, the initial model yielded a “fair” fit to the data [χ²(12) = 38.20, p < .0001; CFI = .92; RMSEA = .07; SRMR = .04]. As with the other Cascade service domains, the initial model had some non-significant hypothesized demographic relationships with Individual Adaptability and Treatment Support Importance. As suggested by modification indices, a revised model was estimated and found to have a satisfactory fit to the data [χ²(15) = 26.95, p = .029; CFI = .96; RMSEA = .043; SRMR = .032], depicted in Figure 6. Again the core of the model stating relationships among Organizational Climate, Individual Adaptability, and Importance and Use of Treatment Support practices was supported; however, there was no direct path between Organizational Innovation/Flexibility and Use of Treatment Support practices. Treatment Job Responsibilities was positively related to Use of Treatment Support. Similar to other cascade domains, White staff were less likely to endorse Adaptability and Treatment Support Use, males were more likely to report lower importance compared to females, and older staff were less likely to report using Treatment Support practices.

Figure 6.

SU Treatment Support: Relationships among Perceived Organizational Context, Staff Attributes, and Best Practice Measures⁺

Discussion

The current study documents perceived importance and use of best practices for SU services among JJ staff and examines how organizational context and individual attributes shape attitudes and reported behavior. When considering the importance of strategies corresponding to the Behavioral Health Services Cascade,⁴⁵ JJ staff report standard referral practices as most important and active referral practices as least important. Although staff who value best practices are also more likely to use them, self-reported practice use is generally lower than ratings of importance on all Cascade domains. The discrepancy might suggest that even when certain practices are supported, JJ staff may find it challenging to execute best practices that are inconsistent with their current job expectations or that require a high level of effort (or time) to execute. For example, clinical assessment requires expertise beyond that of the typical probation officer, and active referral may require significant time commitments for individuals who already have challenging caseloads. Alternatively, the discrepancy could suggest gaps in knowledge regarding what constitutes best practices for youth with SU problems. Standards for screening, clinical assessment, and treatment are available;^22,70,71 but while innovative and successful referral practices are emerging,³⁸ research on referral practices is generally sparse. Research indicates that changing knowledge and attitudes through education and training may predispose service personnel to adopt new practices.⁷² Furthermore, guidance on how to initiate and maintain change can facilitate uptake and sustainability.⁷³ For these reasons, education about the importance of comprehensive assessment and active referral strategies (within a broader best practices framework)²² is incorporated into the JJ-TRIALS intervention.²³

Findings also demonstrate support for hypotheses regarding the role of organizational context and individual staff attributes in potentially shaping JJ staff perceptions of importance and use of SU practices, and identifying remarkably similar models across all Cascade service domains. Consistent with prior research,^48,52,74 both supervisory encouragement and a climate of innovation/flexibility were associated with greater adaptability among staff. Adaptability, in turn, was related to higher importance ratings. Staff who were willing to try new ideas, use new procedures, or adjust quickly to change were more likely to rate best practices for SU as highly important (compared to less-adaptable staff). As expected and consistent with the Theory of Planned Behavior,^42,43 importance was associated with greater reported use of these practices with youth. Thus, organizational climates that support innovation may indirectly affect use of practices through staff attributes and perceptions of the importance of such services. Future studies should examine this premise using data collected over time to determine causality and examine potential bi-directional relationships within the model. In addition, longitudinal studies can examine whether participation in training and practice improvement activities (such as those in JJ-TRIALS) result in increased positivity toward best practices, whether increased perceptions of importance result in increased strategy use, and the degree to which contextual factors such as funding, organization size, and client mix affect attitudes and behaviors regarding SU service provision.

Individuals who were responsible for carrying out screening, assessment, standard referral, or treatment support duties correspondingly rated use significantly higher on the Cascade domain related to their work (compared to other staff). These relationships existed while controlling for the demographic characteristics of gender, race, and age. In some instances, the demographic variables were also directly associated with perceptions of importance and use, with males and older staff reporting greater importance on some domains, younger and White staff reporting lower use on some domains, and White staff reporting lower adaptability in general. These demographic measures may be indicators of an increasingly diverse workforce,⁷⁵ and are potentially correlated with other factors such as job responsibilities, management positions, and prior education/certification. Additional research is needed to further ascertain the role of individual characteristics on attitudes toward innovation,⁷⁶ potential interactions among demographic variables,⁷⁷ and explore strategies for promoting innovation among diverse workgroups.

While it was hypothesized that organizational innovation/flexibility would be directly related to both importance and reported practice use, the strength of these relationships varied depending on the Cascade domain examined. Models for screening and assessment were nearly identical, with organizational innovation directly associated with adaptability, importance, and use. In contrast, the influence of organizational climate appears to be more indirect for referral and for treatment support. Organizational innovation was not related to perceptions of referral importance (either for standard or active practices). Although it was associated with use of standard referral practices, organizational innovation was not significant in the active referral model or the treatment support model. These differences suggest that adoption of best practices for referral and treatment support may be fundamentally different or less consistently valued than those for screening and assessment, and that organizational climate factors may have greater influence on some types of practices than on others.

Findings document two conceptually and psychometrically distinct types of referral practices among JJ staff: standard and active. Standard referral practices include more “passive” approaches to linking youth to services (e.g., providing family with agency contact information, selecting an agency based on the youth’s need; using a referral form to document recommendation), whereas active referral practices involve intentional efforts to facilitate linkage (e.g., making the initial appointment for the family, providing transportation). The fact that no direct paths were evident between organizational innovation and active referral importance or use (and no path between job responsibilities and use) could be attributed in part to the way that active referral practices were measured. Indeed, active referral was ranked lowest on both importance and use and had the largest standard deviations (both scales), suggesting that these practices may be used or perceived as important by fewer staff. The use of active referral strategies are likely at the discretion of individual staff, and may be less embedded in agency organizational contexts, such as leadership and innovation.

Understanding relationships between organizational context and JJ staff referral practices is further complicated by the inter-agency nature of referral practice and apparent lack of manualized or coordinated protocols for referral. Acknowledging the contribution of BH treatment to lowered recidivism,^78,79 policymakers have suggested greater focus on what happens after screening⁸⁰ and cross-systems integration to support referral to care.^38,47,81 Although “off the shelf” availability of evidence-based screening and assessment instruments (or protocols) has no doubt facilitated their implementation, referral and treatment support activities are more a set of practices and tools than a coordinated package. While there is evidence that certain distinct activities (e.g., use of a referral form to facilitate inter-agency handoff), integrated programming,^29,38,82 or role responsibilities for staff charged with service linkage (e.g., case management)^83,84 (linkage specialist)⁸⁵ are effective, these are not available as a coordinated program (for the individual activities) or not widely available in JJ agencies (case management, linkage specialist). Future efforts should focus on development and evaluation of referral/linkage programs, especially for use in JJ settings where families may be less likely to recognize need, since services are not self-selected. Understanding the organizational and individual factors associated with use of active referral and treatment support practices can inform the development of these referral and linkage protocols.

Limitations

A number of limitations in the research design should be acknowledged. First, the data were collected at a single point in time, so causal inferences cannot be made. Second, the individuals who participated work only in JJ agencies, so it is unknown the extent to which these findings may generalize to individuals working in other types of agencies that serve high-risk youth, such as child welfare, behavioral health, or educational organizations. Third, participating agencies were located in only seven US states; it is unknown whether these findings would generalize to JJ agencies in other states. Furthermore, these 36 sites were not selected at random; rather, the sites were specifically recruited to be a part of JJ-TRIALS and agency leadership were aware that their sites would be asked to implement changes in practices over the course of the study. Willingness to be involved in system-level changes may be correlated with an underlying organizational culture of innovation within the agency that may differentiate these JJ agencies from those not participating in initiatives such as JJ-TRIALS. Also, because these data were collected at baseline, it is unknown whether individuals who chose to not respond to the survey differed from respondents in their perceptions of the organizational context, their attitudes toward best practices for SU, and self-reported use of these practices. Finally, all data were collected via self-report from staff, and prompts may have been interpreted differently by staff depending on personal experiences and biases. It is also difficult to determine whether self-reported use of best practices is correlated with actual use, as might be measured via observational data, administrative records or confirmed through focus groups with JJ staff.

Implications for Behavioral Health

As supported by this paper, contextual factors, including organizational innovation and individual adaptability to change, are associated with perceptions and reported use of best practices within JJ agencies. Implementation studies suggest that while some individuals may adopt innovations on their own, many settings may benefit from actively preparing for implementation by first adapting the organizational climate and leadership’s visible support of new ideas and change.^86,87 Established strategies for assessing and improving organizational readiness for change are readily available for agencies.^86,88,89

Earlier work relating probation practices to organizational features has documented the contribution of supervisor leadership to staff use of practices such as those examined here (e.g., use of a screening instrument, onward referral for in-depth assessment).⁴⁷ Across almost all Cascade service domains examined, the direct and indirect associations among organizational context, staff attributes, and reported staff practices were largely consistent. The strongest associations were between importance and use, suggesting that future work should examine strategies to increase staff attitudes regarding the importance of the targeted practices. Indeed, specific change strategies have been found important for the adoption of best practices within correctional settings.^90,91 This suggests that organizational interventions (such as training, facilitated process improvement efforts, etc.) that correspond to domains represented in the BH Cascade have the potential to increase staff perceptions of importance and use of associated practices. The larger JJ-TRIALS study will be able to test the impact of such interventions on staff-level attitudes and practices.

While organizational climate is important in the uptake of best practices by JJ professionals, change in JJ organizations is often guided by the larger context of the agency and state. Leadership and staff “act” according to mandates and policy expectations but “react” according to perceived importance, resource availability and workload. The dissemination of information about “what works” to improve youth outcomes has resulted in a number of states passing legislation mandating that all youth be screened for mental health and substance use needs at the point of entry into the JJ system. Less common, however, are mandates requiring the assessment of youth and the referral of those identified with SU needs to appropriate services. In understanding agencies’ readiness for change, future investigation should consider the ways in which agency leaders attempt to reconcile and integrate external pressures (e.g., outer context) and internal resources (e.g., inner context) to accomplish goals,⁹² as well as other outer context factors that could impact these relationships. For example, county shortage of BH providers has been found to be a strong contributor to youth service access.^29,38

Conclusion

In this study of juvenile justice staff, attitudes about the importance of specific elements of the Behavioral Health Services Cascade were positively associated with the frequency that staff reported using those elements with youth their caseloads. Adaptability within the individual staff member was consistently associated with perceived importance of each Cascade element. While many might consider adaptability to be a personality trait and hence relatively fixed, these findings underscore that adaptability is actually explained, in part, by the organizational context, specifically perceptions of whether supervisors and the broader organization value innovation. Taken together, these findings suggest that there may be value in implementation interventions that not only address discrete training needs about a specific practice but also attend to developing and supporting an organizational context in which supervisors convey the importance of innovative solutions to challenges. The broader JJ-TRIALS study, which tests the added value of facilitated change teams where line staff are empowered to work together to redesign processes, is able to examine whether such an intervention promotes interagency support for innovation, enhancements in individual adaptability, and subsequently greater implementation of the practices embedded within the Cascade.

Table 2.

Assessment Items and Exploratory/Confirmatory Item-factor Relationships

Item	EFA Importance	EFA Use	CFA Importance	CFA Use
G2a. Conducting a comprehensive assessment of substance use and related problems	.82	−.01	.83
G2b. Using a scored, standardized evidence based assessment instrument to identify substance problems	.78	−.04	.77
G2c. Using a clinical assessment that generates a substance use disorder diagnosis	.83	−.05	.91
G2d. Using a clinical assessment that generates a mental health disorder diagnosis	.79	.00	.90
G2e. Interviewing the youth’s family about the youth’s substance problems	.73	.00	.63
G2f. Completing a comprehensive assessment within 30 days after initial offense	.64	.12	.69
G2g. Comprehensive assessments conducted by licensed clinicians	.61	.03	.71
G2h. Using two or more sources of information to identify a substance problem	.63	.08	.66
G2i. Using results from comprehensive assessments to make substance use treatment referrals	.84	−.08	.67
H4a. Conducting a comprehensive assessment of substance use and related problems	−.05	.79		.75
H4b. Using a scored, standardized evidence based assessment instrument to identify substance problems	−.02	.64		.61
H4c. Using a clinical assessment that generates a substance use disorder diagnosis	.07	.76		.86
H4d. Using a clinical assessment that generates a mental health disorder diagnosis	.06	.72		.81
H4e. Interviewing the youth’s family about the youth’s substance problems	−.07	.59		.34
H4f. Completing a comprehensive assessment within 30 days after initial offense	−.04	.71		.63
H4g. Comprehensive assessments conducted by licensed clinicians	.04	.78		.67
H4h. Using two or more sources of information to identify a substance problem	.04	.72		.45
H4I. Using the results from comprehensive assessments to make substance use treatment referrals	.00	.78		.61

Notes: Items with “G” labels referred to Importance Scale; items with “H” labels were the corresponding Use Scale items (How often assessment used in practice).

For EFA (Exploratory factor analysis) results, entries represent values from joint (two-factor) factor pattern matrix. For CFA (Confirmatory factor analysis), entries are estimated relationships for proposed two-factor model. Entries in bold represent values for the intended factor. EFA correlation between factors = .40; CFA correlation = .33.

Table 3.

Referral Items and Exploratory/Confirmatory Item-factor Relationships

Item	EFA Importance	EFA Use	CFA Importance	CFA Use
G3a. Referring the youth with a substance problem to treatment services	.78	−.06	.76
G3b. Initiating referrals within 14 days after needs are identified	.74	.02	.72
G3c. Using standardized procedures for referrals	.73	−.03	.73
G3e. Involving the youth/family in selecting a particular treatment option	.47	.06	.59
G3f. Selecting a treatment option that addresses the youth’s specific need	.71	−.10	.80
G3g. Referring youth to treatment providers that are accredited	.65	.07	.82
G3h. Referring youth to treatment providers that use evidence based practices	.59	.13	.59
G3i. Referring youth to treatment providers based on location, convenience, or accessibility	.59	.05	.52
G3j. Providing the youth/family with the treatment provider’s contact information	.58	−.03	.63
H6a. Referring the youth with a substance problem to treatment services	.02	.67		.72
H6b. Initiating referrals within 14 days after needs are identified	.01	.75		.69
H6c. Using standardized procedures for referrals	.01	.71		.65
H6e. Involving the youth/family in selecting a particular treatment option	.00	.60		.62
H6f. Selecting a treatment option that addresses the youth’s specific need	−.04	.82		.79
H6g. Referring youth to treatment providers that are accredited	.01	.77		.70
H6h. Referring youth to treatment providers that use evidence based practices	.02	.72		.68
H6i. Referring youth to treatment providers based on location, convenience, or accessibility	.02	.60		.59
H6j. Providing the youth/family with the treatment provider’s contact information	−.01	.65		.70

Notes: Items with “G” labels refer to Importance Scale; items with “H” labels were the corresponding Use Scale items (How often used in practice).

For EFA (Exploratory factor analysis) results, entries represent values from joint (two-factor) factor pattern matrix. For CFA (Confirmatory factor analysis), entries are estimated relationships for proposed two-factor model. Entries in bold represent values for the intended factor. EFA correlation between factors = .48; CFA correlation = .40.

Item G3d/H6d, “Basing referral to substance use treatment on type of offense” was omitted based on EFA results (for both importance and use scales).

Table 4.

Treatment Support Items and Exploratory/Confirmatory Item-factor Relationships

Item	EFA Importance	EFA Use	CFA Importance	CFA Use
G4b. Encouraging the youth to initiate treatment within 14 days after referral	.65	.09	.67
G4c. Encouraging the youth to engage in services by attending at least 3 sessions within the first 6 weeks	.63	.10	.62
G4d. Encouraging the youth to participate in treatment services for 90 days or longer	.66	−.04	.60
G4e. Informally sharing information between JJ supervision and treatment provider staff regarding youth participation	.60	−.02	.60
G4f. Formally sharing information between JJ supervision and treatment provider staff regarding youth participation	.71	−.06	.69
G4g. Communicating with youth/family about the youth’s progress in treatment	.69	−.04	.76
G4h. Initiating contact with service provider to confirm first treatment session	.74	−.06	.80
G4i. Initiating contact with service provider to obtain information about a youth’s progress in treatment	.63	.04	.78
H8b. Encouraging the youth to initiate treatment within 14 days after referral	.22	.53		.55
H8c. Encouraging the youth to engage in services by attending at least 3 sessions within the first 6 weeks	.17	.56		.53
H8d. Encouraging the youth to participate in treatment services for 90 days or longer	.19	.51		.63
H8e. Informally sharing information between JJ supervision and treatment provider staff regarding youth participation	−.06	.75		.67
H8f. Formally sharing information between JJ supervision and treatment provider staff regarding youth participation	−.11	.78		.72
H8g. Communicating with youth/family about the youth’s progress in treatment	−.03	.81		.78
H8h. Initiating contact with service provider to confirm first treatment session	.00	.69		.79
H8i. Initiating contact with service provider to obtain information about a youth’s progress in treatment	−.11	.84		.83

Notes: Items with “G” labels referred to Importance Scale; items with “H” labels were the corresponding Use Scale items

(How often used in practice).

For EFA (Exploratory factor analysis) results, entries represent values from joint (two-factor) factor pattern matrix. For CFA (Confirmatory factor analysis), entries are estimated relationships for proposed two-factor model. Entries in bold represent values for the intended factor. EFA correlation between factors = .49; CFA correlation = .42.

Item G4a/H8a, “Encouraging every youth with a substance problem to initiate treatment services” was omitted based on EFA results (both importance and use scales).

Table 5.

Means and Standard Deviations for Cascade Service Scales

		Importance	Use
Service Area	N	M (SD)	M (SD)
Screening	444	42.8 (6.1)	38.4 (8.7)
Assessment	443	41.9 (6.7)	32.8 (9.5)
Standard Referral	444	44.2 (5.2)	39.9 (7.4)
Active Referral	442	30.5 (10.9)	24.7 (10.5)
Treatment Support	441	42.7 (5.9)	38.9 (8.0)