Patterns and predictors of compliance with utilization management guidelines supporting a state policy to improve the quality of youth mental health services

Abstract

Despite a need to improve community mental health services for youths, little is known about compliance with state policies created to improve the quality of services in these settings. This study examined rates, patterns, and predictors of compliance with utilization management guidelines developed by the state of Texas to support a public health policy based on empirical evidence of effective mental health services (i.e., an evidence-based policy). Compliance was defined as authorizing policy-recommended service packages, whereas policy “overrides” occurred when recommended service packages were not authorized. The study sample consisted of 688 youths from ethnically and economically diverse backgrounds. Clinics reported that forty-six percent of youths were not authorized the policy-recommended service package. Overrides were primarily based on level of intensity. Most often, authorized services were less intensive than those recommended by the state guidelines. Higher severity at intake across multiple indicators was associated with authorizing less intensive services than what the policy guidelines recommended. Future studies evaluating system-level efforts such as state mental health policies should pay close attention to levels of service intensity, and their relation to the needs of youth in community settings.

There is a critical need and strong push to improve the quality of mental health services provided to youth in publicly funded mental health settings (Bruns et al, 2016). Consequently, a myriad of quality improvement strategies are being employed to increase the use of evidence-based practice (EBP; Bruns & Hoagwood, 2008; Powell et al., 2015). Despite their potentially wide-reaching effects, quality improvement strategies that promote EBP at the level of mental health systems, such as policies, are less frequently examined empirically than those at the level of individual organizations (e.g. organization culture and climate). System-level approaches are examined less because they are often complex (i.e., composed of multiple components and processes) and occur in uncontrolled, naturalistic settings (Chambers & Rupp, 2015; Finnerty et al., 2009; Hoagwood et al., 2015).

Although under-studied, policies that support EBP are widely advocated as powerful drivers of quality improvement in public sector mental health care (Chambers & Rupp, 2015; Finnerty et al., 2009; Garfield, 2009; Hoagwood et al., 2015; Tanenbaum, 2005). Brownson and colleagues define evidence-based public policy as: “policy developed through a continuous process that uses the best available quantitative and qualitative evidence…to improve public health outcomes” (Brownson, Chriqui, & Stamatakis, 2009, p. 1580). In the United States, the federal government and states have developed several policies to improve mental health services by increasing access to EBP (Chambers, Ringeisen, & Hickman, 2005; Ganju, 2008; Garfield, 2009; Hoagwood et al., 2015; McHugh & Barlow, 2010). Once a quality improvement policy is developed and enacted, compliance with the policy becomes a pivotal aspect of the quality improvement process. As such, examining compliance with policies is an area that is ripe for empirical research.

Texas’ Resiliency and Disease Management (RDM, now known as Resilience and Recovery; Ganju, 2008; Jensen-Doss, Hawley, Lopez, & Osterberg, 2009; Painter, 2009, 2012) initiative is an example of an evidence-based public policy, and is indicative of other research that highlights the importance of initiatives that use research evidence in policy making (Chambers et al., 2005). In 2003, Texas legislature passed House Bill 2292 with the goal of improving the quality and cost efficiency of the state public mental health system by adopting a managed care model (Coleman et al., 2005), and mandating the use of EBPs. To implement the policy, the state of Texas gathered a panel of national experts in youth mental health and EBP who developed utilization management guidelines to facilitate compliance with the policy and aid in treatment decision-making (Hoagwood, 2003), an approach which other states have also used (see Daleiden & Chorpita, 2005). This panel of experts created the guidelines through a consensus process that accounted for syndrome clusters found in the Texas state system.

The Texas utilization management guidelines, known as the “Child and Adolescent Texas Recommended Assessment Guidelines” (CA-TRAG), outlined 8 empirically-based service packages that consisted of a level of service intensity (i.e., dose), and an EBP (e.g., cognitive-behavioral therapy) specific to a youth’s (ages 3 to 18) psychological disorder or “problem type” (Hoagwood, 2003; Texas Department of State Health Services, 2004b, 2005, 2006, 2007, 2008). Service packages were of lower or higher intensity depending on whether the youth’s functional impairment was moderate or high, and thus did or did not require intensive case management in addition to the EBP. At the lowest level of intensity, care involved only medication management (Service Package 4 – Medication Management/After Care Services). The next level of intensity (brief outpatient services) involved the addition of an evidence-based psychosocial treatment (i.e., behavioral parent training or cognitive-behavioral therapy), and the highest intensity service packages involved those interventions plus intensive case management/wraparound services or multisystemic therapy (intensive outpatient services). All service packages offered medication management services. Notably, there could be instances where assessment scores or other circumstances disqualified youth from receiving services. However, specific scenarios under which youth could be deemed ineligible for services were not outlined in the utilization guidelines, and were addressed by authorizers on a case-by-case basis. Services captured by the packages are indicated in Table 1.

Table 1.

Texas 2004-2006 Youth Mental Health Service Packages

Service Package	Disorder	Intensity Level	Service Package Description
1.1	Externalizing Disorders (e.g., ADHD, Conduct or Oppositional Defiant Disorder)	Low	Skills Training. Youths with externalizing disorders and a moderate level of functional impairment. The focus of intervention is on psychosocial skill development and the enhancement of parenting skills, especially in child behavior management… This package is generally considered short-term and time-limited.
1.2	Internalizing Disorders (depressive or anxiety disorders)	Low	Therapy. Youths with depressive or anxiety disorders and a moderate level of functional impairment. The focus of intervention is on using Cognitive Behavioral Therapy (CBT). This package is generally considered short-term and time-limited.
2.1	Externalizing Disorders (MST option)	High	Skills Training-Multi-Systemic Therapy. Youths with externalizing disorders and high levels of severe disruptive or aggressive behaviors who are in the juvenile justice system and at high risk for out of home placement or further penetration in the juvenile justice system due to presenting behaviors. Intensive parent-to-parent peer support is available to the family. The family service plan is developed using a wraparound planning approach. Multi-Systemic Therapy is recommended if available.
2.2	Externalizing Disorders	High	Skills Training + wrap around service. Youths with externalizing disorders and moderate to high functional impairment at home, school or in the community. The need for intensive case management and significant caregiver support is indicated. The family service plan is developed using a wraparound planning approach. Multi-Systemic Therapy is either not appropriate (due to lack of juvenile justice involvement) or unavailable.
2.3	Internalizing Disorders	High	Therapy + wrap around services. Youths with depressive or anxiety disorders and a moderate to high level of problem severity or functional impairment. The focus of intervention is on CBT… Multiple family concerns and significant parental stress indicate the need for intensive case management and the availability of parent-to-parent peer support. The family service plan is developed using a wraparound planning approach.
2.4	Major Disorders: Bipolar, Schizophrenia, Major Depression with Psychosis, or other psychotic disorders	High	Medication + wrap around services. Youth with severe disorders who are not yet stable on medication. The major focus is on stabilizing the youth and providing information and support to the family.
3	Treatment Foster Care	High	Wrap around services. Youth at imminent risk of residential treatment placement. Parents retain custody although the youth may be at high risk of relinquishment of legal custody to the State to access residential mental health treatment or residential placement by the juvenile justice system. It is clinically determined that the child and family can progress with intensive community treatment, including treatment foster placement for the child, in lieu of residential treatment. Extensive training and support are available to the custodial parent or caretakers through clinicians and/or the treatment foster parent. The family service plan is developed throughout a wraparound planning approach.
4	Aftercare Services	Low	Medication Management. Youths who have stabilized in terms of problem severity and functioning and require only medication management to maintain their stability. Service package 4 can only be authorized if: 1) the caregiver refuses the recommended package, wants medication-only services and medication is clinically indicated; or 2) if the youth is NOT Medicaid eligible and the recommended service package is not available due to limited resources, but severe presenting problems that are responsive to medication suggest an authorization for service package 4 during the waiting period.
9	Not Eligible for Services	Low	Client is not eligible for services due to various findings from initial intake assessment. *This is not an official service package*

In terms of resources to aid clinics in the adoption and implementation of EBPs, the state provided EBP training to clinic providers which included workshops and treatment fidelity tools (Jensen-Doss et al, 2009). Furthermore, the publicly managed mental health care system approach in Texas compensates providers through managed care plans for all contracted health services for each person, and through fee-for-service payments (Hogg Foundation for Mental Health, 2016). This Texas mental health care system serves youth and families with private and public health insurance, and makes special provisions for medically indigent individuals.

Service package recommendations were made by an assessor based on a client’s primary diagnosis, functional impairment, and youth and family contextual factors. These domains were assessed via scores on the Ohio Youth Functioning and Problem Severity scales (Ogles, Melendez, Davis, & Lunnen, 2001), and ratings of specific domains such as risk of self-harm and co-occurring substance use on a scale from 1 (no notable limitations) to 5 (extreme limitations). Texas also conducted a validation study of these assessment procedures (Texas Department of State Health Services, 2004a). Currently, Texas continues to use service packages that reflect varying levels of care (Texas Department of State Health Services, 2017a). Under RDM, clinics are asked to follow the guidelines (hereafter referred to as “compliance”), but are allowed to “override” guideline recommendations with justification (hereafter referred to as “overrides”). Compliance and override decisions are made by clinic “authorizes,” who are different from the assessors. The authorizer role was embedded into the RDM policy in response to demands for an authority/provider split. During the development of the RDM policy, some argued that self-interested parties were responsible for determining levels of care, necessitating a position of an authorizer who reports to an independent chain of command. RDM level of care authorizers provide an ostensibly objective pre-authorization of service, and introduce a human element to temper the level of care algorithm, if needed.

Empirically examining compliance with state-developed utilization management guidelines is imperative given that it may have implications for many other states operating under managed care systems (e.g., Chambers et al., 2005; Ganju, 2008; Garfield, 2009; Harburger, Stephan, & Kaye, 2013; Hoagwood et al., 2015; Managed Care, 2016; Schuffman, Druss, & Parks, 2009; Willging et al., 2007; Young, Plotner, Damon, & Hight, 2008). Furthermore, several states (e.g., Arizona, Delaware, Hawaii, Michigan, New Jersey, North Carolina) also allocate mental health services based on the level of service intensity clients need (Daleiden, Pang, Roberts, Slavin, & Pestle, 2010; Fallon et al., 2006; Kraus, Baxter, Alexander, & Bentley, 2015; Pires & Grimes, 2006; Pumariega, Millsaps, Moser, & Wade, 2014). For example, Delaware also developed service packages similar to those used in Texas (Pires & Grimes, 2006), while Hawaii uses the Child and Adolescent Level of Care (CALOCUS), a standardized instrument, to determine assignment of treatment packages (Daleiden & Chorpita, 2005; Fallon et al., 2006; Sowers, Pumariega, Huffine & Fallon, 2003). However, empirical research on service intensity models within states mandating the use of EBP has yet to emerge. Overall, research on compliance with utilization management guidelines is limited, and based on the few published studies that exist, services received often do not match the level of service intensity recommended by guidelines (Bickman, Karver, & Schut, 1997; Cohen, Wiley, Oswald, Eakin, & Best, 1999). This is an important area for further research because examining compliance data will provide useful information to states about actual service allocations under a policy mandate, and potential barriers to meeting client needs.

One previous study on RDM focused on clinician attitudes toward EBP (Jensen-Doss et al., 2009), and two others examined client outcomes after the policy was enacted, and found significant improvements (Painter, 2009, 2012). However, no study to date has examined compliance with the guidelines for all service packages. The youth mental health needs addressed by these guidelines are similar to those encountered in diverse communities across the U.S. where resources are limited, and thus must be allocated efficiently. By offering services at different intensity levels, Texas aims to efficiently provide the needed level of service at the appropriate time (Ganju, 2008).

The purpose of this study was to empirically examine compliance with the Texas utilization management guidelines, thus expanding research on a system-level, quality improvement policy, which is a critically understudied area. Utilizing data during the first two years of RDM implementation, we examined compliance to all service packages available, and hypothesized that overrides would most often be related to service intensity rather than service problem type (for example, a youth might receive less intensive services than recommended, but could still receive a service package designed for their problem type). This prediction is grounded on level of care decision-making research demonstrating service intensity decisions are particularly difficult to judge correctly (Bickman et al., 1997; Cohen et al., 1999), and on the fact that there were less service package options based on problems type. In addition, due to known clinic resource constraints, we hypothesized that compliance with less intensive service packages would be more common than guideline compliance with more intensive service packages.

In line with implementation theory (Mendel, Meredith, Schoenbaum, Sherbourne, & Wells, 2008) and research citing the importance of client-level factors when implementing EBPs in the community (e.g., Ringle et al., 2015), we also examined client-level predictors of compliance. Moreover, client characteristics are of interest because they are the main factors used to determine service package recommendations in Texas. As such, examining client factors sheds light on the types of clients for whom clinics feel they can and cannot comply with the utilization management guidelines. To our knowledge, no study to date has examined client-level predictors of compliance to mental health utilization management guidelines, so prior literature could not guide the generation of hypotheses. However, given that the more intensive service packages were likely more costly and difficult to implement, we hypothesized that the following youth characteristics would be associated with receiving less intensive services than recommended: (a) more impaired functioning, (b) greater problem severity, (c) having a serious mental illness diagnosis (e.g., Bipolar, Schizophrenia), (d) having multiple diagnoses, (e) uninsured status, (f) older age, (g) higher juvenile justice involvement, (h) higher risk of self-harm and harm to others, (i) more family problems, (j) more severe co-occurring substance use, (k) more school problems. Additionally, we also explored whether compliance varied as a function of youth ethnic minority status.

Methods

Participants

As part of a National Institute of Mental Health grant (MH079918, PI: Jensen-Doss) to understand the initial impacts of the psychosocial services included in RDM, medical records were extracted for all child and adolescent clients (N = 727) who received psychosocial services at some point in their first year of services from 2004 to 2006, the first two years after the RDM policy was enacted. Out of the 727 youth, data on all variables pertinent to the present study was available for only 688 youth. The age, gender, and ethnicity of youth who were excluded (n = 39) did not differ significantly from those included. Table 2 described participant demographic information. Of note, client characteristics were used to determine what service package they received, and the only other way that clients could have contributed to compliance decisions was by expressing preference for a treatment other than what was recommended by the guidelines. Client preference was captured as one of the reasons for overrides listed below.

Table 2.

Demographic and Clinical Characteristics

	Full Sample (n=688)	Compliance (n=320)d	Less Intensive (n=313)	More Intensive (n=50)
Variable	Mean (SD, range) or % (n)
Demographic variables
Age	11.14 (3.84, 4-17.75)	11.06 (3.91, 4-17.75)	11.09 (3.73, 4.33-17.74)	11.72 (4.14, 4.05-17.59)
Male	60% (n=413)	59.4% (n=190)	61.7% (n=193)	56% (n=28)
African American	42.6% (n=293)	38.8% (n=124)	47.6% (n=149)	34% (n=17)
Hispanic	37.6% (n=259)	39.7% (n=127)	34.5% (n=108)	46% (n=23)
Caucasian	16.9% (n=116)	18.7% (n=60)	15.3% (n=48)	14% (n=7)
Asian/Other	2.9% (n=20)	2.8% (n=9)	2.6% (n=8)	6% (n=3)
Ethnic Minority Status	83.1% (n=572)	81.3% (n=260)	84.7% (n=265)	86% (n=43)
Insured	67% (n= 461)	65.9% (n=211)	68.7% (n=215)	66% (n=33)
Clinical variables
Functioninga*	36.49 (14.68, 0-80)	37.66 (14.19, 2-69)*	34.41 (14.36, 0-80)*	43.06 (16.78, 4-79)
Problem Severitya*	38.26 (17.64, 1-95)	36.08 (16.43, 1-93)*	41.06 (17.6, 7-95)*	32.86 (21.1, 1-91)
Juvenile Justice Involvementb	1.11 (.46, 1-5)	1.10 (.47, 1-5)	1.11 (.46, 1-4)	1.04 (.20, 1-2)
Risk of Harm to Selfb	1.32 (.66, 1-5)	1.34 (.68, 1-5)	1.28 (.63, 1-4)	1.33 (.63, 1-3)
Risk of Harm to Othersb*	2.83 (.93, 1-5)	2.67 (.90, 1-5)*	3.06 (.87, 1-5)*	2.35 (1.12, 1-5)
Family Problemsb	2.61 (.92, 1-4)	2.56 (.92, 1-4)	2.64 (.95, 1-4)	2.67 (.80, 1-4)
Substance Useb*	1.22 (.66, 1-5)	1.26 (.72, 1-5)*	1.17 (.54, 1-4)*	1.23 (.80, 1-5)
School Problemsb*	3.1 (1.2, 1-5)	2.86 (1.21, 1-5)*	3.4 (1.1, 1-5)*	2.61 (1.35, 1-5)
Diagnosis
ADHDc	50.3% (n=346)	50.6% (n=162)	54% (n=169)	26% (n=13)
Anxietyc	7.6% (n=52)	9.4% (n=30)	5.4% (n=17)	10% (n=5)
Conductc*	27% (n=186)	24.4% (n=78)*	30.7% (n=96)*	20% (n=10)
Diagnosis
Depressionc*	36.9% (n=254)	32.2% (n=103)*	36.4% (n=114)*	70% (n=35)
Serious Mental Illnessc	27.3% (n=188)	26.9% (n=86)	24.3% (n=76)	50% (n=25)
Otherc	24.3% (n=167)	23.8% (n=76)	21.1% (n=66)	44% (n=22)
Multiple Diagnoses	52.2% (n=359)	49.1% (n=157)	52.1% (n=163)	72% (n=36)
Number of Treatment Sessions	8.23 (7.83, 1-53)	9.35 (8.47, 1-44)	7.09 (6.86, 1-53)	8.1 (8.47)
Weeks in Treatment	39.76 (14.7, 4.57-52)	39.38 (14.5, 5.43-52)	40.7 (14.3, 4.57-52)	36.91 (17.67)
Reasons for overrides
Resources	13.7% (n=94)	0% (n=0)	29.4% (n=92)	0% (n=0)
Limitations
Clinician Override	7.4% (n=51)	0% (n=0)	0% (n=0)	100% (n=50)
Client Preference	2.5% (n=17)	0% (n=0)	5.4% (n=17)	0% (n=0)
Other	29.8% (n=205)	0% (n=0)	65.2% (n=204)	0% (n=0)

^aParent-reported Ohio Functioning and Problem Severity scores can range from 0 to 100. Lower functioning, and higher problem severity scores indicate more impairment.

^bDomain was rated by intake assessor on a scale from 1 (no notable limitations) to 5 (extreme limitations). Higher ratings were indicate greater difficulties in these areas.

^cPercentage value denotes % within category; numbers do not add to 100

^dDoes not include 5 instances of only problem-type overrides

^*Significant difference between Less Intensive and Compliance groups. The More Intensive group was not included in analyses as it lacked an appropriate comparison group consisting of youth who were not recommended for services and did not receive them.

Services were rendered at the 4 community mental health clinics of the largest mental health authority in the most populous county in Texas. Like in many other Texas counties, the urban area served by the mental health authority captured in this study is both ethnically and economically diverse. The Texas Department of State Health Services contracts these health authorities across all counties in Texas who in turn are responsible for the allocation of public funding, and the delivery of all mental health services (Texas Department of State Health Services, 2017a).

Procedure

Agencies collected client demographic and clinical data as part of regular clinic procedures. This study utilized demographic (age, gender, ethnicity, insurance status), clinical (diagnosis, problem severity and functioning), and administrative (e.g., guideline compliance) data collected at intake. Clinic staff extracted, de- identified, and provided the data to the research team. The extraction was largely automated due to the existence of variables within the agency’s electronic medical records system indicating, for example, that the agency billed for a session with a therapist. After the data were provided to the research team, extensive data cleaning took place, with the goal of ensuring that all included cases met the study inclusion criteria. The Texas A&M Institutional Review Board and the mental health authority’s Human Subjects Protection Committee approved all study procedures.

Measures

Demographic and clinical variables.

Demographic variables included the youth’s age, gender (male or female), ethnic minority status (Hispanic, African American, Asian/Other or Caucasian), and insurance status (insured or uninsured). Clinical data included the youth’s chart diagnosis, number of diagnoses (one diagnosis vs. more than one diagnosis), weeks in treatment, and scores from two assessment tools. Youth functioning and problem severity were assessed by the Ohio Youth Functioning and Problem Severity Scales, Parent-Report (Ogles et al., 2001). The functioning scale includes 20 items assessing a youth’s ability to complete daily activities and maintain relationships, and the problem severity scale (also 20 items), measures the frequency and severity of emotional and behavioral problems. Items reference the past 30 days and are rated on a 5-point Likert scale from “Not at All” to “All the Time,” with lower functioning scores and higher problem severity scores indicating more impairment. Psychometric data on the Ohio Scales indicate acceptable reliability and validity (Ogles et al., 2001).

The other assessment tool consisted of assessor ratings of a variety of clinical and functional domains. The following domains were rated by intake assessors on a scale from 1 (no notable limitations) to 5 (extreme limitations): (a) juvenile justice involvement (1-no involvement in the last 90 days; 5-rearrested within past 90 days regardless of offense/outcome); (b) risk of harm to self (1-no current suicidal ideation; 2-ideation with intent, plans, and means without adequate safety plan); (c) risk of harm to others (1-interacts with others appropriately and with respect; 5-significant threats to hurt others with weapons; (d) family problems/resources (1-stable family environment; 5-caregiver unable to care for youth); (e) co-occurring substance use (1-no substance use; 5-evidence of addiction to substances); and (f) school problems (1- no behavior problems; 5-suspended/expelled/dropped out of school/daycare). Ratings on these domains were utilized as predictor variables in the current study.

Compliance.

Under RDM, the intake assessment was used to determine each client’s “recommended service package.” Afterwards, a service authorizer determined whether to authorize or not authorize the provision of this service package. Thus, there were two variables in the database: (a) recommended service package, (b) authorized service package. Differences between these two variables were used to generate the guideline compliance variable. Cases in which the authorized service package matched the recommended service package were coded as “compliance.” Cases where the two did not match were coded as “overrides.” Overrides were further broken down into changes based on service intensity and problem type. A service intensity override was operationalized as receiving more or less intensive services. Client problem type override codes included, assigning: (a) someone with externalizing problems to a service package for internalizing disorders or severe mental illness (b) someone with internalizing problems to a service package for externalizing disorders or severe mental illness, and (c) someone with severe mental illness to a service package for externalizing or internalizing disorders. Clinics were also required to report the primary reason for the override from 4 different options: (a) resource limitations, (b) client preferences, (c) clinician override (i.e., clinician judged the recommended service package as not clinically appropriate), or (d) other (e.g., institutionalization in a juvenile justice setting).

Data Analytic Plan

All study analyses were conducted using SPSS 24. Frequency analyses identified rates of compliance and reasons for overrides and chi square tests examined differences in rates of overrides. Clients were nested within 4 clinics and within 6 service authorizers, introducing dependencies into the data. To account for this, we used Generalized Estimating Equations (GEE) with robust standard errors for the analyses predicting compliance (Hanley, Negassa, & Forrester, 2003). Since authorizers accounted for 39% of the variance in compliance, they were used as the level 2 nesting variable, while “clinics,” which only accounted for 2.3% of the variance, was not included in the models¹. Binary logistic GEE was used to predict compliance from client demographic variables (age, ethnicity, insurance status), and clinical variables (functioning, problem severity, diagnosis, number of diagnoses, juvenile justice involvement, risk of self-harm, risk of harm to others, family problems, co-occurring substance use, school problems) at intake. Data missing for variables of interest was low (<5% for all variables), and missing completely at random (MCAR) (χ²= .188, p=.664); therefore, it was handled by list-wise deletion.

Results

Patterns of Compliance and Reasons for Overrides

Forty-six percent (n = 320) of youth received mental health services that complied with the RDM guidelines. Table 3 presents recommendation frequencies and compliance rates for the service packages, and Table 4 patterns of overrides by service package. Service packages 1.1, 1.2 and 2.2 were frequently recommended, while 2.1, 3, and 4 were rarely recommended (less than or equal to3 times). Service package 2.1 (multi-systemic therapy) was recommended 3 times, but was never complied with, although in two of these instances youth did receive service package 2.2, which is of the same intensity level and also for externalizing issues, therefore, for the purposes of this study, these two instances were not considered overrides. Service package 4 (medication management only) was recommended and complied with 1 time, and service package 3 (only wrap around services) was never recommended. Of those that were recommended more than 3 times, 2.4 (93% compliance rate), 1.2 (63% compliance rate) were often complied with, and 2.2 (33% compliance rate) and 2.3 (30% compliance rate) were frequently overridden. Overrides based on service intensity only (n = 356) occurred more frequently than overrides based only on client problem type (n = 5). Overrides based on both service intensity and client problem type occurred 7 times.

Table 3.

Service package frequencies and compliance rates

Service Packag e	Treatment Type	Intensit y Level	Disorder	Recommendatio n Frequency	Complianc e Frequency	Complianc e Rate (%)
1.1	Skills Training	Low	Externalizing	180	84	47
1.2	Therapy	Low	Internalizing	176	111	63
2.1	Skills Training (Multi-Systemic Therapy)	High	Externalizing (MST option)	3	2	66
2.2	Skills Training + wrap around service	High	Externalizing	186	62	33
2.3	Therapy + wrap around services	High	Internalizing	33	10	30
2.4	Medication + wrap around services	High	Serious Mental Illness	60	50	83
3	Wrap around services	High	Treatment Foster Care	0	N/A	N/A
4*	Medication Management	Low	Any disorder	1	1	100
9	Not Eligible for Services	Low	--	49	0	0

^*This service package does not apply to overrides based on problem type and is of lower intensity relative to all service packages except not being eligible for services

Table 4.

Overrides by service package

	Authorized Service Package:
Recommended Service Package:	Intensity level	Disorder	1.1	1.2	2.1	2.2	2.3	2.4	3	4a	9	Total overrides per service package
1.1	Low	Externalizing		1						95		96
1.2	Low	Internalizing	2			1				62		65
2.1	High	Externalizing	1									1
2.2	High	Externalizing	78							46		124
2.3	High	Internalizing	2	18						3		23
2.4	High	SMI*	1	3		2				4		10
3	High	Foster Care										0
4a	Low	Any disorder										0
9	Low	--	19	13		3	2			12		49

^a.This service package does not apply to overrides based on problem type and is of lower intensity relative to all service packages except not being eligible for services Note: Bolded numbers indicate overrides to provide more intensive services, underlined numbers indicate overrides to provide less intensive services, and italicized numbers indicate problem type overrides.

^*Serious mental illness

To understand potential differences among authorizers, we also examined patterns of compliance by authorizer (see Table 5). There was a broad range in the number of cases reviewed by each authorizer (2 to 387), and their compliance rates ranged from 10% (Authorizer 6) to 82% (Authorizer 4). Examination of the specific service package recommendations reviewed and the types of authorizations made by each authorizer suggested possible differences in the types of clients reviewed by authorizers. For example, all of the service package 9 recommendations were reviewed by Authorizer 5, and 90% of the authorizations issued by Authorizer 6 were for medication only. To further explore the possibility that these authorizers were working with different kinds of clients, we examined whether clients reviewed by the 6 authorizers differed significantly on any of the candidate predictor variables, and the only significant difference was on risk of harm to others [F(5,682 = 4.47, p = .001]. Post Hoc Tukey’s tests indicated that Authorizer 6, the authorizer with the lowest compliance rate, authorized services for clients with higher risk of harm to others (M = 3.2, SD = .8) than Authorizer 4 (M = 2.7, SD = .9; Tukey’s p = .003) and Authorizer 5 (M = 2.8, SD = .9; Tukey’s p = .012).

Table 5.

Authorizer service package recommendations, authorizations and overall compliance

Service Package				Recommendation Frequency						Authorization Frequency
	Intensity	Disorder	Authorizer	1	2	3	4	5	6	1	2	3	4	5	6
1.1	Low	Externalizing		0	0	11	40	106	23	1	2	5	70	108	1
1.2	Low	Internalizing		1	0	7	50	101	17	1	0	4	56	81	4
2.1	High	Externalizing		0	0	0	0	3	0	0	0	0	0	0	0
2.2	High	Externalizing		1	2	8	77	84	14	0	0	0	46	24	0
2.3	High	Internalizing		0	0	1	12	20	0	0	0	1	6	5	0
2.4	High	SMI*		0	1	2	29	23	5	0	1	2	29	17	1
3	High	Foster Care		0	0	0	0	0	0	0	0	0	0	0	0
4*	Low	Any disorder		0	0	0	0	1	0	0	0	17	1	152	53
9	Low	--		0	0	0	0	49	0	0	0	0	0	0	0
			Total	2	3	29	208	387	59	2	3	29	208	387	59
Compliance Frequency				1	1	11	170	131	6
Compliance Rate (%)				50	33	38	82	34	10

^*Serious mental illness

The provision of less intensive services than recommended (n = 313) was significantly more common than the provision of more intensive services (n = 50; χ²= 195.968, df= 1, p<.001). Notably, with the exception of one youth, the group of youth who received more intensive services was made up entirely of youth who were not eligible for treatment, but for whom services were still authorized.

Reasons for overrides in descending order included, other (56 %), resource limitations (26%), clinician override (14%), and client preference (4.7%; see Table 2). The reason “other” was frequently used for overrides to service packages 1.1 (45% of the time, n=92), 1.2 (28%, n=57), and 2.2 (23%, n=48), resources limitations was frequently used for overrides to service package 2.2 (75%, n=69), clinician override was frequently used for overrides to “service package” 9 (not eligible for treatment; 96% of the time, n=49), and client preference was frequently used for service packages 2.2 (41%, n=7) and 2.3 (29%, n=5).

Client-Level Predictors of Compliance

Given that overrides based on problem type occurred too infrequently, further analysis of factors associated with compliance were based on service intensity overrides only. Table 2 presents client characteristics of the Compliance, Less Intensive, and More Intensive groups. As mentioned above, all but one youth in the More Intensive group were originally not eligible for mental health services, but still received them. Although anecdotal information from clinics suggests that youth were rarely, if ever, turned away from services, an appropriate comparison group for the more intensive group would be a compliance group comprised of youths who were not recommended for services and did not receive them; as there were no data available for such a group, the more intensive group was not utilized in the predictor analyses. As such, predictor analyses focused on understanding differences between the less intensive (n = 313) and intensity compliance groups (n = 325).

When entered separately, significant predictors of compliance included functioning, problem severity, school behavior problems, risk of harm to others, co-occurring substance use, and diagnosis (see Table 6). Specifically, clients with worse functioning (OR=.985, p<.001), higher problem severity (OR=1.02, p=.015), higher risk of harm to others (OR=1.61, p<.001), more school problems (OR=1.48, p<.001), and who had a diagnosis of depression (OR=1.37, p<.05) or a conduct-related disorder (OR=1.37, p<.05) at intake were more likely to receive less intensive services than those recommended by the RDM guidelines. Conversely, youth with more substance-use issues were more likely to receive services that complied with the guidelines (OR=.779, p=.02). There were no differences in compliance based on other client characteristics including: having multiple diagnoses, ethnic minority status, insurance status, age, juvenile justice involvement, risk of self-harm, and family problems. To understand the independent contributions of these predictors, all significant youth variables were simultaneously entered into a model predicting receiving less intensive services. In this analysis, only higher risk of harm to others (OR=1.3, p<.05), more school problems (OR=1.36, p<.001), less substance use (OR=.76, p<.01), and a diagnosis of depression (OR=1.5, p<.001) were significantly associated with receiving less intensive services. Functioning, problem severity, and a diagnosis of conduct-related problems were no longer significant.

Table 6.

Binomial Logistic Regression Analyses Examining Relations Between Client Factors and Compliance Entered Separately

	Less Intensive Services vs Compliancea
Client variables (at intake)	B	Odds Ratio (95% CI)
Functioning	−.015**	.985 (.98, .99)
Problem Severity	.016*	1.02 (1.00, 1.03)
Juvenile Justice Involvement	.005	1.00 (.67, 1.49)
Risk of Harm to Self	−.153	.858 (.70, 1.05)
Risk of Harm to Others	475***	1.61 (1.28, 2.01)
Family Problems	.080	1.08 (.87, 1.34)
Substance Use	−.250*	.779 (.63, .96)
School Problems	.392*	1.48 (1.41, 1.56)
Age	−.001	.999 (.99, 1.01)
Ethnic Minority Status	.243	1.27 (.99, 1.64)
Insurance Status	.143	1.15 (.89, 1.5)
Diagnosis
ADHD	.208	1.23 (.82, 1.85)
Anxiety	−.383	.68 (.40, 1.16)
Conduct	.319*	1.37 (1.05, 1.81)
Depression	.318*	1.37 (1.02, 1.86)
Serious Mental Illness	−.026	.974 (.66, 1.44)
Other	−.109	.90 (78, 1.04)
Multiple diagnoses	.114	1.12 (.90, 1.39)

^aThe reference category is: Compliance

^*p<.05

^**p<.01

^***p<.001

Post Hoc Analyses to Further Explore Less Intensive Overrides

To better understand the finding that more severe presentations were associated with receiving less intensive services, we conducted post hoc analyses of the less intensive group. We separated the group into three subgroups based on the types of less intensive services overrides: (a) the Highly Intensive to Moderately Intensive group (n = 103) received brief outpatient services and medication instead of also receiving intensive case-management/wraparound services; (b) the Highly Intensive to Medication Only group (n = 53) received medication only instead of receiving all three types of services; and (c) the Moderately Intensive to Medication Only group (n = 157) received medication only instead of also receiving brief outpatient services. Table 7 includes descriptive statistics for client characteristics, as well as reasons for overrides for each less intensive subgroup.

Table 7.

Demographic and Clinical Characteristics of Less Intensive Subgroups

	Highly Intensive to Moderately Intensive (n=103)	Highly Intensive to Medication Only (n=53)	Moderately Intensive to Medication Only (n=157)
Client characteristics	Mean (SD, range) or % (n)
Functioning	35.02 (15.26, 0-69)a	28.83 (14.25, 0-51)a,c	35.89 (13.40, 4-80)c
Problem Severity	42.35 (18.95, 15-95)a,b	49.77 (16.11, 12-90)a,c	37.28 (16.01, 7-82)b,c
Risk of Harm to Others	3.13 (.83, 1-4)	3.87 (.48, 2-5)	2.74 (.82, 1-5)
Substance Use	1.22 (.65, 1-4)	1.17 (.54, 1-4)	1.13 (.45, 1-4)
School Problems	3.67 (1.1, 1-5)a,b	4.04 (.59, 1-5)a,c	3.00 (1.07, 1-5)b,c
Diagnosis
Conduct	43.7% (n=45)b	45.3% (n=24)c	17.2% (n=27)bc
Depression	35.9% (n=37)a	28.3% (n=15)a	39.5% (n=62)
Reasons for overrides
Resources	82% (n=85)	2% (n=1)	3.8% (n=6)
Limitations
Clinician Override	0% (n=0)	0% (n=0)	0% (n=0)
Client Preference	14% (n=15)	0% (n=0)	1.3% (n=2)
Other	2.9% (n=3)	98% (n=52)	95% (n=149)

^a,b,cMeans/percentages with the same superscripts differ significantly from one another, p < .05

Binary logistic GEE was used to conduct pairwise comparisons of the client characteristics of these three groups. Compared to the other two groups, and consistent with initially being recommended to a lower intensity service package, youth in the Moderately Intensive to Medication Only group were the least clinically severe, as indicated by significantly higher functioning, lower problem severity, fewer school problems, and lower likelihood of having a conduct-related diagnosis, (all p’s < .05). Comparing the two groups initially recommended for highly intensive services indicated that clients in the Highly Intensive to Medication Only group had significantly higher problem severity and school problems, lower functioning, and were less likely to have a diagnosis of depression than the Highly Intensive to Moderately Intensive group (all p’s < .05). Examination of the reasons for overrides by group indicated that the two groups who were provided medication only were primarily done so for “other” reasons, whereas the Highly Intensive to Moderately Intensive group was primarily overridden due to resource limitations (82%), although 14% of that group was also overridden based on client preference.

Discussion

This study evaluated compliance with utilization management guidelines created by a state enacting an evidence-based policy to improve the quality of youth mental health services. We found that fewer than half of clients received services that were recommended by the guidelines. Overrides were almost always related to adjusting the intensity of services rather than the problem focus of the services, and most overrides resulted in less intensive services. Receiving less intensive services could be divided into whether youth received brief outpatient services or only medication, when a higher level of care was recommended. Overrides to provide more intensive services occurred only for youth who were not eligible for treatment. Several client characteristics were related to overrides, including higher problem severity and lower functioning, although the only independent predictors were risk of harm to others, school behavior problems, co-occurring substance use, and having a depressive disorder. Present findings are consistent with the few existing studies on levels of care that have found that correctly assigning individuals to the appropriate level of service intensity is usually low (Bickman et al., 1997; Bickman, Noser, & Summerfelt, 1999; Fallon et al., 2006; Friedman & Street, 1985; Sowers, Pumariega, Huffine, & Fallon, 2003).

The fact that most overrides resulted in the provision of less intensive services raises questions about whether these clinics lacked resources to comply with the guidelines, which was provided as a reason for 29% of the overrides. Since RDM was put into place without additional funds to clinics, this highlights the need for resources to align with evidence-based policies (Dickey, 2004; Rapp et al., 2005). Our findings suggest that one priority for states should be to assess clinic needs and potential barriers to compliance so as to prepare for such barriers before implementing an evidence-based policy. This supports the Framework for Dissemination’s proposal that capacity and needs assessment is an essential first step in the effective implementation of EBP (Mendel et al., 2008). Emerging implementation science research suggests that some states and counties reforming their publicly funded child mental health systems are starting to assess and address clinic needs in this way (see Beidas et al., 2013; Lau & Brookman-Frazee, 2016).

When overriding to provide less intensive services, there were two types of decisions that were made. In some cases, individuals were provided routine outpatient services and medication only, rather than also receiving intensive case management or wraparound services. Examination of reasons for overrides suggests overrides of this sort were primarily attributable to resource limitations. Notably, although most families did not request treatments other than what was recommended by the guidelines, when they did, they requested once weekly outpatient psychosocial treatment that included medication management rather than only medication management, indicating a patient preference for combination therapies over medication alone. In addition, youth with a depressive disorder were more likely to receive both outpatient services and medication rather than only medication, which may reflect extra precautions taken by community mental health clinics in regards to monitoring antidepressant use for youth as a response to U.S. Food and Drug Administration warnings (see National Institute of Mental Health, 2017).

The second type of override occurred when individuals received medications only, rather than also receiving outpatient and/or intensive services. These types of overrides were more difficult to understand, as the reasons provided for these overrides was typically “other.” Interestingly, among individuals initially recommended for the highest intensity services packages, those who received medication only had higher initial severity, lower initial functioning, and more school problems than those who received medication and outpatient services. It is possible that authorizers felt these individuals were too impaired to benefit from once weekly outpatient services and instead began their treatment with medication only. To be included in this study, all individuals had to have received psychosocial services at some point during their first year of services, so these individuals were likely moved to more intensive services at a later point. What is not known is whether these additional services were provided because clients were deemed better able to engage in services after being stabilized on medication or because medications were not sufficient to address the clients’ needs. Accordingly, these findings indicate that community mental health professionals are likely faced with decisions regarding the sequencing of medication and psychosocial treatment, which research has started to address. This research suggests benefits to sequencing psychosocial services prior to medication for externalizing conditions such as Attention-Deficit Hyperactivity Disorder (Pelham Jr et al., 2016), while a more nuanced and individualized approach may be needed for anxiety disorders (Keeton & Ginsburg, 2008).

Additionally, more complex youth problems, such as being a danger to others and more severe school problems, may have led clinics to believe that a psychosocial treatment meant for a specific diagnosis, such as the ones indicated by the RDM policy, were not an appropriate first-line option for such presentations. This is in line with concerns previously reported by community clinicians regarding the complex presentations of youth from community settings (Ringle et al., 2015). New effectiveness trials that examine these treatments in community settings where complex clinical presentations are more common are needed, and are starting to emerge (Jensen-Doss et al., 2018). For now, clinics and supervisors attempting to adopt EBPs that have only demonstrated efficacy in university-based clinics should keep in mind that adaptations are possible (Lau et al., 2017), and therefore these treatments could be used as the first-line option rather than only medication.

Although less common, more intensive services than were recommended were also provided. In all of these cases but one, these were youth who were not eligible for services. Unfortunately, we do not have further information on why these specific youths were not eligible for services. Concerns have been raised that utilization management techniques focused on cost effectiveness may lead to decreased service provision (Borenstein, 1990; Koike, Klap, & Unützer, 2000; Miller, 1996). However, present findings demonstrate that even within a managed care system, some youth received services even when not considered eligible. Another positive finding was that youth with more severe substance-use issues tended to receive services that were in compliance with the guidelines. This was the only clinical variable examined for which this was the case, however, based on the data currently available, it is unclear why this finding would be so, and is in need of replication.

Notably, there were some differences among the 6 authorizers who made the final decisions regarding the service packages that youth received. Authorizers differed in the number of service packages they had to make decisions about, and in their compliance rates. Additionally, the authorizer with the lowest compliance rate (10%) had to make service decisions for youths with higher risk of harm to others. This is congruent with the overall finding that more severe client presentations hindered compliance with the RDM utilization management guidelines. Unfortunately, other than their patterns of compliance and data about the youths they served, there was no information about authorizers and clinic/organization factors (there were 4 clinics from one organization) that would allow us to understand whether such factors may have influenced compliance decisions.

Presently most implementation science efforts focus on determining whether a specific EBP is provided, and if provided, whether it is provided with fidelity to a protocol, but this study’s findings suggest that the level of service intensity is also an important variable to account for when serving youths in community settings. In light of these findings, mental health professionals practicing in states under policies similar to RDM or in other community settings can expect to encounter youths with common diagnoses such as depression who also present with other complex problems which resources fall short of addressing. Future research on publicly-funded mental health systems should incorporate these other factors (e.g., risk of harm to others, school problems) that may influence compliance to providing the level of service intensity that best meets a youth’s needs. Additionally, mental health agencies and clinics may benefit from incorporating training that helps mental health professionals correctly determine the needed level of service intensity.

This study had multiple strengths. First, it is one of few studies to use a real-world, mental health services dataset to empirically examine an evidence-based policy (Hoagwood et al., 2015). This administrative dataset provided a large sample of ethnically (83% were of ethnic minority status) and economically (33% were not insured) diverse youth in the community. We also examined a myriad of client-level factors, which have been previously cited as pertinent to community EBP implementation efforts by community clinicians (Ringle et al., 2015). Furthermore, while empirical research on state child mental health initiatives is on the rise, most have focused on adherence to EBP protocols, and have not considered the possibility that measuring compliance more broadly—as done in the present study—may also provide elucidating data about youth mental health policies. Additionally, these findings speak to the latest iteration of the Texas policy, which continues to include treatment options at different levels of intensity, and assignments into these service packages is still determined by results from an intake assessment. Finally, a strong aspect of this study is its generalizability to other states that also employ utilization management strategies such as the one examined here (see Pires & Grimes, 2006).

Despite these strengths, present findings should be considered in light of several limitations. Although clinic staff did not hesitate to report overrides (i.e., they reported overrides for almost half of their clients), self-reported compliance is subject to bias and other errors (Donaldson & Grant-Vallone, 2002), which makes this one of the main limitations of the present study. Unfortunately, validating clinics’ reports was beyond the scope of this project. Another limitation is that the existing dataset that was used for this project only focused on the first two years of the implementation of the RDM policy, and we do not have data after that period. The present study also does not address what the actual, in-session treatment was and its impact on youth outcomes, therefore future studies on the actual treatment and outcomes as well as compliance during subsequent years are needed. Finally, given that this study was a secondary data analysis of a dataset designed to examine RDM’s psychosocial services, clients who were assigned to a medication-only package and remained in it were not included in the study. As such, this study may over-estimate the overall rate of overrides, since there were likely clients who were initially assigned to, and remained in, the medication-only group.

Texas’s RDM policy, along with Hawaii’s 1999’s consent decree to reform publicly funded child mental health services were pioneering, state-level, quality improvement efforts to prioritize EBP in children’s mental health services. Since these policies went into effect, several other states and large public mental health systems have followed suit (see Beidas et al., 2013; Hoagwood et al., 2014; Lau & Brookman-Frazee, 2016), yet more research on state policies is still needed. Present findings suggest that future quality improvement and EBP implementation studies evaluating system-level efforts such as state policies, should pay close attention to levels of service intensity. Although it might be reasonable to assume that pressures to comply with employer requirements might have biased information, it appears that clinics were comfortable reporting overrides. This is relevant to other states that, similarly to Texas, do not have sufficient resources to measure compliance in a more objective and thorough manner (Brunk, Chapman, & Schoenwald, 2015). Findings also point to the importance of conducting a careful assessment of needs and capacity to determine what will help clinics and therapists follow utilization management guidelines. In the present study, there were youth who did not qualify for services, but still received them; these were examples of positive guideline overrides that may have helped youth, suggesting that states should carefully consider the stringency of their guidelines’ exclusion criteria.

This study contributes to the research literature on state policies by examining compliance with utilization management guidelines based on an evidence-based policy created by a state and its relationship to youth factors, which is generalizable to other states employing similar managed care models. Finally, it is of great importance that mental health services researchers continue to pursue partnerships with states and other entities creating policies as this will create access to valuable real-world data that can inform how to best serve youth in community settings.

Highlights.

• Forty-six percent of youths in the present study were not authorized policy-recommended service package.

• Most often, authorized services were less intensive than those recommended by state guidelines. Receiving less intensive services meant youth received brief outpatient services or only medication, when a higher level of care was needed and recommended.

• Client characteristics related to overrides included being a risk of harm to others, having school behavior problems, having co-occurring substance use, and having a depressive disorder.