Treatment effects of parent-child focused evidence-based programs on problem severity and functioning among children and adolescents with disruptive behavior

Abstract

Objective:

This study examined the treatment effects of manualized parent-child focused evidence-based programs (EBPs), characterized by an emphasis on parental involvement and engagement, on functioning and problem severity among a statewide sample of children and adolescents referred to outpatient psychiatric clinic for serious and persistent disruptive behavior.

Method:

Propensity score matching was employed to account for baseline differences between children and adolescents (M_age = 8.4 years; 26% girls; 42% White; 10% black; 42% Hispanic; 5% other) who received parent-child focused EBPs (treatment group; n = 220) and treatment-as-usual (comparison group; n = 2,543).

Results:

Children and adolescents who received parent-child focused EBPs showed significantly greater reduction in problem severity compared to the comparison group, indicative of a drop below the clinical threshold for problem behavior. However, the pattern of improvement in problem severity was not paralleled by differential improvement in ratings of child functioning across treatment and comparison conditions. Finally, there were significant differences between the treatment and comparison groups pertaining to certain child and case characteristics that have potential implications for reaching high-risk populations of children and families.

Conclusions:

These findings support the potential of taking parent-child focused EBPs into scale to promote positive behavioral changes among children and adolescents. Parent-child focused EBPs may serve as an effective remedy that is less restrictive and more conducive to healthy development of children and adolescents.

Disruptive behavior among children and adolescents, including persistent oppositional and aggressive behavior, is among the most common presenting problems for referral to mental and behavioral health treatment services (Frick, 1998; Wilens, Biederman, & Spencer, 2002). Disruptive behavior is particularly concerning given its association with impaired child functioning and poor developmental health trajectories (McCart & Sheidow, 2016). For example, physical aggression in early childhood increases the risk for continued physical violence and delinquency in adolescence (Broidy et al., 2003). Children and adolescents with disruptive behavior also often experience interrelated risk factors, including exposure to trauma, co-occurring mental health problems, and disadvantaged family environment (Abrahamse, Junger, van Wouwe, Boer, & Lindauer, 2016). Thus, treatment of disruptive behavior is often characterized as a multifaceted process that also typically requires the participation of parents.

As primary caregivers, parents have a unique role to influence their children’s behavior (Hoeve et al., 2009). Evidence suggests that interventions that emphasize parental involvement and engagement in treatment provide promising outcomes for children and adolescents with disruptive behavior (Kaminski, Valle, Filene, & Boyle, 2008). However, we know little about the effects of these programs on child functioning and outcomes when implemented in large-scale, statewide settings. This study examined the treatment effects of a range of parent-child focused evidence-based treatments or programs that emphasize parental involvement and engagement in treatment, for children and adolescents with disruptive behavioral difficulties referred to outpatient psychiatric clinics in the state of Connecticut.

Theoretical and Empirical Support for Parent-Child focused Evidence-based Programs (EBPs)

Parent-child focused EBPs are manualized community-based treatment models that utilize training and support for parents to facilitate positive changes in functioning and behavior of their children (Kaminski et al., 2008; Hogue et al., 2015). Examples include Positive Parenting Program (Sanders, Markie-Dadds, & Turner, 2003); Multisystemic Therapy (Henggeler, 2011), Parent Management Training (Kazdin, 1997), Brief Strategic Family Therapy (Santisteban, Suarez-Morales, Robbins, & Szapocznik, 2006; Szapocznik, Hervis, & Schwartz, 2003), Parent-Child Interaction Therapy (Eyberg & Robinson, 1982; Schuhmann, Foote, Eyberg, Boggs, & Algina, 1998), and Parenting with Love and Limits (Smith, Sells, Rodman, & Reynolds, 2006). In these treatment programs, parents are also recipients of intervention services and are actively involved in treatment delivery. The role of parents as change agents in intervention and treatment programs for their children is well-supported by research (Henggeler, 2011; Sanders, Kirby, Tellegen, & Day, 2014; Szapocznik & Williams; 2000).

Social learning theories suggest that parents play a critical role in enabling positive behavioral changes in their children (Bandura, 1969; Burgess & Akers, 1966). For example, ample research shows that certain parenting styles are linked to several measures of antisocial (e.g., substance use, delinquency; Bogenschneider, Wu, Raffaelli, & Tsya, 1998; McMaster, & Wintre, 1996) and prosocial (e.g., academic achievement and performance; Fan & Chen, 2001; Simons-Morton & Chen, 2009) child behaviors. Generally, parents who are able to balance demands from their child and provide a caring and supportive response are able to have better relationships with them (Baumrind, 1991), which can help facilitate prosocial behaviors (Dorius, Bahr, Hoffman, & Harmon, 2004; Steinberg, Blatt-Eisengard, & Cauffman, 2006).

Thus, it is not surprising that EBPs that promote parental involvement and engagement in treatment have garnered support from child-serving agencies, including the child welfare and juvenile justice systems (Barth et al., 2005; Burke, Mulvey, Schubert, & Garbin, 2014; Walker, Bishop, Pullmann, & Bauer, 2015). In Connecticut, some of the most commonly prescribed parent-child focused EBPs include the Parent Management Training, Brief Strategic Family Therapy, Parent-Child Interaction Therapy, and Parenting with Love and Limits. Although these programs rely on different delivery formats and strategies, central to these programs is providing parents the support and skills necessary to promote positive behavioral changes in their children. For example, through positive parent-child interaction, improved communication skills, parenting consistency, and use of problem-solving approaches, parent-child focused EBPs have shown positive treatment effects in reducing disruptive and problem behavior among children and adolescents (Hogue et al., 2015; Kaminski et al., 2008; Thomas & Zimmer-Gembeck, 2007).

Challenges to Implementing Parent-Child Focused EBPs in Community Settings to Scale

Despite a growing support for the effectiveness of parent-child focused EBPs, there is limited research on these programs when delivered to scale. This is problematic because effectiveness studies help build credentials to identify empirically supported programs that can be transported and disseminated to diverse “real world” settings (Forgatch, Patterson, & Gewirtz, 2013). In particular, effectiveness studies in large-scale settings may help address the limitations that initial effectiveness studies might have overlooked or minimized, including “relevant variations in client populations, practitioners, models of service delivery, provider organization, and financing of services” (Schoenwald & Hoagwood, 2001, p. 1192). Effectiveness studies on the wide-scale implementation of EBPs can inform systematic modifications to programs and guide dissemination and sustainability efforts.

Transportability and dissemination research has identified several barriers to bridging this gap with respect to delivery of effective practices in community-based care settings. For example, statewide program dissemination involves expansion to a wider population of clients, often with varying levels of risks and needs (Schoenwald & Hoagwood, 2001). Agencies are often faced with the challenge of adapting services that would cater to a diverse population of clients. Constraints in funding and resources may impede successful implementation and dissemination plans statewide (Milat, Bauman, & Redman, 2015; Hanson, Cleary, Schneider, Tantivess, & Gilson, 2010). The lack of readiness among organizations to adopt and engage in active implementation of the program may also hamper large-scale dissemination efforts (Williams, 2015; Proctor et al., 2009).

Such challenges may be particularly salient in implementation and dissemination of parent-child focused EBPs. In addition to child-specific needs, families often have many needs that require additional supports and services. Parents’ mental health and substance use problems, exposure to domestic violence, and financial difficulties may pose significant barriers to effective implementation and dissemination practices (Forgatch et al., 2013). Thus, in addition to funding and resource limitations that may impede successful statewide dissemination efforts (Milat et al., 2015; Hanson et al, 2010) the capacity of service providers to engage parents in treatment is also a key factor to implement and sustain parent-child focused EBPs. Such complexities and challenges may also limit rigorous evaluation of treatment effects of parent-child focused EBPs in large-scale community settings.

The Current Study

This study extends current literature by examining the effectiveness of parent-child focused EBPs on problem severity and functioning of children and adolescents’ in outpatient treatment presenting with disruptive behaviors. Effectiveness of EBPs is often examined through randomized control trials (RCTs). Indeed, RCTs are considered the gold standard in measuring the efficacy and effectiveness of a program on a given outcome (Austin, 2011). However, certain conditions including financial and ethical limitations proscribe RCTs particularly in large-scale and statewide dissemination settings. Thus, we employed propensity score matching (PSM) to examine treatment effects in a naturalistic setting—across a statewide network of contracted outpatient psychiatric clinics for children (OPCC). PSM statistically adjusts for baseline differences in the treatment and the comparison groups, thereby reducing the effects of selection bias or confounding in an observational study (Hodges & Grunwald, 2005). Based on previous studies (Hogue et al., 2015; Kaminski et al., 2008), we hypothesized that children and adolescents requiring treatment for disruptive behavior who received parent-child focused EBPs (EBP treatment condition) would show reduced problem severity and improved functioning compared to a matched cohort of children and adolescents who received treatment as usual (TAU condition).

METHOD

Data Sources and Participants.

This study was conducted using administrative data submitted by OPCCs to the Programs and Services Data Collection and Reporting System (PSDCRS), a statewide web-based data collection system maintained by Connecticut Department of Children and Families (DCF), the State statutory authority for children’s behavioral health services. OPCCs are community-based children’s mental health facilities contracted by DCF to provide mental health services to youth under eighteen years of age and their families. Providers enter child, family, and case-related data into PSDCRS via a secure login. Data were provided for fiscal years 2012 and 2013 (i.e., July 1, 2011 – June 30, 2013). An index episode, defined as the first episode of service beginning in the evaluation period, was selected for each child (N = 20,841 for fiscal years 2012 and 2013). ‘Evaluation Only’ cases and those with only intake data were excluded from report analyses (19% of the total index episodes). Figure 1 depicts the process of sample identification for the present study.

Figure 1.

Sample selection criteria and final sample size (n = 2,763)

Treatment Condition: Parent-Child focused EBP.

Condition assignment was determined based on a case’s presenting problem (disruptive behavior) and referral to a parent-child focused EBP. Among the 14,788 non-evaluation cases, approximately 42% (n = 6,132) presented with a disruptive behavior problem. Of these cases, 4% (n = 220) were referred to parent-child focused EBPs, 55% (n = 3,369) were referred to non-parent-child EBPs, and 41% (n = 2,543) were referred to treatment-as-usual (TAU). Parent-child focused EBPs include Parent Management Training (n = 128), Brief Strategic Family Therapy (n = 76), Parent-Child Interaction Therapy (n = 37), and Parenting with Love and Limits (n = 6).¹ TAU includes individual and family-based treatments that are not classified as EBPs (e.g., social skills group, supportive therapy, parenting counseling). Cases that were referred to non-parent-child focused EBPs, including child-focused treatment such as cognitive behavioral therapy, aggression replacement training, motivational interviewing, and trauma-focused treatment such as trauma-focused cognitive behavioral therapy, were excluded from analysis. Thus, our final sample consisted of 2,763 index episodes with disruptive behavior as presenting problem; 8% met criteria for inclusion in the EBP treatment condition and the remaining 82% were included in the TAU comparison condition.

Measures

Dependent variables.

Changes in youth problem behavior and functioning were assessed using therapist ratings of the Ohio Scale (Ogles, Melendez, Davis, & Lunnen, 2001) at intake and discharge. The Ohio Scales is comprised of two subscales: problem severity and functioning. Problem severity is a 20-item scale including common behavioral concerns (e.g., getting into fights, fits of anger, using drugs or alcohol) reported by youth in behavioral health treatment. Items are rated on a 6-point scale (0 ‘no at all’ to 5 ‘all the time’). Total problem severity is calculated by summing ratings, with higher scores indicating more problematic behavior. Scores greater than 20 represent clinically elevated problem behavior. Functioning is a 20-item scale that represents level of ability across various domains of daily activity (e.g., interpersonal relationships, recreation, self-direction and motivation). Items are rated on a five-point scale (0 ‘extreme troubles’ to 4 ‘doing very well’). Total functioning is calculated by summing ratings, with higher scores indicating better functioning. Scores less than 50 for parent and therapist ratings indicate clinical levels of impaired functioning. The Ohio Scales have demonstrated satisfactory reliability coefficients across multiple reporting sources (Cronbach’s α range from .65 to .97; test-retest reliabilities range from .43 to .88; Ogles et al., 2000).

To assess clinical relevance of changes in total problem severity and functioning scale scores, researchers have also calculated a Reliable Change Index (RCI; Jacobson & Truax, 1991) score for the Ohio Scales. The RCI is a function of difference scores to the standard error of the difference score for individuals, with an index greater than 1.96 indicative of reliable change. Previous research has estimated a 10-point improvement in problem severity to be indicative of reliable change, and a 5-point improvement indicative of partial improvement (Ogles, Lunnen, & Bonesteel, 2001). Similarly, an 8-point improvement in functioning is indicative of reliable change, and a 4-point improvement indicative of partial improvement.

Propensity score covariates.

Model baseline covariates were extracted from PSDCRS administrative data records. These included basic demographic information such as child age, gender, and race/ethnicity. Intake and discharge dates were used to calculate length of treatment in days. Therapists indicated child clinical diagnoses using DSM-IV Diagnostic Codes, which were re-coded to represent the presence of a behavioral disorder (e.g., oppositional defiant disorder, conduct disorder), mood disorder (e.g., depression), anxiety disorder, posttraumatic stress disorder, or adjustment disorder. Youth history of exposure to six different potentially traumatic events (e.g., witnessing violence, physical or sexual victimization, disrupted attachments, etc.) was used to calculate a sum of traumatic events at intake. Therapists also indicated the presence of caregiver mental illness, substance use, domestic violence, or poverty-related clinical needs. In addition, therapist baseline ratings of Ohio Scales problem severity and functioning were used as model covariates for propensity score models.

Analysis Plan

Propensity Score Methods.

PSM was used to balance pre-treatment characteristics that may introduce selection bias across conditions (i.e., cases receiving parent-child focused EBP treatment and those receiving TAU) to facilitate comparisons of treatment effects (Stuart, 2010). First, covariates (e.g., demographic, case, and family characteristics, intake ratings of child behavior and functioning) that may be associated with selection into the treatment condition and the outcome variable are identified, and covariates are compared across treatment and comparison conditions (Wong & Schonlau, 2013). Significant differences between the treatment and comparison groups may indicate selection effects associated with indicated covariates, which may produce biased estimates of treatment effects. Propensity scores, or the probability of assignment to treatment based on the values of observed covariates (Rosenbaum & Rubin, 1983), are estimated to achieve balance among covariates. Propensity scores are then used to match individuals in the treatment and comparison groups that share approximately similar probability of being selected into the treatment condition.

We compared multiple matching algorithms and considered the following criteria in choosing the most appropriate PSM method (see Guo et al., 2006) – common support, covariate balancing, and median bias. Of the matching methods we considered (i.e., nearest 1–1 with and without replacement; nearest 3–1 with and without replacement; nearest 1–1 with replacement and caliper; nearest 3–1 with replacement and caliper; average treatment effect; and Kernel matching), the Kernel matching approach produced the fewest number of cases dropped within the common support guideline, successfully removed significant differences between the treatment and comparison groups, and produced the lowest median bias after matching. The Kernel matching approach is a nonparametric estimator that uses the weighted average of the outcome for the comparison group, which is then used to compare to the outcome of the treatment group (Caliendo & Kopeinig, 2008; Guo et al., 2006). Kernel matching can result in biased estimates of treatment effect if individual participants with propensity scores outside the area of common support are included; thus, analyses were restricted to those cases in the treatment and comparison groups with substantial overlap in distribution of propensity scores (Wong & Schonlau, 2013; Caliendo & Kopeinig, 2008). Treatment effects were estimated based on the Kernel matched sample with the common support restriction in place.

Difference-in-Difference.

A difference-in-difference (DiD; Heckman, Ichimura, Smith, & Todd, 1998; Lechner, 2010) analytic approach was used to compare rates of change in outcome variables across treatment and comparison conditions. This approach, drawn from the econometrics literature, contrasts change scores across conditions to control for the static effects of unobserved characteristics that similarly effect treatment and comparison conditions. The DiD approach is flexible and may be combined with propensity score methods to increase the robustness of causal effects estimation through increased control of these unobserved covariate effects (Smith & Todd, 2005; Villa, 2016).

RESULTS

Descriptive Characteristics of Treatment and Comparison Groups

Table 1 displays baseline child, family, and case characteristics of the treatment and comparison groups. The sample included children and youth ages 5 through 17. The average age of children and youth in the treatment group was 8.4 years (SD = 3.0), and the average age of children and youth in the comparison group was 9.9 years (SD = 3.6). A majority of the treatment group included White (42%) and Hispanic (42%) boys (74%); for the comparison group, a majority also included boys (68%), though a greater percentage were non-White (75%: Hispanic-49%, Black-18%, other-6%). Children and youth in the comparison group had been exposed to more types of trauma (M = .70, SD = .85) than those in the treatment group (M = .57, SD = .71). The treatment group had a longer length of stay in treatment (M = 189 days, SD = 175) compared to the comparison group (M = 175 days, SD = 158). The treatment and comparison groups showed comparable proportions (i.e., within one percentage point difference) of children and youth diagnosed with behavioral, anxiety, and adjustment disorders. However, a greater proportion of children and youth in the comparison group had diagnoses of PTSD (5%) and mood (16%) disorders compared to children and youth in the treatment group (2% and 9%, respectively). Problem severity and functioning scores were also comparable between the treatment and comparison groups with scores above the clinical range (i.e., above 20 for problem severity and below 50 for functioning) for both groups. Finally, parents’ needs including mental illness and substance use were also comparable between the treatment and comparison groups. However, the comparison group had a smaller proportion of parents who experienced domestic violence (8%), but a higher poverty rate (25%) than the treatment group (11% and 20%, respectively).

Table 1.

Descriptive Statistics for Total Sample and EBP Treatment and Comparison Groups

Demographic, Case, and Family Variables	EBP Treatment Group (n = 220; 8%)	Comparison Group (n = 2,543; 92%)	Total Sample (N = 2,763)
Age***	8.4 (SD = 3.0)	9.9 (SD = 3.6)	9.9 (SD = 3.6)
Gender
Male	162 (74%)	1740 (68%)	1902 (69%)
Female	58 (26%)	803 (32%)	861 (31%)
Race/Ethnicity
White (reference)	92 (42%)	632 (25%)	724 (26%)
Black**	22 (10%)	449 (18%)	471 (17%)
Hispanic	93 (42%)	1241 (49%)	1334 (48%)
Other	11 (5%)	153 (6%)	164 (6%)
Number of Types of Trauma*	.57 (SD = .71)	.70 (SD = .85)	.70 (SD = .84)
LOS in treatment	189 days (SD = 157)	175 days (SD = 158)	176 days (SD = 158)
Mental Health Diagnosis
Behavioral	87 (40%)	1027 (41%)	1114 (40%)
Anxiety	13 (6%)	116 (5%)	129 (5%)
PTSD	5 (2%)	116 (5%)	121 (4%)
Mood**	19 (9%)	412 (16%)	431 (16%)
Adjustment	60 (27%)	657 (26%)	717 (26%)
OHIO Scales at Intake
Problem Severity	26.3 (SD = 11.1)	27.4 (SD = 12.2)	27.4 (SD = 12.1)
Functioning	42.9 (SD = 11.8)	41.7 (SD = 11.2)	41.8 (SD = 11.3)
Parents’ Needs
Mental Illness	39 (18%)	489 (19%)	528 (19%)
Substance Use	11 (5%)	133 (5%)	144 (5%)
Domestic Violence	23 (11%)	211 (8%)	234 (9%)
Poverty	44 (20%)	625 (25%)	669 (24%)

^*p < 0.05;

^**p < 0.01;

^***p < 0.001

Covariate Balance before and after Propensity Score Matching

Kernel matching with common support restrictions was used to balance covariates across treatment and comparison conditions. This restriction resulted in 2 cases outside the common support region being excluded from the treatment group for outcome analyses. Table 2 shows the results from pre- and post-matching t-tests examining differences between the treatment and comparison groups on baseline covariates. After matching, no significant differences were found between the treatment and comparison groups on covariates, described above, indicating that PSM was successful in achieving covariate balance across conditions. Further, the median bias decreased from 10.7 to 2.4; a median bias below 5.0 is indicative of acceptable balancing of pre-treatment differences across condition.

Table 2.

Matching Results Comparing EBP Treatment and Control Groups on Selection Variables Pre- and Post- Adjustment

Covariate	EBP Treatment	Comparison Group
	Group	Unadjusted means		Adjusted means
		Mean	t value	Mean	t value
Age	8.4	9.9	−6.34***	8.8	−1.23
Female (Male)	.26	.32	−1.60	.28	−.37
Race/Ethnicity (White)
Black	.10	.18	−2.74**	.12	−.58
Hispanic	.43	.49	−1.66	.45	−.37
Other	.05	.06	−.56	.05	−.11
Number of Types of Trauma	.56	.70	−2.26*	.60	−.48
LOS in treatment (Days)	191	175	1.40	187	.23
Mental Health Diagnosis
Behavioral	.39	.40	−.36	.39	.02
Anxiety	.06	.05	.94	.06	.02
PTSD	.02	.05	−1.55	.03	−.33
Mood	.08	.16	−2.85**	.11	−.56
Adjustment	.28	.26	.60	.27	.08
OHIO Scales at Intake
Problem Severity	26.1	27.4	−1.52	26.4	−.26
Functioning	43.0	41.7	1.62	42.6	.38
Parents’ Needs (No)
Mental Illness	.18	.19	−.36	.19	−.07
Substance Use	.05	.05	−.07	.05	.03
Domestic Violence	.10	.08	1.19	.10	−.08
Poverty	.21	.25	−1.39	.22	−.24

Note. Reference categories are shown in the parentheses.

^*p < 0.05;

^**p < 0.01;

^***p < 0.001

Estimating Treatment Effects through Difference-in-Difference Models

Difference-in-difference scores for problem severity and functioning were estimated with the Kernel-matched sample to analyze treatment effects of parent-child focused EBPs. Adjusted and unadjusted models of difference-in-difference estimates are presented in Table 3.

Table 3.

Difference-in-Difference Models

Outcome	EBP Treatment Group	Comparison Group	Difference-in-Difference (SE)	T-test
Problem Severity
Unadjusted	−6.76	−4.33	−2.40 (.85)	2.87*
Adjusted	−6.85	−4.23	−2.62 (.91)	2.88*
Functioning
Unadjusted	3.98	3.05	0.93 (.85)	1.09
Adjusted	3.98	2.98	1.00 (.91)	1.09

^*p < 0.05;

^**p < 0.01;

^***p < 0.001

First, we examined the effect of parent-child focused EBP on problem severity. The difference-in-difference estimator indicated that both groups showed a modest decrease in problem severity at discharge. However, the treatment group showed a statistically significant greater reduction in problem severity than the comparison group. This difference remained significant even after the groups were matched on covariates (M_Diff-in-Diff = 2.62, t = 2.88, p < .05).

The results also indicated that the difference scores for both treatment and comparison groups were below the reliable threshold for clinical significance (i.e., Reliable Change Index or RCI is below 10 points; Tam & Healy, 2006; Jacobson & Truax, 1991). However, the treatment group showed partial improvement with a positive change from intake to discharge (M = 6.85) that was greater than half the magnitude of the reliable change threshold and a movement from the clinical (above 20) to the non-clinical (0–20) range (i.e, M_intake = 26.11 to M_discharge = 19.26). On the other hand, the comparison group change scores indicated a minimal level of change, less than half of that required by the RCI (M = 4.23). The results also showed that while there was an improvement in problem severity score among children in the comparison group, they remained within the clinical range (i.e., M_intake = 26.40 and M_discharge = 22.17).

Next, we examined the effect of parent-child focused EBP on functioning. The difference-in-difference model indicated that the level of improvement in functioning did not differ statistically by condition (M_Diff-in-Diff = 1.00, t = 1.09, p > .05). Further, the level of improvement for both conditions was below the RCI, and both groups remained in the clinical range (below 50) for impaired functioning (EBP Treatment Group: M_intake = 43.03 and M_discharge = 47.01; TAU Group: M_intake = 42.61 to M_discharge = 45.45).

DISCUSSION

In this study, we examined the treatment effects of manualized parent-child focused EBPs on functioning and problem severity among children and adolescents referred to outpatient psychiatric clinic for serious and persistent disruptive behavior. There are three main findings with implications for treatment and delivery of services. First, children and adolescents who received parent-child focused EBPs showed significantly greater reduction in problem severity compared to the comparison group, indicative of an RCI in the partial improvement range and dropping below the clinical threshold for problem behavior. Second, the pattern of improvement in problem severity was not paralleled by differential improvement in ratings of youth functioning across treatment and comparison conditions. Finally, there were significant differences between the treatment and comparison groups pertaining to certain child and case characteristics. The comparison group had a greater proportion of children and adolescents who were older, Black, experienced more types of trauma, and were diagnosed with mood disorders.

Treatment Effects on Problem Severity and Functioning

Our results support the effectiveness of manualized parent-child focused EBPs in treating disruptive behavior among children and adolescents in public outpatient settings. Although both treatment and comparison groups showed a decrease in problem severity at discharge, the treatment group had a statistically significant reduction in problem severity compared to the matched comparison group following application of propensity score methods. The adjusted mean difference in problem severity score at discharge between the treatment and comparison group translates to an effect size of d = .20, which is consistent with the effect sizes derived from a meta-analysis of manualized family-based treatments for adolescents with conduct and substance use problems. A meta-analysis by Baldwin, Christian, Berkeljon, Shadish and Bean (2012) examined results of 24 studies comparing BSFT, Functional Family Therapy, Multidimensional Family, and Multisystemic Family Therapy to either treatment as usual or an alternative therapy for adolescents with delinquency or substance use problems and reported statistically significant modest effects (d = .21 to .26).

In terms of clinical significance, our findings suggested partial improvement for the treatment group. This was reflected by the positive improvement in problem severity from intake to discharge as measured by half or more (6.8 points lower) of the magnitude of the reliable change threshold and the movement from clinical to non-clinical range (from 26.1 to 19.3) in problem severity score. Although the comparison group also showed a reduction in problem severity at discharge, the level of change was not consistent with a reliable or clinically significant level of change, and the comparison group’s problem severity score was still within the clinical range (from 26.4 to 22.2). These findings add further support to the utility of implementing parent-child focused EBPs to scale in curtailing problem severity among children and adolescents with disruptive behavior (Hogue et al., 2015).

Problem severity as measured by the Ohio scales reflects various persistent behavioral concerns (e.g., fights, fits of anger, drug and alcohol use, skipping school) that may warrant intervention from various service systems. Considering the potential serious consequences of disruptive behavior, including placement in secure confinement facilities (e.g., congregate care, detention or correctional facilities), parent-child focused EBPs may serve as an effective remedy that is less restrictive and more conducive to healthy development of children and adolescents (Karam, Sterrett, & Kiaer, 2017; Steinberg, Chung, & Little, 2004). For example, parent-child focused EBPs are typically prescribed in community-based settings, which afford children, adolescents, and families to develop and maintain stronger ties. This also allows simultaneous intervention between the children, adolescents, and families that facilitates better parent-child relationships, communication, and parenting practices (Kaminski et al., 2008; Henggeler et al., 2002).

On the other hand, the results for functioning were neither statistically nor clinically significant for either the treatment or comparison groups. Although both groups showed some improvement in functioning at discharge, these scores failed to reach the RCI threshold for reliable change and both groups remained within the clinical range (47.0 and 45.6, respectively for treatment and comparison groups), indicating a lack of clinically significant improvement, as well.

One possible explanation for the discrepancy in outcome patterns may be that problem severity and functioning represent distinct constructs that may require different intervention strategies. Problem severity is comprised of items reflecting common behaviors associated with referral to mental health services, which may be more consistent with the types of behaviors addressed by parent-child focused EBPs. Many such evidence-based prevention and intervention models, for example, emphasize development of executive functioning processes (e.g., behavioral regulation skills such as inhibition, problem-solving, and goal-directed behavior; Greenberg, 2006; MacPhee, Lunkenheimer, & Riggs, 2015). In a meta-analytic review of parent training program effectiveness, Kaminski and colleagues (2008) identified program elements that are helpful for improving behavior and adjustment in children. Their findings suggest that programs that promote parenting consistency and positive parent-child interaction and communication skills consistently yield larger treatment effects compared to programs without those components. There is strong empirical support that such program components may act as safeguards against disruptive and antisocial behaviors through building positive parent-child relationships (Dishion & McMahon, 1998). Conversely, the functioning scale reflects the child’s ability to function across various domains of daily activity including interpersonal relationships, recreation, and motivation. Functional capacity may reflect a more general set of “presenting problems” for children and adolescents with disruptive behavior and may require a different set of individual and parental supports, and thus, may be less amenable to the parent-child focused EBPs included in the present study. This discrepancy highlights the importance of incorporating approaches that target a broader array of intervention strategies to promote adaptability, flexibility, and other relational skills (MacPhee et al., 2015).

Differences between the Treatment and Comparison Groups

Also of note, results of the covariate comparisons prior to matching highlight the characteristics of children and adolescents in a statewide system who are more likely to access parent-child focused EBP treatments, compared to traditional treatment models (i.e., the TAU condition). Compared to those who received parent-child focused EBP treatments, children and adolescents who received TAU services were more likely to be older, Black, experienced more types of trauma, and have higher rates of diagnosed mood disorders. Although we were not able to examine the decision-making context by which children and adolescents in the comparison group received treatment as usual as opposed to a parent-child focused EBP, our findings suggest potential challenges with access to and delivery of services, particularly for underrepresented groups of children and families.

Disparities in service utilization among underrepresented group of children and families is well-documented. Unmet mental and behavioral health needs are particularly striking among racial and ethnic minority groups. Studies have demonstrated that compared to White children and youth, Black and Latino children are less likely to use outpatient services, including specialty mental health and substance abuse services even after accounting for important risk factors (Garland & Besinger, 1997; Garcia, Aisenberg, & Harachi, 2012). Lack of understanding about rules and procedures for obtaining services, financial limitations, and refusal from family members to obtain services may contribute to disparities in service utilization (Cummings, Wen, & Druss, 2013). Given that these barriers may place an already vulnerable child at greater risk for poor health outcomes, there is a critical need to examine the mechanisms by which these disparities operate. Such studies have important implications for implementation and dissemination of parent-child focused EBPs, particularly in terms of the ability to reach high-risk populations of children and families.

Limitations and Future Research Directions

There are several limitations to keep in mind when interpreting these findings. First, due to limitations of using administrative datasets, we were not able to adjust for other potentially relevant child and family characteristics that could influence selection into treatment and the outcomes. Research has shown, for example, that factors such as parenting practices and efficacy (Sanders et al., 2014), treatment fidelity, (Brunk et al., 2014; Henggeler et al., 2002), and dosage are important mechanisms of change in an EBP. Future studies must take these factors into account to better understand the mechanisms by which parent-child focused EBP promote positive outcomes, particularly in large-scale dissemination efforts. Second, this study included five parent-child focused EBPs—52% of children and youth received Parent Management Training, 32% received Brief Strategic Family Therapy, 15% received Parent-Child Interaction Therapy, and 2% received Parenting with Love and Limits. Although providing parents with the support and skills critical to promoting positive behavioral changes in their children are integral to these programs, variations in service delivery formats and strategies may exist. Future research should consider how such variations relate to youth and family outcomes. Third, this study focused on dissemination of parent-child focused EBPs in a single state. Implementation of parent-child focused EBP is typically tailored and adapted specific to the needs and capacities of a state or an agency, which may affect the generalizability of findings to other state contexts or service populations. This limitation, nonetheless, is tempered by the consistency between our findings and those observed in other published studies (e.g., Hogue et al., 2015; Kaminski et al., 2008).

Finally, although PSM offers a promising approach to assess treatment effects in observational data, this method is not without limitations. The capacity of PSM to reduce selection bias and adjust for bias in baseline differences between the treatment and comparison groups is highly dependent on selection of variables for matching (West et al., 2014; Wong & Schonlau, 2013; Guo et al, 2006). Although relevant variables were incorporated into our estimation of propensity scores, we cannot adjust for unobserved predictors of treatment participation and study outcomes (i.e., problem severity and functioning) that were not available in the administrative data systems. Though a difference-in-difference approach strengthens the robustness of estimates (Smith & Todd, 2005; Villa, 2016), to the extent that these unobserved variables may be related to selection into treatment and outcomes measures, PSM methods cannot account such influences on unbiased estimates of treatment effects. Future studies should expand on the types of pre-treatment covariates included in outcome comparisons using PSM.

Despite these limitations, we believe the current analyses demonstrate the benefits of using PSM to assess treatment effects resulting from large-scale or system-wide dissemination efforts—providing a framework to examine program effects in a naturalistic observational design. Although RCTs represent the gold standard with respect to rigor in estimating treatment effects, such designs limit external generalizability that may pose problems for large-scale dissemination trials (Sanson-Fisher, Bonevski, Green, & D’Este, 2007; Davidson et al., 2004).

Implications and Conclusion

In recent years, child-serving agencies have shifted their response to child problem behaviors from a focus on institutionalization (e.g., confinement of children to child welfare, juvenile justice, or mental health facilities) to engaging parents in community-based treatments (Gavazzi, Yarchek, Rhine, & Partridge, 2003; Pennell, Shapiro, & Spigner, 2011). Yet few studies have examined the effectiveness of parent-child focused EBPs when implemented in large-scale community settings. This study adds to the literature supporting the effectiveness of EBPs that emphasize parental involvement and engagement in reducing levels of problem behavior among children and adolescents with disruptive behavior. Children and adolescents who received a parent-child focused EBP showed significant reductions in problem severity compared to those who received treatment as usual, with the levels of improvement resulting in discharge problem severity below the level of clinical severity. Although there were improvements in functioning in both the treatment and comparison, these changes did not reach statistical or clinical significance, suggesting that problem behaviors may be more amenable to parent-child focused EBPs than functioning-related problems. Our findings also identified variations in populations of children and youth who received parent-child focused EBPs and those who received TAU. These findings provide opportunities to inform and improve implementation efforts, better tailor services for, and effect positive outcomes among children and adolescents with serious behavioral problems. Overall, these findings support the potential of taking parent-child focused EBPs into scale to promote positive behavioral changes among children and adolescents.