Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Aug 25.
Published in final edited form as: J Emot Behav Disord. 2014 May 1;23(2):67–77. doi: 10.1177/1063426614532690

Multiple Family Group Service Model for Children With Disruptive Behavior Disorders: Child Outcomes at Post-Treatment

Anil Chacko 1,2,3, Geetha Gopalan 3,4, Lydia Franco 2,3, Kara Dean-Assael 2,3, Jerrold Jackson 5, Sue Marcus 6, Kimberly Hoagwood 3, Mary McKay 2,3
PMCID: PMC4548959  NIHMSID: NIHMS679525  PMID: 26316681

Abstract

The purpose of this study was to determine the benefits of a multiple family group (MFG) service delivery model compared with services as usual (SAU) in improving the functioning of youth with oppositional defiant/conduct disorder in families residing in socioeconomically disadvantaged communities. Participants included 320 youth aged 7 to 11 and their families who were referred to participating outpatient clinics. Participants were assigned to the MFG or the SAU condition, with parent report of child oppositional behavior, social competence, and level of youth impairment as primary outcomes at post-treatment. Family engagement to MFG was measured by attendance to each group session. Caregivers of youth in the MFG service delivery model condition reported significant improvement in youth oppositional behavior and social competence compared with youth in the SAU condition. Impairment improved over time for both groups with no difference between treatment conditions. The MFG led to greater percentage of youth with clinically significant improvements in oppositional behavior. Attendance to the MFG was high, given the high-risk nature of the study population. The MFG service delivery model offers an efficient and engaging format to implement evidence-based approaches to improving functioning of youth with oppositional defiant and/or conduct disorder in families from socioeconomically disadvantaged communities.

Keywords: effectiveness trials, service delivery, child disruptive behavior disorders, socioeconomically disadvantaged communities, inner-city communities

Publicly funded outpatient mental health settings are one of the main contexts where youth with disruptive behavior disorders (DBDs; that is, oppositional defiant disorder [ODD] and conduct disorder [CD]) receive mental health services in socioeconomically disadvantaged communities. Although evidence-based treatments have been developed for treating DBDs (Eyberg, Nelson, & Boggs, 2008), there has been notable challenges in implementing these interventions in publicly funded outpatient mental health settings. This science-to-service or research-to-practice gap has long been recognized as a significant issue—closing the gap between youth in need of mental health care and receipt of the most appropriate, empirically informed care has been identified by the United States Surgeon General as one of the most significant public health issues for the nation to address (U.S. Public Health Service, 2000).

To date, several effectiveness trials have been conducted to determine whether evidence-based treatments, evaluated most often through well-controlled efficacy trials, would have similar benefits if conducted in more routine mental health service settings. For instance, several effectiveness studies have utilized variations of behavioral parent training (BPT) approaches for the treatment of DBDs (e.g., Incredible Years: Axberg & Broberg, 2012; Behan, Fitzpatrick, Sharry, Carr, & Waldron, 2004; Gardner, Burton, & Klimes, 2006; Hutchings et al., 2007; Kleve et al., 2011; Lau, Fung, Ho, Liu, & Gudino, 2011; Parent– Child Interaction Training [PCIT]: Phillips, Morgan, Cawthrone, & Barnett, 2008; Parent Management Training; Costin & Chambers, 2007; Kling, Forster, Sundell, & Melin, 2010; Ogden & Hagen, 2008; Triple P: Gallart & Matthey, 2005). These effectiveness studies have found that various BPT approaches result in statistically and clinically significant improvements in disruptive behavior in youth. The data suggest that there are various parameters in which BPT approaches can be administered, including various formats (e.g., parent-group only; parent–child dyads; parent– clinician individual formats), durations (e.g., 8 weeks; [e.g., Costin & Chambers, 2007] to 14 weeks [e.g., Lau et al., 2011]), and for preschool (e.g., Axberg & Broberg, 2012) through school-age (e.g., Costin & Chambers, 2007) youth. Fortunately, these interventions are also well-manualized and have established training methods that allow for widespread dissemination.

Most recently, Weisz and colleagues (2012) developed a novel approach to treat commonly observed comorbid childhood mental health disorders (e.g., depression, anxiety, DBDs). This approach utilized flexibly delivered, integrated, but free-standing modular treatment components based on the common elements of evidence-based treatments for these various disorders. Data from a randomized clinical effectiveness trial demonstrated that the modular intervention resulted in significantly greater improvements in DBDs as compared with a usual-care condition and a standard evidence-based treatment condition.

At first glance, it appears that there are numerous, effective interventions for the treatment of DBDs in youth that are readily available for widespread dissemination. However, a closer look at the effectiveness literature reveals that the manner in which interventions have been evaluated has made application of study findings to practice settings challenging, particularly for outpatient mental health settings serving families from low socioeconomic backgrounds (Hoagwood, Burns, & Weisz, 2002; Weisz, Ugueto, Cheron, & Herren, 2013). For instance, several recent reviews have suggested that very few treatment studies targeting DBDs in youth were conducted with clinic-referred youth, being treated in routine community-based clinical settings by practicing clinicians in those settings (Michelson, Davenport, Dretzke, Barlow, & Day, 2013; Weisz et al., 2013). As such, there remain questions as to whether findings from effectiveness studies can be readily translated into routine service settings, particularly those with limited resources serving high-risk youth and their families, such as publicly funded outpatient mental health clinics.

In addition, although not often considered, but equally important, is the complexity of the intervention and resultant training and supervision needs, which may directly affect whether an effective intervention can be readily adopted within the constraints of routine clinical practice (Weisz et al., 2013). For example, certain interventions that have demonstrated effectiveness in routine practice settings require 1 week of full-day trainings or more (e.g., Gardner et al., 2006; Weisz et al., 2012). Moreover, given that some BPT approaches (e.g., PCIT) and modular treatment approaches utilize an individual treatment format, it is likely necessary to provide supervision for most if not all treatment cases. The substantial (and often unreimbursed) time commitment required from therapists and supervisors in routine outpatient mental health settings to fully participate in training and ongoing supervision to effectively implement these interventions may be a significant obstacle, particularly in resource-poor outpatient mental health settings. Clearly, both training and supervision requirements for adoption of novel practices are important considerations when determining whether an effective intervention can be adopted.

In response to these sets of issues, there has been a call for a shift in the manner in which interventions are developed and evaluated (Hoagwood et al., 2002; Weisz et al., 2013), with a recommendation that any new service model should begin to be developed and evaluated in the settings in which these service delivery models are ultimately to be deployed. Moreover, the process of developing these service delivery models should attend to the characteristics of youth and families and the service delivery process that often mitigate effectiveness and engagement and impede on adoption of new practices. Collectively, engaging, effective, efficient, and adoptable interventions are necessary to bridge the science-to-service or research-to-practice gap in the treatment of DBDs in youth. This requires thoughtful context-focused development of the intervention with an eye toward parsimonious training and supervision methods and testing within the constraints of the practice setting.

In line with these recommendations, McKay, Gonzales, Stone, Ryland, and Kohner (1996) developed a multiple family group (MFG) service delivery model for youth with DBDs who present to outpatient mental health clinics in socioeconomically disadvantaged communities. The MFG service delivery model was developed in collaboration with families of youth with DBDs and mental health providers from community-based outpatient clinics with the goal of developing an intervention model that can meet the needs of families from complex backgrounds as well as fit within the clinical, financial, and regulatory constraints of routine outpatient mental health settings in socioeconomically disadvantaged communities.

As a foundation, MFG takes a common elements approach (Chorpita & Daleiden, 2009; Garland, Hawley, Brookman-Frazee, & Hurlburt, 2008) by utilizing and integrating core components from the empirical literature regarding effective treatment practices for DBDs. Moreover, MFG incorporates evidence-based engagement techniques demonstrated to improve retention of families from socioeconomically disadvantaged communities in mental health services (Mckay & Bannon, 2004). These empirical literatures are translated into core skills, processes, and methods that are all framed in a strength-based perspective and are delivered in a flexible manner through a partnership model with a family partner/ advocate and clinician. In addition, MFG relies on multiple generations within a family working collaboratively with other families in a group setting as a way of decreasing stigma associated with receipt of mental health services. Lastly, MFG was developed to be a parsimonious model, allowing for greater ease in training and supervising clinical staff. Collectively, MFG was developed to increase engagement in services, provide an efficient and effective service delivery mechanism, and be readily implemented and adopted within routine publicly funded outpatient mental health settings in socioeconomically disadvantaged communities (see the “Method” section for greater details).

The MFG service delivery model is a promising approach to delivering mental health services in routine outpatient mental health care in socioeconomically disadvantaged communities. In a series of studies conducted in socioeconomically disadvantaged community mental health clinics, youth participating in the MFG service delivery model, as compared with those participating in a services as usual (SAU), evinced significantly greater improvements in conduct problems, hyperactivity, impulsivity, and learning problems, as well as greater retention in mental health services (McKay, Gonzales, Quintana, Kim, & Abdul-Adil, 1999; McKay et al., 1996; McKay, Harrison, Gonzalez, & Quintana, 2002; Stone, McKay, & Stoops, 1996). However, the data supporting the efficacy of the MFG service delivery model are based on studies with relatively small samples and implementation of MFG in a limited number of settings. The purpose of this article is to expand the evidence base of the MFG service delivery model by reporting on the results of a large-scale, clinical effectiveness trial of the MFG service delivery model for the treatment of DBDs in school-age youth. Specifically, 320 youth and their families who presented to 1 of 13 outpatient mental health clinics serving youth and their families from socioeconomically disadvantaged communities were assigned to either participate in the MFG service delivery model or to a SAU comparison group. The report herein describes the incremental benefits of participation in the MFG model beyond participation in the SAU comparison group on youth DBD symptoms, social competence, and impairment at immediate post-treatment (i.e., end of MFG group). Moreover, data are presented on the level of engagement to MFG groups.

Method

Participants

Recruitment for the study occurred between October 2006 and October 2010 across 13 community-based outpatient mental health clinics that serve youth and their families residing in socioeconomically disadvantaged communities across the New York City metropolitan area. To be included in the study, youth were (a) to be between the ages of 7 and 11 years and presented for treatment at a participating mental health clinic, and (b) required to have a diagnosis of ODD or CD according to the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR; American Psychiatric Association [APA], 2000) through intake with research staff. A total of 320 youth were enrolled in this study (see Table 1 for demographic information of sample). Written informed consent was obtained from the legal guardian, and verbal assent was obtained from youth. The study was approved by the Institutional Review Board.

Table 1.

Demographic Characteristics of Participants.

MFG (n = 225)
 SAU (n = 95)
Characteristic n % n %
Child age (M ± SD) 209 8.88 ± 1.45 93 8.85 ± 1.34
Caregiver age (M ± SD) 217 35.72 ± 8.39 93 38.18 ± 9.30
Child gender
  Male 148 66 69 73
Child ethnicity
  White/Caucasian 16 7.11 9 9.47
  Black/African American 66 29.33 29 30.53
  Hispanic/Latino 112 49.78 44 46.32
  Native American 3 1.33 5 5.26
  Asian/Pacific Islander 0 0.00 1 1.05
  Other 15 6.67 6 6.32
Family income
  Less than $9,999 91 40.44 34 35.79
  $10,000–$19,999 55 24.44 26 27.37
  $20,000–$29,999 32 14.22 15 15.79
  $30,000–$39,999 15 6.67 8 8.42
  $40,000–$49.999 3 1.33 1 1.05
  More than $50,000 14 6.22 4 4.21
Caregiver marital status
  Single 86 38.22 51 53.68
  Married or cohabitating 81 36.00 24 25.26
  Divorced 7 3.11 7 7.37
  Separated 34 15.11 10 10.53
  Widowed 4 1.78 3 3.16
  Other 4 1.78 0 0.00
Receipt of publicly funded health insurance 150 72 73 77
Open in a new tab

Data are not available for entire sample of 320 participants and their families. MFG = multiple family group; SAU = services as usual.

Procedures

Clinic staff informed youth and adult caregivers at intake about the study. Research staff ascertained DSM-IV diagnosis of ODD and/or CD symptoms through completion of DSM-IV ODD/CD symptom checklist (Pelham, Gnagy, Greenslade, & Milich, 1992) and associated functional impairment through the Children’s Impairment Rating Scale (Fabiano et al., 2006). Participants meeting inclusion criteria were allocated to either receive MFG (n = 225) or clinic SAU (n = 95) based on enrollment at the participating clinic. Specifically, the first six to eight youth screened eligible to participate in this study were assigned to receive MFG, and the next three to four youth who met eligibility requirements were assigned to receive SAU. This allocation was done to ensure sufficient numbers of families for a timely start of the MFG group at the participating clinic (see Goodwin et al., 2001 for similar randomization procedure). Allocation was conducted by research staff who were blind to the youth/family profile. Assessment measures were administered at five time points for each youth and their adult caregivers. This article focuses on the first three time points (baseline, mid treatment at 2 months, and post-treatment at 4 months) to determine immediate benefits of the MFG service delivery model.

MFG

MFG is a 16-week service delivery model that was guided by a manualized protocol. Each group met weekly for approximately a 90- to 120-min/session and included six to eight families, composed of identified youth, their adult caregiver(s), and sibling(s) between the ages of 6 and 18. As a foundation, MFG takes a common elements approach by identifying essential components from the empirical literature from BPT methods (e.g., Chorpita & Daleiden, 2009; Garland et al., 2008) and family therapy (e.g., Alexander, Pugh, Parsons, & Sexton, 2000; Keiley, 2002) regarding core effective practices for treating DBDs, represented as the “4Rs” (i.e., Rules, Responsibility, Relationships, Respectful Communication) and factors related to family engagement in mental health services, represented as “2Ss” (Stress and Social Support). Core components of BPT included in MFG were positive reinforcement (i.e., labeled praise, positive attending, tangible reinforcement/ rewards), which was incorporated into sessions focused on “relationships”; limit setting (i.e., monitoring, effective commands, response-cost; behavioral contracting/goal setting), which was mainly incorporated into sessions focused on “rules” and “responsibility”; psychoeducation and affect education (i.e., learning about, identifying, and labeling stress-related emotions and behavior; developing methods to address common triggers for stress), which was incorporated into sessions focused on “stress.”

Core components of family therapy included in MFG were role identification (i.e., understanding the unique and integrated role each member plays in a family and supporting how family members can support each other in achieving desired family outcomes), which was incorporated into sessions focused on “relationships”; reframing (i.e., developing new strategies to regulate emotions and interactions between family members), which was incorporated into sessions focused on “relationships” and “respectful communication”; communication training (i.e., identifying behaviors [e.g., eye contact] that demonstrate engaging in a conversation, using “I” statements to express needs/wants, utilizing congruent affect and speech when communicating, etc.), which was incorporated into sessions focused on “relationships” and “respectful communication.” Methods to improve within-family and external sources of emotional, tangible, informational, and companionship social supports (e.g., Chacko et al., 2009) were incorporated into the session focused on “social support.” Lastly, given the high-risk nature of the population for poor engagement to treatment, core aspects of evidence-based engagement practices (e.g., aligning expectations for treatment with anticipated treatment benefits, reducing stigma related to receipt of mental health services, etc.; McKay & Bannon, 2004) were also integrated into the MFG program.

Core MFG sessions focused on one of the 4Rs and 2Ss and proceeded with the following processes: (a) creating social networks, (b) information exchange/homework review, (c) group discussion regarding the skill, (d) individual family practice, and (e) homework assignment. MFG content areas (4Rs and 2Ss) were integrated into the program during the first (Sessions 1–8) and the second (Sessions 9–16) halves of MFG to provide opportunities for repeated exposure and practice with content.

Participants in the MFG condition were not prohibited from utilizing any additional services available to them through the outpatient mental health clinic where they were receiving MFG. In this sample, 53% of the participants in the MFG condition did not receive additional interventions. Those 47% of youth who did receive additional services also received outpatient individual services (49%), outpatient medication management (34%), school-based mental health (9%), case management (<1%), and crisis management services (<1%) during the course of the 4-month MFG group. Moreover, for those youth who received additional services beyond receipt of MFG, the majority of youth received one (53%), two (40%), or three (7%) additional services. No youth received more than three additional mental health services.

MFG participating clinics, MFG facilitators, and MFG facilitator training

Recruitment for the study was conducted during two established network meetings of outpatient mental health clinics in the New York City area and meetings with outpatient mental health clinics that had existing relationships with the research investigators. For agencies that were interested in participating in the research study, the research team subsequently met with the clinic administration to discuss the study and requirements for participation. Once a clinic agreed to participate, the research team met with clinical staff at scheduled staff meetings to discuss the study and address any questions/concerns.

MFG groups were delivered by mental health providers and family/peer advocates. Mental health providers were all existing clinical staff at the participating outpatient mental health clinics. Family peer advocates are parents who had previously received services for their children in the mental health system and are employed to help guide and support other parents receiving services for their child within the mental health system. Based on this personal experience, family peer advocates are hypothesized to be able to help families negotiate the mental health system, reduce barriers, and engage the family in services (Hoagwood et al., 2010).

Training for MFG facilitators was conducted by research staff in approximately 5 to 6 hr interspersed across 1 to 2 days (depending on facilitator availability) at the participating clinic site. Training covered the MFG model’s core competencies, group facilitation skills, and addressing engagement challenges. This content was taught through a mix of didactic lecture, group discussion, and role-plays.

SAUs

The SAU condition included all available clinic-based mental health services offered by participating outpatient mental health clinic sites. Typically, this included case management, individual therapy, family therapy, group therapy, and/or medication management. SAUs were provided by the full array of providers at the participating outpatient mental health clinics (i.e., social workers, psychologists, medical doctors, caseworkers, etc.). In this sample, participants in the SAU condition received outpatient individual services (38%), outpatient medication management (33%), outpatient group-based child-focused treatment (10%), outpatient family-based treatment (10%), case management (4%), crisis management services (1%), or inpatient hospitalization (1%) during the course of the 4-month study. Moreover, the majority of youth received one (28%), two (46%), or three (20%) of these services with a minority of youth (6%) receiving more than four services during the course of the study.

Measures

Attendance

Each family’s attendance at each MFG session was coded as present/absent. Percentage of sessions in which families attended MFG was utilized as a measure of engagement to MFG groups.

Iowa Connors Rating Scale–Oppositional/Defiant Subscale (IOWA CRS OD)

The IOWA CRS OD subscale (Waschbusch & Willoughby, 2008) is a widely used brief measure of oppositional defiant behavior in children completed by parents. The IOWA CRS consists of items evaluated using a 4-point Likert-type scale with the following anchors: not at all (0), just a little (1), pretty much (2), and very much (3). For the current study, we utilized the IOWA CRS OD subscale reported at baseline, mid-test, and post-test. Cronbach’s alphas for the baseline, mid-test, and post-test were .80, .83, and .86, respectively.

Social Skills Rating System–Social Skills Subscale (SSRS-SSS)

The SSRS-SSS, of the SSRS (Gresham & Elliott, 1990), focuses on the frequency of occurrence of the social skills of cooperation, assertion, responsibility, empathy, and self-control. Caregivers rate the frequency and importance of specific behavior along a 3-point Likert-type rating, from 0 (never) to 2 (often). Higher frequency scores indicate more frequent use of prosocial social skills. Cronbach’s alphas for the baseline and post-test SSRS-SSS were .88 and .91, respectively.

Impairment Rating Scale (IRS)

The IRS (Fabiano et al., 2006) is a six-item rating scale that asks caregivers to rate the severity of the child’s problems and need for treatment and/or special services in important functional domains (i.e., relationship with peers, relationship with the parent[s], relationship with sibling[s], academic progress, family functioning, self-esteem, overall need). Caregivers place an “X” on a 7-point visual analogue scale to signify their child’s functioning along a continuum of impairment that ranges from 0 (Not a problem at all. Definitely does not need treatment or special services) to 6 (Extreme problem. Definitely needs treatment and special services). IRS scores across each domain were used in the analyses.

Treatment fidelity and supervision

Over the course of the MFG implementation, fidelity monitoring tools were used to assess facilitator adherence to curriculum content (e.g., session topic, information conveyed, competence in guiding discussions) and process of facilitation (e.g., supports for networking across families, information exchange, application of content to specific family need, practice, and follow-up outside the group). These fidelity assessments were standardized and used by research staff during fidelity observations of MFG groups across each participating outpatient mental health clinic study site. In addition, MFG supervisors held weekly group supervision for approximately 60 min per week with each facilitator team leading a MFG group to discuss content and process of the groups, challenges in implementation, and clinical issues that arose during the group session.

Data Analysis

Mixed effects regression was utilized for each of the child outcomes over time using the SuperMix software (Hedeker, Gibbons, du Toit, & Cheng, 2008). Also known as multilevel linear modeling, this type of modeling allows parameters (intercepts and slopes) for measurements over time within cases to vary between cases. The correlation between measurements within cases is also accounted for. Finally, this type of modeling allows for different times and numbers of measurements within cases, which is an appropriate method to model longitudinal change involving data where there is attrition over time with the assumption that the missing data are ignorable (i.e., at least missing at random). As a result, this type of modeling allows for different times and numbers of measurements within cases. Each model included a dichotomous variable for treatment condition (MFG vs. SAU), and time was modeled using dummy variables identifying each assessment period (baseline as the reference category). For analyses involving the IOWA CRS OD subscale, dummy variables for mid treatment and post-treatment were included, while only dummy variables for post-treatment were included in analyses involving the SSRS-SSS and IRS items. Finally, all analyses included the condition by assessment period dummy variable interactions (e.g., Condition × Mid treatment). Furthermore, individual participants were used in a second level function, and because n = 22 families had more than one child enrolled in the study, the third level of analysis involved family-level identification variables. Intercepts were allowed to vary randomly within each model. Linear contrasts were subsequently used to test for significant differences between the MFG and SAU conditions on each outcome variable at post-treatment, as well as within-group changes in outcomes over time from baseline to post-treatment. Finally, as reported in other family-based treatment studies of disruptive behavior disorders (e.g., Chacko et al., 2009), the clinical significance of treatment effects was analyzed to determine whether a child moved out of the clinical range on the IOWA CRS OD subscale based on available norms.

Results

Treatment Fidelity

Research staff directly observed approximately four sessions in 15 out of the total 35 MFG groups conducted (43%) across 13 sites, resulting in 8% of all MFG sessions evaluated for fidelity. Across all sessions, 94% of content and procedures were implemented by the MFG facilitation team (both facilitators) at a high-level, per-treatment integrity and fidelity assessment tools.

Attendance

Attendance to the 16 MFG sessions ranged from 46% to 77% with an average attendance rate of 59% (SD = 7.55%). In addition, approximately 66% of families participating in the MFG condition participated in at least half of the MFG sessions.

Child Outcomes

Table 2 presents pre- and post-treatment means and SDs for all outcomes. Table 3 presents linear contrasts for testing for differences in outcomes by treatment condition at post-treatment. Significant differences were found between the MFG and SAU conditions on the IOWA CRS OD subscale at post-treatment (b = −1.20, SE = 0.48, Z = 2.48, p = .01), where the experimental group exhibited improvements in oppositional and defiant behavior, with an effect size of .35. In addition, MFG experimental group participants also exhibited significantly greater SSRS-SSS scores at post-treatment (b = 3.45, SE = 1.51, Z = 2.28, p = .02), with an effect size of .32. No significant differences between conditions at post-treatment were found on the six items of the IRS. Tests of within-group differences (see Table 4) suggest that both groups experienced significant reductions across areas of impairment at the time of post-treatment assessment. Lastly, 16% (n = 36) of children assigned to the MFG condition were rated as below the clinical cutoff on the IOWA CRS OD subscale versus 3% (n = 5) of children assigned to the SAU condition.

Table 2.

Means and SDs for Outcomes Measures by Time and Treatment Group.

MFG
 SAU
Pre Post Pre Post
IOWA CRS OD 9.28 (3.37) 7.74 (3.75) 9.05 (3.70) 9.01 (3.81)
SSRS-SSS 38.27 (10.64) 43.59 (11.50) 39.66 (11.09) 39.72 (10.77)
IRS #1 3.60 (1.47) 2.96 (1.47) 3.66 (1.43) 2.98 (1.35)
IRS #2 3.91 (1.45) 3.04 (1.48) 3.46 (1.42) 2.98 (1.36)
IRS #3 4.03 (1.55) 3.33 (1.55) 4.16 (1.51) 3.55 (1.53)
IRS #4 3.95 (1.46) 3.16 (1.51) 3.75 (1.45) 3.43 (1.43)
IRS #5 4.15 (1.40) 3.32 (1.47) 3.57 (1.43) 3.23 (1.37)
IRS #6 4.59 (1.15) 3.49 (1.35) 4.45 (1.06) 3.68 (1.39)
Open in a new tab

MFG = multiple family group; SAU = services as usual; IOWA CRS OD = Iowa Connors Oppositional/Defiant Subscale; SSRS-SSS = Social Skills Rating Scale–Social Skills Subscale; IRS #1 = Impairment Rating Scale Impairment Playmates; IRS #2 = Impairment Rating Scale Impairment Parents; IRS #3 = Impairment Rating Scale Impairment Academics; IRS #4 = Impairment Rating Scale Impairment Self-Esteem; IRS #5 = Impairment Rating Scale Impairment Family; IRS #6 = Impairment Rating Scale Overall Impairment/Need for Services.

Table 3.

Test of Group Differences.

Outcome variable Contrast estimate SE Z statistic p value Effect size (Cohen’s d)
IOWA CRS OD −1.20 0.48 −2.48 .01* .35
SSRS-SSS 3.45 1.51 2.28 .02* .32
IRS #1 −0.01 0.19 −0.04 .97 .01
IRS #2 0.10 0.21 0.49 .62 .07
IRS #3 −0.24 0.21 −1.15 .25 .15
IRS #4 −0.20 0.22 −0.92 .36 .14
IRS #5 0.14 0.21 0.66 .51 .10
IRS #6 −0.15 0.18 −0.84 .40 .13
Open in a new tab

OWA CRS OD = Iowa Connors Oppositional/Defiant Subscale; SSRS-SSS = Social Skills Rating Scale–Social Skills Subscale; IRS #1 = Impairment Rating Scale Impairment Playmates; IRS #2 = Impairment Rating Scale Impairment Parents; IRS #3 = Impairment Rating Scale Impairment Academics; IRS #4 = Impairment Rating Scale Impairment Self-Esteem; IRS #5 = Impairment Rating Scale Impairment Family; IRS #6 = Impairment Rating Scale Overall Impairment/Need for Services.

*

p < .05.

Table 4.

Tests of Within-Group Differences (Post-Baseline Assessments)

Outcome variable Condition Contrast estimate SE Z statistic p value Effect size (Cohen’s d)
IOWA CRS OD MFG −1.46 0.28 −5.21 .00** .42
SAU 0.02 0.42 0.05 .96 .01
SSRS-SSS MFG 4.76 0.84 5.70 .00** .44
SAU −0.12 1.18 −0.10 .92 .01
IRS #1 MFG −0.64 0.12 −5.30 .00** .44
SAU −0.68 0.18 −3.82 .00** .47
IRS #2 MFG −0.85 0.12 −6.82 .00** .58
SAU −0.51 0.18 −2.76 .01* .35
IRS #3 MFG −0.70 0.13 −5.39 .00** .46
SAU −0.61 0.19 −3.14 .00** .39
IRS #4 MFG −0.80 0.13 −6.17 .00** .55
SAU −0.37 0.19 −1.93 .05† .25
IRS #5 MFG −0.78 0.12 −6.32 .00** .55
SAU −0.31 0.18 −1.72 .09† .22
IRS #6 MFG −1.06 0.11 −9.44 .00** .94
SAU −0.78 0.17 −4.67 .00** .69
Open in a new tab

IOWA CRS OD = Iowa Connors Oppositional/Defiant Subscale; MFG = multiple family group; SAU = services as usual; SSRS-SSS = Social Skills Rating Scale–Social Skills Subscale; IRS #1 = Impairment Rating Scale Impairment Playmates; IRS #2 = Impairment Rating Scale Impairment Parents; IRS #3 = Impairment Rating Scale Impairment Academics; IRS #4 = Impairment Rating Scale Impairment Self-Esteem; IRS #5 = Impairment Rating Scale Impairment Family; IRS #6 = Impairment Rating Scale Overall Impairment/Need for Services.

†

p < .10

*

p < .05

**

p < .01

Discussion

The primary purpose of this study was to evaluate the effectiveness of the MFG service delivery model for youth with DBDs and their families who present to outpatient mental health clinics in socioeconomically disadvantaged communities. Data demonstrate that participation in the MFG service delivery model as compared with a SAU comparison group results in significantly improved child oppositional/defiant behavior and improved social skills. Importantly, however, the magnitude of these differences is small. Surprisingly, data also support that both MFG and SAU result in improvements in child impairment across multiple domains of functioning including impairment with peers, parents, academic functioning, self-esteem, and overall level of impairment. A minority of youth in MFG also experience clinically significant improvements in oppositional behavior. Engagement to MFG, as measured by rates of attendance, is notably high given the high-risk nature of the study population and relative to established rates of engagement in mental health services in socioeconomically disadvantaged communities (McKay & Bannon, 2004).

Improvements observed following MFG in oppositional/ defiant behavior, social skills, and impairment were not surprising. MFG was developed by integrating literature on parent and family factors associated with the maintenance of DBDs in youth. The four key skills utilized in MFG (i.e., relationships, respectful communication, rules, and responsibility) have a strong foundation in the literature. For instance, much literature has attested to the principle role of developing positive relationships, the establishment of rules and responsibilities (i.e., operationally defined targets for behavior of children), and the use of contingent consequences for rule/responsibility-following-behavior as effective methods for reducing for disruptive behavior in youth (Chorpita & Daleiden, 2009; Garland et al., 2008). As such, the findings of this study further support the value of utilizing these common elements of evidence-based interventions when treating DBDs.

In comparison with other effectiveness trials (see Lee, Horvath, & Hunsley, 2013 for a recent review of this literature), between-group effect size data for the MFG model (d = .35) appear to be in line with what is typically reported (average d = .30 for DBD outcomes), but within-group effect sizes for MFG (d = .42) are smaller than what is typically reported in the effectiveness literature (average d = .68 for DBD outcomes). One-to-one comparisons between effect sizes for MFG and the average effect size data reported from other effectiveness studies, however, are not straightforward. To our knowledge, including this MFG clinical effectiveness study, there are very few effectiveness studies that have been conducted with clinic-referred school-age youth with DBDs receiving treatment by practicing clinicians in routine outpatient mental health clinics. Of those that are more representative of routine practice and focus on school-age youth, MFG has greater effects than found in some studies (e.g., Ogden & Hagen, 2008). Other studies have design limitations (e.g., uncontrolled trial; Costin & Chambers, 2007) that limit the ability to compare effect size findings with those reported herein of MFG.

The data also point to an unexpected finding—impairment in youth improved over time, regardless of treatment condition. This is not a surprising finding for youth assigned to the MFG condition as impairment should be expected to improve as a result of improvements in oppositional/defiant behavior and social skills. For youth assigned to the SAU condition, however, it is not entirely clear why parents rated their child as less impaired given that both oppositional/defiant behavior and social skills did not improve significantly. It may be that SAU affects different behaviors than oppositional/defiant behaviors and social skills, resulting in overall reduction in child impairment. Given that psychiatric comorbidity in youth is high (Angold, Costello, & Erkanli, 1999), a reasonable explanation for this improvement in impairment is the effect of SAU on comorbid conditions rather than DBD symptoms specifically. Moreover, it may be that parents in the SAU condition have changing perceptions and attributions about/tolerance for impairment as a function of receiving mental health services (Canino & Alegria, 2008). Further exploration into the reasons for this finding is necessary.

Attendance to MFG, although variable, was relatively high given the duration of the intervention and the putative risk factors for poor engagement in socioeconomically disadvantaged communities (Gopalan, Goldstein, Klingenstein, Sicher, & McKay, 2010). Importantly, engagement to outpatient mental health clinics in socioeconomically disadvantaged communities is often poor—data suggests that families drop out from services within three sessions (McKay & Bannon, 2004). Although few studies have targeted higherrisk families in parent/family-based treatments for DBDs, data suggest that the 59% average attendance rate observed in this study is as good as what is typically observed in other studies of high-risk families—which is estimated to be approximately 63% (see Ingoldsby, 2010, for a recent review of this literature). Unlike a majority of studies in this area, the current effectiveness trial relied on existing outpatient mental health staff and treatment-seeking families in socioeconomically disadvantaged communities across multiple sites. The relatively high level of attendance suggests that families presenting to various clinics were engaged to the MFG groups, suggesting that qualities of the MFG group rather than qualities of the clinic or clinic staff played an important role in engaging these families.

As we have discussed, adoption of treatments into routine outpatient mental health settings requires not only evidence of effectiveness but also consideration toward the levels of training and supervision required for an intervention to be readily implemented and sustained. In this regard, MFG stands out relative to other effective interventions. MFG training requires 6 hr of face-to-face contact, which is substantially less than other interventions. As an example, evidence-based, modular, individual-based (i.e., Weisz et al., 2012), and other group-based formats (Incredible Years; Gardner et al., 2006) require five full days of training or more. In addition, supervision for MFG requires 60 min per group (six to eight families) per week, which is considerably less than what is needed for other models, particularly those relying on individual delivery formats (cf. Weisz et al., 2012). As such, supervisory needs are considerably reduced because of the MFG group-format delivery.

Moreover, in relation to adoption of novel interventions in clinical practice, consideration must also be given to how an intervention addresses other service delivery constraints observed in publicly funded outpatient mental health clinics. For instance, one glaring issue has been the significant shortage in availability of services in publicly funded outpatient mental health clinics (Gopalan et al., 2010). In the current clinical effectiveness study, where MFG youth were allowed to receive all services available to them at the mental health clinic, half of the participants in the MFG condition (53%) received no other treatments during the course of the study. This finding suggests that MFG may be a very efficient model for providing evidence-based services to families and may address the issue of significant shortage of mental health providers/services. It is likely that other group-based formats (e.g., Incredible Years) can offer the same opportunity to expand the capacity of service provision. However, MFG is a unique group format in that it expands traditional parent-group formats by including the entire family in the therapeutic process. This is not an insignificant distinction. For families that do not have the ability to find child care and therefore must bring their child and their the child’s siblings to treatment, the mental health setting must either provide child care or include the child (and potentially siblings) in treatment. Publicly funded outpatient mental health clinics may not have the resources (staffing or space) to conduct separate groups for parent and children. The MFG model is an ideal format to address this practical service issue.

Limitations

There are notable limitations in this study that require consideration when interpreting findings. First, assignment to treatment condition was not randomized. Importantly, however, there were no baseline differences between the two treatment conditions that were related to treatment outcomes. As such, there is greater confidence that despite the lack of randomization to treatment group, the internal validity of the study is strong. Moreover, dosage of treatment and attendance to treatment were difficult to obtain for youth enrolled in the SAU condition. Although retrospective reports from parents of youth in the SAU condition were gathered regarding this information, it was determined that the data were unreliable. As such, we do not know the intensity of and level of engagement in mental health services for youth in the SAU condition, and comparison between the two treatment conditions is not possible.

An important limitation of this study was the exclusive use of parent-report measures of outcomes. Although outcomes based on multiple methods and multiple reporters would have augmented the assessment battery used in this study, a decision was made in partnership with the outpatient mental health clinic sites that it was not feasible to collect the more traditional outcomes used in family-based treatment studies of DBDs (e.g., observations of parent– child and/or family interactions). Clearly, a broader number of outcomes (e.g., school outcomes; youth-report outcomes) would have provided additional information on the scope of effects of the MFG model. This is a notable limitation of the current study. Importantly, the findings herein are consistent with the smaller scale studies conducted on MFG (McKay et al., 1999; Mckay et al., 1996; McKay et al., 2002; Stone et al., 1996), and psychometrically valid and treatment-sensitive measures that have been utilized in other studies (e.g., Chacko et al., 2009), increasing confidence in the validity of the positive outcomes found in this study.

Clinical Implications and Future Directions

This is one of a handful of studies that have evaluated an evidence-based intervention within the context of routine mental health practices across multiple outpatient clinics that serve youth with DBDs and their families from socioeconomically disadvantaged communities. The small number of treatment fidelity ratings suggests that frontline mental health providers in outpatient mental health clinics may be able to implement MFG with at least adequate levels of fidelity. Clearly, more observations of fidelity are needed to make stronger claims regarding fidelity to the MFG model. Nevertheless, the available data suggest that adequate fidelity to MFG can be achieved across multiple settings and providers, which speaks to the potential ecological validity of MFG and the potential for wide-scale adoption.

Despite statistically significant differences between treatment conditions at immediate post-treatment on improvement in oppositional/defiant behaviors and social skills, the data indicate that these differences are small. This, however, is in line with effects sizes reported in a meta-analysis comparing evidence-based treatments with usual care in community practice (Weisz, Jensen-Doss, & Hawley, 2006) as well as data from effectiveness trials of evidence-based interventions for the treatment of DBDs (Lee et al., 2013). We anticipate, as has been reported in the literature on evidence-based treatments compared with usual care (Weisz et al., 2006), continued gains following MFG and greater effect sizes at follow-up assessment points. Theoretically, greater changes may be observed as families continue to utilize methods and strategies they acquired during the MFG group, thereby continuing to improve child outcomes.

Collectively, the data presented herein suggest that the MFG model is an effective, engaging, and efficient model that has potential for adoption in routine mental health practice. We do not suggest, however, that the MFG model is the appropriate model for all contexts. Rather, considerations regarding the parameters of an intervention (effectiveness, efficiency, complexity, appropriateness for the target population, training/supervision requirements, etc.) must all be considered when making a decision regarding what intervention to use in a given context. The literature offers various well-established interventions (i.e., Incredible Years, Parent–Child Interaction Training, Triple P, Parent Management Training, etc.) and more novel approaches (i.e., modular treatment approaches; Weisz et al., 2012) to consider instead of or in addition to MFG. Importantly, having a “menu” of evidence-based options is an important step toward filling the science-to-service or research-to-practice gap as no one intervention will meet the needs of all youth, families, and clinical contexts.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for this study was obtained through R01 MH072649 (principal investigator: McKay). Salary support for this study also came from National Institute of Mental Health (NIMH; F32MH090614; Gopalan). Dr. Gopalan is also an investigator with the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis, through an award from the NIMH (R25MH080916-01A2) and the Department of Veterans Affairs, Health Services Research and Development Service, Quality Enhancement Research Initiative (QUERI).

Footnotes

The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIMH or the National Institutes of Health.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

  1. Alexander J, Pugh C, Parsons B, Sexton T. Functional family Therapy. In: Elliott DS, editor. Blueprints for violence prevention Functional family therapy. Vol. 8. Boulder, CO: Institute of Behavioral Science; 2000. pp. 185–205. [Google Scholar]
  2. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 4th ed., text rev. Washington, DC: Author; 2000. [Google Scholar]
  3. Angold E, Costello EJ, Erkanli A. Comorbidity. Journal of Child Psychology and Psychiatry. 1999;40:57–87. [PubMed] [Google Scholar]
  4. Axberg U, Broberg AG. Evaluation of “The Incredible Years” in Sweden: The transferability of an American parent training program to Sweden. Scandinavian Journal of Psychology. 2012;53:224–232. doi: 10.1111/j.1467-9450.2012.00955.x. [DOI] [PubMed] [Google Scholar]
  5. Behan J, Fitzpatrick C, Sharry J, Carr A, Waldron B. Evaluation of the parenting plus program. The Irish Journal of Psychology. 2004;22:238–256. [Google Scholar]
  6. Canino G, Alegria M. Psychiatric diagnosis-is it universal or relative to culture? Journal of Child Psychology and Psychiatry. 2008;49:237–250. doi: 10.1111/j.1469-7610.2007.01854.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Chacko A, Wymbs BT, Arnold FW, Pelham WE, Swanger-Gagne M, Girio EL, O’Connor B. Enhancing traditional behavioral parent training for single mothers of children with ADHD. Journal of Clinical Child & Adolescent Psychology. 2009;38:206–218. doi: 10.1080/15374410802698388. [DOI] [PubMed] [Google Scholar]
  8. Chorpita BF, Daleiden EL. Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology. 2009;77:566–579. doi: 10.1037/a0014565. [DOI] [PubMed] [Google Scholar]
  9. Costin J, Chambers SM. Parent management training as a treatment for children with oppositional defiant disorder referred to a mental health clinic. Clinical Child Psychology and Psychiatry. 2007;12:511–524. doi: 10.1177/1359104507080979. [DOI] [PubMed] [Google Scholar]
  10. Eyberg S, Nelson M, Boggs S. Evidence-based psychological treatments for children and adolescents with disruptive behavior. Journal of Clinical Child & Adolescent Psychology. 2008;37:215–237. doi: 10.1080/15374410701820117. [DOI] [PubMed] [Google Scholar]
  11. Fabiano GA, Pelham WE, Waschbusch D, Gnangy E, Lahey B, Chronis A, Burrows-MacLean L. A practical measure of impairment: Psychometric properties of the impairment rating scale in samples of children with attention-deficit hyperactivity disorder and two school-based samples. Journal of Clinical Child & Adolescent Psychology. 2006;35:369–385. doi: 10.1207/s15374424jccp3503_3. [DOI] [PubMed] [Google Scholar]
  12. Gallart S, Matthey S. The effectiveness of Group Triple P, and the impact of the four telephone contacts. Behaviour Change. 2005;22:71–80. [Google Scholar]
  13. Gardner F, Burton J, Klimes I. Randomized controlled trial of a parenting intervention in the voluntary sector for reducing child conduct problems: Outcomes and mechanisms of change. Journal of Child Psychology and Psychiatry. 2006;47:1123–1132. doi: 10.1111/j.1469-7610.2006.01668.x. [DOI] [PubMed] [Google Scholar]
  14. Garland AF, Hawley KM, Brookman-Frazee LI, Hurlburt M. Identifying common elements of evidence-based psychosocial treatments for children’s disruptive behavior problems. Journal of the American Academy of Child & Adolescent Psychiatry. 2008;47:505–514. doi: 10.1097/CHI.0b013e31816765c2. [DOI] [PubMed] [Google Scholar]
  15. Goodwin PJ, Leszcz M, Ennis M, Koopmans J, Vincent L, Guther H, Hunter J. The effect of group psychosocial support on survival in metastatic breast cancer. New England Journal of Medicine. 2001;345:1719–1726. doi: 10.1056/NEJMoa011871. [DOI] [PubMed] [Google Scholar]
  16. Gopalan G, Goldstein L, Klingenstein K, Sicher C, McKay M. Engaging vulnerable youth and their families. Journal of Canadian Child Psychiatry. 2010;19:182–196. [PMC free article] [PubMed] [Google Scholar]
  17. Gresham FM, Elliott SN. Social Skills Rating System. Circle Pines, MN: American Guidance Service; 1990. [Google Scholar]
  18. Hedeker D, Gibbons R, du Toit M, Cheng Y. IT>Supermix: Mixed effects models. Lincolnwood, IL: Scientific Software International; 2008. [Google Scholar]
  19. Hoagwood K, Burns B, Weisz J. A profitable conjunction: From science to service in children’s mental health. In: Burns B, Hoagwood K, editors. Community treatment for youth: Evidenced-based interventions for severe emotional and behavioral disorders. New York, NY: Oxford Press; 2002. pp. 327–338. [Google Scholar]
  20. Hoagwood KE, Cavaleri MA, Olin S, Burns B, Slaton E, Gruttadaro D, Hughes R. Family support in children’s mental health: A review and synthesis. Clinical Child and Family Psychology Review. 2010;3:1–45. doi: 10.1007/s10567-009-0060-5. [DOI] [PubMed] [Google Scholar]
  21. Hutchings J, Bywater T, Daley D, Gardner F, Whitaker C, Jones K, Edwards RT. Parenting intervention in Sure Start services for children at risk of developing conduct disorder: Pragmatic randomised controlled trial. British Medical Journal. 2007;334:678. doi: 10.1136/bmj.39126.620799.55. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Ingoldsby EM. Review of interventions to improve family engagement and retention in parent and child mental health programs. Journal of Child and Family Studies. 2010;19:629–645. doi: 10.1007/s10826-009-9350-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Keiley MK. Attachment and affect regulation: A framework for family treatment of conduct disorder. Family Process. 2002;41:477–492. doi: 10.1111/j.1545-5300.2002.41312.x. [DOI] [PubMed] [Google Scholar]
  24. Kleve L, Crimlisk S, Shoebridge P, Greenwood R, Baker B, Mead B. Is the Incredible Years programme effective for children with neuro-developmental disorders and for families with social services involvement in the “real-world” of community CAMHS? Clinical Child Psychology and Psychiatry. 2011;16:253–264. doi: 10.1177/1359104510366280. [DOI] [PubMed] [Google Scholar]
  25. Kling A, Forster M, Sundell K, Melin L. A randomized controlled effectiveness trial of parent management training with varying degrees of therapist support. Behavior Therapy. 2010;41:530–542. doi: 10.1016/j.beth.2010.02.004. [DOI] [PubMed] [Google Scholar]
  26. Lau AS, Fung J, Ho L, Liu L, Gudino O. PMT with high-risk immigrant Chinese families: A pilot group randomized trial yielding practice-based evidence. Behavior Therapy. 2011;42:413–426. doi: 10.1016/j.beth.2010.11.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Lee CM, Horvath C, Hunsley J. Does it work in the real world? The effectiveness of treatments for psychological problems in children and adolescents. Professional Psychology: Research and Practice. 2013;44:81–88. [Google Scholar]
  28. McKay M, Bannon W. Engaging families in child mental health services. Child and Adolescent Psychiatric Clinics of North America. 2004;13:905–921. doi: 10.1016/j.chc.2004.04.001. [DOI] [PubMed] [Google Scholar]
  29. McKay M, Gonzales J, Quintana E, Kim L, Abdul-Adil J. Multiple family groups: An alternative for reducing disruptive behavioral difficulties of urban children. Research on Social Work Practice. 1999;9:593–607. [Google Scholar]
  30. McKay M, Gonzales J, Stone S, Ryland D, Kohner K. Multiple family therapy groups. Social Work With Groups. 1996;18:41–56. [Google Scholar]
  31. McKay M, Harrison M, Gonzalez L, Quintana E. Multiple-family groups for urban children with conduct difficulties and their families. Psychiatric Services. 2002;53:1467–1468. doi: 10.1176/appi.ps.53.11.1467. [DOI] [PubMed] [Google Scholar]
  32. Michelson D, Davenport C, Dretzke J, Barlow J, Day C. Do evidence-based interventions work when tested in the “real world”? A systematic review and meta-analysis of parent management training for the treatment of child disruptive behavior. Clinical Child and Family Psychology Review. 2013;16:18–34. doi: 10.1007/s10567-013-0128-0. [DOI] [PubMed] [Google Scholar]
  33. Ogden T, Hagen KA. Treatment effectiveness of parent management training in Norway: A randomized controlled trial of children with conduct problems. Journal of Consulting and Clinical Psychology. 2008;76:607–621. doi: 10.1037/0022-006X.76.4.607. [DOI] [PubMed] [Google Scholar]
  34. Pelham WE, Gnagy EM, Greenslade KE, Milich R. Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders. Journal of the American Academy of Child & Adolescent Psychiatry. 1992;31:210–218. doi: 10.1097/00004583-199203000-00006. [DOI] [PubMed] [Google Scholar]
  35. Phillips J, Morgan S, Cawthrone K, Barnett B. Pilot evaluation of parent-child interaction therapy delivered in an Australian community early childhood clinic setting. Australian and New Zealand Journal of Psychiatry. 2008;42:712–719. doi: 10.1080/00048670802206320. [DOI] [PubMed] [Google Scholar]
  36. Stone S, McKay M, Stoops C. Evaluating multiple family groups to address the behavioral difficulties of urban children. Small Group Research. 1996;27:398–415. [Google Scholar]
  37. U.S. Public Health Service. Report of the Surgeon General’s Conference on Children’s Mental Health: A national action agenda. Washington, DC: Department of Health and Human Services; 2000. [PubMed] [Google Scholar]
  38. Waschbusch DA, Willoughby MT. Parent and teacher ratings on the IOWA Conners Rating Scale. Journal of Psychopathology and Behavioral Assessment. 2008;30:180–192. [Google Scholar]
  39. Weisz JR, Chorpita BF, Palinkas LA, Schoenwald SK, Miranda J, Bearman SK …Research Network on Youth Mental Health. Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: A randomized effectiveness trial. Archives of General Psychiatry. 2012;69:274–282. doi: 10.1001/archgenpsychiatry.2011.147. [DOI] [PubMed] [Google Scholar]
  40. Weisz JR, Jensen-Doss A, Hawley KM. Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist. 2006;61:671–689. doi: 10.1037/0003-066X.61.7.671. [DOI] [PubMed] [Google Scholar]
  41. Weisz JR, Ugueto AM, Cheron DM, Herren J. Evidence-based youth psychotherapy in the mental health ecosystem. Journal of Clinical Child & Adolescent Psychology. 2013;42:274–286. doi: 10.1080/15374416.2013.764824. [DOI] [PubMed] [Google Scholar]

RESOURCES