Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Feb 1.
Published in final edited form as: Community Ment Health J. 2013 Jan 8;50(2):158–163. doi: 10.1007/s10597-012-9583-1

Consistency with evidence-based treatments and perceived effectiveness of children’s community-based care

Rachel Haine-Schlagel 1,✉, Danielle L Fettes 2, Antonio R Garcia 3, Lauren Brookman-Frazee 4, Ann F Garland 5
PMCID: PMC3648636  NIHMSID: NIHMS433673  PMID: 23296551

Abstract

This study examined whether delivery of psychotherapeutic strategies consistent with common elements of evidence-based (EB) treatments for child disruptive behavior problems was associated with parents' report of treatment effectiveness. The intensity of delivery of practice elements consistent with EB treatments was coded from a random sample of 538 videotaped psychotherapy sessions with 157 children/families and 75 therapists from six community-based clinics. Multilevel regression analyses tested whether intensity of EB practice elements was associated with parents' report of treatment effectiveness after four months, controlling for intensity of other practice elements. Results indicate parents reported greater perceived treatment effectiveness when community-based treatment included more intensive delivery of practice elements consistent with EB treatments to children. These findings may reassure providers about the acceptability of EB practice elements and may motivate efforts to integrate EB practice elements more intensively into community-based care.

Keywords: community-based care, childhood disruptive behavior problems, youth psychotherapy, perceived effectiveness, common elements

Reviews of community-based mental health services for children have documented generally disappointing outcomes, in particular when compared with evidence-based (EB) treatments (Weisz, Jensen-Doss, & Hawley, 2006; Weisz & Jensen, 2001). Wide-spread pressure exists to improve the quality of care by implementing EB treatments in community settings. However, effective EB treatment implementation has met with some challenges, including (but not limited to) therapist acceptability of EB treatments, implementing manualized interventions with fidelity, and bringing EB treatments to scale in a complex client population (Southam-Gerow, Hourigan, & Allin, 2009; Weisz, et al., 2006; Weisz, et al., 2009).

As a complementary approach to the implementation of individual EB treatment models, researchers have suggested a “common elements” approach to the implementation of EB practices (Chorpita, Daleiden, & Weisz, 2005; Garland, Hawley, Brookman-Frazee, & Hurlburt, 2008). Careful analysis of EB treatment models reveals that many share common treatment content and techniques (i.e., practice elements; Garland, Hawley, et al., 2008; Sburlati, Schniering, Lyneham, & Rapee, 2011). For example, an analysis of common treatment elements within EB treatment models for disruptive behavior problems (DBPs; i.e., aggression, defiance, delinquency, or oppositional behavior) in children examined both common content areas addressed and techniques used (Garland, Hawley, et al., 2008). This analysis indicated that problem solving and social skills, affect management, and focusing on the parent-child relationship were among the common content elements across EB treatment models. Common treatment techniques included modeling, role-playing and assigning and reviewing homework. The common elements approach refers to implementation of these common EB practices within community-based care, in contrast to implementing manualized treatment protocols. The common elements approach may provide a strong ecological fit with community-based care and may address some of the challenges of individual EB treatment protocol implementation given the approach’s flexibility and application to a diverse patient population (Garland, Bickman, & Chorpita, 2010; Garland, Hawley, et al., 2008; Sburlati, et al., 2011).

Recent research has begun to examine the degree to which therapists in community-based settings deliver strategies consistent with common elements of EB treatments (Garland, Brookman-Frazee, et al., 2010). However, little is known about whether naturally-occurring variability in the delivery of common elements of EB treatments is associated with differential outcomes, including consumer perspectives such as parents’ perception of the effectiveness of care. In addition to changes in symptoms and functioning, the consumer perspective on care effectiveness is a key outcome domain in children’s mental health care and a particularly salient outcome domain for providers (Hoagwood, Jensen, Petti, & Burns, 1996). The salience for providers is notable given existing research has found limited associations between consumer perspectives and change in prospectively assessed clinical outcomes (Garland, Aarons, Hawley, & Hough, 2003; Garland, Haine, & Lewczyk Boxmeyer, 2007).

An established literature on client and provider factors associated with consumer perspectives exists (Biering, 2010). However, no research has examined how observed within-session treatment processes, including variability in the intensity of the delivery of common EB elements, may be associated with consumer perceptions of treatment effectiveness in community-based care for children. Given that consumer perspectives such as perceived effectiveness are of importance to provider organizations, greater understanding of this association would be helpful in implementing EB treatments in community-based settings. This paper contributes to this understanding by examining the association between the observed intensity of therapists’ delivery of practice elements common in EB treatments for DBPs and parents’ perception of the effectiveness of care while controlling for the intensity of other practice elements not identified in EB treatments.

Methods

Data are drawn from a longitudinal observational study examining community-based care for children with DBPs, known locally as Practice and Research: Advancing Collaboration (PRAC; Garland, Brookman-Frazee, et al., 2010). Please refer toGarland et al. (2010) for details on practice settings, client recruitment, and procedures. Data were collected from 1) in-person baseline interviews with parents or caregivers (hereafter referred to as parents); 2) telephone interviews with parents four months after their child entered a new episode of outpatient mental health care; and 3) up to four randomly selected videotapes of psychotherapy sessions with each child/family during the first four months of treatment. University of California, San Diego Institutional Review Board approval was obtained. Written informed consent/assent was obtained after a complete description of the study was provided to participants.

Participants

The current subsample included 75 therapists from six publicly-funded clinics in San Diego County. All clinics served patients whose care was funded by Medi-Cal (California’s Medicaid) or the public school system. Therapists were primarily female (n = 65, 86.7%). Therapist race/ethnicity was 65% Caucasian, 9% Hispanic/Latino, 4% African American, and 21% Other/Mixed. The majority of therapists had a Master’s degree (60%), with 37% with a Bachelor’s degree and 3% with a doctorate. Therapist experience averaged 2.7 years (SD = 3.5), with a range from 0 to 25 years.

A total of 157 children with complete data on the variables of interest were included. Eligibility criteria included: 1) entering a new episode of psychotherapy (defined as no therapy for the previous three months) with a participating therapist; 2) age 4–13; 3) presented with a parent-reported DBP (aggression, defiance, delinquency, and/or oppositional behavior); and 4) had an English and/or Spanish speaking parent. The average child age was 9 years old (SD = 2.7; range 4–14), the majority were male (67.5%), and just over half were racial/ethnic minorities (30% Hispanic, 9% African American, 12% mixed/another race). Child primary clinician-assigned diagnosis was 41% ADHD, 20% disruptive behavior disorder, 22% mood disorder, 10% anxiety disorder, 5% autism spectrum disorder, and 1% other; a total of 48% had more than one diagnosis. A total of 66% of children were on medications at the time of the baseline assessment and 75% were still in treatment at the end of four months. Parents of participating children were primarily female (n = 146, 93%) biological mothers (n = 127, 80.9%). A total of 10 fathers (6.4%) participated, and 54% of all the parents were single parents.

Average number of visits in the first four months was 10.0 (SD = 4.6) with a range of 1 to 21. Duration of visits was typically 50 minutes, the standard length of an outpatient session.

Measures

Outcome: Perceived Effectiveness

Perceived effectiveness was assessed by parent report on a subscale of the Multidimensional Adolescent Satisfaction Scale (Garland, Saltzman, & Aarons, 2000) at four months after start of treatment. This subscale includes four items scored on a four-point scale and include, “Has counseling helped your child feel better about him/herself?”; “Has counseling helped your child’s problems get better?”; “Has counseling helped your child learn more about him/herself?”; and “Overall, how much has counseling helped your child?” The subscale was originally developed for adolescent reporters and was adapted for use by parents for this study. The measure has demonstrated good internal consistency, strong test-retest reliability, and convergent, divergent, and predictive validity when used with adolescents (Garland, et al., 2000). Several federally funded research projects including ours are currently using the adapted parent report version (R01-MH66070; R01- MH071483; K23-MH077584). The scale also demonstrates strong internal reliability (α = .90) in the current sample. The score used in this study was the mean item response (range from 1–4; mean = 3.0 (SD = .87).

Predictors: Practice Elements

The predictors of interest are composites of practice elements consistent with EB treatments from the PRAC Therapeutic Process Observational Coding System for Child Psychotherapy–Strategies scale (PRAC TPOCS-S; Garland, Brookman-Frazee, et al., 2010; Garland, Brookman-Frazee, & McLeod, 2008) as observed in community-based care. The PRAC TPOCS-S is an adapted version of the original TPOCS-S (McLeod & Weisz, 2010) based on collaboration with a group of community providers (Garland, Plemmons, & Koontz, 2006). The PRAC TPOCS-S includes codes measuring the delivery of 27 practice elements, divided into elements previously identified as common to EB treatments for children with DBPs (Garland, Hawley, et al., 2008) and other elements utilized by therapists in community-based care. Thirty-one percent (n = 379) of all 1215 coded sessions across the larger study’s 16-month study period were double coded to assess inter-rater reliability and yielded a mean ICC of .78 across all PRAC TPOCS-S codes, which indicates adequate inter-rater reliability (Garland, Brookman-Frazee, et al., 2010). The current subsample consists of 538 coded videotapes within the first four months of treatment.

Up to four tapes were coded per family across the four-month time period (the first tape submitted and up to three additional tapes across the four months). The PRAC TPOCS-S variables used in this study are eight average composites of unique subsets of individual code occurrence/intensity scores (herein referred to as intensity) across the set of coded tapes per child. Intensity reflects both the time spent on the practice element and the thoroughness with which it was pursued. Intensity is rated from 0–6 on a Likert scale with 0 representing no observed delivery of the practice element, 1–2 = low intensity, 3–4 = medium intensity, and 5–6 = high intensity.

The eight PRAC TPOCS-S composites are organized into four EB practice composites, which are the predictors of interest, and four “Other” composites, which control for overall intensity of therapy delivered. The composites are as follows:

  • Child “EB Content” includes problem-solving, affect education, and affect management (M = 1.33; SD = .83).

  • Child “Other Content” includes positive reinforcement principles, family members’ roles, parent-child relationship, improved communication, and addressing the child’s external care (M = .68; SD = .46).

  • Child “EB Technique” reflects active skill building and includes use of positive reinforcement, use of limit-setting, modeling, role-plays, assigning/reviewing homework, psychoeducation, and establishing/reviewing goals (M = 1.32; SD = .63).

  • Child “Other Technique” includes addressing client-therapist relationship, addressing client resistance, interpreting the meaning of behavior, exploring client/family past, identifying strengths, and play/art (M = 1.20; SD = .40).

  • Parent “EB Content” includes problem-solving, affect management, positive reinforcement principles, limit-setting principles, and parent-child relationship (M = .67; SD = .59).

  • Parent “Other Content” includes affect education, family members’ roles, improved communication, addressing medication needs, and addressing parent/family issues (M = .98; SD = .61).

  • Parent “EB Technique” reflects active skill building and includes modeling, assigning/reviewing homework, psychoeducation, and establishing/reviewing goals (M = 1.37; SD = .77).

  • Parent “Other Technique” includes use of positive reinforcement, interpreting the meaning of behavior, exploring client/family past, gathering information, and play/art (M = 1.04; SD = .55).

Means indicate that average intensity was quite low across all eight composites, suggesting that therapists did not provide great depth in their delivery of therapeutic strategies overall (Garland, Brookman-Frazee, et al., 2010).

Analysis Plan

To examine the associations of the four EB composites with perceived effectiveness, accounting for the nesting of clients within therapists, random intercept regression models were conducted using SuperMix v.1.1. Four separate predictor models were analyzed: child content, child technique, parent content, and parent technique with the corresponding EB composite and Other composite entered simultaneously. This allowed for the effect of the predictor of interest, intensity of delivery of the EB composite, to be tested independent of Other practice element intensity.

Results

The first analysis tested the effect of Child Content practice elements on perceived effectiveness. Neither the predictor of interest (Child EB Content) nor the control variable (Child Other Content) was significant (EB composite unstandardized regression coefficient or B = .139, standard error = .096; Other composite B = .238, standard error = .175). For the second model, Child Technique, the EB composite was significant (B = .347, standard error = .123, p<.01) while the Other composite was not significant (B = −.405, standard error = .194). Neither predictor in the third model, Parent Content, was significant (EB composite B = .169, standard error = .143; Other composite B = −.074, standard error = .138). For the fourth model, Parent Technique, the EB composite was a marginally significant predictor (B = .192, standard error = .104, p<.10) while the Other composite was not significant (B = −.130, standard error = .143). In sum, across the four models (child and parent EB content and techniques), only the Child Technique EB practice composite was a statistically significant predictor of perceived effectiveness, and the Parent Technique EB practice composite resulted in a trend in the same direction. None of the Other practice composites approached significance and in three of the four models the EB practice element composite was more strongly associated with perceived effectiveness than the Other composite.

Discussion

The results suggest that parents may feel more positively about the services they and their children receive when those services include higher intensity use of techniques consistent with EB treatments in naturally occurring community-based care. More specifically, parents perceived treatment to be more effective for their child when the treatment techniques were more consistent with EB treatments, particularly when treatment included more intensive use of active skill-building techniques (e.g., modeling, behavioral rehearsal through role play/ practice, homework) with children (and to some extent with parents). A relatively recent meta-analysis of parenting programs also found these techniques to be associated with positive child outcomes (Kaminski, Valle, Filene, & Boyle, 2008). In addition, a qualitative study of consumers indicated that parents and youth value directive, solution-focused treatment (Baker-Ericzén, Jenkins, & Haine- Schlagel, in press).

The finding that average intensity of Other elements was not a significant predictor of perceived effectiveness is notable. Although overall intensity across practice elements in this study was low, the results indicate that more intensive use of practice elements consistent with EB treatments was linked to greater perceived effectiveness rather than just greater intensity on any practice element.

These results support ongoing efforts to enhance parent involvement in community-based care (K. E. Hoagwood, et al., 2010). Parent perceptions about the effectiveness of care may impact ongoing motivation participate in their child’s therapy, which has been linked to treatment engagement (Nock & Photos, 2006). The results further indicate that parents may be more satisfied with care that integrates common elements of EB treatments, which suggests that efforts to enhance parent involvement may benefit from educating consumers about EB treatments. Furthermore, although marginal, the parent EB technique effect suggests the possibility that parents see a benefit to themselves being targeted more actively in therapy.

Strengths of the study include valid and reliable observational coding (vs. self-report) to assess within-session practice and the use of random effects multi-level modeling to account for the nested data structure (i.e., children within therapist clusters). The patient and provider samples were generally representative of other clinical samples nationally (Garland, Brookman-Frazee, et al., 2010); however, the findings have unknown generalizability to other types of service systems in other geographic areas. The study was limited in the ability to test additional child, parent, or therapist variables that may impact the findings due to sample size and power issues related to using multi-level modeling. Also, the observational coding measure did not include nonspecific factors such as warmth and support that may also impact outcomes. In addition, the indicator of treatment effectiveness is limited to the parent’s report of perceived effectiveness. Research has demonstrated that parents’ reports of perceived effectiveness, and satisfaction more broadly, are only minimally associated with other clinical outcomes such as standardized measures of symptoms and functioning (Garland, et al., 2003; Garland, et al., 2007). However, parent perceptions of services is a particularly important outcome for community-based providers (K. Hoagwood, Jensen, Petti, & Burns, 1996) and thus these results provide preliminary evidence that use of practice elements consistent with EB treatments may yield more positive quality of care ratings than other elements. The results are also consistent with recent findings that consumers prefer an EB treatment model to routine community-based care and consider the EB treatment model to be marginally more culturally competent than community-based care (Silovsky, et al., 2011). Given that one of the identified barriers to the implementation of EB practices is providers’ concerns that such treatments will not be well received by patients, these results can help address their concerns about the potential “fit” with community-based clients (Mazzucchelli & Sanders, 2010).

These results are important in the context of the current emphasis to implement EB treatments in diverse community-based settings. The findings may encourage providers to integrate EB elements more intensively into community-based care to boost perceived effectiveness. The results may also help facilitate provider and organizational support for efforts to implement common elements approaches to EB treatments. Although additional research is needed, the findings also support the potential usefulness of the common elements approach as a complementary alternative to implementation of traditional EB treatments in community-based settings.

Acknowledgments

Preparation of this article was supported by the following NIMH grants: R01-MH66070; K23-MH080149; K23-MH077584. The authors thank Michael Hurlburt, Ph.D. for comments on an earlier draft of the manuscript, Scott Roesch, Ph.D. for assistance with data analysis, William Ganger, MA for assistance with data management, Deb Dupuis, MPH and Robin Taylor for project management, and the families and therapists who participated in this study.

Contributor Information

Rachel Haine-Schlagel, San Diego State University and Child & Adolescent Services Research Center, 3020 Children’s Way MC 5033, San Diego, CA 92123; rhaine@casrc.org; 858-966-7703 x3583; 858-966-7704 (fax)

Danielle L. Fettes, University of California, San Diego and Child & Adolescent Services Research Center, San Diego, CA, 3020 Children’s Way MC 5033, San Diego, CA 92123

Antonio R. Garcia, University of Pennsylvania, School of Social Policy & Practice; 3701 Locust Walk; Philadelphia, PA 19104-6214, and Child & Adolescent Services Research Center

Lauren Brookman-Frazee, University of California, San Diego and Child & Adolescent Services Research Center, San Diego, CA, 3020 Children’s Way MC 5033, San Diego, CA 92123

Ann F. Garland, University of San Diego, University of California, San Diego, and Child & Adolescent Services Research Center, San Diego, CA, 3020 Children’s Way MC 5033, San Diego, CA 92123

References

  1. Baker-Ericzén MJ, Jenkins MM, Haine-Schlagel R. Therapist, parent, and youth perspectives of treatment barriers to family-focused community outpatient mental health child services. Journal of Child & Family Studies. doi: 10.1007/s10826-012-9644-7. (in press). [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Biering P. Child and adolescent experience of and satisfaction with psychiatric care: A critical review of the research literature. Journal of Psychiatric and Mental Health Nursing. 2010;17:65–72. doi: 10.1111/j.1365-2850.2009.01505.x. [DOI] [PubMed] [Google Scholar]
  3. Chorpita BF, Daleiden EL, Weisz JR. Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research. 2005;7:5–20. doi: 10.1007/s11020-005-1962-6. [DOI] [PubMed] [Google Scholar]
  4. Garland AF, Aarons GA, Hawley KM, Hough RL. Relationship of youth satisfaction with mental health services and changes in symptoms and functioning. Psychiatric Services. 2003;54:1544–1546. doi: 10.1176/appi.ps.54.11.1544. [DOI] [PubMed] [Google Scholar]
  5. Garland AF, Bickman L, Chorpita BF. Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health. 2010;37:15–26. doi: 10.1007/s10488-010-0279-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Garland AF, Brookman-Frazee L, Hurlburt M, Arnold E, Zoffness R, Haine-Schlagel R, et al. Mental health care for children with disruptive behavior problems: A view inside therapists’ offices. Psychiatric Services. 2010;61:788–795. doi: 10.1176/appi.ps.61.8.788. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Garland AF, Brookman-Frazee L, McLeod B. Scoring manual for the PRAC Study Therapy Process Observational Coding System for Child Psychotherapy: Strategies scale. 2008 [Google Scholar]
  8. Garland AF, Haine RA, Lewczyk Boxmeyer C. Determinates of youth and parent satisfaction in usual care psychotherapy. Evaluation and Program Planning. 2007;30:45–54. doi: 10.1016/j.evalprogplan.2006.10.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Garland AF, Hawley KM, Brookman-Frazee L, Hurlburt MS. Identifying common elements of evidence-based psychosocial treatments for children's disruptive behavior problems. Journal of the American Academy of Child and Adolescent Psychiatry. 2008;47:505–514. doi: 10.1097/CHI.0b013e31816765c2. [DOI] [PubMed] [Google Scholar]
  10. Garland AF, Plemmons D, Koontz L. Research-practice partnership in mental health: Lessons from participants. Administration and Policy in Mental Health & Mental Health Services Research. 2006;35:517–528. doi: 10.1007/s10488-006-0062-2. [DOI] [PubMed] [Google Scholar]
  11. Garland AF, Saltzman MD, Aarons GA. Adolescent satisfaction with mental health services: Development of a multidimensional scale. Evaluation and Program Planning. 2000;23:165–175. [Google Scholar]
  12. Hoagwood K, Jensen PS, Petti T, Burns BJ. Outcomes of mental health care for children and adolescents: I. A comprehensive conceptual model. Journal of the American Academy of Child & Adolescent Psychiatry. 1996;35:1055–1063. doi: 10.1097/00004583-199608000-00017. [DOI] [PubMed] [Google Scholar]
  13. Hoagwood KE, Cavaleri MA, Olin SS, Burns BJ, Slaton E, Gruttadaro D, et al. Family support in children's mental health: A review and synthesis. Clinical Child and Family Psychology Review. 2010;13:1–45. doi: 10.1007/s10567-009-0060-5. [DOI] [PubMed] [Google Scholar]
  14. Kaminski JW, Valle LA, Filene JH, Boyle CL. A meta-analytic review of components associated with parent training program effectiveness. Journal of Abnormal Child Psychology. 2008;36:567–589. doi: 10.1007/s10802-007-9201-9. [DOI] [PubMed] [Google Scholar]
  15. Mazzucchelli TG, Sanders MR. Facilitating practitioner flexibility within an empirically supported intervention: Lessons from a system of parenting support. Clinical Pyschology: Science and Practice. 2010;17:238–252. [Google Scholar]
  16. McLeod BD, Weisz JR. The Therapy Process Observational Coding System for Child Psychotherapy Strategies Scale. Journal of Clinical Child & Adolescent Psychology. 2010;39:436–443. doi: 10.1080/15374411003691750. [DOI] [PubMed] [Google Scholar]
  17. Nock MK, Photos V. Parent motivation to participate in treatment: Assessment and prediction of subsequent participation. Journal of Child and Family Studies. 2006;15:333–346. [Google Scholar]
  18. Sburlati ES, Schniering CA, Lyneham HJ, Rapee RM. A model of therapist competencies for the empirically supported cognitive behavioral treatment of child and adolescent anxiety and depressive disorders. Clinical Child and Family Psychology Review. 2011;14:89–109. doi: 10.1007/s10567-011-0083-6. [DOI] [PubMed] [Google Scholar]
  19. Silovsky JF, Bard D, Chaffin M, Hecht D, Burris L, Owora A, et al. Prevention of child maltreatment in high-risk rural families: A randomized clinical trial with child welfare outcomes. Children and Youth Services Review. 2011;33:1435–1444. [Google Scholar]
  20. Southam-Gerow MA, Hourigan SE, Allin RB., Jr Adapting evidence-based mental health treatments in community settings: Preliminary results from a partnership approach. Behavior Modification. 2009;33:82–103. doi: 10.1177/0145445508322624. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Weisz JR, Jensen-Doss A, Hawley KM. Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist. 2006;61:671–689. doi: 10.1037/0003-066X.61.7.671. [DOI] [PubMed] [Google Scholar]
  22. Weisz JR, Jensen AL. Child and adolescent psychotherapy in research and practice contexts: Review of the evidence and suggestions for improving the field. European Child and Adolescent Psychiatry. 2001;10:12–18. doi: 10.1007/s007870170003. [DOI] [PubMed] [Google Scholar]
  23. Weisz JR, Southam-Gerow MA, Gordis EB, Connor-Smith JK, Chu BC, Langer DA, et al. Cognitive-behavioral therapy versus usual clinical care for youth depression: An initial test of transportability to community clinics and clinicians. Journal of Consulting and Clinical Psychology. 2009;77:383–396. doi: 10.1037/a0013877. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES