Implementation and Evaluation of Evidence-Based Psychotherapeutic Practices for Youth in a Mental Health Organization

Abstract

The current initiative and program evaluation study is a demonstration of the research to practice process in youth-focused psychotherapy. We collaborated within a community-university partnership to create practice and research infrastructure in order to develop, implement, and evaluate two new models of service founded on evidence-based psychotherapeutic practice parameters. The two new service models incorporated validated interventions to address behavior problems in elementary age children, and depression in adolescents, which were delivered in separate but similarly run intensive outpatient programs within a mental health setting. We utilized a rigorous training, technical assistance, fidelity monitoring, and outcome measurement strategy to promote the integrity and quality of services provided. The resultant programs were delivered with acceptable to high fidelity and effects on youth and parenting measures collected during program and from pre to post showed a decrease in targeted problems in youth and positive benefits for families. This initiative and program evaluation adds to the accumulating research-to-practice literature in children’s mental health.

Introduction

There is a compelling research base for child- and parent/family-focused psychotherapeutic interventions for youth exhibiting a wide range of psychological and adjustment problems (Weisz 2004). Meta-analyses of efficacy and effectiveness trials reveal modest to high effects sizes on targeted outcomes when such programs are compared to no treatment or usual care conditions (Weisz et al. 1995, 2006a, b, 2013; Zhou et al. 2015). Emerging literature also shows that psychotherapy using practice elements (i.e., common content and delivery strategies observed across research-validated models) improves outcomes and satisfaction with services when well delivered (Chorpita et al. 2013; Garland et al. 2014; Weisz et al. 2012; Trask et al. 2016).

Although a number of evidence-based child- and parent/family-focused psychotherapies have been developed, their use in real-world settings has proven to be challenging (Beidas and Kendall 2010; Garland et al. 2014; Jensen-Doss et al. 2009), leading to a research to practice gap. Specifically, validated psychotherapies may not always be a good ‘‘fit’’ for some clients, there may be limited buy-in from practitioners regarding their use, practitioners may implement practice parameters in a substandard manner, and administrators may perceive them as expensive, unwieldy, and time consuming (Jensen-Doss et al. 2009; Southam-Gerow et al. 2012). Whatever transpires in practice has to be tailored to clients, must be fiexible and feasible for practitioners, and has to be adequately funded and resourced by a host organization (Southam-Gerow et al. 2012).

Incorporating principles of evidence-based practice (EBP) may help in resolving the research to practice challenges. In EBP, practitioners integrate their knowledge of current research, rely on their accumulating experience-based expertise in their application, and tailor interventions to their clients (American Psychological Association 2006; Kazdin 2008; Mitchell 2011). Furthermore, promoting and measuring fidelity of implementation may be important to the integrity of any practice application of EBPs (Beidas and Kendall 2010; Bloomquist et al. 2013; Schoenwald 2011). Research to practice efforts will likely be more successful if built upon a partnership between community practitioners that routinely delivery psychotherapy and university researchers who validate and disseminate psychotherapeutic protocols (Atkins et al. 2016; Bloomquist et al. 2008; Southam-Gerow et al. 2009).

Two common concerns for youth seeking psychotherapy are behavior disorders in children and depression in adolescents. There is extensive evidence supporting psychotherapy for these problems that could be the basis for EBPs. Specifically, child-focused skills training and parent management training interventions have positive effects for children with behavior disorders (Eyberg et al. 2008; Evans et al. 2014; McCart et al. 2006). Child-focused training, typically emphasizing social-emotional skills, is based on operant, modeling, coaching, social cognitive, and stress inoculation paradigms. Specific techniques or strategies include direct teaching of social behaviors (e.g., sharing, assertiveness), fostering perspective taking, attribution retraining, social problem solving, and coping with strong emotions in interpersonal contexts, as well as administering contingencies to shape specific behaviors and competencies (Garland et al. 2008; Kazdin 2010; Larson and Lochman 2002; Webster-Stratton and Reid 2010). Parent management training is based on social learning theory, applied behavior analysis, and operant methods to alter the child’s environment. Specific techniques or strategies include reinforcing desired behavior, ignoring/punishing undesired behavior, reducing coercive parenting and escalating parent–child interactions, enhancing parental monitoring, and building parent–child bonds (Barkley 2013; Dishion et al. 2012; Forgatch and Patterson 2010; Garland et al. 2008; Kazdin 2010; Sanders 1999; Webster-Stratton and Reid 2010).

With regards to depression symptoms in adolescents, cognitive behavioral therapy (CBT) is well supported (David-Ferdon and Kaslow 2008; Klein et al. 2007; Lewinsohn and Clarke 1999; Weisz et al. 2006a, b, 2013). CBT focuses on changing maladaptive thoughts, feelings, and behaviors that are interrelated and that underlie depression and emotional problems. Most CBT interventions for adolescents emphasize mood monitoring, behavioral activation and pleasant activity scheduling, and cognitive restructuring components with additional secondary foci on problem-solving, relaxation/emotion regulation skills, and assertiveness training (Cuijpers et al. 2009; Hetrick et al. 2015; Rohde et al. 2005). Parent involvement usually entails the parent receiving psychoeducation familiarizing them with what their adolescent is learning. Some protocols involve optional sessions to improve parent-teen interactions (Clarke et al. 1999; Gillham et al. 2012; Wells and Albano 2005).

In addition to the specified content noted above, it is also important to utilize of evidenced-based delivery techniques. The methods most widely used to provide interventions to youth and parents include presenting materials (charts, handouts, etc.), psychoeducation and didactic explanation, modeling and role-playing, assigning and reviewing homework, and reviewing goals and progress (Chorpita and Daleiden 2009; Garland et al. 2008).

There is a need for strategies and demonstrations for effectively transporting EBPs, such as those just described, into practice settings (Hamilton 2015; Southam-Gerow et al. 2012). Mindful of the opportunities and challenges described above, we attempted to infuse and then evaluate research-validated psychotherapies organized as EBPs within the routine practice structure of a large capacity real world mental health organization. This effort is built upon a community-university partnership that developed, implemented, and evaluated two six-week intensive out-patient programs for children with behavior problems and adolescents with depression respectively.

The two intensive outpatient programs were identical in structure and dosage, but the incorporated EBP content was contextualized to a specific targeted population age and presenting problem. The first intensive outpatient program, known internally as the Behavior Development Program (BDP), utilized child-focused skills training and parent management training for children ages 7–12 with behavior problems. The second intensive outpatient program, known internally as the Healthy Emotions Program (HEP), utilized CBT with parent involvement for adolescents ages 12–17 with depression. Services for both BDP and HEP were provided Mondays through Thursday after school for six consecutive weeks (24 days over 6 weeks). Multicomponent program infrastructure and training, technical assistance, and fidelity protocol were adapted from earlier research to practice work by the first author (Author citation 2008, 2013) which guided and promoted the integrity of BDP and HEP. The overarching goal of this initiative is to describe these programs and demonstrate how they were brought into a community mental health setting and evaluated for their effects on youth and family outcomes.

Method

Participants

DSM IV was used when these programs were originally evaluated to determine referrals for programming. It was stipulated that appropriate referrals for BDP were children with a primary diagnosis of ADHD, Oppositional Defiant Disorder, or Behavior Disorder NOS (with co-occurring depression and anxiety disorders allowed) and that appropriate referrals for HEP were adolescents with a primary diagnosis of Major Depressive Disorder, Mood Disorder, or Depressive Disorder NOS (with co-occurring anxiety and behavior disorders allowed). Exclusionary diagnoses for both programs were Autism Spectrum Disorder, Conduct Disorder, and Intellectual Disability. Referred youth were typically experiencing moderate to severe impairment at the time of intake hence the need for an intensive out-patient program. Most of the children and adolescents were taking psychotropic medication that is managed by medical providers not working in BDP or HEP.

Demographic characteristics are presented in Table 1. Overall, participants in the study were likely to come from high-income households, with the majority of parents reporting completion of a Bachelor’s or graduate degree. BDP participants (n = 34) were predominantly males (64.3 %), ranging in age from 7 to 12 with a median age of 10. The majority identified as White, followed by African American and Multiracial. DSM-IV diagnostic data was provided by PrairieCare and is based on the diagnostic code used to bill for insurance for reimbursement of services provided. Most children in BDP were categorized with disruptive behavior disorders diagnosis followed by mood disorders, ADHD, or anxiety. HEP participants (n = 28) were predominantly females (64.7 %), ranging in age from 12 to 17 with a median age of 16. The majority identified as White, followed by Multiracial and Hispanic. The majority of HEP participants’ diagnoses were depressive disorders followed by mood disorders.

Table 1.

Sample characteristics for families in the Behavior Development Program (BDP) and Healthy Emotions Program (HEP)

	BDP (n = 34)a	HEP (n = 28)
Gender
Male	64.3 %	35.3 %
Female	35.7 %	64.7 %
Age
Median	10.00	16.00
Average	10.02	15.70
Average IQ	96.64	107.13
Income	$77,5 85b	$127,501c
Parent education-Bachelor’s degree or higher	50%	67.9 %
Ethnicity
White	58.9 %	75.0 %
African American	17.6 %	-
Hispanic	5.9 %	7.1 %
Asian/Pacific Islander	-	3.6 %
American Indian/Alaska Native	-	-
Other	2.9 %	3.6 %
Multiple	14.7 %	10.7 %
Primary disorder
Depressive disorders	23.9 %	64.8 %
Disruptive behavior disorders	19.0 %	2.9 %
Mood disorders	19.0 %	23.5 %
Attention-Deficit/Hyperactivity Disorder	19.0 %
Anxiety disorders	7.1 %	8.8 %
Opposiu’onal defiant disorder	7.1 %
Adjustment disorder	2.5 %
Reactive attachment disorder	2.4 %

^aNs lower than reported in demographics due to attrition

^bMedian: $59,500; SD: $65,113; range: $8400–$250,000

^cMedian: $110,000; SD: $84,784; range: $25,000–$400,000

Procedure

Forming Practice and Research Infrastructure via a Community-University Partnership

A community-university partnership between PraireCare (comprising PrairieCare LLC and PrairieCare Medical Group), a large capacity mental health organization, and University of Minnesota researchers, was the foundation for the current project. PrairieCare provides a continuum of mental health care to several thousand youth per year and employs approximately 100 fully licensed behavioral health clinicians. Our University of Minnesota team has experience developing and evaluating children’s mental health programs. The partnership was fully collaborative and strove to further each other’s mission and goals in mutually beneficial ways (Israel et al. 1998).

The partnership adhered to recommended guidelines of EBPs (American Psychological Association 2006; Kazdin 2008; Mitchell 2011). This initially involved making decisions based on an integration of shared knowledge of what works, with the experience of PrairieCare’s practitioners, and with consideration of the unique characteristics of PrairieCare’s clients. We decided to develop evidence-based models of service for children with behavior disorders and adolescents with depression given the high rate of those presenting problems within PrairieCare’s practice. We also decided to use an intensive outpatient program model because PrairieCare perceived a gap in its continuum of care and wanted to better serve youth who needed more than outpatient services but did not warrant partial or full hospitalization. The final decision was to incorporate specific EBPs into the services being developed within BDP and HEP (described in next section).

The partners then cooperated in setting up practice and program evaluation procedures. This included the importation of quality assurance methods to support training, technical assistance, fidelity monitoring, and developing methods for subject recruitment, collection of fidelity and response data, and data sharing. The PrairieCare organization adopted BDP and HEP as routine care options for families seeking mental health care including billing insurance for services provided to make it feasible and sustainable. The University of Minnesota researchers assumed primary responsibility for spearheading the quality assurance methods and research.

PrairieCare deployed masters or doctoral-level practitioners to direct BDP and HEP with the assistance of bachelor-level psychiatric technicians and graduate school practicum students. Staff was required to have several years’ experience working with children and families. The masters or doctoral-level practitioners were the ‘‘Lead Practitioners,’’ responsible for intervention implementation and programming coordination. All staff was extensively trained. This included review of the session-by-session manual, 4 h of didactic training with role-playing, 1–2 weeks of shadowing experienced practitioners, and 1–2 weeks of guided practice/feedback with experienced practitioners before being certified as proficient. The Lead Practitioner was supervised by the Program Director (Bloomquist) for a minimum of 60 min per week and the Lead Practitioner supervised the psychiatric technicians and graduate practicum students for a minimum of 120 min per week.

Finally procedures for enrolling families into research were delineated. Youth and families that were in the process of enrolling in BDP or HEP were told about the research study by PrairieCare staff. Families were given the option of being contacted by research staff to participate in research but were told it was not a requirement to receive programming. For those who agreed to be contacted, a research technician then later met with youth and parents in their homes to conduct the consent/assent process, and administer pre-intervention measures. Post-test assessments were also conducted by research staff in each family’s home. All of these research procedures were approved by University of Minnesota and PrairieCare Institutional Review Boards (including a HIPAA compliant provision for PrairieCare to share diagnostic data for each research subject with university researchers).

EBPs Incorporated Into Programming

The intensive outpatient programs were modeled after specific evidence-based protocols but were also extended with related EBPs. The BDP is adapted from the multi-component Early Risers prevention program which is an early intervention system of care framework that has been validated across multiple studies (August et al. 2001, 2006; Bloomquist et al. 2012). Early Risers integrates different evidence-based child skills and parents skills curricula (depending on the study cited above) along with case management support for elementary age children with conduct problems. The BDP provided similar components and a comparable dosage to children who participated in the Early Risers effectiveness trials (August et al. 2006; Bloomquist et al. 2012) and incorporated child social-emotional skills training with parent management training methods informed by the broader EBP literature pertaining to children with behavior problems (e.g., Barkley 2013; Dishion et al. 2012; Forgatch and Patterson 2010; Garland et al. 2008; Kazdin 2010; Larson and Lochman 2002; Sanders 1999; Webster-Stratton and Reid 2010). The HEP was adapted from the Treatment of Adolescent Depression Study cognitive-behavioral therapy program (Treatment for Adolescents with Depression Study or TADS Team 2004, 2007) with similar components and a comparable dosage to adolescents and also utilized CBT strategies with parent facilitation procedures informed by the broader EBP literature for adolescents with depression (Clarke et al. 1999; Cuijpers et al. 2009; Gillham et al. 2012; Hetrick et al. 2015; Rohde et al. 2005; Wells and Albano 2005). Both BDP and HEP were organized around three two-week skills training modules. To make the programs viable and responsive within a real world practice context, youth were enrolled at the beginning of a module every two weeks and then completed six weeks. Tables 2 and 3 shows the practice content elements or procedures organized as modules for BDP and HEP.

Table 2.

Modules and skills training in the Behavior Development Program for children

Module	Child skills	Parent skills
Empathy and behavior	Compassion and accepting differences; listening to others with attention	Behavior management; avoiding power struggles and patiently following through
Emotions	Staying calm; handling uncomfortable social situations	Parental stress management; guiding/reinforcing child
Social	Getting alone with peers; getting along with family members	Family bonding; routines; problem-solving; guiding/reinforcing child

Table 3.

Modules and skills training in the Healthy Emotions Program for adolescents

Module	Adolescent skills	Parent skills
Behavior	Behavioral activation; communicating and being assertive with others	Personally using same skills as teen; guiding/reinforcing teen
Thoughts	Solving general and interpersonal problems; recognizing and replacing unhelpful thoughts	Personally using same skills as teen; guiding/reinforcing teen
Emotions	Recognizing and dealing with stress and overwhelming emotions	Personally using same skills as teen; guiding/reinforcing teen

The Early Risers program and TADS CBT served as organizational substructure upon which EBPs were provided in the BDP and HEP programs respectively. Consistent with an EBP approach (American Psychological Association 2006; Kazdin 2008; Mitchell 2011), some adaptations were made to the Early Risers and TADS CBT protocols from how they were originally deployed (Early Risers was school-and community-based and TADS CBT was weekly outpatient) to fit the current practice context. The adaptations involved delivering similar or increased dosage of psychotherapeutic skills training in a shorter time frame and adding additional practice elements and services consistent with an IOP service model and befitting of the specific populations served within the BDP and HEP at PrairieCare (intervention details described next). The PrairieCare staff was trained in the implementation of programming for BDP and HEP based on adapted versions of the original manuals created by the first author. Both manuals described essential practice methods for each session of youth and parent programming along with corresponding fidelity checklists for staff to adhere to for each session of youth and parent components. The manuals also outlined the program’s schedule, therapeutic, behavior management, and skills training methods, and included all handouts and supplementary materials such as goal, treatment planning, and parent–child activity forms.

In both BDP and HEP, the Lead Practitioner, and two to three co-leaders delivered a 3-h program each day with a cohort of six to ten youth at a time. During the first hour from 3:30 to 4:30 pm, participants engaged in Check-In and Recreational Activities. Youth met with staff to check-in, discuss progress on goals, and participate in engaging and fun activities. During the second hour from 4:30 to 5:30 pm, Structured Skills Training occurred. For BDP the well-validated Second Step curriculum (Frey et al. 2000; Grossman et al. 1997; Taub 2001), and for HEP the public domain cognitive behavioral therapy manual from the Treatment of Adolescent Depression Study (TADS Team 2004, 2007) were incorporated to teach children and adolescents skills respectively (Tables 2 and 3 show specific skills content). Skills were taught using presentation, discussion, behavioral rehearsal, and goal attainment methods. Therapeutic activities occurred during the third hour from 5:30 to 6:30 pm. This final hour consisted of goal updates, academic enrichment, and parent-youth activities. In total all participants in BDP and HEP received 72 h of programming with 24 h exposure to explicit skills training. Throughout all programming staff prompted and reinforced youth to practice newly-acquired skills and encouraged a supportive milieu.

A behavior management strategy was used throughout all programming. In BDP, the behavior management strategy involved children earning and losing points (at a 3:1 ratio) with an opportunity to ‘‘go shopping’’ at a ‘‘BDP Store’’ once per week to exchange earned points for toys, games, books, etc. In HEP, the behavior management approach involved daily review of rules, staff closely monitoring and redirecting youth to minimize negative peer influence, and occasionally individualized behavioral contracts between teens, parents, and staff.

Lead Practitioners in both programs were also responsible for completing brief daily risk assessments with youth each day. Staff evaluated current clinical status, and if there is any potential risk of harm to self or others then staff collaborated with youth (and parents, when necessary) to determine a safety plan. Additionally, staff held sessions with youth and/or parents outside of programming time when clinically indicated (e.g., brainstorming about effective behavioral strategies or risk-assessment). PrairieCare psychiatrists were available to assist staff and families in mental health services planning and respond to individual youth with emergent acute needs or risks. Psychiatry services were provided on an as needed basis for six weeks (adjusted for each family depending on need).

In both BDP and HEP, the Lead Practitioner provided 90-min parenting group sessions, with parent-youth activities, that were offered two afternoons per week (5–6:30 pm) for 18 h of exposure over six weeks. Parents were familiarized with the curriculum being taught to their child or teen in BDP or HEP respectively and learned skills for use with their child or teen. For BDP, the practice elements focused Struggling Kids parent skills training manual (Bloomquist 2013; which evolved from the Early Risers program), and for HEP, an organized and bound guidebook of handouts from the adapted TADS CBT manual (2004, 2007), were provided gratis to parents and served as the basis for parent skills training (Tables 1 and 2 show specific skills content). Skills were taught to parents using presentation, discussion, behavioral rehearsal, and goal attainment methods. There was also an opportunity for parents to receive support and feedback from other parents and practitioners. A half hour was devoted to activities for parents and their child or teen to meet to work on goals or practice skills learned that week.

In BDP and HEP, families received individualized support for their child or adolescent around school concerns and the parent around family concerns on an as needed basis over six weeks (adjusted for each family depending on need). The Lead Practitioner was available to meet with a family and/or coordinate with school personnel. The strategies were goal attainment, advocacy, and case management. In addition, at discharge all families met with the Lead Practitioners to review progress and determine an after program plan to maintain progress and/or address additional mental health or family needs.

The masters or doctoral-level practitioners discussed above were either independently licensed or supervised by independently licensed professionals allowing for billing of private or public third-party payers for professional services rendered. The 3-h intensive outpatient program provided to youth was billed on a daily basis for each youth in attendance. In addition, the parent groups and any family sessions occurring within the context of youth and family support outside of structured programming were billed separately.

Documenting Fidelity

We tracked fidelity of implementation including exposure, which is the quantity of sessions provided; adherence, or how close the content and delivery methods conformed to the manual or the practice elements; and quality, pertaining to how well the intervention was being delivered (Dane and Schneider 1998; Mihalic 2004).

Practitioners completed daily checklists indicating which subjects attended and their own adherence to manual procedures in several different categories. This procedure not only documented fidelity but also likely improved fidelity by continuously reminding practitioners of adherence parameters (see Bloomquist et al. 2013). For the BDP Child Skills groups, practitioners monitored provision of the Second Step curricula for that day, their use of behavior management techniques (e.g., points, time-outs), instructional techniques (e.g., didactic presentation, guiding/dialoguing, modeling and role playing, reinforcing and reviewing of skills), and goal setting. For HEP Teen Skills, practitioners monitored provision of the CBT curricula for that day, and they tracked their facilitation of goal setting and updating, positive behavior reinforcement, and instructional techniques. For the Parent Skills groups in both BDP and HEP, practitioners monitored provision of the Struggling Kids or CBT curricula respectively, and indicated whether they had utilized specific instructional techniques (e.g., didactic instruction, discussion, modeling, role playing), facilitated setting and updating of goals, and encouraged parents to support each other. For each fidelity log item, practitioners checked a box if they had adhered to the procedure or left it blank if they had not.

Trained graduate student raters (trained and supervised by the first author) formally assessed intervention delivery for fidelity of implementation. The BDP and HEP were each assessed six times over the course of approximately one year. Ratings for fidelity items ranged from 1–2 (not at all) to 9–10 (nearly all of the time). Category scores were averaged across three observation points per site. Raters assessed both adherence to manual procedures and quality of delivery (e.g., rapport, enthusiasm, preparedness). Adherence to manual procedures for both Child and Teen Skills groups included behavior management, teaching skills, and content. Adherence to manual procedures for both BDP and HEP Parent Skills groups included teaching skills and content. The quality of delivery composite for Child and Teen Skills and for Parent Skills primarily assessed rapport, enthusiasm, preparedness, and encouragement of participation. Inter-rater reliability for the observations was adequate (κ = .88).

Measures

Progress Monitoring Measures Administered During Programming

PrairieCare staff administered brief rating scales to parents in both BDP and HEP and to adolescents in HEP on a weekly basis to monitor progress.

The Impairment Rating Scale (IRS) (Fabiano et al. 2006) is an 8-item parent-report scale that measures the child’s ability to function across several domains (i.e., relationships with peers, parents, and siblings; emotional and academic functioning). Across several studies, the IRS has been shown to be highly accurate in identifying impairment in children with ADHD and conduct disorders in particular, and demonstrated strong evidence of concurrent, convergent, and discriminant validity overall (Fabiano et al. 2006). The IRS has been found to have good internal consistency (.87) and inter-rater reliability (.78).

The Home Situations Questionnaire (HSQ) (Barkley and Edelbrock 1987; Adams et al. 1995) is a 16-item parent-report scale that measures problems with the child’s frequency and severity of problem behaviors across different home situations, such as homework time, mealtimes, and daily chores (Barkley and Edelbrock 1987). Strong internal consistency and test–retest reliabilities have been documented for the HSQ in both child and adolescent populations, with internal consistency coefficients ranging from .79 to .86 (Adams et al. 1995). Stability coefficients have been found to be adequate for the global scores of Number of Problem Situations (.64–.77), and Mean Severity (.77–.82) (Adams et al. 1995; Altepeter and Breen 1989).

The Strengths and Difficulties Questionnaire (SDQ) (Goodman 2001) measures psychological adjustment in youth. In the current investigation, the Parent and Self-Report Emotional Symptoms and Conduct Problems Sub-scales, each composed of 5 items, were used in the HEP program to measure distress, anxiety, rule-breaking, non-compliance, and oppositional behavior. The SDQ has shown adequate internal consistency for both Emotional Symptoms (Parent: .67; Youth: .66) and for Conduct Problems (Parent: .63; Youth: .60) (Goodman 2001). It has also shown adequate inter-rater reliability between parent-report and self-report scales for both Emotional Symptoms (.37) and for Conduct Problems (.44). In the current investigation, inter-rater reliability was adequate (ICC = .58) for Emotional Symptoms. There was disagreement, however, between parents and adolescents on the Conduct Problems subscales (ICC = −.07), which has been reported similarly for a clinical sample elsewhere (Van Roy et al. 2010). Since pre- post analyses analyzed parent and adolescent ratings separately, this disparity on the Conduct Problems subscale did not affect change score results.

Child and Family Outcomes Administered Before and After Programming

We collected measures of youth mental health and parenting prior to beginning the program, and an average of two months after program discharge.

The Behavioral Assessment System for Children-2-Parent Rating Scale for Children (BASC-2, PRS-C) and Adolescents (BASC-2, PRS-A) (Reynolds and Kamphaus 2004) are questionnaires that evaluate behavioral and emotional functioning in youth. Composite scores for Externalizing Problems, Internalizing Problems, a Behavioral Symptoms Index, and Adaptive Skills are calculated based on the 14 subscales. Gender-specific T-scores are provided with a mean of 50 and a standard deviation of 10. Within a clinical norm group, internal consistency reliability coefficients are generally .90 or above and individual scale alphas ranged from .83 to .89 for PRS-C and from .83 to .86 for PRS-A (Reynolds and Kamphaus 2004). Inter–rater reliabilities for the PRS-C (.69) and PRS-A (.77) were found to be good, and factor analytic techniques have revealed that there is a consistent factor structure and good construct validity (a = .77 to .94) (Reynolds and Kamphaus 2004).

The Behavioral Assessment System for Children-2-Self Report of Personality (BASC-2, SRP) (Reynolds and Kamphaus 2004) is a questionnaire designed to evaluate youth behavioral and emotional functioning, as well as perceptions of self. The SRP has five composite scores: School Problems, Internalizing Problems, Inattention/Hyperactivity, Personal Adjustment, and the Emotional Symptoms Index, which is an overall composite score with both clinical and adaptive scales (Reynolds and Kamphaus 2004). According to Reynolds and Kamphaus (2004), reliability estimates for the composites range from the high .80 s to the middle .90 s, although individual scales have somewhat lower, albeit still adequate, coefficients (alphas in the high .60 s to the high .80 s).

The Delis-Rating of Executive Functions (D-REF) (Delis 2012) is a behavioral rating scale completed by parents to measure a child or teen’s behaviors related to executive function difficulties. The total composite executive functioning score reflects three scales: Behavioral Functioning, Emotional Functioning, and Executive Functioning (Delis et al. 2001a). There is evidence of high internal consistency for the three primary indexes and a composite scale, with test–retest reliability demonstrating moderate to high levels of consistency (α = .50−.80) in executive function measures over time (Delis et al. 2001b).

The Parenting Relationship Questionnaire (PRQ) (Kamphaus and Reynolds 2006) is a 45-item parent-report questionnaire designed to investigate family and parenting relationships. There are 7 scales: Attachment, Communication, Discipline Practices, Involvement, Parenting Confidence, Satisfaction with School, and Relational Frustration. The PRQ has demonstrated internal consistency (α = 82–.87) and test–retest reliability (α = .72−.81).

Data Analyses

Not all participants in BDP and HEP consented for their data to be used in research. Consequently, program-level attendance and fidelity data (which included all participants) were analyzed separately from demographic and pre/post-test data (which included only consenting participants), resulting in different sample sizes for certain measures. The analyses for program-level attendance and fidelity data were based on descriptive and frequency statistical methods (i.e., means and SDs).

Pre- and post-treatment outcomes were compared for BDP and HEP group using t tests for independent means on the aforementioned measures, which is appropriate for small sample sizes such as in the current study (de Winter 2013). Attrition was assumed for participants if there was no data at follow-up. Overall attrition was 18.7 % with specific attrition 19.5 % for the BDP (all outcome measures were missing for 8 out of 41 participants) and 17.6 % for the HEP (all outcome measures were missing for 6 out of 34 participants). Analyses indicated that demographic factors and baseline clinical variables did predict attrition in certain cases. In the BDP program, attrition was more likely for participants from lower income families t (23.24) = 3.95, p = .011. In the HEP program, teens with lower baseline self-report on the Emotional Symptoms Index scores on the BASC-2 SRP were more likely to exhibit attrition t (11.48) = 5.48, p = .019.

Results

Program Fidelity

Fidelity of program implementation was documented. Program attendance, an indicator of exposure, was deemed satisfactory for both BDP and HEP youth by the community-university partnership. The average number of days attended was 18.2 (76 %) for the BDP and 18.7 (78 %) for the HEP, out of a possible 24. The majority of BDP (73.2 %) and HEP (64.7 %) youth attended 70 % or more of programming days. We derived Adherence and Quality of Delivery fidelity indicators of program delivery for BDP and HEP from daily practitioner log checklists and intermittent live research technician observations (Table 4). Results show that both programs were delivered with acceptable to high levels of fidelity by staff. Agreement between raters was high for adherence ratings, and ratings of adherence tended to be somewhat lower for practitioner self-ratings than for observer ratings of practitioners, suggesting that practitioners may underestimate their adherence to manual procedures in retrospect. Adherence percentages for Parent Skills groups also tended to be slightly lower. These scores may have been decreased due to the fact that certain techniques (e.g., role plays) may not be indicated for some of the topics (e.g., determining a bed time) so practitioners did not utilize them. Quality of Delivery criteria were rated by the observers only.

Table 4.

Practitioner adherence ratings to manual procedures: self-report and technician observation

	BDP (% indicated)		(HEP % indicated)
	Daily self-report logs	Intermittent technician observations	Daily self-report logs	Intermittent techniciar observations
Child/Teen skills training
Adherence composite (follow manual procedures)	74	70	77	82
Quality of delivery composite (rapport, enthusiasm, preparedness)	NA	92	NA	85
Parent skills training
Adherence composite (follow manual procedures)	69	68	67	70
Quality of delivery composite (rapport, enthusiasm, preparedness)	NA	91	NA	98

60 % is minimally acceptable fidelity (Bloomquist et al. 2013; Botvin et al. 1990; Fagan and Mihalic 2003)

Weekly Progress Monitoring Ratings

Weekly progress monitoring ratings on the IRS were examined. Parent report of interactions and parent–child relationships significantly improved from Week 1 to Week 6 of programming for the BDP and HEP groups, respectively: t(17) = 3.86, p<.05; t(6) = 2.50, p<.05. In addition, both children’s and adolescents’ overall functioning significantly improved from Week 1 to Week 6 of programming, t(17) = 5.99, p<.05; t(6) = 3. 97, p<.05 (Fig. 1). Weekly progress monitoring ratings on the HSQ were also examined. Although the number of problem settings reported by parents did not significantly decrease from Week 1 to Week 6 of programming, t(15) = .91, p = .380 for BDP, the severity of reported problems did significantly decrease for the BDP group, t(15) = 3.68, p<.05. Similarly, although the number of problem settings reported did not significantly decrease for HEP, t(10) = .1.87, p = .092, the severity of reported problems did significantly decrease for HEP, t(10) = 3.90, p<.05 (Fig. 2). Reports of emotional difficulties on the SDQ did not significantly decrease from the first to the last week of programming for either the self-report t(12) = 1.39, p = .191 or the parent-report, t(12) = 1.55, p = .148 in the HEP group. However, parent reports of teen conduct problems within HEP did significantly decrease, t(12) = 2.81, p = .016, and there was a downward trend in self-reports of teen conduct problems, t(12) = 1.96, p = .074 (Fig. 3).

Fig. 1.

Impairment Rating Scale. This figure illustrates the weekly impairment trajectory of Behavior Development Program (BDP) and Healthy Emotions Program (HEP). Mean scores are depicted for youth’s interaction with parents as well as their overall level of functioning. It should be noted that BDP n’s ranged from 21 to 31 respondents across Weeks 1 through 6, while HEP n’s ranged from 14 to 27

Fig. 2.

Home Situations Questionnaire. This figure illustrates the mean number of problem settings and severity of presenting problem across Behavior Development Program (BDP) and Healthy Emotions Program (HEP). It should be noted that BDP n’s ranged from 18 to 32 respondents across Weeks 1 through 6, while HEP n’s ranged from 14 to 31

Fig. 3.

Strengths and Difficulties Questionnaire. This figure illustrates the mean rating of emotional and behavioral symptoms in the Healthy Emotions Program (HEP). Weekly n’s ranged from 19 to 26

Pre to Post Child and Family Outcomes

Results from analysis of the BASC-2 Parent and Self Report forms revealed that youth in both programs showed improvement across domains of functioning (Table 5). For the BDP, paired (dependent) samples t-tests examining pre-to post-programming changes yielded statistically significant effects in parent report of both the Aggression, t (21) = 2.87, p<.05, and Externalizing, t (21) = 2.90, p<.05, domains, as well as in child self-report of Internalizing, t (10) = 3.50, p<.05. For the HEP, statistically significant effects were found in parent report on all four domains of interest including Depression, t (25) = 3.40, p<.05, Internalizing, t (25) = 2.92, p<.05, Aggression, t (25) = 2.48, p<.05, and Externalizing, t (25) = 2.85, p<.05), as well as adolescent self-report of Internalizing, t (7) = 2.4, p<.05.

Table 5.

Pre–Post outcome measures for Behavior Development Program (BDP, n = 23^a) and Healthy Emotions Program (HEP, n = 27)

Source	Scale	BDP		HEP
		Mean (SD)
		Pre	Post	Pre	Post
BASC-2 youth report	Depression	55.73 (13.68)	49.36 (13.15)	62.90 (18.60)	55.50 (12.78)
	Internalizing	55.09 (9.73)	48.18* (9.96)	66.88 (12.83)	62.00* (11.39)
	Emotional Symptoms Index	55.36 (11.14)	50.36 (12.05)	71.63 (14.35)	65.63 (12.52)
BASC-2 parent report	Depression	73.32 (10.75)	65.50 (15.36)	72.35 (14.01)	64.85* (13.27)
	Internalizing	66.45 (14.85)	61.14 (14.81)	70.58 (13.42)	64.42* (10.96)
	Aggression	71.36 (7.08)	65.91* (11.01)	57.77 (12.89)	54.00* (10.59)
	Externalizing	75.14 (5.74)	68.95* (11.23)	63.85 (16.25)	58.50* (12.79)
D-REF parent report	Behavioral functioning	67.53 (7.10)	65.17 (7.75)	55.48 (10.32)	54.95 (9.55)
	Emotional functioning	69.80 (6.42)	64.50* (8.26)	60.19 (7.70)	58.86 (6.29)
	Executive functioning	67.13 (8.26)	63.27 (10.04)	62.19 (8.26)	59.71 (10.06)
	Total composite	69.87 (6.89)	65.90 (9.13)	60.19 (6.89)	58.52 (8.33)
PRQ parent report	Communication	38.26 (9.70)	40.44 (8.93)	45.14 (10.56)	46.24 (12.06)
	Attachment	36.78 (10.09)	43.26* (6.76)	46.29 (8.02)	48.43 (9.97)
	Relational Frustration	70.56 (10.43)	63.44* (10.26)	61.48 (13.46)	55.19* (10.15)

BASC-2 behavioral assessment system for children, 2nd edition; D-REF delis rating of executive functioning; PRQ parenting relationship questionnaire

^aNs lower than reported in demographics due to attrition * p<.05

On the D-REF, participants in the BDP demonstrated significant improvements from pre- to post-treatment on the Emotional Functioning scale, t(29) = 2.66, p<.05. However, no differences were identified in HEP (Table 5).

The PRQ indicated that parents perceived positive changes in their relationships with their child/adolescent (Table 5). Specifically, for the BDP there were significant differences in pre to post parent report of Attachment, t(26) = −3.43, p<.05 and Relational Frustration, t(26) = 3.16, p<.05, whereas in the HEP, Relational Frustration alone showed significant improvement, t(20) = 2.40, p<.05.

Discussion

The current initiative and program evaluation study is a demonstration of the research to practice process where a community-university partnership applied knowledge and procedures from EBPs for use within a practice context. We collaborated by integrating our knowledge of research, practice, and the unique characteristics of the clients being served (vis-à-vis, American Psychological Association 2006; Kazdin 2008; Mitchell 2011). From that foundation we created practice and research infrastructure to develop, fund (via billing insurance), implement, and evaluate two new models of service founded on EBPs within a practice setting.

The two new EBP service options addressed two very common presenting concerns in mental health centers including behavior problems in children and depression in adolescents. One of the service models was the ‘‘Behavior Development Program (BDP)’’ for children with behavior disorders. The BDP was adapted from the Early Risers program (August et al. 2006; Bloomquist et al. 2012, August et al. 2001), but was extended by incorporating other evidence-based child social-emotional skills with parent management training methods (informed by Barkley 2013; Dishion et al. 2012; Forgatch and Patterson 2010; Garland et al. 2008; Kazdin 2010; Larson and Lochman 2002; Sanders 1999; Webster-Stratton and Reid 2010). The other service model was the ‘‘Healthy Emotions Program (HEP)’’ for adolescents with depression. HEP was adapted from the Treatment of Adolescent Depression Study therapy protocol (TADS Team 2004, 2007), but was also extended by incorporating CBT strategies with parent facilitation (informed by Clarke et al. 1999; Cuijpers et al. 2009; Gillham et al. 2012; Hetrick et al. 2015; Rohde et al. 2005; Wells and Albano 2005). Both BDP and HEP were delivered within an intensive outpatient program to address the needs of youth with more severe problems. A rigorous training, technical assistance, fidelity monitoring, and outcome measurement strategy was used to promote the integrity and quality of services provided (informed by Bloomquist et al. 2013) and a preliminary program evaluation study was conducted.

The results showed that this research to practice demonstration effort succeeded on several counts. The resultant BDP and HEP service models were delivered with acceptable to high fidelity and both programs demonstrated positive effects on some targeted youth and parenting measures. Despite the small sample, multiple youth and parenting outcomes showed statistically significant positive effects both during programming and from pre to post assessments. Parents reported that the severity of targeted problems and domains of impairment across a range of situations and contexts decreased in both BDP and HEP. Parents also reported increased quality of relationships with their children or teens in both programs.

Although the results showed improvement as a whole, many youth still exhibited problems at post-test in both IOPs. Indeed clinical significance, where there is a statistically significant pre-post improvement coupled with a pre-post change from the clinical range (T score ≥ 60) to the normal range (T score ≤ 59) on a measure (Lambert and Ogles 2009) was not observed for BDP and was observed only twice for HEP (i.e., BASC-2 PRF Externalizing and PRQ Relational Frustration scores, see Table 5). Accordingly many youth required additional mental health care after programming and it was common at discharge for participants in BDP and HEP to be referred to follow-up outpatient mental health, school-based, or community-based services to continue treatment. This is not too surprising given that these IOPs typically serve youth with more severe symptoms.

Nevertheless, when examining progress monitoring and pre- to post-test data on the whole, a trajectory of improvement was shown related to participating in either BDP or HEP. In keeping with developmental theory, we hypothesize the youth and their parents, armed with the new skills learned and improved family relationships from these programs would be in position to continue making progress (e.g., Cummings et al. 2000). Continued improvement will be more likely when family’s access recommended follow-up mental health services.

Of course the program evaluation results have to be interpreted cautiously. Since we did not conduct a randomized trial with a comparison group, we cannot definitely attribute the findings to the programs. Furthermore, because there were missing post-test data on some subjects it is possible that the effects observed may be an artifact. Our goal, however, was not to validate psychotherapies via a rigorous trial, but rather bring EBPs into the practice context to potentially optimize the services provided, and our data speak to that primary objective.

Regarding bringing EBPs into practice, we have post hoc anecdotal information pertaining to sustainability of these IOP services after our study. The IOPs are now routine practice within PrairieCare. Data regarding the number of admissions to the IOPs in a recent year period revealed that the BDP had 103 and the HEP had 132. We have also heard from local community referral sources that the BDP and HEP are viewed as positive community assets to address local youth-related mental health needs.

This initiative and program evaluation adds to the accumulating literature in incorporating EBPs in children’s mental health. First, it is rare that EBP intervention protocols are deployed within real world settings with fidelity and outcomes systematically evaluated (Garland et al. 2014; Weisz et al. 2013). The current results show that when carefully planned and executed by a community-university partnership, EBPs can be infused in real world practice with fidelity, and that feasible yet meaningful evaluation can transpire. Second, there has been a call to deploy and evaluate psychotherapies in real world clinical practice settings with youth exhibiting complex clinical presentations (Weersing et al. 2006; Weisz et al. 2013). Accordingly, to meet the needs and goals of the clients and the PrairieCare organization respectively, the current project embedded EBPs into an innovative intensive outpatient program format, delivered services to cohorts of youth and families mostly using group delivery methods, and targeted a moderate to severe population. Finally, it is of interest that this real world deployment of EBPs served a broader net of youth than for what the programs were ostensibly designed. Although program enrollment criteria were specified for BDP and HEP, an examination of the subject characteristics in Table 1 shows there was considerable heterogeneity of participants in the programs. In particular nearly a quarter of the children enrolled in the BDP had a primary diagnosis of a depressive disorder. It appears as though not all youth enrolled in these programs met precise inclusion criteria, but nonetheless the programs met the needs of families and providers who referred them, which is informative about how EBPs may have broader application in practice.

EBPs are promising to the extent that the practice parameters within them can be replicated and evaluated for their effects in the real world practice context (Atkins et al. 2016; Jensen-Doss et al. 2009; Southam-Gerow et al. 2012). Additionally it is also widely accepted that EBPs used in practice must be conducted with fidelity to realize their full impact (Beidas and Kendall 2010; Schoenwald 2011; Schoenwald et al. 2009), and it may well be that supporting and achieving fidelity (synonymous with quality assurance) may be as important as the actual program itself (Carroll et al. 2010; Hershenberg et al. 2012). The present initiative and program evaluation study is another demonstration upon which the field of translation in children’s mental health can continue to build. This research to practice effort focusing on treatment of children and adolescents with externalizing and internalizing problems in a mental health organization warrants replication and extension.

Limitations

The program evaluation study itself has notable limitations. First, most of the measures were self-report which introduces possible expectancy effects. Second, attrition was moderate due to many participants not providing post-test data, though many of these participants did attend the program for the full six weeks. Third, and relatedly, sample sizes of individuals who had completed both pre- and post-test measures were limited, leading to decreased power to detect statistically significant changes. Fourth, the sample of participants was predominantly White and of middle class to affluent socioeconomic status thus potentially limiting generalizability to other populations.