Factors Associated with Use of Evidence-Based Practice Strategies in Usual Care Youth Psychotherapy

Abstract

The purpose of this study was to gain an understanding of how therapists providing usual care (UC) psychotherapy are using elements of treatment common to evidence-based practices (EBPs) for children with disruptive behavior disorders (DBPs) and to identify client and therapist characteristics that may be associated with EBP strategies directed toward children and those directed to their caregivers. Results indicate that certain child, family, and therapist characteristics are associated with use of EBP strategies; however, much of the variability in practice was not explained by the variables examined. These findings highlight the complexity of UC psychotherapy and provide directions for future research on implementation of EBPs in UC.

Children with disruptive behavior problems (DBPs), including oppositional, defiant, aggressive, and/or delinquent behavior, represent the vast majority of youths in the publicly-funded mental health system (Garland et al. 2001; Kazdin and Wassell 2000; Offord et al. 1991). Effective treatment for these youths is essential because children with these problems are at significantly elevated risk for a variety of maladaptive outcomes in adolescence and adulthood, including adult conduct problems, criminal behavior, and psychopathology (Copeland et al. 2007; Earls 1994).

A number of psychotherapeutic treatment models have demonstrated impressive efficacy for this patient population (Eyberg et al. 2008). In fact, much of the child/family intervention research has focused on DBPs and there are more evidence-based practices (EBPs) for DBPs than other childhood disorders (Chambless et al. 1998, 1996; Ollendick and King 2000; Weisz et al. 2006a). Further, several core elements common across efficacious treatment models for DBPs have been identified (Garland et al. 2008b). Common elements of EBPs for this patient population include therapeutic strategies directed toward children (e.g., affect/anger management, role-playing) and strategies directed toward caregivers in parent training models (e.g., principles of positive reinforcement and limit-setting). Although reports suggest that delivery of EBPs is rare in community-based settings (Hoagwood 2003), there is limited research characterizing usual care (UC) psychotherapy for children and families (Bickman 2000; Burns et al. 1999; Weisz et al. 2006b). What is known about UC is that average effectiveness is generally discouraging, reflecting minimal impact on patients’ symptom severity or functional status (Bickman 2000; Andrade et al. 2000; Bickman et al. 1999; Weiss et al. 1999; Weisz et al. 1995; 2006a, b).

Understanding the relative convergence and divergence between EBPs and UC could potentially provide a “road map” for targeted efforts to improve care. Therefore, our research team is pursuing a line of research aimed to characterize UC psychotherapy process and outcomes. The “PRAC” study (Practice and Research: Advancing Collaboration) is a longitudinal, observational study of psychotherapy with a representative sample of therapists, children, and their caregivers in one diverse county. Initial findings from this study related to observed psychotherapy process indicate that there is a great deal of variability in UC treatment for children with DBPs (Garland et al. under review). In-session psychotherapeutic care was observed to be eclectic, as therapists utilized many different therapeutic strategies from multiple theoretical orientations. On average, therapeutic strategies were observed at relatively low intensity. Some therapeutic strategies common in EBP models were observed frequently (e.g., delivering positive reinforcement, psychoeducation), whereas others were observed infrequently (e.g., assigning and reviewing homework).

Given the tremendous variability observed in treatment process, we are interested in examining patterns of practice and the extent to which delivery of treatment strategies common in EBPs is related to specific child, family, and therapist characteristics. Although there is minimal research identifying factors associated with therapeutic processes, there is research examining factors related to treatment process more broadly defined as treatment attendance/retention, therapist attitudes about EBP treatment strategies, and response to treatment. Therefore, we will summarize these areas to provide direction for potential predictors of EBP use in UC psychotherapy.

Therapist Characteristics

In studies of adult psychotherapy, “therapist effects” have been shown to account for modest to large proportion of variability in patient outcomes (Crits-Christoph et al. 1991; Kim et al. 2006). These findings are consistent with other large scale studies examining therapist factors on therapy outcomes for adult populations (Lutz et al. 2007; Mattson et al. 1998). Overall, therapist demographics are poor predictors of outcome with very small effect sizes with the exception of “ethnic match,” which has shown equivocal findings (for review of literature refer to Beutler et al. 1994, 2004). Therapist training, skill, experience and discipline, however, have been shown to have some meaningful effects on outcomes with effect sizes from .08 to .72 depending on the variable and study (Beutler et al. 1994, 2004; Huppert et al. 2001). Recent research addressing child psychotherapy has also found that there are variations in the effects of therapist factors depending on the specific therapeutic approach employed (i.e., cognitive behavioral, nondirective supportive) (Karver et al. 2008).

In a related area of research with child populations, associations between therapist characteristics and treatment process variables have been examined. In our own work examining UC treatment process, for example, therapist discipline was associated with number of treatment visits (Brookman-Frazee et al. 2008), therapists’ year of experience was related to client satisfaction (Garland et al. 2007), and therapist theoretical orientation was associated with client-therapist goal agreement (Garland et al. 2004). Although this research suggests that therapist characteristics may be associated with treatment process, there is no research examining how therapist characteristics are associated with variance in actual delivery of psychotherapeutic treatment strategies.

Research also suggests that therapist characteristics are associated with therapists’ attitudes towards EBPs. Specifically, studies have documented differences in attitudes towards EBPs by therapist level of experience (Aarons 2004) and theoretical orientation (Stewart and Chambless 2007). Although these studies do not measure actual use of EBP strategies, therapist attitudes may influence the treatment process, suggesting that these characteristics are potentially associated with delivery of EBP strategies.

Child Characteristics

Certain child characteristics have also been found to be associated treatment process variables and treatment outcomes. Child sociodemographcis, for example, have been associated with treatment attendance (Armbruster and Fallon 1994). A variety of child clinical characteristics (e.g., diagnosis, comorbidity, presenting problems, symptom severity) have been associated with the delivery of EBP treatment strategies (Jensen-Doss et al. 2007), treatment attendance and retention (Brookman-Frazee et al. 2008; Kazdin et al. 1994; Reyno and McGrath 2006) and response to treatment (Beauchaine et al. 2005; Reyno and McGrath 2006). Overall, the research suggests that child characteristics are important and related to treatment delivery and outcomes; however, the impacts of child characteristics have not been consistently found across studies (Eyberg et al. 2008; Shirk et al. 2008).

Family Characteristics

Although there is no research on how family characteristics may be associated with therapeutic practices such as the delivery of EBP strategies, there is a body of literature on how family characteristics are related to treatment process more broadly defined (e.g., treatment retention; treatment compliance) and treatment outcomes. Parent and family characteristics such as parental psychopathology (especially depression), substance abuse, marital conflict, domestic violence, contextual stress, and socioeconomic status have all been found to be associated with poor treatment compliance and retention (Cobham et al. 1998; Dadds and Mchugh 1992; Dumas and Wahler 1983; Kazdin 1995; Cohen 1977; Waldron et al. 2001; McMahon et al. 1981; Morrissey-Kane and Prinz 1999; Prinz and Miller 1996; Webster-Stratton 1985; Smolen and Lifton 1966; Joanning et al. 1991).

Family characteristics have also been found to moderate treatment outcomes in treatment studies for youth with disruptive behavior disorders (Miller and Prinz 1990; Kazdin 1995; Joanning et al. 1991). A meta-analysis found that low parental education, high maternal psychopathology, and low family socioeconomic status had moderate to large effects on outcomes (Patel et al. 2003). Another study combining six randomized controlled trials found that marital adjustment, maternal depression, and parental substance abuse moderated outcomes (Beauchaine et al. 2005). Additionally, poor marital and family functioning, high levels of parental stress, low treatment expectations, diverse culture and ethnicity, and limited social support have all been shown to influence treatment implementation and outcome (Chronis et al. 2004; Patterson and Chamberlain 1994; Singer et al. 1989; Wahler 1980; Weisz and Weiss 1991). Further, a study comparing a large community sample to samples in EBP studies found high occurrence rates for many parent and family factors (i.e., parent psychopathology, depression, stress, low social support, marital and family problems) in the usual care sample (Baker-Ericzén et al. 2009). These findings suggest that parent and family factors may also be associated with variation in the delivery of EBP strategies within UC settings.

Current Study

Overall, the above-mentioned research suggests that child and family characteristics and therapist characteristics may impact treatment process and outcome. There are no data available, however, on which particular characteristics are associated with use of strategies consistent with EBP treatment models in UC psychotherapy for children. Given that both the children and families served in these settings and the therapists who provide care are highly diverse, understanding how these factors impact treatment delivery offers important information for efforts to improve care.

The goals of this study are to gain an understanding of how UC therapists are using elements of treatment common to EBPs for children with DBPs and to identify patient and therapist characteristics that may be associated with practice patterns with children and with their caregivers. Videotapes for over 1,000 psychotherapy sessions with 191 children have been collected and coded as part of the PRAC study (Garland et al. under review). These data will be used to (1) examine variation in therapists’ delivery of EBP strategies, and (2) identify child, family, and therapist characteristics that may be associated with therapists’ use of EBP strategies.

Methods

Participants

Participating Clinics

The six participating clinics were selected because they represent the largest contractors for publicly-funded, clinic-based out-patient care for children in San Diego County. The clinics are geographically dispersed to maximize representativeness of urban, suburban, and semi-rural areas, as well as racial/ethnic diversity. All clinics serve patients with a wide range of diagnoses and presenting problems.

Therapist Participants

Therapists were randomly selected at the start of the study in late 2003 for recruitment from lists of active therapists. Recruitment proceeded until cells were filled to reflect the distribution of psychotherapists in publicly-funded care in the county by mental health discipline and proportional to the size of the clinic. In subsequent years of recruitment (2004–2006), new therapists were recruited sequentially as they joined the clinics. Of the 163 therapists recruited, 131 (80%) agreed to participate, but only 100 had a child patient participating in the study. Therapists who declined to participate did not differ significantly from participants on age, gender, or race/ethnic distribution, but licensed staff had a slightly lower participation rate (72%) compared to unlicensed staff/trainees (86%). Therapists received an honorarium ($100) for agreeing to participate in the study, regardless of the number of patients who entered the study.

Of the 100 participating therapists, 82 had at least one participating child with videotape data and were considered to be the “primary therapist” for that child (for children who had taped sessions with two or more therapists, the therapist with the most coded videotapes was determined to be the primary therapist; if multiple therapists had an equal number of coded videotapes, the therapist at baseline was considered the primary therapist). The 82 therapists included in the current analyses were primarily female (85.4%), and 65.9% were Caucasian. The range of years of experience was 0–25 years (M = 3 years). Consistent with national samples of therapists in community-based mental health care (e.g., Glisson et al. 2008) the therapists were primarily master’s level clinicians (61.0%), with 2.4% doctoral level and 36.6% Bachelors level. It is important to note that BA-level therapists in these settings are typically graduate students working towards MFT, MSW or Ph.D. degrees. Therapists came from different mental health disciplines: marriage and family therapy (57.3%), psychology (22.0%), and social work (20.7%).

Child Participants

Inclusion criteria for child participants were (a) presenting problems included a disruptive behavior problem (including aggression, defiance, delinquency, oppositional behavior), (b) age between 4 and 13 years at the time of recruitment, (c) primary language for child and parent was English or Spanish, and (d) child was entering a new episode of psychotherapy (defined as no therapy for previous 3 months) with a participating therapist. Clinic administrative staff screened all eligible new patients during the initial call to the clinic for services and obtained permission to share names and contact information with the research team for recruitment. Ten percent of parents declined to be contacted by research staff. Of the 550 who agreed to be contacted and met the inclusion criteria listed above, 55% (n = 292) did not engage in treatment at the clinics, leaving 258 potential participants who were actively recruited into this study. Eighty-five percent (n = 218) agreed to participate in the study. Due to HIPAA restrictions we could not collect data on non-participants, so no information about how non-participants may have differed from participants is available.

Informed written consent was provided by the parent and assent was provided by children ages 8 and older. Family participants were given financial incentives to participate in the study ($40 to the parent and $10 to the child at each interview), and families were assured that their decision regarding participation would not impact treatment. All protocols were approved by affiliated university, hospital, and county research review committees.

The 191 participating children, for whom observational data were available and included in the current analyses, were ages 4–13 years (M = 9.0, SD = 2.7), 32.5% were female and 49.7% were Caucasian, 28.3% Latino, 8.9% African American, 13.1% Mixed or other. Clinician-assigned primary diagnoses were: 38.7% ADHD, 20.4% disruptive behavior disorder, 23.6% mood disorder, 8.9% anxiety disorder, 6.3% autism spectrum disorder, 2.1% other.

Caregiver Participants

Caregivers were primarily female (93.7%), ranging in age from 22 to 69 years (M = 40.1, SD = 10.2). They were primarily biological mothers (77.0%), but also included grandmothers, biological fathers, and foster parents. Marital status was as follows: married or living with a partner (43.2%), divorced (35.3%), never married and single (18.9%), or widowed (2.6%). The percentage who spoke Spanish as their primary language was 16.2%. Mean household income was $36,255.77 (SD = $30,571.20), with the middle 50% of participants reporting income values between $15,600 and $48,000. Care for the majority of families was funded through government sources (72.5%); the remaining families were funded through the school system (27.5%).

Procedures and Data Collection

Child and Family Characteristics

The baseline interview was scheduled as part of the PRAC study after obtaining parental verbal consent to participate. This interview included in-person interviews with youths and caregivers, and occurred within 2 weeks of the first psychotherapy appointment to ensure that reports reflected the child/family’s condition close to the start of therapy. The baseline assessment began with the written informed consent protocol and took approximately 60 min for the caregiver (only caregiver-report data are used in the current study). Clinician-assigned child psychiatric diagnoses were collected from billing records. Psychotherapeutic treatment process data were collected by videotaping and coding sessions (as described below).

Therapist Characteristics

Descriptive data on therapists were based on self-report collected by interview following enrollment of their first patient for that therapist. Specific characteristics recorded included their age, gender, race/ethnicity, number of months practicing psychotherapy, primary theoretical orientation (family systems, behavioral, cognitive-behavioral, psychodynamic, humanistic, eclectic, or “other”), and mental health discipline (marital and family therapy, psychology, or social work).

Observational Measure

Adaptation of Therapeutic Process Observational Coding System for Child Psychotherapy: Strategies Scale

An adapted version of the Therapy Process Observational Coding System for Child Psychotherapy (TPOCS-S) (McLeod 2001; McLeod and Weisz 2005) was used to characterize treatment strategies. The TPOCS-S assesses for a wide variety of intervention strategies that are theoretically (e.g., cognitive-behavioral, psychodynamic, client-centered) and non-theoretically, or cross-theoretically, derived. The content is based primarily on the therapy procedures checklist (TPC) (Weersing et al. 2002) the format is based on the therapist behavior rating scale (TBRS) (Hogue et al. 1996) treatment adherence measure.

The TPOCS-S was adapted for this study in collaboration with the PRAC Therapist Advisory Group (TAG: includes one therapist representative from each of the six participating clinics, see Garland et al. 2006 for full description). The TAG reviewed all original TPOCS-S therapeutic strategy codes and worked with the research team to refine the measure to assess for therapeutic strategies used most commonly in their settings. The final revised PRAC TPOCS-S (Garland et al. 2008a) includes 27 therapeutic strategies, 15 of which reflect therapeutic techniques (e.g., modeling, addressing client-therapist relationship) and 12 of which reflect therapeutic content (e.g., affect management, principles of positive reinforcement). Use of each strategy was coded separately for strategies directed to children versus caregivers. Occurrence indicates whether the strategy was observed during a session. Intensity reflects both the time spent on the strategy and the thoroughness with which it was pursued. “Occurrence/intensity” was rated at the end of each session for each strategy on a Likert scale of 0–6 (0 = did not occur; 1–2 = low intensity, 3–4 = medium intensity, 5–6 = high intensity). For example, a low intensity rating on the strategy “problem-solving/social skills” would reflect a therapist addressing one aspect of problem-solving skills in a limited way, such as generating alternative solutions, but only for one particular experience the child or caregiver faced, and in a somewhat fleeting, or cursory manner. A high intensity rating would be assigned when the therapist thoroughly addresses the multiple steps in problem solving and generalization to multiple problems.

PRAC TPOCS-S EBP Composite Scores

For the current paper, we calculated two EBP composite scores (one for strategies directed to children and one for strategies directed to caregivers) to reflect the average occurrence/intensity rating of the PRAC-TPOCS designated as an EBP strategy. For an individual PRAC TPOCS-S code to be included in the EBP composite scores, it must have (1) been identified as a common element of EBPs in our iterative process of review and expert validation (Garland et al. 2008a, b) and (2) achieved a Kappa > .40 and an intraclass correlation (ICC) > .5 and occurred in more than 1% of sessions. Based on these criteria, 10 strategies were included in the Child EBP composite score and 9 strategies were included in the Caregiver EBP composite score (see Table 1 for specific strategies included in each composite score).

Table 1.

PRAC TPOCS-S Strategies Included in the Child and Caregiver EBP Composite Scores

	Child EBP composite	Caregiver EBP composite
Therapeutic content
Principles of positive reinforcement		X
Principles of effective limit-setting/punishment		X
Parent–child relationship building		X
Problem-solving skills	X	X
Affect/anger management	X	X
Affect education	X
Therapeutic techniques
Delivering positive reinforcement	X
Delivering punishment/limit-setting	X
Psychoeducation	X	X
Assigning & reviewing homework	X	X
Role-play/practicea	X
Modeling	X	X
Establishing and reviewing goals	X	X

^aAlthough role-play/practice is considered a common element of Caregiver EBPs (Garland et al. 2008a, b) the inter-rater reliability of this individual code did not reach the criteria for inclusion in our composite scores

Sampling of Sessions for Coding

With therapist and patient consent, all psychotherapy sessions occurring between the intake interview and 16-month follow-up were videotaped and digitized for efficient coding on a computer. A random sample of up to 10 sessions per child was selected for coding (four tapes within 0–4 month interval, three tapes at 5–8 months, two tapes at 9–12 months, and one tape at 13–16 months to provide the most data during the intervals in which most patients were actively attending). For the larger PRAC study, a total of 1,215 sessions were coded for 191 children and 96 therapists. In the current analyses, we included only sessions with the designated primary therapist since we were testing for therapist effects (see description of therapist participants above), resulting in a total of 1,077 coded sessions of 191 children and 82 therapists.

Coders and Coder Training

Seventeen research assistants served as coders, including three coders fluent in Spanish who coded all sessions in which Spanish was spoken. Coder training was conducted by three of the authors (AG, LBF, RZ) and consisted of review of a detailed training manual, four group and six individual training sessions, coding practice tapes, and ongoing “booster” sessions. All coders met 80% agreement (defined as within 1 point on the Likert scale of intensity) with “gold standard” codes on at least three consecutive training tapes before they coded sessions used for final data analyses.

Inter-Rater Reliability of PRAC TPOCS-S

Of the 1,215 total coded sessions in the PRAC study, 379 (31%) were randomly selected for double-coding to test inter-rater reliability. The ICC at the session level (i.e., across all PRAC TPOCS-S codes) was .78, and average ICC for individual codes was .61, reflecting acceptable reliability (Cichetti 1994) and similar to reliabilities reported in adult psychotherapy process observational research (Malik et al. 2003). For the current study, only EBP codes that achieved a Kappa > .40 and an ICC > .5 and occurred in more than 1% of sessions were included. The mean ICCs for the individual codes included in the Child EBP composite score was .68 and .67 for Caregiver EBP composite score.

Child and Family Baseline Measures

Eyberg Child Behavior Problems Checklist (ECBI: Eyberg and Pincus 1999)

The ECBI was used in this study to measure child symptom severity. The ECBI is a parent-report measure which has been used in many treatment outcome studies for youths with behavior problems, ranging in age from 2 to 16. The ECBI includes 36 items, which are rated on a dichotomous problem scale as well as a 7-point intensity scale. Only the intensity scale is used in the current study. The psychometric characteristics of the EBCI are strong. The intensity scale has demonstrated a 3-week test–retest reliability coefficient of .86 (Robinson et al. 1980), internal consistency coefficients of .98 (Robinson et al. 1980; Eyberg and Robinson 1983) and convergent validity demonstrated with significant correlation coefficients of .75 with the child behavior checklist externalizing problem score among clinic-referred children (Boggs et al. 1990). The established clinical cutoff for the intensity score is 132 (Eyberg and Pincus 1999).

Alcohol Use Disorders Identification Test (AUDIT: Saunders et al. 1993)

The AUDIT is a brief screening scale developed by the World Health Organization to identify individuals with alcohol problems (Saunders et al. 1993). The items distinguish between “hazardous” drinkers (those at risk for alcohol-related physical or psychological damage) and “harmful” drinkers (those already experiencing such problems) (Edwards et al. 1981). The 10-item scale assesses three conceptual domains: alcohol intake items, dependence, and adverse consequences. A review of the psychometric properties of the AUDIT indicate that it is internally consistent across diverse samples and in a broad range of settings, with a median reported Cronbach’s alpha above .80 (Reinert and Allen 2002). Barry and Fleming (1993) determined that the AUDIT is a reliable and valid measure for assessing current alcohol problems, with high internal reliability (α = .86) and a unitary factor structure.

Drug Abuse Screening Test (DAST-10: Skinner 1982)

The DAST is a 10-item instrument used to assess drug use and abuse, yielding an index of the consequences of abuse of drugs other than alcohol in the past year. The instrument defines drug abuse as the use of prescribed or over-the-counter drugs in excess of directions and the use of any non-medical drugs. The minimum score of 10 indicates no evidence of drug-related problems, and the maximum score of 20 indicates substantial problems. The DAST-10 has been shown to have concurrent and discriminant validity (Skinner 1982). The DAST-10 is internally consistent (α = .86) and can discriminate between psychiatric outpatients with and without current drug abuse and dependence (Graham et al. 1986; Stephens et al. 1994). It is considered a standardized instrument for clinical screening (French et al. 2001).

Center for Epidemiologic Studies-Depression Scale (CES-D: Radloff 1977)

The CES-D total score was used in this study to measure depressive symptomatology in caregivers. This self-report questionnaire asks respondents to rate how often they experienced 20 symptoms of depression in the past week along a 4-point scale. The CES-D has strong reliability (alphas range from .85 to .90) and validity for use with adults in the general population, including diverse racial/ethnic groups (Radloff 1977). For a 6-month time interval, test–retest reliability is .54.

Brief Symptom Inventory (BSI: Derogatis and Melisaratos 1983)

The BSI was used to measure caregiver level of psychological distress. Respondents rate 53 items on a 5-point scale indicating the degree to which they have been distressed by various psychiatric symptoms. The BSI yields 9 symptom dimensions (e.g. depression, anxiety, hostility) as well as a global severity index (GSI) which is a combined measure of the number of symptoms and the intensity of perceived distress. The GSI was used in the current study. The internal consistency of the BSI has been demonstrated with alphas ranging from 0.71 to 0.85 across the 9 dimensions (Derogatis and Melisaratos 1983). High convergence between the BSI scales and other common measures of psychopathology has been demonstrated (Derogatis and Melisaratos 1983).

Family Relationship Index (FRI: Holahan and Moos 1983)

The FRI is a 27-item (true/false) index derived from the family environment scale, and is used to assess the quality of family relationships. Three domains are assessed: cohesion (i.e., the degree of commitment and support family members provide for each other), expressiveness (i.e., the extent family members are encouraged to express their feelings directly), and conflict (i.e., the amount of openly expressed anger and conflict among family members). The FRI is based on both parent report. The FRI has demonstrated good construct validity (Hoge et al. 1989) and adequate internal consistency and test–retest reliability for the three subscale scores (Moos and Moos 1981). In this measure, higher scores represent better family functioning.

Family Empowerment Scale (FES: Koren et al. 1992)

The FES is used to measure the broad construct of parent empowerment. The FES is a 34-item instrument that provides three subscales of empowerment: family, service system, and community/political. The specific dimensions of empowerment measured in this scale include parents’ attitudes, knowledge, and behaviors regarding their children. The family subscale measures level of empowerment in the immediate situation at home. The service system subscale measures level of empowerment in the area of interacting with professionals who provide service to the parent’s own child. The community/political subscale measures empowerment related to parent’s advocacy for improved services for children in general, rather than specifically for his/her own child. The FES has alpha coefficients ranging from .87 to .88, test–retest Pearson correlations from .77 to .85, and an overall kappa coefficient of .77 (Koren et al. 1992).

Caregiver Strain Questionnaire (CSQ: Brannan et al. 1997)

The CSQ is a 21-item scale that measures the impact of caring for a child with emotional and behavioral problems in six areas: economic burden, impact on family relations, disruption of family activities, impact on psychological adjustment of family members, stigma, anger and worry/guilt. The CSQ yields 3 domains: objective strain, subjective externalized strain, and subjective internalized strain. A CSQ Global score, representing the mean of all CSQ items, was used to assess caregiver strain in the current study. In a community sample of children with mental health problems, the Cronbach’s alpha for the CSQ was .94 (McCabe et al. 2003). CSQ scores have correlated with parent impairment (.70 for CIS) and depression (.40 for CES-D), as well as youth impairment (.27 for CIS) and DISC-IV diagnosis (.50 for behavior disorders to .16 for mood and anxiety disorders).

Analysis Plan

SPSS (v. 14.0: Release 14.0.2, 2006) was used to calculate sample descriptives and average EBP composite scores at each level. STATA (StataCorp 2005) was used to calculate intraclass correlations to describe the proportion of variance in Child EBP and Caregiver EBP composite scores is attributable to the session, child/caregiver, and therapist levels. HLM 6.06 (Raudenbush et al. 2004) was used to account for the nested structure of the data. The HLM3 function was used to account for session, child/caregiver, and therapist levels in all regression models. Predictor variables were grouped into seven categories: child demographics, caregiver/family sociodemographics, child clinical characteristics, caregiver psychosocial functioning, family functioning, therapist level of experience, and therapist background. Separate HLM3 models were run for each group of predictors.

Results

Descriptives on all potential study predictor variables are presented in Table 2.

Table 2.

Sample descriptives on all study predictor variables

Measure	Mean (SD) or %	Range
Child demographics
Female gender	32.5%
Age	9.0 (2.7)	4–13
Racial/ethnic minority	50.3%
Caregiver/family sociodemographics
Age	40.1 (10.2)	22–69
Racial/ethnic minority	44.5%
Education level
Some high school	17.2%
Some College	66.7%
College/grad School	16.1%
Household annual income	$36,255.8 (30,571.2)	$60–250,000
Child clinical characteristics
Eyberg child behavior inventory problem intensity	146.7 (36.4)	59–237
Primary diagnosis: DBD	20.4%
ADHD	38.7%
Mood	23.6%
Anxiety	8.9%
Other	8.4%
Caregiver psychosocial functioning
Drug abuse screening test	.28 (1.0)	0–8
Alcohol use disorders identification test	2.1 (3.1)	0–21
Center for epidemiological studies-depression	15.4 (10.6)	0–45
Brief symptom inventory	57.1 (11.5)	33–80
Family functioning
Family relationship inventory	9.2 (4.5)	−5–17
Family empowerment scale-family	46.0 (6.8)	29–60
Family empowerment scale-community	27.6 (8.5)	11–50
Family empowerment scale-systems	51.4 (6.5)	28–60
Caregiver strain questionnaire	2.7 (0.8)	1.1–4.6
Therapist demographics
Female gender	85.4%
Age	32.0 (8.7)	23–56
Racial/ethnic minority	34.1%
Therapist level of experience
Months in practice	33.3 (42.7)	0–300
Staff position (v. Trainee)	41.5%
Licensed	13.4%
Therapist background
Discipline: marriage & family therapy	57.3%
Psychology	22.0%
Social work	20.7%
Orientation: cognitive/behavioral	28.0%
Family systems	36.6%
Psychodynamic/humanistic	8.5%
Eclectic/other	26.8%

Note: Sample size for each variable ranges from 136 to 191. Two reasons exist for the missing data. First, several caregiver psychosocial functioning measures (drug abuse screening test, alcohol use disorders identification test, Center for Epidemiological Studies-Depression and family functioning (family empowerment scale) were added as supplemental measures after baseline data collection had already begun. Second, a few participants (5 or less) did not complete all questions on self-report forms including socio-demographics and the Brief Symptom Inventory

EBP evidence-based practices, DBD disruptive behavior disorders (oppositional defiant disorder/conduct disorder), ADHD attention deficit/hyperactivity disorder

Delivery of EBP Strategies

Descriptives of the EBP composite scores for each level of the data structure (session, child/caregiver, therapist) and each target of intervention (child, caregiver) are reported in Table 3. Sample sizes for each level/target combination are also reported. The possible range was zero to six of the EBP composite scores. At all levels, the EBP composite scores were higher for children than for caregivers. For strategies directed to children, at least one EBP strategy was observed in over 99% of sessions for 100% of children by 100% therapists. For EBP delivered to caregivers, at least one EBP strategy was observed in 93% of sessions for 98% of caregivers by 99% of therapists.

Table 3.

Average child and caregiver composite scores

Level	Child EBP			Caregiver EBP
	Mean (SD)	Range	n	Mean (SD)	Range	n
Session	1.424 (.862)	0–5.1	1,077	1.007 (.788)	0–4.4	762
Child/caregiver	1.336 (.560)	.10–3.48	191	.976 (.559)	0–2.58	180
Therapist	1.349 (.554)	.25–3.23	82	.941 (.505)	0–2.33	81

Note: Possible EBP composite score range from 0 to 6

Variability in EBP Delivery Accounted for by Child, Family and Therapist Characteristics

The intraclass correlation coefficient (ICC) provides an estimate of variance for the two dependent variables, Child EBP and Caregiver EBP composite score, accounted for at each level of the data structure. ICCs are summarized in Table 4. All ICCs were above the conventional significance cutoff of .05, indicating that a significant proportion of variability of observed EBP delivery at each level was accounted for by child, family and therapist characteristics (Hox 2002). As the table indicates, the highest ICC (.52) was found when children/caregivers (Level 1) were nested within therapists (Level 2), indicating that 52% of the variance in EBP delivery was attributable to therapist differences.

Table 4.

Intraclass correlations for EBP composite scores at session, child/caregiver, and therapist levels

Level 1	Level 2	Child EBP	Caregiver EBP
Session	Therapist	0.27388	0.14496
Session	Child/caregiver	0.31838	0.24260
Child/caregiver	Therapist	0.52092	0.28555

Predictors of Child EBP

Results of hierarchical linear models predicting Child EBP composite scores are shown in Table 5. Seven models were run (one for each category of predictor variables). Significant child demographic, caregiver/family sociodemographic, and therapist background predictors were found. Specifically, results indicated that child age was positively and significantly associated with Child EBP (B = .029; P = .024), such that older children had higher Child EBP composites scores than younger children. Caregiver education was also significantly associated with Child EBP such that children whose caregivers had some college had higher Child EBP composite scores than children whose caregivers had some high school (B = .259; P = .004). Household income was marginally associated with Child EBP such that higher income families had higher Child EBP composite scores (B = .000; P = .060). Caregiver alcohol use was also significantly associated with Child EBP, such that the more alcohol use problems a caregiver reported, the higher the Child EBP score (B = .022; P = .033). Lastly, therapist self-reported primary orientation was significantly associated with Child EBP composite scores (B = −.335; P = .035). Specifically, children whose therapists self-identified as cognitive-behavioral or behavioral orientations had higher Child EBP score than children whose therapists self-identified as eclectic or other orientations.

Table 5.

Hierarchical linear models predicting child EBP composite scores

Predictor variable	Regression coefficient (standard error)	P-value
Child demographics (n = 1,077 for level 1; n = 191 for level 2; n = 82 for level 3)
Gendera	.096 (.080)	.230
Age	.029 (.013)	.024
Race/ethnicityb	−.002 (.070)	.981
Caregiver/family sociodemographics (n = 1,054 for level 1; n = 185 for level 2; n = 78 for level 3)
Education (reference group = some high school)
Some college	.226 (.086)	.010
College/grad school	.110 (.101)	.275
Income	.000 (.000)	.060
Child clinical characteristics (n = 1077 for level 1; n = 191 for level 2; n = 82 for level 3)
Eyberg child behavior inventory	−.001 (.001)	.338
Primary diagnosis (reference group = DBD)
ADHD	−.029 (.116)	.805
Mood	.001 (.126)	.993
Anxiety	−.076 (.152)	.616
Other	.084 (.173)	.605
Caregiver psychosocial functioning (n = 767 for level 1; n = 134 for level 2; n = 57 for level 3)
Drug abuse screening test	−.067 (.041)	.107
Alcohol use disorders identification test	.022 (.010)	.033
Center for epidemiological studies-depression	−.001 (.006)	.845
Brief symptom inventory	−.004 (.007)	.594
Family functioning (n = 882 for level 1; n = 154 for level 2; n = 64 for level 3)
Family relationship inventory	.005 (.010)	.640
Family empowerment scale-family	.005 (.008)	.516
Family empowerment scale-community	−.007 (.008)	.390
Family empowerment scale-systems	−.007 (.006)	.215
Caregiver strain questionnaire	.019 (.047)	.689
Therapist demographics (n = 1,077 for level 1; n = 191 for level 2; n = 82 for level 3)
Gendera	.094 (.132)	.479
Age	−.005 (.006)	.440
Raceb	−.081 (.139)	.563
Therapist level of experience (n = 1,077 for level 1; n = 191 for level 2; n = 82 for level 3)
Months in practice	.001 (.001)	.587
Positionc	.003 (.134)	.998
Licensured	−.196 (.384)	.384
Therapist background (n = 1,077 for level 1; n = 191 for level 2; n = 82 for level 3)
Discipline (reference group = MFT)
Psychology	.015 (.178)	.934
Social work	−.137 (.173)	.432
Orientation (reference group = CBT)
Family systems	−.194 (.170)	.260
Psychodynamic/humanistic	−.332 (.228)	.149
Eclectic/other	−.335 (.157)	.035

Note: Robust standard error solution is reported, which controls for non-normality in the variables. Each model was run with HLM3 with tape as level one, child/caregiver as level two, and therapist as level three. Regression coefficients reported are unstandardized

DBD disruptive behavior disorders (oppositional defiant disorder/conduct disorder), ADHD attention deficit/hyperactivity disorder, MFT marriage and family therapy, CBT cognitive/behavioral or behavioral

^a0 = Male; 1 = Female

^b0 = White; 1 = Nonwhite

^c0 = Trainee; 1 = Staff

^d0 = Unlicensed; 1 = Licensed

Predictors of Caregiver EBP

Results of hierarchical linear models predicting Caregiver EBP are shown in Table 6. Again, seven models were run (one for each category of predictor variables). No predictor variables were significantly associated with Caregiver EBP composite scores. However, some caregiver sociodemographics, child clinical, and therapist experience characteristics were marginally associated with Caregiver EBP. Specifically, child symptom severity, as measured by the ECBI intensity score, was marginally and positively associated with Caregiver EBP (B = .002; P = .055). Also, caregivers of children with higher ECBI intensity scores had higher Caregiver EBP scores. Lastly, therapist months practiced was marginally, but negatively associated with Caregiver EBP (B = −.002; P = .054), such that therapists with fewer months in practice had higher Caregiver EBP scores.

Table 6.

Hierarchical linear models predicting caregiver EBP composite scores

Predictor variable	Regression coefficient (standard error)	P-value
Child demographics (n = 762 for level 1; n = 180 for level 2; n = 81 for level 3)
Gendera	.053 (.090)	.556
Age	−.023 (.015)	.136
Caregiver/family sociodemographics (n = 741 for level 1; n = 173 for level 2; n = 77 for level 3)
Age	.000 (.003)	.964
Race/ethnicityb	.036 (.089)	.682
Education (reference group = some high school)
Some college	−.059 (.111)	.596
College/grad school	.135 (.134)	.317
Income	−.000 (.000)	.143
Child clinical characteristics (n = 762 for level 1; n = 180 for level 2; n = 81 for level 3)
Eyberg child behavior inventory	.002 (.001)	.053
Primary diagnosis (reference group = DBD)
ADHD	−.119 (.112)	.292
Mood	.004 (.117)	.975
Anxiety	−.171 (.164)	.299
Other	−.014 (.169)	.933
Caregiver psychosocial functioning (n = 534 for level 1; n = 124 for level 2; n = 54 for level 3)
Drug abuse screening test	−.003 (.053)	.949
Alcohol use disorders identification test	.002 (.012)	.854
Center for epidemiological studies-depression	−.006 (.005)	.238
Brief symptom inventory	−.001 (.005)	.867
Family functioning (n = 624 for level 1; n = 143 for level 2; n = 63 for level 3)
Family relationship inventory	.005 (.010)	.587
Family empowerment scale-family	.004 (.008)	.595
Family empowerment scale-community	−.002 (.007)	.795
Family empowerment scale-systems	−.010 (.006)	.105
Caregiver strain questionnaire	.073 (.050)	.151
Therapist demographics (n = 762 for level 1; n = 180 for level 2; n = 81 for level 3)
Gendera	.006 (.005)	.759
Age	.111 (.125)	.379
Raceb	−.122 (.099)	.222
Therapist level of experience (n = 762 for level 1; n = 180 for level 2; n = 81 for level 3)
Months in practice	−.002 (.001)	.054
Positionc	.071 (.120)	.556
Licensured	.168 (.127)	.191
Therapist background (n = 762 for level 1; n = 180 for level 2; n = 81 for level 3)
Discipline (reference group = MFT)
Psychology	.026 (.132)	.846
Social work	−.171 (.114)	.136
Orientation (reference group = CBT)
Family systems	−.061 (.123)	.624
Psychodynamic/humanistic	.178 (.116)	.129
Eclectic/other	.188 (.136)	.170

Note: Robust standard error solution is reported, which controls for non-normality in the variables. Each model was run with HLM3 with tape as level one, child/caregiver as level two, and therapist as level three. Regression coefficients reported are unstandardized

DBD disruptive behavior disorders (oppositional defiant disorder/conduct disorder), ADHD attention deficit/hyperactivity disorder, MFT marriage and family therapy, CBT cognitive/behavioral or behavioral

^a0 = Male; 1 = Female

^b0 = White; 1 = Nonwhite

^c0 = Trainee; 1 = Staff

^d0 = Unlicensed; 1 = Licensed

Discussion

This study used multi-level modeling to identify child, family, and therapist characteristics associated with observed use of psychotherapeutic strategies common in evidence-based practices for children with disruptive behavior problems. The results indicate that the overall intensity of observed delivery of elements of EBP was relatively low for strategies directed towards both children and caregivers. Despite the significant proportions of variation in EBP delivery globally accounted for by child/caregiver and therapist characteristics, few of the specific characteristics measured in this study were significantly associated with EBP delivery. Great Child EBP was associated with older child age, higher caregiver educational level, greater caregiver alcohol use, and having a therapist with a self-reported cognitive-behavioral or behavioral primary theoretical orientation (compared to “eclectic/other”). Although no child, family, or therapist characteristics were significantly associated with Caregiver EBP, certain child symptom, family sociodemographics, and therapist experience characteristics were marginally associated with EBP delivered to caregivers.

Although EBP elements were observed in almost all sessions for almost all children/caregivers and by almost all therapists, overall intensity of EBP elements was relatively low. On a scale with a possible range from 0 to 6, the average EBP composite score was approximately one for use with children and caregivers. This finding is consistent with previous analyses of these data (Garland et al. under review) using different metrics to measure EBP delivery. Specifically, in our previous analyses, we examined the frequency of occurrence and intensity (if observed) separately for individual EBP elements, whereas in the current study we examined, overall, to what degree EBP strategies were observed within a session, child/caregiver, and therapist. In our previous analyses, average intensity when the individual strategies were used was relatively low. Regardless of the method we used to characterize EBP strategies, observed delivery of EBP strategies could be characterized as lacking depth.

The only significant child characteristic associated with EBP delivery was child age. The finding that older children had higher Child EBP scores is consistent with the targeted age groups of evidence-based youth skills training interventions (Eyberg et al. 2008). It is also consistent with our own research indicating that therapists value EBP elements more with older than younger children (Brookman-Frazee et al. 2009). Child symptom severity was also a marginally significant predictor of Caregiver EBP. Specifically, caregivers of children with higher ECBI intensity scores had higher Caregiver EBP composite scores. It may be that therapists perceive more severe problem behavior as requiring more intensive parent training (i.e., Caregiver EBP).

Socioeconomic status indicators did predict Child EBP composite scores. Specifically, caregiver level of education was positively associated with Child EBP and annual household income was marginally positively associated with Child EBP. This may be explained by the demand characteristics of the family. Although speculative, it may be that more highly educated families expect more active (consistent with elements of EBP) treatment for their children.

The only caregiver psychosocial characteristic associated with EBP delivery was alcohol use. The finding that caregiver alcohol use was marginally associated with greater Child EBP may be explained by therapists focusing more intensively on child skills than parent skills in therapy if the caregiver is seen as less engaged in treatment due to their own psychosocial needs.

Certain therapist characteristics were associated with greater EBP delivery. Therapists who self-identified as cognitive-behavioral or behavioral had higher Child EBP composite scores than therapists who identified as eclectic (or “other”) orientations. This finding is consistent with the fact that most of the EBPs for youths with disruptive behavior problems are behavioral or cognitive behavioral (Eyberg et al. 2008). Less experienced therapists (defined by fewer months practiced) had marginally higher Caregiver EBP than more experienced therapists. This is consistent with research indicating that less experienced therapists may hold more positive attitudes towards EBPs (Aarons 2004).

Although there were a few important factors identified as associated with observed delivery of EBP, many factors that previous research suggests may be important, or might be assumed to be related to treatment delivery (e.g., client diagnosis, therapist discipline) were not significantly associated with observed use of EBP. Given that significant proportions in variability of EBP composite scores were associated with child/family and therapist differences, the few significant characteristics (particularly with Caregiver EBP) is striking and indicates that much of the variability in treatment process is explained by characteristics not examined. Key factors that we did not measure include therapist training in EBP models, type and amount of supervision (many of the therapists were unlicensed and were therefore receiving supervision), and previous experience treating children with disruptive behavior problems. Further, we did not include the child’s auxiliary treatment. It may be that receiving other services (e.g., in home behavior support) or being placed in a higher level of care during the course of outpatient treatment impacts the treatment process. Further, we did not include a measure of caregiver-therapist or child-therapist therapeutic alliance, which may be associated with the strategies that therapists employ. Alternatively, the lack of findings may be related to the nature of the data, given the EBP composite scores were positive skewed. Further, the lack of significant associations with EBP strategies directed to caregivers may be due to less power given the reduced sample size in those models. Unfortunately, we cannot determine whether power was an issue given the lack of current technology to conduct three-level power analyses, based on the sample sizes in the Caregiver EBP analyses. Based on our sample sizes in caregiver analyses (81 therapists, 180 caregivers, 762 sessions), however, we would expect to have sufficient power to detect moderate effects in analyses of between-group differences, suggesting that this may not be a significant issue.

One of the primary strengths of this observational study is the large, representative sample of therapists, patients, and psychotherapy sessions. The distributions of clinician education level, gender, race/ethnicity, and trainee status is similar to other studies of community-based mental health providers (Glisson et al. 2008; Hawley and Weisz 2005). Selection bias was minimized by initially recruiting therapists by random selection and then sequentially as they entered the clinic. The patient sample is also comparable to other clinical samples of children in publicly-funded mental health care. Specifically, the male to female ratio, over-representation of race/ethnic minority children and diagnostic distribution are consistent with other studies (Bickman et al. 1995; Eyberg et al. 2008; Foster et al. 2001; Rosenblatt and Rosenblatt 2000; Zima et al. 2005).

Another important strength of this study was our method of characterizing UC psychotherapy. That is, observational data on psychotherapy treatment processes provide the richest information to date on what actually happens in usual care. Despite this strength, the resulting measure only assesses observable therapist behavior. We did not capture therapists’ intentions, goals, or decision-making processes, patients’ responses to different intervention strategies, or communication outside the office or via telephone. Further, although we attempted to minimize the effect of observation by establishing video-taping as a routine practice in the clinics and using small, unobtrusive cameras mounted in the upper corners of therapists’ offices, we do not know how videotaping psychotherapy sessions may have influenced participants’ behavior (see Garland et al. 2009). It is important to note that the PRAC TPOCS-S was not intended to be used by usual care providers to assess practice. Rather, it was developed for use in research studies aimed to rigorously characterize usual care practice. See Garland et al. (2009) for a discussion of methodological challenges to characterizing usual care practice.

In the current study, we differentiated strategies based on whether they had been identified as a common treatment element in EBPs for children with disruptive behavior problems, and conducted analyses to identify characteristics associated with a composite of EBP elements. We designated treatment strategies as “common in EBPs” based on one method of identifying these elements (Garland et al. 2008a, b). Other methods might yield additional or different common elements of EBPs.

Despite these limitations, the results of this study have important implications for the implementation of evidence-based interventions in UC. In particular, these findings highlight the complexity of UC psychotherapy, such that there are no clear patterns of characteristics associated with delivery of EBP elements. Given the generally low intensity of observed elements of EBP, there is a need for training interventions to strengthen therapists’ use of these treatment elements, particularly the individual strategies that are infrequently observed to occur or are observed with low intensity (Garland et al. 2009). These observational data can provide a “road map” for individual EBP treatment elements that require particular attention in therapist training intervention. In addition, the findings suggest that we can not make assumptions about who is delivering what type of service to whom. However, the results can provide some initial direction for efforts aimed at implementing EBP treatment models in usual care settings. For example, extra attention may be paid to encouraging EBP delivery with children in lower socioeconomic families compared to higher socioeconomic families based on the results. Next steps in this line of research include examination of outcome trajectories and analyses to determine how specific therapist strategies may be associated with different child or family outcomes. We will also conduct detailed analyses of the associations between child, family and therapists characteristics and treatment outcomes.