Therapists' Attitudes Towards Psychotherapeutic Strategies in Community-Based Psychotherapy with Children with Disruptive Behavior Problems

Abstract

Little is known about what individual treatment strategies therapists providing usual care psychotherapy consider the most valuable to their practice. The Therapeutic Strategies Survey (TSS) assesses therapists' attitudes about the value of 27 individual treatment strategies in their practice with children with disruptive behavior problems in community-based outpatient psychotherapy. Findings indicate that therapists from multiple professional disciplines highly value many individual psychotherapeutic strategies, and consider strategies common to a majority of evidence-based practices (EBPs) for this population at least as important as strategies not emphasized in EBPs. Implications for developing therapist training and implementation of EBPs are discussed.

Introduction

Researchers and policy makers lament the oft-cited gap between evidence-based approaches to treatment and usual care treatment in community-based settings (Perkins et al. 2007; Weisz et al. 2006; National Advisory Mental Health Council 2001). Outcome studies indicate that usual care treatment for children, on average, has a weaker impact on clinical outcomes for numerous psychiatric disorders or problems than empirically supported treatments (Weisz et al. 2006). While a number of explanations for this discrepancy have been proposed, including differences in service context, provider characteristics, and client characteristics, one natural conclusion is that the treatment processes in community-based treatment differ significantly from treatment processes common in evidence-based practices (EBPs). Unfortunately however, given the limited research characterizing usual care treatment (Bickman 2000), this conclusion is based largely on speculation.

Available research on usual care therapists' practice suggests that there is wide variability in approaches to psychosocial treatment across therapists. Some studies suggest that many therapists prefer eclectic, flexible approaches to psychotherapy, incorporating strategies drawn from multiple theoretical orientations (Baumann et al. 2006; Jensen et al. 1990). Studies also indicate that usual care therapists do not necessarily embrace the utilization of highly structured, empirically-supported treatment manuals, judging them to be too rigid (Addis and Krasnow 2000; Essock et al. 2003). These attitudes have been cited as a barrier to the implementation of EBPs in community care. Gaining greater understanding of usual care therapists' attitudes about and preferences for treatment, and the extent to which these preferences are consistent and inconsistent with elements of EBPs for particular patient populations, is essential for informing efforts to implement evidence-based practices in usual care settings (Aarons 2004; Baumann et al. 2006; Garland et al. 2006).

Theoretical models of individual behavior change such as the Theory of Planned Behavior and Theory of Reasoned Action suggest that attitudes about the expected value of a behavior is related to actual behavior (Perkins et al. 2007). Research on health care professionals' behavior supports this. For example, physicians' attitudes about specific care improvement strategies are highly significant in predicting the success of care improvement implementation efforts (Van Someren 1998). Recent research has examined methods to change mental health therapists' practice. Casper (2007) demonstrated that a training program incorporating direct assessment of, and attention to, therapists' attitudes and intentions regarding practice innovation had a much greater impact on reported practice than did a traditional knowledge dissemination training program. Anecdotal reports from researchers who have attempted to implement EBPs in community service settings support these findings, emphasizing the need to attend to therapists' attitudes and values as significant potential facilitators or inhibitors in the implementation process (Shirk 2004; Weisz et al. 2004). Unfortunately, there is minimal research documenting common attitudes about psychotherapeutic treatment strategy preferences among representative samples of therapists working in community-based settings.

There are several informative studies examining psychotherapists' attitudes towards empirically-supported treatment manuals and EBPs (Aarons 2004; Addis and Krasnow 2000; Baumann et al. 2006; Essock et al. 2003; Rubin and Parrish 2007; Sheehan et al. 2007), as well as studies of psychotherapists' preferred treatment approaches and decision-making more generally (Kazdin et al. 1990; Koocher and Pedulla 1977; Stewart and Chambless 2007; Weersing et al. 2002). Results of studies of psychotherapists' attitudes towards EBPs and manualized treatment generally report ambivalent attitudes. For example, while many therapists endorse the use of research to inform practice (e.g., Addis and Krasnow 2000; Kazdin et al. 1990; Stewart and Chambless 2007), they may lack a clear understanding of the definition of EBPs and/or empirically supported treatment manuals (Aarons 2004; Addis and Krasnow 2000; Baumann et al. 2006). In addition, most studies using quantitative and qualitative methods reveal that therapists have negative attitudes towards evidence-based treatment and manualized approaches, including perceptions of a lack of flexibility, inhibition of clinical skill and autonomy, and lack of “fit” with complex, diverse, community patient samples (Addis and Krasnow 2000; Baumann et al. 2006; Essock et al. 2003; Nelson et al. 2006).

Efforts to identify therapist characteristics associated with attitudes toward EBPs and the value of empirically-derived knowledge to inform practice, not surprisingly, suggest that therapists (psychologists specifically) with more research training and more recent training have more favorable attitudes towards EBPs (Addis and Krasnow 2000; Stewart and Chambless 2007). Further, one study that included providers from multiple disciplines found that interns, as opposed to more experienced staff members, endorsed more positive attitudes about EBPs (Aarons 2004). Few studies have examined attitudes across different mental health disciplines, but one study of psychologists did find associations between reported primary theoretical orientation and attitudes about treatment research (Stewart and Chambless 2007). Specifically, self-reported cognitive-behavioral therapists reported significantly more positive attitudes towards treatment research than did self-reported eclectic or dynamic psychotherapists. While significantly less positive than cognitive-behaviorally identified therapists, Eclectic therapists were significantly more positive than Dynamic therapists (Stewart and Chambless 2007). Other therapist demographic characteristics (e.g., gender, race/ethnicity) have been shown to be unrelated to attitudes about EBP (Addis and Krasnow 2000; Sheehan et al. 2007; Kazdin et al. 1990; Stewart and Chambless 2007).

A related area of research examines therapists' self-report of strategies frequently used with children and families. Results of therapist surveys generally reflect eclecticism in treatment approaches, with psychotherapists often explicitly labeling themselves as “eclectic” or endorsing a variety of treatment approaches derived from different theoretical orientations (Baumann et al. 2006; Kazdin et al. 1990; Koocher and Pedulla 1977; Weersing et al. 2002). While many of these studies assess psychotherapists' reported practices at a relatively broad level of analysis (e.g., theoretical orientation), Weersing et al. (2002) describe the detailed development of the Therapy Process Checklist (TPC), assessing therapists' use of specific therapeutic techniques across multiple theoretical orientations. In this study, therapists reported increased use of behavioral techniques with younger children who have externalizing problems. The TPC has also been adapted for use in other studies (e.g., Baumann et al. 2006) and served as the basis for a comprehensive observational coding system designed to assess therapist use of treatment strategies in psychotherapy with children (McLeod 2005). Baumann et al. (2006), for example, found that therapists reported using family therapy techniques more often than any other type of technique when treating cases of physical abuse.

There are numerous methodological limitations in this burgeoning research area addressing therapists' attitudes and preferences in treatment. Many of the survey studies rely on relatively low response rates, raising questions about the representativeness of the participants' perspectives. For example, all of the studies referenced above report response rates of 56% or lower, except Aarons' study (2004). Several studies report response rates of less than 35% (Addis and Krasnow 2000; Rubin and Parrish 2007; Stewart and Chambless 2007; Weersing et al. 2002). Some of these studies do report on impressively large samples (e.g., Addis and Krasnow 2000: n = 891; Sheehan et al. 2007: n = 446; Stewart and Chambless 2007: n = 591), but, as the authors note, the extent to which the respondents' attitudes are representative of the larger population of therapists is questionable. Similarly, with the exception of studies by Aarons (2004), Baumann et al. (2006), Sheehan et al. (2007) most research in this area reports therapist attitudes for only one professional discipline. Specifically, these studies include samples of exclusively psychologists (Stewart and Chambless 2007; Addis and Krasnow 2000), psychologists and psychiatrists (Koocher and Pedulla 1977), and social work faculty (Rubin and Parrish 2007), as opposed to a representative sample of therapists from a variety of disciplinary backgrounds. Doctoral level mental health professionals (psychiatrists, psychologists, or doctoral level social workers) represent a small minority of the providers of publicly funded psychotherapy (Aarons 2004; Baumann et al. 2006; Glisson et al. 2008; Nelson et al. 2006); therefore, their attitudes about practice may not be representative of the majority of therapists providing usual care.

Finally, there is much variation in approaches to measure therapists' attitudes. One measure of therapists' general attitudes about EBPs (EBPAS; Aarons 2004) and one measure of therapists' preferred treatment strategies (TPC; Weersing et al. 2002) have relatively strong psychometric data to support their reliability and validity. However, most of the studies of therapists' attitudes towards evidence-based treatment strategies and more general surveys about preferred treatment approaches utilize measures with little to no psychometric testing. Most measures have been created for the purposes of the study, and no information about internal reliability, test-retest reliability or validity is provided (e.g., Addis and Krasnow 2000; Kazdin et al. 1990; Rubin and Parrish 2007; Sheehan et al. 2007; Stewart and Chambless 2007).

This study aims to address some of the limitations of previous research on assessing community-based therapists' attitudes towards particular treatment strategies within the context of treatment of children with disruptive behavior problems. We chose to focus on the therapists' general treatment of children with disruptive behavior problems because these children represent the vast majority of the patient population in community-based, out-patient services (i.e., therapists are likely to serve a number of children with disruptive behavior problems). For example, in a study of 874 children in publicly funded care, the prevalence of disruptive behavior disorders, assessed using a DSM-IV-based structured interview, was five times that of anxiety disorders and seven times that of mood disorders (Garland et al. 2001). Early onset of these problems increases the risk of more severe and persistent lifetime problems (Loeber 1990). Further, common elements of EBPs have been identified for this patient population, distilled from multiple, empirically supported, individual treatment models (Garland et al. 2008), making it possible to assess therapists' attitudes towards strategies that are common across individual, empirically supported treatment protocols.

The primary aims of this study were to (a) identify the therapeutic strategies therapists consider most valuable (i.e., important, most effective) in outpatient psychotherapy with children with disruptive behavior problems, (b) compare the rates of endorsement of elements of EBPs compared to all other strategies, and (c) examine how therapist characteristics might be associated with their attitudes towards strategies common in EBPs for this population. To address some limitations of previous research, this study includes a diverse sample of therapists from multiple mental health disciplines, representative of therapists in publicly-funded outpatient clinics serving racially/ethnically and diagnostically diverse children and families. This study aims to characterize community-based psychotherapy practice beyond broad categories of theoretical orientation or general attitudes towards treatment manuals. The focus is on pragmatic, operational aspects of treatment, assessing therapists' attitudes about the array of therapeutic strategies. These data were collected in the context of a large, prospective observational study characterizing psychotherapy process and outcomes for children in publicly-funded, “usual care” psychotherapy for whom at least one presenting problem is a disruptive behavior problem (“Practice and Research: Advancing Collaboration;” PRAC).

Methods

Participants

Survey respondents were 88 therapists practicing in six community-based outpatient mental health clinics primarily or exclusively serving publicly-funded children and adolescents in San Diego County. These clinics represent two larger agencies in our community (three clinics in each agency). These clinics do not specialize in one particular psychiatric disorder, but rather, serve children with a variety of psychiatric problems (Garland et al. 2001). The voluntary survey was administered to all therapists present at regularly attended staff meetings between January, 2007 and July, 2007 by research staff. Approximately 91% of all staff (including employees and trainees/interns) employed at the six clinics completed the survey.

Therapist participants ranged in age from 23 to 64 years old, with an average age of 36. Seventy-five percent of the therapists were female. Therapists had a range of 0 (less than one year) to 29 years of clinical practice, with a mean of 4.8 years. All were currently seeing multiple patients and had been providing care for at least six months when the survey was administered. The sample was ethnically diverse: 64% Caucasian, 18% Hispanic/Latino, 3% African American, 6% Asian, 1% Filipino/Pacific Islander, 4% Other and 3% Mixed. Of the 88 therapists, 64% self-identified as staff and 36% as trainees. In terms of education, 14% of the sample had a BA degree, 74% had a MA degree, and 13% had a Ph.D./M.D. Therapists in this sample varied by discipline and by theoretical orientation. Fifty-three percent of the sample was of the Marriage and Family Therapy (MFT) discipline, 21% Social Work, 17% Psychology, 8% Psychiatry, and 1% Other. Therapists self-identified their primary theoretical orientation as 36% Cognitive Behavioral/Behavioral, 22% Eclectic, 21% Family Systems, 3% Humanistic, 6% Psychodynamic, and 13% Other. It is important to note that therapists in this setting serve all children entering the clinic with a wide range of presenting problems and psychiatric diagnosis (i.e., cases are not assigned to therapists based on diagnosis). On average, therapists self-reported that the majority of their caseloads were represented by children ages 4–13 with disruptive behavior problems.

Survey

As mentioned above, the data from the current study were collected as part of the larger PRAC study characterizing usual care treatment process and outcome for children for whom disruptive behavior is one of the presenting problems. The Therapeutic Strategies Survey (TSS) was developed to measure therapists' attitudes about the value of individual psychotherapeutic strategies for individual/family outpatient treatment of children ages 4–13 with disruptive behavior problems, and was intended to complement the observational data measurement system that is being used as the primary measure of treatment process in the larger PRAC study. As such, the TSS is a self-report version of a modified Therapy Process Observational Coding System for Child Psychotherapy (TPOCS; McLeod 2005). The TPOCS was adapted for the PRAC study to characterize treatment strategies observed during psychotherapy sessions. Specifically, the TPOCS-Strategies section (TPOCS-S) was utilized to characterize a wide variety of intervention strategies that are theoretically (e.g., cognitive-behavioral, psychodynamic, client-centered, family) and non-theoretically, or cross-theoretically, driven (e.g., goals). The TPOCS-S content is primarily based on the Therapy Procedures Checklist (TPC) (Weersing et al. 2002) and the format is based on the Therapist Behavior Rating Scale (TBRS) (Hogue et al. 1996) treatment adherence measure. It was modified for the authors' PRAC study through a collaborative process with six community-based therapists (one from each of the participating clinics) (Garland et al. 2006a, b). These therapist partners were selected to be a part of an ongoing Therapist Advisory Group for the PRAC study, and included a representative group (by mental health discipline) of practicing clinicians employed full time in one of the six practicing clinics. They were primarily master-level clinicians (one Ph.D) and years of experience ranged from 2 to 26 years (Garland et al. 2006a). These therapist partners were not actively engaged in delivering any specific EBP intervention protocol. These community-based therapist representatives met with the research team for over 18 hours to review and critique the TPOCS measure, suggesting refinements in definitions of some therapeutic strategies and additions of strategies not included in the original TPOCS (e.g., coordination of external care), with the aim of capturing the breadth of strategies used in community-based care. The Therapist Advisory Group has continued to meet monthly since the development of the PRAC-TPOCS, however, only one member of the original therapist advisory group was still practicing in a participating clinic at the time that the survey used in the current study was created and administered.

The final revised PRAC-TPOCS includes 27 therapeutic strategies, 15 of which reflect therapeutic Techniques (e.g., modeling, addressing client-therapist relationship) and 12 of which reflect therapeutic Content (e.g., affect management, principles of positive reinforcement). Eighteen of the 27 strategies were retained from the original TPOCS (with some minor word changes to clarify definitions) and 9 new codes were added for this version. The 9 new strategy codes reflect therapeutic techniques and content that therapists reported to be common in usual care (e. g., identifying client/family strengths and coordination of external care). These strategies are listed in Tables 1 and 2.

Table 1.

Rank order of average therapist value ratings for strategies directed to children (n = 88)

Children ages 4–8		Children ages 9–13
Strategies with children	Mean (SD)	Strategies with children	Mean (SD)
Treatment techniques
1. Identifying strengths	3.76 (0.53)	1. Identifying strengths	3.86 (0.46)
2. Delivering positive reinforcement	3.75 (0.58)	2. Modeling	3.64 (0.53)
3. Using play, art, stories	3.63 (0.73)	3. Delivering positive reinforcement	3.53 (0.75)
4. Modeling	3.59 (0.60)	4. Assessing problems/events	3.51 (0.64)
5. Assessing problems/events	3.29 (0.79)	5. Role-playing	3.39 (0.69)
6. Role-playing	3.27 (0.77)	6. Exploring client's past	3.28 (0.79)
7. Delivering punishment/limits	3.20 (0.94)	7. Psychoeducation	3.16 (0.83)
8. Exploring client's past	2.67 (1.04)	8. Using play, art, stories	3.14 (0.83)
9. Establishing/reviewing goals	2.60 (0.92)	9. Interpreting meaning of behavior	3.09 (0.90)
10. Interpreting meaning of behavior	2.52 (0.97)	10. Establishing/reviewing goals	3.08 (0.79)
11. Psychoeducation	2.42 (0.89)	11. Addressing client resistance	3.07 (0.92)
12. Addressing client resistance	2.41 (1.08)	12. Delivering punishment/limits	3.00 (0.95)
13. Assigning/reviewing homework	2.33 (0.94)	13. Assigning/reviewing homework	2.77 (0.83)
14. Addressing client-therapist relationship	2.03 (0.91)	14. Using genograms	2.56 (1.03)
15. Using genograms	1.98 (0.92)	15. Addressing client-therapist relationship	2.53 (0.98)
Therapeutic content
1. Parent/child relationship	3.53 (0.68)	1. Problem-solving	3.82 (0.39)
2. Problem-solving	3.52 (0.62)	2. Improved communication	3.69 (0.57)
3. Affect education	3.48 (0.79)	3. Affect education	3.67 (0.64)
4. Principles of positive reinforcement	3.45 (0.92)	4. Parent/child relationship	3.65 (0.59)
5. Addressing child's external care	3.25 (0.94)	5. Principles of positive reinforcement	3.50 (0.82)
6. Improved communication	3.08 (0.85)	6. Cognitive restructuring	3.45 (0.73)
7. Addressing parent/family issues	3.03 (0.99)	7. Affect/anger management	3.42 (0.74)
8. Affect/anger management	2.94 (0.90)	8. Addressing parent/family issues	3.41 (0.78)
9. Principles of punish./limits	2.93 (0.97)	9. Addressing child's external care	3.35 (0.86)
10. Anticipating setbacks	2.72 (0.96)	10. Family member roles	3.20 (0.77)
11. Family member roles	2.59 (0.94)	11. Anticipating setbacks	3.18 (0.88)
12. Cognitive restructuring	2.45 (0.95)	12. Principles of punish./limits	2.99 (0.89)

Note: Strategies in bold have been identified as common core elements in individual youth skills training evidence-based practices for children (ages 4–13) with disruptive behavior problems (Garland et al. 2008)

Table 2.

Rank order of average therapist value ratings for strategies directed to caregivers (n = 88)

Children ages 4–8		Children ages 9–13
Strategies with caregivers	Mean (SD)	Strategies with caregivers	Mean (SD)
Treatment techniques
1. Identifying strengths	3.78 (0.56)	1. Identifying strengths	3.78 (0.56)
2. Modeling	3.75 (0.49)	2. Psychoeducation	3.77 (0.52)
3. Psychoeducation	3.74 (0.54)	3. Modeling	3.72 (0.50)
4. Assessing problems/events	3.61 (0.60)	4. Assessing problems/events	3.61 (0.60)
5. Exploring client's past	3.51 (0.66)	5. Exploring client's past	3.52 (0.66)
6. Delivering positive reinforcement	3.39 (0.80)	6. Establishing/reviewing goals	3.42 (0.74)
7. Establishing/reviewing goals	3.38 (0.75)	7. Interpreting meaning of behavior	3.40 (0.83)
8. Interpreting meaning of behavior	3.31 (0.85)	8. Delivering positive reinforcement	3.39 (0.80)
9. Addressing client resistance	3.23 (0.78)	9. Addressing client resistance	3.32 (0.78)
10. Role-playing	3.20 (0.80)	10. Role-playing	3.17 (0.83)
11. Delivering punishment/limits	2.97 (1.07)	11. Using genograms	2.92 (0.96)
12. Using genograms	2.83 (0.99)	12. Delivering punishment/limits	2.90 (1.08)
13. Assigning/reviewing homework	2.74 (0.96)	13. Assigning/reviewing homework	2.79 (0.93)
14. Addressing client-therapist relationship	2.59 (1.04)	14. Addressing client-therapist relationship	2.68 (1.01)
15. Using play, art, stories	2.47 (0.99)	15. Using play, art, stories	2.32 (0.93)
Therapeutic content
1. Addressing parent/family issues	3.81 (0.45)	1. Addressing parent/family issues	3.82 (0.42)
2. Parent/child relationship	3.80 (0.46)	2. Improved communication	3.77 (0.50)
3. Principles of positive reinforcement	3.77 (0.54)	3. Parent/child relationship	3.76 (0.50)
4. Improved communication	3.69 (0.59)	4. Principles of positive reinforcement	3.73 (0.58)
5. Problem-solving	3.57 (0.69)	5. Cognitive restructuring	3.69 (0.59)
6. Family member roles	3.52 (0.71)	6. Problem-solving	3.65 (0.66)
7. Addressing child's external care	3.51 (0.68)	7. Family member roles	3.60 (0.65)
8. Cognitive restructuring	3.47 (0.68)	8. Addressing child's external care	3.55 (0.64)
9. Principles of punish./limits	3.42 (0.85)	9. Principles of punish./limits	3.43 (0.85)
10. Affect education	3.38 (0.83)	10. Affect education	3.42 (0.80)
11. Anticipating setbacks	3.30 (0.86)	11. Anticipating setbacks	3.34 (0.87)
12. Affect/anger management	3.18 (0.92)	12. Affect/anger management	3.33 (0.89)

Note: Strategies in bold have been identified as common core elements in evidence-based parent training practices for children (ages 4–13) with disruptive behavior problems (Garland et al. 2008)

As mentioned above, for both practical and organizational purposes, the 27 strategies comprising the PRAC-TPOCS were divided into two categories: therapeutic Content and treatment Techniques. This grouping of codes was recommended by our clinician partners as a meaningful way to organize treatment strategies for descriptive purposes. Content refers to the psychotherapeutic topic or issue that is addressed in the therapeutic intervention (quality of the parent-child relationship, anger management, or problem-solving skills). Technique refers to the method in which the therapist imparts information about the therapeutic content to the client (e.g., psychoeducation, modeling, role-playing). Further, strategies are coded separately when targeted to children or caregivers.

Among the 27 strategies are several that our research team has identified as common to evidence-based practices for children with disruptive behavior problems. These strategies were selected through an iterative process of reviewing eight different, established, empirically supported treatment models, compiling elements of each treatment, and identifying elements that were present in at least half of the youth skill building interventions and/or the parent-training interventions. As described in Garland et al. (2008), expert reviewers, who were authors of the intervention protocol or one of the primary intervention developers, validated the final list of common elements of EBP; more details of this iterative process of identifying common elements of EBP are described in that paper.

In summary, the PRAC-TPOCS is a modified version of the TPOCS that was adapted in collaboration with practicing community-based therapists to measure how frequently and intensively therapists employ evidence-based elements of treatment (e.g., modeling, role-play/practice) and treatment strategies clinicians report are frequently used, but are not necessarily emphasized in evidence-based protocols (e.g., coordination of external care). It includes 27 psychotherapeutic strategies that are coded separately for use with children and use with caregivers.

Although the PRAC-TPOCS and the self-report version used here (TSS) include strategies we have identified as common to EBP, this distinction is not explicit in the listing of therapeutic strategies on the survey itself (i.e., all strategies are listed without any specification of EBP). We intentionally omitted use of the labels “EBP” or “empirically supported treatment elements” on the TSS because previous research, as well as our own collaborative experience with our therapist partners, indicates that therapists may have strong attitudes about the concept and/or label of EBP and manualized interventions (Addis and Krasnow 2000). Thus we attempted to assess attitudes about individual treatment strategies and determined retrospectively the extent to which strategies endorsed as most valuable are common to EBP, rather than simply asking about attitudes toward EBP.

The TSS survey consisted of 3 parts:

1. Background Respondents reported demographic characteristics including age, gender, race/ethnicity as well as professional characteristics including educational level, professional discipline, primary theoretical orientation, staff/trainee status and years of practice.

2. Value Ratings Respondents assigned value ratings indicating how valuable (defined as how important to their practice) each PRAC-TPOCS strategy was on a Likert scale ranging from 1 (not at all valuable/not applicable) to 4 (very valuable). Given that previous research indicates therapists may use different strategies with children of different age groups (Weersing et al. 2002) and the importance of considering developmental stages more generally when designing and evaluating psychosocial interventions (Eyberg et al. 1998), ratings of PRAC TPOCS treatment strategies were divided into two age groups (4–8 and 9–13) for each intervention target (child and caregiver). These groupings reflect children who are younger and those who are older than the mean age of children participating in the PRAC study (M = 9.0 years). Assigning value rating for separate age groups was strongly supported by our clinician partners who indicated that they would expect therapists to value strategies differently for children of different age ranges.

3. Supplemental Forced-Choice Items The Likert scale value ratings were our primary method of measuring therapists' attitudes towards individual treatment strategies. However, given that there are advantages of using different item types (DeVellis 2003) and that there was the risk that therapists would strongly endorse all therapeutic strategies, we also included items presented in a forced-choice format. Respondents selected the three “most valuable” strategies for their practice in four categories: (1) treatment techniques directed to children, (2) treatment techniques directed to caregivers, (3) therapeutic content areas addressed with children, and (4) therapeutic content areas addressed with caregivers. Preliminary analyses revealed that the two methods of assessing therapists' attitudes indicated the same general pattern, such that therapeutic strategies with the highest average value ratings also were selected most frequently as one of the most important forced-choice items. Therefore, only the value ratings are presented in this study and this alternative measurement method offers some support for the validity of the relative ratings.

In sum, the TSS is a self-report version of the PRAC-TPOCS measuring therapists' perceived value of 27 different treatment strategies for treatment of children with disruptive behavior problems (listed in Tables 1 and 2). Eight composite scores are derived from the individual 27 strategies: (1) EBP strategies directed to children ages 4–8 years, (2) EBP strategies directed to children ages 9–13 years, (3) EBP strategies directed to caregivers of children ages 4–8 years, (4) EBP strategies directed to caregivers of children ages 9–13 years, (5) All Other strategies directed to children ages 4–8 years, (6) All Other strategies directed to children ages 9–13 years, (7) All Other strategies directed to caregivers of children ages 4–8 years, and (8) All Other strategies directed to caregivers of children ages 9–13 years.

Analysis Plan

Descriptive statistics were used for analyses of average value ratings of individual strategies. Paired samples t-tests were used to compare composite scores of strategies common in EBPs (including parent-training and individual youth skills training treatments) to composite scores of all other strategies that were not identified as being emphasized in a majority of EBP protocols (Garland et al. 2008). Since preliminary analyses indicate that composite scores for the two age groups significantly differed (in general, average value ratings were higher for older children) all analyses were run separately for the two age groups. Oneway analysis of variance and correlations were used in analyses of therapist characteristics associated with average value ratings of strategies commonly emphasized in EBPs.

Results

Reliability

The TSS was re-administered to a subset of 27 participants to assess test-retest reliability. The test-retest interval was relatively long (mean of 11.7 weeks; SD = 6.9; Range: 3 to 18 weeks) due to scheduling difficulties. The average Pearson's correlation coefficient (r) was 0.63 (range 0.51–.77) for the eight composite scores, suggesting adequate reliability. The composite score with the lowest test-retest reliability was EBP Child 4–8 years (r = 0.51) and the composite with the highest reliability was Parent Other Strategy 9–13 years (r = 0.77). Given that there was variability in the number of days between survey administrations, we also conducted a series of regression analyses to examine the associations between Time 1 and Time 2 composite scores, controlling for the effect of days lapsed. Overall, the partial regression coefficients relating days between administrations and Time 2 composites were small and not statistically significant (β range from 0.05 to 0.29).

Attitudes Towards Individual Strategies

Descriptive statistics were used to summarize the mean value ratings assigned to the individual strategies. The results suggest that therapists value a wide variety of treatment strategies when working with children with DBPs and their caregivers in community-based psychotherapy. Table 1 lists all treatment elements, mean value ratings, and standard deviations for treatment strategies targeting children, separated into younger (4–8 years) and older age groups (9–13). Table 2 lists the same information for treatment strategies targeting caregivers of children in each age group. A mean value rating of 3 on our 4-point Likert scale corresponds to a value rating of “valuable.” The majority of treatment strategies received a mean value rating of 3 or higher. Specifically, 37 out of 54 possible (27 strategies × 2 age groups) treatment strategy ratings for treatment directed to children resulted in a mean value rating above 3; 44 out of 54 strategies directed to care-givers resulted in a mean value rating above 3.

Treatment Strategies Directed to Children

As indicated in Table 1, the most highly valued treatment Techniques for children in younger and older age groups were identifying/addressing strengths (M = 3.76 and M = 3.86). The two lowest rated techniques for both younger and older age groups were addressing client-therapist relationship (M = 2.03 and M = 2.53) and using genograms (M = 1.98 and M = 2.56). The highest rated therapeutic Content strategies for the younger age group were parent/child relationship (M = 3.53), problem-solving/social skills (M = 3.52), and for older children were problem-solving/social skills (M = 3.82) and improved communication (M = 3.69). The lowest rated content strategies were family member roles (M = 2.59) and cognitive restructuring (M = 2.45) for younger children and anticipating setbacks (M = 3.18) and principles of punishment/limit-setting (M = 2.99) for older children.

Treatment Strategies Directed to Caregivers

As indicated in Table 2, the two highest rated treatment technique strategies targeting caregivers of younger children were identifying strengths (M = 3.78) and modeling (M = 3.75). The highest rated techniques for caregivers of older children were identifying strengths (M = 3.78) and psychoeducation (M = 3.77). The lowest rated techniques for caregivers of both younger and older age groups were addressing client-therapist relationship (M = 2.59 and M = 2.68) and using play, art, and stories (M = 2.47 and M = 2.32). The highest rated therapeutic content areas for caregivers of younger children were addressing parent/family issues (M = 3.81) and parent/child relationship (M = 3.80). The lowest rated therapeutic content areas for both age groups were anticipating setbacks (M = 3.30 and M = 3.34) and affect management (M = 3.18 and M = 3.33).

Attitudes about Elements of EBP and Other Strategies

Treatment strategies identified as common to EBP for children with DBP are indicated in bold type in Tables 1 and 2. Many of the strategies ranked high in value ratings (Tables 1 and 2) are common elements of EBP. As indicated in the TSS description, composite scores of average value ratings of strategies common to EBP were compared to composite scores of average value ratings of all other strategies (for each age group/target combination).

A series of four paired samples t-tests were used to examine mean differences between average EBP composite scores to the composite of all other strategies. Results of paired samples t-tests indicate that therapists rate strategies that are common elements of EBPs as high, or higher, than all other treatment strategies. Specifically, average therapist value ratings of elements of EBPs directed to children ages 4–8 (M = 3.06, SD = 0.42) were significantly higher than all other strategies for that age group (M = 2.91, SD = 0.46) (t = 3.72, df = 87, p<.01). Average therapist value ratings of EBP strategies directed to caregivers of children ages 4–8 (M = 3.46, SD = 0.36) were also significantly higher than all other strategies for that age group (M = 3.31, SD = 0.36) (t = 4.26, df = 87, p<.01). Average therapist value ratings of EBP strategies directed to caregivers of children ages 9–13 (M = 3.48, SD = .038) were significantly higher than average value ratings of all other strategies directed to caregivers of children in this age group (M = 3.35, SD = 0.36) (t = 3.65, df = 87, p<.01). The difference between average therapist value ratings of EBP strategies directed to children ages 9–13 (M = 3.31, SD = .039) and average value ratings of all other strategies (M = 3.29, SD = .038) was not significant (t = 0.40, df = 82, p>.01).

Therapist Characteristics Associated with Attitudes about EBP Strategies

Bivariate analyses using independent samples t-tests, univariate ANOVAs, and correlations were conducted to examine the associations between nine therapist characteristics (gender, age, therapist racial/ethnic minority status, professional discipline, primary theoretical orientation, trainee status, months of experience, educational level, and agency) and the four EBP summary composites (EBP strategies to caregivers for both age groups, and EBP strategies to children for each age group). A Bonferroni correction was used to adjust for multiple comparisons, yielding a significance level of .001. At this alpha level, no statistically significant associations were observed, indicating that none of the therapist characteristics were significantly associated with mean therapist value ratings of EBP elements. While this alpha level was used to be conservative, no therapist characteristics were significantly associated with the composite scores at the .05 level. Overall, effects sizes were small (Cohen's d ranged from 0.006 to 0.37; partial eta squared ranged from .004 to .05, and correlation coefficients ranged from 0.03 to 0.19).

Discussion

The results of this study provide valuable information about a generally representative sample of community-based therapists' attitudes towards a variety of treatment strategies, including those that are common across evidence-based interventions for children with disruptive behavior problems and other strategies that are not emphasized in the evidence-based, individual youth skills training or parent training manualized protocols. Therapists were asked to rate how valuable (i.e., important) they consider individual strategies to their practice because perceived value has been identified as an important predictor of behavior (Casper 2007). Overall, therapists highly value many different types of treatment techniques and therapeutic content in their work with this group of children and their caregivers. While there was variability in the Likert ratings, on average, therapists endorsed most of the therapeutic strategies as “fairly valuable” or “very valuable” to their practice. These findings are consistent with previous research indicating that therapists may use a variety of different strategies (Baumann et al. 2006) and prefer an “eclectic” approach to treatment, utilizing a variety of techniques (Kazdin et al. 1990).

The results also indicate that therapists highly value many strategies that are common in EBP protocols for this population. Overall, composite scores of average value ratings of strategies common in evidence-based interventions were higher than composite scores of average value ratings of all other strategies that are not emphasized in the majority of EBPs. In particular, EBP strategies that therapists rated as highly valued in their practice with this population included delivering positive reinforcement, modeling, problem-solving/social skills, and affect education. EBP strategies that therapists rated as highly valued in their practice with caregivers of children with disruptive behavior problems included modeling, psychoeducation, parent-child relationship, and principles of positive reinforcement. This finding is promising, as it indicates that therapists highly value treatment approaches common to EBPs. Interestingly, the composite scores of average value ratings for EBPs and all other strategies were not significantly different for strategies delivered to children ages 9–13. This finding may be related to therapists' perceptions that there are developmental differences in the applicability in common elements of EBP for children with disruptive behavior problems, such that they consider other strategies such as improved communication and cognitive restructuring more valuable with older children than with younger children. It is important to mention that cognitive restructuring is a strategy that was not identified as common to EBPs for disruptive behavior problems, but it may be commonly used in EBPs for internalizing disorders. Overall, the distributions of value ratings for content strategies delivered to children appears slightly different between the two age groups, however, the distribution of value ratings for therapeutic techniques delivered is relatively consistent. It is likely that many of the active techniques, such as modeling and role-play/practice, are applicable to a wide range of content areas addressed in psychotherapy and are applicable to children of different developmental levels. Additional research on how children's developmental level impacts treatment planning and treatment provided would be interesting to pursue (Eyberg et al. 1998).

In our exploratory analyses of associations between therapist characteristics and EBP value ratings, no significant associations were identified, and effect sizes were generally small. This lack of other associations between therapist characteristics and attitudes towards treatment strategies, particularly self-report of theoretical orientation, is notable. One might expect, for example, that therapists who endorse behavioral or cognitive behavioral theoretical orientations might value EBP strategies more than therapists of other orientations, as most of the EBPs for youths with disruptive behavior problems are behavioral or cognitive behavioral. Further, previous research indicates that interns, in contrast to licensed professionals and those with higher educational status, may hold more positive attitudes towards EBPs (Aarons 2004). The lack of significant associations between therapist characteristics and value ratings may be due to the way that attitudes about treatment were measured. Specifically, differences between therapists may be minimized when the focus is on attitudes towards individual treatment strategies rather than towards the concept of manualized treatment or “empirically supported treatments” more generally.

This study adds to the literature in a number of ways. First, while there are a growing number of studies reporting therapists' attitudes toward EBPs, they have typically measured attitudes towards specific EBPs or manualized interventions in general. This line of research suggests that therapists may have different definitions of, and exposure to, manuals, making it challenging to assess their attitudes towards them (Baumann et al. 2006). In the current study, therapists were asked about their attitudes towards individual treatment strategies. The terms “evidence-based practice” and “manual” are not included in the survey instrument, although it includes strategies that have been identified as common to EBPs for children ages 4–13 years with disruptive behavior problems (Garland et al. 2008). It also includes strategies that are not emphasized in EBPs, but that our therapist partners indicate are important to their practice (e.g., identifying client's strengths, assessing problems and events) and may be implicit in EBPs. Focusing on individual strategies provides a common language to assess attitudes, as previous research indicates that there may be significant variability in familiarity with EBPs or manualized interventions (e.g., Aarons 2004; Addis and Krasnow 2000; Baumann et al. 2006). Further, it highlights that therapists may be more positive about EBP strategies when the emphasis is on individual treatment strategies rather than on manualized interventions. Future research might examine how attitudes towards EBPs or manualized interventions more generally are associated with attitudes toward individual treatment strategies.

Another strength of this study is that it assessed attitudes of a generally representative sample of therapists providing community-based mental health services to children in one, large and diverse county and provides a description of the characteristics of therapists practicing in six, representative, community-based, outpatient mental health clinics serving a group of racially/ethnically and diagnostically diverse children and their families. Almost all therapists practicing in six, publicly-funded agencies in a large urban area responded to the survey. The therapists were diverse in their professional discipline, self-reported primary theoretical orientation, and level of experience. While there was some racial/ethnic diversity, the majority were Caucasian. Therapists were mostly masters-level, having practiced five years on average, and over one-third were unlicensed trainees. Therapists represented multiple professional disciplines with approximately one half of the respondents trained in the Marriage and Family Therapy discipline and twenty percent each from the Social Work and Psychology disciplines. The most frequently reported primary theoretical orientations were behavioral and cognitive behavioral. Data from a recent national survey of 1200 clinicians from 100 clinical sites across the United States provides a good comparison sample to the sample of clinicians surveyed in the current study (Glisson et al. 2008). In the national sample, clinicians were primarily female (76%) and Caucasian, with the majority holding a master's degree (67%). This study supports the representativeness of our sample in terms of basic demographics and educational level. Studies including a diverse, representative sample of therapists contribute to our understanding of usual care practice.

There are a number of limitations to this study that should be noted. First, we did not include a measure of observed behavior for the current sample, so we can not say whether the strategies therapists' rate as highly valuable are the strategies that they most frequently use in practice. As part of our larger PRAC research study, however, we are collecting observational data on therapists sampled from the same clinics as the TSS sample (57% of therapists in the current sample are also in the PRAC observational study sample). Preliminary examination of the PRAC observational psychotherapy process data suggests there are a number of consistencies and inconsistencies between therapists' attitudes about the value of treatment strategies and their observed use of treatment strategies. For example, observational data suggests that therapists do use a wide range of treatment strategies, which is consistent with the large number of strategies rated as highly valuable in this study. However, the strategies considered the most valuable are not necessarily the strategies observed most frequently (e.g., modeling was a strategy that was rated as highly valuable, however, it is observed in less half of videotaped psychotherapy sessions). This discrepancy indicates that while therapists may consider certain strategies very important in their work, there may be barriers to implementing these strategies frequently or intensively. Future research will specifically examine these patterns.

Another limitation of this study relates to the measure used to assess therapists' attitudes towards psychotherapeutic strategies. Given our purpose of assessing therapists' attitudes towards individual psychotherapeutic strategies (including those common in EBPs and those that therapists report are common practice) that are consistent with the strategies used in our observational coding system, we did not use one of the few measures with established psychometric properties such as the EBPAS (Aarons 2004; Aarons et al. 2007) or the TPC (Weersing et al. 2002). Instead, we adapted a self-report measure based on the observational measure (PRAC-TPOCS) used in the PRAC study. Thus, the psychometrics of our TSS measure are not well established and further validation of the measure is needed in the future. However, we do have some support for the test-retest reliability, and the two measurement methods lend some support to the validity. A related limitation is that we designated treatment strategies as “common in EBPs” based on one method of identifying these elements (Garland et al. 2008). Other methods might yield additional or different common elements of EBPs. Lastly, this study was conducted in one urban area and, therefore, may not generalize to other areas.

In the current study, we differentiated strategies based on whether they had been identified as a common treatment element in EBPs for children with disruptive behavior problems, and conducted analyses to identify therapist characteristics associated with a composite of EBP elements. In future research, we may examine naturally occurring groups of strategies through exploratory factor analyses and identify therapist characteristics associated with these empirically determined groups of strategies.

It is important to note that we may expect different results if we surveyed therapists regarding their attitudes towards strategies used with a different clinical populations such as children with internalizing problems. This study reports therapists' attitudes towards strategies used with children with disruptive behavior problems. We argue that our emphasis on strategies for children with disruptive behavior problems is important, as it is applicable to the largest portion of patients receiving community-based care (Garland et al. 2001). This study paves the way for researchers to examine therapist attitudes towards elements of treatment targeting other diagnostic groups or presenting problems.

While our therapist sample is likely representative of therapists in other communities based on experience and level of experience (Glisson et al. 2008), certain characteristics of our sample may not be representatives of therapists practicing in other geographic areas. For example, approximately half of therapists in our sample were trained as martial and family therapists (MFTs). While this discipline is very common in California, it may not be as common in other states. Further, the distribution of therapist primary orientation may not be representative of providers in other communities. Fewer than ten percent of therapists in the current sample self-identified as dynamic, differing from certain studies of practicing providers that report approximately one quarter of therapists identifying as dynamic (e.g., Norcross et al. 2002; Stewart and Chambless 2007; Weersing et al. 2002). However, these studies included doctoral-level psychologists only, many of whom were in private practice, rather than a multidisciplinary sample of therapists practicing in publicly-funded settings. Our sample is similar in theoretical orientation distribution to another sample of multidisciplinary therapists working in community-based child welfare practice (Baumann et al. 2006). Overall, distribution of therapist discipline and orientation may vary by geographic location and service setting (community-based care vs. private practice).

Despite these limitations, the results of this study have important implications for the implementation of evidence-based interventions in community-based practice. In particular, these findings have implications for tailoring training efforts in EBP protocols based on the knowledge gained about attitudes towards individual components of EBPs. New research suggests that sustainable changes in mental health providers' behavior are more likely to be achieved when training interventions are tailored to preexisting attitudes and values regarding practice (Casper 2007). Our finding that therapists do value many elements of EBPs indicates that a training approach incorporating attitudes toward select treatment strategies may be effective in addressing potentially ambivalent attitudes toward manualized interventions in general. Understanding therapists' attitudes provides information on characteristics that influence the treatment decision-making process. Given that there may already be “buy in” on individual treatment strategies such as delivering positive reinforcement, training could focus on providing specific feedback on the intensity, timing and generalized use of this strategy. In contrast, more time and effort might be needed to provide a stronger rationale for strategies that therapists did not highly value that are core to EBPs (e.g., assigning/reviewing homework and anticipating setbacks) in order to facilitate training. Essentially, knowledge about therapists' attitudes can provide a roadmap for effective training efforts for individual EBP treatment protocols or proposed common elements approaches (Garland et al. 2008).

In sum, findings from the current study indicate that, when asked about individual treatment strategies used with children with disruptive behavior problems in outpatient, community-based mental health settings, therapists highly value a number of different therapeutic strategies. Further, they value those strategies that are common in EBPs for this population at least a much as strategies not emphasized in EBPs. Differences in attitudes towards EBP strategies cannot be accounted for by differences in therapist demographics, professional discipline, level of training, or primary theoretical orientation. Understanding therapists' attitudes has the potential to significantly improve implementation of EBPs in community-based care.