Child and Family Therapy Process: Concordance of Therapist and Observational Perspectives

Abstract

The objective of this study is to examine the characteristics of outpatient mental health services delivered in community-based outpatient clinics, comparing information obtained from two different sources, therapists serving children and families, and observational coders viewing tapes of the same treatment sessions. Videotaped therapy sessions were rated by therapists and independent coders regarding goals and strategies pursued during each session. Sixty-three sessions were taped of outpatient care provided to 18 children and their caregivers by 11 therapists. Children were 4–13 years old and families were receiving services at least in part due to reported child behavior problems, confirmed by ratings from the Child Behavior Checklist and Conners Parent Rating Scale—Revised. Analyses assessed the frequency, type, and intensity of goals and strategies pursued in therapy sessions from both therapist and observational coders’ perspectives. Reliability of observer ratings and correspondence between therapist and observer reports were also examined. The reliability of observational coding of goals and strategies was moderate to good, with 76% of 39 codes having ICCs of .5 or greater. Therapists reported pursuing 2.5 times more goals and strategies per session, on average, than identified by observational coders. Correspondence between therapists and coders about the occurrence of specific goals and strategies in treatment sessions was low, with 20.5% of codes having a Kappa of .4 or higher. Substantial differences exist in what therapists and independent coders report as occurring in outpatient treatment sessions. Both perspectives suggest major differences between the content of services provided in community-based outpatient clinics and the structure of evidence-based programs, which emphasize intense pursuit of a small number of goals and strategies in each treatment session. Implications of the findings for quality improvement efforts in community-based mental health care settings are discussed.

Introduction

Publicly supported outpatient mental health care for children and families in the United States relies upon networks of community-based (CB) organizations. These organizations can vary considerably in size, structure, and focus based on the niches they serve in an area. However, CB outpatient mental health care organizations also share many common elements. Agencies tend to employ a mixture of clinical staff, many having recently completed Masters level training in areas such as social work, psychology, marriage and family therapy, and nursing (Addis and Krasnow 2000; Hurlburt et al. 2002; Weersing et al. 2002). Most organizations serve populations of children and families with diverse clinical needs (Jensen and Weisz 2002) in a targeted geographic area.

Of greatest significance for this paper, community-based agencies are increasingly experiencing both pressures and incentives to invest in evidence-based mental health interventions. Evidence-based (EB) practices are at the forefront of efforts to improve service outcomes (Kramer and Burns 2008; Raghavan et al. 2008). Accumulating evidence, summarized in the form of meta-analyses and review articles, shows that many different types of structured therapy can improve mental health outcomes for children and adolescents (Brestan and Eyeberg 1998; Casey and Berman 1985; Kazdin and Weisz 1998; Ollendick and King 1998; Pelham et al. 1998; Weisz et al. 1995, 2006). The current emphasis on EBPs as a means for changing practice is reflected in the number of unique organizations supporting collection, summary, and dissemination of information about scientific support for specific practices (Goldman and Azrin 2003), and in policies and legislation that have been adopted in some states (Raghavan et al. 2008).

Implicit in all efforts to encourage adoption of evidence-based practices are assumptions that existing services in CB settings are less likely to yield outcomes as positive as those that could be achieved through adoption of innovative service models having a base of empirical support (Bickman 2002). A recent paper explicitly examined this issue, using meta-analysis to compare the outcomes of evidence-based practices delivered in CB settings with usual care comparison groups (Weisz et al. 2006a, b). Results from that study suggested a modest advantage for EB practices, but also raised questions about the ability to achieve such benefits in CB settings.

In the presence of the significant focus devoted to EBP adoption and implementation, one important issue that has received surprisingly little attention is the content and methods used in standard services in CB settings (Addis 2002; Baumann et al. 2006; Bearsley-Smith et al. 2008; Bickman et al. 1999; Weersing et al. 2002). Understanding the structure and focus of existing services is an essential part of evaluating the potential benefits of investments in any new service quality improvement effort, including utilization of EBPs, and in foreseeing areas in which there might be opportunities or difficulties associated with implementation of new service models (Hogue et al. 2004; Norquist 2001). To date, only a few studies have attempted to measure and describe characteristics of usual services and how they are similar to or different from structured treatment models or guidelines (e.g., Brookman-Frazee, this issue; Garland et al. (2009); Jensen-Doss et al. 2007; Kramer and Burns 2008; Zima et al. 2005).

This paper specifically examines the characteristics of outpatient mental health services delivered by therapists in four CB clinics in a large urban area of the west coast. Two independent perspectives were used to characterize existing services, those of therapists serving children and families, and of observational coders viewing tapes of the same treatment sessions. Results from this study provide some information about the overall characteristics of services in CB settings. However, the paper principally focuses on examining similarities and differences between two potentially quite different perspectives on the characteristics of services delivered. Similarities and differences between therapists’ and observers’ perspectives on the content of services are important in their own regard. Differences that exist between these perspectives could themselves have implications regarding therapists’ attitudes towards evidence-based practices and factors that may affect efforts to encourage their adoption and implementation.

Different methodological approaches are available to understand the structure and content of mental health services in CB service settings. Independent observation of therapy sessions can be used to characterize specific goals and strategies used by therapists. Treatment manuals have been particularly helpful for identifying treatment goals and strategies for observers to code (Chorpita et al. 2005; Kazdin 2000; Luborsky and DeRubeis 1984) that are linked to key hypothesized mechanisms of change. An observational measurement approach has been taken in some psychotherapy adherence measurement studies (Hogue et al. 1997, 2004; Markowitz et al. 2000). Often, observers look specifically for components of intervention models being tested and not a broader array of treatment processes, so the lens of coded therapeutic approaches can be rather narrow (e.g., Ablon and Jones 1999; Hill et al. 1992; Lambert and Ogles 1988; Jones and Pulos 1993).

Therapist self-report has also been used to identify aspects of services provided; this too has been used to measure adherence to specific treatments as part of the psychotherapy research process (Henggeler et al. 1997; Jones et al. 2007; Webster-Stratton and Reid 2003). Others have developed tools that therapists may use to characterize a broader range of goals, strategies, and techniques that occur during the treatment process (Bearsley-Smith et al. 2008; Weersing et al. 2002).

Community-based care is often described as eclectic (see, e.g., Kazdin et al. 1990), or at least encompassing therapists from a number of different treatment perspectives (Weersing et al. 2002). Efforts to more clearly understand CB care must therefore utilize a broader lens to measure treatment process, one that encompasses many of the treatment goals and strategies used by therapists. Fortunately, the documentation of multiple evidence-based programs has revealed that many evidence-based programs share common goals and strategies that can be measured (Chorpita et al. 2005; Garland et al. 2008). Garland et al. (2008) described a systematic intervention review process that was used to identify common elements across evidence-based practices for the purpose of developing treatment process measures and studying usual care services.

The approach taken in this paper is unique in having utilized a common set of treatment process elements as the core for two different measurement methods, therapist self-report and coder observation. As described in the methods section, the treatment goals and strategies included in our treatment process measures have much in common with those of Garland et al. (2008). The Child and Adolescent Treatment Strategies (CATS) study worked collaboratively with the Practice Research, Advancing Collaboration (PRAC) study, described in this volume (Garland et al. 2008; Brookman-Frazee et al. 2008), and in partnership with CB outpatient mental health service organizations interested in learning more about their treatment services. One early aim of the CATS study was to examine areas of consistency and divergence between independent coders’ ratings and therapist self-reports of treatment content in preparation for a larger study of treatment process. The study focused on families presenting for mental health treatment at least in part due to child behavioral difficulties. As described further below, the Child Therapy Process Rating System (CTPRS) was used to measure treatment process and includes a broad range of treatment goals and strategies, but has an emphasis on coding for treatment processes targeting behavioral difficulties in preadolescent and younger children.

Method

Participants

Videotaped therapy sessions were examined of children and families coming to services at least in part due to reported child behavior problems. Children ages 4–13 were eligible if they had a score on the externalizing scale of the Child Behavior Checklist (Achenbach 1991) or the oppositional subscale of the Conners’ Parent Rating Scale—Revised (Conners et al. 1997, 1998) at or above the borderline clinical level, and at least one of the presenting concerns reported by the caregiver was consistent with disruptive behavior problems. These criteria were used to ensure that there was reasonable evidence to support the presence of behavioral difficulties warranting attention, such that therapy sessions would have a high probability of addressing these issues in some way. Participation requirements also included having no evidence in the case record of psychosis, mental retardation, or pervasive developmental delay. Videotapes were made of sessions from ongoing treatment with 18 families, delivered by 11 different therapists, working in four of the largest outpatient specialty mental health clinics in a large west-coast metropolitan area. Sessions recorded occurred at many different points in the treatment process (early, middle, and end of treatment), depending only upon when the child and family were enrolled in the study. In total, up to 4 sessions were videotaped for each family. Table 1 summarizes characteristics of children receiving services who, along with parents and therapists, participated in this study. Therapists participating in the study had a range of training backgrounds, including 1 psychologist, 3 social workers, and 7 marriage and family therapists (MFTs). Seven therapists were licensed staff and 4 were trainees.

Table 1.

Characteristics of children receiving outpatient care whose treatment sessions were rated by therapists and independent coders

Characteristic	Mean (SD) or %
Age	11.18 (2.5)
Gender
Female	22.2%
Male	77.8%
Race/ethnicity
Asian/Pacific Islander	5.6%
White	77.8%
Hispanic	11.1%
More than one reported	5.6%
Diagnoses
ADHD	53.3%
ODD	33.3%
Other behavioral diagnosis	20.0%
Depression NOS or Mood NOS	33.3%
CBCL (T)
Total	71.6 (7.5)
Externalizing	70.4 (8.0)
Internalizing	64.0 (11.5)
Conners (T)
ADHD	71.3 (10.3)
Oppositional	73.5 (11.3)
Total	75.7 (9.7)

Procedures

Enrollment and Video-Recording

The research team first met with each of the participating clinics during staff meetings to explain the project and invite therapists to participate. Interested therapist volunteers completed consent forms. Children and their parents, seen by therapists participating in the study, were invited by clinic staff to receive more information about the study. Interested families met with the study coordinator and completed study consent forms. Of the 35 children age 4–13 coming into contact with participating therapists during the enrollment period, families of 66% (N = 23) agreed to talk with the research coordinator, have their records reviewed, and complete further structured questionnaires. Records were reviewed for diagnostic information and presenting concerns noted in the chart. A total of 21 children were eligible and agreed to participate; ultimately at least 1 taped session was completed with 18 families. No videotaped sessions were available for the other three families due to discontinuation of services and failure to record sessions.

Videotape recordings were made of the next 3–4 sessions following enrollment. After each session, therapists reminded participants that sessions had been recorded and asked permission to provide tapes to the research team. No families declined to provide tapes at this time. At the end of recorded sessions, therapists completed a checklist form of the Child Therapy Process Rating System (CTPRS), described below, that asked about specific goals and strategies the therapist attempted to pursue during the session, and for a rating of the intensity with which each was pursued. Each recorded session was also coded by two independent research team members.

Training of Coders

Coders included students pursuing a Masters degree in a mental health-related field, including social work, marriage and family therapy, and psychology. Masters level students were selected because they tended to have had exposure to multiple theoretical perspectives about mental health treatment, familiarity with the goals and techniques of different treatments, and general interest in learning more about treatment process. Each coder took part in an initial training that included reading the CTPRS coding manual, reading excerpts from treatment manuals that illustrated examples of therapeutic approaches consistent with CTPRS codes, reviewing segments of video-tape that illustrated CTPRS codes, and then coding reference tapes until a threshold level of performance was reached across seven consecutive videotapes. Using an intraclass correlation coefficient, coders were required to have an overall correspondence level with reference coding on training tapes of .7. Upon reaching the accuracy threshold, coders met at least bi-weekly with the principal investigator to discuss coding questions and review examples from coded tapes. The coding procedures were reviewed regularly at ongoing training meetings. Coding software also provided one-click access to code definitions and coders were required to use the definitions in cases of uncertainty during coding, providing another source of ongoing training.

Measures

Demographics

Demographic information was collected about the age, gender, and racial/ethnic backgrounds of children in the study sample.

Child Behavior Checklist and Conners’ Parent Rating Scale: Revised

The child’s primary caregiver completed the Child Behavior Checklist (CBCL) and the Conners’ Parent Rating Scale—Revised (CPRS-R) prior to recording of any treatment sessions. The CBCL is a well-established index of child competencies and emotional and behavioral difficulties. In addition to nine more specific syndrome scales, the CBCL yields broad band internalizing, externalizing, and total problem scores. T-scores are based on population norms. This paper reports means and standard deviations for the three broad scale scores. The CPRS-R is also a well-established index of emotional and behavioral problems among children. The CPRS-R yields scores on seven specific subscales and generates a number of targeted summary scores, including a Global Index and an ADHD index. This paper reports means and standard deviations for the Conners’ Global Index, the Oppositional subscale, and the ADHD index.

Diagnosis and Presenting Concerns

Diagnostic information was abstracted from clients’ case records. Independent assessment of diagnoses was not conducted. It should be noted that concordance between case records and independent structured diagnostic techniques may not be high (Jensen and Weisz 2002; Lewczyk et al. 2003). The diagnostic information gathered describes how children were classified by therapists in CB mental health service settings.

Child Therapy Process Rating System: Overview

The Child Therapy Process Rating System (CTPRS) was developed to measure the methods therapists use, and goals and strategies they pursue with children and families during specialty mental health treatment. It was developed for the broader purpose of describing treatment process and examining the relationship between treatment process and clinical outcomes. Methods were defined as techniques that a therapist could use to accomplish a treatment objective, such as assigning homework, role-playing with a client, reflecting back a client’s thoughts or feelings, or playing games, to name a few. Goals and strategies were considered to be specific aims that a therapist might attempt to accomplish with a client and might be short- or longer-term objectives instrumental in achieving clinical outcomes. This study focuses on goals and strategies pursued in treatment.

The CTPRS includes goals and strategies drawn from multiple sources. These include treatment programs with an established base of empirical support related to children with disruptive behavior problems, more general texts covering different approaches to child mental health services, and transcripts from interviews with therapists asked about the approaches they might take to working with children and families presenting with these concerns. During the initial development phase, the research team reviewed and catalogued in detail the treatment methods and objectives included in treatment manuals or texts. Goals and strategies were abstracted from evidence-based programs, including manuals for The Incredible Years, Parent Child Interaction Therapy, Parent Management Training, Problem Solving Skills Training, and Anger Coping Therapy, programs identified as having strong scientific support at the time of the CTPRS development (e.g., Kendall and Braswell 1993; Hembree-Kigin and McNeil 1995; Larson and Lochman 2002; Shure 2001; Webster-Stratton 2001). Over 30 other texts covering approaches to working with children and families, frequently with a focus on behavioral problems, were also reviewed and abstracted. These included reference texts from disciplines of family systems therapy, narrative therapy, solution-focused therapy, psychodynamic therapy, play therapy, and cognitive-behavioral therapy (e.g., De Jong and Berg 2002; Chazan and Kernberg 1991; Minuchin and Fishman 1981; Glaser and Easley 1998; Kaduson and Shaefer 2000; Smith and Nylund 1997). Additional methods and goals emerged from qualitative abstraction of goals and methods discussed by 21 therapists who took part in semi-structured interviews asking about how they might imagine working with several hypothetical clinical cases. Two independent reviewers, the first author and a Masters level research assistant in social work, reviewed transcripts of interviews with therapists and enumerated goals and strategies that therapists reported they might use. The most common goals and strategies therapists reported were cross-walked with goals and strategies drawn from EBP manuals and other child therapy references. Based on examination of goals and strategies emerging from the three sources, a set of observational codes and their definitions was developed for the CTPRS.

Table 2 lists each of the 39 separate CTPRS goal-target combinations (goal or strategy plus individual to whom it is targeted) that were included in this analysis. Another 19 goals and strategies were also part of the original coding framework but are not included in this analysis because their frequency of occurrence was very low (coded in less then 10 2.5 minute intervals across all 63 session tapes).

Table 2.

Reliability of observer ratings of goals and strategies in the child therapy process rating system (CTPRS)

Goal/strategy	Child ICC	Caregiver ICC
Collect information about…’s mental health status, behavior, and/or functioning in one or more contexts (e.g., home, school, with peers); OR about family structure, such as: (a) who members of the family are and how individuals are related to one another, or (b) the strength of relationships that family members have with one another	.56	.68
Collect information about…’s current psychosocial stressors	na	.68
Collect information about…’s strengths and/or supports	.30	.58
Gather information about high-risk issues and/or assess for the presence or extent of high-risk issues	.89§a	.01§b
Develop and/or facilitate the development of a safety plan or crisis plan to respond to high-risk issues	.89§a	.01§b
Provide an outline or agenda of goals during the session . The agenda could have been verbal or written	.60	.60§c
Review the progress of the child/family toward meeting treatment goals	na	.60§c
Help the child receive appropriate medications . This could occur either by connecting the child/family with an appropriate individual, or by the therapist assessing needs or prescribing him/herself (when appropriate)	.79
Connect at least one individual with other informal/social supports, or strengthen relationships with other informal/social supports (e.g., friends, other parents, informal support groups, etc.)	.51§d	.61§e
Connect at least one individual with other formal services or strengthen their relationships with other formal services (e.g., Head Start, social services, respite care, legal services, alcohol and drug treatment services, etc.). DO NOT COUNT MEDICATION SERVICES HERE	.51§d	.61§e
Help at least one individual learn more about specific skills or abilities that the child is learning or improving	–	na
Help at least one individual to identify and/or counteract his/her cognitive distortions (e.g., irrational beliefs and/or thoughts)	.00	.05
Help at least one individual to improve his/her ability to understand and/or differentiate emotions from one another (e.g., This could involve: (a) recognizing specific emotions, (b) understanding the meaning of and verbal and nonverbal cues associated with specific emotions, etc.)	.63§f	–
Help at least one individual to improve his/her ability to identify his/her own signs or cues that signify specific emotions (e.g., feeling hot, muscular tension, self-denigrating thoughts, accusatory thoughts, might signify anger or sadness)	.63§f	–
Help at least one individual to improve his/her ability to use structured problem solving skills to handle difficult interpersonal situations (e.g., this could include skills such as: (a) recognizing and identifying the problem, (b) coming up with possible solutions, (c) evaluating possible consequences of different actions, (d) choosing an action to take, (e) rewarding oneself for problem solving, etc.)	.40	–
Help at least one individual to improve his/her social and/or communication skills. For example, skills might include: (a) how to listen effectively, (b) how to reflect back what another person has said, (c) how to give equal time to participants in a conversation, (d) how to meet a new person, (e) how to start a conversation, etc	.51	–
Help at least one individual to improve his/her ability to use relaxation skills or techniques. This could involve techniques such as: (a) meditation, (b) deep breathing strategies, (c) progressive muscle relaxation, (d) pleasant imagery, etc	.68§g	–
Help at least one individual to improve his/her ability to reduce own stress level. This could include: (a) relaxation or meditation skills, (b) finding time to take personal time outs, (c) taking part in more pleasant activities, (d) finding ways to anticipate and avoid stressful situations, etc	.68§g	na
Help at least one individual to improve his/her ability to understand another family member’s perspective and/or feelings	.54	.52
Help at least one individual to improve his/her ability to coordinate and be consistent with at least one other individual in how they interact with child (e.g., both respond to hitting in the same way). This could be caregivers with one another or caregiver(s) with teacher(s)	–	na
Help at least one individual to improve his/her ability to: (a) assume adaptive/appropriate parental or caregiver roles and/or responsibilities, or (b) eliminate or reduce inappropriate parental or caregiver roles and/or responsibilities (e.g., acting as the child’s parent, rather than their friend)	–	na
Help at least one individual to improve his/her ability to: (a) assume adaptive/appropriate child roles and/or responsibilities, or (b) eliminate or reduce inappropriate child roles and/or responsibilities (e.g., decrease child’s acting in a parental role or engagement in a coalition against another caregiver, etc.)	na	–
Help at least one individual to improve his/her ability to give effective commands. This could include helping the individual to: (a) state commands succinctly, (b) not repeat commands over and over, (c) use commands that ask the child to start doing something rather than stop doing something, (d) give the child choices in commands, etc	–	.42
Help at least one individual to improve his/her ability to monitor or keep track of child behavior (either appropriate or inappropriate child behavior). For example, this might involve providing supervision of the child, or maintaining contact with teachers, babysitters, daycare providers, etc	–	.03
Help at least one individual to improve his/her ability to respond effectively to inappropriate or negative child behavior. This could involve how to: (a) ignore inappropriate behavior, (b) be consistent over time in responses to behavior, (c) select natural or logical consequences, etc	–	.62§h
Help at least one individual to improve his/her ability to use Time Out	–	.62§h
Help at least one individual to improve his/her ability to respond effectively to appropriate or good child behavior. This could involve methods such as: (a) attending to and/or praising appropriate behavior, (b) looking for approximations of good behavior to praise, etc	–	.79§i
Help at least one individual to improve his/her ability to use reward systems with child. This could involve: (a) selecting behaviors to target, (b) selecting how to reward behaviors, (c) developing a reward chart, etc	–	.79§i
Help at least one individual to experience and/or express affect (especially emotions he/she may not have been able to release). These may be in response to events or circumstances in the past, present, or future	.64	–

Note Intraclass correlations (ICCs) were computed based on 258 double-coded therapy sessions from a larger sample of treatment sessions studied in the Child and Adolescent Treatment Strategies (CATS) study. Each goal and strategy has a more detailed accompanying definition that observers used to identify and code when therapists were pursuing specific goals or strategies. The checklist and definitions are both available directly from the first author. All ICCs greater than or equal to .30 are significant P < .05. § with a letter denotes a code that was combined with another code in the larger CATS study to reduce the total number of CTPRS codes. Goals and strategies with the same letter were combined. ICCs for the combined codes are reported here for each separate code individually. na, information not available about reliability in the broader CATS study because the goal or strategy was not measured as a separate or similar code. –, indicates that ratings were not made about a goal or strategy for a particular target (child or caregiver)

Therapist Ratings of Goals and Strategies

The descriptions listed in Table 2 are the exact items to which therapists responded on a post-session checklist, although arranged in a different order and without the formatting of the therapist checklist. After a session ended, therapists were asked to check off each goal or strategy they had pursued. For all checked goals and strategies, therapists rated the intensity with which they were pursued using the numeric rating system listed in the first column of Table 3, ranging from 1 (pursued fleetingly) to 5 (pursued intensely). A zero was assigned to all goals and strategies not checked.

Table 3.

Intensity rating scale for the child therapy process rating system

Intensity	Description
0	Not pursued in session
1 Pursued fleetingly	Briefest of moments
	Little or no effort to follow up on method or goal
	One minute or less, although this is not an absolute guide
	Coder may have uncertainty about whether or not the goal was pursued
	Client may not know that this was a goal being pursued
2 Pursued minimally	More than the briefest of moments
	Some limited follow-up on the goal, but little effort to follow through on pursuit of goal
	One to three minutes (although possibly less)—minutes are a rough guide
	Coder is reasonably certain goal was pursued
	Client may still not know that this was a goal being pursued
3 Pursued moderately	Goal was clearly pursued for several minutes
	More than minimal effort to follow through on pursuit of the goal, although not very much effort expended on pursuit of the goal
	Two to five minutes—minutes are only a rough guide
	Coder is certain goal was pursued
	Unless there are very unusual circumstances, client is also likely to be clear that goal was pursued
4 Pursued substantially	Goal was clearly pursued for an extended period of time, either in a block or throughout the session
	Substantial attempts were made to pursue goal. This might involve efforts to acquire information of interest or to make sure that important issues were clear with the client
	Generally five minutes or more, although it could be less—minutes are only a rough guide
	Coder is certain goal was pursued
	Unless there are very unusual circumstances, client is also likely to be clear that goal was pursued
	Not quite a 5 rating because the therapist either did not spend an extended period of time on the goal (probably 8 or more minutes, although this is only a rough guide), or because the approach to pursuing the goal did not incorporate examples or specific kinds of details present in the code definition (if these are present)
5 Pursued intensely	Goal was clearly pursued for an extended period of time, either in a block or throughout the session
	Substantial attempts were made to pursue goal. This might involve efforts to acquire information of interest or to make sure that important issues were clear with the client
	Generally eight minutes or more, although it could be less—minutes are only a rough guide
	Coder is certain goal was pursued
	Unless there are very unusual circumstances, client is also likely to be clear that goal was pursued
	If examples or specific details are present in the code definition, the therapist’s actions should be very similar to or cover some of the details in the definition

Note Therapists utilized the rating scale in the first column, including labels. Observers used the same scale but also had descriptions in the second column to inform their intensity ratings

Observer Ratings of Goals and Strategies Pursued

The CTPRS observational coding system allows coders to identify goals and strategies pursued continuously while watching treatment sessions. Coders watched the digitized therapy sessions on a computer using software developed by the principal investigator. While viewing treatment sessions, coders noted goals and strategies pursued by the therapist whenever they appeared. The coding software paused every 2.5 minutes to allow coders to review all CTPRS codes. At the end of the session, the coding software provided a summary report of all goals and strategies coded during the session, the time intervals in which they occurred, and a text summary of qualitative notes taken by the coder viewing the session. For all goals and strategies coded at least once during a session, coders then rated on a 1 to 5 scale the degree to which each was pursued. Observational coders were provided with a further rubric to help anchor their ratings more concretely, as described in the second column of Table 3.

Analysis

Reliability of Observer Ratings

Reliability of the coding system as a whole was established using an intraclass correlation coefficient. All 63 sessions were coded independently by two separate individuals. The intraclass correlation summarizes the proportion of total variation in the intensity ratings attributable to sessions. The ICC is sensitive both to the correspondence between coders’ ratings as well as differences in overall scaling on the intensity ratings. Because the sample of tapes available with both therapist and coder information was relatively small for establishing the reliability of individual codes, an intraclass correlation coefficient was used to estimate the reliability of the coding system overall, based on the 63 double-coded sessions from this work. Then, reliability estimates for individual CTPRS codes are reported from the larger CATS study that continued to use the observational CTPRS with another 258 double-coded therapy sessions, utilizing the same coding procedures.

Goals and Strategies Pursued

Further analyses rely on descriptive statistics to summarize the frequency with which each of the goals and strategies was reported, from both therapist and observer perspectives. The total number of goals and strategies per session reportedly pursued at intensity levels of greater than or equal to 1, 2, 3, and 4 were calculated for ratings by observers and therapists. Correlations, both intraclass and Pearson, were used to compare the total number of goals and strategies pursued as identified by therapists and coders.

For specific goal-target combinations, the percentage of sessions having intensity ratings of 2 or greater and 3 or greater are reported for both rating sources. Ratings of 1 or greater are also reported for observer reports. Kappa statistics were used to examine the correspondence between ratings made by therapists and coders for each specific goal-target combination in the CTPRS.

All work conducted was reviewed and approved by the Institutional Review Board at Rady Children’s Hospital, San Diego.

Results

Participants

Children who were the subjects of services averaged just over 11 years old. The diagnoses recorded in medical records were most often ADHD and disruptive behavior diagnoses, although 33% also had a mood diagnosis listed. Children receiving services had scores on the CBCL and Conners that were well into the clinically significant range, with average T-scores much higher than the clinical cutpoints on all subscales other than the CBCL internalizing subscale, consistent with behavioral concerns being primary reported reasons for receiving services.

CTPRS Reliability for Observational Coders

An intraclass correlation was calculated using goal-target intensity ratings from two coders in each of the 63 double-coded treatment sessions to estimate the reliability of the CTPRS as a coding system. For each session, all 39 goal-target combinations were included in the calculation, with 2457 goal-target pairs (63*39) contributing to the ICC computation. For this overall metric of CTPRS reliability, the ICC was .69, revealing good reliability across the coding system. A significance test could not be used for this broad metric because multiple observations were included from each session. However, tests of significance were used to examine individual code reliability.

The subset of treatment sessions having data from both therapist and coder perspectives was relatively small for the purpose of estimating reliability for individual CTPRS goal and strategy codes. However, Table 2 contains reliability estimates from the larger CATS study that continued using the CTPRS, calculated from 258 double-coded treatment sessions. In total, 76% of individual CTPRS codes had good reliability, with ICCs above .5.

Total Goals and Strategies Pursued

Therapist reports were unavailable for 12 of the 63 total sessions analyzed. Indicators of correspondence between therapist and coder ratings were calculated based on the 51 sessions in which both types of ratings were available. At a global level, therapists reported pursuing more goals in each session than reported by independent coders. Table 4 shows the average number of goal-target combinations per session receiving an intensity rating of at least 1, 2, 3, and 4 for both reporting sources. Summary statistics are presented for the child-focused goal-target combinations, caregiver focused goal-target combinations, and the overall number of goal-target combinations per session. For example, according to coders, on average a treatment session contained 4.8, 2.1, and 1.0 goals and strategies pursued with an intensity of 1, 2, and 3 or higher respectively. Similarly, according to therapists, on average each session contained 11.8, 10.9, and 7.3 goals and strategies pursued with an intensity of 1, 2, and 3 or higher respectively. With paired t-tests, the average number of goals pursued per session recorded by therapists and observers were significantly different for comparisons at all threshold levels, with therapists reporting more goals pursued than recorded by coders for caregiver-focused, child-focused, and total goals.

Table 4.

Average number of CTPRS goals/strategies reported per session by therapists and observers for different intensity thresholds

Goal/strategy types	Intensity Threshold
	≥1	≥2	≥3	≥4
Child-focused
Therapist	6.5a	6.1a	4.0a	1.6a
Observer	3.2b	1.5b	0.7b	0.1b
Caregiver-focused
Therapist	5.4a	4.8a	3.4a	1.9a
Observer	1.6b	0.6b	0.2b	0.0b
Total
Therapist	11.8a	10.9a	7.3a	3.5a
Observer	4.8b	2.1b	1.0b	0.1b

Note The average numbers of goals/strategies at or above each intensity threshold were compared for therapists and observers using paired t-tests. Subscripts within a column indicate averages that differed significantly from one another at the P < .05 level

Intraclass and Pearson correlations were used to examine the correspondence between therapist and observer reports with regard to the total number of goals and strategies pursued per session. For these correlations, goals pursued with an intensity rating of 1 or higher according to coders were counted, and those with a rating of 3 or higher according to therapists were counted. Intraclass correlations of the two reporting sources for the number of child focused, caregiver focused, and total goals were .40, .43, and .32 respectively. The intraclass correlations suggest low to moderate correspondence in the numbers of goals reported by the two sources.

Reports of Specific Goals and Strategies Used and Correspondence Between Rating Sources

Tables 5 and 6 summarize the frequency with which individual goals were pursued from the perspectives of therapists and observers for child-focused and caregiver-focused goals and strategies. The percentage of sessions in which each goal or strategy was pursued is reported for intensity levels of 2 or higher and 3 or higher for therapists, and 1, 2, and 3 or higher for observers.

Table 5.

Percentage of sessions in which child-focused goals were pursued according to therapist and observer ratings and correspondence between rating sources

Goal/strategy	Intensity threshold
	Therapist		Observer
	2+ (%)	3+ (%)	1+ (%)	2+ (%)	3+ (%)	Kappa
Information gathering	90	73	94	78	57	.02
-MH status, behavior, functioning
-Family structure
Information gathering	73	45	41	16	2	.36*
-Psychosocial stressors
Review progress	60	32	18	0	0	.01
Establish treatment/session goals	50	24	32	8	0	−.08
Social/communication/assertiveness skills	43	31	20	8	4	.39**
Experience/express affect	41	31	6	6	0	.12
Family member perspective	41	24	14	0	0	−.08
Identifying signs/cues for emotions	33	24	14	6	0	.55**
Information gathering	28	18	8	2	0	.40**
-High risk issues
Child roles	26	18	2	0	0	.17*
Problem solving skills	24	14	33	18	8	−.14
Stress reduction skills	24	12	0	0	0	–
Understand/differentiate emotions	22	14	8	2	0	.29*
Relaxation skills	22	12	6	0	0	.40**
Cognitive distortions	20	14	8	0	0	−.11
Develop safety or crisis plan	8	8	2	0	0
Appropriate medications	6	4	12	2	2	.47**
Formal supports	6	2	8	2	2	−.03
Informal supports	0	0	0	0	0	–

Note Percentages of sessions in which each goal / strategy was identified as present are reported for sessions rated by therapists (N = 51) and coded by observers (N = 63). Kappa coefficients were calculated dichotomizing the presence/absence of a goal / strategy using intensity cutpoints of 3 for therapists and 1 for observers due to differences in rates of report of goal/strategy presence

* P < .05, ** P < .01

Table 6.

Percentage of sessions in which caregiver-focused goals were pursued according to therapist and observer ratings and correspondence between rating sources

Goal/strategy	Intensity threshold
	Therapist		Observer
	2+ (%)	3+ (%)	1+ (%)	2+ (%)	3+ (%)	Kappa
Information gathering	48	30	10	6	2	.41**
-Psychosocial stressors
Responding effectively to positive/appropriate behavior	39	22	4	0	0	.09
Review progress	38	24	2	0	0	.12
Information gathering	38	21	33	18	8	.28*
-MH status, behavior, functioning
-Family structure
Family member perspective	28	24	16	10	4	.29*
Respond effectively to negative/inappropriate behavior	28	22	25	8	4	.63**
Establish treatment/session goals	26	14	20	4	0	.09
Learn about child skills	26	14	0	0	0	–
Monitor child behavior/improve supervision	24	12	2	2	0	–
Caregiver roles	22	16	2	2	0	.14
Caregiver consistency	22	22	0	0	0	–
Information gathering	22	22	6	2	0	.30*
-High risk issues
Effective commands/limit setting	20	14	6	0	0	.35**
Stress reduction skills	20	18	4	2	0	.13
Reward systems	16	14	6	4	4	.35**
Formal supports	10	8	8	2	0	.46**
Time out	10	10	4	0	0	−.05
Cognitive distortions	8	8	4	2	2	.30*
Develop/facilitate plan	8	8	6	2	0	.54**
Informal supports	8	4	2	0	0	−.03

Note Percentages of sessions in which each goal/strategy was identified as present are reported for sessions rated by therapists (N = 51) and coded by observers (N = 63). Kappa coefficients were calculated dichotomizing the presence/absence of a goal/strategy using intensity cutpoints of 3 for therapists and 1 for observers due to differences in rates of report of goal/strategy presence

* P < .05, ** P < .01

For each CTPRS goal and strategy, therapist reports of frequency and intensity ratings were consistently higher than ratings by observational coders. For example, therapists reported gathering information about psychosocial stressors at an intensity level of 3 (moderately) or higher in 45% of sessions, whereas observers recorded this as occurring at an intensity of 3 or higher in only 2% of sessions. Similarly, therapists reported working on helping parents develop more effective responses to negative behavior in 22% of sessions at an intensity of 3 or higher. Observational coders reported observing this in 4% of sessions at an intensity of 3 or higher.

For each goal in Tables 5 and 6, a Kappa statistic is reported summarizing the association between observer and therapist reports. The kappas reported are based on dichotomizing observer reports and therapist reports regarding the presence/absence of each goal and strategy. To understand the level of correspondence that might be present, cutpoints of 1 and 3 were utilized for observer and therapist reports respectively because therapists tended to report much higher levels of intensity. Eight of the 39 (20.5%) goal-target combinations had Kappa statistics of .4 or greater, indicating moderate agreement about the presence/absence of the goal or strategy in sessions, but 25 Kappa statistics (64.1%) fell below .4, indicating relatively low correspondence between therapist and observer reports. Kappa statistics could not be computed for another 6 goal-target combinations (15.4%) because observers did not record the presence of these goals as occurring with an intensity level of 1 or greater in any of the 51 sessions. In general, therapist / observer agreement for these specific goal-target combinations is estimated to be low because there were no areas of correspondence between therapist and observer reports with respect to the presence of these goals and strategies.

Discussion

This paper examines the correspondence between observer and therapist reports of strategies used and goals pursued when working with families presenting at least in part due to concerns about disruptive behavior problems. It is unique in having used two different approaches to assess treatment process. From a methodological standpoint, comparison of results from the two approaches provides important information about their potential strengths and limitations. Beyond addressing methodological questions, however, comparison of results from the two sources highlights characteristics of CB services that may have important implications for improving clinical care and implementation of innovative services.

Although quite different in character, both the therapist and observer-based measurement strategies yielded detailed information about treatment process. One of the most striking findings was the large difference in intensity ratings between the two sources, across therapeutic goals and strategies. When therapists reported having pursued a particular strategy or goal, they tended to report having pursued it with much greater intensity than identified by observational coders.

The magnitude of difference between therapist and coder ratings of intensity devoted to specific treatment goals was substantial. Even when a cutoff of 4 or higher (pursued substantially) was used for therapists’ ratings, the frequency of occurrence for most goals/strategies was higher according to therapist report than according to coder identification using a lower threshold of 2 or higher (pursued minimally). Consistent with these results, therapists also reported pursuing many more goals and strategies per session than identified by observational coders.

The large differences in intensity ratings do have potentially significant clinical and service implications. For example, the results suggest that therapists perceive that many more goals and strategies are pursued in each treatment session than seen by observers. If the perspectives of observational coders are similar to those of clients, who must also use the therapist’s words and other nonverbal cues to judge time and effort devoted to specific goals and strategies, clients may not be clear about many of the goals that therapists report pursuing.

To the degree that therapists’ perceptions of session focus do in fact differ from that of observers, there are several possible kinds of impacts. If a therapist perceives that a goal has been pursued, it may limit the therapist’s further efforts to pursue the same objective. For example, if a therapist perceives that sessions have already involved substantial work on developing a child’s social, communication, or assertiveness skills, he/she might spend less time or energy pursuing that objective in future sessions. Therapists might also attempt to build upon goals that they perceive to have been a focus in previous sessions, whereas a client might not have registered it as similarly significant. If observers do not perceive goals as having received the same level of focus, it seems appropriate to wonder whether the same is true for clients. Gaps between therapist perceptions and client perceptions about clinical focus have the potential to make it more difficult for clients to know how to approach and achieve desired changes.

At a broader level, if therapists do tend to overestimate the intensity with which goals and strategies are pursued in sessions, this could also impact efforts to improve care quality. For example, clinical supervision is a core component of efforts to continuously improve the care provided by recently trained therapists (Weersing et al. 2002). In many circumstances, staff receiving supervision may comprise 40% of service staff in a CB organization (Hurlburt et al. 2002). Supervision frequently relies heavily on verbal discussion of how cases are progressing and of possible clinical approaches to take, rather than on direct or video-based observation of therapist practices (Hurlburt et al. 2002; Weersing et al. 2002). Because supervision in many settings does rely heavily on self-assessment of session content, therapist reports may be insufficient to serve as a principal basis for guiding the supervision process, which is one of the primary avenues for potential ongoing improvement in therapeutic services.

Therapist self-perceptions that treatment intensity is already high could also have impacts on therapist attitudes towards the potential effectiveness of more structured, evidence-based practices. One hallmark of many evidence-based practices is that they tend to focus on a limited set of treatment goals and strategies across an extended series of sessions, addressing each goal with high intensity across several sessions rather than many goals in each session. Parent training programs illustrate this. Many such programs focus intensively on a relatively limited set of goals such as improving parent-child relationships by helping parents to know how to play more effectively with their children, developing parents’ structured skills for rewarding desired child behaviors, helping parents to master strategies for responding effectively to challenging behaviors (e.g., using ignoring and time out effectively), with some focus also on adjusting parents’ developmental expectations (e.g., Hembree-Kigin and McNeil 1995; Webster-Stratton 2001). Similar short lists of focal goals can be enumerated for other treatment approaches, such as problem solving skills training or treatments for anxiety or depression. A number of studies have found that therapists tend to perceive manualized interventions as not having sufficient breadth to address the complex needs of families served in CB mental health care settings (Addis and Krasnow 2000; Hurlburt et al. 2003). To the degree that therapists perceive the services they already provide are addressing multiple goals with relatively high intensity, many of those goals being consistent with goals articulated in EB treatment manuals, therapists may reasonably view manualized protocols as narrow, possibly having limited applicability, and insufficiently adaptive to the needs of children and families (Kramer and Burns 2008).

Beyond overall differences in intensity ratings, this study revealed that the two different measurement strategies had modest correspondence with regard to the presence/absence of treatment goals and strategies in individual sessions. For some goals and strategies, agreement was relatively good, such as establishing treatment/session goals with children, identifying signs/cues for emotions, developing a safety or crisis plan, helping caregivers to respond effectively to negative/inappropriate behavior, and helping caregivers to develop reward systems. For other goals and strategies, agreement was lower, even for some that would seem to be relatively concrete and observable, such as helping caregivers to use Time Out effectively, reviewing treatment progress with caregivers, identifying and counteracting cognitive distortions with children, and reviewing treatment progress with children. Reasons for differences in agreement across codes were not clear. What was apparent was that agreement tended to be higher when a low threshold was utilized to identify the presence of a goal from the observational perspective and a higher threshold was used for the therapist perspective.

Questions may certainly be raised about the ability of an observational coding approach to accurately classify treatment process, some of which are discussed in the limitations section below, but we concluded from these results that it is likely to be difficult to evaluate the content of mental health services through therapist report without providing therapists with more extensive training to help develop an ability to reflect systematically on the goals and strategies pursued in individual sessions and on the level of time and effort devoted to them. From the perspective of supervision and improvement of service quality, such training might indeed be worthy of further development, but was not within the scope of this study.

Limitations

A number of limitations should be considered in this study. First, the sample size of therapists and sessions was relatively small and based on volunteers into a pilot study. Despite this fact, differences between the two measurement approaches were substantial and consistent across therapists, suggesting that how therapists perceive the foci of their sessions is worthy of further study for many of the reasons already discussed.

Second, the tasks performed by therapists and observational coders were not identical. Therapists reflected on an entire session, at the end of the session, and then decided which of the items on a checklist reflected what they had attempted to accomplish. Sometimes this occurred several days after the session had taken place. Items on the therapist checklists were short prompts as described in Table 2. The instructions in the therapist checklist did provide clear examples regarding how to make intensity ratings, but the overall level of training regarding rating of sessions and the time devoted to making such ratings was quite different from that of observational coders. Coders watched every minute of treatment sessions, assigned codes throughout, and then made intensity ratings at the end, having access to notes they had taken throughout coding a session. Definitions used by coders were more detailed, designed to assist coders in making discriminations during the coding process. Differences in the demands of the tasks may contribute to explaining the lower concordance between therapists and coders regarding whether specific goals and strategies were pursued.

It seems likely that if the tasks of therapists and coders were made more similar, with therapists watching their own sessions and then rating content, that higher levels of correspondence would be observed. For this pilot study, however, we were interested in the degree to which the perceptions of therapists about session content, without adding such intense video review, would correspond with an observational coding approach. Both approaches were designed to assess comparable treatment process categories with the intent of characterizing services in CB mental health settings at the session level. Differences that appear, particularly regarding session intensity and number of goals pursued, provide valuable information about perceptions of goals and strategies pursued in each session.

Third, beyond true differences in the intensity of focus in each area, there are a number of potential explanations for the large differences in the two measurement approaches. For example, the wording of some intensity ratings could have been viewed as having a negative valence. Therapists might have felt awkward reporting that any goal or strategy was pursued “fleetingly” or “minimally” in a session. This could have compressed therapist ratings into the higher intensity levels. However, even if this did occur, it would not explain the much higher frequency with which therapists reported that specific goals and strategies were pursued at all, especially given that therapist checklists provided instructions not to make any intensity rating unless goals or strategies had been pursued for at least two or more minutes. No such threshold was present for observational coders, allowing them to code behaviors that were present for only very brief periods, yet substantial differences in the occurrence of almost every goal and strategy were evident.

Finally, the nature of the coders’ task could have reduced their ability to rate the presence/absence of treatment goals and strategies. Observers were trained to code therapist activity with respect to any treatment strategy or goal to which it could apply. Coders had to monitor for many different goals and strategies simultaneously, which could have led them to miss coding some that were present. To some degree, the solid inter-coder agreement ICCs suggest that this is unlikely to be true to a strong degree, but this cannot be completely ruled out as a source of difference.

Conclusions

The results of this pilot study suggest that therapists and independent observers may have quite different perceptions of the goals and strategies pursued in CB mental health care settings. To the degree that independent observers do capture valid information about the amount of time and effort devoted to specific treatment goals and strategies, these results suggest that large differences likely exist between the intensity and focus on specific goals pursued in CB therapy sessions and in controlled trials of evidence-based practices. From the observer perspective, sessions rarely contain individual goals that are pursued with the level of intensity and focus that would be observed in an evidence-based practice. This could readily contribute to clients having difficulties understanding where to make changes—cognitive, family-systemic, or behavioral—or having insufficient knowledge or direction to accomplish such changes.

From the perspective of therapists, large differences with evidence-based practices also exist. Therapists report pursuing many goals and strategies per session with high levels of intensity. Most EBPs emphasize focusing on a small number of goals and strategies with high intensity in any given treatment session to help ensure that clients understand and can make changes in targeted areas. From either perspective, therapist or observer, treatment process in CB mental health care seems to differ to a large degree from the structure of EBPs.

To the degree that evidence-based practices can improve outcomes in CB settings, this study suggests that one of the avenues through which this would likely occur is an increase in the intensity and focus devoted to specific goals/strategies in each treatment session. The study also suggests that finding new ways to help therapists develop more accurate perceptions of goals and strategies pursued may be worthwhile, both to assist in clinical supervision for trainees and to increase understanding of the differences between CB services and most evidence-based practices. Finally, although much of the current debate about mental health care quality improvement centers on adoption and implementation of specific evidence-based practices, our results raise questions about other possible pathways for improving service outcomes. Enhanced supervision procedures that focus on improving the intensity and focus in each session on goals and strategies that are common across EB practices might contribute to improvements in care for a broader range of clients and help therapists to understand differences between existing service practices and those of EB programs (Chorpita et al. 2005; Bearsley-Smith et al. 2008; Garland et al. 2009). Several different authors have now independently summarized results of efforts to identify common elements of EBPs. The frameworks have much in common and could serve as the basis for such efforts.