Examining Therapist Comfort in Delivering Family Therapy in Home and Community Settings: Development and Evaluation of the Therapist Comfort Scale

Abstract

This study reports on the development and psychometric properties of a new measure assessing therapist comfort in the home treatment context, and the relationship between therapist comfort, related process variables, and therapist characteristics. Data were drawn from a longitudinal evaluation of 185 families treated by 51 therapists using Multisystemic Therapy (MST). Therapist comfort was measured at four time points. Psychometric evaluation indicated that the measure was internally and temporally consistent. Examination of the measure’s validity indicated that therapists’ feelings of safety and comfort during the provision of home-based treatment were associated with family neighborhood characteristics and family socioeconomic factors. Furthermore, the therapist’s reported level of alliance (as measured by the Emotional Bonding subscale of the Working Alliance Inventory) was related to her/his feeling of comfort. Analyses also indicated that therapists with greater belief in the clinical utility of the MST model felt more comfortable when delivering MST. Together the results suggest that economically disadvantaged families treated in home and community settings may be most at risk for erosions in the therapeutic relationship over time as a function of lower therapist comfort. Because therapist comfort was associated with therapeutic alliance - a factor found to be associated with clinical outcomes across studies and treatment models - findings imply that psychotherapists should regularly examine their own level of comfort, especially when providing services in non-traditional settings, and that therapist comfort should be routinely assessed as part of clinical supervision and training.

Providing mental health services in community and home settings is an increasingly common practice of mental health professionals in community agencies (Bruns, Burchard, & Yoe, 1995). Over the last few decades home-based mental health services has been a growing and effective option for treating serious and chronic problems including juvenile delinquency (Henggeler, Melton, & Smith, 1992), adolescent substance abuse (Henggeler et al., 2006), adolescents with poorly controlled Type II diabetes (Ellis et al., 2006), adolescent obesity (Ellis et al., 2010), and HIV positive youth who are nonadherent to their antiretroviral medications (Cunningham, Naar-King, Ellis, Pejuan, & Secord, 2006). One particular treatment that uses a home-based model of service delivery is Multisystemic Therapy (MST; Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 2009). Almost three decades of research and 19 randomized clinical trials (with at least six independent investigators) have validated MST’s effectiveness with youth with serious clinical problems (e.g., chronic juvenile delinquency, substance abuse, severe medical regimen nonadherence) (Henggeler, 2011). Despite widespread use of MST and other empirically-supported treatments (ESTs) that use a home-based model of service delivery, factors related to the effective implementation of this treatment model have not been widely examined and merit further research.

Historically, the EST practice movement has focused primarily on codifying treatment elements and treatment fidelity, and on minimizing therapist variability in implementation of clearly described treatment elements in treatment manuals (e.g., Lebow, 2006). This focus, however, has failed to appreciate and control for the evidence showing that major differences in therapist effectiveness exist even in well-researched and manualized treatments such as the National Institute of Mental Health (NIMH) Collaborative Depression Study (Elkin, 1999). Studies that focus on therapist effects have sometimes demonstrated that therapist variables (e.g., therapeutic alliance) account for more variability in treatment outcome than do treatment-specific factors (Kim, Wampold & Bolt, 2006). The importance of therapist relationship factors led Division 29 of American Psychological Association to develop a Task Force on Empirically Supported Therapy Relationships to identify the evidence-based relationship variables that affect adult treatment outcomes (Norcross, 2002). This task force concluded that effective therapeutic relationship variables included goal consensus and collaboration, the therapeutic alliance, and therapist empathy.

MST is a family-based manualized treatment (Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 2009) that targets a wide array of risk factors with interventions that are individualized to meet the idiosyncratic needs of each youth and their caregivers. MST interventions integrate empirically-supported clinical techniques (e.g., family therapy, behavior therapy, cognitive-behavior therapy) into a broad-based social ecological framework (Bronfrenbrenner, 1979), that addresses relevant risk and protective factors across individual (e.g., positive attitudes toward delinquency and drug use), family (e.g., poor monitoring, inconsistent or lax discipline), peer (e.g., association with drug-using and/or delinquent peers), school (e.g., behavioral problems at school, truancy, academic difficulties), and community (e.g., availability of weapons and drugs, high instability and psychosocial stress) systems. A clear focus of these interventions is to promote behavioral changes in the youth’s natural ecology by providing caregivers with skills and resources to effectively address the difficulties that inevitably arise in raising adolescents.

MST is usually provided by master’s level therapists who participate in an intensive and standardized quality assurance system (Henggeler & Schoenwald, 1998; Henggeler, Schoenwald, Liao, Letourneau, & Edwards, 2002) designed to maintain therapist fidelity to MST’s nine treatment principles (see Henggeler et al., 2009). The MST theory of change is that high therapist adherence to MST results in improved caregiver and family functioning (e.g., parental discipline, monitoring, and family relations), which decreases an adolescent’s association with deviant or delinquent peers, which in turn leads to decreased antisocial behavior (Huey, Henggeler, Brondino, & Pickrel, 2000). Because MST therapists provide services in home, school, and/or neighborhood settings at times convenient to the family (usually during evenings and weekends when youth are prone to engage in antisocial behavior), and are available to respond to clinical problems 24 hours a day, 7 days a week, MST therapists frequently provide services in the family home and in the community. Many MST families come from highly disadvantaged backgrounds, live in poor neighborhoods, and have a wide array of individual and relationship problems. Because of these circumstances, and because many therapists have limited training in delivering therapy in the home, MST therapists may experience varying degrees of discomfort with providing services in the community.

Although studies have not empirically examined therapist comfort in delivering services outside of clinic settings per se, the role of therapist factors more generally has received considerable attention in previous treatment outcome research. Therapist factors that have been found to contribute to differences in treatment outcome include demographic characteristics (e.g., gender, age, ethnicity), personality characteristics, training, experience, and skills and competencies (e.g., Blow, Sprenkle, & Davis, 2007; Huppert et al., 2001). Both clinicians and researchers have acknowledged that establishing a strong therapeutic alliance is an important therapist skill across different psychotherapies (e.g., Friedlander, Escudero, & Heatherington, 2006; Kazdin, Marciano, & Whitley, 2005; Knobloch-Fedders, Pinsof, & Mann, 2007; Martin, Garske, & Davis, 2000). A strong therapeutic alliance has been shown to be related to changes in symptom distress in home-based family therapy (Johnson, Wright, & Ketring, 2002). However, relatively little is known about factors that thwart successful therapist alliance-building (Horvath, 2001) and therapeutic engagement (Foster et al., 2009) in the home-based context.

Recognition of the therapeutic alliance as an important aspect of successful psychotherapy, as well as therapist emotional contributors to alliance, has a rich history in various theoretical schools (e.g., Horwath, 2000). Bordin (1979) suggested a “pantheoretical” definition of working alliance, which consists of having an agreement on tasks and goals, and a mutual fondness, attachment, and trust (i.e., bonding) between the client and the therapist. Bordin’s concept highlights the collaborative nature of the therapeutic alliance. The relational bonding component emphasizes the therapist’s need to establish a strong and stable therapeutic relationship - one that is accepting, empathic, supportive, and genuine. Thus, it is not surprising that much of the early research on therapist factors focused on therapist personal characteristics that can differentially impact the therapeutic alliance, including such attributes as trustworthiness, confidence, openness and honesty (Ackerman & Hilsenroth, 2003). However, little research has systematically examined factors that contribute to therapist reactions to clinical material (Horwath, 2000).

Based on our own clinical experience and the limited research available, we hypothesized that a therapist’s feelings of safety and comfort during home- and community-based treatment would be an important factor in establishing the therapeutic alliance with parents and youth. One qualitative study (Thompson, Bender, Lantry, & Flinn, 2007) of parent and adolescent perceptions of engagement in home-based intervention reported that therapists who appeared calm and comfortable in the client’s home were better able to break down barriers between themselves and their clients successfully. However, it may be difficult for therapists to stay calm and open when they perceive the home environment as dangerous to themselves or their clients. Christensen’s (1995) qualitative exploration of therapists’ perceptions of home-based services identified the home environment and safety as primary concerns. Home factors such as visitors, loud music, and level of hygiene were identified as distractions from therapeutic process. Safety issues were described as making therapists cautious and overwhelmed. Similarly, a focus group study (Adams & Maynard, 2000) conducted with MST therapists and supervisors identified a number of challenges therapists experience in providing this evidence-based treatment. One of the most highly-ranked challenges revolved around crisis intervention and safety (primarily of clients), suggesting that therapist comfort may be particularly salient in the delivery of MST. Even with appropriate training and supervision, therapists working in more challenging home environment may feel and appear to their clients as more guarded, and less trusting and open. In summary, existing studies and clinical experience suggest that a therapist’s comfort in treating families (e.g., physical safety, safety of others) may be an important factor in providing successful home-based treatments (e.g., Cortes, 2004), and low levels of therapist’s comfort may interfere with establishing a strong therapeutic alliance.

Unfortunately, there are no established measures to examine the construct of therapist comfort. The present study reports on the development of a new measure of a therapist’s comfort and provides data on its psychometric properties. The Therapist Comfort Scale (TCS) was designed to assess therapists’ feelings of safety when delivering services in a family’s home and community. The TCS was developed based on qualitative interviews with MST supervisors and therapists to identify sources of therapist discomfort when treating families referred to MST. The TCS includes items addressing issues of physical safety in the neighborhood, comfort in the home, and comfort interacting with family members.

This study had three aims: (a) examine the scoring and reliability (item reliability, temporal consistency) of TCS scores; (b) examine the construct validity of the TCS by examining how these scores correlate with measures of therapeutic alliance and neighborhood characteristics. Because so little is known about therapist and client factors that might contribute to feelings of comfort with home-based service delivery, we also (c) examined whether client pretreatment demographic characteristics (youth age, youth and caregiver gender and ethnicity, ethnic match, Hollingshead scores, receipt of financial assistance), and therapist characteristics (attitude toward MST, perceived adequacy of supervision and training) predict TCS scores early in treatment as well as changes in therapist comfort during the course of treatment.

Data were taken from a longitudinal study of MST treatment in which therapist comfort was measured at four time points: early in treatment (Time 1; T1), twice during mid-treatment (T2 and T3), and at treatment termination (T4). Therapists’ overall perception of comfort was expected to increase over time due to habituation to environmental stimuli over the course of therapy. In addition, TCS scores were expected to correlate with measures of neighborhood characteristics and therapeutic alliance. As such, we predicted that therapists would report lower levels of comfort in more disadvantaged neighborhoods, and more comfortable therapists would report stronger emotional bonds (one aspect of therapy alliance) with caregivers than less comfortable therapists, regardless of phase of treatment.

We made no predictions about relationships between pretreatment client demographic characteristics (youth age, youth and caregiver gender and ethnicity, ethnic match, Hollingshead scores, receipt of financial assistance) and therapist comfort at Time 1, except that ethnic match between a therapist and a caregiver would be related to greater therapist comfort early in treatment, based on previous findings that ethnic match has been associated with better outcomes in MST (Halliday-Boykins, Schoenwald, & Letourneau, 2005). We also explored relationships between pretreatment therapist characteristics (attitude toward MST, perception of adequacy of training and supervision) and therapist’s comfort early in treatment. We expected that a positive therapist attitude toward MST as well as perceived adequacy of training and supervision would be related to greater therapist comfort during treatment, predicated on the assumptions that adequate training and supervision would help therapists deal with any discomfort they experienced, and that therapists who experienced more discomfort might also be more inclined to question the treatment model.

Method

Participants

MST therapists (n = 51) provided data on their treatment experiences with 185 families treated in the family’s home. Thirty-seven therapists (71%) were female; 44 (86%) were White, 1 was African American/Black, 2 were Latino/a, and 4 listed their ethnicity as “other.” Their mean age was 31 years (SD = 7.27). These therapists had spent a mean of 9.51 (SD = 17.35) months using MST when they enrolled in the study, and reported having received their highest degree on average 2.62 (SD = 2.96) years earlier. Eighty-five percent (n = 44) reported having a Master’s degree in such fields as Social Work (50%), Counseling (19%), Psychology (15%), Marital and Family Therapy (12%), and other fields (4%).

To be eligible for participation, MST therapists had to meet their agency’s hiring and training requirements and to have demonstrated compliance with standard onsite MST training. Standard MST training includes an initial five-day orientation, weekly onsite clinical supervision, weekly case consultation with an MST expert, and quarterly booster trainings.

Youth were referred for conduct problems to one of four participating agencies in an urban city in the western US (Denver, Colorado) with licensed MST programs. The majority of the youth (65.4%) were male with a mean age of 15 years (range, 12–17). Participating caregivers were mostly female (85.9%), with a mean age of 43 (range 25–73); 63.2% (n = 117) reported they were the youth’s sole caregiver. Sixty percent of caregivers reported a high school education or less and 41.6% (n = 77) reported receiving financial assistance. Socioeconomic status (SES) was calculated based on caregiver education and occupation using the widely employed Hollingshead SES scale (Hollingshead, 1975) and ranged from 6 to 58, M = 30.28, indicating that the average family was either lower middle class or middle class.

Slightly more than half of the caregivers were White (53%), with the remainder Latino/a (26%), Black (18%) or “other” (3%). Fifty-one percent of caregiver-therapist dyads were ethnically matched; 93% of these involved White therapists paired with White caregivers. Of those dyads who were not ethnically matched, 78% were White therapists seeing minority clients, 9 % involved Latino/a or “other” therapists providing services to white caregivers , and the remainder involved therapists from one minority group conducting MST with members of another minority.

Measures

The Therapist Comfort Scale (TCS) was designed to assess therapists’ feelings of safety and comfort when working with a family in its home and community. TCS items were developed based on semi-structured interviews in which MST therapists (n = 7) and supervisors (n = 3) were asked to identify specific instances when they or a supervisee experienced discomfort while treating families referred for MST (e.g., serious and chronic juvenile delinquents, substance abusing delinquents); the interviewer asked follow-up questions to identify specific descriptions of factors that elicited feelings of discomfort. The second and third authors independently read the interviewer’s written record of each interviewee’s responses and each developed a list of sources of discomfort that were mentioned at least once (by any interviewee). The same two individuals met, reviewed the lists, and consolidated the two lists into themes. Three distinct themes emerged. The first was focused on therapist concerns about their safety and well-being due to the neighborhood or home. For example, one respondent said,¹ “I was told that the apartment complex had been used as a crack house....People came in and out a lot, and there was a lot of reported drug use in the complex.” Another said, “I know the dangers of some of these neighborhoods. I’m spiritual so I pray when I’m going into a neighborhood that could be dangerous.” A second theme in therapist’ comments concerned the homes where they conducted therapy, “Some houses are so dirty, I have to bring something to sit on. I don’t want to sit on furniture,” “Cats walking around and crawling on me make me uncomfortable - I’m allergic.” The final theme reflected therapists’ discomfort in dealing with volatile or other negatively charged interactions in the home. One supervisor stated, “During a session a kid pulled pair of scissors and chased his mom across the room – this was a safety issue for the therapist.” Another described a situation in which a father made advances toward a female therapist. Therapists also sometimes described being uncomfortable when families signaled they were not happy to see the therapist, “One woman didn’t want me in her home - I felt so unwelcome.”

The second and third authors generated 10 TCS items around the three themes that emerged in from the interviews: (a) issues of physical safety in the neighborhood (e.g., “I feel safe in the neighborhoods I need to visit in treating this family”), (b) comfort in the home (e.g., “The cleanliness level of the family's home makes me uncomfortable”), and (c) comfort interacting with family members (e.g., “The ways family members interact in sessions make me nervous”). The appendix contains a copy of the TCS. The therapist rates each item on a 6-point Likert rating scale (1 = strongly disagree to 6 = strongly agree). Higher scores indicate greater therapist comfort.

Two instruments were used to assess the construct validity of the TCS, the Emotional Bonding subscale of the Working Alliance Inventory (WAI; Horvath & Greenberg, 1989) and the Neighborhood Rating Scale (NRS). The WAI possesses adequate psychometric properties (Martin et al., 2000) and predicts treatment outcome (Horvath & Greenberg, 1989). The 12-item Emotional Bonding subscale of the WAI assesses degree of emotional connection the caregiver or therapist experiences in the therapeutic relationship. Therapists rated their relationships with the primary caregiver using the Emotional Bonding items; caregivers likewise evaluated their relationships with the therapists. Scores were internally consistent for therapists (alphas = .82-.90 across time points) and caregivers (alphas = .80-.87). The NRS was developed as a measure of client’s neighborhood characteristics by adopting items from existing instruments (Rains, 2002; Wei, Hipwell, Pardini, Beyers, & Loeber, 2005) and was also used to assess the construct validity of the TCS. The NRS was completed by research assistants (RAs) based on their observation of the neighborhood as they drove to the family’s home. The NRS consists of 13 descriptions of neighborhood characteristics (e.g., “The presence of bars on windows,” “The presence of groups of unsupervised youth”) rated on a 3-point scale (1 = none, 2 = some, 3 = a lot). The RA also has an option to choose “not observed” for characteristics s/he cannot see (e.g., the visit is at night and it is too dark to see graffiti). Higher scores indicate a more disadvantaged neighborhood. The mean item score is calculated for all items that were rated. Internal consistency of these ratings (coefficient alpha) was .86 at time 1, .83 at time 2, .75 at time 3, and .80 at time 4. Correlations between NRS scores for adjacent time points for families who did not move ranged from .62 to .73, ps < .001. Means and standard deviations for therapist and caregiver WAI Emotional Bonding subscale scores and NRS scores at all time points are presented in Table 1

Table 1.

Means and Standard Deviations for Therapist and Caregiver WAI Emotional Bonding Subscale Scores, and NRS Scores at Four Time Points

	Therapist’s WAI	Caregiver’s WAI	NRS
Time 1	5.67 (.69)	-	1.20 (.25)
Time 2	5.63 (.88)	6.04 (.78)	1.20 (.22)
Time 3	5.77 (.87)	5.89 (.85)	1.20 (.19)
Time 4	5.80 (.89)	5.92 (.95)	1.21 (.20)

Note. Time 1 = early in treatment, Times 2 and 3 = midtreatment; Time 4 = termination.

Therapists completed a questionnaire designed to gather therapist demographic information and therapist experience delivering MST. Therapists also rated their attitudes toward MST using 17 questions adapted from Aarons’ Evidence-Based Practice Attitude Scale (EBPAS). The EBPAS is a self-report measure of attitudes toward adopting evidence-based therapy or interventions (Aarons, 2004). We modified items of the original instrument by changing the word “treatment” to “MST,” and added two additional items. Items were rated on a 1–5 scale: 1 = 'not at all,’ to 5 = 'to a very great extent.'

Because these modifications changed the meaning of EBPAS items in important ways, it was not clear whether Aarons’ subscales for the measure would still apply to it. To guide decisions about scoring, an exploratory principal axis factor analysis with varimax rotation was conducted using data from the 76 therapists initially enrolled in the study (some enrolled therapists did not see any families who qualified for and enrolled in the study and hence are not part of the sample used to validate the TCS). Results of this should be considered tentative in light of the small sample size and problems with fit², but nonetheless provided guidance for scoring. Specifically, a three-factor solution fit the data best and made conceptual sense, with 10 attitude items (e.g., “I like to use MST therapy/interventions to help my clients,” “ I would try a new MST intervention even if it were very different from what I am used to doing,” “MST is intuitively appealing”) loading on the first factor (lowest loading = .38, all remaining loadings > .50), and 3 items reflecting therapist ratings of the extent to which they believed they had sufficient training, supervision, and consultation to use MST effectively loading on the second factor (all loadings > .50). Three items related to MST being required by their agency loaded on the third factor but this factor was not relevant to the aims of this study; one item did not load on any of the factors and was dropped. The first two factors were conceptually relevant to the aims of this study and were used in analyses. We averaged item ratings (unit weightings) from the first factor as an index of positive attitudes toward the clinical usefulness of MST (α = .85, M = 4.00, SD = .51, range = 2.67 – 4.89). Items from the second factor provided an index of perceived competence delivering MST (α = .86, M = 3.85, SD = .94, range = 1 - 5). These two scores correlated r = .46, p < .001.

Caregivers completed a demographic questionnaire assessing gender, ethnicity, education, and receipt of financial assistance during the T1 assessment.

Procedures

Therapists completed questionnaires assessing demographic information and attitudes toward MST when they enrolled in the study. To collect information related to individual families and their treatment, research assistants administered measures to therapists, youths, and each youth’s primary caregiver. Participants completed measures soon after referral (T1, with the mean of 3.1 weeks from treatment start to caregiver assessment), twice during mid- treatment (T2-T3; with the means of 9.3 and 15.3 weeks from treatment start to caregiver assessment), and at termination (T4; with the mean of 22.2 weeks from treatment start to caregiver assessment).

Family measures were computer-administered; therapist measures were administered over the telephone or by computer. Therapists completed the TCS and the Emotional Bonding subscale of the WAI (Horvath & Greenberg, 1989) at the T1-T4 assessments. Primary caregivers completed the Emotional Bonding subscale of the WAI at the T2-T4 assessments. The research assistant who visited the family’s home to administer assessments completed the Neighborhood Rating Scale at T1-T4 assessments. Most homes were in Denver and its suburbs. Prior to completing any measures, caregivers and therapists signed informed consent forms and youth completed assent forms affirming their willingness to participate voluntarily in the study.

Therapists delivered MST according to quality assurance and improvement procedures employed by licensed MST programs worldwide. Detailed guidelines for implementing MST for serious antisocial behavior in youth can be found in Henggeler, Schoenwald, Borduin, Rowland, and Cunningham (1998, 2009). Critical features of MST include: (a) integration of empirically-based treatment approaches into a broad-based ecological framework that addresses a range of pertinent risk factors across individual, family, peer, school, and community contexts that directly or indirectly contribute to antisocial behavior (e.g., association with deviant peers, poor parental monitoring); (b) promotion of behavior change in the youth's natural environment, with a major focus of empowering caregivers by providing them with the skills and resources needed to effectively, but independently, address the inevitable challenges in raising teenagers; and (c) quality assurance and improvement system designed to support implementation of MST at multiple levels of implementation-therapist, supervisor, consultant, and organization. The specific components of the MST Quality Assurance and Improvement system (for a detailed description of this system see Schoenwald, 2008), include: (a) manualized treatment components for therapists, supervisors, consultants, and organizations; (b) on-site supervision and weekly case consultation from a distal MST expert; (c) quarterly booster trainings for therapists and supervisors; (d) ongoing program development and support, and (d) web-based implementation tracking using well-validated measures of fidelity for therapists, supervisors, and consultants (Henggeler et al., 2009).

No families dropped out of treatment, although treatment duration varied: 24 families completed treatment before the time 2 assessment (and thus only completed T1 and T4 assessments); 47 additional families completed treatment after T2 but prior to the T3 assessment. Treatment lasted on average 17.5 weeks (SD = 7.7). Differing number of participants at different time points resulted from early terminations, problems with computerized assessments, and therapist changes where the new therapist declined to participate. On rare occasions, families could not be located to be scheduled or declined a particular assessment. No families withdrew entirely from the study.

Results

Scoring and Reliability of the TCS

Because families were nested within therapists (i.e., therapists treated multiple families) and data could not be considered independent, analyses used multilevel procedures. Intraclass correlations for individual items indicated that multilevel procedures were warranted, with values ranging from .02 to .45 across the four time points; 78% of item ICCs exceeded .10.

To examine whether the TCS was better scored using a total versus subscale scores, we examined the factor structure of the TCS with two-level exploratory factor analyses (EFAs) (a “within” variable, families, was nested within a “between” level variable, therapists) at each of the four time points using Mplus (Version 5.21) software (Muthén & Muthén, 1998–2009). The primary concern with all analyses involved family-level results, which provide information about scale performance for individual families controlling for non-independence of families seen by the same therapist, so only family-level results are presented in this paper. An oblique rotation (with the geomin criterion) was used because there were no expectations that factors would be orthogonal. One, two, and three within (family-level) factors were extracted, allowing the between (therapist-level) models to be unrestricted (i.e., no factor structure tested at the between level). The results supported a three factor solution for all four time points, based on reductions in chi-square model fit statistics and improved RMSEA, CFI, and SRMR values.³ However, the same factor structure was not replicated over time, and some items at some time points had high significant loadings on two factors. Because the factor structure did not fully replicate, and in light of the small number of items that made up each factor (3 or 4), total TCS scores were used in subsequent analyses.

Analyses of total TCS scores indicated that conventional coefficient alphas (which do not take non-independence of therapists into account) ranged from .73 to .79. Internal consistency of the total scale was also calculated using HLM v. 6.08 employing the approach described by Raudenbush, Rowan, and Kang (1991) (to control for non-independence) to estimate family-level reliability at each time point and indicated reliabilities ranging from .66 to .74 across time points (see Table 2 for means, standard deviations, and reliability values for all time points).

Table 2.

Means, Standard Deviations and Reliability Values of TCS Total Scores at Four Time Points

	M	SD	HLM Reliability	Coefficient Alpha
Time 1	5.02	.65	.71	.77
Time 2	5.04	.69	.74	.79
Time 3	5.00	.61	.66	.73
Time 4	5.00	.67	.73	.79

Note. HLM reliability estimates calculated using HLM control for nonindependence of data; alphas do not.

Temporal consistency was also examined using multi-level correlations calculated by MPlus v. 5.21. Correlations from one time point to the next were high: .78, .70, .75 for T1-T2, T2-T3, and T3-T4, respectively; ps < .01, showing considerable stability in therapist ratings during the course of treatment.

Validation Analyses

Multi-level correlations examined relationships between TCS total scores and scores from measures of related constructs (see Table 3). As predicted, TCS scores negatively correlated with Neighborhood Rating scores at all time points, indicating that a therapist’s feeling of safety and comfort during home-based treatment was related to the neighborhood characteristics. TCS scores positively correlated with therapist WAI scores at all time points, suggesting that therapist’s level of alliance was related to her/his feeling of comfort. However, caregiver WAI scores significantly correlated with therapists’ level of comfort only at T4.

Table 3.

Multi-level Correlations of TCS Scores with NRS and WAI at Four Time Points

	Time 1	Time 2	Time 3	Time 4
Neighborhood Rating Scale	-.24**	-.27**	-.47***	-.26**
Therapist Working Alliance	.30**	.43**	.46***	.42***
Caregiver Working Alliance	n/a	-.02	.01	.20***

Note. Time 1 = early in treatment, Times 2 and 3 = midtreatment; Time 4 = termination.

^**p < .01.

^***p < .001.

Family and Therapist Predictors of Therapist Comfort

One aim of the study was to examine whether family and therapist characteristics related to therapist feelings of comfort early in treatment, as well as to changes in comfort over time. We examined this aim by using hierarchical linear modeling analyses (HLM v. 6.08 software). Initial HLM analyses indicated that the proportion of variance due to therapists was high (ICC = .62). Therefore, analyses used three-level models with time, family, and therapist as level 1, 2, and 3 nesting variables. Family-level variables (client demographics and ethnic match) were used as Level 2 predictors, and therapist variables (e.g., attitude toward MST, perceived supervision and training support, gender) as Level 3 predictors. Each predictor was examined separately. Predictors were modeled as fixed effects. Estimates were based on calculations using asymptotic standard errors.⁴ Predictors were grand mean centered. Results are presented in Table 4.

Table 4.

Fixed Effects Estimates for Family and Therapist Predictors of TCS Scores

	Intercept		Linear Slope
	Coefficient (SE)	95% CI	Coefficient (SE)	95% CI
Level 1 (family)
Financial assistance	-.31** (.09)	[-1.31, .69]	.02 (.02)	[-.10, .14]
Hollingshead score	.01** (.00)	[-.99, 1.01]	-.001 (.001)	[-.12, .12]
Youth age	.07 (.04)	[-.97, 1.11]	-.01 (.01)	[-.13, .11]
Youth gender	.09 (.10)	[-.97, 1.15]	.03 (.02)	[-.09, .15]
Caregiver gender	.15 (.13)	[-.89, 1.19]	-.04 (.03)	[-.16, .08]
Caregiver ethnicity:
White 0=non white	.04 (.09)	[-1.02, 1.10]	.002 (.02)	[-.12, .13]
Latino/a	-.04 (.11)	[-1.08, 1.00]	-.01 (.02)	[-.13, .11]
Black	-.08 (.13)	[-1.12, .96]	.02 (.03)	[-.10, .14]
Caregiver-therapist ethnic match	.01 (.10)	[-1.05, 1.07]	.02 (.02)	[-.10, .14]
Level 2 (therapist)
Attitude toward MST	.32** (.11)	[-.72, 1.36]	-.01 (.02)	[-.12, .10]
Perceived support	.07 (.07)	[-.99, 1.13]	.01 (.01)	[-.10, .12]
Gender	-.07 (.15)	[-1.13, .99]	.02 (.02)	[-.10, .14]

Note. CI = confidence interval. Gender was dummy coded with 0 = male and 1 = female. Ethnicity was dummy coded with 0 = comparison group and 1 = particular ethnicity (e.g., 0 = Non-White and 1 = White).

^**p < .01.

To decompose the variance components in TCS and WAI scores for each of three levels (time, family, and therapist) we calculated the intraclass correlation coefficients (ICCs) for each level. ICC measures the proportion of variance in the outcome variable accounted for by between group differences at the respective level. ICCs for TCS scores were .19, .62, and .19; for therapist WAI scores were .27, .39, and .34; and for caregiver WAI scores were .53, .48 and .00, for levels 1 (time), 2 (family), and 3 (therapist), respectively. The largest percentage of variance in TCS (62%) was accounted for by differences among families within therapists, suggesting that therapist comfort varied depending on family characteristics. At the same time, therapist differences accounted for 19% of the variance in TCS scores, indicating that some therapists were less comfortable with providing in-home treatment regardless of the specific families with whom they worked. Large amounts of variance in therapist ratings of emotional bond with caregivers were accounted for by family (39%) and therapist differences (34%), suggesting that therapist alliance was related to both family and therapist characteristics. In contrast, the large proportion of variance in caregiver alliance was at the family level (48%), with none at the therapist level. Because therapist differences accounted for a relatively large amount of variance in scores involving therapists as respondents, all analyses involving therapist-generated scores accounted for this source of variance.

Analyses revealed that across families, the slope of TCS scores was not significant, B = -.003, t (182) = -.33, p = .74, indicating that average therapist comfort with particular family did not change over time.⁵ Thus, habituation generally failed to occur across the course of therapy. There was, however, significant variability in slopes of TCS scores, χ² (177) = 279.66, p < .001, indicating that therapists’ patterns of comfort over time varied with different families. Thus, each predictor was examined in terms of whether it predicted TCS scores (intercept) at Time 1 and slope of change over time. The correlation between initial level of therapist comfort (intercept) and rate of comfort change over time (slope) was -.30, p < .001 suggesting that the therapists who started treatment with a higher level of comfort had a lower rate of change in their comfort over the course of treatment.

At the family level, both receipt of financial assistance (scored yes/no) and lower Hollingshead scores significantly predicted less therapist comfort in treating the family (intercept), B = -0.31, t (181) = -2.88, p = .005, and B = .01, t (181) = -2.81, p = .006, respectively. Youth and caregiver gender, youth and caregiver age, caregiver ethnicity (dummy-coded) and ethnic match between a client and a therapist were not related to TCS scores. At the therapist level, therapist reports of positive attitudes toward MST predicted TCS intercept scores, B = .32, t (49) = 2.70, p = .010. These results suggest that therapists with greater belief in the clinical utility of the MST model also felt more comfortable delivering MST at the beginning of treatment. Perceived training and supervisory support with MST did not predict therapist ratings of comfort. None of the other therapist or caregiver variables predicted patterns of change in therapist comfort over time (i.e., slope of comfort ratings) (see Table 4).

Discussion

MST and other empirically-supported treatments that use a home-based model of service delivery present unique challenges for therapists. These include issues related to therapist comfort with providing treatment in a client’s home and community. To date, however, no measures have been available to empirically assess a therapist’s comfort in delivering home-based treatment. The TCS was specifically designed to fill this gap by assessing therapists’ feelings of safety when delivering family services in community-based settings.

Psychometrically the TCS total score showed adequate reliability and validity, suggesting that this scale is a sound measure. As predicted, TCS scores significantly correlated with a measure of neighborhood characteristics completed by research assistants, particularly during midtreatment. TCS scores were also associated with therapist ratings of their emotional bond with caregivers they treated, regardless of phase of treatment.

Findings regarding therapeutic alliance suggest that therapists who are worried about their safety and welfare may find it difficult to engage fully in the therapeutic relationship. Interestingly, caregiver appraisals of the therapeutic relationship were not significantly related to therapist comfort until the end of treatment. Perhaps over time clients become increasingly aware of or unable to tolerate persistent therapist discomfort, which eventually impacts their own feelings of connection with the therapist. Alternatively, persistent therapist discomfort may erode the two-way therapeutic relationship sequentially over time, with therapists failing to engage fully and caregivers eventually responding in kind. Based on previous literature (e.g., Thompson, Bender, Lantry, & Flinn, 2007), these feelings and the associated diminished therapeutic bond might be expected to contribute to poor adherence to the model and to worsened outcomes. Testing whether this is the case is an important direction for future research.

Interestingly, as a group, therapists’ perception of comfort did not change over time. Thus, contrary to expectations, therapists did not seem to habituate to environmental stimuli over the course of therapy with a particular family. At the same time, therapists did show significant variability in their slopes of comfort scores, suggesting that some therapists, generally, became more comfortable over time in delivering MST to high-risk families in their homes while other therapists’ comfort declined or remained unchanged. None of our predictors could explain this variation, however. The statistically significant negative correlation between the initial level of therapist comfort and the rate of comfort change over time suggests that the therapists who started treatment with a higher level of comfort had a lower rate of change in their comfort over the course of treatment. Alternatively, however, this could indicate a ceiling effect (mean scores indicated generally high levels of comfort; around 5 on a scale that could range from 1 to 6). It could also be that those with low initial comfort were more likely to change over time due to regression to the mean.

In addition, analyses of therapist comfort and therapist alliance scores also indicated that therapist differences accounted for meaningful variance in comfort and alliance ratings, over and above variance contributed by individual families. This indicates that some therapists are generally more comfortable delivering home based services than others. Similarly, some therapists tend to rate all of their connections with clients higher or lower than others, at least in the context of MST. These findings could be an artifact of informant variance, an explanation supported by the finding that alliance scores reported by caregivers treated by the same therapist were uncorrelated. Alternatively, these findings might result from some therapists viewing the MST treatment model more favorably than others: positive attitudes toward MST predicted higher comfort ratings when working with families.

Although most caregiver, youth, and therapist demographic characteristics did not predict discomfort, ratings of neighborhood characteristics, receipt of financial assistance, and socioeconomic status were all significantly correlated with TCS scores. This implicates socioeconomic factors as particularly important in therapist reactions, more so than caregiver ethnicity or caregiver-therapist ethnic match in this study. Therapists were particularly likely to experience discomfort treating low-income families in poor neighborhoods. All participating therapist received standard MST training and supervision. However, the clear implication of these findings is that therapists who lack exposure to and comfort with low-income families may benefit from additional supervision and training that are specifically targeted at either reducing their discomfort or ensuring that their unease does not disrupt the therapeutic relationship or therapy processes. It is relevant to note that the MST therapists in our sample had on average a little over 9.5 months experience using MST and were only 2.6 years post completion of their highest degree. Future research is warranted to explore how a level of training and experience may be associated with a therapist’s level of comfort.

These results also highlight the unique challenges of treating high-risk, difficult-to-treat youth and their families. Many, if not all youth referred to MST programs (including the current sample) have had multiple law violations and failed treatment experiences. The parents of these youths, who may also have limited social and financial resources, frequently enter treatment demoralized and hopeless that their child’s behavior will improve (and often with the goal of youth placement outside of the home). Against this backdrop, a low level of therapist comfort may further weaken the potential for a successful intervention. For example, a therapist who has experienced a high level of anxiety traveling into a neighborhood where drug deals are openly made may not be able to focus her attention in the home based session on the family’s unique strengths that could be used to further the therapeutic alliance. One can imagine that in such a context, the therapists’ comfort will be further eroded if there is even a hint of “client resistance,” a term that is often antithetical to strength-based treatment models such as MST.

Results supported the prediction that therapists with more favorable attitudes towards MST also reported being more comfortable in delivering the treatment in community-based settings. However, the hypothesis about a relationship between perceived training and supervisory support with MST and therapist ratings of comfort was not supported. Although we assessed therapist attitudes prior to collecting comfort ratings with families, the relationship between these variables is correlational. Whether attitudes affect later comfort or, alternatively, therapists’ discomfort negatively impacts their attitudes towards MST over time can only be disentangled if both comfort and attitudes are studied longitudinally and simultaneously. Nonetheless, the correlation between therapist attitudes toward MST and comfort suggests that supervisors working with therapists treating families should explore the connections between therapist comfort in delivering treatment in home- and community-based settings and their general feelings about the clinical utility of the treatment they are employing.

The strengths of this study include the longitudinal examination of TCS scores within an effectiveness context in which practicing therapists were employed by licensed MST programs. Despite its contributions, this study was limited in that it only explored comfort in relation to delivering MST. Nonetheless, we would expect comfort and concerns for one’s safety and welfare to come into play whenever professionals intervene with families in home and community settings. In addition, initial factor analyses suggested that subcomponents of comfort may exist, although the small number of items almost assuredly did not capture the full breadth of the components of therapist discomfort. Future research might expand the number of TCS items to permit evaluation and refinement of possible subscales assessing components of discomfort. Another clinically important direction for future research is the examination of the relationship between therapist’s level of comfort and treatment outcome. This is an especially interesting question in the light of this study’s findings that the TCS scores were differentially associated with therapists’ and clients’ perceptions of therapeutic bond. It is not clear from previous studies whether client or therapist perceptions of alliance are more predictive of treatment outcome (see for example, Glebova et al., 2010; Kramer et al., 2008). Investigation of whether a therapist discomfort with delivering home-based services affects treatment outcome will be important for training therapists implementing those services. In addition, exploring whether discomfort reduces therapist adherence to manualized treatment would also be useful in future research.

One of limitations of this study is that youth’s perception of therapeutic bond was not included in the analyses because the caregivers participated in sessions more regularly that the youth did. Inclusion of both caregivers’ and youth’s therapeutic alliance measures is a very important direction for future research of therapist comfort and its relationships to therapeutic process as well as outcome in home-based treatments.

Practical Implications

The results of this study have practical implications for program administrators and clinical supervisors involved in hiring, assessment, and intervention. Program administrators of home-based mental health services should perhaps consider screening potential job applicants for their comfort in delivering interventions in home- and community-based settings. Obviously, hiring decisions should not be based solely on whether a person is “uncomfortable” in working with particular client populations in nontraditional settings. High levels of therapist discomfort, however, may suggest a need for additional training and support. Similarly, clinical supervisors should assess therapist comfort in delivering home-based services, particularly when there are concerns about client engagement and the therapeutic alliance (or ruptures in the alliance). Such assessments may entail querying the therapist specifically about his/her comfort level in working with a particular family. If the supervisor finds that a therapist is indeed uncomfortable working with a particular client in the home or community, interventions designed to improve the therapists’ comfort (e.g., exposure-based strategies, cognitive coping, relaxation training, bringing a colleague along) could then be implemented.

Our findings also have practical implications for therapists themselves. When treatment is not progressing, one issue that a therapist might consider is whether his or her comfort is a contributing factor. If a “self-assessment” reveals that the therapist is uncomfortable working in the home of a particular client (e.g., the home has a distinct order or has an infestation of roaches/lice, high level of verbal aggression/vulgarity), then the onus is on the therapist (and his/her supervisor) to develop strategies “in collaboration” with the family to “improve” the clinical climate/setting. For example, one therapist was very uncomfortable visiting a home that was very disorganized and unclean. The therapist suggested to the family that one way to help their son (with a disruptive behavior disorder) improve his behavior was to “help the family develop daily and weekly routines” to help him get “better organized.” Once the family agreed to this focus the therapists helped the family develop routines around meals, bedtimes, study times, and individual/family chores. Several sessions were devoted to “family organization” where the therapist modeled for the caregiver how to encourage the sons’ chore completion (cleaning his room and other rooms in the house). This therapeutic intervention not only improves parenting skills, but it has the additional advantage of potentially increasing the therapist’s level of comfort in the family’s home.

This study extended existing research on MST and other empirically-supported treatments that use a home-based model of services by identifying a new therapist process variable (comfort) that should be considered in evaluating process contributions to successful delivery of such treatments. Together the findings suggest that disadvantaged families treated in the home may be most at risk for erosions in the therapeutic relationship over time, possibly due to persistent therapist discomfort providing services in the family’s home and community. Furthermore, therapist attitudes toward the utility of an evidence-based intervention and their impact on therapist comfort require further scrutiny in clinical supervision and examination in research studies. Studies with larger samples of therapists who deliver different types of evidence-based treatment in family homes would be a useful future direction for work in this arena.