Early Identification of Treatment Failures in Short-Term Psychotherapy: An Assessment of Therapeutic Alliance and Interpersonal Behavior

Abstract

Early sessions of patients categorized as dropouts (n = 25), good outcome (n = 28), and poor outcome (n = 20) completers of a 40-session protocol of short-term psychotherapy were compared to determine predictive validity of in-session measures of therapeutic alliance and interpersonal behavior (Working Alliance Inventory, Session Evaluation Questionnaire, and Interpersonal Adjective Scale). A number of significant differences were found among the three groups: both patients and therapists in the dropout group rated the relationship as more problematic than those in the good outcome group, and patients in the dropout group also rated the relationship as more problematic than those in the poor outcome group, while therapists' ratings did not distinguish dropouts from poor outcome. Differences between good and poor outcome groups were nonsignificant. These findings have clinical significance, particularly in early identification of patients at risk for treatment failure.

Research on the therapeutic alliance has consistently confirmed the centrality of the alliance construct and its predictive value in the overall outcome of therapy, as well as in the ongoing process of therapy. Bordin,¹ in one of his last works, emphasized that the repair and working-through of strains or ruptures in the alliance is crucial to the process of change. While this is certainly not news to our colleagues in analytic circles, who have been writing about and working clinically with transference, defense, and alliance phenomena for many years, Bordin's transtheoretical language has enabled psychotherapy researchers to begin testing these multifaceted concepts empirically.

Existing Research

Gradually, focus has been shifting toward the study of therapeutic misalliance and the dimensions that predict problematic patient–therapist relationships and poor overall outcome. Such studies have been pursued as a means to more fully understand mechanisms of the change process. However, process studies often compare alliance and interpersonal behavior between good and poor outcome cases but do not include subjects who have prematurely terminated. As Bordin² has suggested, severe alliance ruptures could result in premature and unilateral termination by the patient. Therefore, it could be expected that premature termination cases would be characterized by more extreme ruptures and generally more problematic alliances than would cases resulting in poor overall outcome. Dropout conditions may provide an additional context within which to identify and study features of problematic therapeutic relationships. At a more practical level, the high rates of patient attrition from brief psychotherapy—which average about 47% and range as high as 67%^3,4—suggest that a substantial proportion of people seeking outpatient treatment are probably not receiving adequate care. Attrition rates have been found to be comparatively higher in patients diagnosed with personality disorders.⁵

Problems in Conducting Research

One reason why dropouts are often not included in psychotherapy process research studies may be the variability within this subsample and, hence, inconsistencies in how dropouts are identified. Failed treatments have been defined in the psychotherapy literature in a multitude of different ways, and this lack of consensus has likely contributed to conflicting and confusing research findings. In fact, the defining criteria are least reliable and valid for those patients who terminate therapy prematurely—as compared with patients who complete treatment protocols and have their outcome status defined by pre- and post-therapy ratings of symptoms and personality change.

Frayn⁶ has conceptualized two general categories of dropouts. He found that 50% of patients who dropped out of therapy did so within the first month; the rest of the premature terminators remained in therapy for a longer period of time but ultimately failed to complete the full course of treatment. Frayn observed that an early versus late pattern of attrition was also evident in the NIMH collaborative depression project,⁷ in which 44% of the premature terminators left therapy within the first month. He believes that the “early terminators” dropped out of therapy within the first month because of lack of motivation and the presence of a “powerful . . . negative transference . . . before a significant therapeutic alliance is available” (p. 258). Later dropouts, he hypothesized, are a more heterogeneous group who leave treatment for a variety of conflictual and environmental reasons. These percentages of early dropouts are consistent with rates reported by Garfield and Bergin,⁸ who noted that 50% of dropouts left therapy prior to session eight.

Wierzbicki and Pekarik³ conducted a meta-analysis on studies of premature termination, summarizing studies from January 1974 to June 1990, and found that dropout rates differed significantly as a function of the distinct definition used by each of the authors. In general, the criteria used to define dropout fell into three conceptually distinct categories: 1) termination by failure to attend a scheduled session, 2) therapist's judgment that termination was premature, and 3) low number of sessions attended. Without a reliable and valid definition of dropout, this sample remains a relatively heterogeneous one. For instance, a patient who accepts an unexpected job offer in another city early on in the treatment may have been completely satisfied with the therapy, but would be defined as prematurely terminating because of the small number of sessions completed. As Silverman and Beech⁹ found, subjects who terminated therapy prematurely often benefited from just a few sessions and should not be considered treatment failures based on the number of sessions attended or the disagreement of the clinician. Moreover, although the therapist's judgment may be one of the most valid methods of defining premature termination, it is also the least reliable, since therapists differ widely in the criteria they use for making such an assessment.^10–12

Besides inadequate definition, a second reason for the paucity of empirical studies including dropout samples may have to do with the difficulty in collecting data from these subjects. Patients who feel they are not receiving satisfactory treatment and are considering early termination may be less inclined to spend time completing self-report inventories for the benefit of the study. In support of this supposition, “missing data” was found to be a better predictor than a number of self-report indices in a pilot study that looked at the early identification of treatment failures in brief psychotherapy.¹³ Lack of overall outcome data may be a factor in the emphasis on pretreatment variables (such as demographic and descriptive data) in the literature on dropouts. (In contrast, an “intention-to-treat” model¹⁴ applied to psychotherapy research assumes early attrition and incorporates more frequent assessments into the procedure, so that regardless of where in the treatment protocol a patient withdraws, comparative data are available and the patient can be included in endpoint analyses. Measures collected after each session also allow for ongoing assessment of therapeutic process.)

In terms of pretreatment variables significantly associated with patient dropout, findings seem to most consistently demonstrate an inverse, albeit weak, relationship between dropout and socioeconomic status.^15,16 Wierz-bicki and Pekarik's meta-analysis³ confirmed this statistically. These authors also found that there was a significantly increased risk for dropout among minority patients and those with low levels of education. Although a few patient demographic variables have been reliably demonstrated to predict dropout, overall their clinical utility is rather limited. These global indices do not tell us much about the mitigating factors influencing the relationship between patient and therapist, how each of them thinks about the relationship, and specifically what is going wrong between them. As a number of researchers have now demonstrated empirically,^17–19 the types of interpersonal dynamics and processes that develop in the relationship between a particular patient and therapist will depend on their unique interaction.

Validity Studies

Research examining the validity of in-session alliance or process variables within dropout samples represents a small body of literature to date, and those studies that have been conducted present mixed and inconclusive results. Some studies have found that patient ratings of a variety of alliance measures at pretreatment²⁰ or early in treatment^21–23 significantly predicted dropout, whereas another showed no relationship between patient or therapist ratings of alliance after the first session and premature termination.²⁴ In contrast to their earlier findings, Tryon and Kane²⁵ found that therapist ratings of alliance were predictive of termination type, but patient ratings were not. The results of these dropout studies are inconsistent with the general finding that patients' assessment of the alliance, as compared with ratings by therapists and third-party observers, is the best predictor of overall outcome.²⁶ As mentioned above, the weak or contrasting findings in this literature may be the result of the different methods by which subjects were categorized into outcome conditions. As Wierzbicki and Pekarik³ have recommended, careful operationalization of criteria for defining premature termination is required.

Alliance Rupture Studies

An additional body of theoretical and empirical work that has addressed the issue of problematic therapeutic relationships is the alliance rupture literature. A number of researchers have been focusing simultaneously on the elaboration of aspects of alliance strains or ruptures, approaching the phenomenon from a variety of empirical vantage points. Initial studies were completed by Lansford and Bordin,²⁷ Lansford,²⁸ and Foreman and Marmar¹⁷ identifying poor-alliance cases and describing the components of repair and related improvements. The work of Safran and his colleagues has involved detailed qualitative and quantitative analysis of interpersonal-experiential therapy sessions, describing distinct rupture types and the processes by which these breaches in the alliance are resolved with the therapist.^29–31 Rhodes et al.³² have assessed subjects' retrospective reports of moments when they felt misunderstood by their therapists and how these events were either resolved or not. Interestingly, these different methodologies studied across different types of treatment have resulted in similar, although not identical, resolution models, providing some validation for the construct.

One of the methodological hurdles within this literature has been the definition and operationalization of the alliance rupture itself. Because different patient–therapist dyads play out subtly distinctive interpersonal dances, which also vary greatly in terms of intensity, their components have proven difficult to generalize and therefore to study in traditional, large-sample research designs. Two studies conducted at the Vanderbilt Project^18,19 identified cohorts of therapists who treated both good (high-change) and poor (low-change) outcome dyads and compared them with respect to interpersonal process variables as measured by the Structural Analysis of Social Behavior.^33,34 Overall, results demonstrated a significantly greater degree of hostile interpersonal interactions between patient and therapist in the poor outcome cases. A pattern of more frequent complex communications (defined as a speech utterance conveying more than one interpersonal message) was found in the poor outcome cases, although this finding was not statistically significant because of the small sample size. In the second study,¹⁹ which examined seven therapists, each with a good and a poor outcome case, it was found that hostile therapist introjects were linked to hostile interpersonal behavior and were characteristic of poor outcome dyads. It was demonstrated that “the same therapist, using similar techniques with similar patients, nonetheless might exhibit markedly different interpersonal behaviors in low-change cases as compared to high-change cases”¹⁸ (p. 30). These studies by Henry and colleagues^18,19 demonstrate the validity of ratings of the early therapeutic relationship in predicting differential treatment outcome, and they identify additional components of problematic alliances.

Process/Outcome Studies

To our knowledge, only two previously published studies of brief psychotherapy^35,36 have examined interpersonal process across the continuum of outcome possibilities: in other words, comparing dropout, poor outcome, and good outcome cases within the same design.

As part of the Vanderbilt I study, Hartley and Strupp³⁵ examined therapeutic alliance in a sample of male college students in a 25-session protocol. The subjects were divided into high outcome, low outcome, and premature termination groups depending on their status at the end of treatment. High or low outcome was determined by a composite score that included patient, therapist, and observer ratings of pre- and post-treatment assessment measures. Premature termination was defined as the completion of no more than five therapy sessions. These authors were not able to statistically predict premature termination on the basis of observer ratings of alliance; however, they noticed different patterns of alliance development among the three conditions. Specifically, the mean alliance scores for the dropout group increased in later sessions compared with the two completed-treatment groups, which showed a decrease in mean ratings. The dropout patients were observed “to be slightly less involved in the therapy,”³⁵ (p. 31) which may have served to increase the therapist's level of engagement in the process and accounted for the increase in the mean alliance rating for this condition. Hartley and Strupp note that the small sample of 6 dropout subjects may have accounted for the nonsignificant findings, as well as confounding “external” reasons why subjects terminated therapy prematurely.

The second study³⁶ was part of the Vanderbilt II project, which examined therapist effectiveness before, during, and after psychodynamic training. Najavits and Strupp³⁶ included patient “length of stay” in treatment as an outcome variable, in addition to a composite index of patient symptom change. This study looked at a much larger sample of outpatient psychotherapy subjects (80 patients and 16 therapists) and found that 25% of the correlations between length of stay and in-session ratings of therapist behavior were significant, compared with only 6% of the correlations between outcome (the composite index) and therapist in-session behavior. The authors also found that therapists defined as “more effective” demonstrated more positive behaviors in session, such as warmth, understanding, and helping, and showed fewer negative behaviors, such as belittling, ignoring, and attacking, compared with the “less effective” group. Furthermore, more effective therapists had no patients terminate prematurely (defined here as dropping out prior to session 16), whereas a minimum of 2 of the 5 patients of the less effective therapists prematurely terminated. Thus, therapist in-session behavior was demonstrated to be a significant factor in predicting which patients prematurely dropped out of treatment.

The purpose of the current study was to examine the predictive validity of a number of early, in-session indices of therapeutic alliance and interpersonal behavior in a sample comparing two distinct types of treatment failure conditions (premature termination and poor outcome) and a good outcome condition. In order to clarify the contradictory existing research findings regarding prediction of premature termination from in-session process variables, we attempted to create a more homogeneous and valid treatment sample by employing a more detailed and specific definition of premature termination than has previously been used in the research literature. We hypothesized that with this improved definition, the dropout cases would demonstrate significantly poorer therapeutic alliance scores and significantly more problematic interpersonal behavior patterns than the poor outcome cases, which in turn would demonstrate more problematic alliances than the good outcome cases. We also asked whether patients and therapists would differ with respect to the sensitivity and type of in-session measures of alliance and interpersonal behavior that might predict overall outcome.

Methods

Subjects

Seventy-three patients (43 women; 59%) from the Brief Psychotherapy Research Project, Beth Israel Medical Center, participated as subjects in this study. Their mean age was 40.71 years (SD = 9.36), most had obtained at least a college degree (68; 93%), most were white (66; 90%), and a majority of the patients were either single or divorced (54; 74%). After an initial intake interview, patients were randomly assigned to a therapist for 40 sessions of dynamic, cognitive-behavioral, supportive, or interpersonal-experiential treatment (see Therapists and Treatments section below).

In accordance with the project's inclusion criteria, patients were between the ages of 18 and 65 and described evidence of at least one close personal relationship. Exclusion criteria included evidence of current substance abuse; use of psychotropic medication (such as neuroleptics, antidepressants, or lithium) within the past year; a significant Axis III medical diagnosis; history of recurring psychotic or manic episodes; and history of suicidal, violent, or destructive impulse-control problems. Patients provided informed consent to the research protocol.

Diagnoses were formulated by the research team based on information collected from the Structured Clinical Interview for DSM-III-R–I and II (SCID I³⁷ and SCID II³⁸). Patient DSM-III-R³⁹ diagnoses for the sample were as follows: on Axis I, approximately two-thirds of the sample received a primary depression-related diagnosis (45; 62%), 18 (25%) received an anxiety-related diagnosis, 9 (12%) reported interpersonal problems, and 1 (1%) received an eating disorder diagnosis. Axis II diagnoses were categorized by cluster type, with the majority of subjects receiving either a cluster C diagnosis (27; 37%) or personality disorder not otherwise specified (NOS) with cluster C features (31; 42%). Three patients (4%) received a cluster A diagnosis, and 1 (1%) received a cluster B diagnosis. Eleven patients (15%) presented with no Axis II pathology.

The sample of 73 patients was divided into three groups: 1) dropout (n = 25), 2) completed treatment with good outcome (n = 28), or 3) completed treatment with poor outcome (n = 20). Dropouts were selected if they 1) voluntarily ended therapy within the first third of the protocol and had completed at least four sessions; 2) reported dissatisfaction with some aspect of the treatment and/or therapist and did not terminate exclusively as a result of factors beyond what is typically considered within the therapeutic frame (e.g., moved out of state because of a new job); and 3) had their therapist's acknowledgment of a problematic relationship. The 25 dropouts selected for this group completed a mean of 8.36 sessions (SD = 3.11).

Patients and therapists reported any problems or dissatisfactions with the treatment in response to an open-ended item on the Post-Session Questionnaire⁴⁰ (PSQ). Patient and therapist versions of the PSQ are included in Appendix A (part 1), Appendix A (part 2), Appendix A (part 3), and Appendix A (part 4) and are described in the Process Measures subsection below.

Within the time frame of this study (spanning approximately 3 years), 10 patients who terminated within the first third of treatment did not meet these criteria. The additional 10 dropout patients did not meet criteria for the following reasons: 1 felt much better after nine sessions and terminated because he had solved his presenting problem; 6 dropped out after only one session (1 reported this was due to “scheduling problems,” and the other 5 did not state reasons); and 3 reported having to terminate because they moved out of state. The method used to classify patients as either good or poor outcome is described in the Pre- and Post-treatment Assessment subsection below.

A comparison of patient demographic information among the three treatment groups (Table 1) was conducted by using Fisher exact tests (FI) or analyses of variance. Five variables were tested: age, race, marital status, gender, and education level. There were no significant differences on any of these variables, although a trend was noted toward a greater proportion of women in the dropout group and a greater proportion of men in the poor outcome group (FI = 5.94, df = 2, P = 0.051).

TABLE 1.

Other analyses using the Fisher exact test showed no significant differences on DSM-III-R Axis I and II diagnostic categories across the three groups. On Axis I, the frequency of primary diagnostic categories was as follows: good outcome: depression, 16; anxiety, 10; V codes, 2; poor outcome: depression, 12; anxiety, 5; V codes, 2; eating disorder, 1; dropout: depression, 17; anxiety, 3; V codes, 5 (FI = 7.22, df = 6, P = 0.30). Subjects' primary Axis II diagnoses were as follows: good outcome: cluster C, 15; NOS, 9; none, 4; poor outcome: cluster A, 3; cluster C, 5; NOS, 10; none, 2; dropout: cluster B, 1; cluster C, 7; NOS, 12; none, 5 (FI = 9.03, df = 6, P = 0.17).

Therapists and Treatments

Forty-seven therapists (25 women, 22 men) participated in this study. There were 19 M.D.s, 12 Ph.D.s, and 17 master's-level therapists, with a mean (± SD) of 7.51 ± 8.10 years of clinical experience. The therapists' mean age was 38.01 ± 8.20 years. Overall, the average number of patients per therapist was 1.55 (range 1–5), with no more than 4 patients seen by a single therapist within any outcome group. All therapists were white, and most were married.

Differences in therapist demographic variables were not tested among the three patient outcome groups because of the random occurrence of therapist overlap. In other words, a number of therapists (12; 26%) treated patients who fell within more than one outcome category. As a way to understand any effect of training on outcome, dyads were divided according to training status, with a therapist's first two treatment cases in a modality defined by the research project as training cases (regardless of years of experience). Therapists met weekly for group supervision and didactic seminars, with an additional weekly individual supervision provided for training cases. In fact, no significant difference was found among outcome groups with respect to therapists' brief psychotherapy training status. The number of training cases per condition was as follows: good outcome, 9/19; poor outcome, 4/16; dropout, 9/16 (FI = 1.43, df = 2, P = 0.49).

As mentioned under Subjects above, patients were randomly assigned to one of five manual-based, 40-session therapies. The treatments included two types of dynamic,^41,42 cognitive-behavioral,⁴³ supportive,⁴⁴ and interpersonal-experiential⁴⁵ psychotherapy. All therapy sessions were videotaped.

Pre- and Post-treatment Assessment

Patients completed the Symptom Checklist-90–Revised⁴⁶ (SCL-90-R), and the 127-item version of the Inventory of Interpersonal Problems⁴⁷ (IIP-127) at pre- and post-treatment. The overall mean score of the SCL-90-R was used as a measure of general psychiatric symptomatology. For the IIP-127, the overall mean score was used to reflect the severity of interpersonal dysfunction. Both of these measures have demonstrated adequate psychometric properties and are commonly used in psychotherapy research.

Those patients who completed the 40-session protocol were classified as having either good or poor treatment outcome based on reliable change (RC) scores,⁴⁸ calculated from IIP-127 and SCL-90-R scores obtained pre- and post-treatment. These measures provided an independent classification of treatment outcome. The means and standard deviations of the two self-report indices at intake assessment for the 1) dropout, 2) poor outcome, and 3) good outcome groups, respectively, were as follows: for the IIP-127, 1) 1.04 ± 0.47, 2) 1.49 ± 0.72, 3) 1.26 ± 0.43; and for the SCL-90-R, 1) 0.87 ± 0.51, 2) 0.96 ± 0.56, 3) 1.03 ± 0.50. A multivariate analysis of variance (MANOVA) indicated no significant difference among them (F = 2.16, df = 2,70, P = 0.12).

The RC index is a statistical approach that was used to evaluate the extent to which individual patients demonstrated change over the course of treatment. It is a method of assessing outcome that incorporates the predictive power of large-sample data with ideographic precision. Jacobson and Truax⁴⁸ identify two types of reliable clinical change based on a statistical differentiation between “recovery” (RC > 1.96) and “improvement” (change in the expected direction, but RC <1.96). In order to increase power, cases defined as either recovered or improved in this study were classified as having good outcome. However, we used a more rigorous definition of improvement, where an RC score of > 0.5 (half a standard deviation) but <1.96 was required. Therefore, cases were classified as poor outcome if they demonstrated change in the negative direction or if their improvement was minimal (RC <0.5). Each patient's RC index was computed by comparing the difference in scores on each instrument from pre- to post-treatment, relative to the standard error of difference (S_DIFF) between the two scores:

(RC=X₂ − X₁) ÷ S_DIFF

The S_DIFF was computed from the standard error of measurement (SE) derived from another, comparable sample of patients. The SE was derived from a sample of 118 patients who had completed the 40-session psychotherapy protocol and had participated in other research studies. These patients did not complete the PSQ and were therefore not included as subjects in the present study.

The RC scores from the SCL-90-R and the IIP-127 were averaged to derive a composite index. Using this method, 28 patients were classified as having good outcome and 20 as having poor outcome.

Process Measures

After each therapy session, patients and therapists independently completed equivalent versions of a post-session questionnaire⁴⁰ (PSQ; Appendix A (part 1, part 2, part 3, part 4). This questionnaire was made up of a number of different scales measuring aspects of the therapeutic alliance and interpersonal complementarity, including the Bond, Goal, and Task dimensions of the 12-item Working Alliance Inventory⁴⁹ (WAI-12; Part C on the PSQ), derived from Horvath and Greenberg's original 36-item scale;⁵⁰ Depth and Smoothness indices of the Session Evaluation Questionnaire⁵¹ (SEQ; Part D on the PSQ); and Friendliness and Hostility subscales of the Interpersonal Adjective Scale, short form^52,53 (IAS-S; Part E on the PSQ), developed from Wiggins and colleagues' revised scale.⁵⁴ Although the IAS-S has a total of 16 items, only the patient's ratings of the therapist and the therapist's ratings of the patient were used in this study. (Patients' and therapists' ratings of themselves were not included.) The scales included in this research study were selected because they had been found to be the most predictive of ultimate outcome in pilot studies with a similar patient sample.^13,55

On the basis of this previous research, and the high rate of missing data reported above, patients were closely monitored as to their questionnaire return rate. Prepaid envelopes were provided for patients to mail in completed forms if they were unable to stay and fill them out immediately after their sessions (they were instructed to complete the form as soon after the session as possible), and they were contacted by mail or telephone to inquire about any missing questionnaires. Also, the PSQ was described to patients at the beginning of treatment as a forum for their reactions to the therapist and the therapy program. They were encouraged to be open in their feedback and were reminded that the therapists did not have access to their PSQ responses.

A total of eight process variables, each rated by patients and therapists after every session, were included in the analyses. In order to examine and compare the nature of the therapeutic relationship across the three treatment groups, we identified a window of six early sessions. For the dropout group, mean PSQ scores from the six sessions prior to the last meeting were calculated (for instance, if the patient dropped out at session 9, scores were collapsed across sessions 4 through 9). As a comparison, the window in the good and poor outcome groups was defined as sessions 3 through 8, for three reasons: 1) the average last session in the dropout group was session 8 (which is also consistent with the existing literature in this area, described earlier); 2) it seemed to make clinical sense to give the relationship a few sessions to become established; and 3) it also seemed reasonable to give the patients some experience with the PSQ before collecting data. There is some empirical evidence for this last point: Marziali⁵⁶ found that alliance ratings collected after the first therapy session of a time-limited protocol were less predictive of overall outcome than ratings taken a few sessions later in treatment (after session 5). There were 4 patients in the dropout group who terminated prior to session 8 (2 completed the minimum of four sessions, 1 terminated after session 5, and 1 terminated after session 6). For these cases, the unavailable sessions were treated as missing data.

Results

Discriminant Validity of the Self-Report Measures

A MANOVA comparing the three treatment groups (dropout, good outcome, and poor outcome) was conducted with the eight relationship variables (WAI-12: Bond, Goal, Task, Total; IAS-S: Friendliness, Hostility; and SEQ: Depth, Smoothness), for each set of patient and therapist self-report ratings (reported in Tables 2 and 3, respectively).

Patient Ratings:

Overall, there was a significant difference in ratings of alliance and interpersonal behavior from the patient's perspective among the three outcome groups (F = 11.29, df = 2,68, P <0.001). Specific contrasts indicated that the dropout group was significantly differentiated from both the good outcome group (F = 21.48, df = 1,68, P <0.001) and the poor outcome group (F = 10.53, df = 1,68, P <0.01). However, there was no difference between good and poor outcome groups.

Univariate tests (Table 2) demonstrated that each of the WAI-12 Bond, Goal, Task, and Total scores were significantly different among the three outcome groups, with dropout patients consistently rating the lowest alliance scores, poor outcome patients rating moderate scores, and good outcome patients rating the highest scores. On the IAS-S, patients' ratings of Therapist Friendliness were also significantly different among the groups, with therapists in the dropout dyads being rated as the least friendly, therapists in the poor outcome dyads rated as moderately friendly, and therapists in the good outcome cases having the highest levels of patient-rated friendliness. Patients' ratings of Therapist Hostility did not distinguish outcome groups. Finally, it was found that the Depth factor of the SEQ differentiated the three outcome groups, with good outcome patients reporting the highest Depth scores across sessions, and dropout patients the lowest scores. The Smoothness factor was not significantly different among the groups.

TABLE 2.

Therapist Ratings:

Similar to the analysis of patient-rated variables, there was a significant overall difference of alliance and interpersonal behavior measures among the three outcome groups from the therapist perspective (F = 6.76, df = 2,70, P <0.01). Specific contrasts also demonstrated a significant difference between the good outcome and dropout groups (F = 13.49, df = 1,70, P <0.001) and no difference between good and poor outcome groups. The difference between dropout and poor outcome groups did not quite reach significance (F = 2.12, df = 1,70, P <0.10).

As with the patient ratings, univariate analyses (Table 3) indicated that each of the three WAI-12 factor scores and the Total score were significantly different among the three outcome groups: Bond, Goal, and Task factors and Total score. Mean scores for the three groups fell in the expected direction, similar to the patient scores. The pattern of results from analyses of therapists' ratings on the IAS-S and the SEQ factors were reversed to those found with patients' ratings. Specifically, therapists' rating of Patient Friendliness was not significant, whereas Patient Hostility was: patients in the dropout dyads were rated by therapists as being the most hostile, patients in the poor outcome dyads as being moderately hostile, and those in the good outcome condition as being least hostile. With the SEQ, the Smoothness factor distinguished among the three outcome groups, with the good outcome group showing the greatest degree of Smoothness and the dropout group the least. There was no significant difference on the Depth factor.

TABLE 3.

Post Hoc Analysis

Given the near-significant finding of a gender difference among the outcome groups, a post hoc MANOVA was conducted comparing the eight process variables between male and female patients. There was no significant difference overall.

Discussion

The results of this study indicate that a number of alliance and interpersonal behavior variables measured from patient and therapist perspectives were predictive of treatment outcome and may be useful in the early identification of patients at risk for dropout or poor outcome. The variables that significantly discriminated the three outcome groups in this study included the Total and factor scores of the WAI-12 (rated from both patient and therapist perspectives); the IAS-S (therapist ratings of Patient Hostility and patient ratings of Therapist Friendliness); and the SEQ (therapist-rated Smoothness and patient-rated Depth). Overall, patient ratings significantly distinguished the dropout and good outcome groups and the dropout and poor outcome groups, but they did not discriminate the good from the poor outcome group. In comparison, therapists' ratings distinguished the extreme groups (the dropouts from the good outcome cases), but ratings of poor outcome subjects were not significantly different from those of either of the other two groups. This type of session-by-session monitoring of specific alliance and interpersonal patterns in time-limited psychotherapy—the so-called nonspecific factors of therapy—was demonstrated to be a predictive link to the overall evaluation of treatment progress and may be particularly crucial in the early phase of therapy when the foundation of the therapeutic relationship is being established.

As expected, the therapeutic alliance scores (on WAI-12) of the dropout group were significantly worse than those of the good outcome group, with poor outcome cases falling in between the two. This was consistent for both therapist and patient ratings, although patients' ratings were demonstrated to be more sensitive in discriminating dropout from both good and poor outcome cases. These general findings are consistent with the literature, which has shown the patients' assessments of alliance early in treatment to generally be the best predictors of outcome.²⁶

Results of this current research suggest the case may not simply be that patients are better subjects for rating the therapeutic relationship than therapists, but that patients and therapists pay attention to different aspects of the treatment process. For example, with the IAS-S and SEQ, opposite factors were found to significantly discriminate the outcome groups: for therapists, the Patient Hostility and Smoothness scores discriminated groups; for patients, the Therapist Friendliness and Depth scores were significant. The IAS-S ratings fell in the expected direction, consistent with alliance scores, with the most hostile patients and less friendly therapists found in the dropout group. More specifically, therapists rated patients in the dropout group as being significantly more hostile than good outcome patients, and the poor outcome group scores fell somewhere in the middle. Patients, in contrast, rated the therapists in the good outcome group as significantly friendlier than therapists in the dropout group, with poor outcome scores on this variable falling in between. The items of the Friendliness and Hostility subscales, although typically conceptualized as poles of a single “affiliation” axis, are written as separate items on the questionnaire and, in fact, seem to be pulling for distinctive phenomena occurring between patients and therapists: high Friendliness is not equivalent to low Hostility, and vice versa.

A similar pattern of opposite variables being statistically significant from patient and therapist perspectives was found with the SEQ, where patient ratings of Depth were significantly different among outcome groups (the good outcome group had the highest scores and the dropout group had the lowest), compared with therapist ratings of Smoothness, which distinguished the groups (the good outcome group had the highest ratings, the dropout group the lowest). In other words, patients appear to value sessions that have greater Depth (sessions of good outcome cases were rated as “deep,” “valuable,” “full,” “special”), and therapists place more importance on Smoothness, valuing sessions that are less rough (sessions of good outcome cases were defined as “smooth,” “easy,” and “pleasant”). One possible reason why the Smoothness dimension was predictive for therapists may have something to do with the format of supervision: therapists show their videotaped sessions to their colleagues in the weekly group supervision conferences, which have a strong evaluative component, emphasizing adherence to manuals. Therapists may be focused on producing sessions that clearly and coherently demonstrate the treatment model (Smoothness) and on fitting the material into their agenda rather than responding more spontaneously to what the patient brings into the room each week. These results may be an artifact of conducting short-term, manualized psychotherapies. Additionally, it would be interesting to compare therapist and patient ratings on the PSQ between sessions that are taped and sessions that are not taped to see whether this type of supervision had an effect.

With respect to patient demographic information, two findings contradicted those of previous studies in the dropout literature: 1) the significantly higher number of minority patients in the good outcome group and 2) the finding of no difference with respect to educational status among groups. This sample comprised a highly educated, mostly white group of patients, and the restricted range of these two variables likely affected the results. Including the interaction of demographic categories (all minority patients in this study were highly educated) or weighted variables (education may be more important than racial status) in future research of this kind may produce more meaningful items for comparison among outcome groups. With a much larger and more varied sample, interactions of demographic and diagnostic variables could be tested for effect size and statistical significance.

The nearly significant difference in gender across outcome groups was an unexpected result. In this sample, women were more likely to have either good overall outcome or to drop out of therapy, whereas men were more likely to remain in dyads with either good or poor overall outcome. This result is consistent with some theoretical literatures indicating that females are generally more “object related” than males and may therefore be more attuned to relational cues.^57,58 The women in the present study may have been more sensitive in assessing the quality of the interpersonal match and less likely to remain in a treatment that would result in poor overall outcome, while the men tended to see the therapy through to completion regardless of the quality of the relationship. In a post hoc comparison, no significant differences were found between alliance and interpersonal behavior ratings of men and women in this study, suggesting that it was not the quality of alliance per se but rather its subjective meaning that may have accounted for the skewed distribution of genders across outcome groups. Although the sample size of the present study limits additional comparisons, such as matching patients and therapists by gender, this finding indicates an area for further research.

The next step in the validation of this method will, of course, have to be its application to a new treatment sample. At present, the findings may serve as a preliminary step toward the early, systematic identification of psychotherapy patients who present as being at risk for treatment failure. It is hoped that this type of ongoing assessment of treatment process will help to inform therapists regarding specific intervention choices and ultimately improve the quality of treatment provided.

Acknowledgments

The research presented in this article was supported in part by Grant RO3 MH50246 from the National Institute of Mental Health.

Appendix.

Appendix.

Appendix.

Appendix.