Rupture Resolution Rating System (3RS): Development and validation

Abstract

Objective:

Our aim was to examine the reliability and validity of the Rupture Resolution Rating System (3RS), an observer-based measure of alliance ruptures and resolution processes.

Method:

We used the 3RS to rate early sessions from 42 cases of cognitive behavior therapy. We compared the 3RS to a simplified version of the Structural Analysis of Social Behavior (SASB), as well as patient and therapist self-reports of ruptures and the alliance.

Results:

Coders achieved high rates of interrater reliability on the frequency of confrontation and withdrawal ruptures and resolution strategies (ICCs=.85 to .98), as well as ratings of the therapist’s contribution to ruptures and the extent to which ruptures were resolved (ICC=.92). Predictive validity analyses found that confrontation markers (d=.74), successful resolution (d=.67), and ratings of the therapist’s contribution to ruptures (d=.61) predicted dropout from therapy. Analyses of convergent validity with the SASB failed to meet predictions; however, we observed theoretically coherent relations between 3RS and SASB variables. Confrontation rupture markers were significantly associated with patient self-report of rupture (d=1.54) and therapist self-reported alliance (r = −.50, p = .002).

Conclusions:

This study provides evidence that the 3RS is a reliable and useful tool for examining psychotherapy process and predicting dropout.

The working relationship between the patient and the therapist has long been viewed as central to the process of psychotherapy. Research on the alliance (e.g., Flückiger, Del Re, Wampold, & Horvath, 2018) has provided empirical support for the idea that a strong alliance is essential for effective treatment. A small but growing body of research is building on this alliance literature by focusing on what happens when there is a problem, or rupture, in the alliance (Muran, 2017). Drawing on Bordin’s (1979) tripartite conceptualization of the alliance, ruptures can be characterized as disagreements between patients and therapists on the goals of treatment, failure to collaborate on the tasks of treatment, and/or a strain in the emotional bond. A recent meta-analysis of 11 studies (1314 patients; Eubanks, Muran, & Safran, 2018) found a moderate relation between rupture resolution and positive patient outcome. The relevance of ruptures for treatment outcome points to the potential value of developing and refining reliable and valid measures of alliance ruptures.

To date, the Rupture Resolution Rating System, or 3RS (Eubanks, Muran, & Safran, 2015) is the most commonly used observer-based measure of ruptures (Eubanks et al., 2018). The 3RS began as an adaptation of a coding system developed by Harper (1989a, 1989b). In this coding system, Harper identified two broad categories of alliance ruptures: withdrawal ruptures, in which the patient moves away from the therapist or the work of therapy, and confrontation ruptures, in which the patient moves against the therapist or the work of therapy. She also provided examples of specific patient behaviors that she characterized as markers of withdrawal (such as providing a minimal response to a question asked by the therapist) and confrontation (such as complaining about the therapist). Although we conceptualized ruptures as dyadic phenomena that are co-constructed by patients and therapists (see Safran & Muran, 2000), we began the process of developing an observer based measure of ruptures by adapting Harper’s patient-focused rupture markers. We elaborated on these markers by drawing on the discussion of ruptures in Safran and Muran (2000), and based on examples that arose as we began coding videos of therapy sessions. The final set of 3RS rupture markers is provided in Table 1.

Table 1.

Rupture Resolution Rating System (3RS) rupture markers and resolution strategies.

Withdrawal Rupture Markers		M	SD
Avoidant Storytelling and/or Shifting Topic	Patient tells stories and/or shifts the topic in a manner that functions to avoid the work of therapy.	3.21	3.41
Content/affect split	The patient withdraws from the therapist and/or the work of therapy by exhibiting affect that does not match the content of his/her narrative.	1.58	1.89
Minimal response	Patient withdraws from the therapist by going silent or by giving minimal responses to questions or statements that are intended to initiate or continue discussion.	1.38	2.13
Deferential and appeasing	Patient withdraws from the therapist and/or the work of therapy by being overly compliant and submitting to the therapist in a deferential manner.	.71	1.65
Self-criticism and/or hopelessness	The patient withdraws from the therapist and the work of therapy by becoming absorbed in a depressive process of self-criticism and/or hopelessness that seems to shut out the therapist and to close off any possibility that the therapist or the treatment can help the patient.	.67	1.35
Abstract communication	Patient avoids the work of therapy by using vague or abstract language.	.61	.97
Denial	Patient withdraws from the therapist/work of therapy by denying a feeling state that is manifestly evident, or denying the importance of interpersonal relationships or events that seem important and relevant to the work of therapy.	.21	.52
Confrontation rupture markers
Patient defends self against therapist	Patient defends his/her thoughts, feelings, or behavior against what he/she perceives to be the therapist’s criticism or judgment of the patient.	1.09	1.62
Patient rejects therapist intervention	Patient rejects or dismisses the therapist’s intervention.	1.03	1.29
Complaints/concerns about the activities of therapy	Patient expresses dissatisfaction, discomfort, or disagreement with specific tasks of therapy such as homework assignments or in-session tasks such as empty chair or imaginal exposure.	.86	1.02
Efforts to control/pressure therapist	Patient attempts to control the therapist and/or the session, or the patient puts pressure on the therapist to fix the patient’s problems quickly.	.84	1.76
Complaints/concerns about the therapist	Patient expresses negative feelings about the therapist.	.30	.76
Complaints/concerns about progress in therapy	Patient expresses complaints, concerns, or doubts about the progress that can be made or has been made in therapy.	.11	.30
Complaints/concerns about the parameters of therapy	Patient expresses concerns or complaints about the parameters of treatment, such as the therapy schedule or the research contract.	.08	.26
Resolution Strategies
Therapist responds to a rupture by redirecting or refocusing the patient.		1.13	1.72
Within the context of a rupture, the therapist invites the patient to discuss thoughts or feelings with respect to the therapist or some aspect of therapy.		1.10	1.14
Therapist illustrates tasks or provides a rationale for treatment.		.96	1.15
Therapist changes tasks or goals.		.61	.69
Therapist validates the patient’s defensive posture.		.36	.75
Within the context of a rupture, the therapist discloses his/her internal experience of the patient-therapist interaction.		.33	.76
Therapist clarifies a misunderstanding.		.26	.48
Within the context of a rupture, the therapist acknowledges his/her contribution to a rupture		.19	.59
Therapist links the rupture to larger interpersonal patterns in the patient’s other relationships.		.10	.23
Therapist links the rupture to larger interpersonal patterns between the patient and the therapist.		.01	.05

Note. M = mean; SD = standard deviation. Withdrawal and confrontation markers and resolution strategies are listed in order of frequency of use in this sample.

We also added a set of resolution strategies to the 3RS (see Table 1). We drew on strategies outlined in Safran and Muran (2000), and refined and expanded these based on our experiences of coding. Strategies are defined in terms of therapist behaviors. We chose to ask coders only to rate the occurrence of resolution strategies, not the competence with which they are used or their effectiveness at addressing ruptures. We were concerned that rating the success of a resolution attempt during the course of the session would be difficult, particularly for graduate student coders; even experienced therapists might not be able to tell how well a resolution strategy worked immediately after it was attempted. As early efforts to resolve a rupture may create new ruptures (Safran & Muran, 2000), it may take time for the rupture process to unfold and for an observer to be able to judge how well a rupture or series of ruptures was resolved. However, we recognized that researchers would need some way to judge the success of resolution efforts, so we added a single item asking the extent to which resolution was achieved over the course of the session. This item was modeled after a self-report resolution item on the Post-Session Questionnaire (PSQ: Muran, Safran, Samstag, & Winston, 1992).

As the 3RS was taking shape, we were concerned about the fact that operationalizing ruptures in terms of patient behaviors is contrary to our dyadic understanding of ruptures. In a preliminary effort to begin to examine the role of the therapist in ruptures, we added a single item assessing the extent to which the therapist caused or exacerbated ruptures in the session. We recognized that this single item was not sufficient; our hope was that it would help us begin to identify behaviors that could eventually be added to the measure as therapist rupture markers.

Early versions of the 3RS focused on identifying the frequency of rupture markers and resolution strategies. We realized that in some cases frequency can differ from clinical importance: there could be very frequent rupture markers or resolution strategies that were not very impactful, and conversely, there could be rupture markers or resolution strategies that were pivotal in a session but only occurred once. In an effort to capture this possibility, we added clinical impact ratings that coders complete after watching the entire session. For each type of rupture marker and resolution strategy, coders assess its impact on the alliance over the course of the session.

Preliminary psychometric properties of the 3RS

Several studies that have used the 3RS provide preliminary support for its psychometric properties. Three studies reported interrater reliabilities for 3RS variables, most of which were in the good to excellent range (Cicchetti, 1994). An examination of 38 dyads (201 sessions) that received cognitive behavioral therapy (CBT) at a university clinic in Portugal reported intraclass correlation coefficients (ICCs) of .73 for ratings of the clinical impact of withdrawal ruptures, and .96 for confrontation ruptures; this study did not examine resolution strategies (Coutinho, Ribeiro, Sousa, & Safran, 2014). A study of 44 Australian adolescents (74 sessions) who were diagnosed with borderline personality disorder and received either cognitive analytic therapy or a supportive therapy reported reliability coefficients for rupture and resolution markers ranging from .64 to .90 (Gersh et al., 2017). A study of 77 patients (154 sessions) primarily presenting with mood or anxiety disorders at a university clinic in Switzerland reported ICCs of .74 for rupture markers, and .63 for resolution strategies (Moeseneder, Ribeiro, Muran, & Caspar, 2018). This study also examined whether coders agreed on the incidence of a rupture or resolution attempt within each five-minute segment and obtained a Cohen’s kappa of .46, which indicates only fair reliability (Cicchetti, 1994).

Evidence supporting the predictive validity of 3RS variables can be garnered from studies examining the relations between rupture and resolution processes rated by the 3RS and treatment outcome. A study of 53 patients (82 sessions) comparing sessions in which patients with anxiety or depressive disorders at a university clinic in Germany experienced sudden gains or losses on symptom measures found that more confrontation ruptures occurred in the sessions preceding a sudden loss—in other words, that confrontation ruptures predicted an abrupt increase in symptoms (Ehrlich & Lutz, 2015). A study that used the 3RS to compare early sessions of three good and three poor outcome cases (24 sessions) of dialectical behavior therapy for borderline personality disorder as part of a randomized controlled trial in Canada found some support for the predictive validity of withdrawal rupture markers: unrecovered clients had a higher frequency of withdrawal ruptures than recovered clients (Boritz, Barnhart, Eubanks, & McMain, 2018). The study of adolescents with borderline personality disorder cited above (Gersh et al., 2017) found some evidence for the predictive validity of both rupture markers and ratings of the extent to which resolution was achieved in a session: early treatment ruptures were associated with poor outcome, while greater resolution later in treatment was associated with better outcome. Moeseneder et al.’s (2018) study of patients from a Swiss university clinic examined the relation of resolution strategies to treatment outcome and found that it varied depending on therapists’ use of challenging interventions. Therapist use of challenge during rupture resolution attempts was associated with better outcome than making no resolution attempt or attempting rupture resolution without the use of therapist challenge.

In two analyses of the same dataset of 38 CBT cases (Coutinho, Ribeiro, Fernandes, Sousa, & Safran, 2014; Coutinho, Ribeiro, Sousa, & Safran 2014), Coutinho and colleagues examined the construct validity of the 3RS. They predicted that due to the interpersonal difficulties that characterize personality disorders, patients with personality disorder diagnoses would exhibit more ruptures than patients with Axis I disorders (Coutinho, Ribeiro, Fernandes et al., 2014); they found evidence to support their prediction. They also examined the construct validity of the 3RS by comparing ruptures identified with the 3RS to ruptures identified based on fluctuations in scores on the Working Alliance Inventory (WAI; Horvath & Greenberg, 1989), a patient self-report measure of the alliance (Coutinho, Ribeiro, Sousa, & Safran, 2014). The researchers found some evidence of both convergent and predictive validity: There was some consistency between the two measures with respect to global patterns of ruptures, with cases of patients who dropped out of therapy prematurely showing increases in ruptures on both measures in the sessions just before the patient dropped, as compared to patients who completed therapy. At the same time, the authors proposed that there would also be discriminant validity between observer and self-report measures, with the 3RS demonstrating greater sensitivity than the WAI. Consistent with their prediction, when they focused specifically on significant decreases on the WAI and sessions with high (three or greater on a five-point scale) ruptures on the 3RS, they found that the 3RS identified more rupture episodes than the WAI.

Validation aims and exploratory research questions

In order to build on these early studies and further the validation of the 3RS, we sought to more closely examine the measure’s psychometric properties in the following ways:

1. We sought to demonstrate the interrater reliability of the 3RS at the macrolevel of overall withdrawal and confrontation and the extent to which resolution was achieved, but also the more microlevel of individual rupture markers and resolution strategies.

2. We sought to demonstrate the predictive validity of the 3RS by examining whether the 3RS could predict treatment dropout. Similar to Coutinho, Riberio, Fernandes et al. (2014), we expected that there would be higher rates of confrontation ruptures in dropout cases as compared to completer cases. In addition, we also predicted that there would be higher rates of withdrawal ruptures in drop cases. Finally, we expected to see higher therapist contribution to rupture and less resolution achieved in drop cases as compared to completer cases. In addition, we conducted exploratory analyses to examine whether there were higher rates of individual rupture markers in dropout cases, and to compare dropout and completer cases on resolution strategies. We also assessed the incremental validity of the 3RS by comparing its ability to predict dropout to that of the other process measures employed in this study (the SASB, patient and therapist self-report of ruptures, and the WAI).

3. We also sought to demonstrate the convergent validity of the 3RS by examining the correlation between 3RS variables and specific patient and therapist interpersonal behaviors measured using a simplified version of the Structural Analysis of Social Behavior (SASB: Benjamin, 1974). The SASB has been recognized as a useful tool for measuring patient and therapist behaviors in the alliance (e.g., Benjamin & Critchfield, 2010). Following the use of the full SASB to identify rupture and resolution processes in Safran and Muran (1996), we hypothesized that withdrawal ruptures would be moderately correlated with the SASB patient codes of Follow, Appease, and Avoid, and confrontation ruptures would be moderately correlated with patient Blame. We also predicted that the extent to which resolution was achieved would be moderately correlated with both patient and therapist Express, as well as therapist Affirm and Direct.

In an effort to better understand how the 3RS relates to self-report measures of ruptures and the alliance, we also conducted two additional sets of exploratory analyses:

1. We examined the association between 3RS rupture variables and another measure of rupture, the patient and therapist self-report rupture indices on the PSQ. Given the lack of prior research comparing these measures, we were uncertain whether a comparison would identify convergent validity in the form of a significant association between these rupture measures, or discriminant validity, as the 3RS was specifically designed to capture not only overt ruptures, but also rupture processes that patients and therapists do not self-report and of which they may not even be aware.

2. We also examined the association between 3RS rupture and resolution variables and patient and therapist self-reports of the alliance using the WAI. We regarded this as an exploratory research question, as prior research (Coutinho, Ribeiro, Sousa, & Safran, 2014) found some evidence of convergent validity between the ruptures and lower alliance ratings in some analyses, but also discriminant validity when focusing on more extreme ruptures.

Method

Cases used in the present study were drawn from the archives of an ongoing research program at a metropolitan medical center in the US. Self-report data from 12 (28.6%) of the cases in this study have previously been published in another study (Zilcha-Mano et al., 2016); no prior publication has reported findings derived from session videos of these cases. All patients in this sample provided informed consent to participate in the study and receive 30 weekly sessions of CBT. Patients paid a small fee for each therapy session on an income-based sliding scale in order to approximate a naturalistic treatment setting. All therapy sessions were videotaped.

Cases were chosen based on several criteria. First, cases had to be deemed adherent to CBT based on observer ratings of at least one early session using the CBT subscale of the Beth Israel Fidelity Scale (Patton, Muran, Safran, Wachtel, & Winston, 1998). This yielded a pool of 176 potential cases: 102 (58%) completer cases and 67 (38%) dropout cases. Second, cases had to have completed the 30 session treatment protocol, or unilaterally terminated treatment before the end of the 30 sessions; the seven patients who terminated due to illness, death, or geographic relocation were excluded from the study. As videos of session six were selected for coding, 15 patients who dropped out of treatment prior to session six were also excluded. We chose to focus on session six for a few reasons. First, given our interest in predictive validity, we wanted to capture ruptures in the early phase of treatment, which has been defined in the alliance literature as the first five (e.g., Flückiger et al., 2018) or six (e.g., Muran et al., 2009) sessions. Second, we felt that session six was far enough into treatment that there would be time for a therapeutic relationship to develop and for problems in the alliance to begin to unfold. We were also concerned that dropout prior to session 6 might be more indicative of a patient’s ambivalence about starting therapy rather than factors related to the process of treatment.

The final set of 42 cases was then selected based on the availability of session six videos and self-report questionnaires. In addition, an effort was made to oversample dropout cases in order to have comparable numbers of drop and completer cases. When cases had equivalent quality and availability of data, the final selection was made at random. We compared the 42 cases we selected to the cases that were not selected and found no significant differences with respect to patient demographics or measures of patient pretreatment functioning.

Participants

Patients.

Patients included 25 women and 17 men (N=42), ranging in age from 21 to 78 (M =39.48, SD =16.06). Most patients (83%; n = 35) self-identified as White, with 9.5% (n=4) identifying as Hispanic, 4.8% (n=2) as Asian, and 2.4 % (n=1) as Black. With respect to their highest level of formal education, 11.9% (n=5) were high school graduates, 50% (n=21) were college graduates, and 38.1% (n=16) had graduate degrees. Most patients (64.3%; n=27) were employed, with 28.6% (n=12) unemployed and 7.1% (n=3) retired.

Patient diagnoses were established with the Structured Interview for DSM-IV-Axis I & II (SCID I and II: First, Spitzer, Gibbon, & Williams, 1997; First, Gibbon, Spitzer, & Benjamin, 1997), which was administered prior to the commencement of therapy by trained psychology graduate student research assistants. Two patients did not meet criteria for any diagnoses. All other patients met criteria for at least one diagnosis. On Axis I, 71.4% (n=30) of patients met criteria for a primary diagnosis of mood disorder, and 21.4% (n=9) met criteria for an anxiety disorder. On Axis II, 23.8% (n=10) of patients met criteria for Personality Disorder Not Otherwise Specified; 19.0% (n=8) met criteria for a Cluster C personality disorder, and 2.4% (n=1) met criteria for a Cluster B personality disorder. Exclusion criteria included evidence of organicity, psychosis, mania, or severe major depression, impulse control and compulsive disorder, and any active substance use disorder.

Therapists.

Therapists included 32 women and 10 men (N=42), ranging in age from 23 to 37 (M =28.16, SD =3.55). Most therapists (78.6%; n=33) self-identified as White, with 7.1% (n=3) identifying as multiracial, 4.8% (n=2) identifying as Asian, 2.4% (n=1) identifying as Black, and 2.4% (n=1) identifying as Hispanic. With respect to clinical training, 90.5% (n=38) of therapists were psychology externs, and 9.5% (n=4) were psychiatry residents, and therapists reported a limited amount of clinical experience (M =1.56 years, SD =.92).

Prior to seeing this case, therapists received 16 weeks of didactic training in a cognitive behavioral treatment (Turner, Muran, & Ochoa, 1992/2004) that is based on Beck and colleagues’ (1990) adaptation of cognitive therapy for personality disorders and incorporates Persons’ (1989) case formulation approach. During treatment, therapists attended a 75-minute weekly group supervision, which was led by a highly experienced CBT supervisor. Supervision included role-playing with other trainees, and receiving feedback on videotapes of therapists’ work. Supervisors instructed therapists on establishing case formulations in which core beliefs systems were identified, and implementing CBT interventions such as cognitive restructuring and behavioral exercises.

Measures

Pretreatment measures.

Patients’ pretreatment functioning was assessed via self-report measures administered prior to the commencement of therapy.

Symptom Checklist 90 Revised (SCL-90R: Derogatis, 1983). The SCL-90 is a 90 item self-report inventory that assesses general psychiatric symptomatology. The Global Severity Index (GSI), the overall mean score, was used as a measure of general psychiatric symptom severity. The SCL-90R has acceptable psychometric statistics with test-retest reliability coefficients ranging from .78-.90 and internal consistency coefficients ranging from .80-.90 (Derogatis, 1983). Internal consistency in this sample was .98.

Inventory of Interpersonal Problems- 64 (IIP: Horowitz, Alden, Wiggins, & Pincus, 2000). The IIP is a 64-item self-report scale to assess interpersonal functioning and social adjustment level. The IIP has demonstrated high internal consistency, reliability, and validity (Horowitz, Rosenberg, Baer, Ureno, & Villasenor, 1988) and high test-retest reliability, r = .90 (Hansen & Lambert, 1996). The overall mean score was used. Internal consistency in this sample was .92.

Rupture Resolution Rating System.

The Rupture Resolution Rating System (3RS; Eubanks et al., 2015) is an observer-based measure for identifying alliance ruptures and resolution processes. The 3RS differentiates ruptures into two categories: withdrawal ruptures and confrontation ruptures. The coding system includes codes for seven markers of withdrawal ruptures and seven markers of confrontation ruptures, all defined in terms of observable patient behaviors. In addition, the 3RS assesses ten resolution strategies defined in terms of observable therapist behaviors. See Table 1 for a brief description of the 3RS rupture markers and resolution strategies.

One early session (session 6) from each case was coded using the 3RS. Each session was coded by a pair of coders, drawn from a pool of six graduate students in clinical psychology who were trained in the use of the measure. Coders rated sessions independently, and their scores were averaged. All coders received at least 20 hours of training with the first author of the coding manual, and also engaged in practice coding of therapy sessions not included in this study. Coders were blind to study hypotheses and termination status (drop or completer).

Coders rated the occurrence of rupture markers and resolution strategies in five minute intervals, using ratings of a check for any marker or strategy that occurred within the five-minute segment, and a check minus for instances when they were not entirely sure the behavior fully met criteria. Specifically, when a patient exhibited a behavior that fit the description of a rupture marker, but it was not clear whether the patient was exhibiting this behavior as part of a move away from (withdrawal marker) or against (confrontation marker) the therapist and/or the work of therapy, that patient behavior could be rated with a check minus. For example, a patient might become very quiet and give a very brief answer to a therapist’s open-ended question, and it might not be clear whether the patient was withdrawing from the interaction with the therapist or actually engaging in the interaction by thinking carefully about what the therapist had just asked. The coder could capture this behavior by rating it as a check minus minimal response rupture marker. Similarly, when a therapist exhibited a behavior that fit the description of a resolution strategy (e.g., change task), but it was not clear whether the therapist was engaging in this behavior in order to address a rupture (e.g., by changing a task that the patient found objectionable), or was simply conducting therapy (e.g., moving on to the next task that would normally follow in the therapy session), the coder could capture this by giving a check minus rating for a resolution strategy.

The frequency of rupture markers and resolution strategies was determined by assigning one point to each check and .5 to each check minus within each five-minute segment, and then summing across the segments of a session to determine the weighted frequency of each type of rupture marker and resolution strategy, as well as the overall frequency of all withdrawal markers, confrontation markers, and resolution strategies for that session.

The 3RS also includes global ratings that the coder completes after watching the entire therapy session. Using five-point Likert-type scales ranging from “no significance” to “high significance,” coders rate the extent to which withdrawal markers overall, confrontation markers overall, and each individual type of rupture marker or resolution strategy had a clinically meaningful impact on the alliance over the course of the session. In addition, coders rate the extent to which rupture resolution occurred in the session using a five-point Likert-type scale in response to the question “To what degree were ruptures resolved over the course of the session?” Using a five-point Likert-type scale, coders also rate the extent to which the therapist caused or exacerbated ruptures in the session.

Structural Analysis of Social Behavior.

To assess the construct validity of the 3RS, we also coded therapy sessions with a simplified version of the Structural Analysis of Social Behavior (SASB; Benjamin, 1974), an observer-based measure that defines interpersonal process on the basis of a circumplex model. The SASB has been used as a measure of alliance (Benjamin & Critchfield, 2010; Henry & Strupp, 1994) and as a measure of changes in interpersonal behaviors theorized to be related to ruptures and rupture resolution (Muran et al., 2018; Safran & Muran, 1996). Following the procedure used in Muran et al. (2018), videos of therapy sessions were coded in five-minute intervals. Coders rated the presence of interpersonal behaviors of the patient and the therapist in terms of two surfaces (“Focus on Other” and “Focus on Self”) and two orthogonal dimensions: interdependence (from autonomy to involvement) and affiliation (from hostility to friendliness). Raters coded in terms of the octant SASB model on each surface, resulting in 16 possible dimensions for patient behaviors and 16 for therapist behaviors, a total of 32 SASB dimension ratings for each session. Reliability was calculated at the session level. Trained graduate students in psychology served as coders. About half (N=20) of the sessions were coded by two coders who worked independently and were blind to patient outcome and to the hypotheses of this study. Their scores were averaged. Interrater reliability for these sessions ranged from .75 to .97, with an average intraclass correlation coefficient of .85, indicating excellent reliability. The remaining sessions were coded by one of the two coders.

Following Muran et al. (2018), we used the 32 dimension ratings to calculate variables based on the quadrant version of the SASB. We summed three of the octant items for each quadrant with the middle item (along the circumplex surface) weighted by 1.00 and the two adjacent items weighted by .50. The process yielded the variables Affirm, Direct, Blame, and Ignore on Surface 1 (Focus on Other), and the variables Express, Follow, Appease, and Avoid on Surface 2 (Focus on Self). Table 2 features definitions for each variable based on language from the original SASB model (Benjamin 1974).

Table 2.

Quadrant codes of the simplified SASB, derived from Benjamin (1974).

Surface 1 Focus on Other		Surface 2 Focus on Self
Affirm	Self understands Other, provides emotional support, and encourages autonomy.	Express	Self discloses and expresses innermost self to Other and is straightforward about own position.
Direct	Self teaches and directs Other how to understand and behave.	Follow	Self follows, accepts, and relies on the direction of Other.
Blame	Self criticizes Other to see and behave according to Self.	Appease	Self bottles up emotion and complies with Other to avoid disapproval.
Ignore	Self ignores and neglects needs and interests of Other.	Avoid	Self avoids and disconnects from Other in order to separate.

Self-report measures of treatment process.

Patients and therapists independently completed the PSQ after each session. The PSQ included a 12-item version of the Working Alliance Inventory (WAI: Horvath & Greenberg, 1989; Tracey & Kokotovic,1989), which was used to assess the quality of the therapeutic alliance. The WAI has demonstrated sound psychometric qualities and is widely used in psychotherapy research (Flückiger, et al.). The overall mean score was used for calculations. In this study, both patient-rated (α=.84) and therapist-rated (α =.79) versions of the WAI demonstrated good internal consistency.

The PSQ also includes the following question assessing whether ruptures occurred during the session: Did you experience any tension or problem, any misunderstanding, conflict, or disagreement, in your relationship with your [therapist/patient] during the session? Patients and therapists rated this question on a five-point Likert-type scale ranging from “Not at all” to “Constantly.” Ratings of 1 (not at all) were regarded as indicating no presence of a rupture, and ratings of 2–5 were regarded as the participant indicating that some degree of rupture occurred during the session.

Results

3RS Interrater reliability

Interrater reliability on the 3RS was generally high: for the overall frequency of withdrawal markers reliability was ICC (1, 2) = .85; for confrontation markers ICC (1, 2) = .98; for resolution strategies ICC (1, 2) = .95. Reliability for individual withdrawal markers ranged from .83 to .98, with a mean of .92 (SD =.05), and reliability for individual confrontation markers ranged from .92 to .99, with a mean of .95 (SD =.03). For individual resolution strategies, reliability for the strategy therapist links the rupture to larger interpersonal patterns between the patient and therapist violated reliability model assumptions, resulting in an ICC of −.03 for that strategy. Closer examination revealed that this strategy only received two check minus ratings from one member of a coding pair, each for a different session, in the entire dataset. Excluding that strategy, ICC for individual resolution strategies ranged from .62 to .97 with a mean of .86 (SD =.12).

Coders also achieved good interrater reliability on clinical impact ratings: the ICC (1, 2) was .81 for the clinical impact of withdrawal ruptures overall and .93 for confrontation ruptures overall. Clinical impact ratings for individual rupture markers were also reliable: mean ICC scores for individual withdrawal markers were .89 (SD = .05), and mean ICC scores for individual confrontation markers were .93 (SD = .04). For individual resolution strategies, the aforementioned linking strategy again violated reliability model assumptions resulting in an ICC of −.03 for that strategy. Excluding that strategy, resolution strategies had a mean reliability of .83 (SD =.09).

Interrater reliability for the rating of the extent to which resolution was achieved in the session was ICC (1, 2) = .92. Interrater reliability for the extent of the therapist’s contribution to ruptures was also ICC (1, 2) = .92.

Associations among 3RS variables

In this dataset, correlations between rupture and resolution strategy frequencies and ratings of their clinical impact were very high. Specifically, overall withdrawal frequency and withdrawal clinical impact were correlated r = .79 (p<.001), and overall confrontation frequency and confrontation clinical impact were correlated r =.82 (p<.001). Individual rupture markers and resolution strategies also demonstrated high correlations between the frequency and clinical impact of each marker or strategy, with r’s ranging from .85 to 1.00. Given these high correlations, it appears that in this sample, clinical impact ratings did not yield much information above and beyond frequency ratings; hence, for the sake of parsimony, we will only report findings related to rupture marker and resolution strategy frequency in this study.

Mean frequency ratings of each rupture marker and resolution strategy are presented in Table 1. Coders identified withdrawal markers in every session and confrontation markers in 90.5% of sessions (38 out of 42). If, following Coutinho, Riberio, Sousa et al. (2014), we define ruptures as achieving an overall confrontation or withdrawal score of three or greater on the five-point Likert-type scale of clinical impact, then there were withdrawal ruptures in 31 (73.8%) sessions and confrontation ruptures in 18 (42.9%) sessions.

Predictive validity

Preliminary analyses of dropout.

The dataset included 23 completers and 19 drop cases. The final session for drop cases ranged from session 6 to session 18 (M =10.47, SD =3.82). Before comparing drop and completer cases on any process measures, we first examined whether the two groups differed at intake on demographic variables and found no significant differences between drops and completers on patient or therapist gender, patient or therapist age, patient or therapist race, or therapist years of clinical experience. We also examined whether drops and completers differed on pretreatment measures of symptoms and interpersonal functioning. Independent t-tests comparing drops and completers revealed that they did not differ on symptoms as assessed by the SCL-90R, t(37)=.89, p = .38; nor did they differ on pretreatment interpersonal problems as assessed by the IIP, t(37)=.73, p =.47.

3RS and dropout.

To examine whether 3RS variables could predict dropout from therapy, we conducted independent group t-tests comparing drops and completers on the overall weighted frequency of withdrawal markers and confrontation markers. Consistent with our prediction, confrontation ruptures markers were more frequent in drop cases (M =6.04, SD =4.97) than completer cases (M = 2.86, SD =3.47), t(40) = −2.44, p = .02., with a medium to large effect size (d=.74). Contrary to predictions, drop (M =8.58, SD =4.59) and completer cases (M = 8.24, SD =4.90) did not differ significantly on the overall frequency of withdrawal markers, t(40)= −.23, p = 82, d = .07.

Consistent with our prediction, we found that coders rated therapists of drop cases as contributing more to ruptures (M =2.61, SD =1.34) than therapists of completer cases (M =1.87, SD = 1.06), t (40) = −1.99, p = .05, with a medium effect size (d=.61). We also found support for our prediction of greater rupture resolution achieved in completer cases (M =2.89, SD =.90) as compared to drop cases (M =2.26, SD =.99), t (40) = 2.15, p = .038, with a medium to large effect size (d=.67). As an exploratory analysis, we also examined the overall frequency of resolution strategies in drop cases (M =5.71, SD =5.29) and completer cases (M =4.51, SD =3.46) and found that they did not differ significantly, t (40) = −.88, p = .38, d=.27.

We conducted exploratory t-tests comparing drop and completer cases on the frequencies of each individual rupture marker and resolution strategy. Only two were statistically significant: the confrontation marker patient defends self was more common in drop cases, t (29.8) = −2.24, p = .03), and the resolution strategy therapist validates the patient’s defensive posture was also more common in drop cases than completer cases, t (20.17) = −2.38, p = .03.

Self-report measures and dropout.

In order to assess the incremental validity of the 3RS, we examined whether the other process measures used in this study could distinguish between dropout and completer cases. The association between self-report of rupture and dropout was not significant for patients, χ² (1, N=33) = 2.27, p = .13, nor for therapists, χ² (1, N=37) = .11, p = .74. However, we did find a difference between dropout and completer cases with respect to missing data: eight (42.1%) drop patients were missing the rupture self-report item, compared to only one (4.4%) of the completer patients, a significant association, χ² (1, N=42) = 8.81, p = .003. For therapists, five (26.3%) therapists from drop cases were missing the self-report rupture item; none of the therapists from completer cases were missing this item, a significant association, χ² (1, N=42) =6.87, p = .009.

On the WAI, completer patients (M = 5.97, SD =.70) rated the alliance higher than drop patients (M =5.23, SD =1.22), and this difference approached significance, t (18.48) = 2.08, p = .052. For the therapist-rated WAI, the difference between completer cases (M =5.43, SD =.73) and drop cases (M =4.64, SD =1.05) was significant, t (35) = 2.67, p = .01. Similar to the self-report rupture item, there were differences between dropout and completer cases with respect to missing data on the WAI. With the patient-rated WAI, five (26.3%) drop cases were missing a patient WAI, compared to only one (4.4%) of the completers, a significant association, χ² (1, N=42) = 4.1, p = .04. Therapists of drop cases were also more likely to be missing data from the alliance self-report measure: five (26.3%) therapists from drop cases were missing WAI data, while none of the therapists from completer cases were missing WAI data, a significant association, χ² (1, N=42) = 6.87, p = .009.

SASB and dropout.

To examine whether the simplified version of the SASB used in this study could predict dropout from therapy, we conducted independent group t-tests comparing drops and completers on the SASB patient variables (Express, Follow, Appease, Avoid) and SASB therapist variables (Affirm, Direct, Blame, Ignore). There were no significant differences between drops and completers on these variables.

Convergent validity

3RS and SASB.

To examine our prediction that 3RS withdrawal markers would be associated with the SASB codes of patient Avoid, Appease, and Follow, we correlated overall withdrawal frequency with these variables. The correlation between withdrawal and Avoid was moderate in size but did not reach significance, r =.27, p = .08. Contrary to our predictions, the correlations with Appease (r =.10, p =.53) and Follow (r = −.02, p = .88) were small and not significant. We were unable to examine an association between confrontation frequency and patient Blame because the SASB coders identified no instances of patient Blame in this sample.

To examine our predictions about resolution processes, we conducted a series of correlations with the 3RS variable of resolution achieved. The predicted associations between resolution achieved and patient Express (r =−.03, p = .86), therapist Affirm (r =−.01, p = .97) and therapist Direct (r =.05, p = .76) were not supported. We were unable to examine our prediction of an association between resolution achieved and therapist Express because the SASB coders identified no instances of therapist Express in this sample.

With this failure to support the convergence that we anticipated to find between the 3RS and the SASB, we conducted a series of post hoc exploratory correlations to see if the two measures might be related in ways that we did not anticipate. We correlated overall withdrawal frequency with the SASB codes of patient Express and therapist Affirm, Direct, Blame, and Ignore, and found no significant associations (absolute value of r’s ranged from .00 to .28). We correlated overall confrontation frequency with all the SASB codes, and while we found no significant association with patient Express or therapist Ignore or Direct, we did find significant correlations for patient Avoid (r =.56, p = <.001), patient Appease (r =.37, p = .02), patient Follow (r = −.32, p = .04), and therapist Affirm (r =−.34, p = .03). Thus, confrontation markers appeared to be most strongly related to higher levels of patient avoidant and hostile submissive behaviors and lower levels of patient friendly submissive and therapist affirming behaviors.

In an effort to better understand how the SASB might relate to 3RS codes of resolution processes, we also conducted exploratory correlations of resolution achieved with the SASB variables of patient Avoid, Appease, and Follow, and therapist Ignore and Blame. We found a significant positive correlation between resolution achieved and patient Follow (r =.31, p = .046), and significant negative correlations with therapist Ignore (r =−.35, p = .02) and patient Avoid (r =−.34, p = .03). Thus, the 3RS rating of successful rupture resolution in this sample appeared to be related to higher levels of the patient following the therapist’s direction and lower levels of patient avoidance and therapist neglect.

Exploratory analyses of 3RS and self-report measures

3RS and self-report of ruptures.

To examine the association between 3RS rupture markers and patient and therapist self-report of rupture, we conducted a series of independent groups t-tests comparing overall withdrawal and confrontation frequencies in cases in which a rupture was reported compared to cases in which a rupture was not reported. Contrary to predictions, when comparing cases in which the therapist reported a rupture to cases in which the therapist reported no rupture, there was no significant difference in the frequency of withdrawal markers, confrontation markers, resolution strategies, nor in the ratings of resolution achieved or therapist contribution to ruptures. For patient-reported ruptures, there were also no significant differences with one exception: cases in which the patient reported a rupture had higher ratings of confrontation (M =8.82, SD =4.77) than cases in which the patient reported no rupture (M =2.88, SD =2.62), t (31) = −4.43, p <.001, d=1.54.

3RS and self-report of alliance.

To examine whether there was an association between self-reported alliance and 3RS rupture and resolution codes, we first conducted a series of correlations. Although we found moderate sized correlations between patient-rated alliance and overall withdrawal frequency (r = - .27, p = .11) and resolution achieved (r =.27, p = .11), these failed to reach statistical significance. The negative correlation between patient-rated alliance and overall confrontation frequency approached significance (r = −.32, p = .054). For the therapist-rated WAI, there was a significant negative correlation with the frequency of confrontation ruptures (r = −.50, p = .002) and of resolution strategies (r = −.39, p = .02), and a moderate but nonsignificant correlation with resolution achieved (r =.28, p = .09). Correlations between patient WAI and resolution strategy frequency, between therapist WAI and withdrawal ruptures, and between both patient and therapist WAI and therapist contribution to rupture were all small and nonsignificant (absolute value of r’s ranged from .09 to .16).

Discussion

Using a dataset of early sessions of CBT, we examined the psychometric properties of the 3RS. Consistent with several existing 3RS studies, coders in this study achieved good levels of interrater reliability on the frequency of withdrawal and confrontation ruptures and resolution strategies, as well as ratings of the therapist’s contribution to ruptures and the extent to which ruptures were resolved in the session. This study adds to the existing literature by also demonstrating very good interrater reliability for the frequency and clinical impact of individual rupture markers and resolution strategies, with the exception of the infrequently used strategy of linking the rupture to larger interpersonal patterns in the dyad.

Analyses of predictive validity found that, similar to the findings of Coutinho, Ribeiro, Fernandes et al. (2014), confrontation markers successfully predicted premature termination from therapy. Consistent with Gersh et al. (2017), we also found an association between ratings of resolution and outcome: specifically, we found that lower ratings of resolution predicted dropout. In addition, we found that ratings of therapists’ contributions to rupture predicted dropout. Comparison of the predictive validity of the 3RS to that of the SASB, rupture self-reports, and alliance self-reports provided some evidence of the 3RS’s incremental validity: only 3RS variables and therapist self-reported alliance significantly predicted dropout. It must be noted, however, that there were significant associations between missing data and dropout on self-report measures of ruptures and the alliance. We can only speculate about whether patients and therapists in drop cases were less motivated to complete study questionnaires.

Analyses related to construct validity yielded more mixed findings. Our analyses comparing SASB and 3RS ratings failed to find the evidence of convergent validity that we predicted based on prior research. Exploratory analyses of associations between 3RS variables and patient and therapist self-report found some evidence of convergent validity. Similar to Coutinho et al. (2014), we found some moderate correlations between 3RS variables and self-reported alliance, and we also found a significant association between patient self-report of rupture and confrontation ruptures. However, a number of associations failed to reach significance. Additional research is needed to clarify whether this is evidence of discriminant validity, with the 3RS detecting rupture and resolution processes that patients and therapists do not report, or whether this points to limitations of the 3RS with respect to capturing clinical processes that impact outcome.

It is important to note that across our analyses, we found higher interrater reliability and stronger evidence of predictive and convergent validity with confrontation rupture markers as compared to withdrawal rupture markers. This may reflect characteristics of these types of ruptures: while withdrawal ruptures are conceptualized as being more subtle and harder for patients, therapists, and observers to recognize, confrontation ruptures are typically more overt and are the types of ruptures that therapists tend to self-report the most (Eubanks, Burkell, & Goldfried, 2018). This finding may also be due to treatment and sample characteristics. In this study, withdrawal markers were more common than confrontation markers; this may reflect the diagnostic profile of the sample, which featured many patients with depressive disorders and a number with Cluster C personality disorders, who may be more prone to interpersonal withdrawal. Ehrlich and Lutz (2015)’s study of a sample of patients with anxiety and depressive disorders who received integrative CBT also found greater incidence of withdrawal ruptures overall, and also found that only confrontation rupture markers were associated with outcome in the form of sudden losses. By contrast, Boritz et al. (2018) found higher incidence of confrontation ruptures than withdrawal ruptures overall, and found that only withdrawal rupture markers distinguished between recovered and unrecovered patients receiving DBT for borderline personality disorder. These findings may point to the possibility that ruptures that are “uncharacteristic” for a given population have a greater impact on outcome. More research with different populations and treatments is needed to identify treatment and/or sample characteristics that moderate the relations among confrontation ruptures, withdrawal ruptures, and outcome.

In addition to the possible role of sample characteristics, we should acknowledge the possibility that the relatively weaker psychometric qualities of withdrawal markers in this study indicate limitations in how withdrawal ruptures were conceptualized or coded. Does the higher frequency of withdrawal in this sample indicate that withdrawal was “overcoded,” with patient behaviors that are not related to problems in the alliance being judged as markers of rupture?

The failure to find convergence between the 3RS and the SASB may also indicate limitations in how we were conceptualizing withdrawal. Post hoc analyses revealed that SASB codes that we had conceptualized as related to withdrawal were instead correlated with 3RS confrontation markers. Alternatively, the failure to find the predicted associations between the SASB and 3RS could be due to the fact that the 3RS tracks interpersonal behaviors specifically in the context of alliance ruptures, while the SASB rates all patient and therapist interactions across the session, including in moments of strong alliance and good collaboration. Future research can capitalize on the ratings made every five minutes in order to examine if the measures do converge in moments of rupture and repair. Furthermore, it might be more useful to examine the convergent validity of the 3RS and measures with stronger conceptual links to alliance ruptures, such as the Collaborative Interaction Scale (CIS; Colli & Lingiardi, 2009), an observer-based measure of patients’ and therapists’ contributions to the alliance.

In summary, this study provides preliminary evidence that the 3RS is a useful tool for reliable and valid identification of processes that appear to be important in therapy, such as markers that predict dropout. Although the 3RS was not always related to other measures in the ways that we predicted, the relations we observed seem theoretically coherent.

This study was underpowered, particularly for analyses involving self-report measures of rupture and alliance. Also, the number of analyses conducted raised the risk of alpha inflation; we felt that the potential benefit of exploratory analyses was worth this risk, and we chose not to correct for experiment-wise error, but rather to submit the results to the reader for interpretation and to support future efforts at replication. This study also only looked at ratings for entire sessions: as the 3RS includes ratings every 5 minutes, there are many more sophisticated analyses that could be conducted in future studies to shed more light on rupture and resolution processes. For example, Moeseneder et al. (2018) examined how much of a session was marked by ruptures and resolution strategies, and also investigated the co-occurrence of resolution strategies and other therapist interventions within the same five-minute segment. Future studies should take advantage of codes of more microprocesses in order to better understand how rupture processes develop and evolve over time within particular clinical contexts.

Clearly, an important limitation is the extent to which our findings would generalize to other populations and other forms of treatment. The 3RS profile that emerged in this sample—with avoidant storytelling/shifting topic as the most common withdrawal marker, patient defends self against the therapist as the most common confrontation marker, and therapist redirects or refocuses patient as the most common resolution strategy (see Table 1), probably reflect this particular sample of patients with mood and anxiety disorders and some Cluster C personality disorders receiving CBT; different markers and strategies, and different relations among them, will likely be found in different samples. Our success at obtaining interrater reliability not just at the level of overall withdrawal and confrontation markers but also at the level of individual rupture markers and resolution strategies points to the viability of future research focusing on specific markers and strategies in different populations.

Sessions were selected from the early part of treatment; this may have impacted the types of ruptures and resolutions that were found and their relations to other variables, similar to how Gersh et al. (2017) found relations between outcome and early ruptures and late resolution. For example, the one resolution strategy that failed to demonstrate interrater reliability was a strategy that would likely appear later in treatment, as it required a therapist to link a current rupture to a pattern of ruptures from prior sessions.

This study relied on a US sample of predominantly white patients and therapists. The patients were highly educated, and the therapists were all trainees with limited clinical experience practicing one form of therapy. As research with the 3RS continues, it will also be important to consider how culture impacts how patients and therapists experience and express ruptures, and how observers interpret what they see. For example, Moeseneder et al. (2018) noted that their raters observed subtler presentations of alliance ruptures in their Swiss sample as compared to the examples in the 3RS manual, which were primarily drawn from American cases.

One important future direction for the 3RS is to further refine the coding of resolution processes. There may be additional strategies that should be added to the list. In addition, the way strategies are rated is limited: Currently, the 3RS assesses the frequency and impact of resolution strategies, but not their success at repairing a rupture. The only rating of successful resolution is a single global item. Researchers interested in assessing resolution processes would benefit from more refined ratings and the ability to link resolution attempts to specific rupture markers. Another potential limitation of the resolution strategies is the focus on techniques. Successful resolution may be less a matter of using a particular technique, and more a matter of following certain principles. For example, in our own work we have distinguished between immediate resolution efforts, aimed at just getting treatment back on track, and expressive resolution attempts, aimed at shifting the focus of the session to exploring the rupture and the patients’ underlying relational schemas (Safran & Muran, 2000)—would coding be both easier and yield more clinically useful findings if we focused on that distinction rather than specific resolution interventions? Or, following Goldfried (1980; Eubanks & Goldfried, in press), should we focus on clinical principles at a midlevel of abstraction, such as increasing patient awareness or providing a corrective experience?

An additional factor to consider is the stance of the therapist: to what extent is a general stance of openness, nondefensiveness, and curiosity an essential component of the resolution process, and how best can we capture that with the 3RS? This relates to another important area for further development of the 3RS: the finding that 3RS ratings of therapist contributions to ruptures predicted dropout points to the need for rupture markers of therapist behaviors. Both the Collaborative Interaction Scale and the observer-based version of the System for Observing Family Therapy Alliances (SOFTA-o; Friedlander, Escudero, & Heatherington, 2006) include markers of therapist behaviors that negatively impact the therapeutic alliance. These measures have both demonstrated interrater reliability and construct validity (Colli, Gentile, Condino, & Lingiardi, 2017; Friedlander et al., 2006), which suggests that coding therapist rupture markers is a feasible task. Drawing on the data we have collected with our current therapist contribution item, we hope to create therapist markers of rupture that better capture how both members of the dyad co-construct alliance ruptures (see Safran & Muran, 2000). In this vein, we will also explore the possibility of codes that capture how patients contribute to resolution processes.

Ultimately, it is inevitable that challenging patients, especially patients with interpersonal difficulties, will have difficulty forging a strong therapeutic alliance. It is also inevitable that therapists, being human, will contribute to alliance difficulties. Measures like the 3RS can aid our efforts to better understand how ruptures develop, how to recognize them, and how to respond effectively so that inevitable obstacles are transformed into clinical opportunities.

Clinical or methodological significance:

This study provides evidence of the reliability and validity of the 3RS, an observer-based measure of alliance ruptures and resolution processes. The 3RS can be used to identify problems in the therapeutic relationship that are associated with premature dropout from therapy.