You Never Know What You’re Going to Get: Large-scale Assessment of Therapists’ Supportive Counseling Skill Use

Abstract

Supportive counseling skills like empathy and active listening are critical ingredients of all psychotherapies, but most research relies on client or therapist reports of the treatment process. This study utilized machine learning models trained to evaluate counseling skills to evaluate supportive skill use in 3917 session recordings. We analyzed overall skill use and variation in practice patterns using a series of mixed effects models. On average, therapists scored moderately high on observer rated empathy (i.e., 3.8 out of 5), 3.3% of the therapists’ utterances in a session were open questions, and 12.9% of their utterances were reflections. However, there were substantial differences in skill use across therapists as well as across clients within therapist caseloads. These findings highlight the substantial variability in the process of counseling that clients may experience when they access psychotherapy. We discuss findings in the context of both the need for therapists to be responsive and flexible with their clients, but also potential costs related to the lack of a more uniform experience of care.

There are a number of forms of effective psychotherapy (Wampold & Imel, 2015). Common factors, or treatment elements common to established treatments are responsible for a significant portion of these treatment effects (Alldredge et al., 2021; Constantino et al., 2018; Horvath et al., 2011; Irving et al., 2004; Probst et al., 2019). Consistent with this evidence, a meta-analysis of 31 studies indicated that non-directive supportive counseling could achieve almost 75% of the total pre-post treatment effect size of active psychotherapies (Cuijpers et al., 2012). Ensuring that therapists are prepared to offer clients these foundational skills such as empathy and active listening is critical to providing a base of quality care to clients.

Supportive counseling is characterized by key interpersonal techniques such as active listening, reflections, encouraging, and helping clients explore their experiences and emotions, instead of giving advice or solving problems (see Areán et al., 2010). Taking various forms, supportive counseling is often operationalized in clinical trials as an attempt to provide a supportive, attentive therapist that might be helpful for a client, but not to provide any specific psychological treatment (e.g., CBT, IPT, etc.). Despite the long history of supportive counseling (Conte, 1994; Rockland, 1993; Winston et al., 1986) and evidence that simply listening actively in a supportive, empathic manner can be powerful (Cuijpers et al., 2012), the extent to which therapists regularly use the basic skills that are thought to be common in clinical practice remains unclear.

Assessment of Differences in Therapist Skill Use

Driven by concern that providers rarely use specific evidence-based treatments, work evaluating the skills utilized by therapists in community settings has focused largely on the dissemination and implementation of specific evidence-based treatments (McHugh & Barlow, 2010). Even after training, many providers fail to sustain use of the treatments they were trained to perform, drifting away from the core elements of the treatments over time (Schwalbe et al., 2014; Stirman et al., 2013). For example, in the largest study of its kind, Creed et al. (2016) found in a study of over 300 therapists (and almost 1000 observer-rated sessions), that therapist endorsement of a CBT orientation was not related to use of CBT skills. Therapists’ adherence to a specific treatment is subject to various sources of influence during the therapy session, including both client factors (Boswell et al., 2013; Imel et al., 2011; Snippe et al., 2019) and therapist factors (Snippe et al., 2019; Stirman et al., 2015). Therapists can account for about 13% to 40% of the total variance in adherence and competence to an intervention depending on the setting and type of intervention (Boswell et al., 2013; Imel et al., 2011, 2014). These studies also show that the client that a therapist worked with had a similarly large effect on use of various interventions, accounting for 14% to 32% of the total variance in therapist adherence and competence. Two other studies (Dunn et al., 2016; Hallgren et al., 2018) focused on single-session motivational interviewing interventions also demonstrated substantial variation in the skill use between therapists. Given this evidence for use of specific treatments, it is plausible that there is similar therapist variation in the use of supportive counseling skills.

One factor that might impact estimates of therapist performance in naturalistic settings is a reliance on human observers, which limits the scale of studies such that large representative samples are very difficult and slow to obtain. Specifically, therapists select which recordings to send and only a limited number can be rated. Studies that rely on therapist self-report of evidenced based practices may be subject to social desirability, recall, or other biases. Larger studies like Creed et al. (2016), can take many years to execute given the demands of human coding. However, there are emerging tools that make it possible to assess the therapist’s skill use on a larger scale.

Machine Learning-Based Evaluation of Psychotherapy

Natural language processing refers to a set of technologies used to transform unstructured text data (e.g. therapy transcripts) to more structured and useful information (see Pace et al., 2016 for discussion related to psychotherapy). At present, large-scale efforts to evaluate the quality of psychotherapy rely on human coding efforts that require hiring, training, and supporting teams that directly observe and rate sessions. This process is slow and expensive, and thus limits the scale of coding efforts even in the most ambitious and well-funded studies (Imel et al., 2017). Modern machine-learning-based technologies can utilize human ratings of psychotherapy sessions to train models that allow automatic recognition of key treatment processes, such as session topics (Imel et al., 2015; Dinakar et al., 2015), emotions (Maskit, 2021; Luo et al., 2020; Tanana et al., 2021), motivational interviewing fidelity (Atkins et al., 2014; Can et al., 2016; Imel et al., 2019; Xiao et al., 2012), cognitive behavioral therapy (Ewbank et al., 2020; Hilbert et al., 2020), and discussions of suicide (Walsh et al., 2017). In particular, these technologies can label core aspects of supportive counseling like empathy (Gibson et al., 2016; Xiao et al., 2012; Xiao et al., 2016; Xiao, Imel, Atkins, et al., 2015; Xiao, Imel, Georgiou, et al., 2015), interpersonal skills (Goldberg et al., 2021), open questions (Flemotomos et al., 2022), and reflections (Can et al., 2016); see Aafjes-van Doorn et al. (2020) for a detailed review.

One advantage of machine learning is the ability to scale up the sample size of psychotherapy studies from tens or hundreds to thousands of sessions. For example, Ewbank et al. (2020) trained a machine learning model to recognize indicators of cognitive behavioral therapy (CBT) in text-based counseling. They then utilized this model to evaluate the use of CBT in a large sample (90,934 text-based sessions). As this study relied on text-based counseling, it avoided a number of complications related to capturing session audio reliably, automated speech recognition, utterance segmentation, diarization (i.e., separation of speakers), and role assignment (i.e., determining who the therapist is). However, Flemotomos et al. (2022) described an online system that manages these practical and computation problems and facilitates direct evaluation of spoken language counseling at scale.

Current Study

Foundational education in psychotherapy includes training in basic supportive counseling skills, such as the techniques designed to communicate empathy and facilitate therapeutic relationships. Indeed, these skills represent the necessary background of many evidence-based treatments even if they are not considered the key ingredients. However, studies of therapist skill use in community settings are primarily focused on evaluating the impact of efforts to disseminate some specific type of evidence-based treatment, rather than focusing directly on the core skills that are important across treatments. It is possible that just as with evidence-based treatments, the use of these skills is far less prevalent than stakeholders might hope. As supportive counseling can have a substantial impact on clinical symptoms (Cuijpers et al., 2012), it is then important to assess the degree to which therapists use core supportive counseling skills.

In this study, we conducted the largest study of therapist behavior during in-person counseling to date¹. We used a machine-learning model trained on more than 2.8 million human-coded statements (see Flemotomos et al., 2022) to estimate therapists’ use of several basic counseling skills in a large collection of sessions. The dataset was collected under an IRB-approved research project on motivational interviewing (IRB_00083132). We hypothesized that therapist variability would be larger than previously found in dissemination studies (e.g., Boswell et al., 2013; Dunn et al., 2016; Hallgren et al., 2018; Imel et al., 2011), but that therapist performance would also vary across clients. As mentioned earlier, most of the previous studies were limited to particular treatment types, were limited in size, or relied on sessions obtained during treatment implementation efforts, which may have reduced variability in therapist performance due to training, selection, and supervision efforts. In Hallgren and colleagues’ (2018) study, for example, therapists accounted for 22% of the variability in empathy in training studies, which was most close to a naturalistic setting compared to efficacy and effectiveness studies. We expected therapist variability in our study to be close to or above this number.

Method

Data Source

After cleaning, the dataset for the current study included recordings of 3917 counseling sessions, or 2,624,298 utterances, obtained through an ongoing study of machine-learning-based observation of psychotherapy process at a large counseling center at a university in the western United States (Flemotomos et al., 2022). The dataset contained 1011 clients and 57 therapists. Sessions were digitally recorded in an audio form. Both therapists and clients had given formal consents to the recording of the sessions. The sessions were recorded with two microphones hung from the ceiling of the therapy rooms, one omni-directional and one directed to the therapist. The recorded sessions were processed in an automated pipeline that includes five steps: 1) voice activity detection, 2) speaker diarization, 3) speech recognition, 4) speaker role recognition, and 5) utterance segmentation. The pipeline converts audio recordings to speaker-labeled transcripts for each session, and then assigns therapist skill codes at the utterance level, as well as several additional session level tags, as described below (Flemotomos et al., 2022).

The client sample self-identified as 54.1% female, 42.0% male, and 3.9% other genders. The ethnic composition was 73.2% European Americans, 9.5% Asian Americans, 7.5% Latino/Latina Americans, 5.7% multi-ethnic, and 1.6% African Americans. Ages ranged from 18 to 54, with a mean of 23.4 and a median of 22. The therapist sample had similar demographic properties. For therapists, 56.1% identified as female, 26.3% identified as male, and 3.5% identified as genderqueer. In terms of ethnicity, 56.1% identified as European Americans, 8.8% Latino/Latina Americans, 5.3% Asian Americans, and 3.5% African Americans. Ages ranged from 21 to 69 years, with a mean of 32.8 and a median of 29. In the therapist sample, 18.2% self-identified as cognitive-behavioral therapy oriented, 15.5% as multicultural/feminist oriented, 12.8% as interpersonal therapy oriented, 12.2% as integrated/eclectic, 10.8% humanistically oriented, and 5.4% were psychodynamically oriented.

Measures

Therapist use of basic counseling skills was assessed with a machine-learning model trained on human ratings of psychotherapy sessions (Flemotomos et al., 2022). For this study, we selected specific items that assessed basic active listening and counseling skills from the Motivational Interviewing Skill Code 2.5 (MISC 2.5; Houck et al., 2010). In this study, we used three measures at session level: 1) empathy, 2) count of reflections, and 3) count of open questions. Empathy is defined as the extent to which therapists understand or attempt to understand the client’s perspective. The rating of empathy ranges from 1 (low) to 5 (high). A low rating is defined as the therapist showing little interest in the clients’ own experiences. In contrast, high empathy is characterized by the assessed counselor showing active attentiveness to the clients’ worldview.

Two types of reflective statements, simple reflection and complex reflection, were combined into one overall count of reflections. A simple reflection is defined as the therapist capturing and returning to the client something that the client has said previously in the same session, without adding meaning or emphasis to the client’s statement. Simple reflection is used to merely convey understanding or facilitate conversation. An example of a simple reflection could be a therapist saying “You feel sad; and you want to hide away” after the client said “I feel sad and just want to hide.” A complex reflection enriches what the client conveys by adding significant meaning or emphasis to the client’s statement. For example, a therapist may say: “It sounds like you feel angry when your colleague did that to you” after the client shared that her colleague talked behind her back.

Open questions are questions that leave latitude for responses, instead of the questions that can often be answered with simple responses such as yes or no. An open question oftentimes begins with wh-words and one example could be “What does it mean when you said I just feel it’s impossible?” (Houck et al., 2010).

Because the system predicts reflection and open questions at the utterance level, we calculated the frequency of each measure in a session to obtain the session-level counts. Empathy is scored at the session level and does not need further transformation. The performance of the system was evaluated by accuracy and F₁ score, with the latter being the harmonic mean of precision and recall. The accuracy for predicting high or low empathy was .851. The F₁ score was .342. The F₁ scores for reflections and open questions were .612 and .825, respectively (Flemotomos et al., 2022).

Analysis

Statistical Modeling.

We used generalized linear mixed modeling (GLMM) to examine differences in use of the basic skills highlighted above. GLMM is a generalized version of linear mixed modeling (LMM) or multilevel modeling. LMM fits the nested structure of counseling (Baldwin et al., 2007), where one therapist often provides care to multiple clients, and clients attend multiple sessions. This nested structure violates the independence assumption of commonly used statistical methods such as ANOVA and multiple linear regressions. Compared to LMM, GLMM is more flexible with the choice of model distributions and therefore can model assorted types of data, such as time length (continuous), dropout (binary), or the frequency of using a particular intervention (count). In this study, we constructed three-level empty GLMMs with sessions (Level 1) nested in clients (Level 2) nested in therapists (Level 3). For continuous variables (i.e. empathy), we used a normal distribution. With normal distributions, GLMM is equivalent to LMM. In particular, empathy was modeled with robust-error LMM².

$$\begin{gathered} Y_{ijk}~N\left({\mu }_{ijk},{\epsilon }_{ijk}\right) \\ \mathit{\text{Level}}1:{\mu }_{ijk}={\beta }_{0jk} \\ \mathit{\text{Level}}2:{\beta }_{0jk}={\gamma }_{00k}+u_{0jk} \\ \mathit{\text{Level}}3:{\gamma }_{00k}={\delta }_{000}+v_{00k} \end{gathered}$$

Y_ijk is the raw empathy score for client j and therapist k in session i, μ_ijk is the estimated mean of the empathy scores and ε_ijk is the residual uncounted by the model. β_0jk is the mean score across all sessions of client j, treated by therapist k, representing the mean empathy level with a particular client or client-therapist dyad. γ_00k is the mean score across all clients within therapist k, representing the mean empathy level of a particular therapist. δ₀₀₀ is the grand mean, representing the average empathy level of the entire sample. u_0jk and v_00k are random-effect variables and assumed to be multivariate normal centered at 0 and have a covariance matrix with τ_j² and τ_k² on the diagonal. For count variables (i.e. reflection and open questions), we used a negative binomial distribution with offset terms and a log link³ to model the data.

$$\begin{gathered} Y_{ijk}~NB\left({\mu }_{ijk},\alpha \right) \\ \mathit{\text{Level}}1:\text{ln}\left({\mu }_{ijk}\right)={\beta }_{0jk}+\text{ln}\left(N_{ijk}\right) \\ \mathit{\text{Level}}2:{\beta }_{0jk}={\gamma }_{00k}+u_{0jk} \\ \mathit{\text{Level}}3:{\gamma }_{00k}={\delta }_{000}+v_{00k} \end{gathered}$$

α is a dispersion parameter accounting for the over- or under-dispersion of the data. ln(N_ijk) is an offset term accounting for the varied session lengths that may influence the estimation of count variables (i.e. a longer session may likely have more reflections than a shorter session). Here N_ijk is the total number of therapist talk turns for therapist k in session i with client j. The addition of an offset term under a log link essentially converted the estimation of the mean counts to the estimation of the mean proportions. For example, an estimated δ₀₀₀ of −1.39 for reflection means that, on average, 25% of the therapists’ utterances are reflections (the natural exponent e to the power of −1.39 is about .25).

For each of the three variables, client and therapist effects were assessed using intraclass correlation (ICC).

$$\begin{gathered} IC{C}_j=\frac{{\tau }_j^2}{{\tau }_j^2+{\tau }_k^2+{\tau }_{\epsilon }^2} \\ IC{C}_k=\frac{{\tau }_k^2}{{\tau }_j^2+{\tau }_k^2+{\tau }_{\epsilon }^2} \end{gathered}$$

where ICC_j represents the client effect and ICC_k represents the therapist effect. τ_k² is the between-therapist variance, τ_j² is the within-therapist variance, and σ_ε² = var(ε_ijk) is the variance of the residuals. σ_ε² was estimated by the summation of distribution-specific variance and additive overdispersion (Nakagawa & Schielzeth, 2017). ICC_k can be interpreted as the extent to which therapists vary among each other in their use of the skill assessed. A high ICC_k implies a strong therapist influence on that particular skill. For example, an ICC_k of .13 for empathy means that 13% of the variability in the empathy scores is due to some differences between therapists. Likewise, ICC_j can be interpreted as the extent to which therapists vary in their use of a skill among the clients within their caseloads.

All analyses were conducted in R (R Core Team, 2020). GLMMs were implemented using the lme4 package (Bates et al., 2015). Robust-error LMM was implemented with the robustlmm package (Koller, 2016). ICCs were calculated using the performance package (Lüdecke et al., 2021).

Data Reduction

We conducted a series of data cleaning steps before modeling the data. First, as this data was collected during regular clinical practice, we expected to encounter a number of recordings that were problematic (i.e., accidental recordings, microphone issues, etc.). We inspected the density distributions of the sessions based on session length, number of utterances, and therapist talk time ratio. We set cut-off points to remove long tails in the density distribution plots, which were considered outlier sessions. In particular, we removed any recording shorter than 20 minutes or longer than 70 minutes, or with less than 110 or more than 1300 utterances. These sessions constituted 4.3% of the total sample size. In addition, there were cases where automatic separation of speakers failed - resulting in sessions where speaker labels were almost uniformly therapist or client. To address this problem, we removed sessions where therapist talk time made up either less than 10% or more than 75% of the session. The cleaned dataset contained 3917 sessions, 1011 clients, and 57 therapists, which was 91.8% of the total collected data (N = 4269).

Second, we censored skill labels from brief therapist statements. As described in Flemotomos et al. (2022), the automatic system not only transcribes the session and labels counseling skills, but it also uses a sentence parser to determine when a therapist talk turn starts and stops (i.e., where a label should be applied). This is a challenging problem in spoken dialogue with interruptions, disfluencies, back-channels, within speaker pauses, etc, and thus introduces another source of error - i.e., to what text should the system provide a counseling skill label (indeed, this is a source of error almost never considered in psychotherapy research). As we inspected the raw transcript data, we found a higher preponderance of brief talk turns in auto-generated transcripts compared to human generated transcripts. Specifically, in some cases, the system appeared to be too sensitive in segmenting talk turns. As a result, in some instances, the automatic system has very limited linguistic information to accurately predict a label. To reduce noise from brief talk turns when the system has limited linguistic information, we removed the skill labels of those therapist talk turns with less than 8-words prior to statistical modeling.⁴

Results

The descriptive statistics of the three assessed variables after data cleaning are shown in Table 1. The therapists’ average empathy score was 3.86. On average, there were 36 (12.9%) coded reflections and 9 open questions (3.3%) per session. As an illustration, we have provided sample text of coded statements in two sessions in Figures 1 and 2.

Table 1.

Descriptive Data of Supportive Counseling Skills

	Mean (SD)	Median
Empathy	3.86 (0.39)	3.91
Reflection (counts)	36.44 (16.46)	34
Open Question (counts)	9.16 (5.96)	8

Figure 1.

Therapists’ Use of Open Questions in an Example Session

Note. The blue bars represent the therapists’ utterances coded as open questions. The gray bars are therapists’ utterances other than open questions. The blank areas are clients’ utterances. The length of each bar is proportional to the number of words in the utterance.

Figure 2.

Therapists’ Use of Reflections in an Example Session

Note. The blue bars represent the therapists’ utterances coded as reflections. The gray bars are therapists’ utterances other than reflections. The blank areas are clients’ utterances. The length of each bar is proportional to the number of words in the utterance.

Very brief statements (less than 8 words) represented 61.7% of the total talk turns, but only 25.1% (N = 2,713,071) of the total words. The percentage of open questions before removing the brief talk turns was 7.3% (N = 82,137), and reflections were 34.7% (N = 390,921). If brief responses are entirely removed from the analysis (i.e., not considered in the count of messages), the percentages remain relatively stable with 8.2% in open questions, and 33.0% in reflections. If brief talk turns labels are treated as missing but the messages are still included in session counts, the relative frequency of reflections (3.1%, N = 35,252) and open questions (12.6%, N = 142,575) and open questions are substantially reduced. This indicates that therapists on average spent 3.1% of all their speech using reflections, and 12.6% using open questions.

As indexed by the ICC, therapist variation in the use of each skill was relatively large. Therapists account for 16.9% (ICC_k) of the variability in empathy ratings, 23.4% of variability in open questions, and 25.7% of the variability in reflections. As mentioned earlier, these numbers imply to what extent therapists differed in empathy ratings. For example, 16.9% of the variability in empathy ratings can be explained by the differences in therapists per se.

Differences in average therapist skill use across clients were similarly large, if not somewhat larger than therapist differences. The client ICC_j for empathy was 38.1%, which was about twice the therapist effect of the same rating. The client ICC_j of the counts of open questions and reflections were 27.0% and 31.8%, respectively. This suggests that the differences in client-therapist dyads had a much stronger influence on empathy ratings than therapists. On the other hand, the frequency of the use of open question and reflection was equally affected by both client-therapist dyads and by therapists. The mean and variance estimates of the three variables are shown in Table 2.

Table 2.

Estimates of 3-level Models of Supportive Counseling Skills

	Fixed Effect (Intercept)	Between Therapist Variance (τk2)	Within Therapist Variance (τj2)	Residual (σε2)	Client ICC (ICCj)	Therapist ICC (ICCk)
Empathy	3.829	0.023	0.052	0.061	38.1%	16.9%
Open Question	0.03a	0.082	0.094	0.174	27.0%	23.4%
Reflection	0.124a	0.031	0.039	0.053	31.8%	25.7%

Note. Empathy score was modeled with robust-error LMM; open questions and reflections were modeled with negative binomial GLMM with the count of therapists’ utterances in a session as an offset term.

^aWith offset terms, the estimated intercepts represent average proportions over the therapists’ total utterances in a session. The presented values have been converted back to the original scale through log link.

Discussion

We used generalized linear mixed effects modeling to examine therapists’ use of basic counseling skills in a machine-learning augmented dataset from a counseling center at a large university in the western United States. In line with prior findings (Boswell et al., 2013; Imel et al., 2011), our study demonstrated that therapists varied among each other in their use of basic counseling skills. In addition, therapists also varied in their use of the basic skills in treating different clients. Contrary to our hypothesis, the extent to which therapists and clients varied among each other in skill use did not differ considerably from prior implementation and dissemination studies.

For the empathy rating, the therapist ICC was much smaller than the client ICC (16.9% vs. 38.1%), meaning that therapists’ empathy differed more within their caseloads than it did between their caseloads. This pattern is similar to prior work on therapist differences in adherence to specific evidence-based interventions (e.g., Imel et al. 2011; Boswell et al. 2013) in that clients were a larger source of variance in adherence than therapists. However, differences between and within therapist caseloads were comparable and large for both open questions and reflections.

Two hypotheses may be made on why therapists differed in the use of basic skills across different clients in their caseload. First, particular clients may make it more difficult for therapists to express empathy in general. Alternatively, a particular client might be a poor fit for a therapist such that it is more difficult for the therapist to express empathy with that specific client. This phenomenon could be due to countertransference (Peabody & Gelso, 1982), or some clients might be generally less expressive, making it harder to understand their experience (Zaki et al., 2008). The second possibility is that therapists are adapting to clients’ characteristics or needs at the moment. For example, one client may prefer a more didactic, solution-focused style while another client may prefer an exploratory, flexible style that utilizes reflections and questions extensively (Beutler et al., 1991). Indeed, rigidity to a specific treatment approach can be detrimental to outcomes (Castonguay et al., 2010), while therapists’ flexibility or responsiveness (Constantino et al., 2020; Stiles & Horvath, 2017; Watson & Wiseman, 2021) is linked to positive process and outcomes (Katz et al., 2019, 2020; Li et al., 2020; Owen & Hilsenroth, 2014). In this situation, it is a therapist’s skill in itself to be able to flexibly use these nonspecific skills depending on clients.

However, our study showed that empathy had even greater between-client variability (38.1%) than use of reflection or open questions (27.0–31.8%). If the observed between-client variability was mainly due to therapist flexibility, it should be smaller with empathy. Empathy is broadly considered to be a core common factor and as such, one might expect expression of empathy to be relatively less variable in skilled therapists. Therefore, it is not likely that therapist flexibility alone explains the observed between-client variability.

Future work should focus on assessing the association between between-client variability and treatment outcomes. Hypothetically, the two situations (i.e., therapist flexibility vs. difficulty using specific skills) would show opposite effects. If the between-client variability positively predicts therapy processes and outcomes, it would indicate that therapists are varying skill use appropriately with different clients - actively adapting to clients’ needs. If the variability negatively predicts therapy processes and/or outcomes, it indicates that there is something detrimental in therapists varying their skill use, thus pointing to the situations where differences in the use of particular skills are not helpful.

Our findings also showed that therapists varied among each other in their use of supportive counseling skills. The extent to which therapists varied was consistent with findings from prior studies on specific treatments (Boswell et al., 2013; Imel et al., 2011). Knowing how therapists use these skills can be important to exploring differences in the use of specific treatments, as supportive skills are found broadly across training programs and foundational in many treatment modalities. One possible explanation for the observed therapist differences may be therapists’ affiliation to particular theoretical modalities. For example, therapists who used cognitive behavioral therapy were more likely to offer advice than reflection compared to psychodynamic therapists (Wiser & Goldfried, 1996). Although reflection is seen across theoretical orientations, therapists may have different opinions on when and how to use reflection depending on their theoretical frameworks, thus contributing to the therapist differences observed in this study.

It may also be possible that the differences in the use of specific supportive counseling skills between therapists would potentially pose a concern about the quality of care that clients received, given a literature review showing that these nonspecific skills can result in significant change on their own (Cuijper et al., 2012). Nonspecific skills such as empathy (Elliott et al., 2011, 2018; Greenberg et al., 2001; Nienhuis et al., 2018) and the use of reflection and open questions (Magill et al., 2014, 2018; Pace et al., 2017) are consistently found to predict treatment processes and outcomes. How therapists use these basic skills are possible to play an important role in the quality of care that clients receive. Therefore, it may be of interest to assess how the competence and adherence to these nonspecific skills relate to treatment outcomes. This could help answer the question about whether the finding in the present study, that therapists differed among each other in the use of nonspecific skills, is a concern of quality of care.

Limitations and Future Directions

Several limitations are worth noting here. First, the trained model we used was not perfectly reliable with human raters (human raters are also not perfectly reliable with each other). The F₁ scores for open questions and reflections were acceptable, but there is a large discrepancy between accuracy and F₁ score for empathy (see Flemotomos et al., 2022). Another limitation is the cleaning process we applied to the dataset. We found overcounting of some intervention codes due to very short utterances, which we removed. This method, although intuitive and simple, could potentially overcorrect the issue. However, the estimated ICCs remained relatively stable during this iterative process, which indicates that the distributions of therapist effect and client effect were not affected by this cleaning process. As mentioned, the machine-learning model we used may have been oversensitive in parsing statements, which resulted in the higher proportion of very short utterances compared to human parsing. These short utterances may in turn pose difficulties in the subsequent speech recognition due to limited phonetic information (Kanagasundaram et al., 2011). More recent techniques for short utterance issues in machine learning (e.g., Jung et al., 2019; Kanagasundaram et al., 2011) may benefit the model with a better approximation to the human parsing of statements.

It may also be of interest to investigate if process and outcome variables are associated with therapists’ use of supportive counseling skills. Like empathy, it might be of interest to investigate if the associations based on machine-learning processed large datasets align to those found in meta-analyses, thus offering criterion validity of using such methods. In addition, although explicitly conceptualized in motivational interviewing (Houck et al., 2010), open questioning or reflections are also found in other theoretical orientations to facilitate treatment processes (e.g., Apodaca et al., 2016; Anvari et al., 2020). However, there is a limited number of studies that assess these basic skills in naturalistic settings, which may be an important step given the prevalence of nonspecific skills in all treatments. Given the heterogeneity of the sample in a naturalistic setting, various moderators such as therapeutic orientation can be added to further specify the differential use and effects of these basic skills on treatment outcomes.

In summary, our findings show that therapists varied in their use of supportive counseling skills, not only among each other but also among their caseloads. Using a pre-trained machine learning model (Flemotomos et al., 2022), we scaled up process-outcome research in a naturalistic setting, which had largely been based on human coding in the past and suffered from several limitations (Imel et al., 2017). Our findings replicated what was found in the prior studies regarding the therapist and the client effects in the use of particular skills (Boswell et al., 2013; Imel et al., 2011), which offered support to the potential use of automatic measures in scaling up process-outcome studies in psychotherapy research. This study provided evidence that with reliably trained machine-learning models, the data collected from these models can be useful for process-outcome analyses where researchers would often need human coding.

Clinical Impact Statement.

Question:

We conducted a large-scale observational study to address the knowledge gap about clinicians’ use of basic counseling skills, such as empathy and active listening skills.

Finding:

We found that clinicians varied substantially in their use of basic counseling skills across clinicians as well as within their caseloads.

Meaning:

This variation suggests that quality improvement efforts in behavioral healthcare might focus not only on the implementation of specific treatments, but also ensuring the use of basic counseling skills.

Next Steps:

Future studies will continue to explore the impact of this variation on psychotherapy outcomes.