Skip to main content
NCBI home page
Wiley - PMC COVID-19 Collection logoLink to Wiley - PMC COVID-19 Collection
. 2021 Nov 22;28(6):1403–1415. doi: 10.1002/cpp.2682

Psychotherapists' acceptance of telepsychotherapy during the COVID‐19 pandemic: A machine learning approach

Vera Békés 1,✉, Katie Aafjes‐van Doorn 1, Sigal Zilcha‐Mano 2, Tracy Prout 1, Leon Hoffman 3
PMCID: PMC8652775  PMID: 34723404

Abstract

Objective

This study aimed to develop predictive models of three aspects of psychotherapists' acceptance of telepsychotherapy (TPT) during the COVID‐19 pandemic, attitudes towards TPT technology, concerns about using TPT technology and intention to use TPT technology in the future.

Method

Therapists (n = 795) responded to a survey about their TPT experiences during the pandemic, including quality of the therapeutic relationship, professional self‐doubt, vicarious trauma and TPT acceptance. Regression decision tree machine learning analyses were used to build prediction models for each of three aspects of TPT acceptance in a training subset of the data and subsequently tested in the remaining subset of the total sample.

Results

Attitudes towards TPT were most positive for therapists who reported a neutral or strong online working alliance with their patients, especially if they experienced little professional self‐doubt and were younger than 40 years old. Therapists who were most concerned about TPT were those who reported higher levels of professional self‐doubt, particularly if they also reported vicarious trauma experiences. Therapists who reported low working alliance with their patients were least likely to use TPT in the future. Performance metrics for the decision trees indicated that these three models held up well in an out‐of‐sample dataset.

Conclusions

Therapists' professional self‐doubt and the quality of their working alliance with their online patients appear to be the most pertinent factors associated with therapists' acceptance of TPT technology during COVID‐19 and should be addressed in future training and research.

Keywords: COVID‐19, machine learning, online therapy, telepsychotherapy, therapists, UTAUT model

Key Practitioner Message.

  • Therapists with neutral or strong online working alliance had the most positive attitudes towards telepsychotherapy.

  • Therapists with higher levels of professional self‐doubt and especially with higher vicarious trauma were most concerned about using telepsychotherapy.

  • Therapists with low working alliance were least likely to use telepsychotherapy in the future.

1. INTRODUCTION

Providing psychotherapy remotely has multiple advantages, such as providing access to patients in remote areas and with reduced mobility, avoiding the stigma of visiting a mental health clinic and reduced travel time and associated costs (e.g. commute and childcare; Békés, Grondin, & Bouchard, 2020; Simpson, 2009). The COVID‐19 pandemic has created a unique context in which many therapists have abruptly transitioned to telepsychotherapy (TPT), without much preparation or training in advance. Therapists' experience of this sudden transition to a new therapy format might determine whether they will continue conducting therapy online, once pandemic restrictions are lifted. It is therefore important to understand which therapist, patient and treatment factors contribute to therapists' acceptance of the TPT format and their intention to continue providing therapy online. Various terms have been used to refer to remote therapy (Rochlen et al., 2004), and in this study, based on the most recent literature, we use the term ‘telepsychotherapy’ to refer to psychotherapy sessions provided synchronously online via videoconferencing (Markowitz et al., 2020; Poletti et al., 2020; Rosen et al., 2020; Van Daele et al., 2020).

Preliminary studies have shown that despite the involuntary transition to TPT during the pandemic, therapists have a somewhat positive view of TPT (Feijt et al., 2020, McBeath et al., 2020) and are inclined to use it in the future (Békés & Aafjes‐van Doorn, 2020). In earlier studies, several factors have been proposed that may impact therapists' views of TPT (for a review, see Connolly et al., 2020). One important factor is the perceived ability to develop a strong therapeutic relationship remotely (Roesler, 2017; Sucala et al., 2013). Pre‐pandemic studies showed that in fact, the therapeutic relationship is generally strong in TPT (see review by Norwood et al., 2018) and similar to in‐person therapies (e.g. Bouchard et al., 2020; Watts et al., 2020). This was true not only pre‐pandemic but during the pandemic as well (Aafjes‐van Doorn, Békés, & Prout, 2020; Aafjes‐van Doorn, Bekés, Prout, & Hoffman, 2020). Moreover, during the pandemic, many therapists also reported that they were as authentic and genuine in their online sessions as they were before in in‐person (Békés, Aafjes‐van Doorn, et al., 2020; Békés, Grondin, & Bouchard, 2020).

Before the start of the pandemic, few therapists had attended training in TPT, and most felt uncomfortable using technology to provide psychotherapy (e.g. Essig & Russell, 2017; Glueckauf et al., 2018). Therapists' sense of uncertainty regarding their ability to provide effective therapy online likely further increased during this sudden transition to TPT during the pandemic. Especially among therapists with less clinical and online experience at the start of the pandemic, and those struggling to build strong therapy relationships, professional self‐doubt has been found to be salient (Aafjes‐van Doorn, Békés, & Prout, 2020).

Moreover, another salient factor in therapists' attitudes towards TPT during the COVID‐19 pandemic is their experience of vicarious trauma. Due to the global health crisis and its negative mental health impact, therapists may have been more exposed to more stressful and traumatic material during their sessions than before the pandemic. Indeed, previous studies showed that on average, therapists experienced a moderate level of vicarious trauma during the pandemic and about 15% of therapists experienced high levels (Aafjes‐van Doorn, Bekés, Prout, & Hoffman, 2020).

These preliminary findings suggest that therapists' ability to create a strong therapeutic relationship online, their professional self‐doubt and experienced vicarious trauma likely impact therapists' acceptance of the TPT format. However, little is known about which of these factors are most strongly associated with TPT acceptance and how they interact with one another when relating to acceptance of TPT technology during the pandemic.

Although traditional regression models and machine learning models, such as decision trees, may both base their variable selection on pre‐existing theory, a decision tree model has certain statistical benefits. Traditional statistical approaches test each independent variable as a separate hypothesis, which can lead to erroneous conclusions because of multiple comparisons (inflated Type I errors), model misspecification and multicollinearity. In contrast, machine learning models are not constrained by model assumptions and are particularly helpful for finding patterns in complex datasets (Bi et al., 2019). A decision tree model, for example, is able to predict the mean level of the dependent variable based on exploring interactions between potential independent variables (Loh, 2008). In an attempt to improve the ability of identified models to generalize to new data that will be collected in the future, a machine learning model is usually developed in the training set and subsequently tested in a separate test set (Aafjes‐van Doorn et al., 2021). This process of cross‐validation avoids model overfit on the original data and mimics a replication of the model on an independent dataset, and it thus increases the confidence in the model's generalizability.

The aims of the current study were twofold: (1) to examine the extent of therapists' acceptance of TPT technology during the pandemic and their attitudes towards TPT technology, concerns about the TPT technology and intention to use TPT technology in the future and (2) to develop predictive models for each of these three aspects of therapists' acceptance of TPT technology, based on the therapists' characteristics, reported quality of the therapeutic relationship in TPT and their levels of professional self‐doubt and vicarious trauma during the pandemic by applying decision tree machine learning analyses.

2. METHODS

2.1. Procedures and participants

This cross‐sectional study was based on Qualtrics‐based online survey data about psychotherapists' experiences and attitudes towards TPN during the pandemic. Preliminary results on small subsamples based on less than 20% of the presented data and focused on different research questions have been published in previous reports (omitted for peer review). The present study reports the results on all survey data collected between 25 March and 29 June 2020. Psychotherapists were recruited via professional email listservs, social media and individual contacts across the United States, Canada, China and Europe. Licensed and trainee therapists, currently seeing patients online, were eligible to participate. Interested participants were directed to an online survey platform, which provided additional information about the study. After providing consent, participants were directed to the anonymous questionnaire. Several individual items and standardized scales were administered in a fixed order, taking approximately 15 min to complete. The study was approved by (the local—omitted for peer review) institutional review board.

In the present study, out of the 1911 participants who clicked on the survey link, we included participants who completed all the relevant measures (n = 795). Due to the forced response setting in the survey, there were no missing data. The participating therapists were mostly female (n = 603; 75.8%), and White (n = 562, 70.7%); their average age was 52.72 (SD = 16.36), and more than half of the sample (n = 399, 57.2%) had more than 17 years of clinical experience, with only 7.9% of therapists reporting 4 years of clinical experience or less. This sample is representative of the larger population of US psychologists which is 65% female and 84% White and has a mean age of 45.9 (American Psychological Association, 2018). Most participating therapists were located in North America (n = 650; 81.8%). Detailed demographic data about the study sample are presented in Table 1.

TABLE 1.

Therapist demographics and descriptives (n = 795)

n %
Ethnicity
White European, European American 562 70.7
Asian or Asian Indian 70 8.8
Native Hawaiian or Pacific Islander 2 .3
Hispanic, Latinx, Spanish 30 3.8
Black or African American 6 .8
American Indian or Alaska Native 3 .4
Middle Eastern 9 1.1
Other 40 5.0
Country (declaration of national emergency/lockdown a )
USA (March 13 Trump declares national emergency) 650 81.8
33 4.2
Canada (March 16) 45 5.7
China (January 29) 20 2.5
United Kingdom (March 25) 29 3.6
Other European countries (March 9–March 25) 13 1.0
8 1.0
Other—Latin America (March 16–17) 10 1.3
Profession
Psychologist 362 45.5
Social worker 119 15.0
Counsellor 104 13.1
Medical doctor 115 14.5
Psychoanalyst 69 8.8
Marriage family therapist 17 2.2
Psychotherapist 27 3.4
Other 2 0.3
Licensure status
Licensed 718 90.3
Trainee 77 9.7
Clinical experience in years
0–4 55 7.9
5–8 102 13.0
9–12 82 11.8
13–16 59 8.5
17 or more 399 57.2
Work setting b
Private practice 653 83.0
Outpatient clinic 142 18.0
Hospital 71 8.9
Other 62 7.9
Theoretical orientation b
Psychodynamic 528 67.1
Psychoanalytic 406 51.6
Integrative 280 35.6
CBT 182 23.1
Humanistic 133 16.9
Systemic 99 12.6
Other 106 13.5
Previous experience of providing TPT
No, never 353 44.4
Yes, once or twice 75 9.4
Yes, but only after seeing them in‐person first 223 28.1
Yes several patients 143 18.0
Previous training in TPT
Yes 136 17.1
No 659 82.9
Open in a new tab

Abbreviation: TPT, telepsychotherapy.

b

Multiple answers were possible per respondent.

2.2. Measures

The online survey included individual items that assessed for therapists' demographics and professional activities as well as several standardized scales that assessed for variables of interest, including acceptance of TPT, vicarious trauma, professional self‐doubt and scales about the therapeutic relationship.

2.2.1. Demographics and professional activities

Items included age, gender, race/ethnicity, highest degree, treatment orientation and setting, patient population, licensure, years of experience and number of patients. Additional items inquired about previous TPT experience and training.

2.2.2. Attitudes towards TPT

The novel Unified Theory of Acceptance and Use of Technology Therapist Version (UTAUT‐T; Békés et al., 2021) was used to assess attitudes towards acceptance and usage of TPT. The UTAUT framework (Venkatesh et al., 2003, 2012) offers a comprehensive model of acceptance and subsequent utilization of technological innovations. It has been adapted for various contexts, including occupational and physical therapy (Liu et al., 2015). The UTAUT theoretical framework was recently used in a systematic review of pre‐pandemic empirical studies as an overarching conceptualization of different aspects of acceptance of TPT technology to help frame their inclusion criteria and description of their findings (Connolly et al., 2020; see Venkatesh et al., 2016). In line with the focus of the present study, the phrasing of the items was adapted to reflect TPT. For example, the item ‘People who are important to me think that I should use the system’ in the original UTAUT measure was adapted to ‘People who are important to me think that I should use online therapy’, whereas other items needed more modifications, for example, the original ‘Using the system enables me to accomplish tasks more quickly’ was adapted to therapists as ‘Using online therapy saves me time and/or money’ (for more details on the development of UTAUT‐T, see Békés et al., 2021). The original and extended UTAUT models have been validated in a variety of settings, including academic research (Gruzd et al., 2012), higher education (Pynoo et al., 2011) and healthcare settings (Alapetite et al., 2009; Chang et al., 2007; Liu et al., 2015), for a review, see Venkatesh et al. (2016), whereas the novel UTAUT‐T has not been validated on therapist samples yet.

The UTAUT for psychotherapists includes three scales: Attitudes, Anxiety and Behavioural Intention. The 21‐item UTAUT‐T Attitudes subscale includes items related to performance expectancy, effort expectancy, social influence and facilitating conditions regarding using TPT. The six‐item UTAUT‐T Anxiety subscale includes items regarding feelings of apprehension or concerns about using technology (Compeau et al., 1999). The Anxiety subscale has been shown to have a direct negative effect on actual technology use and is thus seen as an important inhibitory variable (Cenfetelli & Schwarz, 2011). Finally, the two‐item UTAUT‐T Behavioural Intention subscale represents explicitly declared intent and plan to use technology in the future.

As with all UTAUT adaptations, items of the UTAUT‐T scales are scored on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree), with higher scores indicating more positive attitudes towards technology (Attitudes Scale), more concerns (Anxiety Scale) and more intention for future use of TPN (Behavioural Intention Scale). In the present study, the internal consistency of the UTAUT‐T subscales was α = .78 (Attitudes), α = .80 (Anxiety) and α = .94 (Behavioural Intention).

2.2.3. Working alliance

Therapeutic alliance was measured with the Working Alliance Inventory – Short Revised – Therapist (WAI‐SRT; Hatcher & Gillaspy, 2006) which includes four positively worded items for each of three subscales, based on Bordin's (1979) pantheoretical model: bond, agreement on tasks and agreement on goals. The WAI‐SRT uses a 5‐point Likert scale, ranging from seldom (1) to always (5), and has been validated by associations with other alliance measures and by prediction of therapy outcomes (Munder et al., 2010; Zilcha‐Mano, 2017).

A global working alliance rating of 4 (sometimes), the middle point of the scale, is interpreted as a neutral working alliance (Horvath & Greenberg, 1989). The WAI‐SRT has shown adequate reliability and validity (Hatcher & Gillaspy, 2006). The mean score of the WAI‐SRT was used, with higher scores indicating a stronger working alliance. Cronbach's α for the total scale in the current sample was .85.

2.2.4. Real relationship

The Real Relationship Inventory Therapist Form (RRI‐T; Gelso et al., 2005) assesses the genuine human relationship between patient and therapist from the therapist perspective. In Gelso's tripartite model (Gelso et al., 2018), the RR is conceptualized as an ongoing quality of the relationship distinguished from transference and the working alliance. The RR has been found to be highly correlated with the Working Alliance Inventory (WAI; Horvath & Greenberg, 1989), especially with the WAI's Bond subscale (Kelley et al., 2010); however, ratings of real relationship predict treatment progress and outcome above and beyond the variance explained by the WAI (Lo Coco et al., 2011; Marmarosh et al., 2009). The RRI has 24 items that use a 5‐point Likert scale from strongly disagree (1) to strongly agree (5), with higher overall scores reflecting a more genuine and authentic relationship. Reliability has shown to be high, with coefficient alphas ranging from .80 to .90 in various samples (e.g. Fuertes et al., 2013, 2019; Marmarosh et al., 2009). In this study, Cronbach's α was .73.

2.2.5. Professional self‐doubt

The Professional Self‐Doubt Scale (PSD; Nissen‐Lie et al., 2017) is a nine‐item scale derived from the larger Development of Psychotherapists Common Core Questionnaire (DPCCQ; Orlinsky et al., 1999). The PSD assesses the level of uncertainty a therapist has in their ability to help a patient. Items are rated on a 6‐point Likert scale from 0 (never) to 5 (very often), with higher total score indicating more self‐doubt. The PSD includes nine items about how confident and competent the therapist feels, and items are rated on a 6‐point Likert scale from 0 (never) to 5 (very often) and summed for a total score, with higher scores indicating more self‐doubt. In our study, Cronbach's α was .84.

2.2.6. Vicarious trauma

The Vicarious Trauma Survey (VTS; Vrklevski & Franklin, 2008) is a self‐report measure of distress associated with working with traumatized patients. The VTS includes eight items, from which the first two ask about vicarious trauma exposure, whereas the other six ask about distress due to the exposure. Because the first two items are screening questions and do not assess the subjective experience of vicarious traumatization (Aparicio et al., 2013), in the present study, we used the calculated sum of the six distress items. Items are rated on a 7‐point Likert scale from strongly disagree (1) to strongly agree (7), where higher scores indicate more VT. The VTS has strong psychometric properties (Aparicio et al., 2013; Benuto et al., 2018; Michalopoulos & Aparicio, 2012), with alpha .77 for the six‐item version (Aparicio et al., 2013). Cronbach's α for the total scale in the current sample was .81, and for the six‐item subjective VTS, it was .83.

2.3. Data analysis

2.3.1. Machine learning regression decision tree analyses

The primary aim of the analyses was to develop a prediction model of the therapists' acceptance of the TPT format and their attitudes, concerns and intention for future use of videoconferencing. The three dependent variables were UTAUT‐T Attitudes subscale, UTAUT‐T Concerns subscale and UTAUT‐T Behavioural Intention subscale. Based on the aforementioned clinical theory, we entered the following potential independent variables in the models: therapists' age and gender, as well as reported mean ratings of the real relationship, working alliance, professional self‐doubt and vicarious trauma scale. Based on general recommendation for machine learning models (Vrigazova, 2021) and common practice (e.g. Prout et al., 2020), the dataset was randomly split into two sets using PROC SURVEYSELECT of SAS: a training set (70%, n = 556) and a test set (30%, n = 239).

For this purpose, regression decision tree analyses were run with the R party package (Hothorn et al., 2006) using Monte Carlo simulation for multiple‐testing adjustment (Strasser & Weber, 1999). The workings of the decision tree algorithm can be described as follows. First, the null hypothesis of independence between each of the six independent (i.e. input) variables and the respective response dependent (i.e. response) variable was tested. If the independent variable had no association to the dependent variable, this variable was not included in the model. A random selection of independent variables was performed based on the strength of their association with the dependent variable; the first selected independent variable had the strongest association to the dependent variable. This association is estimated by a p‐value corresponding to a test for the partial null hypothesis of a single independent variable and the dependent variable. Subsequently, the algorithm implemented a binary split in the selected independent variable, based on the corresponding p‐value of less than .01. These steps were repeated recursively for each of the independent variables that were significantly associated with the dependent variable, until no remaining variable was associated with the dependent variable. As is common and recommended in prediction models (Loh, 2008), performance of decision trees was evaluated by the root mean square error (RMSE; Kenney & Keeping, 1962) and R 2 (Glantz & Slinker, 2001) of the test set compared to these values in the training set. 1 The R Foundation for Statistical Computing Version (3.6.1) was utilized for the tree analysis.

3. RESULTS

3.1. Therapists' acceptance of TPT technology during COVID‐19

3.1.1. Descriptives

For an overview of the descriptive statistics of all standardized measures, see Table 2. Overall, therapists reported somewhat positive attitudes towards TPT. On average, therapists were relatively concerned about providing therapy as well as undecided as to whether they would like to continue using TPT in the future (i.e. expressed as a neutral response on the UTUAT Behaviour Intention subscale), with large differences among therapists. The real relationship scores were similar or higher compared to scores found in previous studies in in‐person treatments before the pandemic (Bhatia & Gelso, 2018; Gelso et al., 2012). The real relationship has not been previously assessed with regard to online therapies. The working alliance scores on the WAI‐SRT in our sample were similar to therapist‐rated alliance scores on this same measure in previous studies in blended (combined in‐person and online) therapy that was conducted before the abrupt transition to online therapies due to the pandemic (Vernmark et al., 2019).

TABLE 2.

Descriptive statistics for the standardized independent variables and each of the dependent variables in comparison to previously published scores on the same measures (n = 795)

Measure M (SD) Min‐max Published comparison in in‐person therapies Two samples t‐tests a
1. UTAUT‐T Attitude 3.29 (.81) 1.00–5.00 N/A ‐
2. UTAUT‐T Anxiety 2.67 (.81) 1.00–4.67 N/A ‐
3. UTAUT‐T Behavioural Intention 3.17 (1.18) 1.00–5.00 N/A ‐
4. RRI 3.80 (.47) 1.92–5.00

Therapists:

M = 3.81; SD = 1.01

(Bhatia & Gelso, 2018)

 t(1042) = .77, p =. 83

Therapists:

M = 2.94, SD = .12

(Gelso et al., 2012)

 t(812) = 7.96, p < .001
5. WAI‐SRT 3.89 (.63) 1.00–5.00

Therapists in outpatient blended therapy

M = 3.93; SD = .63

(Vernmark et al., 2019)

 t(805) = 0.52, p = .83
6. PSD 2.42 (.84) 1.00–5.78

Trainees:

M = 1.52, SD = .94

(Odyniec et al., 2019)

 t(833) = 28.03, p < .001

Therapists:

M = 1.24, SD = .70

(Nissen‐Lie et al., 2013).

 t(863) = 11.41, p < .001
7. VTS 3.69 (1.17) 1.00–7.00

Social workers:

M = 2.88 SD = 0.99

(Aparicio et al., 2013)

 t(950) = 8.12, p < .001
Open in a new tab

Abbreviations: PSD, Professional Self‐Doubt; RRI, Real Relationship Inventory; UTAUT‐T, Unified Theory of Acceptance and Use of Technology–Therapist Scale; VTS, Vicarious Trauma Scale; WAI‐SRT, Working Alliance Inventory – Short Revised – Therapist.

a

Comparison with previously published scores on the same scales.

On average, therapists experienced professional self‐doubt sometimes or frequently in TPN during the pandemic, which is higher than the level of self‐doubt experienced by therapists in a prior naturalistic study of PSD (Nissen‐Lie et al., 2013) but still on the lower end of the 5‐point Likert scale. The sample in our study reported significantly higher subjective VT compared to previously found pre‐pandemic in‐person sessions (Michalopoulos & Aparicio, 2012).

3.1.2. Preliminary analyses

The UTAUT Attitude Scale significantly correlated with all independent variables, except for age and gender. The UTAUT Anxiety Scale significantly correlated with all independent variables. The UTAUT Behavioural Intention Scale correlated with UTAUT Attitude and Anxiety Scales, working alliance and professional self‐doubt, but correlations with age, gender, real relationship and vicarious trauma were non‐significant. After applying a Bonferroni correction (i.e. p < .005), the correlation between UTAUT Anxiety and gender was no longer significant. All the other correlations remained significant (when applying p < .005; see Table 3).

TABLE 3.

Pearson correlations for the eight variables (n = 795)

Measure 1 2 3 4 5 6 7 8 9
1. Age ‐‐
2. Gender .19*** ‐‐
3. UTAUT‐T Attitude .05 −.01 ‐‐
4. UTAUT‐T Anxiety −.28*** −.08* −.53*** ‐‐
5. UTAUT‐T Behavioural Intention .03 .04 .60*** −.36*** ‐‐
6. Real relationship .01 −.40** .22*** −.20*** .07 ‐‐
7. Working alliance inventory .29** −.04 .28*** −.18*** .10** .55*** ‐‐
8. Professional self‐doubt −.35*** −.12*** −.23*** .53*** −.11*** −.24*** −.25*** ‐‐
9. Vicarious trauma −.25*** −.12*** −.13*** .33*** −.01 −.08* −.09** .47*** ‐‐
Open in a new tab

Abbreviation: UTAUT‐T, Unified Theory of Acceptance and Use of Technology Therapist Version.

*

p < .05.

**

p < .01.

***

p < .001.

3.2. Prediction of therapists' acceptance of TPT technology during COVID

Predicting models were trained on the train set (n = 556) and were assessed for predictive performance on the test set (n = 239). Three separate decision tree models were developed to predict the three aspects of acceptance of TPT technology.

3.2.1. Predicting therapists' attitudes towards TPT

The regression tree model showed that higher alliance ratings (WAI > 3.9) was associated with more positive attitudes. Those with the higher alliance ratings (WAI > 3.9) with relatively low professional self‐doubt (PD < =2.1) and younger than 40 reported the most positive attitudes. Therapists who reported the lowest alliance rates (WAI = <3) also reported the most negative attitudes (Figure 1).

FIGURE 1.

FIGURE 1

Open in a new tab

Regression tree for attitudes towards telepsychotherapy technology (UTAUT‐T Attitudes), n = 556

3.2.2. Predicting therapists' concerns about TPT

The regression tree model showed that therapists with the highest levels of professional self‐doubt (PSD > 2.89) reported the most concern on UTAUT Anxiety. Those with relatively low levels of professional self‐doubt (PSD < 1.67) and low levels of vicarious trauma (VTS < 1.17) reported the least concern about using the online platform (Figure 2).

FIGURE 2.

FIGURE 2

Open in a new tab

Regression tree for concerns about using telepsychotherapy technology (UTAUT‐T Anxiety), n = 556

3.2.3. Predicting therapists' intention to use TPT in the future

The regression tree model showed that therapists who reported higher working alliance (WAI > 3) also reported more intention to use TPT in the future (Figure 3).

FIGURE 3.

FIGURE 3

Open in a new tab

Regression tree for intention to use telepsychotherapy technology in the future (UTAUT‐T Behavioural Intention), n = 556

3.3. Predictive performance

The performance of the model of prediction for UTAUT‐T Attitudes was relatively good in that it had the second best R 2 in the models (0.15) and the best RMSE values (0.51), the test set's RMSE value was 0.48 and the R 2 was 0.46. Prediction of UTAUT‐T Anxiety had the highest R2 in the models (0.54) and the second best RMSE value (0.57), and the test set's predictive criteria values were RMSE = 0.63 and R 2 = 0.46. Prediction of UTAUT‐T Behavioural Intention had the lowest R 2 (0.04) in the models and the highest RMSE values (1.16), and the test set had poorer predictive criteria values that were the same as the train set correspondence values. Taken together, the predictive criteria results of the test set were the same as in the training set when predicting UTAUT‐T Behavioural Intention and UTAUT‐T Anxiety. The ability to predict UTAUT‐T Attitudes was marginally poorer in the test than in the training set. The relatively similar model performances in the training and test set indicate that the model prediction abilities remained when applied to new data. Table 4 summarized the predictive criteria values on the three outcomes. In addition, we tested a dummy model and found higher RMSE for all three subscales (Attitudes RMSE = .51, Anxiety RMSE = .85 and Behavioural Intention RMSE = 1.18) compared to the test set, which indicates that the reported model has better predictive power than the dummy model.

TABLE 4.

Predictive criteria values for training and test sets

Dependent variable Train n = 556 Test n = 239
RMSE R 2 RMSE R 2
UTAUT‐T Attitudes 0.51 0.15 0.48 0.13
UTAUT‐T Anxiety 0.68 0.34 0.69 0.35
UTAUT‐T Behavioural Intention 1.16 0.04 1.16 0.03
Open in a new tab

Abbreviations: RMSE, root mean square error; UTAUT‐T, Unified Theory of Acceptance and Use of Technology Therapist Version.

4. DISCUSSION

In the present study, we examined different aspects of therapists' acceptance of TPT technology (i.e. therapists' attitudes, concerns and intention for future utilization of TPT) as well as the therapists' reported therapeutic relationship with their online patients, their levels of professional self‐doubt and vicarious trauma, as experienced in online sessions during the COVID‐19 pandemic. We explored how these TPT experiences (the therapist‐reported therapeutic relationship with their online patients, their levels of professional self‐doubt and vicarious trauma) might relate to the different aspects of therapists' acceptance of TPT technology. In order to develop and cross‐validate predictive models, we divided our collected survey responses from 795 therapists into a larger training‐set subsample and a smaller test‐set subsample. We then used a machine learning decision tree approach to build predictive models in the training set, before cross‐validating the performance of these models on the test set.

Our survey responses from 795 therapists indicated that therapists had neutral or moderately positive attitudes while also many concerns regarding TPT, and on average, they were undecided about whether or not they would use TPT in the future. Therapists experienced the therapeutic relationship with their patients in their online sessions positively: They reported similar levels of working alliance and higher levels of real relationship in their online sessions during the pandemic compared to in‐person levels in previous studies conducted pre‐pandemic (e.g. blended therapy, i.e. online modules combined with therapist sessions; Vernmark et al., 2019 and in‐person therapy; Bhatia & Gelso, 2018, respectively). Therapists experienced higher levels of professional self‐doubt and higher levels of vicarious trauma than was reported in previous studies on in‐person therapies (Aparicio et al., 2012).

When we tested three aspects of the acceptance of TPT technology (attitudes, concerns, future intention to use), the first model showed that therapist‐reported working alliance was most pertinent to be associated with attitudes towards TPT, in that sufficiently high alliance scores were associated with more positive attitudes, and most positive attitudes were reported by those who also experienced little professional doubt and were 40 years old or younger. Therapists who reported the weakest quality of the working alliance also reported the most negative attitudes towards TPT technology. Therapists' concerns about TPT technology was most pertinently associated with the therapist's professional self‐doubt, in that therapists with very little professional doubt (i.e. the most professionally confident therapists) had the least concerns about TPT, and concerns were the lowest for those who also reported low levels of vicarious trauma during the start of the pandemic. The prediction model for the intention to use TPT technology showed that the only pertinent predictor was working alliance. The majority of therapists who reported at least a medium level of working alliance were inclined to use TPT in the future, whereas a small subgroup of therapists who reported weak alliance also reported the least likelihood to use TPT in the future.

Unlike previous research on in‐person therapies pre‐pandemic that suggested that besides the working alliance, the quality of the real relationship is also important in predicting treatment efficacy (Bhatia & Gelso, 2018), the real relationship did not appear to play an important role in therapists' acceptance of TPT technology. In none of the three models, their reported level of the real relationship (RR) with their online patients was associated with any of the three dependent variables, once the independent variables that were more strongly associated with each of the dependent variables were included in the models, and therefore did not show in the respective decision trees. Similarly, gender of the therapist was not relevant in predicting any of the aspects of the therapists' acceptance of TPT technology.

Overall, these results indicate that therapists' professional self‐doubt and the working alliance that they perceive with their online patients are the most prominent factors in predicting their reported acceptance of TPT. Our study findings partly confirmed previous results that therapists are often reluctant to use TPT due to fears of being unable to build a strong therapy relationship with their patients online (Roesler, 2017). In line with previous studies, based on patient self‐reported alliance measures, that reported that the working alliance in online therapies may be just as strong as in in‐person therapies (Norwood et al., 2018), we found similar ratings of the online working alliance in our participating therapists.

Our results also highlight that concerns about the use of TPT is not pertinent to the therapists' ability to build a strong working alliance online, but rather to their professional self‐doubt and experience of vicarious traumatization in their online work. This might imply that therapists can benefit from their own professional support in order to reduce their professional anxiety during the pandemic and more generally. Moreover, although professional self‐doubt was linked to better treatment outcomes in a previous study of trainees (Nissen‐Lie et al., 2013), our study showed that the experience of more professional self‐doubt was negatively related to therapists' attitudes and positively related to concerns about TPT technology. Thus, professional self‐doubt is an important area to address in practitioners in order to promote the acceptance of TPT as a potential treatment format. Future research is needed to assess whether professional self‐doubt relates specifically to concerns about TPT or to providing therapy in general, including in‐person settings. Future research might also be warranted in examining how best to target and reduce therapists' sense of professional competence and confidence when applying new therapy technology.

Finally, vicarious trauma was only associated with concerns about TPT; it was unrelated to attitudes towards or to future intention to use TPT. And even within the model predicting concerns about TPT, it was not as pertinent as the therapists' experienced professional self‐doubt. Earlier research has shown that vicarious trauma has a negative impact on treatment effectiveness (Sexton, 1999), and this impact is especially strong when therapists and patients experience a disaster simultaneously (Boscarino et al., 2004; Culver et al., 2011). In our study, during the pandemic, providing treatment remotely to patients dealing with traumatic stress represents a serious challenge. Before the pandemic, therapists considered severe psychopathology and acute crisis situations as counter‐indicators of online treatment (Connolly et al., 2020; Simms et al., 2011); however, during the pandemic, there was no alternative to TPT. Our findings about heightened concerns regarding TPT in those with more subjective vicarious trauma experiences possibly reflect the therapists' general sense of anxiety during the pandemic or the impact of therapists' subjective experiences of stress and overwhelm on their ability to be open‐minded to new technology/ways of working. Also, due to the pandemic, therapists were probably more frequently exposed to highly stressful material in their sessions, and given that the stress of the pandemic impacts the therapists as well, they might have less emotional capacity to deal with their patients' trauma presentations. The heightened vicarious trauma experience in itself is an alarming fact, and its impact on concerns regarding TPT need to also be taken seriously when considering ways of using TPT with patients under high stress.

Our results show that despite concerns described in previous literature about relational aspects of TPT (Roesler, 2017; Sucala et al., 2013), the actual experience of working alliance and real relationship in online sessions was relatively positive. In fact, the quality of the real relationship was higher than in previous reports in in‐person therapies (Bhatia & Gelso, 2018; Gelso et al., 2012), even though due to the different study designs, these comparisons should be taken with some caution. Given that the quality of the therapeutic relationship is known to predict positive treatment outcomes, this finding provides some optimism regarding the potential impact of ongoing TPT. However, our study also indicated that therapists' relational challenges are a central factor in determining their attitudes and future intention to utilize TPT. Difficulties with building strong relationships with patients may not only decrease treatment efficacy but also hinder the utilization of TPT in the future. Future research on how therapists' can be helped to build a sufficient working alliance with patients, especially within an TPT format could be important to ensure therapists will feel comfortable enough offering the option of TPT to their patients.

The moderate levels of professional self‐doubt found in this study are understandable in light of the sudden transition to a TPT format and related practical and professional challenges, as well as the general stress of the pandemic on both therapists and their patients, that may have triggered doubts about professional skills and abilities. However, the experiences during the pandemic may have long‐term consequences on the continued use of the TPT format.

4.1. Clinical implications

Our findings might give an indication of which therapists might need more support if required to deliver therapy online. There is a need for training and support on (1) reducing and tolerating professional self‐doubt related to TPT technology; (2) further developing skills to strengthen the relationship in TPT sessions; and (3) identifying therapists who are experiencing vicarious trauma and providing additional support and care.

Providing therapy remotely has its own unique challenges, including the lack of certain aspects of non‐verbal communication, which may hinder understanding of the patients' emotional state or the therapist's ability to sufficiently transmit their own emotions and empathy (Wagnild et al., 2006). Training courses could provide tools to compensate for these aspects of TPT and help therapists to improve their online communication skills to enhance the therapeutic relationship online, and maybe most importantly, their sense of professional competence and confidence. Professional training and peer support groups are also valuable resources for ameliorating the subjective sense of vicarious traumatization while working during a global pandemic that impacts both patient and therapist.

In many countries, therapy is delivered as part of government funded services, so decisions about mode of delivery are likely to be out of clinicians' hands. However, the increased clinical and research interest in TPT might mean that TPT will be more readily accepted by patients, therapists and payers. Given that more positive attitudes towards a treatment process tend to relate to better treatment outcomes, this bodes well for patients' well‐being. Although previously clinicians were required to provide in‐person therapy to ensure governmental funding, in the near future, clinicians might be able to make a clinical decision to determine which therapeutic format might be most beneficial to a patient at a given time. Of course, the use of in‐person therapy or TPT does not have to be a binary clinical decision. Therapists and patients might consider integrating TPT sessions within an in‐person treatment or do the full treatment via TPT technology (Van Daele et al., 2021). Patients may receive a combination of in‐person and remote sessions based on changing needs over the course of therapy (Yellowlees & Nafiz, 2010).

4.2. Limitations

This study had several limitations. First, even though in our study we built the prediction models based on the well‐supported assumption that attitudes towards TPT may be predicted by various factors (Connolly et al., 2020; Venkatesh et al., 2016), including the relationship, professional doubt and vicarious trauma variables, a reverse causation may also be argued for. For example, it is possible that preconceived attitudes towards TPT may prevent therapists from trying to build strong therapeutic relationships in their online sessions.

Second, even though the comparison between our sample and samples reported in previously published studies may be informative, these comparisons need to be interpreted with caution due to various confounders (e.g. different therapist and patient and treatment characteristics in these studies).

Third, we used a convenience sample of therapists willing to report about their experiences during the pandemic in an online survey. Although our sample was relatively diverse, most respondents were female (as is the case in many survey studies; Moore & Tarnai, 2002). At the same time, this is representative of the larger population of psychologists (American Psychological Association, 2015). Therapists who volunteered to participate might also have been especially interested in TPT, which possibly biased the results.

Fourth, a well‐known limitation, common to all survey research, is that all variables were self‐reported responses, which means that the relationship between these variables might have been spuriously inflated. Future research could benefit from using complementary methods to gain information about therapists' acceptance of TPT technology, for example, by having observer codings of behaviour of therapists or by interviewing supervisors and patients to gain their perspective on the therapist's attitudes.

Finally, although the present study reports on a rigorous cross‐validation method for producing results that are likely to be generalizable to the broader population, this must be viewed as internal validation, rather than external validation, as the test set is still a subsample of the original dataset, rather than an entirely new or separate sample (Aafjes‐van Doorn et al., 2021). Thus, it is possible that the identified independent variables may not generalize to new samples of other therapists and that an additional out‐of‐sample external validation would further strengthen the generalizability of our findings (Sammut & Webb, 2017).

4.3. Conclusion

This large survey study of therapists' acceptance of TPT technology during the first weeks of COVID‐19 indicates that therapists had relatively positive attitudes towards TPT technology and some concerns about using TPT in the future. The most consistent factor influencing therapists' acceptance levels was their professional self‐doubt as well as their perceived quality of the online working alliance. This implies that professional support and training might be helpful in increasing therapists' sense of professional and relational competence and confidence online, and therefore generating a more positive attitude, fewer concerns and a higher likelihood of continued use of TPT technology in the future, when it is no longer a health requirement.

CONFLICT OF INTEREST

The authors have no conflicts of interest to disclose.

ACKNOWLEDGEMENTS

The data collected in China were supported by a research grant by the China American Psychoanalytic Alliance (VB, KAVD). We thank Lauren Smith (Kenyon College) for her contributions to updating references and creating the figures in this manuscript.

Békés, V. , Aafjes‐van Doorn, K. , Zilcha‐Mano, S. , Prout, T. , & Hoffman, L. (2021). Psychotherapists' acceptance of telepsychotherapy during the COVID‐19 pandemic: A machine learning approach. Clinical Psychology & Psychotherapy, 28(6), 1403–1415. 10.1002/cpp.2682

Vera Békés and Katie Aafjes‐van Doorn shared first authorship.

Funding information China American Psychoanalytic Alliance

ENDNOTE

1

For RMSE, a lower value suggests a model with a better fit. For R2, a higher value suggests a model with a better fit.

DATA AVAILABILITY STATEMENT

The dataset described in this article is not readily available because the dataset is not approved by the IRB for use by other researchers. Requests to access the datasets should be directed to vera.bekes@yu.edu.

REFERENCES

  1. Aafjes‐van Doorn, K. , Békés, V. , & Prout, T. A. (2020). Grappling with our therapeutic relationship and professional self‐doubt during COVID‐19: Will we use video therapy again? Counselling Psychology QuarterlyAdvance online publication. 10.1080/09515070.2020.1773404 [DOI] [Google Scholar]
  2. Aafjes‐van Doorn, K. , Bekés, V. , Prout, T. A. , & Hoffman, L. (2020). Psychotherapists' vicarious traumatization during the COVID‐19 pandemic. Psychological Trauma Theory Research Practice and Policy, 12, S148–S150. 10.1037/tra0000868 [DOI] [PubMed] [Google Scholar]
  3. Aafjes‐van Doorn, K. , Kamsteeg, C. , Bate, J. , & Aafjes, M. (2021). A scoping review of machine learning in psychotherapy research. Psychotherapy Research, 31(1), 92–116. [DOI] [PubMed] [Google Scholar]
  4. Alapetite, A. , Andersen, H. B. , & Hertzum, M. (2009). Acceptance of speech recognition by physicians: A survey of expectations, experiences, and social influence. International Journal of Human‐Computer Studies, 67(1), 36–49. 10.1016/j.ijhcs.2008.08.004 [DOI] [Google Scholar]
  5. American Psychological Association . (2015). 2015 APA Survey of Psychology Health Service Providers. Retrieved from https://www.apa.org/workforce/publications/15-health-service-providers
  6. American Psychological Association . (2018). 2007‐16: Demographics of the U.S. Psychology Workforce. https://www.apa.org/workforce/publications/16-demographics
  7. Aparicio, E. , Michalopoulos, L. M. , & Unick, G. J. (2013). An examination of the psychometric properties of the vicarious trauma scale in a sample of licensed social workers. Health & Social Work, 38(4), 199–206. 10.1093/hsw/hlt017 [DOI] [PubMed] [Google Scholar]
  8. Békés, V. , & Aafjes‐van Doorn, K. (2020). Psychotherapists' attitudes toward online therapy during the COVID‐19 pandemic. Journal of Psychotherapy Integration, 30(2), 238–247. 10.1037/int0000214 [DOI] [Google Scholar]
  9. Békés, V. , Aafjes‐van Doorn, K. , McCollum, J. J. , Prout, T. R. , & Hoffman, L. (2021). The development of a self‐report scale to assess therapists' acceptance of online psychotherapy via video conferencing. PsyArXiv. doi: 10.31234/osf.io/24w8h [DOI]
  10. Békés, V. , Aafjes‐van Doorn, K. , Prout, T. A. , & Hoffman, L. (2020). Stretching the analytic frame: Analytic therapists' experiences with remote therapy during COVID‐19. Journal of the American Psychoanalytic AssociationAdvance online publication. 10.1080/09515070.2020.1773404 [DOI] [PubMed] [Google Scholar]
  11. Békés, V. , Grondin, F. , & Bouchard, S. (2020). Barriers and facilitators to the integration of web‐based interventions into routine care. Clinical Psychology: Science and Practice, 27. 10.1111/cpsp.12335 [DOI] [Google Scholar]
  12. Benuto, L. , Singer, J. , Cummings, C. , & Ahrendt, A. (2018). The vicarious trauma scale: Confirmatory factor analysis and psychometric properties with a sample of victim advocates. Health & Social Care in the Community, 26(4), 564–571. 10.1111/hsc.12554 [DOI] [PubMed] [Google Scholar]
  13. Bhatia, A. , & Gelso, C. J. (2018). Therapists’ perspective on the therapeutic relationship: Examining a tripartite model. Counselling Psychology Quarterly, 31(3), 271–293. [Google Scholar]
  14. Bi, Q. , Goodman, K. E. , Kaminsky, J. , & Lessler, J. (2019). What is machine learning? A primer for the epidemiologist. American Journal of Epidemiology, 188(12), 2222–2239. 10.1093/aje/kwz189 [DOI] [PubMed] [Google Scholar]
  15. Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research & Practice, 16(3), 252–260. 10.1037/h0085885 [DOI] [Google Scholar]
  16. Boscarino, J. A. , Figley, C. R. , & Adams, R. E. (2004). Compassion fatigue following the September 11 terrorist attacks: A study of secondary trauma among New York City social workers. International Journal of Emergency Mental Health, 6(2), 57–66. [PMC free article] [PubMed] [Google Scholar]
  17. Bouchard, S. , Allard, M. , Robillard, G. , Dumoulin, S. , Guitard, T. , Loranger, C. , Green‐Demers, I. , Marchand, A. , Renaud, P. , Cournoyer, L. G. , & Corno, G. (2020). Videoconferencing psychotherapy for panic disorder and agoraphobia: Outcome and treatment processes from a non‐randomized non‐inferiority trial. Frontiers in Psychology, 11, 2164. 10.3389/fpsyg.2020.02164 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Cenfetelli, R. T. , & Schwarz, A. (2011). Identifying and testing the inhibitors of technology usage intentions. Information Systems Research, 22(4), 808–823. 10.1287/isre.1100.0295 [DOI] [Google Scholar]
  19. Chang, I. C. , Hwang, H. G. , Hung, W. F. , & Li, Y. C. (2007). Physicians' acceptance of pharmacokinetics based clinical decision support systems. Expert Systems with Applications, 33(2), 296–303. 10.1016/j.eswa.2006.05.001 [DOI] [Google Scholar]
  20. Compeau, D. , Higgins, C. , & Huff, S. (1999). Social cognitive theory and individual reactions to computing technology: A longitudinal study. MIS Quarterly, 23(2), 145–158. 10.2307/249749 [DOI] [Google Scholar]
  21. Connolly, S. L. , Miller, C. J. , Lindsay, J. A. , & Bauer, M. S. (2020). A systematic review of providers' attitudes toward telemental health via videoconferencing. Clinical Psychology: Science and Practice, 27, e12311. 10.1111/cpsp.12311 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Culver, L. M. , McKinney, B. L. , & Paradise, L. V. (2011). Mental health professionals' experiences of vicarious traumatization in post‐hurricane Katrina New Orleans. Journal of Loss and Trauma, 16(1), 33–42. 10.1080/15325024.2010.519279 [DOI] [Google Scholar]
  23. Essig, T. , & Russell, G. I. (2017). A note from the guest editors. Psychoanalytic Perspectives, 4(2), 131–137. 10.1080/1551806X.2017.1304111 [DOI] [Google Scholar]
  24. Feijt, M. , de Kort, Y. , Bongers, I. , Bierbooms, J. , Westerink, J. , & IJsselsteijn, W. (2020). Mental health care goes online: Practitioners' experiences of providing mental health care during the COVID‐19 pandemic. Cyberpsychology, Behavior and Social Networking, 23, 860–864. 10.1089/cyber.2020.0370 [DOI] [PubMed] [Google Scholar]
  25. Fuertes, J. N. , Gelso, C. J. , Owen, J. J. , & Cheng, D. (2013). Real relationship, working alliance, transference/countertransference and outcome in time‐limited counseling and psychotherapy. Counselling Psychology Quarterly, 26(3–4), 294–312. 10.1080/09515070.2013.845548 [DOI] [Google Scholar]
  26. Fuertes, J. N. , Moore, M. , & Ganley, J. (2019). Therapists' and clients' ratings of real relationship, attachment, therapist self‐disclosure, and treatment progress. Psychotherapy Research, 29(5), 594–606. 10.1080/10503307.2018.1425929 [DOI] [PubMed] [Google Scholar]
  27. Gelso, C. J. , Kelley, F. A. , Fuertes, J. N. , Marmarosh, C. , Holmes, S. E. , Costa, C. , & Hancock, G. R. (2005). Measuring the real relationship in psychotherapy: Initial validation of the therapist form. Journal of Counseling Psychology, 52(4), 640–649. 10.1037/0022-0167.52.4.640 [DOI] [Google Scholar]
  28. Gelso, C. J. , Kivlighan, D. M. , Busa‐Knepp, J. , Spiegel, E. B. , Ain, S. , Hummel, A. M. , Ma, Y. E. , & Markin, R. D. (2012). The unfolding of the real relationship and the outcome of brief psychotherapy. Journal of Counseling Psychology, 59(4), 495–506. 10.1037/a0029838 [DOI] [PubMed] [Google Scholar]
  29. Gelso, C. J. , Kivlighan, D. M. Jr. , & Markin, R. D. (2018). The real relationship and its role in psychotherapy outcome: A meta‐analysis. Psychotherapy, 55(4), 434–444. 10.1037/pst0000183 [DOI] [PubMed] [Google Scholar]
  30. Glantz, S. A. , & Slinker, B. K. (2001). Primer of applied regression & analysis of variance (2nd ed.). McGraw‐Hill, Inc. [Google Scholar]
  31. Glueckauf, R. L. , Maheu, M. M. , Drude, K. P. , Wells, B. A. , Wang, Y. , Gustafson, D. J. , & Nelson, E.‐L. (2018). Survey of psychologists' telebehavioral health practices: Technology use, ethical issues, and training needs. Professional Psychology: Research and Practice, 49(3), 205–219. 10.1037/pro0000188 [DOI] [Google Scholar]
  32. Gruzd, A. , Staves, K. , & Wilk, A. (2012). Connected scholars: Examining the role of social media in research practices of faculty using the UTAUT model. Computers in Human Behavior, 28(6), 2340–2350. 10.1016/j.chb.2012.07.004 [DOI] [Google Scholar]
  33. Hatcher, R. L. , & Gillaspy, J. A. (2006). Development and validation of a revised short version of the working alliance inventory. Psychotherapy Research, 16(1), 12–25. 10.1080/10503300500352500 [DOI] [Google Scholar]
  34. Horvath, A. , & Greenberg, L. (1989). Development and validation of the working alliance inventory. Journal of Counseling Psychology, 36(2), 223–233. 10.1037/0022-0167.36.2.223 [DOI] [Google Scholar]
  35. Hothorn, T. , Hornik, K. , & Zeileis, A. (2006). Unbiased recursive partitioning: A conditional inference framework. Journal of Computational and Graphical Statistics, 15(3), 651–674. 10.1198/106186006X133933 [DOI] [Google Scholar]
  36. Kelley, F. A. , Gelso, C. J. , Fuertes, J. N. , Marmarosh, C. , & Lanier, S. (2010). The real relationship inventory: Development and psychometric investigation of the client form. Psychotherapy, 47, 540–553. 10.1037/a0022082 [DOI] [PubMed] [Google Scholar]
  37. Kenney, J. F. , & Keeping, E. S. (1962). Root mean square. In Kenney J. F. & Keeping E. S. (Eds.), Mathematics of statistics, pt, 1 (pp. 59–60). Van Nostrand. [Google Scholar]
  38. Liu, L. , Miguel Cruz, A. , Rios Rincon, A. , Buttar, V. , Ranson, Q. , & Goertzen, D. (2015). What factors determine therapists' acceptance of new technologies for rehabilitation – A study using the unified theory of acceptance and use of technology (UTAUT). Disability and Rehabilitation, 37(5), 447–455. 10.3109/09638288.2014.923529 [DOI] [PubMed] [Google Scholar]
  39. Lo Coco, G. , Gullo, S. , Prestano, C. , & Gelso, C. J. (2011). Relation of the real relationship and the working alliance to the outcome of brief psychotherapy. Psychotherapy, 48(4), 359–367. 10.1037/a0022426 [DOI] [PubMed] [Google Scholar]
  40. Loh, W.‐Y. (2008). Classification and regression tree methods. In Ruggeri F., Kennet R. S., & Faltin F. W. (Eds.), Encyclopedia of statistics in quality and reliability (pp. 315–323). Wiley & Sons. [Google Scholar]
  41. Markowitz, J. C. , Milrod, B. , Heckman, T. G. , Bergman, M. , Amsalem, D. , Zalman, H. , Ballas, T. , & Neria, Y. (2020). Psychotherapy at a distance. American Journal of Psychiatry, 178, 240–246. 10.1176/appi.ajp.2020.20050557 [DOI] [PubMed] [Google Scholar]
  42. Marmarosh, C. , Gelso, C. , Markin, R. , Majors, R. , Mallery, C. , & Choi, J. (2009). The real relationship in psychotherapy: Relationships to adult attachments, working alliance, transference, and therapy outcome. Journal of Counseling Psychology, 56(3), 337–350. 10.1037/a0015169 [DOI] [Google Scholar]
  43. McBeath, A. G. , Du Plock, S. , & Bager‐Charleson, S. (2020). The challenges and experiences of psychotherapists working remotely during the coronavirus* pandemic. Counselling and Psychotherapy Research, 20(3), 394–405. 10.1002/capr.12326 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Michalopoulos, L. , & Aparicio, E. (2012). Vicarious trauma in social workers: The role of trauma history, social support, and years of experience. Journal of Aggression, Maltreatment & Trauma, 21(6), 646–664. 10.1080/10926771.2012.689422 [DOI] [Google Scholar]
  45. Moore, D. L. , & Tarnai, J. (2002). Evaluating nonresponse error in mail surveys. In Groves R. M., Dillman D. A., Eltinge J. L., & Little R. J. A. (Eds.), Survey nonresponse (pp. 197–211). John Wiley & Sons. [Google Scholar]
  46. Munder, T. , Wilmers, F. , Leonhart, R. , Linster, H. W. , & Barth, J. (2010). Working alliance inventory‐short revised (WAI‐SR): Psychometric properties in outpatients and inpatients. Clinical Psychology & Psychotherapy, 17(3), 231–239. 10.1002/cpp.658 [DOI] [PubMed] [Google Scholar]
  47. Nissen‐Lie, H. A. , Monsen, J. T. , Ulleberg, P. , & Rønnestad, M. H. (2013). Psychotherapists' self‐reports of their interpersonal functioning and difficulties in practice as predictors of patient outcome. Psychotherapy Research, 23(1), 86–104. [DOI] [PubMed] [Google Scholar]
  48. Nissen‐Lie, H. A. , Rønnestad, M. H. , Høglend, P. A. , Havik, O. E. , Solbakken, O. A. , Stiles, T. C. , & Monsen, J. T. (2017). Love yourself as a person, doubt yourself as a therapist? Clinical Psychology & Psychotherapy, 24(1), 48–60. 10.1002/cpp.1977 [DOI] [PubMed] [Google Scholar]
  49. Norwood, C. , Moghaddam, N. G. , Malins, S. , & Sabin‐Farrell, R. (2018). Working alliance and outcome effectiveness in videoconferencing psychotherapy: A systematic review and noninferiority meta‐analysis. Clinical Psychology & Psychotherapy, 25(6), 797–808. 10.1002/cpp.2315 [DOI] [PubMed] [Google Scholar]
  50. Odyniec, P. , Probst, T. , Margraf, J. , & Willutzki, U. (2019). Psychotherapist trainees’ professional self‐doubt and negative personal reaction: Changes during cognitive behavioral therapy and association with patient progress. Psychotherapy Research, 29(1), 123–138. [DOI] [PubMed] [Google Scholar]
  51. Orlinsky, D. , Ambühl, H. , Rønnestad, M. H. , Davis, J. , Gerin, P. , Davis, M. , Willutzki, U. , Botermans, J.‐F. , Dazord, A. , Cierpka, M. , Aapro, N. , Buchheim, P. , Bae, S. , Davidson, C. , Friis‐Jorgensen, E. , Joo, E. , Kalmykova, E. , Meyerberg, J. , Northcut, T. , … SPR Collaborative Research Network . (1999). Development of psychotherapists: Concepts, questions, and methods of a collaborative international study. Psychotherapy Research, 9(2), 127–153. 10.1093/ptr/9.2.127 [DOI] [Google Scholar]
  52. Poletti, B. , Tagini, S. , Brugnera, A. , Parolin, L. , Pievani, L. , Ferrucci, R. , & Silani, V. (2020). Telepsychotherapy: A leaflet for psychotherapists in the age of COVID‐19. A review of the evidence. Counselling Psychology Quarterly, 1–16. [Google Scholar]
  53. Prout, T. A. , Zilcha‐Mano, S. , Aafjes‐van Doorn, K. , Békés, V. , Christman‐Cohen, I. , Whistler, K. , Kui, T. , & Di Giuseppe, M. (2020). Identifying predictors of psychological distress during COVID‐19: A machine learning approach. Frontiers in Psychology, 11, 3063. 10.3389/fpsyg.2020.586202 [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Pynoo, B. , Devolder, P. , Tondeur, J. , van Braak, J. , Duyck, W. , & Duyck, P. (2011). Predicting secondary school teachers' acceptance and use of a digital learning environment: A cross‐sectional study. Computers in Human Behavior, 27(1), 568–575. 10.1016/j.chb.2010.10.005 [DOI] [Google Scholar]
  55. Rochlen, A. B. , Zack, J. S. , & Speyer, C. (2004). Online therapy: Review of relevant definitions, debates, and current empirical support. Journal of Clinical Psychology, 60(3), 269–283. 10.1002/jclp.10263 [DOI] [PubMed] [Google Scholar]
  56. Roesler, C. (2017). Tele‐analysis: The use of media technology in psychotherapy and its impact on the therapeutic relationship. Journal of Analytical Psychology, 62, 372–394. 10.1111/1468-5922.12317 [DOI] [PubMed] [Google Scholar]
  57. Rosen, C. S. , Glassman, L. H. , & Morland, L. A. (2020). Telepsychotherapy during a pandemic: A traumatic stress perspective. Journal of Psychotherapy Integration, 30(2), 174–187. [Google Scholar]
  58. Sammut, C. , & Webb, G. I. (Eds.) (2017). Encyclopedia of machine learning and data mining (2nd ed.). Springer Science+Business Media. [Google Scholar]
  59. Sexton, L. (1999). Vicarious traumatisation of counsellors and effects on their workplaces. British Journal of Guidance and Counselling, 27(3), 393–403. 10.1080/03069889908256279 [DOI] [Google Scholar]
  60. Simms, D. C. , Gibson, K. , & O'Donnell, S. (2011). To use or not to use: Clinicians' perceptions of telemental health. Canadian Psychology/Psychologie Canadienne, 52(1), 41–51. 10.1037/a0022275 [DOI] [Google Scholar]
  61. Simpson, S. (2009). Psychotherapy via videoconferencing: A review. British Journal of Guidance & Counselling, 37(3), 271–286. [Google Scholar]
  62. Strasser, H. , & Weber, C. (1999). On the asymptotic theory of permutation statistics. Mathematical Methods of Statistics, 8(2), 220–250. [Google Scholar]
  63. Sucala, M. , Schnur, J. B. , Brackman, E. H. , Constantino, M. J. , & Montgomery, G. H. (2013). Clinicians' attitudes toward therapeutic alliance in E‐therapy. The Journal of General Psychology, 140(4), 282–293. 10.1080/00221309.2013.830590 [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Van Daele, T. , Best, P. , Bernaerts, S. , Van Assche, E. , & de Witte, N. A. (2021). Dropping the E: The potential for integrating e‐mental health in psychotherapy. Current Opinion in Psychology, 41, 46–50. 10.1016/j.copsyc.2021.02.007 [DOI] [PubMed] [Google Scholar]
  65. Van Daele, T. , Karekla, M. , Kassianos, A. P. , Compare, A. , Haddouk, L. , Salgado, J. , Ebert, D. D. , Trebbi, G. , Bernaerts, S. , Van Assche, E. , & de Witte, N. A. (2020). Recommendations for policy and practice of telepsychotherapy and e‐mental health in Europe and beyond. Journal of Psychotherapy Integration, 30(2), 160–173. [Google Scholar]
  66. Venkatesh, V. , Morris, M. , Davis, G. , & Davis, F. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. 10.2307/30036540 [DOI] [Google Scholar]
  67. Venkatesh, V. , Thong, J. , & Xu, X. (2012). Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology. MIS Quarterly, 36(1), 157–178. 10.2307/41410412 [DOI] [Google Scholar]
  68. Venkatesh, V. , Thong, J. Y. , & Xu, X. (2016). Unified theory of acceptance and use of technology: A synthesis and the road ahead. Journal of the Association for Information Systems, 17(5), 328–376. 10.17705/1jais.00428 [DOI] [Google Scholar]
  69. Vernmark, K. , Hesser, H. , Topooco, N. , Berger, T. , Riper, H. , Luuk, L. , Backlund, L. , Carlbring, P. , & Andersson, G. (2019). Working alliance as an independent variable of change in depression during blended cognitive behaviour therapy. Cognitive Behaviour Therapy, 48(4), 285–299. 10.1080/16506073.2018.1533577 [DOI] [PubMed] [Google Scholar]
  70. Vrigazova, B. (2021). The proportion for splitting data into training and test set for the bootstrap in classification problems. Business Systems Research: International Journal of the Society for Advancing Innovation and Research in Economy, 12(1), 228–242. 10.2478/bsrj-2021-0015 [DOI] [Google Scholar]
  71. Vrklevski, L. P. , & Franklin, J. (2008). Vicarious trauma: The impact on solicitors of exposure to traumatic material. Traumatology, 14(1), 106–118. 10.1177/1534765607309961 [DOI] [Google Scholar]
  72. Wagnild, G. , Leenknecht, C. , & Zauher, J. (2006). Psychiatrists' satisfaction with telepsychiatry. Telemedicine Journal & e‐Health, 12(5), 546–551. 10.1089/tmj.2006.12.546 [DOI] [PubMed] [Google Scholar]
  73. Watts, S. , Marchand, A. , Bouchard, S. , Gosselin, P. , Langlois, F. , Belleville, G. , & Dugas, M. J. (2020). Telepsychotherapy for generalized anxiety disorder: Impact on the working alliance. Journal of Psychotherapy Integration, 30(2), 208–225. 10.1037/int0000223 [DOI] [Google Scholar]
  74. Yellowlees, P. , & Nafiz, N. (2010). The psychiatrist‐patient relationship of the future: Anytime, anywhere? Harvard Review of Psychiatry, 18(2), 96–102. 10.3109/10673221003683952 [DOI] [PubMed] [Google Scholar]
  75. Zilcha‐Mano, S. (2017). Is the alliance really therapeutic? Revisiting this question in light of recent methodological advances. American Psychologist, 72(4), 311–325. 10.1037/a0040435 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

The dataset described in this article is not readily available because the dataset is not approved by the IRB for use by other researchers. Requests to access the datasets should be directed to vera.bekes@yu.edu.

Articles from Clinical Psychology & Psychotherapy are provided here courtesy of Wiley

RESOURCES