Skip to main content
NCBI home page
Wiley - PMC COVID-19 Collection logoLink to Wiley - PMC COVID-19 Collection
. 2022 Feb 2;29(4):1342–1354. doi: 10.1002/cpp.2713

Impact of COVID‐19 social distancing measures on routine mental health care provision and treatment outcome for common mental disorders in the Netherlands

Edwin de Beurs 1,2,✉, Matthijs Blankers 1,3,4, Jaap Peen 1, Clara Rademacher 2, Alicja Podgorski 2, Jack Dekker 1,5
PMCID: PMC9015637  PMID: 35068003

Abstract

Objective

The uptake of digital interventions in mental health care (MHC) has been slow, as many therapists and patients believe that in‐person contact is essential for establishing a good working relationship and good outcomes in treatment. The public health policies regarding social distancing during the coronavirus disease‐2019 (COVID‐19) pandemic forced an abrupt transformation of MHC provisions for outpatients: Since mid‐March 2020, nearly all in‐person contact was replaced with videoconferencing. The COVID‐19 crisis offered a unique opportunity to investigate whether MHC with videoconferencing yields inferior results as compared to in‐person interventions.

Method

In a large urban MHC facility in the Netherlands, measurement‐based care is routine practice. Outcome data are regularly collected to support shared decision making and monitor patient progress. For this study, pretest and post‐test data were used to compare outcomes for three cohorts: treatments performed prior to, partially during and entirely during the COVID‐19 lockdown. Outcomes were compared in two large data sets: Basic MHC (N = 1392) and Specialized MHC (N = 1040).

Results

Therapeutic outcomes appeared robust for COVID‐19 conditions across the three cohorts: No differences in outcomes were found between treatments that were conducted during lockdown compared to in‐person treatments prior to COVID‐19, or treatments which started in‐person, but needed to be continued by means of videoconferencing.

Discussion

Videoconferencing care during the COVID‐19 pandemic had similar outcomes compared to traditional in‐person care. These real‐world results corroborate findings of previous randomized controlled studies and meta‐analyses in which videoconferencing and in‐person care has been directly compared in terms of clinical effectiveness.

Keywords: anxiety, COVID‐19, depression, effectiveness, videoconferencing treatment

Key Practitioner Message.

  • Under COVID‐19 circumstances, treatments were almost exclusively provided through videoconferencing.

  • Videoconferencing treatment was not less effective than in‐person treatment.

  • Treatment outcome for common mental disorders appears robust for the impact of COVID‐19.

1. INTRODUCTION

The digitalization of health care is rapidly progressing (Fairburn & Patel, 2017), a process amplified by the recent coronavirus disease‐2019 (COVID‐19) epidemic (Pierce et al., 2021; Probst et al., 2020; Wind et al., 2020; Wosik et al., 2020). The widespread use of internet, computers and smartphones has facilitated remote contact between individuals (Goodman‐Deane et al., 2016), and digital tools such as videoconferencing closely mimic in‐person communication (Krouwel et al., 2019). The implications for the delivery of health care are substantial: Treatment providers use patient portals to facilitate the transfer of information to and from patients, easily accessible (medical) information abounds, and many applications dedicated to health education and management have become available. These new tools are particularly interesting as they bring about new options for delivery of psychological treatment. Clients can now be treated remotely through the use of these digital tools. For instance, delivery of treatment through videoconferencing has been applied to various formats and types of psychotherapy (Backhaus et al., 2012).

There is a long‐standing tradition with internet‐based mental health care (MHC) in the Netherlands, starting in 1996 with Interapy, an asynchronous email‐based approach with writing assignments for PTSD patients (Lange et al., 2001). Videoconferencing treatment has also been around for quite some time in MHC (Richardson et al., 2009), but its uptake is slower in MHC as compared to other fields of health care, and widespread implementation in routine psychotherapeutic care is still lacking (Vis et al., 2018). A potential barrier for the implementation of treatment via videoconferencing may be the perspective that in‐person communication is a requisite for therapeutic change in psychotherapy, which deems in‐person contact crucial to the establishment of a good therapeutic working relation (Connolly et al., 2020). Many practitioners in MHC feel that treatment delivery in a videoconferencing format is ultimately inferior to in‐person therapy (Humer et al., 2020). Practitioners also state that they feel less confident when delivering therapy in this form (Aafjes‐van Doorn et al., 2020; Békés et al., 2021). The latter authors mention a decrease in confidence among therapists because they are afraid of not being fully attentive themselves or that their clients are easily distracted. Furthermore, this lack of confidence also results from the fear of encountering technical difficulties, which used to accompany treatment via videoconferencing (Fletcher‐Tomenius & Vossler, 2009). The limitations of videoconferencing, as perceived by therapists, are also related to the absence of non‐verbal cues, the possibility of missing out on important information and the difficulty in dealing with possible crisis situations through a digital medium (de Beurs et al., 2021b). Finally, compared to blended digital treatments (a mix of in‐person and videoconferencing or other forms of digital care), treatments without any in‐person contact are perceived as less advantageous and risky (Schuster et al., 2018). This concern is substantiated with research on patients' adherence to digital MHC, which has found that adherence can be poor, especially to free‐to‐access programs (Christensen et al., 2009). However, adherence to treatment is substantially higher when digital MHC is provided in the context of a randomized controlled trial (RCT) (Cuijpers et al., 2016), and adjunctive measures, such as weekly phone calls, appear to further improve adherence (Mohr et al., 2010).

To a large extent, patients mirror the concerns of their therapists. Two reviews of research into client preferences show that the vast majority of patients prefer in‐person contact over treatment delivered via videoconferencing (March et al., 2018; Meurk et al., 2016), mainly because they expect more benefits from in‐person treatment. However, when asked if they would like to give it a try, many clients are open to treatment via videoconferencing, and those who have already gained experience with this form of treatment are indeed more likely to opt for it again (March et al., 2018). Recently, we conducted a survey among patients of our institution (Arkin, the largest MHC provider of Amsterdam, the Netherlands) on how they experienced videoconferencing treatment under COVID‐19 social distancing measures and found that patients reported benefits from the treatment. Still, most patients preferred a return to in‐person treatment or opted for blended in‐person and videoconferencing when social distancing was no longer required (de Beurs et al., 2021a). In sum, it seems that progress with the implementation of remote care has been hampered mainly by negative perceptions, both among professionals and clients.

Negative perceptions of therapists and clients towards treatment through videoconferencing are partly due to negative expectancies of its benefits. Research, investigating the effectiveness of digital treatments, does not support the negative attitude towards videoconferencing treatment. Results of several RCTs suggest that guided internet‐based interventions are equally effective as in‐person treatment of common mental disorders (Andrews et al., 2010; Cuijpers et al., 2017; Karyotaki et al., 2018). Based on this research, treatment via videoconferencing seems to be a good alternative to the conventional in‐person approach to therapy. But professionals and clients are not convinced. They note that these findings are predominantly based on RCT studies (e.g., Andrews et al., 2010; Cuijpers et al., 2017). RCTs are the gold standard for evaluating new treatments or modes of treatment delivery, as randomization can effectively counter biasing effects of confounders, thus optimizing internal validity of a study comparing modes of treatment delivery. However, optimizing internal validity can be at the expense of external validity. When the effectiveness of videoconferencing treatment is evaluated under controlled conditions, participants of the study are randomly assigned to either in‐person or videoconferencing treatment conditions. In order to be randomized, both client and therapist must be open to both forms of treatment, which might cause a selective inclusion of study subjects where only therapists and clients partake with a favourable view on videoconferencing treatment. Consequently, it is unclear whether the similar outcomes for videoconferencing and face‐to face treatment can be generalized to everyday clinical practice. It is important to investigate how conventional in‐person treatment and delivery via videoconferencing compare in everyday clinical practice. The recent COVID‐19 has offered a prime opportunity to do so.

Wind et al. (2020) describe the recent outbreak of COVID‐19 as the ‘black swan virus’ for e‐mental health care. A ‘black swan’ is an unforeseen event that changes everything (Blumenstyk, 2020). Wind et al. (2020) imply that the viral outbreak resembles such an event and speculate that it introduced a shift from conventional in‐person therapy to the use of videoconferencing, which will, at least partly, persist in the future. This shift is likely to benefit MHC, as the use of videoconferencing has the prospect of providing treatment more efficiently and will facilitate accessibility of treatment. Videoconferencing facilitates delivery of treatment to clients who live in rural areas and thus have difficulty attending in‐person therapy (Schopp et al., 2006). For clients, the advantages of videoconferencing are as follows: no need to travel or in case of blended care, less often; and homework assignments can be executed more interactively as they can receive direct feedback from their therapist, and thus integrate therapy more easily in their everyday live. In addition, some clients may find it easier to disclose sensitive information online instead of in the therapists' room (Gega et al., 2004)—lack of self‐disclosure is a common issue in psychotherapy (Knox & Hill, 2016)—while others find it harder to establish sufficient trust online (Cataldo et al., 2021).

In any case, the outbreak of COVID‐19 has offered a unique opportunity to investigate the effectiveness of treatment via videoconferencing under the natural conditions of everyday practice. The virus originated in China and within the course of only a few months led to the development of a global pandemic. As COVID‐19 is highly communicable, drastic measures were taken to reduce human contact (Wu et al., 2020). For the delivery of psychotherapy, in‐person therapy was replaced by videoconferencing in the Netherlands after the COVID‐19 measures came into effect at mid‐March 2020. The measures sharply restricted person to person contact. Therefore, in the period from 2019 to 2021, some clients received their entire treatment in‐person, some were treated initially in‐person therapy but switched to delivery via videoconferencing, whereas others were treated entirely through videoconferencing. This study aims to compare the process of treatment and treatment outcome of patients who received treatment provided online in a synchronous manner via videoconferencing to patients who received in‐person treatment prior to the corona crisis, and to patients who received a mix of videoconferencing and in‐person treatment.

In sum, the current study aims to investigate the effectiveness of treatment via videoconferencing in everyday practice, using as a natural experiment the opportunity offered by the recent COVID‐19 measures limiting in‐person contact in the psychotherapeutic context. Based on therapists' and clients' expectancies, we hypothesized the following effect of the treatment delivery mode: a diminished outcome for treatments that were conducted via videoconferencing, as compared to in‐person treatments or treatments that started as in‐person but continued via videoconferencing on 16 March 2020.

2. METHOD

2.1. Setting

This study took place at two ambulatory clinics of Arkin, the major MHC provider in Amsterdam. One is called Basis GGZ and provides ‘Basic MHC’ (B‐MHC) focusing on time‐limited primary MHC for patients with mild singular mood or anxiety disorders, with a low risk of suicide or dangerous behaviour, targeting patients who are expected to benefit from relatively short‐term Cognitive Behavioural Therapy (CBT) interventions. B‐MHC works in close collaboration with primary care physicians and their practices. Treatments are provided predominantly by supervised master‐level psychologists. The other clinic was PuntP, which provides ‘Specialized MHC’ (S‐MHC) to patients requiring multidisciplinary care for more severe or complex problems, also with mood disorders or anxiety disorders as the most common primary treatment diagnoses, but commonly with comorbid conditions. These patients were also predominantly treated with CBT, sometimes combined with pharmacotherapy. Here, treatments are provided by master‐level psychologists, so‐called ‘health care psychologists’ (master's degree plus 2 years of practical training), and specialists in clinical psychology or psychotherapists (health care psychologists with at least 4 years of postlicensure training).

2.2. Participants

This study made use of a two convenience samples of patients with common mental disorders: One sample was comprised of patients (N = 1392) receiving short‐term treatment (a maximum of 12 sessions) in B‐MHC. The other sample (N = 1040) was comprised of patients receiving long‐term treatment in S‐MHC. Only treatments with a first session on 1 January 2019 or later and a last session at 25 May 2021 were included. According to Dutch law, use of anonymized data, which is routinely collected as part of everyday treatment and intended to support shared decision making, quality control and research, is exempt from an informed consent procedure.

In the B‐MHC sample, 66.7% of the participants identified themselves as female and were between 18 and 83 years of age (M = 36.9, SD = 13.4); 37.5% of the clients had a diagnosis of mood disorder, and 20.8% were diagnosed with an anxiety disorder. The remaining 41.7% of clients were diagnosed with other disorders, such as PTSD and adjustment disorder, or did not meet criteria for a DSM‐5 disorder.

In the S‐MHC sample, 64.5% of the participants identified themselves as female and were between 18 and 72 years of age (M = 36.4, SD = 12.5); 54.5% had a mood disorder, and 26.1% were diagnosed with an anxiety disorder. The remaining 19.4% of clients were diagnosed with other mental disorders, such as a psychotic disorder or no disorder. Table 1 provides an overview of the characteristics of the two study samples.

TABLE 1.

Samples description of clients with complete outcome data

N B‐MHC S‐MHC
1392 1040
M SD M SD
Age 36.9 13.4 36.4 12.5
Pretest severity (OQ‐SD) 52.4 15.1 56.8 14.7
Functioning (GAF) 57.3 5.9 55.6 5.4
N % N %
Female gender 926 66.7 700 64.5
Primary diagnosis N % N %
Mood disorder 520 37.5 592 54.5
Anxiety disorder 289 20.8 283 26.1
PTSD 173 12.5 9 0.8
Adjustment disorder 128 9.2 7 0.6
Psychotic disorder 129 9.3 94 8.7
Personality disorder 15 1.1 14 1.3
Other/missing 134 9.7 87 8.0
Comorbidity N % N %
No comorbidity 806 58.1 449 41.3
Axis 1 comorbidity 552 39.8 560 51.6
Axis 2 comorbidity 30 2.2 77 7.1
M SD M SD
Number of sessions 10.2 5.6 31.3 20.3
Treatment duration (days) 142.5 77.4 287.7 85.0
Open in a new tab

Regarding demographics, both samples were similar; regarding severity and diagnoses, S‐MHC sees more severe and complex cases with more comorbidity. There were more adjustment disorders and (mild) PTSD in B‐MHC as compared to S‐MHC. The low number of primary PTSD diagnoses in S‐MHC is due to the fact that Arkin has a specific clinic specialized in the treatment of PTSD. Mean treatment duration in S‐MHC was almost twice as long with three to four times more sessions.

2.3. Procedure

The current study was a longitudinal observational study. All participants were assessed repeatedly with self‐report questionnaires through Routine Outcome Monitoring (ROM; Carlier et al., 2012; de Beurs et al., 2011). ROM is a method to monitor treatment progress by continuously or periodically measuring the nature and severity of the client's problems. The frequency of ROM assessments varies from a high‐frequency, session‐by‐session approach to low frequency (e.g., every 6 weeks or 3 months). Low‐frequency ROM is routine practice in the clinics of Arkin. According to the ROM approach implemented in B‐MHC and S‐MHC, the severity of symptoms and level of functioning were assessed at the beginning of treatment, and assessments were repeated approximately every 3 to 4 months during treatment to monitor symptom change and functioning.

In the S‐MHC sample, we used the ROM assessments that were closest to the date of the first and the last treatment session to evaluate outcome. In order to homogenize the cohorts regarding their treatment duration, we limited our analysis in S‐MHC to the first year of treatment; later ROM assessments were excluded. We analysed only data from the first year in order to evaluate treatment (sections) of similar duration. By May 2021, treatments provided under COVID‐19 conditions had a maximum duration of 14 months. Otherwise, due to lengthy treatments, the three cohorts may diverge in average treatment length, which would provide an alternative explanation for differences in outcome among the cohorts. This was only relevant for S‐MHC where one third of the treatments continue for longer than 1 year. In B‐MHC, lengthy treatments (>1 year) were not present, and here, the first and the last ROM assessments were also used for pretest and post‐test data.

Although ROM is routine practice in the Arkin clinics, not all patients completed all the assessments. Only treatments with complete pretest and post‐test data for the first year were included, about 55% of all treatments that had been started, 43.8% of the S‐MHC and 70.3% of the B‐MHC treatments. We compared fully assessed treatments to treatments with incomplete data on pretest patient characteristics to check for selective ROM non‐response (de Beurs, Warmerdam, & Twisk, 2019).

2.4. Cohorts relative to 16 March 2020

For B‐MHC and S‐MHC, we compared outcomes among three cohorts of participants that were based on the timing of their treatment interval relative to the date of the implementation of lockdown measures: (i) in‐person treatments conducted and concluded prior to the date that COVID‐19 restrictions regarding interpersonal contact came fully in effect (16 March 2020), (ii) treatments conducted partially under COVID‐19 restrictions (having started as in‐person treatment prior to 16 March and continued by means of videoconferencing on 16 March and later) and (iii) treatments executed entirely under COVID‐19 restrictions (started on 16 March or later and predominantly through videoconferencing; see Figure 1). Demographics, clinical characteristics and ROM response rates of the three cohorts in the B‐MHC and S‐MHC samples are presented in Tables 2 and 3, respectively.

FIGURE 1.

FIGURE 1

Open in a new tab

Composition of three cohorts based on the first and last treatment session date relative to 16 March 2020

TABLE 2.

Comparison of treatment duration and demographic and clinical characteristics of the three treatment cohorts in B‐MHC (N = 1392)

Relative to COVID‐19 Total
Prior Partially Entirely
1001 338 640 1979
Initial sample size (N) N % N % N % χ 2(2) p
Treatment dropouts 108 10.8 55 16.3 97 15.2 10.03 .007 1 < 2 = 3
ROM response 721 72.0 231 68.3 440 68.8 2.79 .25 1 = 2 = 3
M SD M SD M SD F(2) p Post hoc
Age 36.6 13.0 37.4 14.0 37.0 13.6 0.35 .71 1 = 2 = 3
Pretest severity (OQ‐SD) 53.4 15.4 51.2 14.1 51.6 15.0 2.84 .06 1 = 2 = 3
Functioning (GAF) 57.1 6.0 57.2 6.2 57.6 5.7 0.71 .49 1 = 2 = 3
N % N % N % χ 2(2) p
Female gender 484 67.2 155 67.7 287 65.4 0.53 .77 1 = 2 = 3
Diagnosis N % N % N % χ 2(12) p
Depression 307 42.6 81 35.4 132 30.1 30.31 .003 1 > 2 > 3
Anxiety 139 19.3 58 25.3 92 21.0 2 > 3 > 1
PTSD 77 10.7 32 14.0 64 14.6
Adjustment disorder 59 8.2 15 6.6 54 12.3
Pers disorder 9 1.3 0 0.0 6 1.4
Psychotic disorder 62 8.6 23 10.0 44 10.0
Other 67 9.3 20 8.7 57 10.7
Comorbidity N % N % N % χ 2(6) p
No comorbidity 406 56.4 133 58.1 267 60.8 2.81 .59 1 = 2 = 3
Axis 1 comorbidity 296 41.1 91 39.7 165 37.6
Axis 2 comorbidity 18 2.5 5 2.2 7 1.6
M SD M SD M SD F(2) p Post hoc
Number of sessions 15.0 7.8 18.5 7.0 15.8 8.7 16.73 <.001 2 > 1, 2 > 3, 1 = 3
Mean treatment duration 124.6 73.5 200.8 73.3 141.5 70.7 95.67 <.001 2 > 1, 2 > 3, 3 > 1
Open in a new tab

TABLE 3.

Comparison of treatment duration and demographic and clinical characteristics of the three treatment cohorts in S‐MHC (N = 1040)

Relative to COVID‐19 Total
Prior Partially Entirely
1374 735 256 2374
Initial sample size (N) N % N % N % χ 2(2) p
Treatment dropouts 183 13.3 48 6.5 36 13.6 10.03 .007
ROM response 551 40.1 355 48.3 134 50.6 18.61 <.001 1 < 2 < 3
M SD M SD M SD F(2) p Post hoc
Age 36.9 12.5 35.3 12.1 37.0 13.0 2.17 .12 1 = 2 = 3
Pretest severity (OQ‐SD) 56.7 15.3 57.0 14.0 56.8 14.3 0.07 .93 1 = 2 = 3
Functioning (GAF) 56.1 5.6 55.1 5.4 54.6 4.0 6.51 .002 1 > 2, 1 > 3, 2 = 3
N % N % N % χ 2(2) p
Female gender 373 62.5% 228 64.2% 99 73.9% 6.22 .045 1 = 2 < 3
Diagnosis N % N % N % χ 2(12) p
Depression 332 55.6% 195 54.9% 65 48.5% 29.82 .003 1 = 2 > 3
Anxiety 150 25.1% 94 26.5% 39 29.1% 1 = 2 < 3
PTSD 4 0.7% 2 0.6% 3 2.2% 1 = 2 < 3
Adjustment disorder 3 0.5% 3 0.8% 1 0.7%
Pers disorder 4 0.7% 4 1.1% 6 4.5%
Psychotic disorder 57 9.5% 21 5.9% 16 11.9%
Other 47 7.9% 36 10.1% 4 3.0%
Comorbidity N % N % N % χ 2(6) p
No comorbidity 259 43.4% 144 40.6% 46 34.3% 3.86 .43 1 = 2 = 3
Axis 1 comorbidity 297 49.7% 186 52.4% 77 57.5%
Axis 2 comorbidity 41 6.9% 25 7.0% 11 8.2%
M SD M SD M SD F(2) p Post hoc
Number of sessions 54.6 31.6 67.5 41.3 51.4 32.6 19.9 <.001 2 > 1, 2 > 3, 1 = 3
Mean treatment duration 279.4 85.3 325.6 60.9 224.7 91.5 86.65 <.001 2 > 1, 2 > 3, 1 > 3
Open in a new tab

2.5. Measures

At pretest, patients' level of functioning was evaluated on the Global Assessment of Functioning scale (GAF; Endicott et al., 1976). All patients were diagnosed according to the DSM‐5 (American Psychiatric Association, 2013). Data on premature treatment termination (dropout) were derived from patients' electronic records, where therapists register the reasons for discontinuation.

Treatment outcomes were measured at the pretest and post‐test with the Symptomatic Distress scale of the Outcome Questionnaire (OQ‐45; Lambert et al., 1996). The OQ‐SD scale measures psychological symptoms with 25 items and has good reliability and validity indicators. In addition, treatment outcome was operationalized by the clinical status of the patient at post‐test in four levels: recovered, merely improved, no reliable change or deteriorated. To do so, two indicators were used: the Reliable Change Index (RCI) and Clinical Significance (CS). The RCI determines whether the change from pretest to post‐test was statistically reliable (Jacobson & Truax, 1991). We converted scores to T scores and used a cut‐off of 5 T‐score points as proposed by de Beurs, Carlier, and van Hemert (2019). In addition, in order to decide whether a change is clinically meaningful, a cut‐off score of T < 55 (de Beurs, Carlier, & van Hemert, 2019) was used to categorize a patient as functional or dysfunctional at the post‐test (CS). When RCI and CS are combined, clients can be categorized into four levels of outcome: (1) recovery and reliable change, (2) reliable change but no recovery, (3) no change and (4) deterioration (reliable change in the ‘wrong’ direction) (de Beurs et al., 2016).

2.6. Statistical analysis

First, pretest and post‐test scores were screened for outliers with boxplots. Next, assumptions for a one‐way analysis of variance (ANOVA) were checked. Normality was tested by visual inspection of histograms and QQ plots; homogeneity of variances was checked with scatterplots of regression standardized results. The number of treatments concluded per month was checked to investigate whether there was a spike in treatment termination around the introduction of the COVID‐19 measures in March 2020.

We checked with MANOVA and χ 2 tests for differences in demographics and clinical characteristics between patients with and without complete ROM data. With ANOVA and χ 2 tests, we checked whether the three cohorts within each treatment setting differed concerning demographics, pretest severity, functioning and regarding their diagnoses and comorbidity. We also checked whether the three cohorts differed regarding their mean duration of treatments and regarding the dropout rate.

Finally, to compare outcome among the three cohorts, for each sample, a repeated measures ANOVA was performed with time as within factor and group as between factors. An a priori power analysis demonstrated that for a small effect size (η 2 = 0.01), a total sample size of 387 would provide 95% power (G‐power; Faul et al., 2009). With a sample size of 1000+, ample power can therefore be assumed and the chance of a Type 2 error (erroneously deciding that there is no difference in outcome among the three cohorts) is small. The statistic most pertinent to the research question is the interaction between group and time: A significant interaction indicates a difference in course of symptoms over time (i.e., treatment outcome) between the cohorts. Pairwise comparisons were planned, comparing treatment during COVID‐19 through videoconferencing to the other two cohorts (2 > 3, 1 > 3). Cohort 3 was expected to yield inferior results to Cohorts 1 and 2, where treatment had started in‐person, providing an opportunity for the therapist and the client to become acquainted in. In case of significant pretest differences among the cohorts, demographics, diagnoses, functioning and treatment length could be included as covariates into the main analysis, as pretest differences may confound the results. In addition, treatment outcome categories (recovery, improvement, etc.) were compared between the three cohorts with a χ 2 test.

We used administrative data regarding the treatment delivery mode to assess videoconferencing use in the three treatment cohorts. As an additional inspection of how videoconferencing may influence treatment outcome, the association between the percentage of videoconferencing and the pretest‐to‐post‐test difference was calculated in the B‐MHC sample and in the S‐MHC sample and, within each sample, for the three cohorts. A significant association between the proportion of in‐person contact and outcome, especially expected for the overall samples and for Cohort 2, would support our hypothesis of a diminished treatment outcome for videoconferencing treatment.

3. RESULTS

Table 2 presents demographics, pretest clinical characteristics and ROM response rates of the three cohorts within B‐MHC and S‐MHC. Cohorts differed regarding their dropout rates: Less patients terminated treatment prematurely in the cohort that was treated prior to COVID‐19 (10.8%), compared to the other cohorts (16.3% and 15.2%). There was no difference in ROM response rate among the three cohorts in the B‐MHC setting, and comparison of included and excluded patients due to ROM non‐response did not reveal meaningful differences. Furthermore, at pretest, cohorts were similar in age, gender, pretest severity of symptomatic distress (F(2) = 2.84; p = .06) and functioning. Regarding diagnosis, depression was somewhat lower in the COVID‐19 cohorts; anxiety was somewhat higher in the cohort that was partially treated during COVID‐19. Regarding comorbidity, cohorts were again similar. Treatments comprised of more sessions and were longer in duration in the cohort that was partially treated during COVID‐19. The treatment duration of the COVID‐19 cohort fell in between both other cohorts.

For the S‐MHC sample, cohorts differed regarding the dropout rate. However, in S‐MHC, less patients terminated treatment prematurely in the cohort that was treated partially during COVID‐19 (6.5%), as compared to the other cohorts (13.3% prior to COVID‐19 and 13.6% for those treated entirely during COVID‐19). Furthermore, a small difference in ROM response was found among the cohorts with response increasing over time (from Cohorts 1 to 2 and to 3). Regarding demographic and clinical characteristics, the included patients were similar to those without sufficient ROM assessments. At pretest, cohorts were similar in age and pretest severity of symptomatic distress but differed in functioning with the cohort treated prior to COVID‐19 functioning better (F(2) = 6.51; p = .002). Women were overrepresented in the COVID‐19 cohort in both settings. Regarding the primary diagnosis, depression was somewhat lower, and anxiety and PTSD were somewhat higher in the COVID‐19 cohort. Regarding comorbidity, cohorts were again similar. Treatments comprised of more sessions and were longer in duration in the second cohort, to aid with the transition from in‐person to videoconferencing treatment.

Table 4 shows the mean (and SD) on the OQ‐SD in T scores for the three cohorts in B‐MHC (upper half) and S‐MHC (lower half) and the results of the repeated measures ANOVA. For B‐MHC, a time effect was found (F(1,1385) = 617.65, p < .001, partial η 2 = .308, on average clients improve) but no group effect (F(2,1385) = 2.58; p = .08, partial η 2 = .004) and no group‐by‐time interaction (F(1,1385) = 0.10; p = .91; partial η 2 < .001). Post hoc pairwise comparisons, comparing treatments during corona with treatment prior to corona and with treatments partially conducted during corona also did not reveal a difference in outcome between these pairs (p = .19 and p = .87, respectively).

TABLE 4.

Treatment outcome in the three cohorts relative to COVID‐19 in Basic MHC and Specialized MHC

Relative to COVID‐19
Prior Partially Entirely
B‐MHC M SD M SD M SD F(2,1385) p Contrast
Pretest T 73.1 11.2 71.5 10.2 71.8 10.9 0.10 .91 1 = 3, 2 = 3
Post‐test T 64.7 14.4 63.0 12.4 63.6 13.4
N % N % N % χ 2(6) p
Recovered 168 23.3 58 25.3 85 19.4 5.92 .43
Improved 247 34.3 70 30.6 167 38.0
Unchanged 248 34.4 84 36.7 155 35.3
Deteriorated 57 7.9 17 7.4 23 7.3
Relative to COVID‐19
Prior Partially Entirely
S‐MHC M SD M SD M SD F(2) p Contrast
Pretest T 75.4 11.3 75.7 10.3 75.5 10.5 0.62 .54 1 = 3/2 = 3
Post‐test T 65.1 13.7 66.1 12.8 65.6 14.1
N % N % N % χ 2(6) p
Recovered 118 19.8 61 17.2 22 16.4 4.78 .57
Improved 269 45.1 163 45.9 88 50.7
Unchanged 162 27.1 109 30.7 37 27.6
Deteriorated 48 8.0 22 6.2 7 5.2
Open in a new tab

For S‐MHC, we also obtained a time effect (F(1,1083) = 569.67; p < .001, partial η 2 = .345) but no group effect (F(2,1083) = 2.58; p = .66, partial η 2 = .001) and no group‐by‐time interaction (F(1,1083) = 0.62; p = .54; partial η 2 < .001). Again, pairwise comparisons did not reveal a difference in outcome either (Cohort 3 vs. 1 p = .95; Cohort 3 vs. 2: p = .98).

Figure 2 displays the decrease in OQ‐SD T score from pretest to post‐test for the three cohorts in B‐MHC and S‐MHC. Proportions of recovered, improved, unchanged and deteriorated patients (also depicted in Table 4) did not differ among the treatment cohorts in B‐MHC (χ 2(6) = 5.92; p = .43) and S‐MHC (χ 2(6) = 4.78; p = .57). Figure 3 displays these various proportions in B‐MHC and S‐MHC. Finally, Table 5 shows for B‐MHC and S‐MHC, overall and per cohort relative to COVID‐19, correlation coefficients for the association of the proportion of sessions with videoconferencing in the treatment (0% to 100%) with outcome (operationalized as the decrease in symptomatic distress according to the OQ‐SD between the pretest and post‐test). The correlation coefficients indicated only small associations, and the direction of the two statistically significant associations was positive, suggesting a better treatment outcome when a larger part of the treatment was conducted through videoconferencing.

FIGURE 2.

FIGURE 2

Open in a new tab

Change over time of OQ‐SD‐based T scores for three cohorts relative to COVID‐19 measures in B‐MHC (left) and S‐MHC (right)

FIGURE 3.

FIGURE 3

Open in a new tab

Proportion of patients in each outcome category in three cohorts relative to COVID‐19 measures in B‐MHC (left) and S‐MHC (right)

TABLE 5.

Correlation between the proportion of videoconferencing sessions and outcome (DeltaT)

Overall Relative to COVID
Prior Partially Entirely
B‐MHC .07 * .03 −.01 .16 **
S‐MHC .02 −.06 .03 .16
Open in a new tab

Note: A positive association indicates more treatment effect with a higher proportion of videoconferencing. For all other r: p > .16.

*

p = .014.

**

p = .001.

4. DISCUSSION

4.1. Main findings

For both B‐MHC and S‐MHC, we found no evidence of diminished effectiveness of treatments that were provided predominantly through videoconferencing during the COVID‐19 lockdown. Also, no clear association was found between the proportion of videoconferencing sessions in the treatment and outcome. Thus, the conclusion seems justified that fears of a diminished effect of videoconferencing treatment are not supported by the present findings.

For B‐MHC, the result of the pairwise comparison came close to a significant difference, with the cohort treated prior to COVID‐19 having somewhat higher scores at pretest and post‐test compared to the other cohorts. However, the slope of decrease of symptoms over time was similar in the three cohorts (see Figure 2, left), demonstrating equally effective treatment in all cohorts. Among S‐MHC cohorts, there were also no significant differences found.

The finding of noninferior outcomes for videoconferencing therapy was unexpected and striking, especially because conditions for its implementation were far from ideal: The transition was unplanned and sudden, and therapists were not trained in this delivery mode and had no experience with the software that was used; the latter also applied to the clients. Our surveys among professionals (de Beurs et al., 2021b) and clients (de Beurs et al., 2021a) revealed that there were initially problems with the use of software. Furthermore, the COVID‐19 pandemic has consequences, beyond its implication for treatment delivery, such as a negative impact on mental health and well‐being (Brooks et al., 2020; Fiorillo & Gorwood, 2020). However, this also does not appear to lead to worse outcomes for MHC treatment for common mental disorders, as outcomes in this observational study remained the same in the three cohorts for both B‐MHC and S‐MHC. Importantly, patients treated predominantly through videoconferencing did not differ in other aspects (demographically or clinically) from the cohort treated predominantly through conventional in‐person treatment prior to the COVID‐19 pandemic or those treated initially in‐person but shifted to treatment by videoconferencing during COVID‐19.

Videoconferencing treatment is still viewed as an inferior treatment option to in‐person by many professionals and patients (Békés et al., 2021). In our own surveys, among both groups, a majority indicated a preference for a return to in‐person treatment or in‐person blended with videoconferencing when COVID‐19 lockdown restrictions are lifted (de Beurs et al., 2021a, 2021b), stating that interpersonal contact and information transfer is leaner as non‐verbal and implicit communication is lacking. However, a recent literature review (Cataldo et al., 2021) demonstrated divergence between professionals and patients in how videoconferencing impacts the therapeutic relationship: Many professionals highlight difficulties in establishing an effective therapeutic relationship in treatment via videoconferencing, whereas patients are generally satisfied with the relationship in videoconference therapy. Békés et al. (2021) mentioned that therapists differ in their ability to establish a strong therapeutic bond through videoconferencing treatment and also investigated professional self‐doubt (in general and with this new technology). They concluded that both the view of the therapeutic relationship and professional self‐doubt generate hesitance among therapists to use videoconferencing. Future research should investigate the influence of therapist factors such as age, having received training (Pierce et al., 2020) or (prior) experience with digital MHC (Glueckauf et al., 2018), professional confidence and therapeutic orientation (Probst et al., 2021) on openness to this treatment delivery mode.

4.2. Different dropout rates among cohort and samples

In B‐MHC, we found a significant difference between the cohorts in premature terminations of treatment: A higher dropout rate occurred among treatments conducted (partially) during COVID‐19 compared to treatments prior to COVID‐19 (1.5 more dropouts). Furthermore, in B‐MHC, the higher dropout rates during COVID‐19 suggest that treatment via videoconferencing may be less acceptable for certain patients or their engagement or involvement may be diminished compared to in‐person treatment. Similar findings have been reported for patient with PTSD (Valentine et al., 2020).

In the S‐MHC sample, where dropout was generally lower, we also found a difference in dropout rate between cohorts, but here, dropout was lower among patients who had to transition from in‐person to videoconferencing treatment. Thus, dropout rates in the longer S‐MHC showed a different pattern from what was found in B‐MHC. We have no specific information regarding motives of premature termination. Future research should focus on characteristics of dropouts, and extra care should be taken in short B‐MHC videoconferencing treatments to sustain patients' engagement.

As dropouts occurred predominantly in the initial phase of treatment, no post‐test scores were available for these patients, and all subsequent comparisons between cohorts were done on completers. This implies that the present findings of similar outcomes can only be generalized to patients who remain in treatment (Tierney & Stewart, 2005).

4.3. Alternative explanations

Did a substantial group of patients discontinue treatment on 16 March 2020? If so, this could suggest a rival explanation for the lack of a difference in outcome among the treatment cohorts. We looked at administrative data for a spike in aborted treatments in mid‐March 2020, and there was not any. In the survey conducted among our clients in April/May 2020, many indicated that they appreciated the option of continuing their treatment, and although considering videoconferencing second best, they showed an understanding that there was no alternative due to the lockdown (de Beurs et al., 2021a).

4.4. Differences in lengths of treatment between the cohorts

In S‐MHC, we limited the scope of the outcome data to the first year of treatment to homogenize treatment duration among the cohorts. We collected data until June 2021, and treatments with a duration of more than 14 months could not be included in the videoconferencing cohort. Therefore, we decided to limit the outcome data for all cohorts to the first year of treatment.

An alternative explanation for a lack of the expected diminished treatment effect in the videoconferencing cohort might be that these patients were treated longer or more intensely. We compared the number of sessions and treatment duration among the cohorts and found that treatments of patients who transitioned from in‐person to videoconferencing (Cohort 2) were longer. One might speculate that the transition to videoconferencing required an additional effort. However, the number of sessions and treatment duration of patients treated entirely via videoconferencing was similar to treatments delivered prior to COVID. Perhaps therapists gained more experience with and grew more accustomed to videoconferencing over time. At any rate, the results showed that no longer or more intense treatment was provided in the cohort treated entirely during the COVID‐19 crisis. In S‐MHC, the average treatment duration was even somewhat shorter in the videoconferencing cohort, which could have resulted from more efficient contact between therapist and patient. In the survey, therapists reported that they felt more ‘business‐like’ and mentioned that videoconferencing was more personally intense and demanding (de Beurs et al., 2021b). Cost‐effectiveness studies suggest that guided digital interventions for the treatment of depression and other disorders (Donker et al., 2015) have the potential to be a cost‐effective complement to treatment as usual, although more methodologically sound studies are needed (Paganini et al., 2018). In the future, when the COVID‐19 pandemic and its lockdown is further behind us, a comparison of the length of full‐treatment trajectories can shed more light on the potential greater efficiency of videoconferencing psychotherapy.

4.5. Strengths and limitations

A strength of the present study is its naturalistic design: Pretest and post‐test data of ROM (de Beurs et al., 2011) were used to evaluate outcomes of treatments delivered in everyday clinical practice of B‐MHC and S‐MHC for common mild to moderately severe psychiatric disorders, such as depression and anxiety disorders. The pre–post change reported in this study is comparable to what has been reported before for B‐MHC (van Mens et al., 2017) and S‐MHC (de Beurs, Warmerdam, & Twisk, 2019) in the Netherlands and so are the proportions of recovered, improved, unchanged and deteriorated patients (de Beurs et al., 2016). The external validity of the study findings is amplified by these features. At the same time, the observational nature of the study also brings forth some limitations: There is less control over the collection of data compared to what is customary in a controlled study. For instance, only treatments with complete ROM data (i.e., a pretest assessment and a post‐test assessment) were included. As ROM response was around 70% in B‐MHC and 45% in S‐MHC, results may be biased by selective non‐response (selective inclusion at the pretest or selective attrition at the post‐test). Previous research has shown that when ROM response is >50%, the biasing effects are limited (de Beurs, Warmerdam, & Twisk, 2019; Gomes et al., 2015). Furthermore, it is unlikely that the lack of a difference in outcome between the modes of treatment delivery is explained by selective non‐response, as response levels for ROM were quite similar in the three cohorts within each sample. Our analyses did not reveal systematic differences between included and excluded patients in demographic of clinical characteristics. Nevertheless, potentially, treatment outcomes could be overestimated by selective inclusion of well‐treatable patients at pretest or by selective attrition of treatment failures at post‐test. Overestimated outcomes may obscure differences in outcome between the cohorts, which would otherwise become apparent. In particular, in S‐MHC, the findings may be biased, as in S‐MHC, the overall response rate was below 50%. Low‐frequency ROM is implemented in the Arkin clinics, which implies that ROM assessments were completed periodically (every 3 months); of these, we used only pretest and post‐test assessments. Consequently, possible differences between cohorts in the trajectory of change were not detected. Finally, outcome was only assessed by a self‐report questionnaire, the OQ‐45. Although the subscale for symptomatic distress (OQ‐SD) of the OQ‐45 has good psychometric properties (Lambert et al., 1996), using clients' self‐reports as the only source of information may render the findings vulnerable to bias. It should be noted that self‐reports do not necessarily lead to overestimated outcomes, as evidenced by Cuijpers et al. (2010) who undertook a meta‐analysis to compare ratings by clinicians to self‐reports by patients. They found that self‐reports render a more conservative estimate of treatment efficacy in controlled trials compared to outcome ratings of clinicians. Still, future studies might benefit from including an assessment of treatment benefit from the professional or from an independent rater.

A further limitation is that the form of digital care evaluated in the present study—videoconferencing—uses only one of the common channels for live contact between a therapist and client. In contrast, digital care may involve varying amounts of therapist contact blended with computerized tasks. Sometimes therapist contact is limited to asynchronous messages, such as email. Indeed, the present study primarily compared two modes of treatment delivery: in‐person versus synchronous videoconferencing. Other forms of eHealth, such as blended eHealth, in which part of therapist involvement is replaced by interaction with the computer or asynchronous messaging, were not evaluated.

Another limitation is related to the observational design of the study. Its findings will have to be evaluated considering a major confounder: All data for treatment outcome of therapy conducted through videoconferencing were taken from time intervals within the COVID‐19 crisis. The assumption that mental health patients, and the human population in general, were mentally more troubled during this time is undeniable. Indeed, Robillard et al. (2021) found that during the COVID‐19 crisis depression and anxiety symptoms as well as suicidal ideation have significantly increased. Professionals were also hindered by the lockdown measures in performing therapy from home. Our survey demonstrated that many complained about lack of privacy and distraction from family members not going to work or school (de Beurs et al., 2021b). Thus, the COVID‐19 pandemic, its concurrent requirements of social distancing resulting in social isolation and COVID‐19 associated fears, anxiety and worries may by itself lead to diminished outcomes of therapeutic efforts, irrespective of the delivery mode of the treatment. As the nature of this study is observational, the possible effect of the viral outbreak on psychological symptoms cannot be separated from the effect of switching to videoconferencing treatment. This is called ‘history’ in the listing of threats to internal validity by Campbell and Stanley (1966): an external event occurring during the data gathering phase of a study that effects the outcome. A limitation of the present research is its quasi‐experimental and uncontrolled design. Future research should investigate the comparative efficacy of in‐person and videoconferencing treatment in an RCT. Furthermore, other forms of eHealth, such as blended forms, should be evaluated in a controlled design.

4.6. Conclusion

The present results of this quasi‐experimental study suggest that videoconferencing may be a viable alternative to in‐person treatment for patients who find this an acceptable form of treatment. Concerns that treatment via videoconferencing might yield a diminished treatment effect are not substantiated by the ROM data. Moreover, we found no evidence for the inferiority of videoconferencing compared to in‐person treatment, despite the adverse COVID‐19 conditions under which the videoconferencing treatment took place. Thus, less personal contact, reduced transfer of non‐verbal information, lower expectations of videoconferencing treatment effectiveness or the circumstances related to the COVID‐19 conditions itself were not associated with a diminished outcome of treatment.

CONFLICT OF INTEREST

No conflicts to declare.

TRANSPARENCY STATEMENT

All authors confirm that they have not published before and/or have not something in press around this data set. The manuscript has not been previously published and is not currently under consideration by any other journal. Additionally, all authors have approved the contents of this paper and agree to the CPP's submission policies.

ACKNOWLEDGEMENT

None.

de Beurs, E. , Blankers, M. , Peen, J. , Rademacher, C. , Podgorski, A. , & Dekker, J. (2022). Impact of COVID‐19 social distancing measures on routine mental health care provision and treatment outcome for common mental disorders in the Netherlands. Clinical Psychology & Psychotherapy, 29(4), 1342–1354. 10.1002/cpp.2713

DATA AVAILABILITY STATEMENT

Data obtained for the study will be made available upon request.

REFERENCES

  1. Aafjes‐van Doorn, K. , Békés, V. , & Prout, T. A. (2020). Grappling with our therapeutic relationship and professional self‐doubt during COVID‐19: Will we use video therapy again? Counselling Psychology Quarterly, 34(3–4), 473–484. 10.1080/09515070.2020.1773404 [DOI] [Google Scholar]
  2. American Psychiatric Association . (2013). Diagnostic and statistical manual of mental disorders (DSM‐5®). American Psychiatric Association. [Google Scholar]
  3. Andrews, G. , Cuijpers, P. , Craske, M. G. , McEvoy, P. , & Titov, N. (2010). Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: A meta‐analysis. PLoS ONE, 5(10), e13196. 10.1371/journal.pone.0013196 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Backhaus, A. , Agha, Z. , Maglione, M. L. , Repp, A. , Ross, B. , Zuest, D. , Rice‐Thorp, N. M. , Lohr, J. , & Thorp, S. R. (2012). Videoconferencing psychotherapy: A systematic review. Psychological Services, 9(2), 111–131. 10.1037/a0027924 [DOI] [PubMed] [Google Scholar]
  5. Békés, V. , Aafjes‐van Doorn, K. , Zilcha‐Mano, S. , Prout, T. , & Hoffman, L. (2021). Psychotherapists' acceptance of telepsychotherapy during the COVID‐19 pandemic: A machine learning approach. Clinical Psychology & Psychotherapy, 28(6), 1403–1415. 10.1002/cpp.2682 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Blumenstyk, G. (2020). Why coronavirus looks like a “black swan” moment for higher ed. The Chronicle. [Google Scholar]
  7. Brooks, S. K. , Webster, R. K. , Smith, L. E. , Woodland, L. , Wessely, S. , Greenberg, N. , & Rubin, G. J. (2020). The psychological impact of quarantine and how to reduce it: Rapid review of the evidence. The Lancet, 395(10227), 912–920. 10.1016/S0140-6736(20)30460-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Campbell, D. T. , & Stanley, J. C. (1966). Experimental and quasi‐experimental designs for research. Ravenio Books. [Google Scholar]
  9. Carlier, I. V. , Meuldijk, D. , van Vliet, I. M. , van Fenema, E. , van der Wee, N. J. , & Zitman, F. G. (2012). Routine outcome monitoring and feedback on physical or mental health status: Evidence and theory. Journal of Evaluation in Clinical Practice, 18(1), 104–110. 10.1111/j.1365-2753.2010.01543.x [DOI] [PubMed] [Google Scholar]
  10. Cataldo, F. , Chang, S. , Mendoza, A. , & Buchanan, G. (2021). A perspective on client‐psychologist relationships in videoconferencing psychotherapy: Literature review. JMIR Mental Health, 8(2), e19004. 10.2196/19004 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Christensen, H. , Griffiths, K. M. , & Farrer, L. (2009). Adherence in internet interventions for anxiety and depression: Systematic review. Journal of Medical Internet Research, 11(2), e13. 10.2196/jmir.1194 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Connolly, S. L. , Miller, C. J. , Lindsay, J. A. , & Bauer, M. S. (2020). A systematic review of providers' attitudes toward telemental health via videoconferencing. Clinical Psychology: Science and Practice, 27(2), e12311. 10.1111/cpsp.12311 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Cuijpers, P. , Cristea, I. A. , Karyotaki, E. , Reijnders, M. , & Huibers, M. J. (2016). How effective are cognitive behavior therapies for major depression and anxiety disorders? A meta‐analytic update of the evidence. World Psychiatry, 15(3), 245–258. 10.1002/wps.20346 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Cuijpers, P. , Kleiboer, A. , Karyotaki, E. , & Riper, H. (2017). Internet and mobile interventions for depression: Opportunities and challenges. Depression and Anxiety, 34(7), 596–602. 10.1002/da.22641 [DOI] [PubMed] [Google Scholar]
  15. Cuijpers, P. , Li, J. , Hofmann, S. G. , & Andersson, G. (2010). Self‐reported versus clinician‐rated symptoms of depression as outcome measures in psychotherapy research on depression: A meta‐analysis. Clinical Psychology Review, 30(6), 768–778. 10.1016/j.cpr.2010.06.001 [DOI] [PubMed] [Google Scholar]
  16. de Beurs, E. , Barendregt, M. , de Heer, A. , van Duijn, E. , Goeree, B. , Kloos, M. , Kooiman, K. , Lionarons, H. , & Merks, A. (2016). Comparing methods to denote treatment outcome in clinical research and benchmarking mental health care. Clinical Psychology & Psychotherapy, 23, 308–318. 10.1002/cpp.1954 [DOI] [PubMed] [Google Scholar]
  17. de Beurs, E. , Blankers, M. , Goudriaan, A. E. , de Koning, M. , & Dekker, J. J. M. (2021a). Beeldbellen in de GGZ: de mening van cliënten. Tijdschrift voor Psychotherapie, 47(3), 176–184. [Google Scholar]
  18. de Beurs, E. , Blankers, M. , Goudriaan, A. E. , de Koning, M. , & Dekker, J. J. M. (2021b). Zorgverlening in tijden van corona: de uitkomsten van een enquête over zorg‐op‐afstand onder medewerkers in de GGZ [Mental health care in times of corona: Experiences of professionals with eHealth]. Tijdschrift voor Psychiatrie, 63(4), 242–250. https://www.tijdschriftvoorpsychiatrie.nl/assets/articles/63-2021-4-artikel-debeurs.pdf [PubMed] [Google Scholar]
  19. de Beurs, E. , Carlier, I. V. , & van Hemert, A. M. (2019). Approaches to denote treatment outcome: Clinical significance and clinical global impression compared. International Journal of Methods in Psychiatric Research, 28, e1797. 10.1002/mpr.1797 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. de Beurs, E. , den Hollander‐Gijsman, M. E. , van Rood, Y. R. , van der Wee, N. J. , Giltay, E. J. , van Noorden, M. S. , van der Lem, R. , van Fenema, E. , & Zitman, F. G. (2011). Routine outcome monitoring in the Netherlands: Practical experiences with a web‐based strategy for the assessment of treatment outcome in clinical practice. Clinical Psychology & Psychotherapy, 18(1), 1–12. 10.1002/cpp.696 [DOI] [PubMed] [Google Scholar]
  21. de Beurs, E. , Warmerdam, L. , & Twisk, J. W. (2019). Bias through selective inclusion and attrition: Representativeness when comparing providers performance with routine outcome monitoring data. Clinical Psychology & Psychotherapy, 26(2), 430–439. 10.1002/cpp.2364 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Donker, T. , Blankers, M. , Hedman, E. , Ljótsson, B. , Petrie, K. , & Christensen, H. (2015). Economic evaluations of internet interventions for mental health: A systematic review. Psychological Medicine, 45(16), 3357–3376. 10.1017/S0033291715001427 [DOI] [PubMed] [Google Scholar]
  23. Endicott, J. , Spitzer, R. L. , Fleiss, J. L. , & Cohen, J. (1976). The Global Assessment Scale: A procedure for measuring overall severity of psychiatric disturbance. Archives of General Psychiatry, 33, 766–771. 10.1001/archpsyc.1976.01770060086012 [DOI] [PubMed] [Google Scholar]
  24. Fairburn, C. G. , & Patel, V. (2017). The impact of digital technology on psychological treatments and their dissemination. Behaviour Research and Therapy, 88, 19–25. 10.1016/j.brat.2016.08.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Faul, F. , Erdfelder, E. , Buchner, A. , & Lang, A.‐G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. 10.3758/brm.41.4.1149 [DOI] [PubMed] [Google Scholar]
  26. Fiorillo, A. , & Gorwood, P. (2020). The consequences of the COVID‐19 pandemic on mental health and implications for clinical practice. European Psychiatry, 63(1), e32. 10.1192/j.eurpsy.2020.35 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Fletcher‐Tomenius, L. , & Vossler, A. (2009). Trust in online therapeutic relationships: The therapist's experience. Counselling Psychology Review, 24(2), 24–34. [Google Scholar]
  28. Gega, L. , Marks, I. , & Mataix‐Cols, D. (2004). Computer‐aided CBT self‐help for anxiety and depressive disorders: Experience of a London clinic and future directions. Journal of Clinical Psychology, 60(2), 147–157. 10.1002/jclp.10241 [DOI] [PubMed] [Google Scholar]
  29. Glueckauf, R. L. , Maheu, M. M. , Drude, K. P. , Wells, B. A. , Wang, Y. , Gustafson, D. J. , & Nelson, E.‐L. (2018). Survey of psychologists' telebehavioral health practices: Technology use, ethical issues, and training needs. Professional Psychology: Research and Practice, 49(3), 205–219. 10.1037/pro0000188 [DOI] [Google Scholar]
  30. Gomes, M. , Gutacker, N. , Bojke, C. , & Street, A. (2015). Addressing missing data in Patient‐Reported Outcome Measures (PROMS): Implications for the use of PROMS for comparing provider performance. Health Economics, 25(5), 515–528. 10.1002/hec.3173 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Goodman‐Deane, J. , Mieczakowski, A. , Johnson, D. , Goldhaber, T. , & Clarkson, P. J. (2016). The impact of communication technologies on life and relationship satisfaction. Computers in Human Behavior, 57, 219–229. 10.1016/j.chb.2015.11.053 [DOI] [Google Scholar]
  32. Humer, E. , Stippl, P. , Pieh, C. , Pryss, R. , & Probst, T. (2020). Experiences of psychotherapists with remote psychotherapy during the COVID‐19 pandemic: Cross‐sectional web‐based survey study. Journal of Medical Internet Research, 22(11), e20246. 10.2196/20246 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Jacobson, N. S. , & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19. 10.1037//0022-006x.59.1.12 [DOI] [PubMed] [Google Scholar]
  34. Karyotaki, E. , Ebert, D. D. , Donkin, L. , Riper, H. , Twisk, J. , Burger, S. , Rozental, A. , Lange, A. , Williams, A. D. , Zarski, A. C. , Geraedts, A. , van Straten, A. , Kleiboer, A. , Meyer, B. , Ünlü Ince, B. B. , Buntrock, C. , Lehr, D. , Snoek, F. J. , Andrews, G. , … Cuijpers, P. (2018). Do guided internet‐based interventions result in clinically relevant changes for patients with depression? An individual participant data meta‐analysis. Clinical Psychology Review, 63, 80–92. 10.1016/j.cpr.2018.06.007 [DOI] [PubMed] [Google Scholar]
  35. Knox, S. , & Hill, C. E. (2016). Introduction to a special issue on disclosure and concealment in psychotherapy. Counselling Psychology Quarterly, 29(1), 1–6. 10.1080/09515070.2015.1095156 [DOI] [Google Scholar]
  36. Krouwel, M. , Jolly, K. , & Greenfield, S. (2019). Comparing Skype (video calling) and in‐person qualitative interview modes in a study of people with irritable bowel syndrome—An exploratory comparative analysis. BMC Medical Research Methodology, 19(1), 219. 10.1186/s12874-019-0867-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Lambert, M. J. , Burlingame, G. M. , Umphress, V. J. , Hanssen, N. , Vermeersch, D. , Clause, G. , & Yanchar, S. C. (1996). The reliability and validity of the Outcome Questionnaire. Clinical Psychology & Psychotherapy, 3, 249–258. [DOI] [Google Scholar]
  38. Lange, A. , van de Ven, J.‐P. , Schrieken, B. , & Emmelkamp, P. M. G. (2001). Interapy. Treatment of posttraumatic stress through the internet: A controlled trial. Journal of Behavior Therapy and Experimental Psychiatry, 32(2), 73–90. 10.1016/S0005-7916(01)00023-4 [DOI] [PubMed] [Google Scholar]
  39. March, S. , Day, J. , Ritchie, G. , Rowe, A. , Gough, J. , Hall, T. , Yuen, C. Y. J. , Donovan, C. L. , & Ireland, M. (2018). Attitudes toward e‐mental health services in a community sample of adults: Online survey. Journal of Medical Internet Research, 20(2), e59. 10.2196/jmir.9109 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Meurk, C. , Leung, J. , Hall, W. , Head, B. W. , & Whiteford, H. (2016). Establishing and governing e‐mental health care in Australia: A systematic review of challenges and a call for policy‐focussed research. Journal of Medical Internet Research, 18(1), e10. 10.2196/jmir.4827 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Mohr, D. C. , Duffecy, J. , Jin, L. , Ludman, E. J. , Lewis, A. , Begale, M. , & McCarthy, M. Jr. (2010). Multimodal e‐mental health treatment for depression: A feasibility trial. Journal of Medical Internet Research, 12(5), e48. 10.2196/jmir.1370 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Paganini, S. , Teigelkötter, W. , Buntrock, C. , & Baumeister, H. (2018). Economic evaluations of internet‐ and mobile‐based interventions for the treatment and prevention of depression: A systematic review. Journal of Affective Disorders, 225, 733–755. 10.1016/j.jad.2017.07.018 [DOI] [PubMed] [Google Scholar]
  43. Pierce, B. S. , Perrin, P. B. , & McDonald, S. D. (2020). Demographic, organizational, and clinical practice predictors of U.S. psychologists' use of telepsychology. Professional Psychology: Research and Practice, 51(2), 184–193. 10.1037/pro0000267 [DOI] [Google Scholar]
  44. Pierce, B. S. , Perrin, P. B. , Tyler, C. M. , McKee, G. B. , & Watson, J. D. (2021). The COVID‐19 telepsychology revolution: A national study of pandemic‐based changes in US mental health care delivery. American Psychologist, 76(1), 14–25. 10.1037/amp0000722 [DOI] [PubMed] [Google Scholar]
  45. Probst, T. , Haid, B. , Schimböck, W. , Reisinger, A. , Gasser, M. , Eichberger‐Heckmann, H. , Stippl, P. , Jesser, A. , Humer, E. , Korecka, N. , & Pieh, C. (2021). Therapeutic interventions in in‐person and remote psychotherapy: Survey with psychotherapists and patients experiencing in‐person and remote psychotherapy during COVID‐19. Clinical Psychology & Psychotherapy, 28(4), 988–1000. 10.1002/cpp.2553 [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Probst, T. , Stippl, P. , & Pieh, C. (2020). Changes in provision of psychotherapy in the early weeks of the COVID‐19 lockdown in Austria. International Journal of Environmental Research and Public Health, 17(11), 3815. 10.3390/ijerph17113815 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Richardson, L. K. , Frueh, B. C. , Grubaugh, A. L. , Egede, L. , & Elhai, J. D. (2009). Current directions in videoconferencing tele‐mental health research. Clinical Psychology: Science and Practice, 16, 323–338. 10.1111/j.1468-2850.2009.01170.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Robillard, R. , Daros, A. R. , Phillips, J. L. , Porteous, M. , Saad, M. , Pennestri, M.‐H. , Kendzerska, T. , Edwards, J. D. , Solomonova, E. , Bhatla, R. , Godbout, R. , Kaminsky, Z. , Boafo, A. , & Quilty, L. C. (2021). Emerging new psychiatric symptoms and the worsening of pre‐existing mental disorders during the COVID‐19 pandemic: A Canadian multisite study. The Canadian Journal of Psychiatry, 66(9), 815–826. 10.1177/0706743720986786 [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Schopp, L. H. , Demiris, G. , & Glueckauf, R. L. (2006). Rural backwaters or front‐runners? Rural telehealth in the vanguard of psychology practice. Professional Psychology: Research and Practice, 37(2), 165–173. 10.1037/0735-7028.37.2.165 [DOI] [Google Scholar]
  50. Schuster, R. , Pokorny, R. , Berger, T. , Topooco, N. , & Laireiter, A.‐R. (2018). The advantages and disadvantages of online and blended therapy: Survey study amongst licensed psychotherapists in Austria. Journal of Medical Internet Research, 20(12), e11007. 10.2196/11007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Tierney, J. F. , & Stewart, L. A. (2005). Investigating patient exclusion bias in meta‐analysis. International Journal of Epidemiology, 34(1), 79–87. 10.1093/ije/dyh300 [DOI] [PubMed] [Google Scholar]
  52. Valentine, L. M. , Donofry, S. D. , Broman, R. B. , Smith, E. R. , Rauch, S. A. , & Sexton, M. B. (2020). Comparing PTSD treatment retention among survivors of military sexual trauma utilizing clinical video technology and in‐person approaches. Journal of Telemedicine and Telecare, 26(7–8), 443–451. 10.1177/1357633x19832419 [DOI] [PubMed] [Google Scholar]
  53. van Mens, K. , Lokkerbol, J. , Janssen, R. , van Orden, M. L. , Kloos, M. , & Tiemens, B. (2017). A cost‐effectiveness analysis to evaluate a system change in mental healthcare in the Netherlands for patients with depression or anxiety. Administration and Policy in Mental Health and Mental Health Services Research, 45(4), 530–537. 10.1007/s10488-017-0842-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Vis, C. , Mol, M. , Kleiboer, A. , Bührmann, L. , Finch, T. , Smit, J. , & Riper, H. (2018). Improving implementation of eMental health for mood disorders in routine practice: Systematic review of barriers and facilitating factors. JMIR Mental Health, 5(1), e20. 10.2196/mental.9769 [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Wind, T. R. , Rijkeboer, M. , Andersson, G. , & Riper, H. (2020). The COVID‐19 pandemic: The ‘black swan’ for mental health care and a turning point for e‐health. Internet Interventions, 20, 100317. 10.1016/j.invent.2020.100317 [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Wosik, J. , Fudim, M. , Cameron, B. , Gellad, Z. F. , Cho, A. , Phinney, D. , Curtis, S. , Roman, M. , Poon, E. G. , Ferranti, J. , Katz, J. N. , & Tcheng, J. (2020). Telehealth transformation: COVID‐19 and the rise of virtual care. Journal of the American Medical Informatics Association, 27(6), 957–962. 10.1093/jamia/ocaa067 [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Wu, Y.‐C. , Chen, C.‐S. , & Chan, Y.‐J. (2020). The outbreak of COVID‐19: An overview. Journal of the Chinese Medical Association, 83(3), 217–220. 10.1097/JCMA.0000000000000270 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Data obtained for the study will be made available upon request.

Articles from Clinical Psychology & Psychotherapy are provided here courtesy of Wiley

RESOURCES