Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2025 Jul 21.
Published in final edited form as: Psychother Res. 2024 Jul 21;35(6):990–1000. doi: 10.1080/10503307.2024.2375231

A cohort study of engagement in telehealth psychotherapy versus in-person services

MARY BETH CONNOLLY GIBBONS 1, LANG DUONG 1, RACHEL Y CHIU 1, PAUL CRITS-CHRISTOPH 1, ROBERT GALLOP 2, DAVID MANDELL 1, OLGA BARG 1, CORY F NEWMAN 1, LILY A BROWN 1, MARIA A OQUENDO 1
PMCID: PMC11746968  NIHMSID: NIHMS2034869  PMID: 39034438

Abstract

Objective

Although telehealth psychotherapies have been studied for over 20 years, mental health services remained largely delivered in person until the COVID-19 pandemic forced clinics to reconsider the utility of telehealth psychotherapy. This study aims to compare patient engagement in in-person versus telehealth services in outpatient psychotherapy for mood and anxiety disorders.

Method:

A cohort investigation was conducted, using a propensity score matched sample, extracted from an electronic health record (EHR) to compare engagement in psychotherapy for 762 patients who used in-person services before the pandemic to a cohort of 762 patients who used telehealth psychotherapy after the onset of COVID-19. The authors compared cohorts on initial engagement in psychotherapy services following an initial intake, number of psychotherapy sessions attended, and the rate of missed sessions.

Results:

There was a 26% increase in the total number of individual psychotherapy sessions attended when the clinics transitioned to telehealth services (p < .001). In addition, patients who received telehealth psychotherapy were five times more likely to not cancel or miss any scheduled sessions (p < .001).

Conclusion:

These results indicate that telehealth services may result in improved treatment engagement for outpatient centers focused on brief evidence-based psychotherapies for mood and anxiety disorders.

Keywords: telehealth, psychotherapy, service engagement, service utilization

Early attrition from psychotherapy, prior to receiving the full course of treatment agreed upon with the provider and obtaining optimal clinical benefit, has been an intractable problem that influences treatment outcomes and efficient clinic functioning (Barrett et al., 2008; Olfson et al., 2009; Swift et al., 2017). An estimated 19% of individuals receiving mental health services in a nationally representative sample (Olfson et al., 2009) and 22% of individuals receiving psychotherapy as part of randomized controlled trials (Swift et al., 2017) terminate psychotherapy prematurely. People in outpatient mental health treatment receive a median of only seven psychotherapy sessions, less than is traditionally considered an adequate dose of most evidence-based psychotherapies for mood and anxiety disorders. In fact, 70% of patients who discontinue psychotherapy early attend only one or two sessions (Olfson et al., 2009). Early attrition from psychotherapy has been associated with poor treatment outcomes (Lambert, 2017), especially for patients who withdraw very early from treatment (Pekarik, 1992).

Although most evidence-based psychotherapies for mental disorders have traditionally been delivered in person, efforts have been made over the past two decades to improve patient engagement in services by exploring the feasibility and effectiveness of remotely delivered psychotherapies. Synchronous telehealth psychotherapies, including both telephone and video conference-based psychotherapies, have been designed to address obstacles to in-person services (Andrade et al., 2014; Hoge et al., 2004). Telehealth treatments may allow patients to conquer structural barriers like a need for transportation, childcare responsibilities, and difficulties obtaining time off (Hoge et al., 2004). Telehealth interventions also provide flexible solutions for individuals who live a distance from mental health centers or who need to travel for their work (Mohammadi et al., 2020; Zinzow et al., 2012). In addition to these practical barriers to care, telehealth psychotherapy may help individuals concerned with stigma engage in psychotherapy (Hoge et al., 2004; Mohammadi et al., 2020; Zinzow et al., 2012). Patients may feel that engagement in mental health services may change how others perceive them and may even harm their career trajectory (Hoge et al., 2004). Telehealth treatments provide flexible and private care that eliminates the perceived stigma of being physically present in mental health clinics (Mohammadi et al., 2020; Zinzow et al., 2012).

The COVID-19 pandemic led to a renewed focus on using telehealth services to increase access to care (Betancourt et al., 2020; Centers for Disease Control and Prevention, 2020). In-person sessions for mental health care decreased by 40%, and telehealth sessions increased ten-fold following the onset of the COVID-19 pandemic (Cantor et al., 2023). A recent literature review indicated that clinical outcomes, patient satisfaction, and the therapeutic alliance did not differ between synchronous telehealth-delivered behavioral health interventions and in-person psychotherapies (Bellanti et al., 2022). In addition, the authors reported that treatment utilization was similar across telehealth and in-person psychotherapies. However, the studies reviewed included a wide range of specific diagnostic groups, including eating disorders, traumatic brain injury, homebound older adults, individuals living with HIV, and post-traumatic stress disorder in veteran populations. Only one trial was conducted in a general outpatient psychiatry clinic and this trial focused only on engagement in psychiatric consultation for individuals with a range of depressive and anxiety disorders (De Las Cuevas et al., 2006). This study, like many of the trials reviewed, operationalized utilization simply as a dichotomy comparing completers to non-completers in time-limited treatments within randomized controlled studies. A more comprehensive evaluation of psychotherapy use for broad patient populations in naturalistic outpatient settings is needed to inform decisions regarding the continued implementation of telehealth services post-pandemic.

The current investigation used the broader term engagement to encompass a range of clinically relevant psychotherapy utilization indicators, including initial engagement in services following treatment intake, the number of sessions attended, and the rates of missed or canceled appointments. We conducted a cohort study comparing psychotherapy engagement for in-person services delivered for two years prior to the onset of the COVID-19 pandemic compared to telehealth services provided for two years after the transition to telehealth interventions. Data was collected from two outpatient mental health centers at a large urban medical center that deliver evidence-based cognitive behavioral psychotherapies for individuals with mood and anxiety disorders.

Although the number of sessions attended may be an important indicator of engagement, it should be noted that it may not be a good indicator of response for all patients. The dose effect model (Howard et al., 1986) predicts that more sessions of psychotherapy will lead to better treatment outcomes due to greater exposure to the active ingredients of psychotherapy. Howard et al. (1986) demonstrated a linear dose-effect relationship in a review of 114 studies with 75% of patients showing measurable improvement by 26 sessions. The authors acknowledge that this doesn’t necessarily mean that these patients achieved optimal benefit by 26 sessions and interestingly there were diminishing returns for additional sessions beyond 26. In contrast, some have argued that more sessions of psychotherapy are not always associated with continued improvements. The good-enough level model of therapeutic change (GEL model; Barkham et al., 2006) predicts that the number of sessions will not be related to treatment outcome because some patients have a higher rate of change requiring fewer sessions and other patients have a lower rate of change requiring relatively more sessions of psychotherapy to achieve a good enough level of improvement. While multiple studies support the good enough level model (Barkham et al., 2006; Reese et al., 2011), multiple investigations demonstrated that the rate of improvement was related to the number of sessions but support the GEL model by demonstrating that the dose–response curve was not negatively accelerating (Baldwin et al., 2009; Falkenström et al., 2016).

The literature on telehealth versus in-person care further suggests that neither modality meets the needs of all patients. One study demonstrated higher discontinuation rates from telehealth therapy for patients with post-traumatic stress disorder (Morland et al., 2020), while another showed higher discontinuation rates for in-person psychotherapy for patients with depression (Mohr et al., 2012). Beyond diagnostic groups, few other baseline predictors of psychotherapy engagement have been evaluated. There is, however, accumulating evidence that patients with higher symptom severity, comorbidity (Stiles-Shields et al., 2015), hopelessness (Luxton et al., 2016), and greater age (Smolenski et al., 2017) have poorer outcomes in telehealth psychotherapy, suggesting that demographic and clinical characteristics should be explored to see how treatment formats might be personalized to optimize engagement in services.

Objective

Our primary aim was to compare engagement for psychotherapy services delivered via telehealth to services delivered in-person. We hypothesized that telehealth services, by addressing barriers to service utilization, would result in improved engagement relative to in-person services. Our secondary aim was to explore demographic, clinical, and social vulnerability variables as predictors and moderators of psychotherapy engagement across telehealth and in-person services to inform the personalization and adaptation of services.

Method

Overview

We conducted a retrospective cohort study using de-identified data extracted from Penn Medicine’s electronic health record (EHR). The University of Pennsylvania Institutional Review Board deemed this study exempt and waived informed consent because data were de-identified. To ensure the quality and accuracy of EHR data (Greiver et al., 2012; McGinn et al., 2011; Price et al., 2012), we implemented a comprehensive analysis of data accuracy prior to hypothesis testing (Dziadkowiec et al., 2017; Kahn et al., 2012).

Samples and Treatment Services

Participants received services from one of two outpatient mental health clinics that offer cognitive–behavioral therapies by highly trained psychologists and psychology trainees (masters or doctoral level clinicians). Clinic 1 primarily treats posttraumatic stress disorder, obsessive-compulsive disorder, social anxiety disorder, panic disorder, agoraphobia, generalized anxiety disorder, and specific phobias using evidence-based cognitive–behavioral treatment protocols designed to be implemented across 12–20 sessions depending on the diagnosis and case complexity. Clinic 2 offers services for a range of mood and anxiety disorders using protocols that are active, directive, and focused on skill building. Although most treatments are designed and delivered in a short-term format, the patient and therapist work collaboratively to decide the optimal treatment length.

Cohorts

Before the onset of the COVID-19 pandemic, almost all psychotherapy sessions at these clinics were provided in-person. Telehealth services were provided only for rare emergencies. In March 2020, both clinics rapidly transitioned to synchronous telehealth services, including both telephone-based and video sessions. We defined two retrospective cohorts. Cohort 1 included all patients scheduled for in-person services during the two years before the pandemic during 2018 and 2019. Patient who received an intake session between January of 2018 and June 2019 were included to allow adequate time for completion of six months of treatment. Cohort 2 included patients who received telehealth services during the two years after the onset of the pandemic, defined as July 2020 through July 2022. In Cohort 2, all treatments were delivered predominantly via telehealth, although clinicians did have the option of providing in person sessions occasionally. Those who received a new patient visit between July 2020 and December 2022 were included to allow adequate time to complete six months of treatment. We did not include patients who received services from January 2020 through June 2020 since this was a time in which engagement was likely strongly influenced by the spread of COVID-19 and the sudden transition of care to telehealth services.

Measures of Psychotherapy Engagement

Psychotherapy engagement was estimated by counting the total number of individual psychotherapy sessions attended and calculating the rate of missed sessions. For the count of sessions, we examined both whether or not each patient attended at least one psychotherapy session following the intake appointment to represent initial engagement in treatment and the total number of individual psychotherapy sessions attended following the initial intake session to represent continuity of care. The rate of missed sessions was calculated using the formula for missed opportunity rate proposed by Jacobs et al. (2019) that divides the number of missed individual psychotherapy sessions (either through patient cancelation or no-show) by the total number of visits scheduled. We examined both whether or not each patient missed any scheduled sessions and the missed opportunity rate across patients who missed at least one session.

Predictors and Moderators of Engagement

Demographic Variables.

We examined age at treatment intake, legal sex, race, Hispanic ethnicity, and marital status. Demographic categorical variables were dichotomized for use in predictive analyses. Race was dichotomized as minority versus non-Hispanic White. Secondary analyses examined engagement for the subsets of Black and Asian patients versus White. Marital status was dichotomized as currently married or cohabitating versus not.

Clinical Characteristics.

Psychiatric comorbidity was represented by the number of psychiatric diagnoses at the intake and medical comorbidity was the number of medical diagnoses documented in the EHR during the cohort period. We further evaluated the presence versus absence of a diagnosis of depressive disorder, anxiety disorder, personality disorder, or substance/alcohol use disorder at intake.

Social Vulnerability Indices.

We included four social vulnerability variables that were derived from geocoded zip codes as well as two variables that were automatically calculated within the EHR based on service use. The four variables based on zip code included the percentile rankings for socioeconomic status, household composition and disability, minority status and English language proficiency, and housing type and transportation (Centers for Disease Control and Prevention / Agency for Toxic Substances and Disease Registry, 2022). Socioeconomic status included ratings of poverty level, employment, income, and education. The household composition and disability ratings included age and disability status. The minority and language ranking was based on minority status and ability to speak English. The housing type and transportation ranking included evaluation of the type of housing, crowding in housing, and access to a vehicle. For these social vulnerability indices, census tracts were ranked across the United States. We used the percentile ranking values that ranged from 0 to 1 with higher scores indicating greater vulnerability. The preventative care gap score was calculated by adding the number of preventative care screenings an individual qualified for but did not complete. The general adult risk score was calculated based on factors that contribute to an increased risk of adverse health events.

Data Analysis

Data was extracted from the EHR for 1806 patients, including 1022 in cohort 1 and 784 in cohort 2, on 4 April 2023. All patients in the first cohort received an in-person intake and individual psychotherapy sessions. Following the onset of COVID-19, all treatments were offered and initiated via telehealth. Clinicians provided predominantly telehealth sessions but were able to offer occasional sessions in-person if approved by the administration. Cohort 2 includes cases that received predominantly telehealth treatment, allowing for occasional in-person sessions, representing how telehealth is implemented in a naturalistic sample. Ninety-seven percent of patients in cohort 2 (n = 762) received at least 80% of their sessions via telehealth and were retained for primary analyses.

Propensity Score Matching of Cohorts.

We used propensity score matching across the 17 variables listed in Table I to mitigate differences in the cohorts. We log-transformed two non-normal variables, the number of concurrent psychiatric diagnoses and the number of medical diagnoses. Logistic regressions for categorical variables and linear regressions for scale variables estimated missing values for eight intake variables from the other intake variables across 40 imputations (Azur et al., 2011; Graham et al., 2007). We followed the guidance of Graham et al. (2007) who suggests 40 imputations to improve power. Final estimates for variables with missing values were computed as the average predicted value across the 40 imputations.

Table I.

Patient demographic and clinical characteristics by clinic by cohort at the new patient visit in a matched dataset.

In-person cohort (n = 762) Telehealth cohort (n = 762)
Characteristic N available
(N = 1524)		 Clinic 1
(n = 389)		 Clinic 2
(n = 373)		 Clinic 1
(n = 421)		 Clinic 2
(n = 341)
Demographic Characteristics
 Gender, n (%) 1524
  Male 151 (38.8) 124 (33.2) 152 (36.1) 121 (35.5)
  Female 238 (61.2) 249 (66.8) 269 (63.9) 220 (64.5)
 Race dichotomized, n (%) 1224
  Minority 73 (20.6) 48 (18.7) 78 (21.1) 43 (17.6)
  White 282 (79.4) 209 (81.3) 292 (78.9) 202 (82.4)
 Ethnicity dichotomized, n (%) 1437
  Hispanic/Latino 29 (7.6) 9 (2.6) 23 (5.7) 11 (3.7)
  Not Hispanic/Latino 353 (92.4) 342 (97.4) 383 (94.3) 287 (96.3)
 Marital status dichotomized, n (%) 1524
  Married/cohabitating 123 (31.6) 79 (21.2) 138 (32.8) 72 (21.1)
  Not married/cohabitating 266 (68.4) 294 (78.8) 283 (67.2) 269 (78.9)
 Age, mean (SD) 1524 33.1 (11.7) 30.2 (13.4) 33.6 (11.7) 29.79 (12.6)
Clinical characteristics
 Number of DSM-V diagnoses, mean (SD) 1524 1.04 (0.55) 1.38 (0.70) 1.11 (0.67) 1.38 (0.73)
 Number of Medical diagnoses, mean (SD) 1524 0.07 (.30) 0.02 (.15) 0.06 (0.27) 0.03 (0.21)
 Depression diagnosis, n (%) 1524 229 (58.9) 67 (18.0) 250 (59.4) 53 (15.5)
 Anxiety diagnosis, n (%) 1524 126 (32.4) 341 (91.4) 142 (33.7) 310 (90.9)
 Substance use diagnosis, n (%) 1524 15 (3.9) 1 (.3) 16 (3.8) 6 (1.8)
 Personality disorder diagnosis, n (%) 1524 26 (6.7) 1 (.3) 48 (11.4) 2 (.6)
Social vulnerability indices
 General adult risk score, mean (SD) 1210 0.99 (1.24) 0.85 (1.02) 0.94 (1.13) 0.75 (1.20)
 Prevention care gap score, mean (SD) 1252 3.41 (1.50) 3.26 (1.36) 3.26 (1.44) 3.33 (1.34)
 Socioeconomic theme summary percentile ranking, mean (SD) 1518 .47 (.35) .36 (.33) .42 (.34) .39 (.34)
 Household composition theme summary percentile ranking, mean (SD) 1518 .26 (.27) .24 (.24) .23 (.26) .27 (.27)
 Minority status language theme summary percentile ranking, mean (SD) 1518 .74 (.14) .70 (.17) .73 (.14) .71 (.16)
 Housing type transportation theme summary percentile ranking, mean (SD) 1518 .68 (.25) .62 (.28) .67 (.26) .62 (.27)
Open in a new tab

Propensity scores were calculated using a logistic regression model with a logit link function predicting cohort from the 17 baseline variables listed in Table I. A greedy nearest neighbor matching without replacement within a specified caliper width of 0.20 was used to match patients from the in-person cohort with each of the 762 patients in the telehealth cohort (Austin, 2014). Following the recommendations of Austin (2014), the nearest neighbor matching within a specified caliper was selected as this method demonstrates less biased estimates. The caliper was set at.20 following the recommendations of Austin (2011). This method applied without replacement performed at least as well as other propensity matching algorithms and was recommended by Austin (2014).

Balance was examined following the recommendations of Austin (2011) using the MatchIt version 4.5.5 package in R (Ho et al., 2011). First standardized mean differences were examined for each of the 17 potential covariates. In the unmatched sample, standardized mean differences ranged from .00 to .23 indicating some treatment group selection bias. In the matched sample, the 17 standardized mean differences ranged from 0 to .09, all below the .1 cutoff defining optimal balance (Austin, 2011), indicating that the matched sample eliminated observed systematic differences between cohorts. The distribution of baseline covariates across cohorts was examined based on density plots, which confirmed that matching eliminated differences across cohorts across the full distribution of each covariate. Finally, we examined quintile side by side boxplots of the propensity scores per cohort, confirming balance in the final propensity score across the full distribution.

Evaluation of Engagement.

The analytic framework included zero-inflated negative binomial (ZINB) models to address the high proportion of zero counts, confirmed as the approach that would provide a superior fit using the test developed by Vuong (1989). Differences between cohorts are described using odds ratios (OR) for the zero versus nonzero component and rate ratios (RR) for the count portion of the zero-inflated models. All analyses included terms for cohort, clinic, and the interaction of cohort by clinic in the prediction of engagement indices. Analyses did not include a term for therapist as therapists were partially nested and partially crossed with cohort and there were a high number of therapists who saw a single patient.

In the in-person cohort, 16% (n = 159) of patients continued sessions through the acute onset of COVID-19 during the transition from in-person to telehealth services. For the telehealth cohort, we obtained only data up until the end of the cohort in June 2022. As a proxy for who may have continued sessions beyond the cohort, we identified 17% of patients in the telehealth cohort (n = 136) who attended at least one session in June of 2022 who likely were still engaged in services after the cohort window. Primary analyses included the full matched sample while sensitivity analyses evaluated the subsample of patients known to have completed services within the respective cohort windows.

Predictors and Moderators of Engagement.

To control for type I error in the exploratory analyses evaluating predictors and moderators of engagement, we used a p-value of .003 (.05 adjusted for 17 potential predictor variables) for analyses of predictors. For each predictor, a ZINB model included clinic, cohort, and the predictor with a separate model including the predictor by cohort interaction to evaluate moderation.

Results

Characteristics of Study Population

The majority of patients were female (64.0%), white (80.5%), non-Hispanic (95.0%), not married or cohabitating (73.0%), with an average age of 32 (SD = 12.42) years (see Table I). Patients received the following diagnoses: depressive disorder 39.3%, anxiety disorder 60.3%, substance/alcohol use disorder diagnosis 2.5%, and personality disorder 5.1% (see Table I).

Analysis of Engagement in Psychotherapy Services

The results of the ZINB regressions for the prediction of the total number of individual psychotherapy sessions and missed opportunity rate are presented in Table II. There were no significant interactions between cohort and clinic in the prediction of engagement indices. There was no significant difference between cohorts in early engagement in psychotherapy following the initial intake. Between 9 and 16% of patients dropped out of treatment after the initial intake across clinics and cohorts. There was a statistically significant 26% increase in the total number of individual sessions attended for the sample that attended at least one session for the telehealth cohort compared to the in-person cohort (z = 4.85, p < .001; RR = 1.26 [1.20, 1.32]). Patients in the in-person cohort attended on average 14.40 (SD = 12.61) sessions while patients in the telehealth cohort attended on average 17.77 (SD = 14.10) sessions.

Table II.

Engagement in psychotherapy by cohort by clinic across in-person and telehealth services in the propensity score matched sample.

In-person cohort Telehealth cohort
Engagement index Clinic 1 Clinic 2 Clinic 1 Clinic 2 z OR/RR 95% CI (LL–UL) p
Number of sessions
 Any, n (%) 328 (84.3) 312 (83.6) 382 (90.7) 287 (84.2)
 Cohort −1.59 .76 0.63–0.90 .112
 Clinic 2.15 1.46 1.22–1.74 .032
 Cohort*clinic 1.75 1.87 1.30–2.67 .081
 Count, mean (SD) 13.35 (12.44) 15.50 (12.72) 17.59 (14.72) 18.01 (13.23)
 Cohort 4.85 1.26 1.20–1.32 <.001
 Clinic 1.89 1.09 1.04–1.15 .059
 Cohort*clinic −1.40 .88 0.96–1.18 .163
Missed opportunity rate
 Any, n (%) 76 (19.5) 38 (10.2) 16 (3.8) 10 (2.9)
 Cohort 7.25 5.14 4.10–6.45 <.001
 Clinic 3.40 1.92 1.58–2.32 <.001
 Cohort*clinic −0.99 .63 0.40–1.01 .325
 Count, mean (SD) .14 (.11) .10 (.08) .09 (.08) .12 (.14)
 Cohort −1.04 .82 0.68–0.99 .299
 Clinic −1.49 .79 0.68–0.93 .136
 Cohort*clinic 1.68 1.90 1.30–2.79 .092
Open in a new tab

The majority of missed sessions (91%) were sessions that the patient scheduled but did not attend without notifying the clinic prior to the appointment, while the remaining percentage of missed sessions were due to advance cancelations from either the patient or the clinician. The odds of having no missed opportunities was five times higher for the telehealth cohort compared to the in-person cohort (z = 7.25, p < .001; OR = 5.14 [4.10, 6.45]). There were no significant effects for the count component predicting missed opportunity rate. Sensitivity analyses confirmed that all significant effects obtained for the full matched sample were also evident in the subset who were known to have completed treatment.

Predictors and Moderators of Psychotherapy Engagement

The results for predictors and moderators of psychotherapy engagement can be found in Table III. At a corrected p value of .003, patients with more medical comorbidities were twice as likely to attend no further sessions after the initial intake regardless of whether the service was provided via in-person or telehealth (z = 3.04, p = .002; OR = 2.24 [1.72, 2.92]). Patients who attended at least one psychotherapy sessions for a depressive diagnosis attended 18% more sessions than those without a depressive diagnosis (z = 3.10, p = .002; RR = 1.18 [1.12, 1.25]) and patients with anxiety disorders were 63% more likely to attend one or more psychotherapy sessions following the new patient visit (z = −4.11, p < .001; OR = 0.37 [0.28, 0.47]) regardless of treatment format.

Table III.

Zero-inflated negative binomial analyses of predictors and moderators of engagement in psychotherapy in the propensity score matched sample.

Total individual sessions Missed opportunity rate
Zero vs. non-zero Count Zero vs. Non-Zero Count
Variables z p z p z p z p
Demographic variables
 Gender 1.39 .164 1.25 .213 2.33 .020 −1.09 .275
 Race (dichotomized) 1.89 .059 1.70 .089 1.27 .204 −1.93 .054
 Ethnicity (dichotomized) −1.21 .227 −.19 .849 −1.26 .208 1.65 .098
 Marital status (dichotomized) −1.86 .063 −1.59 .113 1.29 .196 −0.48 .633
 Age −2.01 .044 −2.71 .007 −0.35 .724 −0.69 .488
 Cohort*gender −1.11 .266 0.74 .457 −0.55 .585 −1.90 .058
 Cohort*race (dichotomized) 0.84 .399 0.05 .961 0.20 .845 0.20 .845
 Cohort*ethnicity (dichotomized) 0.17 .863 0.39 .699 0.79 .427 −0.15 .881
 Cohort*marital status (dichotomized) −.48 .634 2.63 .009 1.10 .270 −0.84 .399
 Cohort*age 2.63 .009 2.30 .022 −0.58 .561 −3.02 .003
Clinical variables
 Number of psychiatric diagnoses −1.11 .266 1.56 .119 −1.19 .235 −0.50 .616
 Number of medical diagnoses 3.04 .002 −2.73 .006 −0.07 .943 −1.32 .188
 Any depressive diagnosis 2.42 .016 3.10 .002 −1.99 .047 1.48 .138
 Any anxiety diagnosis −4.11 .001 −0.15 .878 −0.09 .928 −1.38 .167
 Any substance use diagnosis 0.63 .532 0.34 .736 0.67 .506 −2.12 .034
 Any personality disorder diagnosis −1.31 .189 1.50 .134 −1.57 .116 0.46 .645
 Cohort*number of psychiatric diagnoses 0.13 .894 0.25 .799 −1.73 .083 −0.36 .720
 Cohort*number of medical diagnoses −1.51 .131 −0.11 .914 −0.57 .570 0.29 .775
 Cohort*any depressive diagnosis −0.38 .707 2.69 .007 −0.03 .979 −2.36 .018
 Cohort*any anxiety diagnosis 1.79 .074 −2.26 .024 −1.16 .246 1.48 .139
 Cohort*any substance use diagnosis −1.54 .124 0.07 .945 0.31 .753 –a –a
 Cohort*any personality disorder diagnosis −0.77 .439 0.32 .751 0.58 .561 −2.48 .013
Social vulnerability index variables
 Preventative care gap score 2.15 .032 −0.42 .674 −1.42 .156 −0.26 .793
 General adult score −0.03 .974 −1.36 .173 −0.67 .503 1.55 .120
 Socioeconomic theme percentile −0.97 .332 0.40 .690 0.17 .867 0.32 .747
 Housing composition theme percentile −0.96 .336 −0.37 .710 −0.94 .346 −0.66 .506
 Minority status theme percentile 0.23 .816 −0.28 .779 0.90 .367 2.23 .026
 Housing type and transportation theme percentile −0.79 .428 −0.13 .896 1.80 .071 0.68 .500
 Cohort*preventative care gap score −0.09 .929 −0.45 .650 1.83 .067 −1.31 .189
 Cohort*general adult score 1.05 .294 1.51 .130 0.43 .669 0.71 .477
 Cohort*socioeconomic theme percentile −1.80 .072 −0.36 .723 −0.56 .575 −2.52 .012
 Cohort*housing composition theme percentile −1.29 .196 −0.05 .962 −0.09 .932 −1.39 .164
 Cohort*minority status theme percentile −.1.29 .197 −0.25 .801 −1.54 .124 −1.01 .312
 Cohort*housing type and transportation theme percentile −0.61 .545 −0.56 .573 −1.72 .085 0.41 .681
Open in a new tab
a

The number of non-zero cases is too small to fit to the model.

There was a significant interaction between age and cohort in the prediction of the missed opportunity rate for the count component (z = −3.02, p = .003; RR = 0.95 [0.94, 0.97]) indicating that older patients missed 5% fewer scheduled sessions in telehealth. Using a median split, patients less than 32 years old who missed at least one scheduled psychotherapy session had on average a missed opportunity rate of 11% in both the in-person and telehealth cohorts. In contrast, patients greater than or equal to 32 years old had on average a missed opportunity rate of 14% in the in-person cohort but only 7% in the telehealth cohort.

Finally, there were no significant effects for race evaluating engagement for White patients compared to minority patients. In secondary analyses, Black patients attended 29% fewer sessions than White patients regardless of treatment format (z = −2.98, p = .003; RR = 0.71 [0.63, 0.80]). Black patients attended on average 12.29 (SD = 10.90) sessions while White patients attended 16.57 sessions (SD = 14.04). There were no significant differences between Asian patients and White patients on engagement indices.

Discussion

Our results confirm our hypotheses that the transition to telehealth psychotherapy from primarily in-person services resulted in improved engagement. Although the move to telehealth services did not improve initial engagement following the intake appointment, there was a 26% increase in the number of individual psychotherapy sessions attended and patients were five times more likely to attend all of their scheduled sessions. Canceled appointments and session unexplained absences present a tremendous problem for clinics that are attempting to reduce costs by maintaining high clinician productivity. Missed sessions and sporadic attendance in psychotherapy can also have a significant impact on the process of therapy. Our results suggest that telehealth may improve clinic productivity and could optimize therapeutic progress by engaging patients more consistently in treatment.

Future studies will need to elucidate whether the improved engagement demonstrated in the telehealth format leads to optimized outcomes for all patients. The dose effect model (Howard et al., 1986) suggests that the increased number of sessions attended in the telehealth format will increase patient exposure to the active ingredients of psychotherapy and thus optimize their clinical benefit. Patients in the current sample attended on average 14 sessions in the in-person format compared to 18 sessions in the telehealth format. The dose effect model further predicts a diminishing return for additional sessions beyond 26 sessions, indicating that patients are not improving as much across these additional sessions. It is possible that the telehealth format encourages patients to remain in treatment longer, to the point where the benefit of the additional sessions is minimal. In our sample, telehealth increased average engagement to 18 sessions, well before the diminishing return demonstrated by Howard et al. (1986) at 26 sessions and within the number of sessions recommended for optimal clinical benefit within evidence-based cognitive behavioral interventions for mood and anxiety disorders. The GEL model (Barkham et al., 2006) further suggests that for some patients who have higher average rates of change, fewer sessions would be needed to achieve a good enough level of improvement. It is possible that the telehealth format encouraged even those patients who achieved a good enough level of improvement earlier in treatment to remain in treatment longer than needed.

Our analysis of predictors and moderators of engagement indicates that telehealth psychotherapy may be especially important for relatively older patients who had significantly lower rates of missed sessions in the telehealth cohort compared to the in-person cohort. It may be that relatively older patients have more job and family responsibilities that interfere with traveling to sessions in person. However, medical comorbidity may impact engagement in services regardless of whether the service is provided in-person or via telehealth. Patients with medical comorbidity were twice as likely to not engage in psychotherapy following the initial intake. Although telehealth may break down some practical barriers to engagement like transportation and childcare needs, patients with high medical comorbidity may have medical appointments and physical symptoms that interfere with psychotherapy engagement in either format.

Our results further indicate that race may be an important predictor of psychotherapy engagement regardless of format. Although a broad variable representing minority versus White patients did not significantly predict or moderate engagement, secondary analyses indicated that Black patients attended significantly fewer sessions compared to White patients. Our sample included small sample sizes for specific minority groups precluding a thorough exploration of how racial groups differ in engagement in psychotherapy across in-person and telehealth services. Further research is needed to fully explore barriers to engaging in services across racial groups, including whether patient and clinician match on racial background influences engagement in services.

Our results were inconsistent with previous studies that demonstrated higher discontinuation rates from telehealth psychotherapy for veterans with anxiety disorders (Morland et al., 2020) and higher discontinuation rates from in-person treatments for patients with depression treated in primary care settings (Mohr et al., 2012). In contrast, our results indicate that patients with depressive disorders received more sessions of psychotherapy and patients with anxiety disorder diagnoses were more likely to engage in sessions following an intake regardless of whether the treatment was delivered via telehealth or in-person. This contrast in results indicates that differences in treatment settings and patient populations may influence engagement in services across these treatment formats.

Study limitations include our quasi-experimental design. Without randomization, we cannot confirm that other variables that differentiated our cohorts were not responsible for the cohort effects obtained. However, our use of a propensity-score matched sample reduced potential bias and improved our ability to make causal inferences. Some limitations to propensity score matching have been noted in the literature. King and Nielsen (2019) warned that propensity score matching for well-balanced data prior to implementation may result in imbalance post matching. However, we examined imbalance both prior to and post matching and found that matching improved the balance in our sample. One limitation of using greedy nearest neighbor matching without replacement is the potential for not all participants to be matched within the respective caliper. However, all individuals in our telehealth cohort were matched with someone in the in-person cohort within our prespecified caliper, ensuring that our results generalize to the population receiving telehealth.

Our analyses of predictors and moderators of engagement were limited to the demographic, clinical, and social vulnerability indices available in our EHR. It’s possible that the telehealth format is not optimal for other subgroups of patients that were not identified in our sample. Further studies will be needed to confirm that improved engagement leads to decreased clinic costs and optimal patient benefit. Our results were further limited by an exclusive focus on data available in the EHR. Although we were able to focus on important indices of psychotherapy engagement, we did not have access to ratings of initial symptom severity or change in symptoms across treatment. It will be important for future investigations to explore the effects of telehealth delivered services on engagement, treatment process, and outcome. In addition, we could not distinguish between cancelations initiated by the patient versus cancelations initiated by the clinician. Thus, any differences in the missed opportunity rates between cohorts could have been influenced by clinician reactions to the treatment format. Finally, our results may only generalize to outpatient mental health clinics at a major medical center. It may be that additional practical barriers experienced by vulnerable populations, like access to technology and adequate wireless networks, may negatively impact engagement in telehealth services.

Conclusions

In conclusion, telehealth options may lead to better patient engagement in psychotherapy and may decrease the costs and wait times resulting from high numbers of missed and canceled psychotherapy sessions. Although telehealth utilization remains elevated since the onset of the COVID-19 pandemic, in-person mental health visits are returning to pre-pandemic levels (Cantor et al., 2023). Insurers and policy makers will need to continue to weigh the benefits of telehealth services against a number of factors. Increased engagement in services has the potential for increasing healthcare spending (Mehrotra et al., 2021) which will need to be carefully considered in light of the potential for telehealth treatments to engage patients in longer treatments than might be needed to obtain good enough level outcomes. Broad dissemination of telehealth services in community settings will need to take into account both patient and therapist preferences for in-person versus telehealth care as well as practical barriers to telehealth like access to devices and Wi-Fi in community settings. The emerging literature on engagement in psychotherapy and treatment effectiveness for telehealth services suggests the need for regulatory flexibility to optimize mental health outcomes. Not all patients will benefit from a single treatment format, rather access to both in-person and telehealth psychotherapy may be needed to address stakeholder preferences and optimize outcomes for all patients in all settings.

Clinical or methodological significance of this article:

Patients who were offered telehealth psychotherapy services attended 26% more sessions than those who were offered in-person services in clinics providing short-term evidence-based psychotherapies for mood and anxiety disorders. Telehealth psychotherapy resulted in significantly fewer missed or canceled sessions across short-term evidence-based psychotherapies. Access to telehealth psychotherapy may be especially important for relatively older patients who may face barriers to in-person services.

Footnotes

Disclosure Statement

No potential conflict of interest was reported by the author(s).

References

  1. Andrade LH, Alonso J, Mneimneh Z, Wells JE, Al-Hamzawi A, Borges G, Bromet E, Bruffaerts R, de Girolamo G, de Graaf R, Florescu S, Gureje O, Hinkov HR, Hu C, Huang Y, Hwang I, Jin R, Karam EG, Kovess-Masfety V, & Levinson D (2014). Barriers to mental health treatment: Results from the WHO World Mental Health surveys. Psychological Medicine, 44(6), 1303–1317. 10.1017/S0033291713001943 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Austin PC (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424. 10.1080/00273171.2011.568786 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Austin PC (2014). A comparison of 12 algorithms for matching on the propensity score. Statistics in Medicine, 33(6), 1057–1069. 10.1002/sim.6004 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Azur MJ, Stuart EA, Frangakis C, & Leaf PJ (2011). Multiple imputation by chained equations: What is it and how does it work? International Journal of Methods in Psychiatric Research, 20(1), 40–49. 10.1002/mpr.329 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Baldwin SA, Berkeljon A, Atkins DC, Olsen JA, & Nielsen SL (2009). Rates of change in naturalistic psychotherapy: Contrasting dose–effect and good-enough level models of change. Journal of Consulting and Clinical Psychology, 77(2), 203–211. 10.1037/a0015235 [DOI] [PubMed] [Google Scholar]
  6. Barkham M, Connell J, Stiles WB, Miles JN, Margison F, Evans C, & Mellor-Clark J (2006). Dose-effect relations and responsive regulation of treatment duration: the good enough level. Journal of Consulting and Clinical Psychology, 74(1), 160–167. 10.1037/0022-006X.74.1.160 [DOI] [PubMed] [Google Scholar]
  7. Barrett MS, Chua WJ, Crits-Christoph P, Gibbons MB, & Thompson D (2008). Early withdrawal from mental health treatment: Implications for psychotherapy practice. Psychotherapy: Theory, Research, Practice, Training, 45(2), 247–267. 10.1037/0033-3204.45.2.247 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Bellanti DM, Kelber MS, Workman DE, Beech EH, & Belsher BE (2022). Rapid review on the effectiveness of telehealth interventions for the treatment of behavioral health disorders. Military Medicine, 187(5–6), e577–e588. 10.1093/milmed/usab318 [DOI] [PubMed] [Google Scholar]
  9. Betancourt JA, Rosenberg MA, Zevallos A, Brown JR, & Mileski M (2020). The impact of COVID-19 on telemedicine utilization across multiple service lines in the United States. Healthcare, 8(4), 380. 10.3390/healthcare8040380 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Cantor JH, McBain RK, Ho PC, Bravata DM, & Whaley C (2023). Telehealth and in-person mental health service utilization and spending, 2019 to 2022. JAMA Health Forum, 4(8), e232645–e232645. 10.1001/jamahealthforum.2023.2645 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Centers for Disease Control and Prevention. (2020, June 10). Using telehealth to expand access to essential health services during the COVID-19 pandemic. http://web.archive.org/web/20220105100151/https://www.cdc.gov/coronavirus/2019-ncov/hcp/telehealth.html.
  12. Centers for Disease Control and Prevention / Agency for Toxic Substances and Disease Registry. (2022, October 26). CDC/ATSDR social vulnerability index 2020 database US. https://www.atsdr.cdc.gov/placeandhealth/svi/data_documentation_download.html.
  13. De Las Cuevas C, Arredondo MT, Cabrera MF, Sulzenbacher H, & Meise U (2006). Randomized clinical trial of telepsychiatry through videoconference versus face-to-face conventional psychiatric treatment. Telemedicine and e-Health, 12(3), 341–350. 10.1089/tmj.2006.12.341 [DOI] [PubMed] [Google Scholar]
  14. Dziadkowiec O, Callahan T, Ozkaynak M, Reeder B, & Welton J (2017). Using a data quality framework to clean data extracted from the electronic health record: A case study. eGEMs (Generating Evidence & Methods to improve patient outcomes) (Washington DC), 4(1), 1201. 10.13063/2327-9214.1201 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Falkenström F, Josefsson A, Berggren T, & Holmqvist R (2016). How much therapy is enough? Comparing dose-effect and good-enough models in two different settings. Psychotherapy, 53(1), 130–139. 10.1037/pst0000039 [DOI] [PubMed] [Google Scholar]
  16. Graham JW, Olchowski AE, & Gilreath TD (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8(3), 206–213. 10.1007/s11121-007-0070-9 [DOI] [PubMed] [Google Scholar]
  17. Greiver M, Barnsley J, Glazier RH, Harvey BJ, & Moineddin R (2012). Measuring data reliability for preventive services in electronic medical records. BMC Health Services Research, 12(1), 1–9. 10.1186/1472-6963-12-116 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Ho D, Imai K, King G, & Stuart EA (2011). MatchIt: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42(8), 1–28. 10.18637/jss.v042.i08 [DOI] [Google Scholar]
  19. Hoge CW, Castro CA, Messer SC, McGurk D, Cotting DI, & Koffman RL (2004). Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. New England Journal of Medicine, 351(1), 13–22. 10.1056/NEJMoa040603 [DOI] [PubMed] [Google Scholar]
  20. Howard KI, Kopta SM, Krause MS, & Orlinsky DE (1986). The dose–effect relationship in psychotherapy. American Psychologist, 41(2), 159–164. 10.1037/0003-066X.41.2.159 [DOI] [PubMed] [Google Scholar]
  21. Jacobs JC, Blonigen DM, Kimerling R, Slightam C, Gregory AJ, Gurmessa T, & Zulman DM (2019). Increasing mental health care access, continuity, and efficiency for veterans through telehealth with video tablets. Psychiatric Services, 70(11), 976–982. 10.1176/appi.ps.201900104 [DOI] [PubMed] [Google Scholar]
  22. Kahn MG, Raebel MA, Glanz JM, Riedlinger K, & Steiner JF (2012). A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Medical Care, 50, S21–S29. 10.1097/MLR.0b013e318257dd67 [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. King G, & Nielsen R (2019). Why propensity scores should not be used for matching. Political Analysis, 27(4), 435–454. 10.1017/pan.2019.11 [DOI] [Google Scholar]
  24. Lambert MJ (2017). Maximizing psychotherapy outcome beyond evidence-based medicine. Psychotherapy and Psychosomatics, 86(2), 80–89. 10.1159/000455170 [DOI] [PubMed] [Google Scholar]
  25. Luxton DD, Pruitt LD, Wagner A, Smolenski DJ, Jenkins-Guarnieri MA, & Gahm G (2016). Home-based telebehavioral health for U.S. military personnel and veterans with depression: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 84(11), 923–934. 10.1037/ccp0000135 [DOI] [PubMed] [Google Scholar]
  26. McGinn CA, Grenier S, Duplantie J, Shaw N, Sicotte C, Mathieu L, Mathieu L, Leduc Y, Légaré F, & Gagnon MP (2011). Comparison of user groups’ perspectives of barriers and facilitators to implementing electronic health records: A systematic review. BMC Medicine, 9(1), 1–10. 10.1186/1741-7015-9-46 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Mehrotra A, Bhatia RS, & Snoswell CL (2021). Paying for telemedicine after the pandemic. JAMA, 325(5), 431–432. 10.1001/jama.2020.25706 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Mohammadi R, Tabanejad Z, Abhari S, Honarvar B, Lazem M, Maleki M, & Garavand A (2020). A systematic review of the use of telemedicine in the military forces worldwide. Shiraz E-Medical Journal, 21(11), 1–8. 10.5812/semj.99343 [DOI] [Google Scholar]
  29. Mohr DC, Ho J, Duffecy J, Reifler D, Sokol L, Burns MN, Jin L, & Siddique J (2012). Effect of telephone-administered vs face-to-face cognitive behavioral therapy on adherence to therapy and depression outcomes among primary care patients. JAMA, 307(21), 2278–2285. 10.1001/jama.2012.5588 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Morland LA, Mackintosh MA, Glassman LH, Wells SY, Thorp SR, Rauch SA, Cunningham PB, Tuerk PW, Grubbs KM, Golshan S, Sohn MJ, & Acierno R (2020). Home-based delivery of variable length prolonged exposure therapy: A comparison of clinical efficacy between service modalities. Depression and Anxiety, 37(4), 346–355. 10.1002/da.22979 [DOI] [PubMed] [Google Scholar]
  31. Olfson M, Olfson R, Mojtabai NA, Sampson I, Hwang B, Druss PS, Wang KB, Wells HA, & Pincus RC (2009). Dropout from outpatient mental health care in the United States. Psychiatric Services, 60(7), 898–907. 10.1176/ps.2009.60.7.898 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Pekarik G (1992). Posttreatment adjustment of clients who drop out early vs. late in treatment. Journal of Clinical Psychology, 48(3), 379–387. 10.1002/1097-4679(199205)48:3<379::AID-JCLP2270480317>3.0.CO;2-P [DOI] [PubMed] [Google Scholar]
  33. Price M, Bowen M, Lau F, Kitson N, & Bardal S (2012). Assessing accuracy of an electronic provincial medication repository. BMC Medical Informatics and Decision Making, 12 (1), 1–7. 10.1186/1472-6947-12-42 [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Reese RJ, Toland MD, & Hopkins NB (2011). Replicating and extending the good-enough level model of change: Considering session frequency. Psychotherapy Research, 21(5), 608–619. 10.1080/10503307.2011.598580 [DOI] [PubMed] [Google Scholar]
  35. Smolenski DJ, Pruitt LD, Vuletic S, Luxton DD, & Gahm G (2017). Unobserved heterogeneity in response to treatment for depression through videoconference. Psychiatric Rehabilitation Journal, 40(3), 303–308. 10.1037/prj0000273 [DOI] [PubMed] [Google Scholar]
  36. Stiles-Shields C, Corden ME, Kwasny MJ, Schueller SM, & Mohr DC (2015). Predictors of outcome for telephone and face-to-face administered cognitive behavioral therapy for depression. Psychological Medicine, 45(15), 3205–3215. 10.1017/S0033291715001208 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Swift JK, Greenberg RP, Tompkins KA, & Parkin SR (2017). Treatment refusal and premature termination in psychotherapy, pharmacotherapy, and their combination: A meta-analysis of head-to-head comparisons. Psychotherapy, 54 (1), 47–57. 10.1037/pst0000104 [DOI] [PubMed] [Google Scholar]
  38. Vuong QH (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57(2), 307–333. 10.2307/1912557 [DOI] [Google Scholar]
  39. Zinzow HM, Britt TW, McFadden AC, Burnette CM, & Gillispie S (2012). Connecting active duty and returning veterans to mental health treatment: Interventions and treatment adaptations that may reduce barriers to care. Clinical Psychology Review, 32(8), 741–753. 10.1016/j.cpr.2012.09.002 [DOI] [PubMed] [Google Scholar]

RESOURCES