Virtual Behavioral Health Treatment Satisfaction and Outcomes Across Time

Abstract

Purpose

The COVID-19 pandemic continues to have major and long-lasting impacts on health care delivery and mental health. As health care shifted to telehealth, legislation was adjusted to expand telehealth allowances, creating a unique opportunity to elucidate outcomes. The aim of this study was to assess long-term patient and clinician satisfaction and outcomes with virtual behavioral health.

Methods

Data were obtained over 16 months from surveys to patients and clinicians receiving/providing virtual treatment. Outcomes data also were collected from medical records of adults receiving in-person and virtual behavioral health treatment. Data were summarized using descriptive statistics. Groups were compared using various chi-squared tests for categorical variables, Likert response trends over time, and conditional independence, with Wilcoxon rank-sum or Jonckheere trend test used to assess continuous variables. P-values of ≤0.05 were considered statistically significant.

Results

Patients gave high ratings to virtual treatment and indicated a preference for virtual formats. Both patient and clinician preference for virtual visits increased significantly with time, and many clinicians perceived virtual services to be equally effective to in-person. Virtual programs had higher completion rates, attendance rates, and number of treatment visits, suggesting that virtual behavioral health had equivalent or better outcomes to in-person treatment and that attitudes toward telehealth changed over time.

Conclusions

If trends found in this study continue, telehealth may emerge as a preferred option long term This is important considering the increase in mental health needs associated with the COVID-19 pandemic and the eventuality that in-person restrictions ease as the pandemic subsides.

COVID-19 has significantly impacted the mental health of Americans and prompted major changes in behavioral health utilization.^1,2 Psychological illnesses increased^3,4 — both as direct and indirect products of COVID-19^1,5 — and symptoms worsened for patients with preexisting psychiatric illness,^4,6,7 significantly increasing the need for behavioral health services. Historically, most behavioral health treatments have been conducted in-person.² However, COVID-19 pressured health systems to incorporate telehealth options.^8,9 Clinics transitioned quickly, clinicians adapted to virtual services while upholding quality of care, and the scope of insurance coverage expanded to encompass telehealth.^10,11 Overall, patient and physician satisfaction for telehealth has been positive across medical specialties,^11–16 which is critical given established correlations between patient satisfaction and outcomes.^17–19

One field uniquely capable of adopting telehealth was behavioral health.^20,21 Pre-COVID-19 studies suggested that virtual behavioral health treatment was effective and comparable to in-person care,^22–30 although some patients had concerns regarding quality and privacy.³¹ Given the now-widespread use of telehealth for behavioral health treatment, it is imperative to assess patient and physician satisfaction with telehealth in a clinical setting across time.

The purpose of this quality-of-care study was to elucidate patient and clinician satisfaction with virtual behavioral health care delivered within an integrated health system across an extended duration. The secondary purpose was to compare outcomes between virtual and in-person mental health intensive outpatient treatment programs (IOP). IOP were targeted because they had parallel virtual and in-person options, thereby allowing comparison of virtual and in-person formats within a single program type. In contrast, partial hospitalization programs only briefly offered virtual treatment, thereby limiting the ability to compare virtual versus in-person versions of that program.

METHODS

This study design was retrospective and quantitative. Data were initially collected for internal quality purposes and included responses to surveys by clinicians and patients as well as electronic health record (EHR) characteristics. The study was deemed non-human subject research by the relevant institutional review board. Virtual visits across the behavioral health system utilized HIPAA-compliant telephone or Zoom modalities.

Study Population

The target population for the patient surveys analysis was all adult patients completing at least one virtual behavioral health visit for mental health and/or substance use treatment between May 4, 2020, and July 25, 2021, within a large integrated health system. This included all available levels of behavioral health care acuity that offered a virtual component, meaning virtual partial hospitalization programs, virtual IOP, and virtual outpatient programs. Although clinical care teams were asked to provide all eligible patients with links to the surveys, the anonymity of the survey responses precluded confirming which teams were consistently sending survey links to their patients. Further, patients could enter treatment more than once, thus potentially resulting in more than one survey submission per patient. Therefore, it is unknown which patients received the survey link and also went on to complete the survey.

The target population for the clinician survey analysis was all behavioral health clinicians providing virtual mental health and/or substance use treatment between June 2020 and May 2021. This included psychiatrists, psychologists, therapists, social workers, and other professionals providing direct virtual behavioral health care. Although all clinicians providing virtual treatment were eligible to take the survey, surveys were sent by clinical leadership and responses were anonymous. Therefore, it is unknown whether every leader consistently forwarded the survey to all of their eligible clinicians and whether all clinicians who received the survey also completed it.

The study population for the retrospective EHR analysis included all patients who started mental health IOP treatment on or after April 1, 2021, and were discharged by July 2, 2021, with data collection starting at the patient’s first completed visit. This relatively brief time period was chosen to control for temporal changes in extraneous factors such as overall treatment program quality and social/environmental variables (eg, changes to community-level COVID-19 restrictions).

Data Collection

Patient Surveys

Two surveys were used to collect patient feedback: a daily survey and a survey at discharge. Each provider could choose their own process for sending survey links, such as sending via email or via the health system’s patient portal. The daily survey asked patients to rate the quality of their treatment that day on a scale of 1 to 10. This survey was delivered to patients each day after treatment (Online Appendix A, Survey 1). The discharge survey was sent to patients after program discharge and used a Likert scale to collect feedback about ratings and preferences comparing virtual and in-person treatment (Online Appendix A, Survey 2).

Anonymous responses were collected in Microsoft Forms. Patients submitted 2479 responses to the daily survey, which could be submitted across multiple visits during treatment, and 414 responses to the discharge surveys, which were only submitted once per patient at treatment discharge for each program admission.

Clinician Surveys

Clinicians were surveyed around every 2 months (Online Appendix A, Survey 3). The clinician survey was sent via a survey link in an email from each team member’s clinical leadership 6 times between August 2020 and May 2021, and 381 responses were collected overall. The quantitative Likert scale survey was anonymous, and therefore the number of unique clinicians represented in this population across time is unknown. Assuming each clinician respondent completed no more than one survey for any individual request, unique clinician participation for each of the 6 survey time points ranged from 37 to 84 responses.

EHR Data

Retrospective patient EHR data were collected manually to assess no-show, length of stay (completed visit count within the episode of care), and program completion rates for patients enrolled in the virtual or in-person versions of equivalent mental health IOP at a single psychiatric hospital. All eligible patient records were assessed, resulting in 116 admissions to in-person mental health IOP treatment and 81 admissions to virtual mental health IOP treatment, for a total of 197 admissions across 194 unique patients (3 patients were admitted twice).

Program completion data were measured as a binary variable determined by whether the patient successfully completed the program based on clinical discharge notes (eg, “Patient completed program,” “Patient lost to follow-up”). Length of stay was measured as the number of completed visits within the episode of care. To calculate no-show rates, data included the total number of scheduled appointments, appointments attended, and appointments that were no-shows or cancelled late (ie, patient did not show up, indicated they were ill, or indicated they did not have transportation). No-show rates were calculated by dividing the total number of no-show/cancelled visits by the total number of scheduled visits.

Statistical Analyses

Descriptive statistics including frequencies, percentages, medians, and quartiles were used to summarize data. Beta regression was used to model proportion data. Chi-squared test of association was used to compare groups of categorical variables. Wilcoxon-Mann-Whitney test was used to compare groups of continuous variables. Cochran-Mantel-Haenszel chi-squared test for linear trend (M²) was used to examine trends in responses to Likert-type questions over time,³² with its accompanying Breslow-Day statistic for homogeneous association used to test conditional independence. Jonckheere-Terpstra test (Z-score) was used to assess trends in nonnormal continuous variables over time. Data were analyzed using Prism version 8.0.0 (GraphPad Sofware) and SAS 9.4 (SAS Institute Inc.) software. P-values of ≤0.05 were considered statistically significant.

RESULTS

Patient Survey Outcomes

Patient daily surveys demonstrated consistently high satisfaction ratings (on 1–10 scale) across the 16-month survey period (Figure 1A). The mean rating over the study period was 8.9 out of 10. A Jonckheere-Terpstra test indicated that there was a significant upward trend in mean ratings over time (Z=5.37; P<0.0001).

Figure 1.

Patient virtual behavioral health survey responses. Virtual patient daily score submissions (Panel A) and perceptions at program discharge (Panels B and C) across time.

Across the 16-month data collection period, most discharge survey responses (53.6%) indicated that virtual treatment was very or somewhat similar to in-person treatment experiences, and there was no significant trend in the responses over the study period (M²(1)=0.51; P=0.48) (Figure 1B). Most responses (61.3%) indicated a preference for virtual treatment over in-person treatment or indicated no preference (22.4%), and there was a significant increasing trend in the data path of preference for virtual care delivery over time (M²(1)=6.63; P=0.01) (Figure 1C).

Clinician Survey Outcomes

Across the study period, the majority of clinician responses (54.6%) rated virtual treatment as very easy/easy, and an additional 25.7% considered it neither easy nor difficult (Figure 2A). There was no significant trend in perception of ease of implementing virtual treatment across time (M²(1)=1.73; P=0.19). At the inception of virtual treatment in June 2020, more clinicians preferred in-person services (49.4%) compared to virtual (31.8%), yet there was a significant increasing trend of clinician preference for virtual services over time (M²(1)=21.49; P<0.0001) (Figure 2B). By the last survey period in May 2021, the majority preferred virtual services (60.0%) compared to only 13.3% who preferred in-person and 26.7% who had no preference.

Figure 2.

Aggregated clinician perceptions of virtual behavioral health treatment across time per responses to multiple survey questions (Panels A–E).

There was a significant increase in perceptions of virtual treatment effectiveness over time, from 13.1% in June 2020 to 30.0% in May 2021 (M²(1)=15.52; P<0.0001) (Figure 2C). Nearly half of clinicians perceived virtual and in-person treatments as equally effective. On average over the study period, most clinicians (74.3%) indicated they would choose to continue offering virtual treatment, with a significant upward trend from 61.2% at study inception to 88.3% at study end (M²(1)=5.77; P=0.02) (Figure 2D). Clinician ratings of their experience providing virtual treatment services were high, with a mean rating of 7.7 out of 10 and a significant upward trend over time (Z=3.84; P<0.0001) (Figure 2E).

Patient Outcomes and Demographics

Of the 197 mental health IOP patient admissions pulled from the EHR, 59.9% were female, the majority were White, and the median age was 35 years (Table 1). Groups were demographically comparable (race/ethnicity and age) except that women were significantly more likely to be in virtual treatment (69.1%) than in-person treatment (χ²(1)=4.89; P=0.03). Patients in virtual treatment had significantly higher rates of program completion (60.5%) than their in-person counterparts (44.0%) (χ²(1)=5.21; P=0.02).

Table 1.

Patient Demographics and IOP Outcomes (N=197)

Characteristic	In-person IOP, n=116	Virtual IOP, n=81	P
Age in years, mean (IQR)	35 (25.3–49.0)	35 (24.0–46.5)	0.78
Sex, n (%)			0.03
Female	62 (53.5%)	56 (69.1%)
Male	54 (46.5%)	25 (30.9%)
Race, n (%)			0.43
White, not of Hispanic origin	83 (74.1%)	52 (70.3%)
Black, not of Hispanic origin	19 (17.0%)	14 (18.9%)
AI/AN, AAPI, multiracial, or unknown	10 (8.9%)	8 (10.8%)
White, of Hispanic origin	4 (3.6%)	7 (9.5%)
Attendance outcomes
Completed treatment, n (%)	51 (44.0%)	49 (60.5%)	0.02
Rate of no-shows or late cancellation, median (IQR)	11.0% (0–33.0)	6.0% (0–12.0)	0.02
Length of stay by visit count, median (IQR)	9.5 (6.0–14.0)	11.0 (7.0–15.0)	<0.001

Note: Categorical data are given as count with percentage and continuous data as median with IQR.

AI/AN, American Indian or Alaskan Native; AAPI, Asian American Pacific Islander; IQR, interquartile range; IOP, intensive outpatient treatment program.

Given that women were more likely to receive virtual treatment and the virtual treatment group was more likely to complete treatment, the degree of association was assessed between program type and treatment completion while controlling for sex. Program completion was associated with treatment type, independent of patient sex, and patients in the virtual group had almost twice the odds of completing treatment compared to their in-person counterparts regardless of sex (odds ratio: 1.98, 95% CI: 1.10–3.56).

Median visit no-shows and late cancellation rates were 6.0% for virtual admissions and 11.0% for in-person admissions (P=0.024). Again, sex had no effect on the no-show and late cancellation rates, and the virtual group was less likely to miss a scheduled visit compared to in-person (odds ratio: 0.68, 95% CI: 0.49–0.95; P=0.025). Patients who did not complete the program had significantly higher median no-shows and late cancellation rates (23%) compared to those who completed (6%) (Z=6.14; P<0.0001). Each unique patient admission in the virtual group averaged more visits (14) compared to in-person admissions (9.5) (Z=2.90; P=0.004).

DISCUSSION

Results of study data captured early in the pandemic through mid-2021 suggest patients were consistently satisfied with telehealth, which aligns with recent research.^{26–28,31,33,34} However, exposure to virtual treatment resulted in increased preferences for telehealth across time. These preferences likely stemmed from improved treatment access and barrier removal.^{26,31,35–37} In addition to patients who preferred virtual treatment, many patients had no preference between virtual and in-person options. This finding concurs with a previous study in which 83% of virtual behavioral health patients had a preference for a hybrid treatment program³⁸ and suggests opportunities to further improve treatment using hybrid formats.³⁶

EHR data analyses comparing virtual versus in-person mental health IOP indicated that virtual patients had significantly higher rates of attendance, number of visits, and completion of treatment, which was similar to literature correlating telehealth programs with similar or improved attendance rates^39,40 and treatment outcomes.^{26–30,39,41} This is critical given the correlation of appropriate and planned transitions between treatment programs and lower readmission rates.^42–44

Women were significantly more likely to be admitted to virtual treatment than in-person treatment, a finding similar to other studies^45,46 and consistent with women’s tendency to prefer virtual care as compared to men.⁴⁷ This skewing may be due to personal provider preferences, as clinicians determined program type (virtual or in-person) based on each patient’s clinical needs, as well as patient preferences. Although no data were available to assess why women were more likely to enroll in virtual treatment, previous studies suggest women may experience more barriers to travel than men. Specifically, women tend to have more family/caregiving responsibilities,⁴⁸ which have increased during the pandemic,^49,50 and women may be more likely than men to avoid COVID-19 exposure.⁵¹

Many clinicians were satisfied with telehealth and perceived it as effective, with that trend increasing over time. The percentage of clinicians perceiving virtual treatment as less effective early in the transition to virtual was similar to previous reports⁵² and consistent with data on clinicians new to virtual care reporting a desire to return to in-person.³⁶ However, clinician satisfaction with virtual treatment increased with exposure,^9,13,36 as did the desire to continue telehealth after the pandemic.^9,13 Further research is needed to better understand clinicians’ experiences and preferences, as providers are an integral part of implementing telehealth services.

Limitations

Given the rapid transition to virtual care, survey questions were not validated using evidence-based questionnaire methodology. Another limitation is that surveys were anonymous. This limited the ability to differentiate responses based on acuity level, reason for treatment, or patient/provider demographics and prevented the ability to track who received the surveys or capture changes within an individual responder over time. Approximate sample sizes for the patient and clinician surveys were not able to be ascertained given the paucity of relevant population sizes, thus potentially limiting the internal validity of the results due to potential selection bias by a nonrepresentative sample of these populations. In addition, EHR data were retrospective, and therefore patient assignment to virtual/in-person programs was not randomized, as evidenced by the higher frequency of women in virtual treatment. Although analyses controlled for sex, other variables could similarly be skewed across groups, and future studies should control for this using prospective, randomized designs. Future studies also could include qualitative data to assess reasons for patient preferences and/or concerns related to virtual treatment, especially targeting previously identified concerns related to quality and privacy.³¹

CONCLUSIONS

The COVID-19 pandemic forced the establishment of virtual behavioral health as a treatment mechanism. This long-term study found that both behavioral health patients and clinicians perceived virtual treatment as appealing and effective. Importantly, virtual care had better attendance outcomes compared to in-person care, suggesting that virtual behavioral health treatments may be superior for some patients.

Patient-Friendly Recap

• As the COVID-19 pandemic shifted more care delivery to “telehealth,” those receiving or providing treatment for mental health conditions increased their use of virtual visits.

• Authors assessed changes in patient and clinician satisfaction with virtual behavioral health visits over time and also compared program attendance and completion rates for virtual versus in-person treatment.

• They found that patients and clinicians both perceived virtual behavioral health treatment as appealing and effective. Moreover, the superior outcomes seen from virtual care suggest such visits should continue to be offered after the pandemic.