Patient Experiences With Telehealth During Versus After a System-Wide Telehealth Mandate During the COVID-19 Pandemic

Abstract

This study examines whether patients’ telehealth experiences differed during a health system mandate for telehealth encounters due to the COVID-19 pandemic versus after the mandate was relaxed. Patient experience surveys from telehealth visits across 17 adult (age 18+) primary care sites at a large, urban public health system were analyzed during two periods: when a mandate was active (March 1, 2020-June 30, 2020) and when the mandate was relaxed and any appointment modality was available (July 1, 2020-November 30, 2021). Primary outcomes were odds ratios (ORs) comparing top-box percentages of survey responses at multiple levels: individual questions, four domains, and all questions together as a composite. Key findings:

1. Patients had higher odds of selecting top-box answers in the elective telehealth period for the Care Provider (1.09 [95% confidence interval 1.03, 1.16]) and General Assessment (1.13 [1.02, 1.24]) domains and the survey composite (1.08 [1.04, 1.13]), but there was no difference for individual questions.

2. Women reported more positive experiences during the elective telehealth period in the Access (1.22 [1.01, 1.47]), Care Provider (1.32 [1.17, 1.50]), and Telemedicine Technology (1.24 [1.04, 1.50]) domains.

3. Our findings suggest that patients had better telehealth experiences when mandates were relaxed.

Introduction

During the coronavirus disease 2019 (COVID-19) pandemic, telehealth usage in the United States increased drastically as regulatory limits on telehealth were paused and many health systems mandated telehealth usage to minimize viral transmission.^1,2

Previous studies have explored patients’ telehealth experiences during the pandemic and compared in-person with virtual care.^3–5 However, none have examined telehealth patient experiences with regard to health system mandates that required visits be completed virtually. Given the impact that a telehealth mandate may have on patients’ experiences, we aimed to compare patients’ experiences with telehealth appointments during such mandates to telehealth appointments that occurred after such mandates were lifted.

We hypothesized that patients would have more positive telehealth experiences when telehealth visits were elective instead of mandated, as optionality would allow patients and providers to self-select into telehealth encounters based on personal preference and growing familiarity with virtual care.

Methods

Setting and Participants

In this retrospective study of survey data, we used aggregated survey results from routinely collected telehealth patient experience surveys across 17 adult primary care sites at a large, urban public health system from visits completed during two periods: a mandated telehealth period when the health system mandated preferential scheduling of telehealth visits, with guidelines to schedule 80+% of appointments virtually (March 1, 2020 through June 30, 2020), and an elective telehealth period when both in-person and telehealth visits were freely available (July 1, 2020 through November 30, 2021). During these two time periods, there were no systemic changes to the telehealth delivery model. Surveys stemmed from both audio-only and video-enabled telehealth encounters and were considered together. We included all primary care telehealth encounters by patients 18 years old and older during these periods.

This study was exempted from full review and granted a waiver of informed consent by the Biomedical Research Alliance of New York institutional review board (#21-12-194-373).

Survey Instrument

The surveys contained 17 Likert scale questions (1 = “very poor” to 5 = “very good”) grouped into four domains (Access, Care Provider, Telemedicine Technology, and General Assessment) and were distributed to patients by Press Ganey Associates via e-mail within one day of their visit. An e-mail reminder was sent to nonrespondents five days afterward. Surveys could be completed up to one year from initial receipt. Surveys were available in English and Spanish. Survey questions are included in the caption of Figure 1.

Figure 1.

(A) Forest plot of the odds ratios of selecting a top-box score on the telehealth patient experience survey during the mandated and elective telehealth periods. (B) Distribution of aggregate total (N) and top-box (n) question responses on the telehealth patient experience survey by survey domain and survey composite. * Access Questions: Ease of arranging your video or telephone visit; Ease of contacting (eg, email, phone, web portal) us. † Care Provider Questions: Concern the care provider showed for your questions or worries; Explanations the care provider gave you about your problem or condition; Care provider's efforts to include you in decisions about your care; Care provider's discussion of any proposed treatment (options, risks, benefits, etc); Likelihood of recommending this care provider to others. ‡ General Assessment Questions: How well the staff worked together to care for you; Likelihood of your recommending our practice to others. § Telemedicine Technology Questions: Ease of talking with the care provider over the video or telephone connection; If you had a video visit, how well the video connection worked (please only answer if your visit was by video); How well the telephone connection worked during your visit, whether by phone or video.

Primary Outcome

Our primary outcomes were unadjusted ORs comparing survey questions’ top-box percentages in the elective period relative to the mandated period. In top-box methodology, the most positive response option is defined as the “top-box” response, and all other options are non-top-box.

Statistical Analysis

We used contingency tables to generate unadjusted ORs, and Fisher's exact test was used to generate P values and confidence intervals. We also compared top-box ORs across time periods at the survey domain level and when all questions were considered together as a survey composite.

We compared respondent age, sex, race, and preferred language between time periods using χ² tests. We also completed domain-level subgroup analyses stratified by these demographics.

Results

There were 1145 completed surveys in the mandated period and 5761 completed surveys in the elective period (response rates 12.5% and 9.9%, respectively; Table 1). There were no significant differences in respondent age, sex, race, or preferred language between time periods.

Table 1.

Demographic Characteristics of Telehealth Patient Experience Survey Respondents During the Mandated and Elective Telehealth Periods.

Demographic characteristic	Mandated Telehealth period March 2020-June 2020 (N = 1145)	Elective Telehealth period July 2020-November 2021 (N = 5761)	P value
	n (%)	n (%)
Age (years)			.88
18–34	111 (9.7)	521 (9.1)
35–49	345 (30.1)	1737 (30.2)
50–64	489 (42.7)	2511 (43.6)
65+	200 (17.5)	991 (17.2)
Sex			.74
Male	402 (35.1)	2052 (35.6)
Female	743 (64.9)	3709 (64.4)
Race			.19
Asian	81 (7.1)	505 (8.8)
Black or African American	381 (33.3)	1792 (31.1)
Other, Unknown, or Choose not to disclose	578 (50.5)	2940 (51.0)
White	105 (9.2)	520 (9.0)
Preferred Language			.97
English	757 (66.1)	3806 (66.1)
Spanish	388 (33.9)	1955 (33.9)

There was no significant difference in the odds of a patient selecting a top-box score for any individual question between time periods. Analysis by domain indicated higher odds of selecting a top-box score in the elective period versus the mandated period for Care Provider (1.09 [95% confidence interval 1.03, 1.16]) and General Assessment (1.13 [1.02, 1.24]) (Figure 1). Higher odds for top-box scores were also observed when considering all questions together as a survey composite (1.08 [1.04, 1.13]).

In subgroup analyses by demographics, women were more likely to select a top-box score in the elective period versus the mandated period in the domains of Access (1.22 [1.01, 1.47]), Care Provider (1.32 [1.17, 1.50]), and Telemedicine Technology (1.24 [1.04, 1.50]). There were no differences observed for other subgroups at the domain level.

Discussion

We describe differences in patients’ telehealth experiences over two periods during the COVID-19 pandemic: when encounters were mandated to occur via telehealth and when telehealth was optional. There were no significant changes across these periods for individual survey questions. In contrast, there were modestly higher odds of selecting a top-box score during the elective period in the domains of Care Provider and General Assessment and for the survey composite, but not for the domains of Access and Telemedicine Technology.

With previous work showing that demographic characteristics are associated with different telehealth preferences, ⁶ it is interesting to note that the demographic composition of respondents did not change with the ability to elect into telehealth. Differences in experiences over time were observed for female patients, but not for male patients or when examined by age, race, or preferred language.

These findings support our hypothesis that patients had more positive telehealth experiences when telehealth mandates were relaxed. No significant difference was observed in the domain of Telemedicine Technology. This suggests that even when patients and providers were able self-select into telemedicine care delivery, patient experience does not improve significantly, and there is a limit to gains from platform familiarity.

Limitations

Individual-level data were not available; thus, we are unable to control for the effect of multiple demographic factors simultaneously. Patients of different demographic groups may have different preferences related to telehealth usage^7,8 and may also vary in their participation in patient experience surveys. ⁹ The surveys were only available by email and in two languages, limiting generalizability. Finally, our response rate was relatively low; however, this response rate is on par with patient experience survey response rates in safety-net settings. ¹⁰

Conclusion

The COVID-19 pandemic led to many fundamental shifts in healthcare provision, including the rapid adoption of telehealth. Investigating how patients’ experiences of telehealth evolved as this care modality transitioned from being mandatory to being optional offers insight into which patients may continue to prefer telehealth as a means of receiving care. Further research is needed to understand why some survey domains saw changes in their top-box scores over this transition while others did not. Such research would benefit from analysis at the individual (vs aggregate) level, as this could allow for the development of targeted interventions to improve the telehealth experience of identifiable populations.