Patient satisfaction with telehealth versus in‐person visits during COVID‐19 at a large, public healthcare system

Abstract

Rationale

During the coronavirus disease pandemic, audio‐only and video telehealth visits became more widely available, but the relative patient satisfaction between telehealth and in‐person modalities is not well‐described.

Aims and Objectives

Our objective was to compare patient satisfaction with audio‐only, video, and in‐person adult primary care visits at a large, urban public healthcare system.

Methods

In this cross‐sectional study, we used aggregated data from Press Ganey patient satisfaction surveys at 17 primary care facilities at New York City Health + Hospitals for visits between 1 June 2021 to 30 November 2021. We compared mean scores for questions common to surveys for each modality in domains of Access, Care Provider, and Overall Assessment using pairwise comparisons with two‐tailed t‐tests.

Results

There were 7,183/79,562 (9.0%) respondents for in‐person visits and 1,009/15,092 (6.7%) respondents for telehealth visits. Compared to respondents for in‐person visits, respondents for telehealth visits were more likely to be aged 35–64 years, Asian, and speak English as their primary language, and less likely to be ≥65 years old, Black or other race, and speak Spanish or another language as their primary language (p < 0.001). Patients reported higher mean satisfaction for Access measures for telehealth visits than in‐person visits (p < 0.001). For Care Provider satisfaction questions, video visits generally had higher mean scores than in‐person and, in turn, audio‐only visits. For Overall Assessment questions, video visits had higher mean scores than in‐person and, subsequently, audio‐only visits.

Conclusion

Of the visit modalities, video visits had the highest mean satisfaction scores across all domains. Telehealth may improve experiences with access, but audio‐only visits may provide poorer visit experiences.

1. INTRODUCTION

Patient satisfaction with their healthcare experience is an important outcome that is associated with clinical outcomes, ¹ patient ² and clinician ³ retention, and malpractice risk. ⁴

During the coronavirus disease pandemic, audio‐only and video telehealth visits became more widely available. On the one hand, patients may prefer the potential convenience of telehealth visits; on the other hand, patients may be discontented by new barriers related to technology usage or unmet expectations of what can and should be achieved during a medical encounter (e.g., physical examination). Prior studies have suggested that patient satisfaction for telehealth visits may be on par with or better than in‐person visits. ⁵ , ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ However, whether this is true in a safety‐net, adult primary care population is not well‐established. Additionally, differences in satisfaction between audio‐only and video visits are not well‐described.

Our objective was to compare patient satisfaction with audio‐only, video, and in‐person adult primary care visits at a large, urban public healthcare system.

2. METHODS

In this cross‐sectional study, we used aggregated data from Press Ganey patient satisfaction surveys ¹¹ at 17 primary care facilities at New York City Health + Hospitals for visits conducted from 1 June 2021 to 30 November 2021. During this time period, telehealth visits were available to patients electively. When scheduling appointments, clinicians and patients used their discretion to determine whether an in‐person, audio‐only, or video visit was appropriate or preferred for a given concern. There were no explicit clinical, demographic, or other exclusion criteria for scheduling telehealth visits.

Patient satisfaction surveys were provided via mail, text message, or email (sequentially) to a random sample of patients who completed in‐person visits and via email only to all patients who completed telehealth (audio‐only or video) visits. Visit modality (in‐person, audio‐only, video) was determined using scheduling data. Patients with multiple visits during the time period were not sampled more than once every 90 days. Patients had up to 1 year from the time of the completed visit to respond to a survey. One reminder is sent for paper surveys and up to two reminders are sent for electronic surveys.

We describe patient demographics for survey respondents using data linked from the electronic health record.

Our primary outcomes were mean scores for questions from the Press Ganey survey that were used for both telehealth and in‐person encounters targeting domains of Access, Care Provider, and Overall Assessment. Each question is answered on a 5‐point Likert scale with corresponding point values (Very Poor [0], Poor [25], Fair [50], Good [75], Very Good [100]). We compare mean satisfaction scores for each question between in‐person, audio‐only, and video visits using pairwise comparisons with two‐tailed t‐tests.

We used p < 0.05 as a threshold for statistical significance. All analyses were conducted using Stata, version 15. This study was exempted from full review and granted a waiver of informed consent by the Biomedical Research Alliance of New York institutional review board.

3. RESULTS

Of 79,562 surveys administered for in‐person visits, there were 7,183 (9.0%) respondents; of 15,092 surveys administered for telehealth visits, there were 1,009 (6.7%) respondents.

Compared to respondents for in‐person visits, respondents for telehealth visits were more likely to be aged 35–64 years, Asian, and speak English as their primary language, and less likely to be ≥65 years old, Black or other race, and speak Spanish or another language as their primary language (p < 0.001; Table 1).

Table 1.

Patient demographics by modality

Characteristic	In‐person, N = 7183	Audio‐only televisit, N = 458	Video visit, N = 551	p
Age, years				<0.001
18–34	590 (8.2)	32 (7.0)	47 (8.5)
35–49	1980 (27.6)	139 (30.4)	179 (32.5)
50–64	2982 (41.5)	212 (46.3)	236 (42.8)
≥65	1631 (22.7)	75 (16.1)	89 (16.2)
Female	4529 (63.1)	302 (65.9)	362 (65.7)	0.414
Race				<0.001
White	627 (8.7)	46 (10.0)	38 (6.9)
Black	2576 (35.8)	171 (37.3)	182 (33.0)
Asian	551 (7.7)	45 (9.8)	72 (13.1)
Something else/unknown	3428 (47.7)	196 (42.8)	259 (47.0)
Language				<0.001
English	4603 (64.1)	323 (70.5)	393 (71.3)
Spanish	2338 (32.6)	135 (29.5)	158 (28.7)
Something else	240 (3.3)	0 (0)	0 (0)

On average, patients reported higher satisfaction for Access measures (ease of scheduling/arranging an appointment, ease of contacting the clinic) for telehealth visits than in‐person visits (p < 0.001), though there were no significant differences between audio‐only and video (p = 0.54, p = 0.34, respectively; Table 2).

Table 2.

Patient satisfaction by modality

Question	Mean score (SD), in‐person, N = 6975	Mean score (SD), audio‐only, N = 455	Mean score (SD), video, N = 551	p, in‐person vs audio‐only	p, in‐person vs video	p, audio‐only vs video
Access
Ease of scheduling your appointment/Ease of arranging your video or telephone visit	81.3 (23.7), n = 6644	87.4 (21.3), n = 445	88.2 (19.1), n = 537	<0.001	<0.001	0.54
Ease of contacting (e.g., email, phone, web portal) the clinic	77.4 (25.5), n = 6261	83.8 (24.7), n = 442	85.2 (21.6), n = 539	<0.001	<0.001	0.34
Care provider
Concern the care provider showed for your questions or worries	88.1 (18.7), n = 6731	86.6 (23.6), n = 446	91.2 (17.5), n = 548	0.11	<0.001	<0.001
Explanations the care provider gave you about your problem or condition	88.4 (18.6), n = 6694	86.8 (23.1), n = 448	90.3 (17.8), n = 546	0.08	0.02	0.007
Care provider's efforts to include you in decisions about your care	88.4 (18.6), n = 6615	86.7 (23.2), n = 447	90.1 (18.1), n = 545	0.07	0.04	0.01
Care provider's discussion of any proposed treatment (options, risks, benefits, etc.)	87.9 (19.1), n = 6491	85.5 (24.1), n = 441	89.4 (18.4), n = 546	0.01	0.08	0.004
Likelihood of recommending this care provider to others	88.4 (20.2), n = 6583	85.4 (25.7), n = 448	90.1 (19.9), n = 542	0.003	0.003	<0.001
Overall assessment
How well the staff worked together to care for you	88.3 (16.9), n = 6727	86.4 (23.4), n = 437	89.7 (17.6), n = 541	0.03	0.06	0.01
Likelihood of your recommending our practice to others	88.6 (18.6), n = 6669	86.7 (24.0), n = 445	90.4 (18.8), n = 544	0.04	0.03	0.007

For Care Provider satisfaction questions, video visits generally had higher mean scores than in‐person and, in turn, audio‐only visits (Table 2). Differences in in‐person and audio‐only were not statistically significant for concern the care provider showed (p = 0.11), explanations the care provider gave (p = 0.08), and efforts to include patients in care decisions (p = 0.07). Differences in in‐person and video were not statistically significant for discussion of proposed treatments (p = 0.08). Meanwhile, differences between audio‐only and video visits were consistently significant (p ≤ 0.01).

For Overall Assessment questions (how well staff worked together, likelihood of recommending the practice to others), video visits had higher mean scores than in‐person and, subsequently, audio‐only visits (Table 2). The difference between in‐person and video visits was not statistically significant for how well the staff worked together (p = 0.06).

4. DISCUSSION

In this cross‐sectional study of patient satisfaction surveys in primary care at a large, public healthcare system, we found that patients reported similarly higher satisfaction with telehealth (audio‐only and video visit) Access compared with in‐person visits, and that video visits specifically had higher satisfaction than in‐person and audio‐only visits, respectively, for Care Provider and Overall Assessments. Our study uniquely compares audio‐only and video modalities and focuses on an urban, primary care safety‐net population.

Our findings of higher satisfaction with Access in telehealth visits than in‐person visits are consistent with the notion that telehealth visits may improve access to care. Despite patients’ satisfaction with Access not being significantly different between telehealth modalities, satisfaction scores with Care Provider and Overall Assessments were different between audio‐only and video visits, with audio‐only visits consistently scoring lower. We hypothesize that having a visual connection between patient and clinician contributes to a more positive virtual visit experience, particularly in domains such as the clinician showing concern for the patient. Interestingly, video visits also scored higher in these domains than in‐person visits, suggesting that video visits may make patients feel heard and cared for, perhaps because, unlike during in‐person visits when a provider may be turned away from a patient to document, during video visits, they are often facing the camera throughout the appointment.

For the Overall Assessment, video visits had higher satisfaction scores than in‐person visits, and audio‐only visits had the lowest scores. Given that the staff workflows (i.e., registration processes, contacting the patient for scheduling follow‐up appointments, referral management and arranging diagnostic testing) for audio‐only and video visits are similar, we were surprised that there was a difference in satisfaction with how well the staff worked together to care for patients between audio‐only and video visits, especially when the other domain most related to nonclinical staff processes (Access) did not have differences between telehealth modalities. One possibility is that a “halo effect” from having a more positive clinical encounter improved perception of the overall experience.

Though the scores were not discrepant in an extreme manner (i.e., less than one whole Likert category lower), the finding that audio‐only visits scored lower on patient experience than in‐person and video visits is important to consider in the conversation about implementing and scaling different care modalities to balance ease‐of‐access, equity, efficacy, and user experience. Prior studies have suggested that audio‐only visits may be favorable for their ease‐of‐access and potentially more equitable access. ¹² However, if this is at the expense of an optimal care experience, additional work may be needed to calibrate service design to ensure positive experiences for all. Finally, while satisfaction may be relatively higher, this may not necessarily translate into patients being comfortable with continuing care with telehealth after the pandemic. ⁶ , ¹³

4.1. Limitations

One of the limitations of this study is the low response rate, which may introduce bias due to patients who responded potentially being systematically different from patients who did not respond. ¹¹ , ¹⁴ , ¹⁵ Further, telehealth visit surveys were conducted by email only, which may affect response rates from less technically savvy patients. However, though our response rate is lower than commonly published averages for Press Ganey surveys (~12%), ¹¹ , ¹⁴ , ¹⁵ it is consistent with previously demonstrated lower rates for low‐income patients (~6%). ¹⁴ Because of the low response rate, findings may not be generalizable, but they still capture a slice of patient experiences and can serve as a starting point in understanding relative care experiences with in‐person and telehealth services. Since we use aggregated data, we do not account for differences in baseline characteristics for in‐person, audio‐only, and video visit respondents. Clinicians and patients could electively decide the modality of their visits, thus the characteristics of the population that participated in telehealth may be influenced by self‐selection or implicit biases. Different population segments may participate in telehealth differently ¹⁶ , ¹⁷ and engage with patient experience surveys differently. ¹⁸ , ¹⁹ No specific clinical guidelines were given regarding scenarios appropriate for telehealth, so patients may have had variable experiences with telehealth services if telehealth was sub‐optimally applied to their clinical situation. Nonetheless, we would expect this to reflect more negatively in satisfaction scores for telehealth services. We used scheduling data rather than billing data to identify visit modality, which may overestimate the proportion of video visits completed; however, this would bias differences between audio‐only and video visits towards the null since the two groups would be more similar. These patient satisfaction data were captured during a pandemic, when patients’ expectations of care availability and experience and sentiment towards healthcare providers may be different than during nonpandemic times.

5. CONCLUSIONS

Audio‐only visits, while perhaps conducive to improved access to care, may lead to poorer visit experiences. Meanwhile, video visits may balance improved access with better visit experience. These differences are important to consider when designing primary care services that center around the patient experience. Future studies should assess demographic differences in satisfaction with telehealth and also evaluate clinical outcomes of telehealth visits and corroborate them with patient satisfaction.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

Chen K, Lodaria K, Jackson HB. Patient satisfaction with telehealth versus in‐person visits during COVID‐19 at a large, public healthcare system. J Eval Clin Pract. 2022;1‐5. 10.1111/jep.13770

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.