Comparison of Patient Experience with Telehealth vs. In-Person Visits Before and During the COVID-19 Pandemic

To the Editor

In response to the COVID-19 pandemic, utilization of telehealth services became an essential modality for providing care.¹ Health care providers, who often minimally used telehealth before the pandemic, rapidly transitioned to meet the health needs of their patients (< 10% prior to COVID to > 70% at the peak).^{1 , 2} However, with the swift conversion to telehealth, the impact on patient experience is unknown.^{3 , 4} Our objective was to compare patient experience for telehealth vs. in-person visits both before and during the COVID-19 pandemic.

Methods

Patient experience was assessed using a novel electronic, Web-based survey at 405 ambulatory clinics (1,920 clinicians) covering all specialties in a 9-hospital health care system (1 academic medical center, 7 community hospitals, 1 critical access hospital). Two questions, each scored on an 11-point scale from 0 (not likely at all) to 10 (extremely likely), were sent to patients, by text message and/or e-mail, after each clinic visit to evaluate the likelihood to recommend (LTR) (1) “the clinic for care” and (2) “the provider for care.” The LTR question used in our survey is very similar to the LTR measure used widely across the country in ambulatory patient experience surveys.⁵ To develop the survey questions, cognitive interviews were conducted with a diverse group of patients to assess overall survey coherence and clarity. The survey was iteratively revised and retested in a larger sample of patients from multiple institutions.

We compared patient experience metrics for those who had telehealth visits during the early COVID-19 era (defined as March 17–April 28, 2020) vs. two comparison groups: (1) patients contemporaneously having in-person visits (March 17–April 28, 2020) and (2) patients having in-person visits prior to COVID-19 (November 1, 2019–March 16, 2020). The visit type was determined by a combination of a required billing code modifier and visit type recorded in the electronic health record. Our definition of telehealth visits includes synchronous visits, conducted by telephone or video. Patient and provider characteristics were matched to each patient encounter.

Multivariable logistic regression models were developed to predict patient dissatisfaction (LTR scores ≤ 8), adjusting for patient-reported gender, age, patient type (new patient vs. returning visit), race, operating units, provider gender, provider specialty, and provider years in practice. Robust standard errors clustered by provider specialties were calculated. Stratified models were run for each of the three groups, adjusting for the same factors. Sensitivity analyses were carried out for new patients. The Northwestern University Institutional Review Board deemed this study exempt.

Results

Of 844,483 eligible encounters, 200,987 surveys were completed between November 1, 2019, and April 28, 2020—an overall response rate of 23.8%. Comparing telehealth to in-person visits during the COVID era, LTR scores for the provider were similar for in-person visits and telehealth visits (mean LTR: 9.72 vs. 9.74, p = 1.00) (Table 1 ). Both in-person and telehealth visit scores during the COVID era were significantly higher than pre-COVID in-person visits (mean LTR: 9.64 p < 0.001 for both comparisons). When comparing telehealth vs in-person visits, there were no differences when separately examining new patient visits; no differences were detected in scores between pre-COVID and COVID era for new patients. Results for clinic LTR were comparable to those for provider LTR.

Table 1.

Comparison of Pre-COVID Era and COVID Era for In-Person and Telehealth Visits

	Pre-COVID Era	COVID Era
	Nov. 1, 2019 – Mar. 16, 2020	Mar. 17, 2020 – Apr. 28, 2020
	In-Person Visit	In-Person Visit	Telehealth Visit
No. of survey responses	169,682	8,179	23,126
No. of unique patients*	121,928	7,363	21,731
Patient type
Returning visit %	72.9	72.9	85.2
New visit %	22.7	19.1	13.0
Specialty %
Family medicine	12.2	16.0	7.9
OBGYN	7.1	3.7	15.7
Internal medicine	44.1	57.2	32.3
Dermatology	6.7	3.2	4.5
Pediatrics	3.4	1.3	6.
Surgery	2.9	1.5	4.1
Orthopaedic surgery	4.9	1.8	8.1
Neurology	3.5	6.9	1.6
Urology	1.8	1.9	2.2
Other	13.4	6.5	17.1
Provider experience
Mean LTR score (SD)†	9.64 (1.19)‡,§	9.72 (1.04)‡	9.74 (0.94)§
Dissatisfied% (score ≤ 8)	7.3	5.8	5.5
Net promoter score||	90.2	92.4	93.0
Clinic experience
Mean LTR score (SD)†	9.49 (1.28)‡,§	9.63 (1.12)‡	9.62 (1.02)§
Dissatisfied % (score ≤ 8)	12.1	8.5	9.4
Net promoter score||	84.7	89.1	88.8
Comment rate %#	74.3	71.5	76.8

^⁎For each patient in each group and time frame, the first visit is used for analysis.

^†Kruskal-Wallis test for mean rank was significant (p < 0.001).

^‡Post hoc pairwise Dunn's test with Bonferroni adjustment was significant (p < 0.001) for COVID in-person visit vs. pre-COVID in-person visit.

^§Post hoc pairwise Dunn's test with Bonferroni adjustment was significant (p < 0.001) for COVID telehealth visit vs. pre-COVID in-person visit.

^||Net promoter score is calculated by multiplying the difference between promoter (a score of > 8) % and detractor (≤ 6) % by 100.

^#Comment rate is the percentage of responders who wrote a comment in the optional free-text comment box.

OBGYN, obstetrics/gynecology; LTR, likelihood to recommend; SD, standard deviation.

Similarly, there was no significant difference between COVID-era telehealth visits and COVID in-person visits when adjusting for patient and clinician factors (5.5% vs. 5.8% dissatisfaction; odds ratio = 1.06, 95% confidence interval = 0.94–1.20) (Table 2 ). Patients were more likely to report dissatisfaction if they were female, younger, non-white, new visits, or seeking care during the pre-COVID era. Clinicians who were in practice 10 years or less and those in practice more than 30 years received lower ratings than those with 11–30 years of experience. Use of telehealth during the COVID era resulted in increased LTR scores (that is, decreased dissatisfaction) among Black patients (pre-COVID in-person, 8.8%; COVID in-person, 9.4%; COVID telehealth, 6.6%) (Table 2).

Table 2.

Factors Associated with Lower Patient Experience Scores for the Provider

Variable	Level	Patient Experience with Provider – Reporting Dissatisfaction (score ≤ 8)
		Overall		Stratified
				Pre-COVID In-Person		COVID In-Person		COVID Telehealth
		%	OR (95% CI)	%	OR (95% CI)	%	OR (95% CI)	%	OR (95% CI)
Patient gender	Male	6.5	reference	6.7	reference	6.2	reference	5.6	reference
	Female	7.3	1.10 (1.04–1.17)	7.8	1.15 (1.06–1.26)	5.5	0.72 (0.58–0.91)	5.4	0.90 (0.76–1.07)
Patient type	Return visit	6.1	reference	6.5	reference	5.0	reference	4.6	reference
	New visit	10.1	1.53 (1.43–1.65)	10.1	1.45 (1.37–1.54)	8.8	1.60 (0.99–2.58)	10.9	2.37 (1.93–2.91)
Patient age	< 18 years (guardian)	6.6	1.19 (0.95–1.48)	7.0	1.19 (0.93–1.52)	5.3	1.84 (0.73–4.65)	3.9	0.87 (0.36–2.09)
	18–30 years	11.0	1.96 (1.81–2.12)	11.8	2.06 (1.82–2.33)	8.8	2.37 (1.60–3.52)	6.8	1.29 (1.10–1.50)
	31–50 years	8.5	1.56 (1.49–1.63)	8.9	1.58 (1.44–1.74)	7.4	1.91 (1.21–3.03)	6.6	1.36 (1.10–1.67)
	51–64 years	6.8	1.22 (1.16–1.28)	7.1	1.23 (1.14–1.32)	5.4	1.27 (1.00–1.61)	5.6	1.19 (1.04–1.36)
	65+ years	5.5	reference	5.8	reference	4.2	reference	4.7	reference
Patient race	White	6.6	reference	7.0	reference	5.3	reference	5.2	reference
	Black/African American	8.5	1.32 (1.27–1.38)	8.8	1.30 (1.23–1.39)	9.4	1.98 (1.52–2.58)	6.6	1.30 (1.00–1.69)
	Asian/Pacific Islander	8.6	1.24 (1.12–1.37)	9.1	1.26 (1.13–1.41)	6.2	1.14 (0.78–1.67)	6.4	1.14 (0.93–1.41)
	Other/Unknown	8.1	1.10 (1.01–1.19)	8.5	1.10 (1.02–1.20)	7.0	1.22 (0.87–1.71)	6.0	0.96 (0.85–1.09)
Operating unit	MPG location 1	7.1	1.05 (0.56–1.97)	7.4	1.06 (0.61–1.84)	5.3	1.28 (0.31–5.30)	5.6	0.74 (0.27–2.03)
	MPG location 2	6.5	1.02 (0.55–1.88)	6.8	1.03 (0.62–1.73)	6.4	1.63 (0.39–6.90)	4.9	0.67 (0.23–1.94)
	MPG location 3	6.3	1.15 (0.62–2.11)	6.6	1.14 (0.67–1.95)	5.6	1.78 (0.56–5.66)	5.1	0.88 (0.31–2.50)
	MPG location 4	10.3	1.99 (1.04–3.83)	11.1	2.07 (1.16–3.68)	7.4	2.32 (0.57–9.38)	7.3	1.23 (0.45–3.40)
	MPG location 5	6.7	reference	6.9	reference	4.2	reference	6.3	reference
Specialty	Family medicine	5.4	reference	5.9	reference	3.1	reference	3.7	reference
	OBGYN	7.7	1.13 (1.04–1.22)	7.6	1.01 (0.95–1.08)	6.8	2.13 (1.73–2.63)	9.4	2.19 (1.91–2.52)
	Internal medicine	6.5	1.32 (1.18–1.47)	6.7	1.25 (1.10–1.42)	5.4	2.37 (1.89–2.97)	5.5	1.57 (1.45–1.70)
	Dermatology	8.4	1.72 (1.51–1.97)	8.7	1.69 (1.47–1.95)	7.3	3.29 (2.41–4.50)	5.5	1.47 (1.32–1.63)
	Pediatrics	6.6	1.31 (1.06–1.61)	7.1	1.30 (1.07–1.58)	4.2	1.14 (0.53–2.46)	4.0	1.40 (0.67–2.94)
	Surgery	5.8	1.04 (0.94–1.16)	5.7	0.95 (0.85–1.07)	5.3	1.98 (1.65–2.37)	7.4	1.89 (1.76–2.03)
	Orthopaedic surgery	9.2	1.85 (1.69–2.02)	9.8	1.84 (1.64–2.06)	5.6	4.87 (3.99–5.95)	5.4	1.67 (1.61–1.74)
	Neurology	9.3	1.92 (1.62–2.29)	10.3	2.00 (1.62–2.47)	11.8	1.58 (1.09–2.30)	6.6	1.52 (1.36–1.70)
	Urology	8.2	1.62 (1.44–1.83)	9.0	1.57 (1.37–1.80)	3.4	2.96 (2.48–3.54)	5.5	1.61 (1.51–1.71)
	Other	8.3	1.64 (1.50–1.79)	8.5	1.64 (1.48–1.83)	7.2	2.10 (1.54–2.86)	6.5	1.40 (1.28–1.53)
Provider gender	Female	7.0	reference	7.4	reference	6.0	reference	5.4	reference
	Male	7.0	1.01 (0.94–1.08)	7.3	1.01 (0.93–1.10)	5.6	0.91 (0.73–1.12)	5.5	1.01 (0.94–1.09)
Years in practice	≤ 10 years	8.5	1.24 (1.08–1.42)	8.8	1.22 (1.06–1.40)	7.4	1.28 (0.98–1.66)	7.3	1.45 (1.15–1.82)
	11–20 years	6.4	reference	6.7	reference	5.7	reference	5.1	reference
	21–30 years	6.9	1.13 (1.02–1.25)	7.3	1.15 (1.04–1.28)	5.1	0.89 (0.61–1.31)	5.3	1.07 (1.00–1.14)
	30+ years	7.0	1.23 (1.09–1.39)	7.4	1.25 (1.08–1.44)	6.1	1.24 (0.94–1.62)	5.1	1.15 (0.95–1.38)
Study group	Pre-COVID in-person	7.3	1.26 (1.13–1.41)	–	–	–	–	–	–
	COVID in-person	5.8	0.94 (0.83–1.06)	–	–	–	–	–	–
	COVID telehealth	5.5	reference	–	–	–	–	–	–

OR, odds ration; CI, confidence interval; MPG, Multispecialty physician group; OBGYN, obstetrics/gynecology.

Discussion

Patient experience scores during the rapid initial transition to telehealth because of the COVID-19 pandemic resulted in comparable patient experience scores between in-person and telehealth visits. It is unclear why COVID-era scores were higher than pre-COVID scores, but in general, there may have been a tendency for patients to be more benevolent to health care workers during the early part of the COVID-19 pandemic,⁶ though less so for new-visit patients who showed no difference in patient satisfaction comparing telehealth vs. in-person visits during pre-COVID vs. COVID era. Limitations of this study should be acknowledged. First, there is an inability to assess the telehealth visit modality (telephone vs. video) during the early stage of the pandemic. Second, the clinics examined are part of a single health care system within a single state, so results may not be generalizable. Third, although the response rate may appear low for a survey, this rate is higher than that of typical patient experience survey response rates nationally.⁷ Fourth, the effects of the broad utilization of telehealth on diagnostic errors, care delays, and quality of care were not examined. Nonetheless, these results suggest that the rapid shift to telehealth generally resulted in favorable patient experience, and possibly some improvement in racial disparities; however, attention should be paid to new patient visits, as these may not be as ideal via telehealth.