Patient Experience with In-Person and Telehealth Visits Before and During the COVID-19 Pandemic at a Large Integrated Health System in the United States

Abstract

Background

Information about telehealth versus in-office visits and how patient experience before compared to during the COVID-19 pandemic is important for healthcare planning.

Objective

To compare patient experience by visit type and before and during the pandemic.

Design

Survey of patients assessing ambulatory care before and during the pandemic.

Participants

A total of 58,500 adult patients (13,928 primary care and 44,581 specialty physician visits) at a large integrated health system with 197 clinics on the west coast of the United States. The majority were female (59%), 55 or older (65%), and non-Hispanic White (55%), and had an in-office visit (87%) while 10% had a tele-video and 3% a phone visit.

Main Measures

Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Survey 3.0 doctor communication, care coordination, access, and office staff composites; an overall rating of the doctor; and whether the patient would recommend the doctor to family and friends.

Key Results

Patient experience with telehealth visits was as positive as or more positive than that with traditional office-based visits. Doctor communication on tele-video visits was viewed as slightly more positive than that of in-office or phone visits. Tele-video visits were also slightly more positive than in-office visits for care coordination, overall rating of the doctor, and willingness to recommend to family and friends. Office staff were viewed less positively on the phone than tele-video or in-office visits. Patient experience was similar before and during the COVID-19 pandemic (e.g., on a 0–100 possible range with a higher score being better, doctor communication was 94.4 before and 94.9 during).

Conclusions

The positive experiences with telehealth, especially tele-video, may be due to patient appreciation of efforts made to maintain access, the focused nature of telehealth visits, and help by staff for navigation technical issues. Lessons learned about delivering responsive telehealth care can be used to ensure high-quality care after the pandemic.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11606-021-07196-4.

INTRODUCTION

Patient experience measures are key aspects of health care quality assessment in the United States (U.S.). Many policy initiatives for the improvement of care quality and the facilitation of patient choice include public reporting of providers’ performances on patient experience measures.¹ Some initiatives have incentivized providers to improve care through pay-for-performance programs.² In addition, ambulatory patient experience performance measurement is often included in the U.S. government and commercial alternative payment model contracts.³

The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) surveys are the most widely administered patient experience measures in the U.S. The CAHPS Clinician and Group Survey (CG-CAHPS) is used to assess care from physician groups and/or individual providers.⁴ Variants of the CG-CAHPS survey are reported on the Centers for Medicare & Medicaid Services Physician Compare web site and used to evaluate accountable care organizations participating in the Medicare Shared Savings Program. CG-CAHPS surveys have been administered to millions of patients and thousands of U.S. medical practices to assess patient experiences with care received from providers and staff in primary, specialty, and ambulatory care settings.⁵

A randomized trial of 497 adult men followed over 2 years showed that substitution of phone calls for some in-clinic visits led to a reduction in utilization and total expenditures and improvement in physical function for the subgroup of patients with fair or poor overall health at the baseline.⁶ Six years later, Wasson commented that it is “difficult to determine the circumstances for which phone care is likely to be effective …emerging literature about telephone care has significant gaps” (p. 646).⁷ More recently, Polinski et al.⁸ reported that one-third of 1734 patients at CVS MinuteClinics in 2014 preferred tele-video over in-person visits. Similarly, a study of 254 adult patients at the Massachusetts General Hospital telehealth program found that 63% perceived no difference in the overall quality of virtual video visits compared to office visits.⁹ In addition, a study of primary care in Israel found no evidence of missed diagnoses or adverse health outcomes related to telehealth ¹⁰.

There has been an increase in the use of telehealth associated with the COVID-19 pandemic.^11,12 Information about how these experiences compare to traditional in-office health care is of great interest. Ackerman et al.¹³ assessed patient satisfaction with electronic consultation with specialists by their primary care provider versus conventional in-person specialty visits at Dartmouth-Hitchcock, University of Virginia, University of Iowa, University of Madison, and five University of California sites (San Francisco, San Diego, Irvine, Davis, and Los Angeles) from March to November of 2016. The authors found that patients were as satisfied with recommendations based on these electronic consultations to their provider as they were with conventional referrals based on a simple yes/no satisfaction question: “Are you satisfied with the care you received?” Miller et al.¹⁴ reported that video-based visits during the pandemic tended to be most positively rated compared to office and phone visits in the Kaiser Permanente Mid-Atlantic States health system, but the study included only two survey questions: How well your needs and schedule were taken into consideration when this (office, phone, video) appointment was scheduled, and How well your needs were met in this appointment/visit. Authors of a review article concluded that “patients and healthcare providers have a high level of satisfaction with use of telehealth…and reported willingness to continue using telehealth after the pandemic.”¹⁵

In this study, we examine patient experiences with ambulatory care in a large sample of adults receiving care in an integrated health system to compare telehealth with in-office visits, and before and during the COVID-19 epidemic. Pre-pandemic, the health system began a deliberate but measured movement toward the use of telehealth by enabling tele-video within the electronic health record, physician and staff education and training, and sequenced rollout of these functionalities. Due to regulatory restrictions, it was anticipated that telehealth use would be limited. This rollout was rapidly expanded to respond to patients’ needs at the onset of the pandemic enabled by the loosening of regulatory restrictions and other barriers.

METHODS

Sample

The CG-CAHPS survey 3.0 was administered using mixed mode (interactive voice response, phone, web) to a sample of 58,509 adult patients in a large integrated health system on the west coast of the United States with 13,928 primary care and 44,581 specialty physician visits at 197 clinics. Patients were asked about care received in the prior 6 months. Patients were eligible if they had at least 1 visit with their primary care or specialist physician (named in the survey) during the 12 months before the survey fielding date. Continuous sampling was used covering adult patients with at least 1 visit from July 1, 2018, through December 31, 2020. Patients who completed a survey were not eligible to be sampled for another survey for 6 months.

Measures

The CG-CAHPS survey 3.0 includes multi-item composite measures: a 4-item doctor communication, 3-item care coordination, 3-item access, and 2-item office staff composite (Table 1). In addition, the survey includes a 0–10 overall rating of the doctor. We also included a question about whether the patient would recommend the doctor to family and friends. We coded each of these items on a 0–100 possible range with a higher score representing better patient experience. We averaged together the items within the composites, so all scores are on the same 0–100 possible range.

Table 1.

Content Assessed in Survey

Access

- Patient got appointment for urgent care as soon as needed   - Patient got appointment for non-urgent care as soon as needed   - Patient got answer to medical question the same day he/she contacted provider’s office

Doctor communication

- Doctor explained things in a way that was easy to understand   - Doctor listened carefully to patient   - Doctor showed respect from what patient had to say   - Doctor spent enough time with patient

Care coordination

- Doctor knew important information about patient’s medical history   - Someone from doctor’s office followed up with patient to give results of blood test, x-ray, or another test   - Someone from doctor’s office talked about all prescription medications being taken

Office staff

- Clerks and receptionists were helpful   - Clerks and receptionists were courteous and showed respect

Doctor rating

- Rating of doctor

Would recommend

- Would recommend doctor to family and friends

Access, doctor communication, care coordination, and office staff items were administered using Never/Sometimes/Usually/Always response options. The doctor rating item was administered with a 0 (worst doctor possible) to 10 (best doctor possible) response scale. The recommend item was administered using a definitely yes/somewhat yes/somewhat no/definitely no response scale

The survey asked about age, gender, education, race and ethnicity, self-rated general health, and self-rated mental health using the standard C-G CAHPS Survey 3.0 items. It included a question about whether the respondent was of Hispanic or Latino origin or descent and a separate question about race (White; Black or African American; Asian; Native Hawaiian or Other Pacific Islander; American Indian or Alaska Native; Other). In addition, administration data indicated the type of visit (in-person, tele-video, or phone) and the clinic where care was received.

Analysis Plan

We provide information about the reliability and construct validity of the CAHPS measures. We estimate internal consistency reliability¹⁶ of the CG-CAHPS composites and clinic site of care-level reliability for the composites and two single items using one-way ANOVA: (mean square between clinic − mean square within clinic)/(mean square between clinic). We then estimate product-moment correlations among the composites and global rating items and regress the global rating items on the CG-CAHPS composites.

We estimate CG-CAHPS composites and global rating items by type of visit and compare patient experience before (July 1, 2018–March 15, 2020) and after (March 16, 2020–December 31, 2020) the COVID-19 pandemic. Because the CG-CAHPS report items use a 6-month recall period, we also run analyses restricting those in the COVID-19 period to September 16, 2020 through December 31, 2020. We use generalized linear models to estimate least square (adjusted) means for the CG-CAHPS composites and global rating items, controlling for gender, age, race and ethnicity, education, self-rated general health, self-rated mental health, and type of visit. Site of care (clinic) is also adjusted for in models that estimate the CG-CAHPS composites and global rating items. Differences between subgroups were evaluated using Tukey–Kramer adjustment for multiple comparisons.^17,18 Differences of 1, 3, and 5 points are considered small, medium, and large, respectively ¹⁹.

Adjusted CAHPS scores by the seven quarters before the pandemic did not indicate systematic seasonality effects. Four of the six CAHPS measures differed significantly by quarter, but effects were inconsistent (e.g., the adjusted doctor communication mean was highest in one winter and lowest in the other winter). In addition, the mode of survey administration was not significantly uniquely associated with CAHPS scores in multivariate models and was, therefore, not included in the regression models.

Statistical significance was based on p < 0.05. Analyses were conducted using SAS 9.4 (SAS Institute, Cary, NC). The secondary analyses reported in this study were deemed exempt by the UCLA Human Subjects Protection Committee (IRB #20–002382).

RESULTS

Figure 1 shows the change in visit type associated with the onset of the pandemic through December 2020. There was a rapid drop-off of in-person visits and an increase in telehealth visits within a few weeks. Over time, in-person visits increased, total visits went back to baseline, and telehealth visits stabilized at about 20% of the total visits.

Fig. 1.

Total visits by visit type in 2020. Large drop in total visits and rapid increase in telehealth visits at the onset of the pandemic in this health system. The number of total visits returned to pre-pandemic levels with a mix of visit types and persistent use of telehealth

The response rate was 28%. The vast majority (71%) of the sample completed the CAHPS survey using interactive voice response, followed by web (23%) and phone (6%). The characteristics of the study sample are provided in Table 2. The majority was female (59%), 55 or older (65%), non-Hispanic White (55%), highly educated (64% with a 4-year college degree or higher), with good or very good self-rated general health (64%), and with very good or excellent mental health (65%), and had an office visit (87%) rather than a tele-video (10%) or phone (3%) visit.

Table 2.

Sample Characteristics (n = 58,509)

Characteristic	Percent
Gender
Female	59%
Age
18–24	3%
25–34	9%
35–44	10%
45–54	13%
55–64	20%
65–74	26%
75 and older	19%
Race and ethnicity
Hispanic (English primary language)	10%
Hispanic (Spanish primary language)	2%
Non-Hispanic
White	55%
Black	5%
Asian	6%
Pacific Islander, American Indian, or Alaskan Native	1%
Another race	5%
Missing race and ethnicity	16%
Education
8th grade or less	1%
Some high school	1%
High school graduate	8%
Some college	25%
4-year college degree	26%
More than 4-year college degree	38%
Self-rated general health
Poor	4%
Fair	16%
Good	32%
Very good	32%
Excellent	16%
Self-rated mental health
Poor	2%
Fair	9%
Good	24%
Very good	33%
Excellent	32%
Type of visit
In-office	87%
Tele-video	10%
Phone	3%

Provider administration data was available about gender, age, and race for all adult patient visits during the study period. Hispanic ethnicity was not available in the administration data. Race and ethnicity were combined in the sample characteristics reported above. The percentage of the sample who reported White race regardless of ethnicity was 60%. Administration data indicate that the gender and race composition for all patient visits was the same as it was for the sample: 59% were female and 60% were White. However, a larger percentage of the sample was 55 or older: 65% for the sample versus 56% for all adult patient visits.

Reliability

Reliability at the clinic site of care level exceeded 0.80 for all four composites and the two global rating items: doctor communication (0.82), care coordination (0.83), access to care (0.87), office staff (0.82), overall rating of doctor (0.82), and would recommend (0.81). Internal consistency reliability was larger than 0.70 for three of the four composites (0.73 for access, 0.75 for office staff, and 0.88 for doctor communication, but it was low (0.47) for the care coordination composite.

Associations Among CG-CAHPS Measures

Product-moment correlations among the four composites ranged from 0.34 (coordination of care with office staff) to 0.55 (coordination of care with doctor communication). In addition, access to care correlated 0.35 with both coordination of care and doctor communication; communication correlated 0.36 with office staff and 0.37 with access to care. Correlations of the composites with the global rating items ranged from 0.31 (office staff) to 0.68 (doctor communication) for the overall rating of the doctor, and from 0.30 (office staff) to 0.71 (doctor communication) for the would recommend item.

Unique Associations of C-G CAHPS Composites with Global Rating Items

Appendix Table 5 shows that doctor communication had the largest association of the four C-G CAHPS composites with the overall rating of the doctor in the main effect regression model (standardized betas in parentheses): communication (0.61), coordination (0.15), access (0.06), and office staff (0.04). Doctor communication had the largest association of the four C-G CAHPS composites with the would recommend item in the main effect regression model (standardized betas in parentheses): doctor communication (0.67), coordination (0.10), access (0.03), and office staff (0.01).

CG-CAHPS Composites and Global Rating Items by Type of Visit

Table 3 provides adjusted means by visit type. Doctor communication was significantly more positive for tele-video than office or phone visits, and care coordination, global rating of the doctor, and willingness to recommend the doctor to family and friends were more positive for tele-video than office visits. In addition, reports about office staff were less positive for phone than for office or tele-video visits. The size of the significant differences ranged from 0.8 (care coordination, doctor rating, and would recommend) to 1.3 (office staff).

Table 3.

CG-CAHPS Survey 3.0 Composites and Global Rating Items by Visit Type

CG-CAHPS	Office	Tele-video	Phone	F statistic	p value
Access	84.5a	85.0a	83.4a	2.47	0.0842
Doctor communication	94.5b	95.6a	94.5b	11.58	< 0.0001
Care coordination	85.7b	86.5a	86.5a,b	3.48	0.0308
Office staff	93.3a	93.2a	92.0b	5.01	0.0067
Doctor rating	93.0b	93.8a	93.8a,b	8.80	0.0002
Would recommend	94.3b	95.1a	94.7a,b	3.67	0.0254

Data were collected for visits from July 1, 2018–December 31, 2020. Models are adjusted for gender, age, race and ethnicity, education, self-rated general health, self-rated mental health, and clinic site of care. The F statistic is for the 2 degrees of freedom test of type of visit. When visit types share the same superscript in a row, then the adjusted means do not differ from one another. Superscript “a” represents more favorable patient experience than superscript “b”

C-G CAHPS Scores Before and During the COVID-19 Pandemic

There were several significant differences favoring patient experience during the COVID-19 pandemic. Results were robust to whether those included in the pandemic subgroup were based on a start data of March 16, 2020, or restricted to 6 months later (September 16, 2020), to correspond to the 6-month recall period of the CAHPS survey items. The largest difference was for access to care: 1.5 points (Table 4, panel A) and 1.2 points (Table 4, panel B). All other significant differences ranged from 0.5–0.7 points.

Table 4.

CG-CAHPS Survey 3.0 Composites and Global Rating Items Before and During the COVID-19 Pandemic

CG-CAHPS	Before	During	F statistic	p value
Panel A: during the period that begins with the start of the COVID-19 pandemic
Access	83.8	85.3	29.92	< 0.0001
Doctor communication	94.4	94.9	8.28	0.0040
Care coordination	85.6	86.0	3.22	0.0727
Office staff	93.0	93.7	21.13	< 0.0001
Doctor rating	92.8	93.4	13.96	0.0002
Would recommend	94.1	94.7	10.99	0.0009
Panel B: during the period limited to account for the survey’s 6-month recall
Access	84.0	85.2	9.35	0.0022
Doctor communication	94.4	95.0	6.01	0.0143
Care coordination	85.6	86.1	2.01	0.1560
Office staff	93.0	93.7	8.96	0.0028
Doctor rating	92.8	93.5	8.85	0.0029
Would recommend	94.1	94.7	3.41	0.0647

Data “Before” was collected for visits from July 1, 2018, to March 15, 2020, and “During” was March 16, 2020–December 31, 2020 (Panel A) and September 16, 2020–December 31, 2020 (Panel B). Models are adjusted for gender, age, race/ethnicity, education, self-rated general health, self-rated mental health, type of visit, and clinic site of care. The F statistic is for the 1 degree of freedom test of before versus during the COVID-19 pandemic

DISCUSSION

This study indicates that patient experience with telehealth visits was as positive or more positive than with traditional office-based visits. Reports about doctor communication, care coordination, global rating of the doctor, and recommending the doctor to family and friends were perceived to be significantly better for tele-video visits than office visits. Adjusting for type of visit and several other variables, patient experience scores during the COVID-19 pandemic were often significantly better than before but the magnitude of differences was small.

At the height of the pandemic in the spring of 2020, the total visits were reduced by 51%, and 58% of the visits were telehealth visits. The relatively favorable patient experience during this time could be due in part to patient appreciation of the efforts made to maintain access to providers via telehealth,²⁰ as well as some practice site help by staff for patient navigation of technical issues. All telehealth visits were scheduled within the electronic health record.

Study limitations should be considered when interpreting the study findings. Although participating patients were randomly selected to participate in the CG-CAHPS survey, their willingness to participate involves self-selection and could confound the associations with CG-CAHPS composites and ratings reported here. The response rate was 28% and this may have resulted in nonresponse bias, and information about nonrespondents was unavailable. Administration data indicated that the distribution of gender and race for all adult visits during the study time interval was the same as the study participants, but participants tended to be older. While response rates are only weakly associated with nonresponse bias,²¹ the possibility of nonresponse bias remains. Nonrespondents tend to be less healthy and less positive in their evaluations of health care.²² Nonetheless, such bias is minimized in CAHPS surveys when case-mix adjustment is employed.²³ In addition, patients may not remember the details of all their care experiences in the last 6 months that are asked about on the C-G CAHPS Survey 3.0. Finally, because this study was limited to adults in one system of care, the results may be different in other clinic settings. Examination of the extent to which the findings of this study generalize to other settings is needed.

In summary, patient experience during the COVID-19 pandemic, which included a rapid increase in the use of telehealth, especially tele-video virtual care, was at least as good as or better than that prior to the onset of the pandemic. The persistence of telehealth visits as total visits returned to normal in this health system suggests that virtual care is addressing meaningful patient needs. We believe that government and commercial health plans should continue to both allow and embrace telehealth services. These results suggest that offering telehealth services to patients as part of an overall care model is well received by patients. Telehealth is often promoted to increase access to care. Our results show that access to care for telehealth visits was like other types of visits and that access to care overall was perceived to be significantly better during the pandemic than before. Access to care may have been especially meaningful to patients during this study period before widespread vaccination of the public began. In addition, we found that doctor communication and global ratings of the doctor were slightly better for tele-video visits. This may reflect a more focused and intimate experience of care. The findings of this study add to the understanding of the patient experience with telehealth and provide further indication that it can result in positive patient experiences with care.

Supplementary Information

Below is the link to the electronic supplementary material.

Funding

Hays was supported in part by a cooperative agreement from the Agency for Healthcare Research and Quality (U18 HS025920). The assistance of the UCLA Health Office of Health Informatics and Analytics is greatly appreciated.