Patient and Provider Perceptions of COVID-19-Driven Telehealth Use From Nurse-Led Care Models in Rural, Frontier, and Urban Colorado Communities

Abstract

The purpose of this study was to characterize the patient and provider engagement in the sudden telehealth implementation that occurred with the onset of the COVID-19 pandemic. Patients and providers from 3 nurse-led models of care (federally qualified health centers, nurse midwifery practices, and the Nurse-Family partnership program) in Colorado were surveyed. Data from the Patient Attitude toward Telehealth survey and Provider Perceptions about Telehealth were collected. Patient respondents (n = 308) who resided primarily in rural or frontier communities were female, white, and Hispanic. Patients in urban areas used telehealth more frequently than in rural or frontier areas (P < .001). Rural/Frontier patients had significantly lower attitude scores than urban patients across each of 5 domains assessed. Telehealth modality differed across location (P < .023), with video calls, used more frequently by urban providers, and phone calls used by rural/frontier providers. Our data highlight differences in telehealth access and attitudes across rurality. These findings may contribute to future policy while addressing barriers to telehealth access and delivery.

Introduction

The use of telehealth to provide clinical care increased by 154% during the early phase of the COVID-19 pandemic (1). This sudden surge was influenced by the Centers for Disease Control's recommendation consistent with social distancing practices as well as the expansion of Medicare and Medicaid telehealth to allow beneficiaries to receive care in their homes and providers to be paid at the same rate as in-person visits (2). Among privately insured patients, telehealth claim lines increased by 4.347% between March 2019 and March 2020 (3). A rapid review of telehealth implementations between January 2004 and May 2018 revealed telehealth to be equivalent to in-person care for a wide variety of chronic and behavioral health conditions, with the potential to improve access for rural residents or those with transportation barriers (4).

Mortality among rural residents tends to be higher than their urban counterparts due to socioeconomic deprivation, provider shortages, and lack of health insurance (5). The regionalization of healthcare services has resulted in differential access to care, sometimes creating barriers to those living in underserved, rural communities. These disparities in access contribute to inconsistent and poorer healthcare outcomes. Telehealth seems to pose a solution to improve provider, patient, and family satisfaction (6), enhance quality of care and patient safety, and reduce overall costs of care. Although evidence supports telehealth as an effective option for care delivery, careful evaluation of its effectiveness, accessibility, patient adoption, and unintended consequences across ethnically, geographically, and economically vulnerable populations is required (7). The purpose of this statewide study was to characterize patient and provider engagement in telehealth. The term “engagement” refers to the process of building the capacity of patients, … as well as health care providers, to facilitate and support the active involvement of patients in their own care (8) (p. 6). The specific question addressed was, how did patients and providers perceive their engagement in the sudden shift to telehealth that occurred as a result of the COVID-19 pandemic?

Method

Design

The onset of COVID-19 set the stage for a natural experiment to evaluate the impact of the sudden shift to telehealth on patients and providers in nurse-led care models in urban, rural, and frontier communities across the state of Colorado. Three care models included were federally qualified health centers (FQHC) (9), which provide integrated primary care and mental health services, certified nurse–midwifery practices (10), and the Nurse-Family Partnership (NFP) home visitation program for first-time, low-income mothers and their babies (11).

Sample

The nurse-led models involved in this study provide access to care for approximately 50 000 low-income, ethnically diverse Coloradans with a mixture of health insurance, who live in a geographically vast area covering 104 185 square miles in the eighth largest state in the US. Eligible patient participants were at least 18 years old and experienced healthcare during the COVID-19 stay at home phase in Colorado or were established patients who canceled visits during that time. Tailored recruitment materials with a link and QR code to complete the online survey were distributed to patients by the participating sites through patient portals, email, or text messages. Printed recruitment materials were available for face-to-face visits. Behavioral and physical health providers were emailed the survey link by their site administrator. Patients and providers gave their informed consent for study participation. Survey participants were assured their choice to participate would not affect their treatment (for patients) or employment (for providers). Across all sites, 389 patients started the survey (clicked on the link) and 308 of those (79.2%) consented and completed the survey. All surveys were completed anonymously, and no personal health information or personally identifiable information was collected.

Procedure and Instruments

Survey data were collected between April and May 2021 using the secure Research Electronic Data Capture (REDCap) data management system hosted at the University of Colorado. Patients 18 years and older were asked to complete the PAtient Trust Assessment Tool (PATAT), a 25-item, valid and reliable survey that measures patient trust in the use of telemedicine services, with Cronbach's alpha ranging from .79 to .91 and significant predictive modeling with an R² of .68 (12). The 5 key domains for the patient survey included Trust in the care organization (patient's belief that the healthcare organization acts in the patient's best interests); Trust in care professional (belief that the health care provider acts in the patient's best interests); Trust in treatment (belief that the treatment they are receiving is effective); Trust in technology (belief that a specific technology is safe and secure); and Trust in telehealth service (belief that a specific telehealth system is effective, in this case phone, vs computer-based technology or smartphone). Patients rate individual items such as “When I use telehealth, I am in control,” on a 5-point scale with 1 = Disagree and 5 = Agree.

Clinician perception of telehealth

To assess clinician attitudes and perceptions regarding the use of telehealth technology, we employed the Clinician Perception of Telehealth survey (13). This instrument, developed by researchers at the MGH Center for Teleheath and the Mongan Institute Health Policy Center, consists of 12 items describing patient encounters (such as “ability to communicate effectively”) for which the provider rates whether a virtual visit or office visit is better, or no difference exists.

Patient Survey Translation and Cultural Adaptation

Due to the high number of Spanish speakers in the targeted patient population, it was imperative to provide a linguistically appropriate survey for these patients. The use of the PATAT required a thorough method of translation and cultural adaptation to reach equivalence and conceptual consistency between the original and the translated version of the survey (14,15). The process described by Gance-Cleveland and colleagues (16) was utilized for this adaptation. Consent forms, recruitment materials, and surveys were originally translated by a contracted certified translator. Spanish-translated surveys were back-translated into English, and later corroborated and adapted linguistically through an iterative process. Two bilingual nursing scientists with expertise in this research as well as working with the Spanish-speaking communities in Colorado, discussed each iteration to provide the best translation given the intent of the question and possible audience.

Institutional Review Board Approval

This study was designed to comply with ethical standards for human subjects’ studies and approved by the Colorado Multiple Institutional Review Board (protocol # 20-1513).

Statistical Analysis

Data were extracted for analysis and assessed for data integrity upon completion of data collection. Patient and provider responses were extracted from REDCap. All statistical tests were conducted on de-identified data using R statistical software (version 4.1.1) (17). Demographic characteristics were summarized using descriptive statistics and assumptions were checked for all statistical tests. For continuous variables, mean values and standard errors or medians were presented, while percentages were tabulated for categorical data. Comparisons between urban versus rural/frontier (aggregated due to low numbers) patients and providers were evaluated (χ² and 2-sample t test), with nonparametric statistical methods used when needed (Fisher exact test, Kruskal-Wallis Test) due to non-normal/skewed distributions and/or limited sample size. Reported P values were corrected for multiple testing using the Benjamini-Hochberg False Discovery Rate (FDR) method. Post hoc analyses were evaluated to further dissect key results, as described below.

Results

Patient Characteristics

Patients from the 3 nurse-led care models lived in geographically diverse rural, frontier, and urban Colorado counties. As shown in Table 1, almost 2 of every 3 patient respondents reported their county of residence as rural/frontier (67.3%). Over three-fourth of respondents were females (79.5%), 18.5% were males, and 2.0% identified as gender diverse. The gender distribution of females among rural/frontier respondents (73.5%) was significantly different than in urban settings (91.8%, P < .001). The racial distribution was similar among the location categories, with most survey participants being of white race (89.3% vs 86.7%). Overall, 11.6% identified as being multiracial and people of color (American-Indigenous, Asian, Black, and Pacific Islander were aggregated due to low numbers), which is parallel to the overall racial distribution in Colorado. The ethnic distribution among survey respondents also paralleled the overall population in Colorado, with around 30% of the respondents who identified as Hispanics or Latino across geographic locations. Survey participants ranged in age from 20 to over 60 years old, with a younger population of respondents in the urban clinical settings: 44.0% of urban patients versus 10.7% of rural/frontier patients were 20 to 29 years old, while 7.0% of urban patients versus 39.0% of rural/frontier patients were over 60 years old. The majority (96.1%) completed the survey in English and spoke English at home (93.4%), irrespective of location. Of note, although almost twice as many patients spoke Spanish at home in urban than rural/frontier areas (9.1% vs 5.4%), this difference was not statistically significant. Most patients lived less than 30 min from their clinic, with no statistical differences in distance from clinic across location (78.6% and 73.3%). Patients served by the NFP model of care were not included in this analysis as their visits typically occurred in the home.

Table 1.

Characteristics of Patients Served Across 3 Nurse-led Care Models in Rural and Urban Settings (n = 308).^a

Characteristic	Total		Rural/Frontier				Urban		P Value
	N	%		n	%	n		%
Gender	303			204	67.3	99		32.7	<.001
Female	241	79.5		150	73.5	91		91.8
Male	56	18.5		52	25.2	4		4.1
Gender Diverse	6	2.0		2	1.0	4		4.1
Race	267			177		90			.723
Multiracial and people of color (American Indigenous, Asian, Black, Pacific Islander)	31	11.6		19	10.7	12		13.3
White	236	88.4		158	89.3	78		86.7
Ethnicity	296			196		100			.951
Hispanic	88	29.7		59	30.1	29		29.0
Non-Hispanic	208	70.2		137	69.9	71		71.0
Survey language	308			207		99			.627
English	296	96.1		200	96.6	90		90.9
Spanish	12	4.9		7	3.4	5		5.0
Language at home	304			205		99			.457
English	284	93.4		194	94.6	90		90.9
Spanish	20	6.6		11	5.4	9		9.1
Age	305			205		100			<.001
20-29 years	66	21.6		22	10.7	44		44.0
30-39 years	63	20.7		32	15.6	31		31.0
40-49 years	51	16.7		37	18.0	14		14.0
50-59 years	38	12.5		34	16.6	4		4.0
Over 60 years	87	28.5		80	39.0	7		7.0
Travel time to visitb	243			187		56			.627
<30 min	181	74.5		137	73.3	44		78.6
39-59 min	47	19.3		38	20.3	9		16.1
>60 min	15	6.2		12	6.4	3		5.4
Telehealth encounter	307			206		101			<.001
No	77	25.1		65	31.6	12		11.9
Yes	230	74.9		141	68.4	89		88.1
Telehealth encounter type	229			141		88			<.001
Phone call	130	56.8		110	78.0	20		22.7
Video visit	99	43.2		31	22.0	68		77.3
Desktop	10	10.1		7	22.6	3		4.4
Laptop	31	31.3		6	19.4	25		36.8
Smartphone	53	53.5		15	48.4	38		55.9
Tablet	5	5.0		3	9.7	2		2.9

Bold type represents a statistically significant result.

^a Fisher exact test was used for Survey Language and Gender due to limited sample size. The χ² test was used for all other categorical variables. A 2-sample t test was used for age, which was treated as an ordinal variable. Some variables were collapsed to be binary comparisons: (1) Race—White versus non-white, (2), Gender—female versus male, and (3) Travel Time—<30 min versus ≥30 min. Presented P values were corrected for multiple comparisons using the Benjamini-Hochberg False Discovery Rate (FDR) method.

^b NFP modality involves solely visits at home, therefore the question “travel time to visits” was not captured in the survey for NFP patients.

Patients’ Engagement With the Virtual Visit

A significantly higher proportion of urban patients reported having a telehealth encounter in the previous year compared to rural/frontier patients (88.1% vs 68.4%, P < .001) (Table 1). Among those patients who had a telehealth visit, most participated via phone call in rural/frontier areas (78.0%), which was significantly higher than phone telehealth visits in urban areas (22.7%, P < .001). Conversely, a video telehealth visit occurred significantly more frequently in urban than rural/frontier areas, with smartphones and laptops (55.9% and 36.8%) being the technology most frequently used for video visits in urban areas. There was a significantly lower proportion of urban patients (11.9%) who did not have a telehealth visit in the previous year when compared to rural/frontier patients (31.6%, P < .001). Of the 77 patients (25.1%) who did not have a telehealth encounter in the previous year, the main reasons for not engaging in telehealth were “preference to seeing the provider in person” and “telehealth not being an option for me.”

Patient Trust Assessment Tool Scores

Overall, PATAT scores were associated with location (Urban, Rural/Frontier), as depicted in Figure 1. Rural/Frontier patients had significantly lower PATAT scores than urban patients across each of the 5 domains: Trust in the care organization (median urban score = 5.0 vs median rural/frontier score = 4.2, P < .001); Trust in care professional (4.8 vs 4.2, P < .001); Trust in treatment (4.4 vs 4.0, P < .001); Trust in technology (4.5 vs 4.0, P = .002); and Trust in the telemedicine service (4 vs 3.7, P = .001). Although these last 2 scores were calculated similarly, patient responses were specific to the type of telehealth service they utilized, either phone or video telehealth visits.

Figure 1.

Patient perceptions regarding trust in telehealth services provided by nurse-led sites by place of patient's residence (location). Each P value tests if the urban distribution is significantly different than the rural/frontier distribution for each PATAT score using the Kruskal-Wallis test. Missing data due to lack of self-reporting varied by item. Actual counts per paired variable score are frontier/rural (N = 186-191) and urban (N = 77-88). Presented P values were corrected for multiple comparisons using the Benjamini-Hochberg False Discovery Rate (FDR) method.

We hypothesized that the significant associations of location with Trust in technology (P = .003) and Trust in the telemedicine service (P = .001) may be confounded by age or visit type (phone vs video), as our rural/frontier patient sample was significantly older (P < .001, Table 1) and had much higher use of phone over video visits (P < .001, Table 1) compared to urban patients. To evaluate these possible confounding effects, we conducted post hoc analyses using quantile regression (specifically comparing medians), due to the highly skewed nature of the Trust in technology and Trust in telemedicine service scores. Trust in technology and Trust in telemedicine service were modeled separately as dependent variables, and age, visit type (phone vs video), and location (rural/frontier vs urban) were modeled as independent variables. Multivariable quantile (median) regression models were considered separately for each outcome. None of the variables were associated with Trust in technology and only visit type was significantly associated (P < .05) with Trust in telemedicine service, in the multivariable models.

Provider Characteristics

Of eligible providers (physicians, advanced practice registered nurses, registered nurses, pharmacists, and counselors) who conducted at least one virtual visit during the COVID-19 pandemic in the previous year, 88 (28.8%) completed the online surveys (only provided in English). As shown in Table 2, most respondents were white (94.2%), non-Hispanic (84.1%), and spoke English at home (97.7%), with no significant differences between urban or rural/frontier areas of residence. Most respondents were female (87.5%), with a slightly higher representation in urban versus rural/frontier settings (96.2% vs 74.3%, P = .04). Although providers in urban areas seem to be slightly younger than those in rural/frontier areas (36.5% vs 11.8%, respectively, were in the 30-39 years range), this difference was not statistically significant. With no differences across location, most respondents provided clinical services (71.6%) rather than behavioral, pharmacy, or other services, and most had offered telehealth in the past year (97.7%). The modality of telehealth services was significantly different across location (P = .023), with urban providers using video calls more frequently than phone (77.4% video vs 42.9% phone), and rural/frontier providers mainly using phone calls (57.1%). Additionally, urban providers conducted the bulk of their visits over telehealth (75.4% provided >51% of their visits over telehealth), while the frequency of visits provided via telehealth was significantly lower among rural/frontier providers (45.7% provided >51% of their visits over telehealth, P = .023).

Table 2.

Characteristics of Providers Participating in the Telehealth Surveys (n = 88).^a

Characteristic	Total		Rural/Frontier		Urban		P Value
	N	%	n	%	n	%
Gender	88		35		53		.040
Female	77	87.5	26	74.3	51	96.2
Male	11	12.5	9	25.7	2	3.8
Race	86		34		52		.797
White	81	94.2	31	91.2	50	96.2
non-white	5	5.8	3	8.8	2	3.8
Ethnicity	86		34		52		.983
non-Hispanic	74	84.1	28	82.4	46	86.8
Hispanic	14	15.9	7	17.6	7	13.2
Language at home	88		35		53		.961
English	86	97.7	35	100.0	51	96.2
Spanish/English	2	2.3	0	0.0	2	3.8
Age	87		34		53		.961
20-29 years	6	6.9	3	8.8	3	5.8
30-39 years	23	26.4	4	11.8	19	36.5
40-49 years	26	29.9	10	29.4	16	30.8
50-59 years	17	19.5	9	26.5	8	15.4
Over 60 years	14	16.1	8	23.5	6	11.5
Provider type	88		35		53		1.000
Clinical	63	71.6	25	71.4	38	71.7
Behavioral health	14	15.9	7	20.0	7	13.2
Pharmacy	1	1.1	1	2.9	0	0.0
Other	10	11.4	2	5.7	8	15.1
Provide telehealth encounter	88		35		53		1.000
Yes	86	97.7	34	97.1	52	98.1
No	2	2.3	1	2.9	1	1.9
Telehealth encounter type	88		35		53		.023
Phone call	32	36.4	20	57.1	12	22.6
Video visit*	56	63.6	15	42.9	41	77.4
Smartphone	31		8		23
Laptop	41		9		32
Desktop	12		5		7
Tablet	7		5		2
Percent of visits using telehealth	88		35		53		.023
<25%	14	15.9	12	34.3	2	3.8
26%-50%	10	11.4	2	5.7	8	15.1
51%-75%	19	21.6	7	20.0	12	22.6
100%	37	42.0	9	25.7	28	52.8

Bold type represents a statistically significant result.

^a Fisher's exact test was used for Race, Gender, Home Language, and “Do you provide telehealth” due to sample size constraints. The χ² test was used for all other categorical variables. The t test was used for age, whose categories were treated as ordinal values. Some variables were collapsed to be binary comparisons: (1) Race—White versus non-white, (2), Service type—Clinical versus nonclinical, and (3) Percentage of Telehealth—<50% versus >50%. Presented P values were corrected for multiple comparisons using the Benjamini-Hochberg False Discovery Rate (FDR) method.

When asked about their perception of whether the telehealth visit was equal or better than the in-person visit, there were no significant differences in provider responses across location for the various topics asked: Timely access to follow-ups; Efficiency of Visit; Length of visit required to assess and treat patient; Workflow outside of the visit (before and after); Ease of scheduling follow-up visits; Continuity of care and coordination with other providers; Ease of accessing patient record during the visit; The ability to see a physical problem; The ability to communicate effectively; Personal connection I feel with the patient; Patient access to care; and Overall quality of the visit.

To better characterize clinicians’ preferred aspects of both office and virtual visits, we conducted a post-hoc analysis testing if the proportion of providers preferring Virtual Visits versus Office Visits was significantly different for each clinician perception question (Figure 2). Differences for each question were tested using χ² goodness-of-fit test assuming a multinomial distribution, conditioned on the number of providers who reported “No Difference” for each question. In this analysis, providers significantly preferred office over virtual visits with respect to the quality of the visit, the connection with the patient, the ability to communicate effectively, the ability to see a problem, continuity and care coordination, and the appropriate length of the visit (3-100-fold, all P < .001). Conversely, providers reported that ease of access to patient records, scheduling follow-up visits, and timely follow-ups were all better accomplished during virtual visits than office visits (2-4.5-fold higher, all P < .001).

Figure 2.

Clinician perceptions—percent of providers who thought office versus virtual visit was better for each visit feature. Given P values test if the proportion of providers (N = 83) who prefer Virtual Visits is significantly different than the proportion of providers who prefer Office Visits for each question. A χ² goodness-of-fit test assuming a multinomial distribution, conditioned on the number of providers who reported “No Difference,” was used to test this hypothesis for each question. *P values were conducted as post hoc analyses so were not corrected for multiple testing.

Discussion

The regionalization of healthcare services has resulted in differential access to care, sometimes creating barriers for those living in underserved, rural communities. These disparities in access contribute to inferior healthcare outcomes. Telehealth poses a solution to improve patient and provider satisfaction, increase measures of quality of care and patient safety, and reduce overall costs of care. Results from patients who received care through 3 nurse-led care models across Colorado demonstrated more phone versus video visits among rural and Hispanic populations. The ethnic distribution among the survey respondents related to the overall population in Colorado, with 32.7% of the respondents who identified as Hispanics or Latino in both rural/frontier and urban areas. The preponderance of female respondents (79.5%) was not surprising considering 2 of the 3 models of care were directed toward maternal health and a recent study documented women used telemedicine in primary care significantly more than men (18).

Although the use of telephone is consistent with research on low-income populations (19), it raises important policy implications for broader use of telehealth services. Media reports during the pandemic indicated that many families in rural settings may have only had one phone which was used not only for communication and health visits but also for remote schooling of children. In addition to the lack of available equipment, the availability of broadband or cell coverage is also an issue in remote areas (20). The finding that trust in telehealth was lower among patients who used phone versus video further substantiates the need to provide consistent, high-quality video capability for those in rural areas.

The post hoc quantile regression results suggest differing Visit Type use drove the Location by Trust in telehealth service association previously observed and looks to be more important in determining a patient's trust in telehealth service score than age and location. Specifically, video users had significantly higher trust in telehealth service scores than phone users (coefficient = 0.333, P = .015).

From a provider perspective, there were no significant differences concerning perceptions of telehealth by location. Further, providers across the state were consistent in identifying visit features that worked well virtually (such as improved ease of follow-up), as well as those for which an office visit is preferable (including visits requiring a physical exam) as is consistent with current literature (21).

Limitations

A few limitations of the study are noteworthy. First, self-reported assessments were used; however, both were used in previous studies. Second, results are limited to patients providing data through an electronic survey, rather than all patients served, therefore, the perception of overall access and ability to use technology might be biased. Our attempted solution was to offer financial support to clinical sites to facilitate data collection when patients presented for care. However, we were not able to successfully reach those patients for whom telehealth was not a solution. Third, given the small numbers in frontier counties, patients in rural and frontier settings were aggregated, which might not necessarily demonstrate they were demographically identical. Moreover, the evaluation focused on data from nurse-led care models, which may not generalize to care received by all patients in the state.

Conclusions

As we emerge from the pandemic and analyze the lessons learned, telehealth can be beneficial, but it must be available to the most vulnerable. Access to Internet services has been referred to recently as a social determinant of health (22). The explosion of telehealth during the COVID-19 pandemic exposes the need for innovative and comprehensive approaches to address disparities created by the digital divide. Using a health equity lens, policy makers must consider equitable access and work toward innovative care solutions that are available to all.