Physician Perspectives about Telemedicine: Considering the Usability of Telemedicine in Response to Coronavirus Disease 2019

Supplemental Digital Content is available in the text

ABSTRACT

Objective:

Use of telemedicine in pediatric gastroenterology has increased dramatically in response to the coronavirus disease 2019 (COVID-19) pandemic. The goal of this study was to systematically assess the usability of telemedicine in the field of pediatric gastroenterology.

Methods:

The previously validated Telehealth Usability Questionnaire was distributed to physician pediatric gastroenterologist members of North American Society for Pediatric Gastroenterology Hepatology and Nutrition. Physician demographic and practice characteristics were collected. Data were analyzed using descriptive, linear mixed-effect, and ordinary least squares regression methods.

Results:

One hundred sixty pediatric gastroenterologists completed the survey. The majority were from academic practice (77%) with experience ranging from trainee (11%) to over 20 years of clinical practice (34%). Most (82%) had no experience with telemedicine before the pandemic. The average usability score (scale 1–5) was 3.87 (σ = 0.67) with the highest domain in the usefulness of telemedicine (μ = 4.29, σ = 0.69) and physician satisfaction (μ = 4.13, σ = 0.79) and the lowest domain in reliability (μ = 3.02, σ = 0.87). When comparing trainees to attending physicians, trainees’ responses were almost one point lower on satisfaction with telemedicine (trainee effect = −0.97, Bonferroni adjusted 95% confidence interval = −1.71 to −0.23).

Conclusion:

Pediatric gastroenterologists who responded to the survey reported that the technology for telemedicine was usable, but trainees indicated lower levels of satisfaction when compared to attending physicians. Future study is needed to better understand user needs and the impacts of telemedicine on providers with different levels are experience to inform efforts to promote implementation and use of telemedicine beyond the pandemic.

What Is Known/What Is New

What Is Known

• There has been a rapid acceleration of telemedicine in response to coronavirus disease 2019.

• Telemedicine Usability Questionnaire is a validated survey to assess physician usability.

• Poor healthcare information technology has been linked to poor patient safety outcomes, clinician workarounds, and clinician burnout.

What Is New

• Pediatric gastroenterologists reported overall satisfaction with the usability of telemedicine.

• Pediatric gastroenterology fellows agreed telemedicine was useful but were less satisfied with its use as a care delivery model.

There has been a rapid acceleration in the use of telemedicine, defined as the electronic exchange of medical information for healthcare delivery, among practicing pediatric gastroenterologists in the United States in response to the emergence of the novel coronavirus disease, coronavirus disease 2019 (COVID-19) (1–3). Beginning in March 2020, directives from the US Secretary of Health and Human Services (HHS) enabled broad utilization and reimbursement of telemedicine during the COVID-19 pandemic; this led to widespread efforts to implement telemedicine capabilities and increase its use across practices and providers (4–6). For instance, one large, urban, free-standing children's hospital in the Midwest that completed 19 telemedicine visits in all of 2019 reported completing 8000 visits during the first 22 days of the COVID-19 pandemic (5). A similar sized hospital in the South with greater telehealth experience before the pandemic also reported a 100-fold increase in telemedicine visits per day in the weeks following the World Health Organization (WHO) declaration (4).

Despite efforts by specialty medical societies, such as the North American Society for Pediatric Gastroenterology Hepatology and Nutrition (NASPGHAN) and the American Gastroenterology Association (AGA) to provide subspecialty-specific education around the use of telemedicine (2,7,8), most pediatric gastroenterologists have found themselves challenged by the need to adapt to new workflows, new software, and new care delivery models (9,10). Although most pediatric gastroenterologists having now gained at least some personal familiarity with telemedicine, evaluation of their collective experience using it to deliver healthcare for children with digestive diseases has been limited.

One important concern has been ensuring that pediatric gastroenterologists are comfortable with the “usability” of telemedicine as an information technology (IT) tool. Usability is generally defined as “the effectiveness, efficiency, and satisfaction in which the intended users can achieve their tasks in the intended context of [a product's] use (11).” Poor usability in health IT has been linked to poor patient safety outcomes, clinician work-arounds, and clinician burnout (12). Another issue has been determining the appropriateness of care delivered via telemedicine for pediatric gastroenterology (GI) patients. Prior studies in pediatric GI have suggested that telemedicine may be used for established patients to supplement care delivery when there is a specific focus on symptom monitoring and mental health (1,3,13,14). With the rapid expansion of telemedicine during the pandemic, however, this care modality is now being used more broadly--in new patient encounters, as a primary care delivery method, and across all settings (2).

Notably, a baseline for the usability of telemedicine in pediatric GI has not been established. The abrupt transition to telemedicine as a primary care delivery model over the past year therefore presents an important opportunity to improve our understanding of provider perspectives about this topic. To fill this knowledge gap, we developed a survey to explore provider satisfaction with video-based telemedicine that incorporated a validated telemedicine usability survey. We aimed to identify areas of clinical care as well as characteristics of providers that may indicate where telemedicine is more or less useful for providing care to children with gastrointestinal conditions.

METHODS

Study Design

Our study was designed to investigate physician experience with telemedicine both before and during the pandemic, and to learn what physicians perceived to be the utility of telemedicine for their clinical work in pediatric GI. We developed a 62-item survey including these topics as well as basic demographic questions. The survey was designed to be deployed electronically via Qualtrics. The survey was approved by NASPGHAN leadership for distribution to all members in June 2020, about 3 months after the WHO declared COVID-19 a “global pandemic” on March 11, 2020 (15). The study was determined to be exempt by The Ohio State University Institutional Review Board (IRB).

Survey Measures

To assess telemedicine usability, the survey included the Telehealth Usability Questionnaire (TUQ), a previously validated tool designed to assess both the quality of user interface and telemedicine interaction across five domains: usefulness, reliability, ease of use, effectiveness and satisfaction. The TUQ has been shown to have strong content validity and internal consistency (16,17), and has been validated with both patients and clinicians. All corresponding survey items are rated on a 5-point Likert scale from “strongly disagree” to “strongly agree”, with lower scores indicating a more negative assessment.

Additional survey sections asked demographic questions including physician age and years in clinical practice, as well as practice characteristic questions including clinical practice type, geographical region, electronic medical record (EMR) system, and available telemedicine software. Respondents were also asked about their general comfort level with the EMR at their institution, and for the average number of telemedicine visits per week before COVID-19 emerged and during April and May 2020.

Data Analysis

Survey data were imported to R version 4.0.2 (18) for analysis. The five usability scales were each assigned a numerical value corresponding to the mean score of the respective survey items. Each physician was also assigned an overall score corresponding to the average of all usability items. Furthermore, since telemedicine visits before April 2020 occurred only for a small proportion of respondents, a dichotomous variable was created to discriminate between physicians who used telemedicine before the COVID-19 pandemic and those who did not. The geographical location for clinical practice was binned into four regions for respondents based in the United States; one additional category was created for physicians practicing outside the United States.

The TUQ was reported as responses to individual questions, overall score, and the five domains, consistent with guidance about how to use the validated tool (16). Specifically, Usefulness was defined by questions 1–3, Satisfaction by 4–7, Ease of use by 8–13, Effectiveness by 14–18, and Reliability by 19–21. Mean (μ) and standard deviation (σ) were reported for normally distributed data.

Ordinary least squares (OLS) regression analysis was used to investigate the association between physician characteristics and the survey usability scales. For this analysis, missing data were handled by listwise deletion and a P-value <0.05 was considered significant. Linear mixed effect (LME) regression was also used to model the possible clustering of responses within geographic regions. LME and OLS results were compared via likelihood ratio tests. Finally, a marginal mean calculation from the OLS models was applied to statistically significant physician characteristics, and pairwise comparisons of the estimated scale scores were performed using Holm-adjusted p-values (19).

RESULTS

A total of 160 unique pediatric gastroenterologists completed the online survey and were included in our analysis (Table 1). A majority of respondents reported working primarily in academic practices (77%). U.S. respondents (95%) were distributed across all four regions; eight respondents (5%) were from neighboring countries. Physician experience level ranged from trainee (11%) to over 20 years of clinical practice (34%). Most respondents (91%) reported themselves to be “very” or “extremely comfortable” with the primary EMR in use at their institution. Notably, the great majority of respondents (82%) reported no experience with telemedicine before the COVID-19 pandemic. A total of 24 individual telemedicine software programs were identified by respondents as preferred for this care modality.

TABLE 1.

Physician respondent characteristics (N = 160)

Characteristic	n (%)
Years of clinical practice
20+ years	55 (34)
15–19 years	20 (13)
10–14 years	20 (13)
5–9 years	26 (16)
1–4 years	21 (13)
Trainee/fellow	18 (11)
Primary clinical practice type
Academic	123 (77)
Community-based practice	12 (8)
Mixed	24 (15)
Other (please specify)	1 (1)
Geographical region
Northeast	32 (20)
Midwest	43 (27)
South	38 (24)
West	36 (23)
Outside US	8 (5)
Unknown	3 (2)
Electronic medical record (EMR) brand
Cerner	32 (20)
Epic	102 (64)
Other	19 (12)
Unknown	7 (4)
Comfort using EMR
Extremely comfortable	60 (38)
Very comfortable	79 (49)
Somewhat comfortable	12 (8)
Not so comfortable	1 (1)
Not at all comfortable	0 (0)
Unknown	8 (5)
No. of office visits before COVID-19
<10	18 (11)
10–19	23 (14)
20–29	31 (19)
30–39	31 (19)
40+	45 (28)
Unknown	12 (8)
Used telemedicine before COVID-19	29 (18)
No. of office visits in person
<10	51 (32)
10–19	36 (20)
20–29	28 (18)
30–39	16 (10)
40+	13 (8)
Unknown	16 (10)
No. of office visits via telemedicine
<10	50 (31)
10–19	48 (30)
20–29	25 (16)
30–39	10 (7)
40+	9 (6)
Unknown	18 (11)

COVID-19 = coronavirus disease 2019.

The average score across all TUQ survey questions was 3.87 (σ = 0.67) (Table 2). The highest scores were related to the usefulness of telemedicine (μ = 4.29, σ = 0.69) and physician satisfaction (μ = 4.13, σ = 0.79), while the lowest scores were related to the effectiveness of telemedicine in providing care to children with digestive disorders (μ = 3.66, σ = 0.80) and to its reliability (μ = 3.02, σ = 0.87).

TABLE 2.

Responses to items and scales from Telehealth Usability Questionnaire (TUQ), by survey domain and overall (N = 160)

Item	Mean (SD)
1. Telehealth improves my patient's access to healthcare services	4.30 (0.82)
2. Telehealth saves my patients time traveling to a hospital or specialist clinic	4.65 (0.67)
3. Telehealth provides for my patient's healthcare needs	3.93 (0.91)
Usefulness scale summary (items 1–3)	4.29 (0.69)
4. I feel comfortable communicating with the patient using the telehealth system	4.19 (0.80)
5. Telehealth is an acceptable way to deliver healthcare services	4.06 (0.93)
6. I would use telehealth services again	4.26 (0.85)
7. I am overall satisfied with this telehealth system	4.00 (0.97)
Satisfaction scale summary (items 4–7)	4.13 (0.79)
8. It was simple to use this system	4.19 (0.88)
9. It was easy to learn to use the system	4.28 (0.76)
10. I believe I could become productive quickly using this system	4.17 (0.93)
11. The way I interact with this system is pleasant	3.92 (0.96)
12. I like using the system	3.76 (1.06)
13. The system is simple and easy to understand	4.14 (0.83)
Ease of use scale summary (items 8–13)	4.08 (0.77)
14. This system is able to do everything I would want it to be able to do	3.51 (1.14)
15. I could easily talk to the patient using the telehealth system	4.03 (0.85)
16. I could hear the patient clearly using the telehealth system	3.83 (0.90)
17. I felt I was able to express myself effectively	4.08 (0.75)
18. Using the telehealth system, I can see the patient as well as if we met in person	2.86 (1.24)
Effectiveness scale summary (items 14–18)	3.66 (0.80)
19. I think the visits provided over the telehealth system are the same as in-person visits	2.66 (1.19)
20. Whenever I make a mistake using the system, I could recover easily and quickly	3.63 (0.95)
21. The system gave an error message that clearly told me how to fix problems	2.77 (1.03)
Reliability scale summary (items 19–21)	3.02 (0.87)
Overall usability (Total average)	3.87 (0.67)

Survey scales and their corresponding items. Survey items were rated on a 1- to 5-point Likert scale from “strongly disagree” to “strongly agree”, respectively. Scales summaries were calculated as means of the survey item numeric rates. A total average is also provided as an overall score for telemedicine usability.

Associations between satisfaction scores and physician characteristics were analyzed via OLS and suggested three positive correlations: between clinical experience, clinical volume and telemedicine satisfaction; between comfort with EMR use and telemedicine ease of use; and between the volume of telemedicine performed and its perceived usefulness (see Appendix, Supplemental Digital Content).

Estimated marginal means calculated from the OLS models (Table 3) indicated that satisfaction was lowest for the physicians with the highest number of weekly office visits. The ease of use of telemedicine was lower for physicians who were less comfortable with their EMR. In addition, telemedicine was estimated to be more useful by physicians who conducted between 10 and 39 weekly e-visits, although usefulness declined beyond that number of visits. Interestingly, with adjustment for multiple comparisons, these relationships did not remain statistically significant.

TABLE 3.

Adjusted estimates of Telehealth Usability Questionnaire (TUQ) domains by respondent characteristic

	Satisfaction
Characteristic	Estimate	95% CI∗
Years of clinical practice
20+ years	4.01	3.41, 4.60
15–19 years	4.08	3.39, 4.77
10–14 years	3.93	3.31, 4.55
5–9 years	4.25	3.61, 4.89
1–4 years	4.01	3.32, 4.71
Trainee/fellow	3.09	2.28, 3.90
No. of office visits/week before COVID-19
<10	4.36	3.61, 5.12
10–19	3.86	3.16, 4.56
20–29	3.79	3.14, 4.45
30–39	3.76	3.14, 4.39
40+	3.69	3.03, 4.35
Geographical region of practice
Northeast	4.05	3.42, 4.67
Midwest	3.85	3.18, 4.53
South	3.55	2.93, 4.17
West	4.13	3.48, 4.77
Outside United States	3.89	3.03, 4.76

Characteristic	Ease of use
	Estimate	95% CI∗
Comfort using EMR
Extremely comfortable	4.36	3.84, 4.88
Very comfortable	4.24	3.71, 4.77
Somewhat comfortable	3.59	2.84, 4.34
Not so comfortable	2.17	0.41, 3.92
Geographical region of practice
Northeast	3.75	3.05, 4.44
Midwest	3.66	2.91, 4.40
South	3.29	2.60, 3.98
West	3.69	2.96, 4.41
Outside United States	3.56	2.60, 4.52

Characteristic	Usefulness
	Estimate	95% CI∗
No. of telemedicine office visits/week
<10	4.00	3.49, 4.52
10–19	4.33	3.75, 4.91
20–29	4.28	3.68, 4.88
30–39	4.35	3.77, 4.93
40+	3.90	3.22, 4.58
Geographical region
Northeast	4.28	3.74, 4.83
Midwest	4.25	3.66, 4.83
South	3.93	3.39, 4.47
West	4.39	3.83, 4.96
Outside United States	4.01	3.26, 4.77

COVID-19 = coronavirus disease 2019; EMR = electronic medical record.

^∗CI = confidence interval.

Our results did show an association between years of clinical experience and satisfaction with telemedicine (Fig. 1). When comparing trainees to attending physicians, we found trainee ratings of telemedicine were almost one point lower on the satisfaction scale (trainee effect = −0.97, Bonferroni adjusted 95% CI = −1.71 to −0.23). The post-hoc test showed that all pairwise comparisons with Trainees were significant, except the one between trainees and physicians with 10–14 years of experience.

FIGURE 1.

Pairwise comparisons for adjusted estimated scores of the satisfaction scale associated to the time spent in clinical practice. Adjusted estimates calculated using all respondent characteristics as covariates.

We also found geographic differences in univariate analyses. Physicians practicing in the southern states of the United States scored telemedicine usability lower than those practicing in other regions across all dimensions except reliability; however, those differences were not statistically significant once we accounted for multiple comparisons. Moreover, likelihood ratio tests comparing our results with those of mixed-effects models did not reveal any improvements in considering a hierarchical data structure connected to the geographical locations of respondents.

DISCUSSION

Across all experience levels, pediatric gastroenterologists reported satisfaction with telemedicine as a healthcare delivery model as well as reporting it was easy to use and effective at providing care to children with gastrointestinal conditions. We found high general satisfaction with the usability of telemedicine, with the highest scoring domain of usability as useful as it relates to access to care, time savings from travel, and ability to provide for a patient's needs. There was no difference in academic versus community practice setting. As it is currently estimated that there are an average of 9.2 pediatric gastroenterologists per 1,000,000 children, with a density of pediatric gastroenterologists varying by urban versus rural location and across geographic regions (20), these findings of satisfaction and usefulness of telemedicine provide some comfort about the rapid increase in the use of this technology, particularly given the potential of telemedicine to allow children to access needed subspecialty care.

The results of our study suggest that trainee status may impact physicians’ perceptions about the usability of telemedicine as a health information technology (HIT) tool. We found that trainees agreed that telemedicine was useful, although they were definitively less satisfied with it as a health care delivery model compared to their attendings. This may be reflective of reduced availability of attendings during telemedicine visits, as shown by Mallon et al's recent study which reported that in June 2020, 76% of pediatric GI programs had fellows using telemedicine but only 27% of those programs indicated an attending was present for the duration of telemedicine appointment (21). Future study will be needed to better understand the reasons for this difference in satisfaction, as well as potentially investigate differences by year of training.

Although the majority of our respondents had no experience with telemedicine before COVID-19, these trained subspecialists had reportedly adapted to using telehealth and incorporated it into their medical practice fairly early in the global pandemic. There was no difference in perceptions about overall or domain-specific usability for respondents with and without prior telemedicine experience. Our finding of relatively high usability scores along with high rates of adoption and use of the new technology are consistent with the results of many prior studies of HIT (12).

We found that the lowest usability domain scores in our study were related to the reliability of telemedicine, and particularly to the equivalence of telemedicine with in-person office visits. Even under optimal circumstances, telemedicine can lead to unintended consequences including disrupted workflows, fragmentation of services, changes in responsibilities of providers or staff, and barriers from ill-adapted regulatory frameworks (22). During the pandemic-related telemedicine expansion, many health care organizations reported the need for staff to take on new tasks, which created confusion and the need for creating new service roles (2,23). It is unclear how these factors may have influenced providers’ perceptions of usability, especially when considered in the context of the equivalence of office visits.

Finally, our results suggest that there may be room for improvement in the graphical interface design of telemedicine software that is used to deliver care in the field of pediatric GI. Our respondents were reportedly less satisfied with how pleasant the system was to use, enjoyment of using the system, and instructions on how to recover from an error message. These findings suggest that a structured usability evaluation of the software may be needed to improve user satisfaction. As our study participants listed 24 different telemedicine platforms they had used, we were unable to assess which might have been preferred; however, with the temporary HIPAA waiver for using non-HIPAA compliant telemedicine software due to expire, hospital systems choosing to continue telemedicine may be looking for new telemedicine software. Incorporating a usability evaluation of new software may therefore inform the decision-making process around telemedicine software and increase the likelihood of provider satisfaction with both the software and this care delivery modality.

Limitations

An important limitation of this study is our response rate. Despite having approval for distribution to the NASPGHAN general body, our low response rate may be a reflection of competing demands of potential respondents who were concurrently undergoing significant changes to their general practice, or due to survey fatigue given deployment of other surveys at the same time. This survey does not address provider satisfaction as a factor of patient diagnosis. Another limitation is that the survey was not designed to assess comparative usability with respect to the 24 different telemedicine platforms and software.

CONCLUSION

Pediatric gastroenterologists responding to a survey about the usability and use of telemedicine reported that the technology was very usable, but trainees indicated lower levels of satisfaction with this modality of care delivery than did attendings. Future study will be important to learn about the impacts of telemedicine on trainees and providers as well multi-disciplinary members of the GI team including dietitians and psychologists, and to better understand user needs for this technology to inform the design of the next generation of telemedicine software. Future analysis to understand provider comfort with respect to patient diagnosis will help facilitate the use of this care delivery approach in the pediatric gastroenterology field.