Skip to main content
Journal of General Internal Medicine logoLink to Journal of General Internal Medicine
. 2021 Sep 7;36(11):3497–3502. doi: 10.1007/s11606-021-07061-4

Teaching Telemedicine in the COVID-19 Era: a National Survey of Internal Medicine Clerkship Directors

Bruce L Henschen 1,, Harish Jasti 2, Michael Kisielewski 3, Amber T Pincavage 4, Diane Levine 5
PMCID: PMC8423074  PMID: 34494207

Abstract

Background

Prior to the COVID-19 pandemic, telemedicine (TM) experiences in undergraduate medical education were uncommon. When students’ clinical experiences were interrupted due to the pandemic, TM education provided opportunities for students to participate in clinical care while adhering to social distancing guidelines.

Objective

To assess the prevalence of TM experiences in the internal medicine (IM) core clerkship experience prior to the COVID-19 pandemic, during interruption in clinical clerkships, and following the return to in-person activities at US medical schools.

Design

The Clerkship Directors in Internal Medicine (CDIM) survey is a national, annually recurring thematic survey of IM core clerkship directors. The 2020 survey focused on effects of the COVID-19 pandemic, including a section about TM. The survey was fielded online from August through October 2020.

Participants

A total of 137 core clinical medicine clerkship directors at Liaison Committee on Medical Education fully accredited US/US territory–based medical schools.

Main Measures

A 10-item thematic survey section assessing student participation in TM and assessment of TM-related competencies.

Key Results

The response rate was 73.7% (101/137 medical schools). No respondents reported TM curricular experiences prior to the pandemic. During clinical interruption, 39.3% of respondents reported TM experiences in the IM clerkship, whereas 24.7% reported such experiences occurring at the time they completed the survey. A higher percentage of clerkships with an ambulatory component reported TM to be an important competency compared to those without an ambulatory component.

Conclusions

The extent to which TM was used in the IM clinical clerkship, and across clinical clerkships, increased substantially when medical students were removed from in-person clinical duties as a response to COVID-19. When students returned to in-person clinical duties, experiences in TM continued, suggesting the continued value of TM as part of the formal education of students during the medicine clerkship. Curricula and faculty development will be needed to support TM education.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11606-021-07061-4.


The use of telemedicine (TM) grew substantially in response to the COVID-19 pandemic.1,2 In March 2020 alone, the number of telehealth visits increased by 154% compared to the same period in 2019.3 That rise in TM coincided with a contraction of in-person experiences for medical students in the USA.4 When medical students throughout the country were removed from clinical rotations to reduce the spread of COVID-19, medical schools responded in a variety of ways, with some suspending clinical clerkships entirely while others developed alternative experiences.5 Some healthcare organizations incorporated trainees into TM-related patient outreach programs and other ways that students could contribute to patient care remotely.6,7 Overall, the COVID-19 pandemic highlighted the need for medical trainees to become proficient in TM as part of holistic medical education.8

Even prior to the pandemic, leading medical organizations called for medical students to gain experience in TM.9 In 2019, some US medical schools offered didactic learning experiences, exercises with standardized patients, and opportunities for telemedicine patient encounters, though these experiences varied based on location.10 Most formal TM training experiences described in the literature focus on students in clinical clerkships.1012 For example, Jonas et al. described a TM-focused curriculum embedded into clinical clerkships, including didactic content and interactive patient contact. Students reported improvement in TM-related competencies and 80% reported future plans to practice TM.13 Additionally, national organizations have made online modules available to medical trainees and clinicians on TM competencies.14 However, there is great variability in the prevalence of TM curricula and experiences, with several states in 2019 reporting that no medical schools offered formal TM training.10

In response to COVID-19, many regulatory requirements governing the use of TM were relaxed, providing new opportunities for TM in medical education.15 Jumreornvong et al. proposed a framework for incorporating TM education into medical schools, highlighting the importance of formal training to provide future physicians with the competencies to implement safe, ethical, and legal TM practices.8 However, the number of medical students who received formal education or experiences in TM during the COVID-19 pandemic has not been quantified.

This study is based on a nationally representative survey of internal medicine (IM) clerkship directors (CDs), conducted to understand the prevalence of TM education and clinical experiences for clinical clerkships in relation to the COVID-19 pandemic. We assessed what training was provided to students and how TM competencies are assessed, if at all; and described the challenges and best practices in TM education following the onset of the COVID-19 pandemic.

METHODS

Survey Development

Clerkship Directors in Internal Medicine (CDIM) is a charter organization of the Alliance for Academic Internal Medicine (AAIM), a nonprofit professional association that includes academic faculty and leaders responsible for third- and fourth-year undergraduate medical education (UME). CDIM has conducted annual research surveys on topics essential to UME since 1999.

On March 17, 2020, the Association of American Medical Colleges (AAMC) recommended that students leave clinical clerkships due to the COVID-19 pandemic.4 In response, the 18-member CDIM Survey and Scholarship Committee focused the 2020 Annual Core Clerkship Directors Survey on how clerkships responded to the pandemic. Committee writing groups, comprised of faculty and faculty leaders with experience in UME, authored thematic subsections based on their relevant expertise. Following approval by the CDIM Governing Council, the survey was programmed into the web survey suite Qualtrics (Qualtrics, Provo, UT) and pilot tested. Questions were revised for clarity and content validity.

This study is based on the TM subsection of the survey and focuses on 10 questions, including multiple-choice (single-selection and select-all-that-apply), 5-point Likert scale, and open-text response options, with logical skip and display patterns (Appendix 1). The final questions asked respondents to report the prevalence of TM experiences in the IM clerkship and in other locations in the curriculum: pre-clerkship years, other clerkships, electives, and fourth-year coursework. Additional questions were included to assess what specifically students engaged in during TM encounters, the perceptions of respondents of the importance of TM as an educational tool, and the utility of using TM for assessment. To ensure validity in reporting between schools, we defined the “pre-pandemic” period as the time prior to March 2020; “clinical interruption” as the period between March 17, 2020, and June 1, 2020, in which most students nationally were not participating in in-person activities; and “clinical re-entry” as the first 6 weeks after June 1, when students returned to in-person clinical activities.

Survey Distribution

The original study population included CDIM members designated as “clerkship director” at all 142 US LCME fully accredited medical schools with CDIM membership (one designated clerkship director per school). CDIM Annual Surveys exclude medical schools with Liaison Committee on Medical Education (LCME) preliminary or provisional accreditation because those schools would not necessarily be able to provide representative data about training experiences for third- or fourth-year medical students. The population size was adjusted to 137 after removing two medical schools confirmed not to have renewed their CDIM membership as of August and after three schools were determined to have preliminary LCME accreditation. Unique participant URLs were disseminated via email invitation to the population on behalf of the CDIM Survey and Scholarship Committee.

The survey launched on August 19, 2020, and closed on October 20, including four e-mail reminders to non-respondents. No incentives were offered for participation. All email communications included voluntary opt-out links and the survey landing page included an informed consent statement and critical study information about human subject research protections. Respondents who communicated that they no longer were their medical school’s clerkship director were replaced by AAIM Surveys personnel (MK) with the current clerkship director of record or most appropriate contact (e.g., an assistant clerkship director).

Data Analysis

Upon closure, MK merged the survey dataset with the complete survey population database to incorporate demographics and medical school characteristics before de-identifying the dataset. Descriptive statistics were used to report the summary results. Pearson’s chi-square or Fisher’s exact test (alpha level: 0.05 for statistical significance) was used to conduct bivariate tests for associations between categorical variables. Tests for association between curriculum offerings at different time periods defined in the survey were conducted using paired or (where applicable) two-sample t-tests adjusted for unequal variances. Comparisons for goodness-of-fit for each period (pre-pandemic, clinical interruption, and currently) to all curriculum offerings were made using Pearson’s chi-square with Sidak-adjusted p values to minimize type I errors. Internal consistency of multiple-item Likert-scale questions was confirmed by reporting Cronbach’s α with average inter-item correlation. Data analysis was conducted in Stata 16.1 SE (StataCorp LLC, College Station, TX). Due to survey conditional logic or item non-response, denominators for certain questions or cross-comparisons of variables will not sum to the total number of survey respondents. We elected to exclude responses of “not sure” in the denominator for each analysis, consistent with prevailing methodology.16,17

The study (#20-AAIM-112) was declared exempt by Pearl IRB (US DHHS OHRP #IRB00007772) according to FDA 21 CFR 56.104 and 45CFR46.104(b)(2): (2). Only MK had access to the survey population and survey software during fielding.

RESULTS

Participant Characteristics

The survey response rate was 73.7% (101 of 137 possible respondents). There were no statistical associations between survey respondents and non-respondents based on core demographic and institutional characteristics including medical school classification (public/private), size (number of enrolled students), US Census region, or clerkship director gender (p>0.05 for all test results) (Appendix 2). Additionally, there were no statistical associations between clerkships with/without ambulatory components, or clerkships with telemedicine components, with respect the above characteristics (Appendices 3 and 4, respectively). None of these medical school characteristics were associated with differences in TM experiences detailed below.

TM Prevalence in IM Clerkships

Most respondents (98.0%; 99/101) reported an interruption from all in-person clinical rotations (66.3%, 67/101) or complete suspension of all IM clerkship activities (31.7%, 32/101) at the start of the pandemic. During the pre-pandemic period, no respondents (0/90; 11 reported “not sure”) reported a TM curriculum and/or clinical experience on the IM clerkship (Table 1). For the 67 clerkships that did not fully suspend clerkship activities during clinical interruption, 61 respondents reported which curricular content or clinical experiences in TM were offered during that period; six reported they were not sure. Of those 61, 39.3% (24/61) reported TM experiences in the IM clerkship during the clinical interruption period. At the time of survey completion, when students had returned to clinical clerkships for more medical schools, 24.7% reported TM experiences (23/93; 8 reported “not sure”).

Table 1.

Responses to the 2020 CDIM Annual Survey Question “Was / Is There a Curriculum and / or Clinical Experience in Telemedicine at Your School?”

Time period
Pre-pandemic (n=90)* During clinical interruption (n=61) Currently (n=93) p value§
No. (%) No. (%) No. (%)
Yes, in the pre-clerkship years 5 (5.6) 6 (9.8) 22 (23.7) <0.01
Yes, in the internal medicine clerkship 0 (--) 24 (39.3) 23 (24.7) <0.01
Yes, in another clerkship 4 (4.4) 25 (41.0) 49 (52.7) 0.01
Yes, in an elective 4 (4.4) 17 (27.9) 30 (32.3) <0.01
Yes, in the post-clerkship year 4 (4.4) 12 (19.7) 22 (23.7) <0.01
In any aspect of the curriculum 11 (12.2) 46 (75.4) 71 (76.3) <0.01

*An additional 11 respondents reported “not sure”

An additional 6 respondents reported “not sure”

An additional 8 respondents reported “not sure”

§Tests for goodness-of-fit (Pearson’s chi-square with Sidak-adjusted p values) across each period

One-half (50.5%; 51/101) of CDs reported that their core IM clerkship does not have an ambulatory component, which may affect how respondents perceive TM. Among CDs who reported that their IM clerkship has an ambulatory component (42/90 pre-pandemic; 33/61 during clinical interruption; 46/93 at the time of the survey), the percentages who reported TM experiences in any aspect of the curriculum were 14.3% (6/42) pre-pandemic; 72.7% (24/33) during clinical interruption; and 76.1% (35/46) currently. There was no statistical association between the presence of an ambulatory component and the prevalence of offering TM at any time period.

TM in Other Curricular Elements

CDs reported that TM experiences existed in other clerkships and electives throughout the UME curriculum prior to the pandemic, during clinical interruption, and at the time of the survey (Table 1). Respondents reported more TM experiences in any aspect of the curriculum during clinical interruption and at the time of the survey, compared to the pre-pandemic period (Table 1). TM experiences in any aspect of the curriculum during any of the periods studied were reported by 82.5% (80/97) of respondents. All goodness-of-fit comparisons for each time period were statistically significant, suggesting an association between the increased use of TM during clinical interruption and currently, compared to the pre-pandemic “baseline” period.

Types of TM Experiences

The 24.7% (23/93) of respondents who reported a TM curriculum or experience in the medicine clerkship at the time of the survey characterized how students participate in TM. The most commonly self-reported means were by conducting (87.0%; 20/23) or documenting (78.3%; 18/23) TM patient visits. Respondents reported a wide range of hours spent conducting TM visits and in didactics, lectures, and modules related to TM. Roughly one-third of respondents reported students spending 1–5 h on clinical TM encounters (39.1%; 9/23) and 1–5 h on didactics (34.8%; 8/23); 60.9% (14/23) reported students spending 10 or more clinical TM hours; one-third reported 10 h or more of didactic content (30.4%; 7/23). With respect to how TM curriculum in the medicine clerkship is delivered, the most commonly reported means were student direct participation in TM visits (78.3%; 18/23) and student observation of telehealth visits (47.8%; 11/23). For a small number of institutions, students participated in mock TM visits with standardized patients or in small group discussions related to TM (13% and 30%, respectively).

TM as Assessment

All respondents were asked to describe their perceptions of TM as an assessment tool. Among 99/101 respondents who answered this question, 35.4% (35/99) believed that a competency in TM was “important” or “very important”; 33.3% (33/99) reported “neutral”; and 31.3% (31/99) reported “unimportant” or “very unimportant.” A similar distribution of attitudes toward TM was observed when respondents were asked whether TM should be a core Entrustable Professional Activity (EPA). However, respondents recognized that TM can be used to assess many of the current EPAs (Fig. 1). Most respondents reported that TM can, to either a “moderate” or “great extent,” be used to evaluate EPA1 (history and physical), EPA2 (prioritize a differential diagnosis), EPA3 (recommend/interpret diagnostic tests), EPA4 (enter and discuss orders and prescriptions), EPA5 (document a clinical encounter), and EPA6 (provide an oral presentation of a clinical encounter) (Cronbach’s α for the responses to all items above: 0.90; average inter-item correlation: 0.34).

Figure 1.

Figure 1

Responses to the 2020 CDIM Annual Survey question “With respect to the following core Entrustable Professional Activities, to what extent can telehealth be used to assess a student’s ability to accomplish the following?”*. *Cronbach’s α (for all items): 0.8968; average inter-item correlation: 0.3431.

A higher percentage of CDs whose clerkship included an ambulatory component reported that TM is either “important” or “very important” compared to those without an ambulatory component (Table 2). Responses did not differ among self-reported attitudes about EPAs. Practically, however, many clerkships did not use TM for assessment: 18 of 43 eligible respondents (41.9%) reported that they did not evaluate any TM-related competencies and 6.3% (6/95) of respondents reported using TM assessments as part of the clerkship final grade during clinical re-entry.

Table 2.

Attitudes About Telemedicine Among Respondents to the 2020 CDIM Annual Survey, Based on Whether Respondents Had an Ambulatory Clerkship Component

Ambulatory component (n=50) No ambulatory component (n=49) Total (n=99)
No. (column %) No. (column %) No. (column %) p value*
Competency in telehealth for students on the medicine clerkship
  Very unimportant or unimportant 7 (14.0) 24 (49.0) 31 (31.3) <0.001
  Neutral 19 (38.0) 14 (28.6) 33 (33.3) 0.395
  Important or very important 24 (48.0) 11 (22.5) 35 (35.4) 0.011
That telehealth be a core Entrustable Professional Activity (EPA) for entering residency
  Very unimportant or unimportant 12 (24.0) 19 (38.8) 31 (31.3) 0.133
  Neutral 18 (36.0) 14 (28.6) 32 (32.3) 0.521
  Important or very important 20 (40.0) 16 (32.7) 36 (36.4) 0.532

*Bivariate test (Fisher’s exact)

DISCUSSION

In 2020, IM clerkship directors (CDs) responded to unprecedented challenges in medical education posed by the COVID-19 pandemic. Among many alterations to curriculum and course design, instruction and clinical experiences in TM provided opportunities for students to participate in clinical care while abiding by national guidelines. In our survey, we found that the use of TM in clerkships expanded markedly in early 2020. Over time, as in-person instruction became possible, those who used TM continued it as part of their curricular experiences.

There are many explanations for this expansion of TM curricula. Clerkship directors may have perceived value in teaching TM as part of a core clerkship experience, given its rapid rise to prominence in clinical practice.18 Clerkship directors were likely limited in the clinical sites and experiences available to students early on in clinical re-introduction, thus necessitating the use of remote learning. Finally, it is possible that these attitudes, although assessed during the pandemic, were also assessed during a time of rapid flux in medical practice. Attitudes about, and opportunities for, TM may change in response to further waves of incident cases.

Among clerkships with TM experiences, CDs reported that students were engaged in directly interviewing patients and other active learning experiences critical for skill acquisition and professional development.19 As curricula in TM evolve, observatory experiences such as shadowing, reported by almost half of the survey respondents, should be replaced with active, experiential learning opportunities that can drive both learning and assessment. Training and deliberate practice, via interactive modules, standardized patients, or simulation, are necessary to ensure students acquire skills needed for active participation in TM.8 When used effectively, TM can also be an effective assessment tool, as many respondents reported.

Despite its more prominent use, many respondents also reported that competency in TM was either “unimportant” or “very unimportant” for students on the IM clerkship. Although some may have presumed that competency could be derived elsewhere in a UME curriculum, a similar percentage also perceived TM to be unimportant as an EPA. We found that CDs from clerkships with an ambulatory component reported perceiving TM as an important competency. CDs who rely more heavily on inpatient experiences may view TM as less important as a student competency as this clinical modality is more difficult to replicate in the inpatient setting. For clerkships without an ambulatory component, it may be important to ensure that TM competencies are met in other clinical experiences throughout the UME curriculum.

The strengths of our study include a survey conducted in the midst of a pandemic with a response rate that is broadly (statistically) representative of the survey-eligible population. Our work contains certain limitations. First, the final survey population size was slightly reduced to 137 of 142 LCME fully accredited US medical schools with CDIM membership. Second, the pandemic’s compression of training periods and clerkship length at many institutions might have resulted in some degree of measurement error due to questions asking about modifications to training experiences in a condensed timeframe. Specifically, it is possible that questions about curricular offerings prior to the pandemic were subject to recall bias, although we note that 81% of responding core clerkship directors reported to be in their position for at least 3 years at the time of survey completion. Finally, we recognize that there was some degree of item non-response or “not sure” responses at different time periods, which might have slightly reduced the statistical power of the results. However, the rate of item non-response or “not sure” was roughly constant (about 90% completion for those items at all time periods) and thus unlikely to change our overall results.

Overall, in response to COVID-19, many IM clerkships adapted by including TM as part of the curricular structure. Many continued teaching about TM after students were fully reincorporated into the clinical environment, demonstrating its potential as an educational tool in the pandemic era. Of those clerkships that used TM, most directly involved students in the visit itself, providing experiential training in a skillset that will likely be an important aspect of their future training and practice environments. Finally, clerkship directors recognize the potential for TM as a setting for assessment of clinical skills, including direct observation. However, many did not utilize this modality for assessment. Further research should be conducted regarding the use of TM as a means of assessing student progress in pre-established EPAs and in TM-specific competencies. As TM is likely here to stay in the context of clinical care,2022 it is critical to ensure that we are preparing students appropriately for TM in residency and beyond.

Supplementary Information

ESM 1 (39.5KB, docx)

(DOCX 39 kb)

Acknowledgements

We wish to thank our colleagues from the CDIM Survey and Scholarship Committee for assistance with drafting and reviewing the survey content. We extend our appreciation to the CDIM Council for its review of the survey instrument and for encouraging us to conduct this research during a challenging time, and to Jordan Ortiz of the Alliance for Academic Internal Medicine for her invaluable assistance as project staff. Finally, we thank the medicine clerkship directors who completed the survey for taking the time to provide critical data as they faced multiple competing priorities due to the COVID-19 pandemic.

Declarations

Ethics Approval

This study and its protocol (Number: 20-AAIM-112) were submitted to Pearl IRB (registered with the US Department of Health and Human Services Office for Human Research Protections as IRB00007772) for exemption determination in accordance with FDA 21 CFR 56.104 and DHHS 45 CFR 46.104 regulations and were deemed exempt according to FDA 21 CFR 56.104 and 45CFR46.104(b)(2): (2) Tests, Surveys, Interviews on 07/27/2020.

Conflict of Interest

The authors declare that they do not have a conflict of interest.

Footnotes

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ESM 1 (39.5KB, docx)

(DOCX 39 kb)


Articles from Journal of General Internal Medicine are provided here courtesy of Society of General Internal Medicine

RESOURCES