Improvement of Medical Student Performance in Telemedicine Standardized Patient Encounters Following an Educational Intervention

Abstract

Background and Objectives:

The Association of American Medical Colleges identifies telemedicine competence as an important skill for graduating medical students, but which educational methods are effective in improving student performance is unclear. We aimed to assess the impact of two educational interventions on student performance in telemedicine standardized patient encounters.

Methods:

Sixty second-year medical students participated in the telemedicine curriculum during their required longitudinal ambulatory clerkship. Students first completed a preintervention telemedicine standardized patient (SP) encounter in October 2020. They subsequently were assigned to two intervention groups (ie, a role-play intervention, N=30; a faculty demonstration, N=30) and completed a teaching case. In December 2020, they completed a postintervention telemedicine SP encounter. Each case was a unique clinical scenario. SPs scored the encounters across six domains based on a standardized performance checklist. We compared the median scores for these domains and the median total score pre- and postintervention (using Wilcoxon signed rank and rank-sum tests) and the difference in median score by intervention type.

Results:

Students scored highly in history-taking and communication performance but had low physical exam (PE) and assessment/plan scores. Postintervention, median scores in PE (ie, median score difference 2, interquartile ranges [IQR] 1-3.5, P<.001), assessment/plan (ie, median score difference 0.5, IQR 0-2, P=.005), and overall performance improved significantly (ie, median score difference 3, IQR 0-5, P<.001).

Conclusions:

Early medical students had low performance at baseline in telemedicine PE and assessment/plan skills, but both a role-play intervention and faculty demonstration led to significant increases in student performance.

INTRODUCTION

Telemedicine, “the use of electronic information and communications technologies to provide and support health care when distance separates the participants,” 1 is an increasingly used form of health care delivery. 2 The Association of American Medical Colleges (AAMC) identified the need for telemedicine training to ensure a supply of competent providers. 3 Prior assessments have demonstrated gaps in trainee telemedicine skills, 4, 5, 6 highlighting the need for foundational education early in training. Although nearly 90% of surveyed medical schools reported incorporating telemedicine education into their 2020-2021 curricula, 7 limited data on learner-specific needs are available.

Proposed methods for telemedicine education include asynchronous teaching, discussion groups, standardized patient (SP) encounters, and hands-on experience. 8 Though which educational interventions can improve performance is unclear, several studies have demonstrated increased self-reported preparedness and confidence with telemedicine curricula. 9, 10, 11

We aimed primarily to assess the effect of an educational intervention on performance outcomes in SP telemedicine encounters and secondarily to assess differences in performance across two educational interventions.

METHODS

We performed a prospective cohort study of second-year medical students at Johns Hopkins University School of Medicine during their required longitudinal ambulatory clerkship, which consists of weekly 4-hour ambulatory clinics that include in-person and telemedicine encounters. Sixty students participated in the virtual telemedicine sessions during the clerkship (Figure 1).

Figure 1 . Schematic of Telemedicine Sessions.

With support from the Johns Hopkins Telemedicine Consortium, the clerkship course directors developed the materials for the sessions, including three unique cases (ie, two cases for the pre- and postintervention telemedicine SP encounters and a teaching case), performance checklists for the SP encounters, and educational interventions. Performance checklists for each SP encounter included content items within six domains: (1) history of present illness (HPI); (2) review of systems (ROS); (3) medical history, including past medical, social, and family histories (Hx); (4) physical exam (PE); (5) assessment/plan; and (6) interpersonal communication skills (ICS). Three nonaffiliated ambulatory physicians reviewed the material to ensure that the cases were equivalent in difficulty with appropriate content items. The educational interventions, role-play, and demonstration were selected because they are appropriate educational methods for skills-based objectives and require minimal resources. 12

Students completed the preintervention SP encounter in October 2020. Immediately following the encounter, they were randomly assigned to two intervention groups to complete a virtual teaching case. Group 1 completed a role-play session and Group 2 completed a faculty-led demonstration. During the faculty-led demonstration, a facilitator completed a telemedicine demonstration with a patient and a parent. During the role-play session, students were assigned roles as student doctor, patient, or parent and completed the role-play with the help of facilitators. All students then completed a postintervention SP encounter in December 2020. Following the pre- and postintervention SP encounters, SPs completed the performance checklist. In a 4-hour session, SPs were trained by education specialists to standardize scoring.

We described the characteristics of the two intervention groups using descriptive statistics and calculated the students’ median scores and interquartile ranges (IQR) across the six performance domains in the pre- and postintervention cases. We used a Wilcoxon signed rank test to compare performance outcomes pre- and postintervention, and a Wilcoxon rank sum test to compare the medians of the two intervention groups. We used Pearson correlation coefficient to measure the association between telemedicine experience and postintervention score. We used SAS software, version 9.4, for analysis. Two-tailed P<.05 was considered significant. The project was approved by the Johns Hopkins Medicine Institutional Review Board (IRB00270053).

RESULTS

Sixty students participated in the virtual telemedicine sessions between October and December 2020; the students completed 57 total clinical telemedicine encounters as a part of the longitudinal ambulatory clerkship during this time (Table 1).

Table 1. Distribution of Student Longitudinal Ambulatory Experiences by Intervention Group.

	Group 1 (N=30)		Group 2 (N=30)
	n (%)		n (%)
Setting
Academic	15 (50)	-	18 (60)	-
Community	15 (50)	-	12 (40)	-
Specialty
Internal medicine	10 (33)	-	16 (53)	-
Internal medicine: pediatrics or family medicine	9 (30)	-	2 (7)	-
Pediatrics	10 (33)	-	6 (20)	-
Other*	1 (3)	-	6 (20)	-
Ambulatory experience	n	Median (IQR)	n	Median (IQR)
Patient telemedicine encounters**	30	1 (0-3)	27	0 (0-2)

Abbreviations: IQR, interquartile range

^*Surgical or medical subspecialty clinic

^** Patient telemedicine encounters refers to the total number of video visit encounters in which students actively participated during their 4-hour ambulatory clinics.

In the preintervention case, students received the highest median scores in HPI, ROS, Hx, and ICS, and the lowest median scores in PE and assessment/plan (Table 2). Postintervention, the scores of 73.3% of students increased and 23.3% decreased. The median postintervention score significantly increased in PE (ie, median score difference 2, IQR 1-3.5, P<.001), assessment/plan (ie, median score difference 0.5, IQR 0-2, P=.005), and overall performance (ie, median score difference 3, IQR 0-5, P<.001). We found no significant difference in median scores between intervention Groups 1 and 2 for all performance domains. We also found no significant correlation between the overall postintervention performance and the number of patient telemedicine encounters (r=0.0040, P=.98).

Table 2. Median Difference in Performance Scores for Telemedicine Standardized Patient Encounters Pre- and Posteducational Intervention.

Performance domain (maximum score)	Preintervention median score (IQR)	Postintervention median score (IQR)	Median score difference [p re - post] (IQR)	P value *
History of present illness (6)	5 (5-6)	5 (5-6)	0 (–1-1)	.7949
Review of systems (6)	3 (2-4)	3 (2-4)	0 (–1-2)	.6087
Medical history (6)	5 (4-5)	5 (4-5)	0 (–1-1)	.9671
Physical exam (7)	1 (0-2)	3 (2-4.5)	2 (1–3.5)	<.0001
Assessment/plan (4)	1 (0-2)	2 (1-3)	0.5 (0-2)	.0047
Interpersonal communication skills (8)	8 (8-8)	8 (8-8)	0 (0-0)	.7961
Total (37)	23 (21–25)	26 (23-28)	3 (0-5)	<.0001

Abbreviations: IQR, interquartile range

^*Calculated byWilcoxon signed rank test

DISCUSSION

The AAMC states that, by graduation, a competent medical student “obtains history,” “conducts [an] appropriate physical examination” and provides a “clinical assessment and treatment plan during a telehealth encounter.^” 3 In our population, second-year medical students achieved these competencies in history-taking but not in PE or assessment/plan domains at baseline. Performance in these domains improved with role-play and faculty-led demonstrations but was not associated with clinical telemedicine experience. Because both interventions were effective, either can be feasibly implemented in low-resource, virtual learning environments.

The PE is an important educational domain, especially because the virtual PE differs from the in-person PE. 13 Students previously identified PE content in a telemedicine curriculum as the most useful component. 9 We note that while student performance in both PE and assessment/plan improved postintervention, median scores remained low. Similarly, in non-telemedicine objective structured clinical examinations, second-year students scored the lowest in their differential diagnosis. 14 Further research is needed to determine whether student performance is due to telemedicine-specific deficiencies or low general competency at this stage of training.

Our study has several limitations. We studied one class of medical students in one institution with a moderate sample size. Case specificity may have impacted performance; we addressed this by using matched samples and independent physician review of the cases, but alternating the case order in future studies could better address this limitation. The use of unique cases also confers a benefit, because it eliminates case memorization from contributing to performance. Additional strengths include the fact that students demonstrated improved performance 3 months after the intervention, indicating longer-term retention of these skills.

In conclusion, we are the first to describe educational interventions to improve medical student performance during telemedicine SP encounters. Future studies should assess performance trajectory throughout medical training, particularly in more advanced domains.