The TeleHealth OSCE: Preparing Trainees to Use Telemedicine as a Tool for Transitions of Care

Abstract

Background

Telemedicine holds promise to bridge the transition of care between inpatient and outpatient settings. Despite this, the unique communication and technical skills required for virtual encounters are not routinely taught or practiced in graduate medical education (GME) programs.

Objective

To develop an objective structured clinical examination (OSCE) case to assess residents' telemedicine-specific skills and identify potential gaps in our residency program's curriculum.

Methods

As part of a multi-station OSCE in 2019, we developed a case simulating a remote encounter between a resident and a recently discharged standardized patient. We developed an assessment tool comprising specific behaviors anchored to “not done,” “partly done,” and “well done” descriptors to evaluate core communication and telemedicine-specific skills.

Results

Seventy-eight NYU internal medicine residents participated in the case. Evaluations from 100% of participants were obtained. Residents performed well in Information Gathering and Relationship Development domains. A mean 95% (SD 3.3%) and 91% (SD 4.9%) of residents received “well done” evaluations across these domains. A mean 78% (SD 14%) received “well done” within Education/Counseling domain. However, only 46% (SD 45%) received “well done” evaluations within the Telemedicine domain; specific weak areas included performing a virtual physical examination (18% well done) and leveraging video to augment history gathering (17% well done). There were no differences in telemedicine-specific skill evaluations when stratified by training track or postgraduate year.

Conclusions

We simulate a post-discharge virtual encounter and present a novel assessment tool that uncovers telemedicine-specific knowledge gaps in GME trainees.

Introduction

The transition of care between inpatient and outpatient settings represents a particularly vulnerable time for patients.^1,2 Telemedicine, the provision of health care remotely, holds promise to bridge this transition, expand access to care, and potentially reduce subsequent hospital admission.³ Among the many applications of telemedicine, the potential for real-time post-discharge surveillance makes telemedicine a timely and effective means of providing post-hospital care. In 2020, with widespread social distancing regulations due to the COVID-19 pandemic, telemedicine has become an even more essential modality for patients to access care.

Despite this, telemedicine-specific communication and technical skills required for successful virtual encounters are not routinely taught in graduate medical education (GME). While didactics and elective clinical telemedicine experiences exist in some medical schools, few are required in core curricula, and published data regarding the specific content of such offerings are limited.^4,5 Many advocate for adoption of telemedicine competencies to facilitate training of residents^6–8; however, there are few telemedicine curricula or structured assessment tools in GME.

We developed an objective structured clinical examination (OSCE) case and new assessment tool to evaluate residents' telemedicine-specific skills and identify potential gaps in our residency program's curriculum.

Methods

This study was conducted with internal medicine (IM) residents at New York University (NYU) Grossman School of Medicine, a large, urban, multisite, university-based training program. A telemedicine case was developed for the 2019 multi-station OSCE, a formative assessment conducted over several dates throughout the academic year. Participants comprised residents from NYU's categorical, primary care, and Brooklyn community health tracks.

The case consisted of a telemedicine (video) visit between a resident and recently discharged standardized patient (SP). Two experienced SPs underwent 3 hours of case- and assessment-specific instruction (provided as online supplemental material). Residents were instructed in advance with relevant SP clinical information for the 10-minute scenario (provided as online supplemental material); however, they were not primed with the assessment items. SPs and residents were stationed in different rooms and communicated via video conference. Participants had not previously taken part in any didactics or clinical experiences regarding telemedicine. Participants were debriefed at the conclusion of the OSCE program and were provided with optional post-encounter self-assessment forms.

We developed a behaviorally anchored assessment tool that evaluated core communication skills and unique telemedicine skills (Table 1). Core communication items used in our assessment tool have been previously described⁹ and are widely used to evaluate these domains. Specifically, the core communication assessment items are internally consistent in OSCE encounters across multiple samples of leaners, settings, and clinical scenarios, have demonstrated sufficient test/retest reliability and interrater reliability and show consistent patterns over time.

Table 1.

Frequency Distribution of Resident Evaluations for Each Domain and Descriptors of “Well Done” Behaviors

Domain	Checklist Item	Frequency of Each Item, % (n)			Mean % Well Done (SD)	Behavioral Descriptor of “Well Done”
		Not Done	Partly Done	Well Done
Information gathering	Elicited your responses using appropriate questions	0 (0)	9 (7)	91 (71)	95 (3.3)	Facilitated the telling of your story by asking questions one at a time without leading you in your responses
	Managed the narrative flow of your story	0 (0)	6 (5)	94 (73)		Elicited full narrative by asking questions that facilitated natural flow of story
	Clarified information by repeating to ensure understanding	0 (0)	4 (3)	96 (75)		Repeated information and directly invited you to indicate whether accurate on an ongoing basis
	Allowed you to talk without interrupting	0 (0)	1 (1)	99 (77)		Did not interrupt and allowed time to express thoughts fully
Relationship development	Displayed understanding of social situation and intent to help	0 (0)	14 (11)	86 (67)	91 (4.9)	Actions and words conveyed intention to help/concern
	Acknowledged emotions appropriately	0 (0)	12 (9)	88 (69)		Acknowledged and responded to your emotions in ways that made you feel better
	Was accepting and nonjudgmental	0 (0)	1 (1)	99 (77)		Made comments and expressions that demonstrated respect
	Used words you understood and/or explained jargon	1 (1)	8 (6)	91 (71)		Provided no opportunity for misunderstanding by avoiding or spontaneously explaining jargon
Education/counseling	Asked questions to check your understanding	0 (0)	21 (16)	79 (62)	78 (14)	Checked your understanding through specific questioning and/or asking you to repeat back information
	Provided clear explanations/information	0 (0)	8 (6)	92 (72)		Provided small bits of information at a time and summarized to ensure understanding
	Collaborated with you in identifying next steps	1 (1)	36 (28)	63 (49)		Elicited your views on next steps, shared their ideas, and mutually developed plan of action
Telemedicine skills	Confirmed patient identifiers	9 (7)	86 (67)	5 (4)	46 (45)	Asked patient to confirm name/date of birth, callback number, and location
	Used nonverbal communication to enrich communication on camera	0 (0)	6 (5)	94 (73)		Maintained eye contact with webcam throughout encounter, sat squarely in front of camera, and at appropriate distance
	Actively optimized technical aspects of the virtual encounter	76 (59)	20 (16)	4 (3)		Assessed sound quality, video quality, and backup plan if audio/video failed
	Exhibited comfort and confidence using video interface	0 (0)	10 (8)	90 (70)		Confident on camera, acknowledged and moved forward from technical glitches, and did not let video interface detract from natural conversation
	Utilized live video to augment information gathering	13 (10)	70 (55)	17 (13)		Attempted to do 2 or more: visually reconcile meds, witness reproducible symptoms, talk with onsite collateral, assess the home
	Partnered with patient to perform physical examination	82 (64)	6 (5)	12 (9)		Asked patient to perform maneuvers or access peripheral monitoring device (home blood pressure cuff, FitBit/apple watch, glucometer), followed by verbal confirmation of findings
	Maintained appropriate computer etiquette during encounter	1 (1)	0 (0)	99 (77)		Paused video or provided clear explanation while documenting, searching another website, or having another screen open for the purpose of patient care

Telemedicine-specific assessment items were developed to capture key behaviors necessary for successful virtual encounters. To generate these items, 2 authors (D.J.S. and S.R.Z.) convened focus groups with experienced telemedicine clinicians from NYU Virtual Urgent Care, Steven A. Cohen Military Family Center, and “Doctor on Demand.” One author (D.J.S.) directly observed a series of virtual visits and interviewed clinicians to generate telemedicine behavioral anchors. Our assessment tool captures similar skills described by Cantone and colleagues,¹⁰ mirroring the following key skills: evaluating/optimizing technical aspects of virtual visits, adjusting posture/camera to maintain eye contact, verbalizing actions while documenting, and acknowledging technical glitches with the interface. The assessment tool also reflects a relevant subset of proposed nursing telehealth entrustable professional activities,¹¹ proposed telepsychiatry competencies,⁸ and skills described by the American Telemedicine Association.¹²

Table 1 includes specific descriptors that represent “well done” behaviors. SPs evaluated residents across each domain with responses anchored to “not done,” “partly done,” and “well done.” Evaluations are presented as “% not done,” “% partly done,” and “% well done” for each assessment item, as well as mean % well done (SD) for each domain as a whole. Telemedicine-specific domain items were further stratified by resident training track and postgraduate year (PGY); 3×3 Pearson's chi square tests were used to assess for association of either training track or PGY with frequency of not done, partly done, and well done telemedicine evaluations.

This project met NYU's criteria for certification as a quality improvement and not a human subject research project and was exempt from institutional review board review.

Results

Seventy-eight residents comprising all PGY and 3 training tracks participated in the TeleHealth OSCE case. Evaluations from all 78 participants were obtained. When core communication domains were analyzed in aggregate, a mean 95% (SD 3.3%) of residents received well done within the Information Gathering domain, and 91% (SD 4.9%) received well done within Relationship Development. A total of 78% (SD 14%) of residents received well done within the Education/Counseling domain. There were only 2 assessment items evaluated as not done (Table 1).

In contrast, resident performance was variable within the Telemedicine domain. Only 46% (SD 45%) of residents received well done evaluations in this domain. Specifically, 24% (n = 19) of residents assessed technical barriers during the encounter, and 18% (n = 14) attempted a virtual physical examination. Only 17% of residents (n = 13) received well done evaluations for using video to augment information gathering—a key item that included virtual medicine reconciliation and discussing care plans with onsite caregivers providing collateral (Table 1). Interestingly, self-assessments, provided by a subset of residents (n = 23), demonstrated that residents felt confident with performance despite SP evaluations. Most residents (91%, 21 of 23) reported that they felt prepared for this telemedicine encounter, and only 9% (2 of 23) of residents reported that the encounter could have gone better.

Our program's training tracks reflect distinct areas of focus that may influence telemedicine proficiency; therefore, we stratified evaluations to assess whether telemedicine-specific evaluations differed by track. There were no significant associations between telemedicine skill evaluations and training track (X = 2.88, DF 4, P = .57). There were no significant associations between telemedicine skill evaluation and training year (X = 1.26, DF 4, P = .91; Table 2).

Table 2.

Mean Percentage “Well Done” Telemedicine Skills Assessments Stratified by Training Track and Postgraduate Year (PGY)

Telemedicine Skills
Resident Grouping	Mean % Well Done	P Value
Categorical (n = 44)	44	.58
Primary care (n = 24)	48
Community health (n = 10)	47
PGY-1 (n = 20)	44	.87
PGY-2 (n = 51)	45
PGY-3 (n = 7)	47

Discussion

This post-discharge telemedicine OSCE demonstrated resident achievement of core communication competencies but revealed deficiencies in several telemedicine-specific skills across all tracks and PGY levels in a large IM residency program. Residents self-assessed their telemedicine performance higher than the SPs who rated resident performance with a new behaviorally anchored assessment tool.

To our knowledge this is the first assessment of specific telemedicine skills among IM residents, such as performing a virtual physical examination, appropriately identifying patients remotely, optimizing the audio-video interface, and using video to augment history taking. Our study supports assertions that telemedicine requires distinct interpersonal and technical skills that warrant dedicated assessment and training.^7,13 Somewhat striking are our findings that trainee's self-reported confidence with telemedicine differs significantly from their objective telemedicine-specific performance. GME trainees may not recognize that telemedicine represents far more than medicine via FaceTime⁷ and requires distinct skills.

Limitations of our study include the single institution sample of participants that may not be generalizable to other settings or specialties. In addition, the case focused on a post-discharge telemedicine visit, which may not reflect skills required in virtual encounters in other settings, such as urgent care. As we did not examine interrater reliability, the 2 different SPs may have evaluated residents differently. Lastly, we had a rather low self-assessment response rate (n = 23 of 78) and thus conclusions regarding perceived telemedicine skill apply to a subset of learners.

Future studies will focus on expanding assessment of residents' telemedicine skills to different types of telemedicine OSCE encounters, for example those requiring urgent evaluation and triage and those focusing on chronic disease surveillance. These will provide further evaluation of the reliability of our assessment tool.

Conclusions

This study found that IM residents participating in a post-discharge telemedicine OSCE assessed using a new behaviorally anchored assessment tool demonstrated good core communication skills but were deficient in several telemedicine-specific skills, regardless of training year or training track. Residents did not recognize their lack of telemedicine-specific skills.