Leveraging Technology to Overcome Challenges with In-Person Direct Observations (DOs)

Abstract

There are several challenges to conducting quality Direct Observations in a timely manner—notably the constraint placed on the observer’s schedule. Our single-institution study found that a virtual DO format was attractive to students and allowed for efficient and effective feedback while also reducing the time burden on the observer.

Direct Observations (DOs) are critical for assessing a learners’ skill development in a competency-based medical education environment. The Liaison Committee on Medical Education (LCME) requires medical students to receive one DO on each clerkship rotation to ensure authentic insight into their clinical performance. When done well, DOs facilitate the transfer of actionable and timely feedback which fosters skill development. Despite the LCME’s emphasis on DOs and the role that they play during a student’s clinical year, previous studies suggest that DOs occur infrequently and insufficiently across the medical education continuum [1, 2]. Several barriers to DOs have been identified, including time and scheduling, as well as student stress with an observer present [3]. Some prior work has investigated the ways technology can be used to overcome these challenges; however, it has focused on the outpatient setting [4].

During the 2023 academic year, The University of Chicago Pritzker School of Medicine piloted a new approach to DOs in which technology was used to record third-year students’ clinical encounters on their pediatrics inpatient rotation. During the clerkship, a convenience sample of 65 students (of an eligible 88) used a school-owned iPad to record themselves during an interview with a hospitalized patient — showing their history-taking skills. Consent for the interviews was obtained by the medical student using the Medical Center’s media consent document. Then, students uploaded their recordings to a secure online folder using the university’s Box account. The use of an institution-owned device and secure folder on the Box website allowed for maintenance of patient safety and confidentiality. Within a week of the encounter, a faculty member watched the recordings asynchronously and the student and faculty member met for feedback about their history-taking skills. At the end, each student completed a brief survey that used a 5-point Likert scale to investigate the effectiveness and feasibility of the virtual DO format.

Figure 1 shows that most medical students identified the virtual DO experience as easy to access (85.5%) and use (82.3%), user-friendly and effective (92.0%), and an efficient way to receive feedback (90.4%). Only 12 students (18.4%) reported preferring an in-person DO experience as opposed to the virtual format. Additionally, 74.6% of students affirmed that the virtual format did not disrupt their clinical encounter.

Fig. 1.

Likert responses from student perspectives on virtual DO format

In the open-ended responses to our survey, students suggested that the virtual experience was less stressful than in-person observations and allowed the student to be more engaged. One student stated, “I completely forgot that I was being supervised and was able to immerse myself in the encounter,” indicating that removing the physical presence of an observer may enhance authentic student performance. Our work suggests that virtual DOs can potentially reduce student performance anxiety — a barrier to DOs that educators have long struggled with combating.

From the perspective of the observer, the virtual DO experience was invaluable. With the sessions pre-recorded, the observer could “watch the patient interactions at [their] own pace, pause and rewind the encounter, and formulate extensive feedback.” The virtual DO reduced the scheduling demand of the observer by allowing them to watch the encounter at a time that suited their schedule — “creating a seamless balance between my responsibilities as a clinician and an educator.” The recorded system enabled the viewer to get a true sense of the student’s clinical skills while also “avoiding the awkwardness and stress that an in-person DO can create.”

In summary, the virtual DO format provided a successful way to overcome previously identified barriers to in-person DOs. Students felt that the virtual format was easy-to-use and did not interfere with their patient interaction, while the physician noted that the virtual format helped reduce scheduling demands. This virtual DO format has the potential to improve the way medical schools conduct DOs, allowing educators to assess DOs in a nonthreatening manner while overcoming scheduling challenges of the inpatient setting. Future work will be crucial in determining other ways technology can facilitate patient observation in medical education.

Funding

This research was made possible with generous support through a philanthropic donor, who prefers to remain anonymous.

Data Availability

The data that supports the findings of this study are available upon reasonable request to the corresponding author, [AR]. The data are not publicly available in order to maintain the privacy of the students involved.

Declarations

Competing Interests

The authors declare no competing interests.