Going Virtual: Objective Structured Teaching Exercises as an Innovative Method for Formative Resident Education

Abstract

Background

The coronavirus disease 2019 pandemic forced residency programs to adapt teaching to the virtual arena. Objective Structured Teaching Exercises (OSTEs) are a simulation-based session we previously implemented in our in-person pediatric curriculum. We aimed to assess feasibility of and resident satisfaction with the transition to virtual learning for simulation-based OSTEs.

Methods

The pediatrics residency program at our hospital has a weekly academic half-day for residents where the OSTEs were held annually in person 2018 to 2019 and virtually in 2020. Surveys were collected from participating residents and faculty to compare teaching experience, feedback quality, and satisfaction with the session.

Results

Over 3 academic years, there were 159 total teaching sessions, 3 of which were OSTEs. The OSTE session was highly rated each year and was the second highest rated virtual session. Residents felt the OSTEs improved their teaching regardless of the virtual versus in-person platform (P = .77), and the quality of feedback as rated by the resident teacher was higher for virtual sessions (P < .001).

Conclusions

Transitioning the OSTE to a virtual platform was both feasible and effective when compared to the in-person OSTE. In the transition to virtual learning, educators should consider opportunities for simulation-based teaching such as OSTEs.

What's New.

Objective Structured Teaching Exercises have emerged as a tool to develop clinical teaching and interpersonal skills for residents. This study demonstrates their feasibility and acceptability in preparing residents for clinical teaching and as a model for virtual learning.

Alt-text: Unlabelled box

The coronavirus disease 2019 pandemic forced residency programs to adapt teaching curricula to the virtual arena overnight. Online learning has many potential barriers, such as “Zoom fatigue” and increased distractions.^{1 , 2} Evidence-based best practice suggests that engaging students in active learning helps overcome these barriers.³ One educational experience that heavily utilizes active learning is simulation, which is a cornerstone of medical education in-person curricula, is well received, and positively impacts learning.^{4 , 5} However, studies are only beginning to explore whether virtual simulation can overcome barriers to online learning.⁶

The pediatric residency program at our hospital has a weekly academic half-day called longitudinal curriculum (LC) utilized for postgraduate-year (PGY)-specific education. In 2018, we developed and implemented in-person, pediatric-specific Objective Structured Teaching Exercises (OSTEs) into the PGY-2 curriculum. OSTEs are a simulation of clinical teaching that have emerged as a tool to develop clinical teaching and interpersonal skills.7, 8, 9 OSTEs draw from the Objective Structured Clinical Examination (OSCE) used to evaluate medical students’ clinical skills. OSTEs and OSCEs are often used for summative assessment; however, both have utility in formative assessment to develop rather than assess skills. We implemented OSTEs to develop teaching skills early in the PGY-2 year as residents transition to leadership roles.

OSCEs have already made the transition to the virtual platform, and emerging data suggest they continue to be an effective measure of student performance.⁶ However, it is unknown whether this transition is feasible or effective for OSTEs. This study's objective was to assess the feasibility, acceptability, and perceived impact on clinical teaching of transitioning OSTEs to a virtual platform.

Methods

OSTE Setup

OSTEs were implemented into the standard, PGY-2 LC curriculum at an urban, academic, mid-sized pediatric residency program. OSTEs traditionally consist of one participant practicing their teaching,1, 2 standardized trainees (STs) portraying learners, and one faculty observer. Our session is unique in that residents switch roles halfway through the session to participate as both teachers and STs. All role groups underwent a brief, day-of orientation for their roles. Teachers met with their individual assigned faculty member to set goals for the teaching skills they wanted to practice and develop. A schema of the OSTE experience is described in Figure 1 .

Figure 1.

Breakdown of in-person and virtual OSTE experience. OSTE indicates Objective Structured Teaching Exercise.

At the beginning of cases, teachers were provided a prompt prior to entering each room (eg, teach a student in the emergency room requesting a chalk talk on abdominal pain). They then entered the room to participate in an 8-minute case simulating a teaching scenario, followed by 7 minutes of immediate feedback from the faculty and ST. Teachers rotated through 3 exercises: Feedback on a Presentation, Teaching on Rounds, and Giving a Chalk Talk.

While acting as STs, residents were given guidance on the learner they portrayed, including the level of training, knowledge base, personality traits (eg, over-confident), and standardized scripted answers to common questions. We explicitly encouraged STs to provide teachers with feedback alongside faculty.

Faculty were provided debriefing guides illustrating frameworks for delivering effective feedback relevant to the specific OSTE (eg, the advocacy-inquiry model¹⁰), topic-specific teaching strategies such as the SNAPPS model,¹¹ and how to use visual aids thoughtfully.

Case content was not altered for the virtual format. Instead of physical rooms, participants were placed in virtual breakout rooms. Each participant received a personalized email that included links necessary to complete the case, such as videos, case prompts, surveys, and debriefing materials, as appropriate. All materials (cases, ST guides, debrief guides) are available in Appendix 1.

Data Collection

OSTE participation was required for all residents scheduled to attend that week's LC; however, study participation was voluntary. Residents completed the standard, residency-administered, anonymous evaluation for LC sessions within 1 week. The survey asks residents to rate how the session compared to other LC sessions on a Likert scale from 1 (worst) to 7 (best). We analyzed sessions from July 2018 (the OSTEs’ first year) through January 2021.

Residents who opted to complete additional study-administered surveys received a $25 gift card. Faculty similarly completed a study-administered survey. All study surveys were completed during the session. These were developed through iterative expert review and piloted in a larger ongoing OSTE study (Appendix 2). Resident surveys assessed subjective impact of OSTEs on teaching ability and perceptions of teacher and ST experiences. In 2019 and 2020, feedback quality was assessed by resident teachers and self-assessed by faculty and STs via a single 100-point global assessment scale. Global scales were utilized as a feasible tool and based on evolving evidence showing global rating scales perform similarly to larger, more cumbersome checklists.^{12 , 13} Questions about interacting virtually were added in 2020. In questions asked across all surveys, 2018 and 2019 data were combined as “in-person” and 2020 data are reported as “virtual.”

Data Analysis

Two-tailed student's t test was used to compare resident LC scores by year and to compare participant experiences. The assessment of feedback was analyzed in 2 ways: first, an unadjusted mean feedback score across the 3 cases was calculated for each participant, and 2-tailed student's t test were used to compare scores. Second, 3 linear mixed models were used, one each for the teacher, STs, and faculty assessment of feedback. All models used format (in-person vs virtual) as a fixed-effect and participant giving feedback as a random-effect to account for repeated measurements of the same individual giving feedback. Thus, this model adjusted for some faculty participating in both formats. All calculations were performed using JMP 14.0 (SAS Institute Inc, Cary, NC). The Massachusetts General Brigham Institutional Review Board reviewed and deemed this study exempt.

Results

OSTE Ratings

From July 2018 through January 2021, there were 159 LC sessions; 116 in-person (2 OSTEs) and 43 virtual (1 OSTE). Each LC session was rated by 6 to 14 residents, with only the mean LC score for each session available for analysis. Excluding OSTEs, the mean in-person rating was 5.85 (out of 7) (standard deviation [SD] = 0.75) compared to 5.57 (SD = 0.91) virtually (P = .061; effect size of virtual = −0.34 by Cohen's d). The mean in-person OSTE rating was 6.31 (SD = 0.12; effect size compared to all in-person = 0.48) and the virtual OSTE rating was 6.71 (1.2 SD above the 2020 LC mean). The virtual OSTE was the second-highest scoring virtual session. There was no statistical difference between non-OSTE and OSTE sessions (5.77 vs 6.44; P = .15); however, the effect size of an OSTE was 1.12 (Cohen's d) indicating a large effect.

OSTE Participants

Across 3 years, 30 PGY-2 residents participated as teachers. Twenty-eight residents completed all surveys and one additional resident had partial responses. There were no differences in baseline characteristics across years.

While all 30 residents also participated as STs, an additional 6 PGY-3 residents on electives (2 each year) participated as dedicated STs to improve session logistics. All 36 residents completed postcase feedback surveys; however, dedicated STs did not complete the final survey given they did not participate as teachers.

Sixteen total and 10 unique faculty members (5–6 each year) participated in giving feedback and all 16 completed surveys.

Participant Attitudes Around OSTE Impact on Teaching

On a 5-point Likert scale (1 = strongly disagree and 5 = strongly agree), residents felt that OSTEs prepared them for clinical teaching equally well whether in-person or virtual (in-person 4.25 vs virtual 4.33, P = .77) and the feedback debrief after the case was effective at improving teaching (4.35 vs 4.56, P = .38). All 8 in-person (2019) residents and all 10 virtual residents thought participation as a ST was educational (not asked in 2018).

Quality of Feedback

The unadjusted quality of feedback perceived by the residents was 93.0 (SD = 7.6) in the virtual format versus 80.0 (SD = 13.0) in-person (P = .015) with similar findings in the linear mixed model (92.7 vs 80.1, estimated effect of virtual format = 8.4; 95% confidence interval [CI] 5.4–11.4; P < .001). There was no difference in the faculty's unadjusted self-assessment of the feedback (69.2 [SD = 12.7] vs 62.1 [SD 2.5]; P = .26); in the model, feedback was statistically higher in the virtual format (mean 69.2 vs 62.2; estimated effect of virtual format = 6.3; 95% CI 3.2–9.4; P < .001). Increased statistical power was driven by 3 of the faculty members giving feedback in both years. STs’ self-assessment of feedback was not statistically different; unadjusted 67.0 virtual (SD = 18.3) vs in-person 51.6 (SD = 13.3) (P = .07) with similar results in our model (mean 68.9 vs 48.5, estimated effect of virtual format = 8.0; 95% CI -0.5 to 16.5; P = .067; Fig. 2 ).

Figure 2.

Results of 3 linear mixed models comparing in-person to virtual feedback. Each model used format (in-person vs virtual) as a fixed-effect and participant giving feedback as a random-effect to account for repeated measurements of the same individual giving feedback across cases and formats. The estimated effect of the virtual formal and P value are reported for each model.

Virtual Platform Barriers and Opportunities

Eight of 9 responding residents who participated virtually felt that the virtual platform was not a barrier to learning during OSTEs; all 10 STs and 5 faculty agreed that the virtual format was not a barrier to effective feedback. Two of 10 STs felt acting over zoom was difficult. All 10 residents agreed that the virtual session improved their overall teaching skills.

Discussion

This study demonstrates the feasibility and acceptability of utilizing OSTEs to prepare residents to teach clinically, particularly highlighting the ease and potential benefit of transitioning to the virtual platform. There are many potential downsides to learning virtually, and our residents trended toward rating virtual LC sessions lower than nonvirtual; however, sessions such as OSTEs with active, simulation-based, 1:1 learning may represent an effective way to engage learners virtually and possibly provide additional benefits beyond the in-person experience. As residency programs continue to utilize remote learning, this study highlights the benefits of such learning opportunities.

Both in-person and virtual OSTEs harness the theory of deliberate practice: residents set teaching goals applicable throughout the teaching scenarios at the beginning of the LC session (eg, assessing where my learner is). They then learned through the immediate feedback they received and incorporated as they refined their approach to reach those goals over the course of the three cases.¹⁴ Residents reported that OSTEs and the associated feedback were useful for preparing them to more effectively teach on the wards regardless of in-person versus virtual format. The virtual format was specifically helpful for feedback, as faculty self-reported and resident-perceived feedback improved virtually, and ST self-reported feedback showed a trend toward improvement. One hypothesis is that teachers felt more comfortable when faculty were not physically present and could participate with their video off. Residents and faculty may also have felt added comfort delivering feedback in the safety of their own environment. Alternatively, given the virtual format occurred after the in-person format, faculty may have become more adept at providing feedback.

While virtual OSTEs have several benefits, there are limitations such as how to give feedback on nonverbal components of teaching and skills that require touch. Programs should think critically about how to give feedback on these important facets of teaching.

Notably, residents reported learning new teaching skills from acting as STs. This speaks to social cognitive theory, in which people learn from observing and imitating others’ behavior.¹⁵ The ST role provides residents the opportunity to experience 3 different teaching approaches and how learners might respond to them. Importantly, using residents as STs instead of actors nearly cut in half the number of participants necessary to run the OSTEs. It also did not require any funding to compensate actors. This method represents an opportunity to decrease barriers to OSTE implementation.

Study limitations include generalizability, as these data are from one residency program. Given the significant time commitment from faculty to participate, programs may be limited in how frequently they can offer OSTEs. LC ratings were obtained within 1 week, which may limit residents’ ability to evaluate the sessions impact more fully, although residents participate in LC during PGY-1 year so are globally aware of session quality. Finally, this study measured resident self-perception rather than an objective measurement of the impact of OSTEs on resident teaching skills.

Despite these limitations, collectively our findings suggest that both virtual and in-person OSTEs lead residents to feel more prepared for clinical teaching. Additionally, implementing simulation and/or 1:1 based learning like OSTEs into virtual curricula may provide unique benefits for learning and is well-accepted by residents and faculty. Educators should embrace simulation-based sessions, especially during virtual teaching sessions.

Future work will measure the OSTEs’ impact on real-life clinical teaching. Lastly, this work is part of a larger ongoing study using OSTEs for summative evaluation of resident teaching.