Adaptive change in simulation education: Comparison of effectiveness of a communication skill curriculum on death notification using in‐person methods versus a digital communication platform

Abstract

Background

Mandates to social distance and “shelter in place” during the COVID‐19 pandemic necessitated the exploration of new academic content delivery methods. Digital communication platforms (DCP; e.g., Zoom) were widely used to facilitate content delivery, yet little is known about DCP’s capacity or effectiveness, especially for simulation.

Objective

The objective was to compare the experience, outcomes, and resources required to implement a simulation‐based communication skill curriculum on death notification to a cohort of learners using in‐person versus DCP delivery of the same content.

Methods

We used the GRIEV_ING mnemonic to train students in death notification techniques either in person or utilizing a DCP. For all learners, three measures were collected: knowledge, confidence, and performance. Individual learners completed knowledge and confidence assessments pre‐ and postintervention. All performance assessments were completed by standardized patients (SPs) in real time. Wilcoxon rank‐sum test was used to identify differences in individual and between‐group performances.

Results

Thirty‐four learners participated (N = 34), 22 in person and 12 via DCP. There was a statistically significant improvement in both groups for all three measures: knowledge, confidence, and performance. Between‐group comparisons revealed a difference in pretest confidence but no differences between groups in knowledge or performance. More preparation and prior planning were required to set up the DCP environment than the in‐person event.

Conclusions

The in‐person and DCP delivery of death notification training were comparable in their ability to improve individual knowledge, confidence, and performance. Additional preparation time, training, and practice with DCPs may be required for SPs, faculty, and learners less familiar with this technology.

INTRODUCTION

The pandemic of 2020 brought many challenges to medical education and provided opportunities to explore and advance nontraditional instructional methods. One example is delivering academic content through digital communication platforms (DCPs) like Zoom. The use of DCPs created opportunities to bring faculty and students together from remote distances and meet mandates to “shelter in place.”¹, ², ³ The requirement for shelter in place presented a particularly difficult challenge for interactive curricula such as those delivered by simulation. Traditional in‐person medical simulation provides an immersive experience that allows students to explore medical decision making and outcomes with real‐time feedback in an environment that is safe for both learners and patients. Because of the pandemic, this method of training was unavailable to many. We modified our educational delivery method to a DCP format rather than deprive students of the needed education. The use of a DCP to remotely deliver simulation‐based experiential learning has largely been unexplored.

We used a particular method of medical simulation, rapid cycle deliberate practice (RCDP; (Figure 1), which has been demonstrated to improve residents’ clinical performance and skill acquisition in resuscitation and difficult communication.⁴, ⁵, ⁶, ⁷ This approach differs from traditional simulation experiences that typically involve the learners completing an entire uninterrupted simulation scenario, followed immediately by debriefing from faculty on their performance. In the RCDP approach, the faculty member intermittently pauses the simulation scenario, provides formative feedback, and restarts the scenario, allowing the learners to incorporate feedback immediately and demonstrate improved performance.⁸ This pause with feedback can happen several times during the RCDP scenario concluding with a much shorter debriefing period at the end (Figure 1). This approach is optimal for skills with a very scripted or linear approach.⁹ We previously applied the RCDP simulation method to GRIEV_ING,¹⁰ a validated death notification tool, to first‐year emergency medicine residents, with significant improvement in knowledge, performance, and confidence.⁶

FIGURE 1.

RCDP methodology model. RCDP, rapid cycle deliberate practice

The DCP delivery of this content was provided to fourth‐year medical school students in their final 2 weeks of medical school. This created a natural experiment that allowed us to compare two very different delivery methods, in person and DCP (Zoom), to two very similar learner groups. The purpose of this study was to compare the learner and faculty experience, outcomes, obstacles, and resources needed to implement a simulated communication skill curriculum on death notification via DCP to a cohort of remote medical students as compared to in‐person learners.

METHODS

The Michigan State University Institutional Review Board declared the study exempt, and all students consented to have their data deidentified and used in aggregate for this activity. We compared two formats of educating learners in death notification techniques using the GRIEV_ING mnemonic developed by one of the authors (CH).¹⁰ Two cohorts of subjects were taught death notification using the GRIEV_ING method. The first in‐person cohort was composed of 22 first‐year emergency medicine residents at the Indiana University School of Medicine (IUSM). The DCP cohort was composed of 12 fourth‐year medical school students in their final 2 weeks of medical school who were enrolled in a Michigan State University College of Osteopathic Medicine (MSUCOM) elective titled “Readiness for Residency, a Simulation‐based Competency Evaluation.” The IUSM group was trained in person in July 2019, and the MSUCOM group was trained via the DCP of Zoom in April 2020, utilizing identical faculty and staff (see Figures 2A and 2B).

FIGURE 2.

Rapid cycle death notification training: (A) in person; (B) DCP

A RCDP Mastery Learning Model of education was utilized for both events. Sessions began with an identical (30 minutes) prebrief of the event to occur and surveys on baseline confidence (self‐efficacy) and knowledge on the topic of death notification. Next, each participant completed a baseline performance evaluation via an initial one‐on‐one encounter with a standardized patient (SP), delivering a death notification based on information provided via a “door note.” This simulation was 10 minutes max, ended at the discretion of the participant. This was followed by 1 hour of didactic content in the form of a lecture utilizing the standardized GRIEV_ING mnemonic. Participants were formed into small groups of three to four with a dedicated faculty member and with a different SP and given 1‐hour to practice a new death notification scenario using the RCDP method. After the formative curriculum, participants completed a final summative performance assessment, which included a third encounter with a different SP, utilizing a different door note (see Figure 3). There was a total of 47 possible points on each of the performance assessments, graded in real time by the SP immediately following both the pre and postencounter. Immediately following the session, participants completed a postsurvey on self‐efficacy and knowledge. The confidence survey and knowledge assessment were developed by the authors for this study. A final course evaluation was conducted with the MSUCOM students regarding the utility and effectiveness of DCP training.

FIGURE 3.

Flow diagram of study interventions

Educational materials and assessment measures for the GRIEV_ING death notification method are open access and found on MedEdPortal.¹¹ Preparation for the in‐person event included simulation case and door note development, performance assessment tool and pre/post survey creation, development of the didactic content for delivery, scheduling of the event, SP training, preevent run through with faculty and staff, and finally conducting the event followed by an additional debrief. This required approximately 65 hours.

Significant additional preparation was necessary for the telepresence encounter utilizing a DCP. The preperformance run through required a DCP meeting to assure all devices being used by faculty and SPs from an environment outside the simulation laboratory connected effectively. Additional time was required to ensure that all SPs were familiar with the platform and had the means to record the encounters and download the recordings before exiting and then confirm the capability to load these recordings into a secure site for later review if needed. Surveys needed to be delivered to DCP learners by an online platform to be completed in a timed manner. For this study, we utilized the learning management platform available at Michigan State University, D2L. The prebrief was a separate Zoom meeting to allow all learners and faculty to communicate and answer questions. Door note delivery entailed the creation of communication via email or other method timed to be delivered approximately 10 minutes before the SP encounter for the learner to read before entering the digital SP room. Each SP room was a separate DCP Zoom meeting, timed to allow participants to join when it was their turn. The SP was now the host, which allows for recording of the event, as would happen in a standard simulation laboratory experience. A separate DCP Zoom meeting or webinar was created for the didactic lecture to allow the host to change to the lecturer and all participants to be in the same DCP space. The process for the deliberate practice experience involved yet another DCP meeting to be set up for each of the faculty and their assigned SP and learners, with the faculty as host, and finally the second one‐on‐one SP experience was conducted similarly to the first, with the same setup requirements. Final surveys of knowledge and comfort need to be delivered via the same platform as the presurveys with advanced setup (see Table 1 for a detailed list of steps necessary to provide this DCP encounter).

TABLE 1.

Steps to create a digital death notification RCDP training module

Task	Implementation comments
Enter quiz questions/surveys into LMS.	Consider precisely timed delivery and completion.
Distribute email instructing learners to complete self‐efficacy survey and knowledge‐based multiple‐choice test.	Consider making this a time‐limited activity.
Create a digital (Zoom) meeting to deliver the prebrief and provide an opportunity to answer questions.
Set up timed emails containing door notes to be read immediately prior to entering the meeting with the first SP.	Door notes should be opened no more than 10 minutes before the encounter. SPs can perform three interactions per hour.
Create digital (Zoom) meetings for each SP to host.	If the waiting room feature is used, one meeting will accommodate multiple learners. The learner will have 15 minutes to deliver notification of death, and the SP will have 5 minutes to evaluate the learner.
If future viewing is required, instruct SPs to record and save the meeting.	Participants may also require identification if they are to be evaluated by outside faculty or instructors.
Determine a means of sharing saved meetings.	Video file sizes tend to be quite large and may require secure transmission for sharing.
Create digital (Zoom) meeting with lecture faculty as host to deliver didactic content.
Set up timed emails containing door notes to be read immediately prior to entering the meeting with the second SP for RCDP.
Create digital (Zoom) meeting with simulation faculty as host for RCDP.	This meeting will have three to four learners and one SP with one faculty member.
Set up timed emails containing door notes to be read immediately prior to entering the meeting with the third SP.
Create digital (Zoom) meetings for each SP to host for the second one‐on‐one SP experience.	If the waiting room feature is used, one meeting will accommodate multiple learners. The learner will have 15 minutes to deliver notification of death, and the SP will have 5 minutes to evaluate the learner.
If future viewing is required, instruct SPs to record and save the meeting.	Participants may also require identification if they are to be evaluated by outside faculty or instructors.
Set LMS or survey instrument to open immediately following the third SP session for final surveys.	Consider making this a time‐limited activity.
Instruct SPs to submit their score sheets for the first and third simulation experience.	Ideally, SPs would evaluate three different learners in the first and third session.

Abbreviations: LMS, learning management system; RCDP, rapid cycle deliberate practice; SP, standardized patient.

Data analysis

The Wilcoxon rank‐sum test was used to test for differences in scores for individual and between‐group performances, pre‐ and postintervention. The Wilcoxon rank‐sum test was chosen due to the data's nonparametric distribution and rank‐ordered attributes. All statistical analysis was performed using SAS 9.4.

RESULTS

Twenty‐two in‐person and 12 DCP learner encounters were conducted using the above process. Data collected for each encounter were collected in this sequence and consisted of: two preintervention assessments for confidence and knowledge, baseline one‐on‐one SP encounter performance assessment, post–one‐on‐one SP encounter performance assessment, followed by completion of the two postintervention assessments of confidence and knowledge. Finally, a comprehensive evaluation of the DCP environment as a vehicle for content delivery was completed by the MSUCOM learners.

Learners were predominantly male, 55% (IUSM 57%, MSUCOM 50%). Table 2 displays the pre and post results for each cohort across the three measures: confidence (self‐efficacy), knowledge, and performance. The data demonstrate statistically significant improvement in confidence, knowledge, and performance in both the in‐person and the DCP arms of the study.

TABLE 2.

Individual group assessments pre‐ and postintervention

	In person (n = 22)		p‐value	DCP (n = 12)		p‐value
	Pre	Post		Pre	Post
Confidence	3 (3–4)	4 (4–5)	<0.0001	2 (2–3)	4 (4–5)	<0.0001
Knowledge, %	65 (40–80)	90 (80–90)	<0.0001	70 (65–80)	90 (85–95)	<0.0001
Performance	82.1 (76.8–87.5)	86.6 (83.9–94.6)	<0.0303	72.3 (67.0–78.7)	86.2 (78.7–92.6)	<0.0373

Data are reported as median (IQR).

Abbreviations: DCP, digital content platform; IQR, interquartile range.

Table 3 displays the data comparing the two groups of learners across the three activities, confidence (self‐efficacy) scores, knowledge scores, and performance scores. Statistical analysis comparing in‐person training to DCP across confidence, knowledge, and performance demonstrates no statistical significance.

TABLE 3.

Between‐group comparison of assessments pre‐ and postintervention

	Pre		p‐value	Post		p‐value
	In‐person	DCP		In‐person	DCP
Confidence	3 (3–4)	2 (2–3)	<0.0001	4 (4–5)	4 (4–5)	0.2795
Knowledge, %	65 (40–80)	70 (65–80)	0.0801	90 (80–90)	90 (85–95)	0.6164
Performance	82.1 (76.8–87.5)	72.3 (67.0–78.7)	0.0381	86.6 (83.9–94.6)	86.2 (78.7–92.6)	0.6782

Data are reported as median (IQR).

Abbreviation: DCP, digital content platform; IQR, interquartile range.

DISCUSSION

This study utilized near‐peer learner groups to assess the pre and post confidence, knowledge, and performance scores of in‐person versus DCP training in a RCDP death notification simulation curriculum. For both learner groups, in‐person and DCP, there was a statistically significant improvement in confidence, knowledge, and performance in death notification skills. Additionally, there was no significant difference in the sub‐analysis between the traditional in‐person and DCP models. This study demonstrated the effectiveness of the DCP to provide death notification training with SP’s to medical students and use the RCDP approach.

There was a significantly lower preconfidence score for the medical students compared to the interns despite less than a month of formal training difference between both groups. This may result from prior training, selection of emergency medicine as a specialty, focused areas of training during the first few weeks as a physician related to communication skills, or other factors that were not assessed. The confidence level for medical students rose to that of the interns by the completion of the educational experience. This was likely facilitated by gaining experience in the one‐on‐one encounter with a professional actor and the subsequent opportunity for immediate feedback from their faculty during the formative training period. Additionally, during the small group training of three to four students with the instructor and the SP, the students could see their colleagues “in the hot seat” with subsequent feedback from their instructor on their opportunities to improve, as is typical during the RCDP approach.

The increase in knowledge in death notification skills was likely the result of the 1‐hour didactic lecture and the opportunity to immediately put the information to practice during the simulations with SPs. This curriculum was designed to expose the students to different teaching methodologies to accommodate different learning styles. The skill of death notification or giving bad news is one where you can “know what to say” but truly fail at executing effectively. Poor performance of death notification can result in secondary trauma to the family of the deceased.¹² We did not anticipate or hypothesize that there would be any significant difference in providing the didactic material remotely versus in person and this was confirmed.

The DCP cohort demonstrated notably greater performance improvement versus the traditional in person cohort. Although both groups improved with statistical significance, the precurricular performance score was far lower for the medical students than the interns. Several students anecdotally reported feeling somewhat intimidated by a live and very skilled SP and by having to perform this skill with no previous formal training. For the medical students, their prior simulation exposures were all comprised of objective structured clinical examination type structured experiences in which they knew the purpose of the event and could practice prior to performing. In addition, feedback to faculty after the session indicated they felt a time‐constrained pressure to read the door note and then have to open their Zoom session and immediately begin the SP experience, all within 5 minutes.

Interestingly, both cohorts’ final score was almost identical, as scored by the SPs. The SPs in this study were the same for both cohorts to limit variability in scoring. The loss of the “power of touch,” a key component in communication, particularly in osteopathic medicine, impacted faculty's ability to both model and evaluate nonverbal empathetic gestures and posture as well as learners’ ability to practice this technique of empathy.

The DCP did not provide any obvious disadvantage to the medical students regarding their ability to learn how to execute a death notification. This approach may be an avenue to achieve increased student training experiences “through simulation” without being physically present. Often the rate‐limiting step to increasing the number of student training experiences is the lack of physical space in a simulation center in which to conduct them. Utilizing the DCP where appropriate may also provide several strategic opportunities for the simulation center in scheduling, meeting strategic partners’ educational needs, and generating revenue. By focusing on medical education areas effectively delivered via a DCP, simulation would not have to come to a halt when situations dictate isolation from others. In the future, there may be opportunities for a mix of DCP and in‐person experiences to be conducted simultaneously, allowing training groups to remain intact even when one or more members of the group are unable to participate in person. This platform's success may help shift “brick‐and‐mortar” simulation centers to focus their time and attention on skills that are genuinely best taught in‐person and do not translate well to the DCP format. Concomitantly conducting experiences utilizing a DCP and in person would allow increased learner numbers and improve efficiency. In addition, the DCP format can expand the geographical footprint of simulation, providing the same experience for all learners regardless of physical location. Further research can help identify other skills traditionally taught in a simulation that can effectively be delivered utilizing a DCP.

The setup time was substantially longer for the DCP version than the in‐person training. This was multifactorial. The DCP required the need to set up meetings for each encounter and learner instead of bringing learners together in person, completing the prebrief and the surveys, and then moving to their rooms in an ambulatory fashion. All content that was quiz based also had to be entered into a learning management system for the students to access at the appropriate time, a step not necessary in the in‐person setting. In the future, some of this time difference may be mitigated by utilizing the DCP differently and using other learning management systems. This activity used what was available at the time of the event to the faculty involved. Both approaches required a presession practice with the faculty and SP to evaluate the content and assure understanding of the processes to be employed. Additional time was necessary for the DCP environment to troubleshoot device performance and location performance to ensure stable Internet services and verify that the SP can also record, save, and upload the videos produced if they are to be used for grading or research purposes. The vetting of the SP’s ability to save the simulation recording, quickly provide a formalized assessment with a checklist, and quickly transition to a new participant must be carefully evaluated. An SP less familiar with navigating or troubleshooting this software may result in significant delays or loss of data necessary for providing feedback or research studies.

This study provides a curricular framework for simulation educators who desire to provide formal instruction to their learners on death notification. Medical school and residency programs interested in developing and executing such a curriculum will improve their chances of success by pooling time and financial resources to make this process less demanding. Once the curriculum is established, rerunning this training via DCP will require far less time for faculty. Additionally, this method of instruction should result in cost savings. Participants, SPs, and instructors can engage remotely, avoiding facility fees charged by many simulation centers.

LIMITATIONS

Our limitations included the small sample size. We performed these two exercises at two institutions, limiting generalizability. Additionally, we utilized similar but not identical levels of trainee cohorts. Furthermore, one group was homogeneous in career specialty goals while the other was heterogeneous. By only using the DCP Zoom product for the telepresence exercise, we could not determine if it was the most effective tool for this activity type.

CONCLUSIONS

The digital delivery of the death notification training curriculum utilizing the GRIEV_ING mnemonic as the conceptual framework via rapid cycle deliberate practice was comparable in its ability to improve confidence, knowledge, and performance as an in‐person training in a simulated environment. Simulation scores were similarly improved across both methods. Standardized patient technology skills, including the ability to utilize and troubleshoot a digital communication platform during a session with a learner, record, save, and upload this recording for future viewing if needed, may affect your choice of standardized patients.

This telepresence model can allow delivery to broader audiences and be more cost‐effective using digital communication platform such as Zoom thus mitigating the need for travel costs and utilization of the simulation laboratory. We have demonstrated that both in‐person and digital communication platform delivery methods can be implemented effectively without losing student engagement and skill acquisition.

AUTHOR CONTRIBUTIONS

All authors meet all the following criteria: substantial contributions to the conception or design of the work; acquisition, analysis, or interpretation of data for the work; drafting the work or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Hughes M, Gerstner B, Bona A, et al. Adaptive change in simulation education: Comparison of effectiveness of a communication skill curriculum on death notification using in‐person methods versus a digital communication platform. AEM Educ Train. 2021;5:e10610. 10.1002/aet2.10610