The Equivalence of Video Self‐review Versus Debriefing After Simulation: Can Faculty Resources Be Reallocated?

Abstract

Introduction

Traditional simulation debriefing is both time‐ and resource‐intensive. Shifting the degree of primary learning responsibility from the faculty to the learner through self‐guided learning has received greater attention as a means of reducing this resource intensity. The aim of the study was to determine if video‐assisted self‐debriefing, as a form of self‐guided learning, would have equivalent learning outcomes compared to standard debriefing.

Methods

This randomized cohort study consisting of 49 PGY‐1 to ‐3 emergency medicine residents compared performance after video self‐assessment utilizing an observer checklist versus standard debriefing for simulated emergency department procedural sedation (EDPS). The primary outcome measure was performance on the second EDPS scenario.

Results

Independent‐samples t‐test found that both control (standard debrief) and intervention (video self‐assessment) groups demonstrated significantly increased scores on Scenario 2 (standard—t(40) = 2.20, p < 0.05; video—t(45) = 3.88, p < 0.05). There was a large and significant positive correlation between faculty and resident self‐evaluation (r = 0.70, p < 0.05). There was no significant difference between faculty and residents self‐assessment mean scores (t(24) = 1.90, p = 0.07).

Conclusions

Residents receiving feedback on their performance via video‐assisted self‐debriefing improved their performance in simulated EDPS to the same degree as with standard faculty debriefing. Video‐assisted self‐debriefing is a promising avenue for leveraging the benefits of simulation‐based training with reduced resource requirements.

Procedural sedation is a core competency for the practice of emergency medicine comprising a specific competency milestone in the Accreditation for Graduate Medical Education Next Accreditation System.1 Despite advances in technology such as end‐tidal CO₂ monitoring, the safety profiles of commonly used medications, and the implementation of published practice guidelines, adverse respiratory events are still observed.2, 3, 4

Immersive simulation provides a safe training environment for learning procedural sedation. The traditional learning model for simulation‐based training includes a postsimulation debriefing in which learners’ experiences are explored and reflected upon with the goal of improving future performance.5, 6, 7 Such a format is resource‐intensive due to the higher faculty–trainee ratio when compared to other educational strategies, and there is increasing pressure to evaluate and report the costs associated with simulation‐based educational interventions.8

Shifting the degree of primary learning responsibility from the faculty to the learner through self‐guided learning has received greater attention as a means of reducing resource intensity.9 While the benefits of reducing the need for faculty resources is a benefit of self‐directed learning, the optimal uses of simulation for self‐guided learning in clinical education in this manner has not yet been fully explored.10, 11, 12 The purpose of the study was to determine if video‐assisted self‐debriefing, as a form of self‐guided learning, would have equivalent learning outcomes compared to standard debriefing.

The Society for Simulation in Healthcare defines debriefing as a formal, collaborative, reflective process within the simulation learning activity that encourages participants’ reflective thinking and provides feedback about their performance while various aspects of the completed simulation are discussed.13 We propose that self‐reflective video “debriefing” is a form of debriefing because it meets the criteria of the above definition. Specifically, it is formal in that it is a structured experience, it is collaborative in that the checklist is a derivation of a modified Delphi panel, and it is reflective in that viewing one's own performance on video with a checklist provides a framework for comparison within the simulation learning activity.

Methods

Study Design

This randomized cohort study consisting of 49 PGY‐1 to ‐3 emergency medicine residents compared performance after video self‐assessment utilizing an observer checklist versus standard debriefing for simulated emergency department procedural sedation (EDPS). The study was approved by the institutional review board under expedited review criteria. Data for research were only extracted from participants who voluntarily provided consent.

The study consisted of 49 PGY‐1 to ‐3 emergency medicine residents. Participation in the residency program's formal biannual training in procedural sedation was included in demographic data. Eligible study subjects (all emergency medicine residents) were randomized to intervention or control groups prior to enrollment. Learners participated in individual simulations that were deployed as part of weekly residency conference. Residents were approached by designated research personnel prior to participation to obtain consent to include their deidentified data.

An observer checklist (Data Supplement S1, Appendix S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10372/full) was created that assessed clinical, pharmacologic, and communication areas of EDPS with a total of 49 checklist items. The checklist was developed using a two‐round modified Delphi process.14 The checklist was piloted with PGY‐3 residents and revisions were made to increase reliability. Inter‐rater reliability was determined with five EDPS videos using the Brennan‐Prediger kappa variant, and the overall mean kappa for all questions and subtypes was 0.90.

The “Quality of Patient Care Assessment” utilized a five‐point Likert scale. This was a global assessment performed by both individual learners and two emergency medicine core faculty members.

Video‐assisted self‐debriefing was structured as a self‐guided postevent debriefing12 in which learners viewed video recording of their performance while completing the observer checklist. Standard debriefing was defined as postevent facilitator‐guided,12 using the “debriefing with good judgment” framework.15 All debriefers (GJT, LTB) were emergency medicine physicians trained in debriefing. Debriefers completed the observer checklist during the simulation to use as a debriefing guide.

Scenario Design

Two variants of an EDPS scenario were created for the study. Scenario 1 involved a 6‐year‐old child with a dog bite to the thigh that required suturing. Scenario 2 involved an 18‐month‐old child with an abscess of the inner thigh that required incision and drainage.

Both variants had the same objectives:

1. Perform standard elements of presedation evaluation for a pediatric patient requiring procedural sedation and analgesia (PSA);

2. Correctly identify and correct medication error embedded in scenario;

3. Correctly identify and correct hypoventilation from oversedation embedded in scenario;

4. Provide standard‐of‐care sedation (preprocedural, procedural, and postprocedural) for a child with a condition requiring PSA (see checklist).

To increase fidelity and allow the test subjects to focus solely on the procedural sedation, three trained standardized participants (SPs) were utilized in each case. Their specific training was uniform and the scripted dialogue is included in Data Supplement S1, Appendix S2. There were no more than three different SPs in the role of the parent, limiting variation in performance. In addition, each of the SP parents had an earpiece to provide any necessary correction during the simulation. These SPs remained the same throughout the study.

Protocol

All eligible subjects participated in the simulated EDPS. The intervention group (n = 25) used video self‐assessment while the control group (n = 24) underwent standard debriefing. Randomization assignment was revealed directly after participation in the simulation so that raters were blinded until the simulation scenario was complete. For the video group, research personnel provided acclimation to the video playback features and the observer checklist. The same performance checklists were used for self‐guided learning and in faculty debriefing to ensure the same content was covered.

The primary outcome measure was performance in the second EDPS scenario. Case order was the same for both groups. All participants ran Scenario 2 (incision and drainage) within a period of 4 months from their participation in Scenario 1. Both groups experienced facilitator‐led debriefing for Scenario 2. Change in checklist scores between the two simulations for each group was calculated and compared using paired and independent‐samples t‐test. Statistical analysis was performed with R software (The R Foundation).

RESULTS

Demographic information for the control and intervention groups is shown in Table 1. The lack of a significant score difference between randomizations for Scenario 1 indicates that they started at the same baseline.

Table 1.

Demographics for Control and Intervention Groups

	Video Self‐assessment Group (n = 25)	Standard Debriefing Group (n = 24)	p‐value
PGY level
1	22 (88.0)	17 (70.8)	0.17
2	2 (8.0)	4 (16.7)	0.42
3	1 (4.0)	3 (12.5)	0.35
Number of days from sedation course to Scenario 1	394.04 (245.58)	416.67 (231.41)	0.74
Scenario 1 score	55.88 (±16.33)	62.15 (±18.62)	0.22
Number of days between Scenario 1 and Scenario 2	79.0 (±51.9)	114.1 (±97.4)	0.17

Data are reported as n (%) or mean (±SD).

The means and standard deviations (SD) for the faculty evaluations of the standard and video randomizations for Scenario 2 (incision and drainage) can be seen in Table 2 below. Independent‐samples t‐test found that both groups demonstrated significantly increased scores on Scenario 2 (video—t(45) = 3.88, p < 0.05; standard—t(40) = 2.20, p < 0.05).

Table 2.

Postintervention Performance Ratings

	Video Self‐debriefing
Scenario	n	Mean	SD	Mean Increase
Video self‐debriefing
Lac	25	55.88	16.33	16.93
I&D	22	72.81	13.11
Standard debriefing
Lac	24	62.15	18.62	11.98
I&D	18	74.13	15.79

I&D = incision and drainage; Lac = laceration repair.

A linear regression was used to determine the effect of randomization, scenario, and the interaction between the two on performance score. No significant associations were found between score and randomization or between randomization and scenario. These results indicate that the observed increase in performance on the second scenario (laceration repair) compared to the first scenario (incision and drainage) was independent of randomization. The conditional plots in Figure 1 graphically display the relationship between randomization, scenario, and performance score.

Figure 1.

Conditional plot illustrating relationship between randomization, scenario, and performance score.

There was a baseline difference between the video and standard debriefing group for the laceration repair. The standard debriefing group started out with a better performance (though not statistically significant). Even though the video group started out five points below the baseline, they demonstrated clear improvement (there was no overlap in error bars between Scenario 1 and Scenario 2).

Paired Resident Self‐evaluation Versus Faculty Evaluations

Chi‐square and Fisher's exact tests were used to determine if there was a difference in the frequency at which resident self‐evaluation and faculty assessments identified specific items on the checklist as “performed”. Of the 21 performed checklist items that had both faculty and resident evaluations, a significant difference between resident self‐evaluation and faculty evaluations was found only for the “jaw thrust performed” item with 72% of residents classifying it as performed in contrast to only 40% of faculty (χ²(1,N = 50) = 5.19, p < 0.05).

Total checklist scores were calculated by assigning one point for each completed action. Team performance questions were excluded from this analysis. There was no significant difference between faculty and residents self‐assessment mean scores (t(24) = 1.90, p = 0.07).

There was a large and significant positive correlation between faculty and resident self‐evaluation (r = 0.70, p < 0.05), although a significant difference was found in the distribution of the quality of patient care ratings given by residents and faculty (FET, p < 0.05). As can be seen Figure 2, the residents’ ratings tended to be mostly average while the faculty ratings were spread out more toward the high and low ends of the scale.

Figure 2.

Comparison of resident vs faculty self‐evaluation.

Discussion

Kolb described the use of experiential learning in the “critical linkages among education, work, and personal development. His study of the similarities of several learning theories emphasized the linkage of experience to reflection, which in turn allows the learner to “[adapt] to one's total life situation.”16 In this study, residents are prompted to engage in a deeper and guided self‐reflection assisted by video. From a pedagogic standpoint, this self‐reflection is a transforming experience leading to “new implications for action.” This study addresses, in part, the exploration of a resource‐sparing reflective process which retains the benefit of the experience for the learner.

Video‐assisted self‐debriefing may be one strategy within a larger context of self‐regulated learning (SRL). A key feature for success in SRL is deliberate planning of activities to support individual progress rather than merely “learning alone.”17 In our study, both scenarios lent themselves to the use of a performance checklist for assessment. Similarly, proper scaffolding of self‐guided mastery learning has demonstrated educational outcomes equivalent to instructor led mastery‐learning interventions for ACLS training of internal medicine residents with lower associated costs.18

The immediate appeal of self‐directed learning is the logistic flexibility and reduction of costs. A recent study calculated significant cost savings for video‐assisted self‐debriefing when compared to instructor‐led debriefings of anesthesiology residents participating in “perioperative crisis scenarios.”18 While there is cost associated with purchasing and utilizing video recording equipment, simulation recording is a common practice and part of the baseline equipment at our institution. Furthermore, cost‐effectiveness must take into consideration the intangible expenses associated with faculty time commitment, faculty scheduling burden, and training faculty to become skilled in simulation debriefing. In addition, there is a significant time investment required to develop, deploy, and study a checklist‐based educational intervention. As a checklist is developed it may be validated then utilized across multiple learner groups with similar educational goals (e.g., residents in emergency medicine, pediatrics, surgery)

Video‐assisted debriefing can be accomplished in almost any environment—the learner sits at a laptop with headphones and views his or her recording with the checklist. Cost savings for faculty debriefing time could be a significant benefit for appropriately selected simulation scenarios amenable to video checklist debriefing. In addition, time saved by video debriefing may be spent on learner remediation, scenario refinement, or other educational activities. In our study, each scenario had an average length of 14 minutes and 18 seconds per participant.

While these results demonstrate positive impact of video‐assisted self‐briefing on learners’ subsequent performance, video review in and of itself is not the panacea. Martin et al.19 report discordant ability to assess self versus others performance of a simulated patient communication encounter via video. However, the assessment tool supporting video review in that study was a behavioral rating scale, which allows for greater rating subjectivity. Our data demonstrate that for procedural sedation, video checklist debriefing can result in self‐assessments highly correlated with faculty assessments and, more importantly, similar improvement in performance when compared with standard debriefing. Our study may have been able to detect the improvement between scenarios because a checklist was used instead of a behavior rating scale.

Our results suggest that video self‐review may be more beneficial for novice (lower score) learners. Expert learners in the standard debriefing group started higher and had a smaller improvement which suggests that expertise is asymptotic (it takes proportionally much more effort to get from 90 to 100 than it does from 10 to 100) and that expert learners may benefit from a more nuanced faculty guided debriefing.

In addition, our data demonstrate that for procedural sedation, video checklist debriefing can result in self‐assessments that correlate highly with faculty assessments. More importantly, this equates to a similar improvement in performance when compared with standard debriefing. Our study may have been able to detect the improvement between scenarios because a checklist was used instead of a behavior rating scale.

Limitations

Several limitations are present. There were six individuals who were lost to follow‐up in the standard debriefing group. Despite the significant positive impact on subsequent performance, the inclusion of these subjects may have potentially changed the results for the standard debriefing group relative to the intervention group. Our scenarios involved only two cases with PSA as the substrate for clinical care. The singular focus of PSA and the ability to create a binary checklist may not be generalizable to other case scenario types. Additionally, both case scenarios involved PSA for simple procedures (laceration repair and abscess drainage), which may not represent the cognitive load for more complex PSA cases or other case scenarios. Furthermore, the selection of cases may have contributed to a significant improvement between the laceration case and the abscess case. Sedation for laceration repair may be more cognitively intuitive than for incision and drainage. The difference may be also experiential if the learners experienced the laceration scenario second.

Another limitation of our study is that residents may not focus on the same aspects of performance as faculty raters. We did not define faculty and resident self‐assessment performance elements and therefore these may be looking at different aspects of clinical care. This, however, was not a primary goal of our study and is a potential area of future research.

Since our study population was not large enough to detect subgroup differences, it is possible that the benefit is greater for more novice learners. Subsequent research should define the optimal contexts and adjuncts to realize the maximal benefits of video‐assisted self‐debriefing.

Conclusions

In conclusion, emergency medicine residents receiving feedback on their performance via video‐assisted self‐debriefing improved their performance in simulated emergency department procedural sedation to the same degree as with standard faculty debriefing. Video‐assisted self‐debriefing is a promising avenue for leveraging the benefits of simulation‐based training with reduced resource requirements. Future work should delineate the relevant content areas and supportive structures need to optimize its impact on learner competency.

We would like to acknowledge the staff at JUMP simulation that made this study possible including Toufic Khairallah, research coordinators Kim Cooley and Kelsey Balcer, our simulation specialists (especially Dustin Holzwarth), and our standardized participants.

Supporting information

Data Supplement S1. Supplemental material.

AEM Education and Training 2020;4:36–42.