360 virtual reality pediatric mass casualty incident: A cross sectional observational study of triage and out‐of‐hospital intervention accuracy at a national conference

Abstract

Objective

With adolescent mass casualty incidents (MCI) on the rise, out‐of‐hospital readiness is critical to optimize disaster response. We sought to test the feasibility and acceptability of a 360 Virtual Reality (360 VR) platform for disaster event decisionmaking.

Methods

This was a cross‐sectional observational assessment of a subject's ability to triage and perform out‐of‐hospital interventions using a 360 VR MCI module. A convenience sample of attendees was recruited over 1.5 days from the American College of Emergency Physicians (ACEP) national conference in San Diego, CA.

Results

Two hundred and seven (207) subjects were enrolled. Ninety‐six (46%) subjects identified as attendings, 66 (32%) as residents, 13 (6%) as medical students, 4 (2%) as emergency medical technicians and 28 (14%) as other. When comparing mean scores between groups, physicians who were <40 years old had mean scores higher than physicians who were >40 years old (8.7 vs 6.5, P < 0.001). Residents achieved higher scores than attendings (8.6 vs 7.5, P = 0.005). Based on a 5‐point Likert scale, participants felt the 360 VR experience was engaging (median = 5) and enjoyable (median = 5). Most felt that 360 VR was more immersive than mannequin‐based simulation training (median = 5).

Conclusion

We conclude that 360 VR is a feasible platform for assessing triage and intervention decisionmaking for adolescent MCIs. It is well received by subjects and may have a role as a training and education tool for disaster readiness. In this era of distanced learning, 360 VR is an attractive option for future immersive educational experiences.

1. INTRODUCTION

1.1. Background

An unfortunate truth is that high school shooting events are becoming more common. ¹ , ² , ³ At the same time, individual practitioners have relatively low encounter rates with critically injured children versus adults. ⁴ This emphasizes the need for disaster preparedness and mass casualty incident (MCI) training specific to pediatric and adolescent patients. Traditionally, MCI training has included live simulation and video. Although these have been shown to reinforce knowledge and skills, in the setting of a MCI they have had less clear success. ⁵ , ⁶ Additionally, with the current emphasis on reducing large gatherings, it is likely that in person disaster training sessions will decrease in availability and popularity. This opens the opportunity for alternatives.

360 Virtual Reality (360 VR) is an emerging media platform that has already been widely applied across a diverse range of industries including entertainment, higher education, pilot training, the military, and medicine. ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ This burgeoning technology can facilitate highly effective interventions such as “Just in Time” learning. ¹² Thanks to rapid advances in technology and increased affordability, 360 VR experiences are becoming more accessible and ripe for study.

Several studies comparing simulation to 360 VR have concluded that 360 VR achieves equivalency to live simulation in realism and meeting teaching milestones. ¹³ , ¹⁴ , ¹⁵ It also provides realistic physiological and psychological responses similar to those evoked with live simulation. ¹³ , ¹⁵ , ¹⁶ , ¹⁷ These benefits have been found to impact trainees at all experience levels. ¹³ , ¹⁴ , ¹⁷ , ¹⁸ , ¹⁹

360 VR offers several benefits over simulation, given its ease of distribution, high engagement, and portability. ⁹ , ¹¹ , ²⁰ , ²¹ In contrast to live simulation, 360 VR modules can be accessed at any time, at the learners’ leisure. This ability to train and be assessed asynchronously is a key advantage of 360 VR in helping refresh skills and prevent knowledge decay. ¹⁰ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶

1.2. Importance

Although there has been prior virtual reality work for disaster readiness training and education using virtual environments, and computer‐generated imagery, our search of the literature did not reveal studies using 360 VR for pediatric disasters or MCIs. ¹³ , ¹⁷ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² This novel approach may prove to be a valuable training and education tool for disaster preparedness.

1.3. Goals

In this study, we explore the feasibility of using a 360 VR experience for the assessment of adolescent disaster readiness and also assess its acceptance among users.

The Bottom Line

This is a descriptive study of the development of the 360 VR module in MCI triage training and the exemplary utility based on experiences of learners with differing levels of experience.

2. METHODS

2.1. Study design and setting

This study was an observational cross sectional analysis of subjects’ medical decisionmaking skills required for triage and field intervention assessed in a 360 VR experience of an adolescent multi‐casualty, mass shooting incident. It was approved by the Stanford Investigational Review Board (47397).

2.2. Creation of 360 VR

A total of 150 high school student actors were oriented to the MCI experience and moulaged to simulate traumatic injuries. These students were taking part in an emergency medicine summer program (Advanced Emergency Medicine, Stanford, California, USA) where they learned about topics in emergency medicine. They were taught the Simple Triage and Rapid Treatment (START) protocol and learned about life‐saving out‐of‐hospital interventions such as needle thoracentesis, hemostasis, splinting, and tourniquets. ²⁷

The scenario took place at a casualty care point after a mass shooting at a high school. The actor playing the MCI “responder” was an emergency physician with specific training in tactical medicine and response to MCIs. The 360 video was shot using an Insta360 Pro (Insta360, Irvine, CA) camera, which is an all‐in‐one, video recording device. Audio was obtained via a Sennheiser Ambeo VR microphone (Sennheiser, Germany). The 360 video footage was stitched together using Insta360 Stitcher and was edited using Adobe Premiere (Adobe Software, San Jose, CA). Once edited, each patient scenario was prepared for use as a 360 VR module by adding an interactive graphical user interface (GUI) in Unity (Unity Technologies, San Francisco, CA) which is a mobile platform development engine. Post‐production voiceover was recorded by an emergency medicine attending and resident in Audacity (Audacity Technologies) using a Blue Yeti USB (Blue Designs, Westlake, CA) microphone.

The 360 video was reviewed by a group of 3 board eligible or board certified emergency physicians (Cynthia Peng, Christopher Winstead‐Derlega, Henry Curtis) and a pediatric emergency physician (Jason Lowe). Expert consensus among this group yielded 9 high quality scenarios (Table 1). Five triage scenarios (55.6%) were designated as immediate, 2 (22.2%) minor, 1 (11.1%) delayed, and 1 (11.1%) expectant based on the START categorizations.

TABLE 1.

Virtual reality MCI patient scenarios

Victim	Injury	Triage	Intervention	Time limit (s)
1	Abdominal evisceration	Immediate	Wet gauze dressing	20
2	Lower limb fracture	Minor	None	18
3	Tension pneumothorax	Immediate	Needle decompression	15
4	Lower limb fracture	Minor	None	13
5	Upper extremity arterial bleed	Immediate	Tourniquet	9
6	Open chest wound	Immediate	Occlusive dressing	13
7	Open chest wound	Immediate	Occlusive dressing	9
8	Devastating injury	Expectant	None	6
9	Pelvic fracture	Delayed	Pelvic binder	5

For each patient, participants could choose a triage category and a field intervention from drop‐down menus. Triage options included “Minor,” “Immediate,” “Delayed,” or “Expectant.” Options for field interventions included needle decompression, 3‐sided dressing, completely occlusive dressing, direct pressure, tourniquet, splint, pelvic binder, bandage, or nothing. The platform was programmed with an integrated database to collect and to store subjects’ scores and demographic information.

2.3. Selection of participants

Conference attendees on the exhibit floor over 18 years of age were recruited as a convenience sample of subjects at the American College of Emergency Physicians (ACEP) 50th annual conference in San Diego, California from October 1–3, 2018 (6 hours/day). There was no compensation given for participation in the study.

2.4. Exposures

After participants consented to the study, they were oriented to the 360 VR headset and controls, using either an Oculus Go (Facebook) or Daydream VR (Google) device (Figure 1). Upon donning the headset, participants self‐reported demographic information and clinical experience. Participants were then introduced to the 360 VR environment via a tutorial where they performed one triage and intervention. Users could repeat the tutorial as often as desired. Once comfortable with the interface, they continued on to the assessment module.

FIGURE 1.

Participant wearing oculus go and view from within virtual reality headset experience

As the scenario began, participants were immersed visually and auditorily in a scene of numerous casualties in an open grassy field (Figure 1). Participants identified the next patient of interest in their 360 visual field by listening and physically turning in the environment. After making observations of the casualty's initial evaluation, participants used the hand controller to choose a triage level and field intervention from a drop down menu toolbar. The heads‐up display included a countdown timer for patient evaluations. Each subsequent encounter had a shorter time limit. Participants could not pause the experience and faced a new patient whenever the time ran out. All patient encounters were presented to subjects in the same order and 1 point was awarded for each correct triage or field intervention answer. No penalty was given for wrong answers. In total there were 9 patient encounters with the highest possible score being 18.

After participants completed the module, they returned the 360 VR headset and the device was cleaned. Research personnel then offered participants an anonymous post‐simulation 7‐question, 5‐point Likert survey to describe and rate their experience.

2.5. Analysis

Variables were evaluated for normality using the Shapiro Wilk test. Non‐parametric data were analyzed with Wilcoxon's rank sum test. Multivariable analysis was done using analysis of variance with a subsequent Tukey's test for post‐hoc analysis for positive findings. All statistics was performed using R statistical software (https://www.r-project.org/).

3. RESULTS

3.1. Characteristics of study subjects

Of the 207 study participants, 63 (31%) identified as <30 years old, 85 (41%) 31–40 years old, 33 (16%) 41–50 years old, and 24 (12%) >50 years old. Two participants were excluded from age calculation due to improper entry. A total of 96 (46%) identified as attending level, 66 (32%) as residents, 13 (6%) as medical students, 4 (2%) as emergency medical technicians (EMTs), and 28 (14%) as other. Geographically, subjects were well represented across the United States with 58 (28%) from the Northwest, 24 (12%) from the Midwest, 57 (28%) from the West, 50 (24%) from the South and 18 (7%) from outside the United States. Sixty‐two (30%) said they had previous mass casualty experience and 145 (70%) had not. Fifty‐nine percent used the Oculus VR device and 41% used the Daydream VR.

3.2. Main results

Analysis of combined scores for all participants produced a Gaussian distribution curve. Results are presented in Table 2. When comparing mean combined scores between groups, physicians who were <40 years old had mean scores higher than physicians who were >40 years old (8.7 vs 6.5, P < 0.001). Subjects who had been to medical school scored higher overall versus those who had not been to medical school (7.9 vs 5.9, P = 0.0002). Residents produced higher scores than attendings (8.6 vs 7.5, P = 0.005) for total scores (triage plus intervention) and intervention scores alone (3.5 vs 2.5, P = 0.0004) but not for the triage scores alone (5.2 vs 4.9, P = 0.19). A separate analysis comparing residents versus attendings <40 years old showed no significant difference in total score, triage, or field intervention scores separately (8.7 vs 8.3, P = 0.34). Those who reported previous MCI experience did not score differently than those who did not claim such experience (7.7 vs 8.1, P = 0.35). No difference was found in total scores based on device used (7.6 vs 7.7, P = 0.87).

TABLE 2.

Correctly answered triage and intervention scores based on level of medical experience

	Resident (n = 66)		Attending (n = 96)		Non‐physicians a (n = 45)		Total (n = 207)
	Triage	Intervention	Triage	Intervention	Triage	Intervention	Triage	Intervention
Victim	n (%)	n (%)	n (%)	n (%)	n (%)	n (%)	n (%)	n (%)
1	37 (56)	16 (24)	56 (58)	19 (20)	23 (51)	10 (22)	116 (56)	45 (22)
2	43 (65)	13 (20)	65 (68)	5 (5)	29 (64)	4 (9)	137 (56)	22 (11)
3	55 (83)	49 (74)	81 (84)	61 (64)	33 (73)	24 (53)	169 (81)	134 (65)
4	36 (55)	34 (52)	50 (52)	25 (26)	19 (42)	13 (29)	105 (51)	72 (35)
5	28 (42)	35 (53)	46 (48)	41 (43)	22 (49)	18 (40)	96 (46)	94 (45)
6	55 (83)	45 (68)	72 (75)	59 (61)	27 (60)	14 (31)	154 (74)	118 (57)
7	41 (62)	20 (30)	52 (54)	23 (24)	21 (47)	6 (13)	114 (55)	49 (24)
8	48 (73)	15 (23)	52 (54)	12 (13)	16 (36)	7 (16)	116 (56)	34 (16)
9	2 (3)	1 (2)	1 (1)	0 (0)	1 (2)	0 (0)	4 (2)	1 (0)

^aNon‐physicians = EMT, medical student, and other.

Results of an anonymous 5‐point Likert survey based on the experience revealed the following results (Figure 2). The VR experience was engaging (median = 5) and enjoyable (median = 5). Participants felt more prepared for adolescent MCI (median = 4). Most felt that VR should be an integral tool for medical education (median = 5) and specifically that they would like to see more VR used for disaster training (median = 5) and pediatric VR training (median = 5). Finally, most felt that VR was more immersive than mannequin‐based simulation training (median = 5).

FIGURE 2.

Post‐analysis survey with 5‐point Likert scale

4. LIMITATIONS

We acknowledge the limitations of our work. Although we provided a 360 VR tutorial that could be repeated as often as desired, some participants may have benefited from further familiarization to the VR platform. They may have had difficulty logging responses solely because of difficulty with the interface and not because of lack of knowledge. As 360 VR becomes more ubiquitous, we believe there will be fewer concerns with technology familiarization.

Our study population may have been subject to selection bias. We suspect that the participants were receptive to and intrigued by 360 VR overall, and this interest may have existed regardless of content. Additionally, the majority of ACEP attendees can be presumed to be enthusiastic about education in general and thereby interested in our work.

The cohort of participants who were categorized as non‐physicians included medical students, EMTs, and Others. Over half of this group was classified as Others and were not further sub‐categorized into occupation or training level. Without this information, we are unable to explain why this group scored higher than attending physicians on some interventions. Future studies should better delineate occupation and training of all participants.

Furthermore, there was the potential for reporting bias. All demographic data were self‐reported. This included experience and level of training, which may be subject to some interpretation by the entrant.

Finally, to prioritize enrollment of new participants, the Likert scale was presented in an analogue fashion. This resulted in anonymized data sampling, which did not allow for cross‐matching with other types of data such as performance, headset type, and demographic information. Despite this limitation, our survey was able to assess the basic acceptability of the 360 VR platform.

5. DISCUSSION

Our study demonstrates the feasibility and acceptability of a 360 VR experience created to assess for disaster triage and field intervention knowledge in adolescent MCIs. Our results demonstrate that this medium has the ability to discriminate between different levels of knowledge in adolescent MCI medical decisionmaking. It is also well accepted by users.

The combined triage and field intervention scores showed a normal distribution. The knowledge and skills required to obtain a correct answer was such that participants with little to no medical decisionmaking ability could be differentiated from participants with emergency medical knowledge. Given this, we believe that immersive VR experiences can be an effective means of assessing levels of knowledge.

Age was a significant variable in the difference in the mean scores of participants. Residents and attendings younger than 40 years old may have performed better for multiple reasons. While previous studies of VR have demonstrated differences in performance across age groups in which older age groups score less well than younger groups, ²⁸ , ²⁹ it is possible that the experience of more senior clinicians may not be properly captured by our study design. Further investigation on the role of age in performance is warranted.

Interestingly, residents scored higher overall than attendings. To the authors’ knowledge, there are no studies in the emergency medicine literature comparing the disaster preparedness of resident physicians to that of attending physicians. The observed difference in this study may be due to being in closer temporal proximity to formal disaster training. Because of this, we postulate that continued training may be to protect against knowledge degradation.

Residency‐trained emergency physicians scored much higher than non‐physicians. That is, subjects who have had exposure to disaster training in residency scored better than non‐physicians. This difference was not affected by geographic location or age. This is similar to previous research comparing VR to live simulation. It is consistent with others' findings that 360 VR can effectively differentiate learners based on training level. ⁷ , ⁸ , ⁹ , ²¹

Post‐survey Likert scale scores showed high acceptance of 360 VR as a training modality and viewed it as engaging and enjoyable. Overall subjects had overwhelmingly positive interactions with VR, and were enthusiastic for it to be used in further VR learning. This was true across all age groups and is consistent with other studies. ³⁰ , ³¹ , ³² This acceptance, coupled with the previously mentioned ease of distribution, makes 360 VR a target for additional experiences to assess preparedness for disasters and other areas of medicine as well.

It is not overly complicated to design this method of disaster readiness assessment. The prosumer camera that captures 360 video is simple to use and can be operated by educators with little filmmaking knowledge. Editing and programming is less challenging for 360 VR than other forms of virtual reality such as computer‐generated imagery. The team was able to produce the entire project for $7000 using funds from a small innovation grant. When comparing this cost to a disaster drill designed for a large scope and scale, this required fewer human resources, equipment, and time. Additionally, once a scenario has been created, it can be reused in perpetuity by infinite users in many locations.

6. CONCLUSION

Our study results suggest that 360 VR can be used as a means of assessing readiness for adolescent MCIs and other medical procedures and pathways. With an increasing focus on virtual options as a means to decrease physical contact and individualize education, VR offers an increased level of interaction that can be distributed widely, without barriers. Future studies should collect more specific demographic data and be powered to better delineate differences between groups of subjects. Modules dedicated to inculcating knowledge and skills also may be viable. With more refined measurements, the scope of 360 VR as a training and education tool can be further defined and applied.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

AUTHOR CONTRIBUTION

JL takes the final responsibility of the article.

Biography

Jason Lowe, DO, is Clinical Assistant Professor of Emergency Medicine and Clinical Assistant Professor of Pediatrics at Stanford University.

Lowe J, Peng C, Winstead‐Derlega C, Curtis H. 360 virtual reality pediatric mass casualty incident: A cross sectional observational study of triage and out‐of‐hospital intervention accuracy at a national conference. JACEP Open. 2020;1:974–980. 10.1002/emp2.12214