Responding to a simulated disaster in the virtual or live classroom: Is there a difference in BSN student learning?

Abstract

Aim:

This study aim was to investigate if prelicensure baccalaureate nursing students gained more knowledge from a live or virtual disaster simulation. The study goal was to inform the use of e-learning or traditional textbooks in undergraduate nursing population health courses.

Background:

Weather-related disasters have increased in frequency and severity in the past ten years, with 2020 being the most active storm season ever seen (National Oceanographic and Atmospheric Administration, 2021.) Even with advances in early warning systems and mitigation efforts, educating student nurses in disaster response remains a priority. Due to the impact of Covid-19 quarantine policies, many in-person student learning labs and clinical experiences were cancelled. However, virtual simulation offers an alternative to developing nursing student skills and clinical reasoning ability (Aebersold, 2018; Fogg et al., 2020).

Design:

A randomized quasi-experimental, repeated measures 2 × 2 crossover design (Kim, 2018) was applied, which allowed students to participate in both the live and virtual simulations.

Methods:

Analysis was conducted using paired samples t-test to evaluate knowledge gains. To measure students’ self-assessment of knowledge, Unver et al. (2018) 12-item survey was administered. To explore students’ own perceptions about the disaster simulations, semi-structured interview questions were offered through private Wiki postings. The responses were analyzed using Saldanã’s in vivo coding (2015) and thematic analysis.

Results:

Students retained more empirical knowledge following the virtual assignment as compared to the disaster simulation, except in two items addressing triage. Neither age, years of education, or GPA impacted test results. However, students’ own assessment of learning did not differ between live and virtual simulations. In all but three items, students perceived a significant increase (p < .05) in their learning following the simulation, regardless whether it was live or virtual.

In narrative responses, students overwhelmingly cited the benefit of an in-person simulation. However, they did not believe that they were prepared adequately for the live simulation. They also expressed that they would be more prepared if the simulation was repeated. Students expressed discomfort, even distress, regarding not being able to care adequately for everyone, even though it was a simulation (See Table 5). This highlighted that live simulations can affect students emotionally, and follow-up debriefing is essential to help in both acknowledging and processing student feelings.

Conclusion:

These findings, which support the use of virtual disaster training in nursing education, are especially important in the light of Covid-19 and increasing threat of storm disasters.

1. Introduction

Weather-related disasters have increased in frequency and severity in the past ten years, with 2020 being the most active storm season ever seen (National Oceanographic and Atmospheric Administration, 2021). The winter months leading into 2021 culminated in the deadliest and costliest winter storms on record, with 125 deaths and 10 billion dollars in costs (National Oceanographic and Atmospheric Administration, 2021). Even with advances in early warning systems and mitigation efforts, educating student nurses in disaster response remains a priority. However, as the number of nursing schools increase, the demand for student clinical experiences has grown beyond what is often available. Narrowed preceptor-student ratios mandated by hospital partners, and lack of willing and experienced preceptors who were not already fatigued from teaching, have further limited the availability of clinical sites (Billings, 2015).

Prior to the pandemic, simulation was becoming more prevalent in prelicensure nursing education, as the benefits of learning critical skills using simulation are well documented (Curl et al., 2016). National standards for nursing education now support up to 50% of clinical education occurring in simulation (Curl et al., 2016). The Covid-19 pandemic of 2020 placed further demands for alternatives to face-to-face experiential learning.

In early work, Kaplan (2012) demonstrated the effectiveness of disaster simulations based on both quantitative and qualitative indications of positive (95%) student learning and colleagues. Other researchers demonstrated that virtual reality simulation consistently provides a realistic, immersive, and genuinely convincing environment for disaster training (Farra and Miller, 2013). Virtual reality simulation opportunities have continued to expand rapidly to meet the learning styles of today’s nursing students. This includes the iGeneration or Gen Z (born between 1995 and 2015) and Millennials or Gen Y, (born between 1980 and 1994), both of whom are well-known for their technological sophistication (Bell, 2013). Metaanalyses of published studies within the past five years revealed that disaster simulation experiences contribute to and increase student learning in several ways (Cant and Cooper, 2017), but more experimental research is needed to measure effects on learning outcomes (Staykova et al., 2017)). Furthermore, virtual simulation is receiving more attention as a teaching/learning modality due to the impact of Covid-19 quarantine policies: Many in-person student learning labs and clinical experiences were cancelled due to pandemic-related school closures and hospital restrictions. However, virtual simulation offers an alternative to developing nursing student skills and clinical reasoning ability (Aebersold, 2018; Fogg et al., 2020). Virtual simulation is also referred to as learning online, distance learning, web-based learning, or e-learning. E-learning indicates the use of electronic (online) textbooks, which often include interactive components, in which the user must respond to questions using the computer keyboard to move forward through the content or activities. Virtual simulations can be conducted any time, without face-to-face interaction. In contrast, live simulations are usually conducted during class time after traditional textbook reading assignments and in-person didactic teaching by faculty that are known to the students.

The motive for this study emerged during the piloting of a new e-learning textbook in an undergraduate nursing population health class. The students were offered the choice to participate in an online “e-learning” virtual disaster simulation format over reporting to campus for a live simulation. The majority of the students (75%) voted for the live disaster simulation. However, the textbook publisher was offering incentives to pilot their new online “e-learning” textbook which contained virtual disaster simulations. This consequently raised the question “Is a virtual disaster simulation as effective as a live disaster simulation in achieving similar learning among today’s Gen Y and Z students?” The study purpose was to determine if the type of disaster simulation (live or virtual) made a difference in a) student scores on a post-simulation formal assessment by examination and b) student perceptions of learning. The hypotheses, based on the students’ initial preferences and performances related to face-to-face disaster training, was that (A) live simulations would produce higher examination scores than the virtual simulations, and that (B) students would perceive that greater learning occurred from the live simulations. The specific aims were (1) to identify differences or similarities in students’ examination scores, and (2) to explore student perceptions of learning, after participating in both the live and virtual simulations. The primary expected outcome was that both student test scores and perceptions of learning would be higher in the live simulation group than the virtual simulation group. This predicted outcome was based on past experiences of high assessment scores averaging 88%, using traditional examination methods after face-to-face disaster simulations. The secondary outcome was that there would be a clear indication of which type of simulation students preferred. The expected goal was to inform the choice of assigning an e-learning textbook with virtual simulations, or a less-costly traditional textbook with a live simulation, for future undergraduate population health courses.

This education-focused research proposal was motivated by the Association of Community Health Nurse Educators (ACHNE) priority of “improving the public health nursing workforce” by increasing competency during disasters. This proposal also addressed a current goal of the funding agency, Wolters Kluwer, offered through ACHNE, to explore best educational practices in the classroom related to a public health topic of disaster preparedness and management.

The theories of Situated Cognition (Brown et al., 1989) and Nursing as Caring (Boykin and Schoenhofer, 2001) served as the conceptual frameworks for this study. The three key components comprising situated cognition are Embeddedness, Extension, and Embodiment. Farra et al. (2015) demonstrated the usefulness of these concepts in virtual simulations (see the definitions below in Fig. 1). For our study, “Situated Cognition” (Brown et al., 1989) was also adaptable for the live simulation, as represented in Fig. 1. The word “virtual” could be represented also as “live” as in the example, The Learner enters the Live Environment rather than the Virtual environment (by engagement in a live simulation.) Additional similarities between the live and virtual simulations included cognition linkage to sensorimotor brain and body, visual representation, and psychomotor and affective learning outcomes. Differences lay in the ability for repetition (first rectangle) and repeated practice (second rectangle) that are available in the virtual simulation but were not in the live simulation (Fig. 1).

Fig. 1.

Situated Cognition. (Farra et al., 2015). Virtual reality disaster training: Translation to practice. Nurse Education in Practice, 15(1), 53–57).

The holistic concepts grounding the Nursing as Caring theory, such as patient-centered care and the importance of seeing beyond what may be immediately obvious was applied when building the qualitative inquiry. The qualitative questions posed to the students in the Wiki discussion are available in Fig. 2. Boykin & Schoenhofer’s specific concepts were discussed between the researchers during the in vivo analysis, including that all persons are unique, that it is the nurse’s responsibility to determine “what matters most” to the person being nursed, and that authentic presence is essential in answering the call to nurse (2001). Focusing on these concepts during the reading and rereading of the students’ statements facilitated the identification of general themes and selection of in vivo codes that were more likely to reflect the participants’ meanings.

Fig. 2.

Qualitative inquiry debriefing questions.

2. Literature review

Using a combination of all of the search terms, online disaster simulations in nursing education, across all data bases available to the university (including PubMed, CINAHL, & Science Direct) and criteria of “peer-reviewed, nursing and allied health discipline, within the past five years in full-text” a total of 390 articles were available. Narrowing the search to include the Boolean phrase “disaster simulation nursing” or “related text” such as emergency preparedness within the title or full text, 60 relevant articles published from 2013 to the present emerged. The search was updated this year in preparation for this article using the equivalent search terms.

Of those 60 studies, fifteen were eliminated for duplicity including metaanalyses, or reported as conference proceedings. Articles were eliminated if the primary focus was on a related topic rather than the simulation itself, such as discussion of models to use in triage, targeting of post-graduate audiences, or general discussion on simulation. Those focusing on specific populations such as veterans (2) and pediatrics (3) or for a different primary purpose other than an localized incident such as interprofessional team-building (5), use of case studies or debriefing (5), widespread mass casualty involving multiple agencies (5), safety (3), ethics (2), transport issues (2), and evaluating a new scale (1) were excluded as a means of narrowing the scope of this review. Only two additional articles meeting the research criteria were found when searching the above references.

In total, eleven studies were located that met the research criteria. Four studies focused on virtual simulations, three on live, and four on both live and virtual simulations. The number of overall articles found were congruent with an integrative review conducted during the previous ten years by Jose (2014) searching in three similar databases (Ovid rather than Science Direct) using the key words of disaster, preparedness, nursing, education. From Jose’s review, 109 articles emerged, but only ten articles met the research criteria. The summarized findings from these articles are reported as follows.

2.1. Live simulations

Following a didactic lecture and in-class live simulation, researchers found a significant improvement in t-test results (p < .01) between pre-post test scores (Alim et al., 2015). They also concluded from observer ratings and student interviews that the classroom training (n = 309) and drill (n = 25) improved student learning. The simulation included students and no-student actors with moulage and multiple hospital departments. This study targeted associate and diploma degree students.

A study conducted by Davis et al. (2020) in a southeastern United States nursing college included 391 BSN students. The researchers conducted a quasi-experimental, pre-post test design using high fidelity simulation to assess the effectiveness on student knowledge and preparedness. Results indicated an increase in scores after participation in the simulation at every level. In their scale of Disaster Knowledge Competency Scores, there was a statistically significant (p < 0.01) difference in post test scores for Junior I (p = 0.0022) and (p < 0.0001) for Senior II. Although increases in knowledge were reported for the Senior Level I, it did not reach statistical significance.

At a university which had a Disaster Simulation Lab, a full-scale mass casualty incident (MCI) was implemented (Kim and Lee, 2020). The simulation environment included pre-hospital and hospital sections, with videos displayed on a large screen and sound effects played on loudspeakers. The results showed that participants were initially likely to triage insufficiently prior to the intervention. There was a statistically significant difference (p < .001) in effective triage, and positive attitudes after the intervention. Self-reported teamwork was high, with leadership and team coordination scoring the highest. In addition, awareness of roles, communication, and satisfaction with the simulation were rated highly.

Unver et al. (2018) also conducted a quasi-experimental, pre-post test design using a high-fidelity simulator among associate degree nursing students to examine the effectiveness of a live simulation upon student learning. Results included that prior to the intervention, less than half (43%) of students felt confident in their disaster preparedness. A statistically significant (p < 0.05) difference was found in their scores on the Scale of Perception of Disaster Preparedness among Nurses in the post-disaster evaluation. Confidence in disaster training is of paramount importance.

In a large teaching hospital with a state-of-the-art simulation lab, a full-scale mass casualty incident (MCI) was conducted (Fletcher et al., 2015) Fifteen students received moulage and played disaster victims following a bus accident. Eight other students were divided into groups and assigned the task of conducting 30-second triage. All participants felt it was a valuable learning experience. The extensive moulage (such as dried chicken bones for compound fractures, and glass shards embedded in wigs), fog machines, strobe lights and emergency siren sounds were utilized. However, adequate attention to other tasks, such as establishing a safe zone, delegating others to create staging areas by severity of injury, recording persons and injuries, finding family members, and arranging for transport were not included. During the debriefing, the students talked about their feelings of abandoning victims for whom death was imminent, feelings of helplessness, and trying to cope with bystanders was discussed.

2.2. Virtual simulations

Mixed methods research conducted among 82 nursing students demonstrated that computer simulations were extremely well received (Donovan et al., 2018). Themes that emerged from the analysis of narrative data included improved prioritization, benefits of role-modeled nursing care, engaged critical thinking, and decreased anxiety levels. Quantitative results also supported positive student performance. However, the computer simulations were used as preparatory work prior to the static and high-fidelity simulations, and not as stand-alone learning.

Researchers examined 14 databases between 2010 and 2015 for proof of effectiveness of virtual simulation for any health provider learners, and found overwhelming evidence that “Online virtual simulation was comparable or superior to traditional simulation methods where increased engagement with learning occurred in a safe environment with convenient access” (Duff et al., 2016, p. 383). Students consistently reported that virtual patients were more realistic than standard actor patients or students, and that they preferred the increased length of scenarios and engagement with learning. This information is imperative to the advantages of virtual simulations.

Another group of researchers investigated online learning by comparing web-based module instruction (the control group) with web-based modules and virtual reality simulation (Farra et al., 2015). Knowledge was evaluated using a 20-item measure that underwent content validity appraisal by both education and disaster experts, and reliability testing using test-retest (r = .72). Three measurement points (pre, post, and two-months post) showed that in the web-based module, knowledge actually decreased more over time, and in virtual reality simulation, knowledge was retained, with the most significant difference (p < .0001) between the two groups at the most distant third measurement. The researchers also cited numerous studies that have “explored the use of virtual reality simulation for disaster training with ”great success” (p. 55). They did not compare virtual simulation with live training.

Two studies were found that compared live with virtual simulations. In the first, researchers used mannequins and virtual scenarios (Wilson et al., 2014). Faculty provided classroom didactic training first, then the online virtual simulation to 54 BSN students. They also used a quasi-experimental crossover design with random assignment to a computer-based case simulation and a human patient simulation using mannequins. Although overall scores were higher in the human patient simulation group, the researchers found that different phases in each elicited better performance, suggesting that a combination of both online virtual simulation and live virtual simulation may be ideal for learning.

In the second article comparing live with virtual simulations, researchers compared the effectiveness of virtual reality to clinical simulations among a variety of health profession students in executing the START (Simple Triage and Rapid Treatment) triage model. They also examined the levels of salivary cortisol (a-amylase) in measuring the stress produced in each. The found that the virtual reality method was as efficient as clinical simulation for training on the START model execution, and caused less stress. Specifically, the percentage of victims that were triaged correctly averaged 88.1% (SD = 9.) for the Clinical Simulation with Actors groups, and 87% (SD = 7.2) for the Virtual Reality Simulation group, with no significant differences (p = 0.6) between both groups. However, the increase in salivary cortisol was significantly greater (p < 0.001) for the Clinical Simulation with Actors group (182.2, SD = 148.7 U/L) than the Virtual Reality Simulation group (80.7, SD = 109.7 U/mL).

In summary, multiple researchers highlighted in their studies that 1) disaster preparedness education through simulations has been found to be highly effective for learning and, 2) more rigorous studies are needed in this area due to the wide array of methodologies, data collection, and analysis. Students transitioning to the registered nurse role who may be called upon to lead during times of disaster events must receive adequate training to appropriately and confidently apply disaster management skills. This study added to the current literature by comparing two methods that would be feasible in the general classroom, particularly when access to large simulation laboratories or external emergency care and/or interdisciplinary venues are limited.

3. Methods

A randomized quasi-experimental repeated measures 2 × 2 cross-over design (Kim, 2018) was applied, which allowed students to participate in both the live and virtual simulations. Analysis was conducted using paired samples t-test to compare pre-post knowledge scores. Pearson’s correlational analysis was applied to determine if any of the sociodemographic variables (age, years of education, and nursing courses GPA) were associated with scores. To measure students’ self-assessment of knowledge, Unver and colleague’s (2018) 12-item survey regarding disaster simulation was administered. To explore student perceptions about learning from disaster simulations, and not just their knowledge, semi-structured interview questions were offered through private Wiki postings. The interview responses were analyzed using Saldaña’s in vivo coding (2015) and thematic analysis. This qualitative inquiry contributed to a greater understanding regarding potential barriers or successes in learning between the two platforms (live and virtual).

3.1. Setting

Students attending a Florida public university in their junior year, first semester of a traditional undergraduate BSN track, and accelerated (prior health science degree holding) students in their second of five semesters participated in this study. The XXX county setting included a diverse community. Over 23% of the general population are > 65. African Americans comprise 19.4%, while Hispanics 21.5%. Over 30% are speakers of a non-English language, as compared to the national average of 21.5%, including 4.8% Creole speakers, which is unusually high. This diversity was represented in the university’s nursing student body of 43% African American, 13% Hispanic, and 4% Asian. The average GPA for admission in this class was 3.75.

3.2. Sample

The target population was 90 junior baccalaureate science (BSN) nursing students from varied ethnicities and race, but with similar education backgrounds. A minimum GPA of 3.5 was required for admittance to the BSN program at this university. Applying GPower 3.1 paired samples a-priori parameters of.5 medium effect size and 95% power, differences were calculated between the two-dependent means/matched pairs criteria (Faul et al., 2007). This resulted in a recommended sample size of 45 for each group, which was similar to the design of a 2 × 2 crossover design in a nursing simulation involving 82 students in comparing roleplay versus lecture (Kim, 2018), and other online disaster simulation studies (Donovan et al., 2018; Duff et al., 2016). Students were randomized into either the live simulation or the virtual simulation as their first experience by drawing folded slips of paper with a choice from a basket. Neither the student nor the faculty was aware to which group the student was being assigned. Inclusion criteria were any juniors taking population health during the Fall 2019 semester. Exclusion criteria were any students who would be unable to participate in both forms (virtual and live simulations), or who were members of previous population health classes.

The importance of adhering to ethical standards of nursing conduct by faculty included supporting the student’s autonomy to engage in class activities, avoiding maleficience (harm) that could be caused by faculty disregard or disapproval of students’ live simulation performances, and upholding beneficence by promoting safe, student-centered learning. Students were informed that likewise, their conduct was expected to be congruent with the American Nursing Association Code of Conduct (Haddad and Geiger, 2021) ethical principles, such as veracity (truth--telling), integrity (thoroughly completing all activities in each section of the virtual disaster simulation), and fidelity to their future communities through a full commitment to learning. Participation was voluntary, and informed consent was obtained from students prior to their participation in the simulation. Student protections and confidentiality parameters were followed per the university’s Insititutional Review Board (#719852.) There was no penalty for not participating in the randomized trial. Data Collection occurred following completion of voluntarily signed consents. Following the intervention, all students (both groups) were offered the opportunity to participate in the mode of simulation to which they did not participate earlier. Gift cards were offered to help offset the cost of the textbook with the added virtual simulation.

The overall Goals of Student Learning are available in Fig. 3.

Fig. 3.

Goals of learning for disaster simulations.

3.3. Measures

To meet the specific aims, measures included a sociodemographic survey, pre-and post-test based on an Elsevier test bank, and a self-assessment of knowledge gained. Each of these measures were reviewed by three other faculty members with at least five years experience in teaching public health nursing, and one faculty member whom has worked with supervising disaster shelters for over 40 years, and two who were CERT trained. Content validity of both the formal assessment and the live simulation were also examined by the public health nurse educators, discriminant validity was examined to some degree by the pre-test (measuring the knowledge base of students without disaster training). The principal investigator was familiar with conducting the live disaster simulation, which she designed and taught to the population health class during the past three years.

The sociodemographic survey included questions regarding age, gender, race, ethnicity, current nursing courses GPA, and three items regarding the type and length of previous disaster emergency management in these categories: 1) education/training, 2) professional experience during disasters, and 3) personal experience with disaster. The pre- and post- tests were each twenty questions taken from the Disaster Management chapters testbank in the e-textbook, which were derived from empirically validated and widely tested questions. Two sets of tests were created to accommodate for students participating in both types of simulations. Psychometric analysis was conducted on the student score results from the tests. Unver et al. (2018) 12-item survey regarding disaster simulation was administered as a means of measuring students’ self-assessment of learning.

3.4. Simulation procedures

Procedures for both the live and virtual simulations consisted of three phases; preparation, simulation, and debriefing. To prepare for the live simulation, the students drew cards naming their roles. During the first hour, they applied moulage to victims, and reviewed supplies that were available with the First Aid kit, including triage tape. In preparation for the virtual simulation, students were directed to complete the textbook readings and modules (Disaster Assessment and Response, Module 3, and Public Health Nursing in Post Disaster Recovery, Module 4). For the live simulation phase, students reported to their assigned rooms. The scenario presented to students was that they were in a bus returning home from a university academic competition when they encountered a tornado. You-tube footage from a local news station was shown of an approaching tornado from a bus window that actually occurred in our state. Each index card that students drew for their assigned role also included instructions for acting out the role on one side, and brief tips for the triage nurse to consider if needed. For the virtual simulation phase, students completed the disaster simulations that were available in Modules 3 and 4 of the of the text.

For the live simulation debriefing phase, faculty met with students and conducted open-ended discussions regarding “what went well”, “what could have been improved”, and how they felt when acting out their assigned roles. For the virtual simulations, students completed the seven open-ended questions presented in private Wiki online discussions with faculty that explored the students’ lived experience of both the simulation (See Fig. 2). For both simulations, students were asked “what mattered most” (Boykin and Schoenhofer, 2001) to them when engaging in the simulations.

3.5. Data analysis

Test scores were analyzed in terms of mean, median, mode, frequencies, and percentage of both individual and total of item scores. Descriptive and Pearson’s r correlational analyses were used to examine the relationship between sociodemographic variables (age, gender, race, ethnicity, self-reported GPA, prior personal or professional disaster experience, years of post-high school education) with student test scores. Multiple linear regression analysis was applied for evaluating if any of the independent variables significantly predicted student scores. Paired samples t-test were used to quantify differences in learning based on pre-and post-intervention summative assessments. In addition, narrative student responses from the qualitative semi-structured interview questions as a function of debriefing were explored using in Saldaña’s vivo coding and thematic analysis to add to the understanding of student perceptions of barriers and successes to learning.

4. Results

The majority of students participated in both phases of the study; N = 80 of 90 (89%). Six students who chose not to engage in the study cited the cost of purchasing the virtual textbook, even with the gift card support. Four others did not offer a reason for refraining from enrolling in the study. All ten students received the disaster didactic content via the traditional lecture format in the classroom with their colleagues. The sociodemographics of the sample (Table 1) revealed that the participants, as expected, were a highly racially and ethnically diverse group, as the university is the most racially and ethnically diverse public university in the state.

Table 1.

Participant Characteristics (N = 80).

Categorical	n	%
African American	17	21
Afro Caribbean	18	22
Asian	3	.4
Hispanic or Latino	19	24
European Caucasian	23	29
Female	57	70.1
Male	12	5.8
Other/Undisclosed	11	13.1
Prior training in education in disasters	25	32.0
Professional experience in disaster management	10	12.5
Continuous	M	SD	Min.	Max.
Age	26.5	3.49	16	37
GPA	3.9	.12	3.6	4.0
Personal experience with disasters	45	55.5

4.1. Correlations between sociodemographic variables and test scores

Using Pearson’s r, there were no significant correlations found between the continuous dependent variables of age or GPA and the independent variable of test score following either the live or disaster simulation. Neither was there any significant correlation using Pearson chi-square analysis between the categorical variables of gender, disaster training, or professional experience with students’ knowledge scores. As expected, there were significant correlations between students with higher knowledge scores and professional work experience. (r = .56, p. =.01). However, only four students reported personal experience with disasters, although 21 had prior Red Cross table-top training. Of the 50 students indicating no prior training or education in disaster management, 35 (70%) of them indicated that the virtual simulation greatly improved learning (Table 2). Results were very similar among students who reported no personal experience; 61% reported that virtual simulation greatly improved learning (Table 2).

Table 2.

Effect of prior professional training or personal experience on students’ perceptions of improved knowledge after participating in disaster simulations.

	Students with prior professional disaster training/education
	None	Yes
Improved knowledge from both simulations	Disagree	3	6
	Somewhat Agree	12	6
	Greatly Agree	35	9
Total student responses	50	21
	Students experiencing disasters personally
	None	Yes
	Disagree	12	0
	Somewhat Agree	12	0
	Greatly Agree	38	4
Total student responses	62	4

4.2. Perception of learning between live and virtual simulations

As seen in Table 3, and using Unver et al. (2018) measure, significant correlations were found overall between the pre-test and post test scores after the live simulation using a paired samples t-test (t = 3.57, 78, P = .001). Significant increases in perceptions of learning were demonstrated across all but three items;.

Table 3.

Comparison of significance in learning between virtual and live simulations.

Items measuring perceptions of learning (Unver et al., 2018)	Virtual simulation (Sig. p value)	Live simulation (Sig. p value)
1. Combined theoretical & practical knowledge	.016	.68
2. Improved critical thinking skills	.017	.04
3. Improved decision-making skills	.001	.03
4. Allowed connection with real life facts	.001	.001
5. Allowed recognition of learning needs	.003	.001
6. Improved self-esteem before a clinical assignment	.027	.039 *
7. Felt like a nurse	.06	.166
8. Increased motivation	.001	.001
9. Understood significance of communication/collaboration	.005	.001
10. Improved disaster knowledge in debriefing activity	.02	.03
11. Pleased with disaster training	.001	.03
12. Recommend using this format in other trainings	.001	.001

Note:

^*= students did not have access to a clinical assignment prior to the simulations.

#1: “Combined theoretical and practical knowledge” demonstrated student perception that the virtual simulation learning was significant (p = .02), but the live simulation learning was not (p = .68).

#6: “Improved self esteem before a clinical assignment” did not achieve a significant difference in perceived learning in either the live or virtual simulation. However, the students did not have any clinical assignment associated with this learning activity.

#7: “I felt like a nurse” showed students rating the virtual activity as near significant learning (p. =.06) compared to the live simulation, which did not show any trend toward significance (p = .166).

4.3. Knowledge scores

Comparing the total test scores on a 25-item test derived from an Elsevier test bank revealed that students retained more knowledge following the virtual assignment as compared to the disaster simulation, except for two items addressing triage (establishing priority of care and use of triage tape). There was a five-point difference in average test scores between the live (M = 15.93, SD = 6.44) and virtual (M = 20.55, SD = 4.75). Reliability testing of the formal assessment was favorable. Following the pre-test for both the live and virtual simulation, the Cronbach’s alpha was a = .84 and.83 respectfully. The post-test results were similar, with a Cronbach’s alpha of.82 and.81 (Cronbach, 1951).

4.4. Qualitative inquiry

The participants answered a series of open-ended questions (See Fig. 2) in the format of an online Wiki-post assignment in the course learning management system. In vivo coding (Saldana, 2015) was the style of content analysis used to study student responses qualitatively to gain more understanding of their perceptions about disaster simulation training. This approach of content analysis relies on the participants’ own words to serve as themes rather than the investigator’s interpretation, thus preserving the participant’s language and perspective.

Three of the researchers independently read the transcripts, reread them, and began classifying the content into categories of main ideas. They then selected a phrase by the participant that reflected the category, prior to meeting to identifying similarities and reconciling differences. Three in vivo codes were agreed upon each type of simulation from the participants’ own verbiage as illustrated in Table 4.

Table 4.

In vivo coding to student responses in qualitative inquiry.

Examples of live simulation survey responses	In Vivo codes
• I looked around and it was so chaotic and I was not sure where to start.	“so chaotic”
• The live simulation was overwhelming for those who had to determine the severity of everyone’s injuries. Sometimes there was not a lot information on the persons condition, and it could be difficult deciding where to group everyone.
• It was organized chaos.
• I think it was a better to see the chaos than to sit back and relax at home and complete an assignment.
• It was easier to feel the chaos of a real disaster. It made me think of new things that I wouldn’t have thought about with just a virtual simulation.
• It was easier to feel the chaos of a real disaster. It made me think of new things that I wouldn’t have thought about with just a virtual simulation.
• I felt overwhelmed in the live simulation as it was unexpected. I did not know how to respond.
• At the beginning, I wasn’t sure of what I was doing but throughout the simulation I understood the process better.	“did not know how to respond”
• It was difficult not knowing exactly what was going on.
• I was one of the “victims” so I was not sure what to expect.
• I was on the response team, so it was a bit hectic at the beginning trying to figure out how to do things. I also realized my own lack of knowledge about how to address certain situations.
• I liked that the live simulation was more realistic to what we would have access to in a real life scenario. It also required us to treat those who we could temporarily. That allowed for additional learning.
• I enjoyed the experience as we all came together in the simulation. It was insightful to get a sense of what to expect and practice on how to react. I learned a lot about myself and what skills I need to work on.	“allowed for additional learning””
• I really the fact that everything was happening right on the spot. It prepares you for real-life situations like this event. In the event of a disaster we will know what to expect and how we can assist.
• I got to use my clinical techniques and nursing skills without restriction.
• It took me into the moment and gave me a great understanding of the pressure, sense of urgency and actual nursing interventions and awareness that a disaster requires.
• Even though I wasn’t triaging, it got me thinking about what I would have done or how I might have handled the situation. Planning and forethought are key, and I feel the live simulation facilitated more of that style of learning.
• It was easy to see the magnitude of the injuries.
• I got to witness and later listen to the rationales behind decision-making for the rescue team and survivors, which was great for applying this exercise to possible disasters in the future.
Examples of virtual simulation survey responses	In Vivo Code
• There was a safety associated with doing it virtually	“safety”
• I liked how you could practice a disaster response without hurting anyone.
• I could return to patients and place them into a different category if I felt that i originally labeled them wrong.
• It helps to understand the topic better. I also like how simple and easy it was to use and understand.
• I liked that in the virtual simulation they gave write ups and records of each patient needing triage. I think by having background information it enabled me to be able to triage them correctly.	“easy to use and understand”
• I liked that I had more time with the patients, and could really think about where I should place these patients.
• It alone gave a lot of learning.
• The virtual simulations were the best type of modules in my opinion. I found that they were fun and reinforced the book knowledge.
• I really enjoyed the virtual simulations and felt that they gave a great understanding of what it is like working in jails, disasters, doing windshield surveys, etc.	“gave a lot of learning”

5. Discussion

The purpose of this study was to answer the research question “Is a virtual disaster simulation as effective as a live disaster simulation in achieving similar learning among today’s Gen Y and Z students?” Several key findings were elicited from this study. First, both hypotheses were disproved, as students retained more knowledge following the virtual assignment as compared to the disaster simulation, except for two items addressing triage. This outcome was based on results from both the formal assessment (examination scores) conducted after the student’s first live or virtual simulation. In addition, students’ own perceptions of learning did not differ between live and virtual simulations, as shown by the results from the Unver et al. (2018) survey. In all but three items, students perceived a significant increase (p < .05) in their learning following the simulation - regardless whether it was live or virtual. Furthermore, neither age, years of education, or GPA impacted formal assessment results, as indicated by exam scores.

Another unexpected finding was found during the qualitative exploration of the live simulation. Although the students overwhelmingly cited the benefit of an in-person simulation, they did not believe that they were prepared adequately. They also expressed that they would be more prepared if the simulation was repeated. Students expressed discomfort, even distress, regarding not being able to care adequately for everyone, even though it was a simulation (See Table 5). This highlighted that simulations can affect students emotionally, and follow-up debriefing is essential to help in both acknowledging and processing student feelings.

Table 5.

Examples of survey responses to item asking about future simulation recommendations.

• Both because I feel that the virtual simulation helps to not feel so lost during the live one

• I think that both should be used in future classes because the virtual simulation prepares you for the live one.

• I am the type of learner where I need to see it and then be able to put it into practice.

• I felt more prepared for the live simulation as I had already completed the virtual simulation prior to the live one. I also prefer to learn in-person and hear/see the processes going on around me in a situation, making the live simulation more adaptive and appealing to me.

• I felt that I was more prepared for the live simulation, since we had an idea of what we all should expect from the virtual simulation

• The live, partly because I did the virtual early on and did the live drill after the benefit of the class learning.

• I think that both should be used in future classes because the virtual simulation prepares you for the live one

In contrast, students wrote in the Wiki posts that the virtual approach was a safe, fun, and effective means of learning how to act as a nurse. Students also noted the high quality and comprehensive content available in the virtual simulations. This suggests that perhaps more effort needs to be made by faculty to improve the didactic content and delivery associated with the live simulation. Of note was that 35% of students offered comments recommending that both the live and virtual simulation would be ideal, with the virtual offered first. Examples of these comments are provided in Table 5. However, the cost of the e-learning textbook may be a barrier to future use. To access the e-learning textbook, students must enter a code that is provided after purchase. Traditional textbooks are often less costly initially, and previously owned books are frequently available for purchase online at an even greater reduced rate.

5.1. Limitations

Limitations of this study included the cross-over effect; students participated in either the live or virtual simulation prior to engaging in the other. Thus, experience with the first type of simulation likely had some effect on the second. This was minimized somewhat by administering the formal assessment (examination) after each student’s first disaster simulation. The Unver survey and qualitative reflection were completed after both types of simulation were completed, and provided important insights.

Other limitations included testing in only one school of nursing, and inability to control for extraneous influences, such as technical difficulties, logistics in conducting ten groups of eight students each through a live simulation, and availability of persons experienced in applying moulage. Other potential sources of bias may have been present in the strength and consistency among test bank questions used to evaluate didactic learning, and lack of student honesty or adequate reflection in answering the narrative inquiries following the intervention. Furthermore, this was a fairly homogeneous group in terms of high GPAs and maturity; all students had completed a prior baccalaureate degree in a health-related field, so these results are not generalizable to nursing student bodies with different levels of academic prowess.

An important restriction to adopting the textbook with the virtual simulation was the cost of the e-textbook. Numerous students commented on this in the qualitative survey. Many students verbalized in class that if it had not been for the grant support, that they would not have been able to afford the additional $75 for the text, which is not resellable with the e-book option. Discussion is underway with the publisher to ascertain if the e-book subscription could be limited to six months rather than a year, and the cost decreased by half.

Additional limitations that potentially may have occurred based on reports in the current literature include the following: Alfred et al. (2015) reported on multiple strategies used in preparing nursing students in disaster management. Their work, and five other recent studies were not reviewed in this proposal, due to their emphasis on interdisciplinary/interagency disaster drills, which is outside the scope of this study. Rafferty-Semon et al. (2017) created and tested a novel and effective simulation for training students to serve at the point of collaboration (POC) i.e. disaster shelters, for which tabletop trainings have occurred, but not simulations. Chilton and Alfred (2017) investigated the use of both virtual simulations and live training for personal emergency preparedness among undergraduate students and RNS, but did not use online simulations with pre-licensure students. Researchers also tested a new topic in disaster simulation; offering a simulation that addresses personal readiness, including a “to go” bag. None of these additional three types of disasters a) interdisciplinary. b) points of collaboration, and c) personal disaster preparedness, were incorporated into this proposal. Any of these would no doubt be beneficial, but this study was limited in scope. The ten students who did not participate in the study were given a separate assignment to identify five learning points after completing the traditional textbook assignments, which could be another means of evaluating learning after a virtual simulation.

5.2. Conclusion

These findings, which support the use of virtual disaster training in nursing education, are especially important in the light of Covid-19 and increasing threat of storm disasters. Our results were congruent with those reported in a new systematic review of 69 studies by Foronda et al. (2020). The majority (86%) of authors concluded that virtual simulation was an effective pedagogy for meeting learning outcomes. Through both the quantitative and qualitative student responses, we learned that significant learning was achieved with the virtual simulation.

It would be helpful to incorporate student suggestions in live simulations, such as strengthening didactic content, particularly in regards to triage and priority-setting. An interesting suggestion was made managing color-coded systems when tape is not available: The triage nurse could write the letter of the corresponding triage tape letter on the bottom of the shoe, e.g., Y for yellow (meaning needing eventual but not critical medical attention), when highlighters or tape are not available. An important note is that the most important learning mechanism following any simulation is the debriefing period. The effectiveness of achieving positive knowledge and self-efficacy outcomes through simulation debriefing regardless of mode; in person, on paper, or by self-reflection, has shown to be very effective (Verkuyl et al., 2018). It is essential to include a debrief with instructor involvement and feedback regardless of the mechanisms used for the simulation. In addition to reinforcing learning, debriefing can facilitate student expression of feelings and concerns emerging from the simulation, and provide an opportunity for faculty to address “what matters most” (Boykin and Schoenhofer, 2001) to the student following a disaster learning experience.