High-fidelity simulation and virtual reality: an evaluation of medical students’ experiences

Abstract

Background

Simulation technology is widely used in medical education, providing an environment in which students can develop and practise a multitude of skills that are relevant to clinical practice, without the risk of harm to patients.

Methods

We conducted a mixed methods cross-over study with quantitative and qualitative outcomes. This analysed students’ perceptions of two simulation technologies: a high-fidelity patient simulator and virtual reality. Twenty final year medical students completed a questionnaire after having experienced both simulation modalities.

Results

Students scored the patient simulator higher in domains such as developing team working and ‘ABCDE assessment skills’, whereas the virtual reality simulation was more immersive and fun. Participants found the patient simulator more useful in preparing them for clinical practice.

Conclusion

Medical students in this study expressed that a high-fidelity patient simulator, in a simulated clinical environment, was of greater value to their preparation for clinical practice than virtual reality simulation of a similar environment. However, the virtual reality simulation offered a near comparable experience, and was found to be was enjoyable, immersive and easily portable.

What is already known on this subject.

• Simulation is used widely in medical education, with many different simulation technologies being used to facilitate acquisition of clinical skills.

• Novel simulation technologies, such as virtual reality simulation, have been developed to address some of the known limitations of simulators, including high-fidelity manikins.

What this study adds.

• This mixed-methods study evaluates two types of simulation technology: a high-fidelity manikin and virtual reality software. Recognising that learning experience is an important aspect of quality education, this research compared student perceptions of the two technologies across a range of difference domains.

• This research also explores the two simulation technologies from the perspective of implementing the technologies into an undergraduate medical curriculum.

Introduction

Simulation is used in many countries to facilitate teaching and learning in undergraduate and postgraduate medical education.¹ Due to the nearly limitless range of skills potentially recreated in a simulated environment without impacting on patient safety, medical schools across many countries have formally adopted simulation into their medical education curriculum. Virtually all UK medical schools use a variety of simulation techniques to support their curricula,^{2 3} with many using at least some of the features of high-fidelity patient manikins such as SimMan 3G.

While high-fidelity manikins incorporate many features which support the immersivity of the scenario, other aspects such as pallor, heat and the general ‘look’ of the patient are features that are difficult to replicate with a manikin.^4–7 These limitations are well recognised and have contributed towards the creation of a more realistic and immersive experience in medical simulation, in the form of virtual reality.^{4 8} Virtual reality is already used in some aspects of healthcare, such as surgical training and as a therapeutic intervention for some forms of mental illness.⁹ This evolving form of simulation has the potential to provide a more immersive experience for students and has numerous possible applications in medical education, not least practising emergency patient management.⁸ Although there is little research comparing virtual reality to high-fidelity patient simulators in medical education, there is some evidence to suggest that complete immersion, such as that delivered by virtual reality with a head-mounted display, helps to improve knowledge acquisition and conceptual learning.^{10 11}

With the aims of addressing some of the limitations of high-fidelity simulation and potentially providing a more immersive experience for students, virtual reality technology, in the form of Oxford Medical Simulation (OMS)’s simulation package,¹² has been introduced into the final year medical student simulation sessions at the Hull York Medical School. The simulation package provided by OMS is designed to be used with a head-mounted display, incorporating auditory input, in order to provide a fully immersive experience. The package includes a variety of simulation scenarios, each allowing the student to virtually examine, investigate and to treat a patient in real (simulation) time, with physiological responses to actions and interventions as the scenario progresses. However, understanding of how best to use these evolving virtual reality medical simulation scenarios in undergraduate education is limited. Although simulation using interactive computer programmes presented on a screen has previously been compared with high-fidelity simulation using a manikin,¹³ we are not aware of any published studies comparing truly immersive virtual reality using a modern headset to traditional manikin-based simulation. Our aim was to conduct an initial pilot study to compare the two technologies across different domains and consider how the two modalities may be used within the curriculum to optimise their capabilities. We hypothesised that the students would score the manikin based high-fidelity patient simulator higher in areas such as team working skills and communication, with there being little or no difference between the two modalities in other domains, while immersivity may be increased using the virtual reality environment.

Methods

Study population

The research took place in the simulation suite at Castle Hill Hospital, Hull University Teaching Hospitals NHS Trust, one of the main clinical teaching bases of the Hull York Medical School. All final year medical students were invited to participate in the research. At the time of the study, students had completed their final year of the MBBS programme. They were in their assistantship period prior to graduation. Recruitment was primarily through online advertisements which were displayed on the students’ virtual learning environment. It included full details of the research, including a participant information leaflet.

An a priori sample size calculation was not performed as there was no data on the SD of the outcome measure on which to base a power calculation. A total of 20 participants volunteered for involvement in the research. No students met the exclusion criteria, defined by exclusions outlined in the product licensing of the virtual reality system.¹⁴ Thus all 20 students participated in the research, following written informed consent.

Study design and interventions

The research team leading the simulation sessions had all received formal training and had previous experience in delivering teaching sessions using both SimMan 3G and the OMS virtual reality software. Four scenarios were chosen for the study: acute asthma, upper gastrointestinal bleeding, bacterial meningitis and anaphylaxis, based on the pre-designed scenarios in the OMS software, which were predominantly medically based acute care scenarios. The four scenarios reflected core content of the curriculum, and the chosen scenarios were then matched using SimMan 3G.

This study was run alongside standard simulation training for final year students. In this simulation training, students are formatively assessed against competency domains in relation to conducting an ABCDE assessment on an unwell patient. In addition, the study aimed to capture student perceptions about the technology while these competencies were being addressed during training.

All students were shown the official introductory video for the OMS virtual reality technology prior to completing the virtual reality scenario.¹⁵ This video introduced the simulated environment, instructed on how to interact within the virtual environment and explored the options available during the scenario. Students were already familiar with SimMan 3G and the simulation suite having used it on a number of previous occasions. One of the key differences identified was the differences in prompts and cues within the scenario. In contrast to prompts being provided by a real nurse assisting in the SimMan 3G scenario, the OMS virtual reality software uses a dropdown selection process for clinical decisions. By using the introductory video prior to virtual reality being introduced, we aimed to address these differences so that participants could solely focus on the response and the underlying clinical reasoning for their decisions.

This was an initial scoping study, using a cross-over study design. Twenty students were randomly divided into two groups: Group A and Group B. The study was performed over five small-group simulation sessions. The students in Group A each completed one of the individual simulation scenarios using the OMS virtual reality technology first, while the students in Group B completed an individual scenario using the SimMan 3G first. Each student completed a different scenario to their peers in the same study group. Following a break, the two groups then completed a matched clinical scenario using the other simulation technology (figure 1).

Figure 1.

Flow of participants through the study.

Assessment measures

Students completed a questionnaire (available in online supplemental materials) after each scenario, which aimed to evaluate the two forms of technology at Kirkpatricks’ level one (reaction),¹⁶ by asking students about their experiences of each of the simulation technologies. The questionnaire comprised two sections. The first asked students to rate their perceptions of the simulation technology over nine different domains, including teamworking, immersivity and ease of use, using five-point Likert items. The second part asked open questions to gain qualitative feedback about students’ experiences. In addition, once students had completed the scenario in both technologies, there was a final question asking students to choose which technology they felt was most useful in helping them to prepare for real clinical practice. The questionnaires were pseudonymised using numerical identifiers.

We used non-parametric statistics for analysis of the quantitative data on the basis that our questionnaire comprised individual Likert items, as opposed to a Likert scale.^{17 18} The cross-over pairs were analysed by the Wilcoxon signed-rank test using a nominal level of 5% statistical significance (two-tailed), performed with Stata software. A thematic analysis of the qualitative feedback was performed independently by two investigators. For the purpose of this study, the SimMan 3G simulation was regarded as the ‘gold standard’.

Results

All 20 students completed the study, with no missing data for the Likert items or qualitative questions. There was one missing data point for the final question, which has been addressed in our analysis.

Quantitative data

A summary of the Likert item analysis is shown in table 1. Students agreed or strongly agreed that both simulation technologies improved their confidence in performing an ABCDE assessment, but with the SimMan 3G simulation scoring significantly higher (p<0.01). More students agreed or strongly agreed that the SimMan 3G simulation provided a useful learning experience as compared with the OMS virtual reality simulation (p<0.02). A higher proportion of students agreed or strongly agreed that the SimMan 3G simulation provided a realistic experience (p<0.01). Interestingly, 20% of students disagreed that the OMS virtual reality simulation provided a realistic experience. Despite this difference in students’ perceptions towards the realism of the experience, there was no significant difference in reported immersivity (p=0.1). Notably, 15% of students disagreed that during the SimMan 3G simulation, they felt fully immersed in the clinical scenario, rather than being aware of their peers observing. With regards to demonstrating team working skills, the SimMan 3G simulation was regarded more positively than the OMS virtual reality simulation (p<0.05), with 40% disagreeing or strongly disagreeing that the OMS virtual reality simulation permitted the student to demonstrate their team working skills.

Table 1.

A summary of the Likert item analysis

Question	Proportion of students who agreed or strongly agreed for SimMan 3G (%)	Proportion of students who agreed or strongly agreed for VR (%)	Prob>|z|
I feel more confident in performing an ABCDE assessment on an unwell patient	95	70	0.002
I enjoyed the learning experience provided by the simulation technology	100	90	0.500
The simulation technology used provided a useful learning experience	100	90	0.017
The simulation technology used provided a realistic experience	85	50	0.005
I felt fully immersed in the clinical scenario rather than being aware of my peers observing	75	95	0.101
I felt there was a significant degree of prompting during the scenario	35	75	0.005
I felt the scenario allowed me to develop and demonstrate my team working skills	90	25	<0.001
I felt the scenario allowed me to develop and demonstrate my clinical reasoning skills	90	85	0.047
I found the simulation technology easy to use	95	70	0.006

VR, Virtual reality.

Facilitating the development of students’ clinical reasoning skills was also explored, with more students agreeing or strongly agreeing that SimMan 3G simulation allowed them to develop these skills (p<0.05). For both categories, the proportions were overall similar between the technologies but with more students strongly agreeing that they could develop clinical reasoning in the SimMan 3G scenario (SimMan 3G 55% vs virtual reality 25%). More students agreed or strongly agreed that SimMan 3G was easy to use when compared with the OMS virtual reality (p<0.05). Students enjoyed using both technologies. All students either agreed or strongly agreed that they enjoyed SimMan 3G scenarios, and 90% of students said they felt this way about their experience of the OMS virtual reality simulation (p>0.05). Although this suggests that student enjoyment of the OMS virtual reality was minimally less, more students (65%) strongly agreed that they enjoyed their experience with OMS virtual reality technology compared with 45% of respondents using the same rating to describe their enjoyment of the SimMan 3G scenarios.

With regards to prompting, 75% of students agreed or strongly agreed that there was a significant amount of prompting during the OMS virtual reality scenario (p<0.01). In contrast, just 35% of students either agreed or strongly agreed that this was the case within the SimMan 3G, with an equal number disagreeing or strongly disagreeing with this statement. Finally, students were asked to decide which simulation technology they felt was most useful overall in helping them to prepare for real clinical practice. Nineteen of the 20 participants answered this question, with a significant majority, 18 out of the 19 students, deciding that the SimMan 3G was superior in achieving this aim.

A summary of the Likert item analysis is shown in table 1.

Qualitative data

Students were asked to comment on the positive aspects of the simulation technology, the negative aspects or limitations of the technology and if they had any further comments. A thematic analysis of the students’ responses showed two main themes: their perceptions of the simulation experience and how the simulation relates to clinical practice (figures 2 and 3). Feedback on the simulation experience was further divided into the common themes of realism, technical aspects, prompting, immersivity of the experience, enjoyment of the scenario and variety of the options available within the scenario. The themes that were identified in relation to clinical practice were communication and handover skills, drug prescribing, ability to perform a systematic assessment, access to guidelines and investigations and gaining practical experience of performing examinations and assessments (figures 2 and 3).

Figure 2.

Summary of feedback for SimMan 3G.

Figure 3.

Summary of feedback for Oxford Medical Simulation virtual reality technology.

A summary of the qualitative feedback is provided in tables 2 and 3.

Table 2.

A summary of the qualitative feedback for SimMan 3G

SimMan 3G qualitative feedback
Theme	Subtheme	Positive comments	Negative comments
Clinical practice	Communication and handover skills	‘Able to communicate with nurse and patient’14 ‘Good to practice SBAR handover’11	–
	Drug prescribing	‘Good to prescribe initial drugs and fluids’10	–
	Ability to perform a systematic assessment	‘I could organise my thoughts more easily’8	–
	Access to guidelines and investigations	–	‘No immediate results for bloods/scans and so on therefore can only take the simulation so far’15
	Gaining practical experience of performing examinations and assessments	‘Having a physical patient allowed you to carry out the actions, making them more memorable’9	–
Simulation experience	Realism	‘Good life-like experience of what it’s like in a hospital situation’1	‘Difficult to imagine clinically what the patient looks like to help give an impression of how the patient is’18 ‘Aspects that cannot be created, for example, rash/skin tone’5
	Technical aspects	‘Good camera angles’1	‘Never know if your (sic) not seeing a sign due to it not being present or if the sign just isn’t able to be produced by the model’7
	Prompting	‘No prompts/less (sic) prompts making simulation more realistic’15 ‘Allowed to think for yourself without prompting’13	–
	Immersivity	–	‘Not quite as immersive having people that you know ‘acting’ in scenarios (eg, being the nurse assistant)’2
	Enjoyment	–	–
	Variety	–	–

SBAR, Situation, Background, Assessment, Recommendation.

Table 3.

A summary of the qualitative feedback for virtual reality

Virtual reality qualitative feedback
Theme	Subtheme	Positive comments	Negative comments
Clinical practice	Communication and handover skills	–	‘No practise of communication’15 ‘Much more difficult than speaking and discussing A–E and management’13
	Drug prescribing	‘…easy to use and prescribe’10	‘Needs opportunity to look up drugs and doses instead of suggesting them’17 ‘No thinking involved in choosing Ix (investigations) and meds’6
	Ability to perform a systematic assessment	‘Felt that it allowed for structured approach’1 ‘Experience that worked through A–E’7	‘It made it difficult to structure assessment’8
	Access to guidelines and investigations	‘The access to guidelines/high-quality investigations results was very good.’2	–
	Gaining practical experience of performing examinations and assessments	–	‘Physically doing it is much more helpful—you can’t pick the questions or feel an abdomen’17
Simulation experience	Realism	‘It could give certain signs that sim man would not provide.’8 ‘… provided a better simulation of a ward environment than other types of simulation’9	‘Lots of thing (sic) were readily available that is, reg call, guidelines etc so this made it a bit unrealistic’8 ‘Time pressure unrealistic’19
	Technical aspects	‘Good system, easy to use’10	‘Due to the pressure of the mask on my glasses I had trouble focusing on high detail parts of the screen’1 ‘you have to really search for what you want to do, in real life you just do it!’17
	Prompting	‘Able to make clinical choices and have prompts for this’15	‘Maybe too much prompting, as inexperienced clinicians its good to learn the A–E w/o prompts’3
	Immersivity	‘Good to be immersed into virtual ward’14 ‘Very immersive, forget your (sic) in a room of people’11	–
	Enjoyment	‘Different and fun way to practise simulation’15	–
	Variety	‘I felt the range of options for communication, management and others were very good’1 ‘…possible to create more varied scenarios than with Sim Man’2	–

Discussion

Previous high-fidelity patient simulator research has shown that this technology can help to improve the acquisition of relevant knowledge and skills and improve student confidence in patient assessments.^19–22 In our study, final year medical students experienced both a high-fidelity patient simulator and a new virtual reality simulation environment when training for acute emergency scenarios. The high-fidelity patient simulator was used as the benchmark to assess the utility of the newer and evolving virtual reality technology in day-to-day clinical education. Overall, the students scored SimMan 3G higher across several domains than the newer OMS virtual reality software environment. The students reported that the physical high-fidelity manikin, with haptic feedback, provided a more useful learning experience and improved their confidence more in managing acute emergency scenarios than the virtual reality simulation environment. The students also felt that the high-fidelity manikin was better at providing a familiar and realistic experience, provided less prompting during the scenario, and was also easier to use.

This is understandable for two reasons. First, these final year students had been exposed to SimMan 3G on at least four previous occasions in their final year, whereas it was their first exposure to the virtual reality simulator. Second, their physical presence in a simulated emergency room with a nurse, played by a member of faculty, and a physical manikin is a very familiar environment for medical students. This can be contrasted with the more novel environment of the virtual reality emergency room, avatar patient and avatar nurse. It is therefore interesting to see how highly the virtual reality simulator scored across the domains assessed, even though most of these scores were comparatively lower than the high-fidelity simulator.

The students also perceived SimMan 3G to be better in helping them to develop their team working and clinical reasoning skills, both of which are important for dealing with emergencies in clinical practice.²³ This aligns with existing research, which has highlighted the benefits of high-fidelity simulation in developing non-technical skills, such as teamwork.²⁴ Students commented that they found the ‘hands-on’ experience contributed positively to their learning experience, with some students specifically highlighting the benefits of using prescription charts in the scenario for drug prescribing practice. This could only be simulated in the virtual reality environment by using dropdown lists to choose individual drugs or intravenous fluids. These findings are not surprising considering the earlier comments about the two simulation environments. With regards to the negative aspects of SimMan 3G, student feedback highlighted common themes previously recognised in the literature, predominantly a perceived lack of immersivity within the scenario and the inability to recreate aspects that may be important to the clinical picture, such as a rash and skin tone.^{4 5 7} Ultimately, despite the SimMan 3G high-fidelity patient simulator having many features which are reflective of a real patient, such as palpable pulses and audible breath sounds, the fact that it remains a heavy, relatively static and cold plastic manikin is recognised by students as a significant limitation of this technology.⁶

The students reported the OMS virtual reality technology to be generally more enjoyable and immersive than the SimMan 3G. The immersive setting that is created within the OMS virtual reality technology system is linked to the fact that, due to the nature of the technology, elements of acute care scenarios that are particularly challenging to recreate outside of the virtual environment may be replicated. For example, the exposure to a myriad of sounds mimicking those of a real-life emergency room.²⁵ Furthermore, and perhaps most importantly, the ‘sick’ avatar patient that is at the centre of the student’s experience of the scenario, and ultimately learning, is more realistic; students are able to interact with their patient, being able to see them talk, look ‘sick’ and also how they change, for the better or for the worse, over time and dependent on their actions. This may be an important factor in relation to suspension of belief, and therefore student engagement and sense of presence in the scenario, which could be more variable and subject to additional external factors within a manikin-based scenario.^{26 27} Students further commented on the usefulness of having guidelines and results of investigations within the OMS virtual reality system scenarios. The students felt the perceived higher degree of prompting within the virtual reality simulator to be a potentially negative factor. This occurs due to the nature of the virtual reality interface, requiring the use of dropdown menus for selection by the student, whereas in the high-fidelity simulator environment subtle prompting can come verbally from the faculty member present as the nurse. The inability to practise verbal communication and hand-over skills within the virtual environment were other notable areas highlighted by the students. This is likely to be addressed in the future with the evolution of the virtual reality environment when multiple participant scenarios are implemented in multiplayer environments, and emerging research has also suggested that team training using virtual reality was non-inferior to live simulation.²⁸ This application of virtual reality also has the potential for participants not to be physically co-located.

The students felt that SimMan 3G was more helpful in preparing them for real clinical practice, although that they had not yet reached that point. However, the ability to use the OMS virtual reality technology as a learning tool cannot be underestimated. For example, the fact that students felt there was more prompting with the OMS virtual reality system may render this technology more useful to students at an earlier stage of their training. For example, it may provide a means of introducing an immersive experience of managing emergency scenarios to other students, notably those in year 4 of our undergraduate programme. OMS virtual reality technology may provide an excellent opportunity for students to experience, and practice managing, a range of acute emergency scenarios in a more guided and supportive environment. For earlier year clinical students who are developing diagnostic and clinical reasoning skills, but are less confident about patient management, the OMS virtual reality simulation scenarios, with prompting from guidelines and dropdown prescribing options, is likely to be very helpful. In this example, rather than using virtual reality as an educational tool for learning focused on higher cognitive processes such as analysing or evaluating, it could be used for acquisition of knowledge or more less complex cognitive processes, such as remembering and understanding, as described within the revision of Bloom’s taxonomy.²⁹

One important limitation of our study is that our students were already familiar with simulation using the SimMan 3G and this may have impacted on the students’ perceptions of both technologies. For example, familiarity with SimMan 3G could easily have influenced question responses, for example, about ease of use or usefulness, when compared with the less familiar virtual reality environment. Most of the students were naïve to the OMS virtual reality technology and despite orientation using the introductory video from OMS about the virtual reality environment, there may have been an impact on responses related to differential familiarity. We directly supervised the virtual reality simulation in an identical way to our procedures for performing SimMan 3G simulation, including debriefing. We did not use the inbuilt debriefing tool within the OMS technology, which can be used by students without direct supervision to practice the scenarios and increase their scores, as assessed by the internal algorithm of the virtual reality software. Enhanced familiarity with the virtual reality environment may well increase the experiential benefit of this form of technology and benefit from the computer derived performance metrics permitting students to track their progress. An additional limitation of our study was that the outcome data reflected only student perceptions of the simulation technologies, reflecting Kirkpatricks’ level one, as opposed to data that demonstrate achievement of learning outcomes or Kirkpatricks’ level two data.¹⁶ While student perceptions of learning are an important component of the educational experience, it is recognised that a more robust evaluation of how the technologies help to achieve or assess specific competencies would provide more useful information for educators to help inform decisions about how best to use different forms of simulation.

As an initial exploratory study, we recognise that the results of this study are not definitive, and that further work may be required in this area across a number of disciplines in order to provide more robust data. Further evaluation of the simulation technologies and the experiences of final year medical students that are non-naïve to both technologies is currently in progress by our research team. The next steps of our research aim to not only address this limitation of this study, but also to further explore how the two technologies can address different aspects of the medical curriculum, such as prescribing and interpreting clinical investigations.

From a practical perspective, the OMS virtual reality kit can also be easily transported between clinical teaching sites. This is of particular value given that teaching at the Hull York Medical School and most other UK medical schools, occurs over a range of hospital sites. Similarly, the OMS technology software has been designed to allow students to log into the scenarios from their personal computers. Although this facility is not currently being used by our students, it is important to recognise that this technology can potentially be utilised in this way. With an increasing focus on self-directed and technology-enhanced learning, the value of providing students with an opportunity to practise the OMS technology emergency scenarios many times, and at their leisure, cannot be underestimated.

Conclusions

In our study, final year students felt that emergency scenarios experienced using the SimMan 3G within a simulated clinical environment were of greater value in preparing them for clinical practice than those created within OMS virtual reality system. However, the OMS virtual reality technology offers an enjoyable and easily portable alternative. Learning outcomes, such as team working and communication skills, are better triggered with SimMan 3G due to the nature of the technology, however, there is a trend towards greater immersivity within the OMS virtual reality technology environment. This study has helped to further understand student perceptions of these two simulation technologies, and has informed plans for further research to address the limitations of this study further and to further consider the relevance of these simulation technologies to different competencies and learning outcomes.

Data availability statement

Data are available upon reasonable request. A summary of our research results is provided in the article. Additional data that are currently held by the authors of the article include the full breakdown of all results, qualitative comments and the research protocol.

Ethics approval

The study was approved by the Hull York Medical School Ethics Committee (protocol number 18 45, approved 12 March 2019).