Virtual, Augmented, and Alternate Reality in Medical Education: Socially Distanced but Fully Immersed

Abstract

Background

Advancements in technology continue to transform the landscape of medical education. The need for technology-enhanced distance learning has been further accelerated by the coronavirus disease (COVID-19) pandemic. The relatively recent emergence of virtual reality (VR), augmented reality (AR), and alternate reality has expanded the possible applications of simulation-based education (SBE) outside of the traditional simulation laboratory, making SBE accessible asynchronously and in geographically diverse locations.

Objective

In this review, we will explore the evidence base for use of emerging technologies in SBE as well as the strengths and limitations of each modality in a variety of settings.

Methods

PubMed was searched for peer-reviewed articles published between 1995 and 2021 that focused on VR in medical education. The search terms included medical education, VR, simulation, AR, and alternate reality. We also searched reference lists from selected articles to identify additional relevant studies.

Results

VR simulations have been used successfully in resuscitation, communication, and bronchoscopy training. In contrast, AR has demonstrated utility in teaching anatomical correlates with the use of diagnostic imaging, such as point-of-care ultrasound. Alternate reality has been used as a tool for developing clinical reasoning skills, longitudinal patient panel management, and crisis resource management via multiplayer platforms.

Conclusion

Although each of these modalities has a variety of educational applications in health profession education, there are benefits and limitations to each that are important to recognize prior to the design and implementation of educational content, including differences in equipment requirements, cost, and scalability.

Advancements in technology and the arrival of the next generation of learners are transforming the landscape of medical education. In particular, given the increased focus on patient safety and tightened supervision standards, simulation has emerged as an essential element of clinical education. Widespread forced adoption of distance learning in the setting of the coronavirus disease (COVID-19) pandemic accelerated a shift that was already in progress—expanding simulation experiences out of the traditional “simulation laboratory” (e.g., a central location with high-fidelity patient simulators, procedural task trainers, and live standardized patients encounters) and into the virtual realm.

Emerging technologies, including virtual reality (VR), augmented reality (AR), and alternate reality present unique approaches to teaching clinical skills to all levels of learners using computer-generated simulation. Computer-generated simulation uses a human–computer interface to create environments for participants to experience and interact with and varies in terms of immersion, fidelity, and interactivity (1). In VR, immersive, highly visual inputs create realistic digital representations of the real world. The user interacts with the virtual world via head-mounted displays (i.e., VR headsets), motion sensors, controllers, keyboards, and voice recognition software (2). In contrast, AR superimposes computer-generated stimuli on real-world environments or objects, such as computer-generated anatomical structures superimposed on a manikin (2). Alternate reality platforms create a parallel world in which users can interact with and influence a narrative through their decisions (3). In alternate reality, participants interact with virtual worlds using real-world tools, such as interacting with patient data in electronic health record simulations. The degree of immersion into the virtual world is what largely differentiates VR from AR and alternate reality, although the forms exist on an overlapping continuum sometimes known as “mixed reality.” A graphic representation of the various modalities is provided in Figure 1.

Figure 1.

Graphic representation of virtual reality, augmented reality, and alternate reality, including optimal educational applications.

In VR, variable auditory, visual, and sensory inputs create partially or fully immersive interactive experiences for learners. The degree of immersion can range from screen-based (previously desktop-based, now available on various devices) to head-mounted displays or immersive VR rooms (4). Screen-based VR can have a range of appearances, including two-dimensional (standard) video, 360-degree video, or completely computer-generated three-dimensional (3D) environments. In the most fully immersive VR environments, 3D images projected via headset or multiple projectors and motion controllers to track motion and provide haptic (sensory or tactile) feedback allow users to interact with the virtual world while simultaneously blocking input from the real world (5). Virtual worlds typically incorporate avatars, which are 3D graphic representations of the user in the virtual environment, whose movements and actions are controlled by the user (5). For example, a user could be asked to interact with and manage a simulated computer-generated patient with a particular diagnosis. Through their avatar, the learner could perform a history and physical exam, order and review diagnostic tests, and administer treatment in the virtual clinical environment.

In contrast, AR refers to technology that integrates computer-generated content into real-world settings (6). Commonly, this is done using glasses or projectors to overlay images onto what the eye is seeing naturally, but it could also include theoretical expansion to smell, touch, or sound. Perhaps the simplest application might be using AR glasses to add animated content or videos to a classic textbook (7), whereas some of the more advanced current applications include 3D renderings of anatomy and haptics while using a laparoscopic simulator (8). This can be used to either increase fidelity of a simulated environment or to allow educational content to be simultaneously experienced in a manner not possible in the real world.

Alternate reality is an emerging modality that uses computer-generated parallel worlds as a learning and practice environment. Using its broadest definition, many of the fictive encounters (e.g., case-based learning and standardized patients) currently employed in medical training qualify as low-technology alternate reality (9). Virtual patient encounters in VR simulation could also be interpreted as alternate reality; however, alternate reality experiences typically have deeper, more-networked fictional worlds and often use gamification (10). The term most commonly refers to alternate reality games (ARGs) and gamified, or game-informed, training platforms. An ARG or game-informed training platform includes several essential elements: an underlying narrative (story, events, and rules) that drives the game, interaction or dialogue between the users and the platform, and a challenge or set of challenges (11). Although alternate reality experiences do not necessarily require computers, applications in healthcare education tend to be computer based to maintain fidelity to real-world activities. Examples in healthcare education include electronic health record–based simulations and virtual hospital and online patient generator games (12).

Mixed reality technology has been in development since the 1960s and was first explored for applications in medicine in the 1990s (1). Following its reemergence in the entertainment industry and commercial availability in 2016, there has been renewed interest in mixed reality and its applications in medical education (1, 13). Although many educators believe “the future of medical education is no longer blood and guts, its bits and bytes” (14), there are inherent limitations to what these novel technologies can accomplish. Certain educational tasks will be better suited to a VR environment, an AR platform, or an ARG. The optimal choice for technology will be based on specific learning objectives and end-user needs. The strengths and weaknesses of each modality can be broken down into upfront investment, whether the degree of fidelity achieves the necessary level for learner immersion or assessment, and whether the technology lends itself to customization for specific local learning objectives. Examples of this comparison and optimal applications of each technology can be found in Figure 2.

Figure 2.

Comparison of virtual, augmented, and alternate reality modalities regarding cost, customizability, and fidelity. EHR = electronic health record; POCUS = point-of-care ultrasound.

In this review, we will explore the potential applications, evidence for the use, and strengths and weaknesses of VR, AR, and alternate reality in healthcare profession education.

VIRTUAL REALITY

In contrast to traditional manikin-based simulation, VR simulations can take place asynchronously whenever and wherever is most convenient for the learner (15). The virtual world setting (e.g., bronchoscopy suite, hospital room, or outpatient clinic) can be easily varied to meet the educational objectives (5), allowing for more streamlined integration into everyday education (15). Although initial costs for content development and equipment may be high, VR’s scalability and flexibility can allow for overall long-term cost savings and favorable cost-utility compared with traditional simulation modalities (15, 16).

Although VR-based education is relatively new, the evidence base for its applications across various educational domains is ever growing. VR has provided highly rated learning experiences and has been preferred in certain settings to standard didactics because of its increased realism and ability to provide data-rich analyses by tracking every user’s input and interactions (4). Compared with standard screen-based learning, fully immersive virtual environments have demonstrated significantly higher knowledge retention (15, 17). VR also has utility in portraying complex clinical scenarios in problem-based learning formats, allowing for flexibility in training across geographically diverse locations (18, 19).

VR-based education has also been shown to be a feasible and reliable means for technical and nontechnical skills assessment (20). To date, the efficacy of VR in procedural training has been demonstrated and more broadly adopted in surgical specialties than in medicine and nursing; however, there is increasing evidence of VR’s utility in teaching a broad range of nonsurgical procedures (15, 21). Given improvements in procedural skills as well as patient comfort observed in a series of randomized trials, VR-based endoscopy training is now becoming the standard for beginning endoscopic training in gastroenterology (22). In addition, a small pilot study among cardiology trainees demonstrated improved performance in virtual transvenous pacer lead extraction compared with conventional training alone (23).

Among pulmonary and critical care trainees, VR-based bronchoscopy training improves both skill acquisition and time needed to complete the procedure (24). It has also demonstrated improved bronchoscopy training efficiency compared with the traditional apprenticeship model (25). VR-based bronchoscopy education affords the learner tactile feedback and realistic 3D visualization, including that of the bronchoscope in the airway (26). As expected, the most significant improvements in skill acquisition have been seen with novice bronchoscopists (27). VR’s efficacy has also been shown in the performance of bronchoscopic-guided intubations, including improved time to intubation (28).

A variety of studies have demonstrated VR’s utility in cardiopulmonary resuscitation (CPR) and advanced cardiac life support training using techniques including avatar-based gamification and interprofessional team training (5, 29–32). Virtual interprofessional education (IPE) sessions completed with computer-generated virtual teammates have been shown to be effective and scalable solutions to imparting foundational IPE to health profession students (33). Such simulations have also been shown to be an effective means for stress inoculation among resuscitation team leaders, improving situational performance and reducing anxiety (34). A pilot study also demonstrated improved situational awareness during virtual CPR training (35); however, VR-based CPR training may be inferior in conferring high-quality CPR skills compared with traditional face-to-face training (36).

In addition to IPE, VR can also be used for doctor–patient communication training through the use of virtual standardized patients (VSPs). VSPs are avatar-based representations of human standardized patients with which the learner can interact, often using natural language (2). VSPs have been effectively utilized in continuing medical education for practicing physicians, nurses, and other allied health professionals, as they can be effective tools in teaching clinical skills and improving clinical decision-making (even in the absence of faculty facilitation) (37). The use of VSPs has resulted in significantly greater post-training improvement in motivational interviewing compared with academic study alone with successful transference of skills to human standardized patients interactions (38). The ability for trainees to participate in high-yield self-directed learning modules may be important in reducing the overall demands on faculty time that high-quality interactive learning demands. Enhanced communication training in the virtual world has been further demonstrated in breaking bad news (39). More recently, fully immersive VR simulations have been used to build empathy among faculty and health system leadership and address the issues of systemic racism and inequity faced by many patients (40).

Despite VR’s benefits, there are limitations to the technology, including reduced haptic fidelity compared with manikin-based simulation (36). Although users can perform critical actions using hand-held controllers, the tactile sensation associated with certain physical interventions, such as placing a peripheral intravenous catheter or providing chest compressions, has not yet achieved adequate fidelity in the virtual world. In addition, the fully immersive nature of VR can provoke “cybersickness,” which has symptoms similar to motion sickness and is more commonly seen in women (41). Symptoms can include nausea, disorientation, postural instability, and headaches. The issues associated with this condition are complex, as the onset can be impacted by the design of the virtual environment, individual responses to stimuli, tasks performed, and the type of technology used (41). Technical issues with various VR platforms are also reported and can be viewed as a barrier to uptake (19).

AUGMENTED REALITY

The technology required to make AR effective is still relatively new (42). With that, however, it has rapidly been implemented in areas relevant to critical care, particularly procedural training and diagnostic imaging. By beginning with a genuine environment, such as a clinical setting, AR enables immersion at high fidelity, without requiring the creation of the virtual scene from the ground up. The focus can then be to have the synthetic content integrate almost seamlessly into the educational experience. This can be used to increase the fidelity of a simulator, such as by having the otherwise blank ultrasound screen integrate the real-world movement of the ultrasound transducer by the trainee on a manikin or actor (43).

Similarly, adding a visual layer to a simulation increases the fidelity of procedural training for venous access or surgical techniques (44, 45). Multiple studies have used AR-assisted surgical techniques to enhance surgical planning and navigation (46–48). This allows the benefits of existing simulation technology (physical touch and investments in setting) to be incorporated into advanced immersion technology. This additional educational material can change the learning experience, allowing a trainee to “see” the radiographic and anatomic relationships overlaid on their own reflection in a mirror as they move (49). More intriguing, however, AR can change the experience of learning a procedure to include prompts at the time of use, such as by having warning images, prompts for the next steps, or a video of optimal techniques overlaying the real world (50). In general, the limitations to AR in this sphere include the significant upfront costs of purchasing high-fidelity devices and generating the simulated content that will be applied. Private companies are developing packages for common applications such as point-of-care ultrasound, but this is not currently a technology easily customized for the specific learning or assessment metric (49).

Procedural training no longer can rely only on the availability of highly supervised clinical opportunities, and AR has a broad array of applications wherein it is superior to VR or simulation alone. The ability to overlay anatomy otherwise hidden, especially when combined with a haptic feedback system, generates confidence in procedures more quickly than traditional simulation alone, as was demonstrated in needle guidance for spinal procedures (51). Although not yet rigorously tested, the ability to turn a checklist into a nonobtrusive nudge allows for both improved education and assessment while the procedure is performed, specifically for central venous catheterization (50). The future state, however, is perhaps the most exciting, wherein AR is used not just in the educational arena but also as a part of every procedure, allowing for the integration of diagnostic imaging and navigation systems into an intuitive interface. The Food and Drug Administration has approved the use of AR in procedural guidance for spinal surgery, with the potential for more generalized procedural assistance (52).

For diagnostic imaging, especially point-of-care ultrasound, the learner often must choose between the fidelity of scanning a healthy person and the experience of visualizing pathology. With AR, however, the experience of transducer manipulation can be combined directly with pathological images integrated into a clinical scenario without losing fidelity. Integration of AR into the educational process of learning echocardiography, for example, has been shown to effectively allow novices to acquire high-quality images (53) and to assist nurses in both acquisition and image capture at a level comparable with echosonographers (54). As devices continue to improve, AR applications of teleguidance can allow distance supervision and assessment without the same degree of resource allocation required currently (55, 56).

Alternate Reality

Alternate reality can be created at significantly lower cost points than AR or VR and does not necessarily require hardware or equipment investments. It creates a unique experience in which participants use real-world tools and interfaces to discover and interact with a parallel world, thus creating a fun and low-stress environment for problem-solving and learning (3). Game building (creating the elements, structure, and logic for the game) can require a wide range of expertise and resources, depending on the complexity of the parallel world and/or game (11). One proposed advantage of ARGs is that the use of existing platforms (e.g., electronic health record playground, email, online biomedical library, etc.) can reduce the overall resources and technical skill required to build the world (9). Alternate reality platforms can either run automatically (requiring more complex software) or can be monitored and managed by a game runner who adapts the narrative based on input from the users. Although alternate reality experiences can be individual, they are often employed for group collaboration and/or use multiple players or teams to create a friendly competition environment. Alternate reality platforms employ game features to build and practice communication skills, data gathering and interpretation, and problem-solving. A limitation of alternate reality is that learners typically interact with data rather than patient representations; however, VR experiences can be embedded into alternate reality environments to meet this need.

Applications of alternate reality–based experiences to medical education have been growing in recent years; however, rigorous research and evaluation of efficacy remains in its infancy (57). There is, however, a growing body of literature supporting the use of gamification and game-informed platforms, often referred to as serious games, in medical education (12). Gamification has been successfully used to increase learner engagement (58), improve performance (59), and reduce the overall cost per hour of simulator use (60). The use of game design elements such as leaderboards, badges, and points or more complex elements such as evoked emotions, narratives, and competition has improved attention and engagement (61). It has been well accepted among millennial learners and has been successfully used for team training in multiplayer formats (62–65).

One proposed advantage of alternate reality is the potential for deep user engagement and subsequent enhancement in learning outcomes, facilitated by the challenge and fun engendered by gamification. Alternate reality simulations may be best suited for activities that are more cognitive in nature, such as practicing clinical reasoning skills (66), longitudinal management of chronic conditions (67), and/or patient panel management (68). A study of surgical trainees performing minimally invasive surgery showed improved problem-solving skills and ability to troubleshoot equipment malfunction as compared with a standard curriculum (65). Multiplayer games have been used to simulate organizational challenges in disaster preparedness, including a virtual simulation of a mass-casualty situation for emergency physicians and nurses, which was shown to improve confidence in managing incidents and perceived as improving clinical skill (29). Similar disaster training games have been shown to be a comparable alternative to drills using standardized patients for emergency medicine learners (69). Longitudinal and electronic health record–based simulation games have been employed to help students learn organizational and practical skills for outpatient practice, including at the medical student level (72). Isolating the cognitive and organizational aspects of the skill can create a more focused deliberate practice experience.

Conclusions

As computational capacity rapidly advances and becomes more affordable, the ability to integrate computer-generated simulation into medical education increasingly becomes the standard. Although the technology continues to evolve, these emerging approaches to simulation represent an opportunity for innovation in health profession education that is reproducible, scalable, and exportable. Although there is an initially steep learning curve for users and content creators, potential physical limitations such as cybersickness, and time investment in creating virtual and mixed reality cases, there is also potential opportunity for innovation. The specific strategy of integration into a curriculum, either in the creation of fully immersive VR environments, AR sessions with computer-generated components, or the creation of alternative reality games, should be tailored to specific educational objectives and learner needs. As learning becomes increasingly socially distanced and on the schedule of the learner, traditional simulation centers will need to adapt mixed reality modalities into the overall learning toolset. The identification of available resources, acceptable costs, and previously developed components is necessary for the educator to choose the best methodology to meet their specific goals.