Emerging Applications of Virtual Reality in Cardiovascular Medicine

Summary

Recently, rapid development in the mobile computing arena has allowed extended reality technologies to achieve performance levels that remove longstanding barriers to medical adoption. Importantly, head-mounted displays have become untethered and are light enough to be worn for extended periods of time, see-through displays allow the user to remain in his or her environment while interacting with digital content, and processing power has allowed displays to keep up with human perception to prevent motion sickness. Across cardiology, many groups are taking advantage of these advances for education, pre-procedural planning, intraprocedural visualization, and patient rehabilitation. Here, we detail these applications and the advances that have made them possible.

Central Illustration

For many years, extended reality technologies have promised physicians the ability to move beyond 2-dimensional (2D) screens, allowing them to understand organ anatomy in 3-dimensions (3D) noninvasively. However, this promise has been stymied by bulky equipment that was incapable of displaying high-quality virtual images coherently enough to prevent user motion sickness. Recent advances in high-resolution display technology, exponential increases in computational power, and miniaturization of components led by mobile device manufacturers have enabled a new class of head mounted display (HMD) devices (1). These low-cost, comfortably-worn devices can display high-quality clinical data at response times that are fast enough to be used for extended periods of time, overcoming longstanding barriers to adoption in the medical community.

Advances in digital light projection, organic light emitting diode, and optics manufacturing have resulted in thinner, lower-power, and brighter display systems (2). Speech recognition and generation advancements brought the earliest forms of augmented aural reality; the online digital assistant now known as Siri (Apple, Cupertino, California) or Google’s assistant (Google, Mountain View, California) are in use daily, along with automated transcription systems. Sensor technology advancements in positioning and navigation systems originally designed to function with the global positioning system have been extended to include satellite-free indoor navigation, tracking user position via their mobile device by leveraging software and hardware such as Project Tango and ARCore (Google) or ARKit (Apple). Eye and hand tracking provides new human machine input capabilities for understanding natural intent with less burden on the user to understand the language of a specific manufacturer. This combination of hardware and software innovation has enabled new classes of 3D platforms.

Based on these advances in 3D platforms, the number of clinical applications has grown exponentially in the areas of education, pre-procedural planning, rehabilitation, and even intraprocedural visualization. Here, we focus on the application of virtual reality (VR) and related technology for clinical cardiac practice, focusing on what is possible based on current technology and what barriers still exist for widespread adoption.

Defining Reality

Extended reality describes the spectrum, or “virtuality continuum” (3) from fully immersive, curated digital experiences in VR, to unobtrusive annotations within easy access of the operator in augmented reality (AR) (Table 1). It encompasses 2D annotations on real-time video, 3D models, and true interference-based holograms, like animated versions of those seen on baseball cards. Although most headsets refer to their models as “holograms,” HMDs typically create the perception of depth for 3D models through stereoscopy, simulating depth without generating true holograms.

Table 1.

The Extended Reality Spectrum

Virtual Reality	Merged Reality	Mixed Reality	Augmented Reality
Interactive virtual objects	Interactive virtual objects	Interactive virtual objects	Virtual objects
Virtual background	True background	True background	True background
Immersive display	Immersive display	See-through display	See-through display
Fully Immersive Experience		Unobstructed Experience

VR provides complete control over the wearer’s visual and auditory experience as they interact within a completely synthetic environment. This control over the environment can provide virtual experiences of either subdued or amplified versions of reality. Commercially available VR platforms from Oculus, HTC, and Sony, among others, use high-resolution displays to fully replace the wearer’s visual field. These immersive displays have been applied to pain management (4), exposure therapy (5), stroke rehabilitation (6), education, and surgical planning (Surgical Theater).

Conversely, AR allows the wearer to see their native environment while placing 2D or 3D images within it through a “window-on-the-world” (3). This annotated window-on-the-world can be displayed on an unobtrusive HMD or on a mobile device, using the onboard camera to provide a live view of the environment. Perhaps the most successful consumer application of AR technology has been Pokémon Go (7) in which a mobile phone camera feed was annotated with avatars and contextual game data. These AR applications minimally interfere with the normal field of vision, providing useful information only when called upon by the user. In the medical setting, contextually relevant graphics, reference data, or vital information is presented alongside (rather than in place of) the physical surroundings. The first, and most widely publicized commercial platform, Google Glass for example, was shown to display patient vital signs, relevant history, and prescription information from a patient’s electronic health record during a visit (8). More recently, other platforms have been developed for education, patient point of care (Evena [9]), emergency response, and telemedicine (AMA Xperteye [10]).

VR and AR denote the 2 bookends of the continuum of experiences, and as the industry has grown, 2 new classes of experiences have emerged: merged reality (MeR) and mixed reality (MxR). Both approaches achieve a similar experience: to allow for interaction with digital objects while preserving a sense of presence within the true physical environment. MeR captures a user’s surroundings and re-projects them onto on a VR-class HMD, which can mediate the environment up or down as desired. This allows for a more seamless transition between mediated and unmediated virtuality and reality. For consumers, this is portrayed as the ability to transport users to a completely different room and back to their living room with the same device (Intel Alloy [11]), which could also be applied to patients in hospital rooms. MxR accomplishes a similar experience by projecting digital objects onto a semitransparent display. As such, the MxR platforms do not obscure, or mediate, the physical environment, allowing the wearer to maintain situational awareness of their surroundings as well as maintain normal interactions with those not participating in the MxR experience. This advance has opened a window of opportunity for this type of technology for intraprocedural use, allowing physicians to remain in their environment while viewing the virtual image (Central Illustration). Currently, MxR displays commercialized by Microsoft and DAQRI (Los Angeles, California) have been demonstrated in medical education and medical imaging (12).

Central Illustration.

Mixed Reality Cardiac Electrophysiology Workflow

Mixed reality allows for the display and interaction with existing displays within the cardiac catheterization suite, including integration with fluoroscopy (top left), electroanatomic mapping systems (top center), electrocardiograms (top right), as well as previously acquired and computed tomography– or magnetic resonance–derived 3-dimensional (3D) anatomic models (middle row). Although augmented reality platforms (bottom left) can show 2-dimensional (2D) data unobtrusively, mixed reality platforms (bottom center) allow for hands-free 2D and 3D visualization as well as direct sterile control of these data without otherwise obstructing the normal visual field, as in virtual reality (bottom right).

Several additional prominent 3D display platforms have been developed without using an HMD, including 3D flat-panel displays, and interference-based computer-generated holography. Flat-panel 3D displays, first introduced in 2010 (13), were primarily displays only and lacked input devices for manipulating 3D data. Hewlett Packard’s (Palo Alto, California) Zvr 3D display and input device is used in conjunction with EchoPixel’s (Mountain View, California) software to provide diagnostic quality images (14). A second example, computer interference-based holography, generates realistic 3D images by shaping light waves using a combination of complex processing, specialized computer-controlled light sources, and optics. Although real-time display of holograms was first demonstrated in 1992 (15), recent advances by RealView imaging have enabled their practical use, and they have explored several clinical applications using this technology, including cardiology 16, 17.

Cardiac Applications of Virtual Reality

Various cardiac applications of virtual reality are depicted in Figure 1.

Figure 1.

Emerging Applications of Extended Reality Applications

The x-axis is a condensed representation of the virtuality continuum, with virtual reality (VR) on the left, mixed reality (MxR) and merged reality (MeR) in the center, and augmented reality (AR) on the right. The y-axis categorizes the applicability of each solution. Intraprocedural applications are the most sensitive platform type, and currently are only being developed for MxR.

Education

Extended reality provides a wide range of possibilities for educational and training applications. Some applications leverage the immersion that VR enables to simulate the entire operating environment along with the educational material. Another class of applications brings the existing medical simulations for tablets and mobile phones to VR as the next platform that trainees will have access to. These VR-use cases are generally available across most consumer VR platforms. Other applications take advantage of the presence of MxR to allow multiple wearers to interact and discuss with each other while viewing the same educational material in a natural environment. These applications rely on the view through nature of MxR, combined with the freedom of untethered headsets to walk around and communicate naturally.

Stanford virtual heart

The Stanford Virtual Heart Project (18), working with Lighthaus, Inc., uses an immersive VR headset for educational purposes. This project has a few distinct arms. The first is geared at patient and family education to help families better understand their child’s cardiac anatomy, which is currently limited to drawings and plastic models. This improved depth of understanding should help parents better participate in their child’s complex medical care. This application has been expanded to Stanford medical students and trainees, who can visualize normal and abnormal anatomies and understand how congenital anomalies affect physiology. Using a fully immersive VR headset, the students can inspect, manipulate, and walk through the models, providing a more complete understanding of the anatomy and physiology. A library of approximately 2 dozen common congenital lesions is available to the trainees. The aim of these experiences is to provide a deeper anatomic understanding of these lesions, improving the understanding and speed of learning of these complex abnormal physiologies and hemodynamic sequelae.

The final application is the use of a 3D monitor, Echopixel (as discussed later in the section “Pre-Procedural Planning”), in the cardiothoracic operating room. A 3D workstation in the surgical suite may allow for accurate assessments of intracardiac anatomy and geometry, which may be difficult to see after patients are placed onto cardiopulmonary bypass and the heart is decompressed.

HoloAnatomy

At Case Western Reserve University, investigators are using the HoloLens (Microsoft) to change medical student education, particularly anatomy (19). The ability to better understand 3D anatomic relationships not only eases the learning curve, but also encourages students to “think like a doctor.” In conjunction with the Cleveland Clinic, the team at Case Western Reserve University is developing a curriculum, HoloAnatomy, that will allow medical students to perform holographic dissections to better visualize and understand the body’s organs and systems. Preview versions of this software are freely available to download.

Pre-procedural planning

EchoPixel

One of the first 3D displays to gain approval from the U.S. Food and Drug Administration is the True 3D system that has been developed by Echopixel, which is integrated into a diagnostic grade DICOM workstation. The system provides 3D visualization using a technique similar to that used in 3D movie theaters and early 3D consumer televisions: by providing different images to each eye using specialized glasses. A single Echopixel user wearing polarized glasses can additionally manipulate the onscreen image using a handheld wand. Initial cardiology studies include using the Echopixel system to visualize arteries in patients with pulmonary atresia with major aortopulmonary collateral arteries. In this study (20), cardiologists evaluated patients who had undergone computed tomography angiography either by using the 3D display or a traditional readout. Cardiologists using the True 3D display had interpretation times of 13 min compared with 22 min for those that used a traditional display. Both groups were similarly accurate in their interpretations when compared to catheter angiography.

The True 3D display is also being used as part of the Stanford Virtual Heart Project (see the previous section, “Education”).

Intraprocedural visualization

Enhanced electrophysiology visualization and interaction system (Project ĒLVIS)

Currently, visualization in the electrophysiology laboratory relies on a combination of fluoroscopy, electroanatomic mapping systems (EAMS), and echocardiography (intracardiac echocardiography and transesophageal echocardiography), with most laboratories using EAMS plus other tools. Although improvements in visualization have been a source of research and development over the years, there have not been equal gains in improvements in interaction. Our prototype, the Enhanced ELectrophysiology and Interaction System (ĒLVIS), not only empowers the interventional electrophysiologist to visualize patient-specific 3D cardiac geometry with real-time catheter locations, but also allows direct control of the display without breaking sterility, which is a key advance (21).

ĒLVIS can display data that is exported from an EAMS or that is obtained pre-procedurally via computed tomography or cardiac magnetic resonance imaging. To date, we have connected ĒLVIS to the EnSite Velocity EAMS (Abbott, Abbott Park, Illinois) via the CoHesion module to display electroanatomic data, including cardiac geometry, catheter localization data, and electroanatomic maps, including local activation time and voltage maps. Most recently, we have demonstrated the ability to display historical cases for review, as well as a live case observed in real-time from the control room. In addition to the ability to improve visualization, the system allows the user to utilize gesture, gaze, or voice control for sterile control of the display. This improved interaction allows the interventionalist to directly control this single cohesive model in manner that is optimal for a given procedure.

Sharing functionality provides a single shared cardiac holographic model for as many as 5 users with the model remaining fixed in the room, allowing all users to visualize the model from their vantage points. Using the sharing system, there is a single person in control of the system at any given time with the ability to pass controller privileges to other users. Supplemental Video 1 demonstrates the current functionality of the prototype, including sharing, gesture control, and the display of intracardiac geometry and catheter movement that were obtained from the EnSite system.

Realview

In 2016, the pediatric cardiology group at Schneider’s Children’s Medical Center partnered with Realview Medical Imaging (Yokneam, Israel) to assess the feasibility of creating real-time 3D digital holograms in a standard cardiac catheterization laboratory (16). The Realview computer-generated holography (CGH) were created using 3D rotational angiography coupled with 3D transesophageal echocardiography. A total of 8 patients were enrolled in this study, including patients with structural heart disease and post-operative cardiac patients. In all patients, the team generated real-time 3D holograms with high accuracy (as measured by instructing 4 independent observers to identify anatomic landmarks within the hologram and typical cardiac imaging) with “very easy” interactions including image marking, cropping, zoom, rotation, movement of hologram, and slicing. This is the first study of its kind to demonstrate feasibility within the cardiac catheterization laboratory.

Rehabilitation

MindMaze

MindMaze (San Francisco, California) is creating both hardware and software in the VR space with a current medical application in neurorehabilitation. Their current solution, MindMotion PRO, is cleared by the U.S. Food and Drug Administration for use in post-stroke patients, combining virtual reality, brain imaging, and gaming technologies to retrain the brain to improve upper limb mobility. In acute post-stroke patients, a clinical study has enrolled patients who engage in 20- to 30-min sessions as soon as 4 days post-hospitalization without continuous supervision, with increasing training intensity over time (22). All patients had a positive user experience using the MindMotion PRO, with 90% reporting an improvement in movement capacity (22).

Extended Reality Hardware

Significant advances in extended reality devices have enabled the previously mentioned applications, yet the technology continues to evolve rapidly. In the following text, we describe currently available systems and future devices that will certainly expand the number of opportunities for extended realities to improve patient care.

Displays

Typical displays project either a single coherent image to 1 or both eyes as a near-eye display, or in a stereoscopic pair for 3D simulation. Monocular systems are either opaque or view through, and avoid disruption of normal vision by positioning a small display outside of the normal working visual field. This allows a user to access the information contained on the display “at a glance.” Equivalent capabilities can be achieved using biocular displays, or the displays can provide 2 different images to provide stereoscopic image pairs. Most HMD platforms generate image pairs by using a panel display or small projector and a series of optics to enlarge the image and simulate projection at a distance to ease focusing, as illustrated for several common systems in Figure 2. For a given projected distance and resolution, as the apparent display and field of view (FOV) is increased, apparent pixel size is also increased.

Figure 2.

Simplified Image Formation Diagrams for the HVS and Common Extended Reality Displays

Sample extended reality optics configurations for extended reality displays for visualizing a gray sphere at the top of each diagram, as perceived by an observer eye at the bottom of each diagram. (A) The normal human visual system (HVS) observes the sphere directly, and can accommodate variable focal distances. (B) Virtual/merged reality systems magnify an opaque, distortion-compensated display through a plano-convex type (shown) or Fresnel equivalent lens (not shown). (C) In reflective half mirror (“birdbath”) view through systems, light from the display reflects off a semitransparent splitter and focus/combiner to the eye. (D) In computer-generated holography (CGH), an incoming light wave front (right) is shaped through the spatial light modulator (SLM) (center) to match the wave front of the sphere from the correct focal distance. (E) In polarizing reflector waveguides, the input display is coupled into the waveguide with a polarizing mirror and reflected internally. Polarized reflectors selectively decouple light from waveguide to the eye. (F) In an array of half mirrors, light from the display is internally reflected through the waveguide, and is reflected to the eye through an array of small, semi-reflective mirrors (shown as a bold line) rather than an equivalent large mirror (not shown). (G) In diffraction waveguides, light is coupled from the display through diffraction structures such as surface grating diffraction (SGD) shown or holographic optical element (HOE), not shown. Light is then reflected internally through the waveguide and decoupled out through a corresponding output diffraction structure. Note that only the normal HVS (A) and CGH (D) support multiple focal planes as shown.

3D display

Most 3D displays use either active or passive shutters to simulate stereoscopic 3D without a near-eye display. These displays simulate depth similarly to head-mounted displays, by projecting a displaced image to each eye, controlled by either shutters or polarization in the glasses to provide the perception of depth. These displays can be used for navigating and interacting with medical images in a conventional office setting using a stylus for interaction (EchoPixel, Hewlett Packard’s Zvr). Current displays are readily available and can be integrated into hospital workflows; however, disadvantages include the required use of glasses, the capacity to support only a single user at a time, and a limited depth volume for tracking and display.

Light field displays

Light field displays use the projection of light directly onto the retina. The primary advantage of these techniques is that they encode both the position of light and the angle, providing a more realistic image by recreating depth. True light field displays require much more computational capacity to render a point in space and generate the bundles of light necessary. The Avegant Light Field display (Belmont, California) is not yet commercially available and is not a true, full, light field display, but circumvents this complexity by accommodating a limited number of fixed focal depths. Displays of this type fundamentally trade spatial or temporal resolution in the viewing plane for resolution in the depth plane in addition to FOV of traditional stereo HMDs.

Interference-based holography

Holographic displays refer to interference-based holography, generated using a spatial light modulator (SLM) to create a hologram in space, at video framerates. The SLM shapes the incoming reference light to replicate the wave front that would originate from a real object at the appropriate position in 3D space, creating a true, multifocal, 3D hologram. This approach is being used by Realview to display 3D images of the heart (described in the previous text). The process of generating an interference pattern for a given 3D model and generating instructions for the SLM is computationally complex, but the resulting hologram best satisfies the requirements of the human visual system. In general, CGH, however, is restricted by render volume, viewing angle, and brightness.

Waveguides

There are many different variants of waveguide-based displays (23), but all fundamentally rely on reflecting the output of a display through reflection to a view-through display in front of the user’s eye. The fundamental tradeoff of these displays is that the cost and complexity of a given design increases as FOV increases. The relatively uncomplicated design of the reflective half mirror, utilized by devices like the ODG R-9 (San Francisco, California), uses a single reflective half mirror and a reflector/combiner, first developed for aircraft head-up displays. Improvements on this design rely on miniaturizing the mirror using embedded polarizing reflectors (Lumus, Rechovot, Israel), arrays of microstructure mirrors (Optinvent, Monte Sereno, California), and diffractive etched surface (HoloLens, Microsoft; Vuzix, Rochester, New York) or holographic gratings (Digilens, Sunnyvale, California; WaveOptics, Oxfordshire, England). Microstructure mirrors provide a compact, cost-effective view-through display, whereas diffractive waveguides, although currently more expensive to produce, can achieve complex optical systems in a relatively thin package. These view-through displays allow a clinician to visualize data in either 2D or stereo 3D without otherwise obstructing the normal visual field.

Virtual interaction

The breadth and depth of interactions with these different display platforms have expanded as sensor and processing capabilities have improved, ranging from marker-free tracking to neural and voice inputs. The simplest form of interaction common to most HMDs is through the movement of the head, measured by accelerometers in the display. This angular movement updates the display based on the direction the head is pointing, and a display cursor representing the center of the user’s focus is rendered in a fixed position relative to the display. This gaze cursor is usually rendered on the vertical midline of the display, although not necessarily in the center of the FOV depending on the manufacturer and ergonomics. If the gaze cursor is on an interactive element, such as a button, there are generally 2 methods of interaction with it. The first gaze-based interaction is referred to as gaze-dwell, and is triggered by holding the gaze cursor on the interactive element for an application-determined amount of time. If the HMD is equipped with internal or external hand tracking hardware, the user could also initiate activation of the interactive element using a recognized hand gesture, which in combination with gaze is referred to as a gaze-gesture command.

In addition to these gaze-based commands, some displays can recognize more complex hand gesture or controller gesture commands by tracking the hand or a controller in the hand. Controllers have the added benefit of supporting physical buttons in place to further enhance interaction. Finally, most platforms support a microphone to support varying levels of voice command automation. Although gesture interaction offers a rich, efficient means to interact with the environment, voice and gaze-dwell inputs have the advantage of being hands-free, and controllers lose the advantage of being touch free.

Sensors

Compact inertial measurement units (IMUs), optical tracking, depth sensing cameras (Intel Real Sense, Microsoft Time of Flight) and voice recognition (Google Assistant, Microsoft Cortana) provide the basis for navigation and control for most platforms. Most commercial VR displays use headset-mounted or external-fixed optical tracking (Optitrack, Corvallis, Oregon; Polaris, Medina, Minnesota; Scopis, Cambridge, Massachusetts) to provide accurate positioning and tracking within a volume, known as “outside-in” tracking (Figure 3, left). AR platforms may rely on the same optical tracking, or use onboard cameras to track fiducial markers within the environment. IMUs provide high-resolution angle and rate information for updating the view on screen, which is coupled with optical tracking information to provide correction updates to the tracking system, known as “inside-out” tracking (Figure 3, right). Network communications can provide access to highly trained voice recognition algorithms, or simpler local models can be used for specific commands. Near-eye pupil tracking (Pupil Labs, Berlin, Germany) cameras are available for simple tracking as an input device or as an enhancement to gesture input. Ultrasound arrays (Ultrahaptics, Bristol, England) can provide sterile haptic feedback through free air by using ultrasonic speakers to induce sensation on skin. This can help overcome the lack of haptic feedback when interacting with digital objects.

Figure 3.

Positional Tracking Systems

The perception of depth is influenced by the distance at which the eyes converge and the focal distance of the eye, referred to as vergence (left) and accommodation (right), respectively. 3-dimensional displays, which rely on stereo vision, can only influence vergence by adjusting the stereo disparity between the left and right eye, and the focal plane of the optics remains fixed. The eyes (bottom) perceive the vergence depth of the image of the sphere on the displays by the distance of the intersection of the displayed images. On the right, the accommodation distance is the distance to the focal plane at which the cube is in focus. Objects at closer and farther focal distances appear out of focus. Vergence and accommodation conflicts cause discomfort due to the disagreement in vergence and accommodation distances, particularly at close distances where accommodation cues are most influential.

Challenges

The applicability of different modalities of extended reality to education, pre-procedural planning, intraprocedural guidance, and therapeutic use depends on their inherent advantages and limitations, particularly with respect to isolation from the natural physical environment. VR and MeR completely occlude the normal visual field, whereas AR and MxR enhance the visual field. This is most apparent when power is removed from the display; VR and MeR are completely opaque and must be removed to be able to see, whereas AR and MxR are transparent and only the digital additions are removed. VR, for example, can allow an individual student to interact in a fully immersive simulation, isolated from outside distraction. However, AR, MeR, and MxR will allow interventionalists to maintain presence in a physical room to perform procedures as well as maintain the ability to interact with the patient and cooperate with their supporting personnel. MeR platforms, however, present a potential safety risk during a procedure in the event of power loss, which would cause complete obstruction of the normal visual field.

Extended reality platforms are constrained primarily by cost, size, weight, and power to achieve the highest visual quality, mobility, processing speed, and interactivity. Visual quality is dependent on resolution, brightness, focal depth, and FOV. Display technology is the most demanding aspect of extended reality, and is generally the largest design and cost constraint (23). For 3D systems, this is compounded by requiring stereoscopic pairs of images to generate the perception of depth through vergence, or the angularity disparity between 2 displays (Figure 4, left). The display system must compromise size and cost with providing the maximum visual quality to match the capabilities of the human visual system (HVS). The lower bound of a normal human visual acuity (roughly 20/20 or 6/6) is 1 arc min/pixel with an approximately 150º to 170º by 135º to 150º elliptical FOV 24, 25, 26. A display system that achieves this angular resolution contains pixels that are considered indistinguishable, commonly referred to by Apple as a Retina Display. This equates to a roughly 9,000 by 8,100 pixel/eye requirement to fully emulate and immerse the human visual system. By comparison, 4K HMD displays contain 3,840 × 2,160 pixels and commonly require workstation class graphics for processing. Understandably, this resolution and FOV is currently not economically feasible with current optics and display technologies. As a result, device manufacturers compromise FOV, pixel density, and display brightness to achieve the optimum capabilities for a given application (Table 2). Cost, size, weight, and power will only decrease as technology advances, as demonstrated by the recent release of the Apple iPhone X (27), which contains miniaturized, lower-power versions of the depth sensors, IMU, and processing required for advanced, handheld AR.

Figure 4.

Principles of Vergence and Accommodation

Extended reality platforms rely on positional tracking systems to track head mounted display, hand, and instrument positions. Tracking systems are either external (“outside-in”) systems (left) or on-board (“inside-out”) systems (right). Outside-in optical systems use visual and time-of-flight cameras to track the relative angles to triangulate positions and poses of active or passive marker arrays mounted to the headset, controllers, and tools. Inside-out tracking systems track the relative angles, hands, tools, and landmarks to triangulate the position and pose of the headset relative to landmarks, and the position of hands and tools relative to the headset without any external hardware.

Table 2.

Technical Resolution and Field of View for Various Displays

	Angular Resolution (arc-min)	Diagonal Field of View (º)	Tethered
Human eye (20/20)	1.0	∼150
60-inch 4K UHD @ 40”	1.0	73.7
24-inch 1,080P @ 24” (ZvR)	1.4	53.1
Oculus Rift (VR)	4.9	132.2	Yes
HoloLens (MxR)	1.4	34.5
Meta2 (MxR)	2.8	90.0	Yes

MxR = mixed reality; UHD = ultra high-definition; VR = virtual reality.

Depth at close distances is also perceived through accommodation (Figure 4, right), or the perception of depth due to disparity in focal depth, which is a challenge facing all conventional stereoscopic displays. Accommodation is required to allow a user to focus on instruments and digital objects at the same simulated distance (28) (e.g., surgical guidance overlays, within “personal space” and “action space”). Disparity between vergence depth displayed by HMD and accommodation expected by HVS is referred to the vergence and accommodation conflict (VAC) and is responsible for discomfort at these close working distances (29). Most display systems only support a single, fixed, focal plane for all digital elements (Figure 4), although some emerging technologies can provide multiple fixed focal planes by employing adaptive optics.

Platform Differentiation

Every design decision to mitigate these challenges affects applicability for use in each procedural environment. AR and MxR displays provide the best compromise between digital annotation and clear FOV. The high pixel density, large FOV devices designed for desktop or office use, require an umbilical to a high-powered workstation to support the processing required for their displays. These devices provide a larger digital display and a higher-resolution display, but at the expense of complicated setup before procedures and limited maneuverability during procedures. Untethered platforms generally have a reduced FOV and require battery power, but allow for unrestricted movement. CGH platforms provide the most realistic, true holograms within close working distance, but still require large supporting systems tethered to the display, and have limited working volume. These CGH displays and others that compensate for VAC are well suited where near-field interaction between the digital and physical is critical (e.g., projection of pre-procedural imagery onto a surgical field). Systems that cannot simulate accommodation can avoid discomfort by placing digital objects farther away, where the HVS is less sensitive to disparity in accommodation. Platforms utilizing external optical tracking achieve consistent tracking volumes but require additional equipment and clear lines of sight between cameras and devices, which increases initial installation and maintenance complexity. Mobile AR platforms with inside-out tracking of both position and gestures provide the most flexible platforms for intraprocedural use, and can mitigate VAC through careful placement of digital objects.

Conclusions

Rapid hardware advances driven by the revolution in mobile computing have finally brought devices that are tractable for medical applications into existence. These devices have the potential to provide physicians with a sterile interface that allows them to control 3D images. Early data show that this improved visualization will allow the physician to learn more quickly, interpret images more accurately, and accomplish interventions in less time. These improvements in physician performance based on better information will most likely translate into lower-cost procedures and better outcomes for patients.

Appendix

Supplemental Video 1