Exploring virtual reality technology and the Oculus Rift for the examination of digital pathology slides

Abstract

Background:

Digital slides obtained from whole slide imaging (WSI) platforms are typically viewed in two dimensions using desktop personal computer monitors or more recently on mobile devices. To the best of our knowledge, we are not aware of any studies viewing digital pathology slides in a virtual reality (VR) environment. VR technology enables users to be artificially immersed in and interact with a computer-simulated world. Oculus Rift is among the world's first consumer-targeted VR headsets, intended primarily for enhanced gaming. Our aim was to explore the use of the Oculus Rift for examining digital pathology slides in a VR environment.

Methods:

An Oculus Rift Development Kit 2 (DK2) was connected to a 64-bit computer running Virtual Desktop software. Glass slides from twenty randomly selected lymph node cases (ten with benign and ten malignant diagnoses) were digitized using a WSI scanner. Three pathologists reviewed these digital slides on a 27-inch 5K display and with the Oculus Rift after a 2-week washout period. Recorded endpoints included concordance of final diagnoses and time required to examine slides. The pathologists also rated their ease of navigation, image quality, and diagnostic confidence for both modalities.

Results:

There was 90% diagnostic concordance when reviewing WSI using a 5K display and Oculus Rift. The time required to examine digital pathology slides on the 5K display averaged 39 s (range 10–120 s), compared to 62 s with the Oculus Rift (range 15–270 s). All pathologists confirmed that digital pathology slides were easily viewable in a VR environment. The ratings for image quality and diagnostic confidence were higher when using the 5K display.

Conclusion:

Using the Oculus Rift DK2 to view and navigate pathology whole slide images in a virtual environment is feasible for diagnostic purposes. However, image resolution using the Oculus Rift device was limited. Interactive VR technologies such as the Oculus Rift are novel tools that may be of use in digital pathology.

INTRODUCTION

Mixed reality technologies, including augmented reality (AR) and virtual reality (VR), either supplement or completely replace the real world with computer-generated data to facilitate user interaction via one's natural senses. Both VR and AR technologies have been shown to be beneficial in providing contextual learning experiences, enabling the exploration of complex networks of information, and constructing new models of understanding.[1] VR technology, which was all the rage among technophiles in the early 1980s and 1990s, has been utilized in various applications from gaming to engineering.[2,3,4] However, early prototypical VR systems came with a hefty price tag. Additionally, early VR systems often featured large, cumbersome head-mounted displays (HMD), which were hampered by narrow fields of view (FOV), low frame rates, and suboptimal latencies. As a result, the ‘VR experience’ offered by such systems were not very immersive and rather uncomfortable to wear for extended periods of time. Thus, work with VR technology was essentially limited to researchers and developers with access to these proprietary tools.

More recently, Oculus VR, a company acquired by Facebook following a successful Kickstarter campaign, is on the verge of releasing the first consumer version of it's low-cost, lightweight VR platform, the Rift. The Oculus Rift is a VR HMD with a wide FOV (110°), high-definition quality video resolution (2160x1200), and a high frame rate (90Hz), ensuring little to no lag with head movement tracking. In addition, the Oculus Rift relies on an embedded infrared tracking system that enables positional and rotational tracking of the HMD, which has implications for clinical research.[5] Putting on the headset replaces an individual's field of vision with a digital image, where each eye looks through a different lens to create a stereoscopic three-dimensional (3D) effect.

A few studies have investigated the role of VR and AR technologies as tools for medical education and training, where rich contextual learning environments are required.[6,7,8,9,10,11,12,13] Much of this work stems back to the early 2000s, when these technologies were still in their infancy. A handful of studies have purported investigating the role of VR technology in pathology; however, there are practically no investigations that leverage the current iterations of VR or AR technology.[13,14]

Several pathology laboratories are currently in the process of transitioning to using digital imaging systems, such as whole slide imaging (WSI).[15,16] WSI platforms emulate traditional light microscopy by digitizing glass slides using specialized hardware (e.g., whole slide scanners) and software (e.g., vendor-specific or vendor-agnostic digital slide viewers). Initially, WSI platforms were used primarily for educational or research purposes. More recently, WSI solutions are being deployed for diagnostic purposes.[17,18,19,20,21,22,23] Digital slides obtained from WSI platforms are typically viewed in two dimensions either using a desktop personal computer monitor/display or mobile device (e.g., tablet). Current complaints from users of WSI are that navigation of digital slides is slow and tedious, especially with input devices such as a computer mouse.[24] Indeed, employing gaming controllers such as those used by Sony PlayStation to create an ultra-high-speed WSI viewing system appear to improve user viewing satisfaction.[25] Pathologists also complain that when viewing a digital slide on a computer monitor, they lose the benefit of peripheral vision they are used to when looking at a glass slide with a conventional light microscope. This may partly explain why pathologists like using very large displays made up of many monitors (e.g., power wall) to interact with virtual slides.[25,26]

To the best of our knowledge, we are not aware of any studies regarding regarding digital pathology slides being viewed in a VR environment. Therefore, we sought to explore the use of the Oculus Rift for examining digital pathology slides.

METHODS

Virtual Reality Setup

An Oculus Rift Development Kit 2 (DK2) was connected through USB cables to a 64-bit computer (HP Z440 Workstation, Intel Xeon CPU E5-1650 v3, 32GB DDR4-2133, NVIDIA GeForce GTX Titan X GPU, 512GB PCIe SSD, 1TB 7200RPM HDD) running Microsoft Windows 10 (64-bit). Whole slide images acquired for this study were stored on a remote server and viewed using ImageScope (Leica Biosystems, Wetzlar, Germany) software. Additional software was necessary to display two-dimensional (2D) whole slide images in a stereoscopic 3D virtual environment. Initially, we investigated the possibility of creating our own VR software using the development kit provided by Oculus (SDK 0.8.0.0) and a unity 3D plugin (http://www.unity3d.com/unity), a popular cross-platform game engine. A base example was built in the form of an empty, cubic room with static images used to wallpaper the walls [Figure 1]. For ease of use, we opted to instead use commercially available Virtual Desktop software (v0.9.34) (http://www.vrdesktop.net/) for navigation of digital slides. Navigation of digital whole slides with the Oculus Rift was tested using keyboard controls, an optical mouse, and a Leap Motion controller (https://www.leapmotion.com/product/vr). The Leap Motion controller relied on two monochromatic infrared cameras and three infrared light-emitting diodes to generate 3D positional data. Leap Motion enabled users to perform hand gesture-controlled navigation of digital slides in a VR environment.

Figure 1.

Example of a stereo rendering constructed using the Oculus SDK and unity three-dimensional plugin

Study Design

De-identified glass slides from twenty cases (n = 20) were obtained from a lymph node teaching set. Cases were randomly selected by two pathologists who were nonparticipants in the viewing phase of the study. Ten of the twenty cases harbored malignant diagnoses including metastatic ductal carcinoma, lobular carcinoma, signet ring cell carcinoma, leiomyosarcoma, papillary thyroid carcinoma, malignant melanoma, squamous cell carcinoma, prostatic adenocarcinoma, endometrioid endometrial adenocarcinoma, and Hodgkin's lymphoma. The remaining ten cases harbored benign diagnoses including reactive paracortical lymphoid expansion, normal lymph node, fatty infiltration of an inguinal and pelvic lymph node, tuberculous lymphadenitis, Epstein–Barr virus-associated lymphadenitis, cat scratch disease, HIV-associated lymphadenitis, sarcoidosis, and sinus histiocytosis.

A single glass slide from each of the twenty cases was digitized using an Aperio ScanScope XT WSI scanner (Leica Biosystems, Wetzlar, Germany) with an Olympus UPlanSAPO 20x/0.75 objective. The images were acquired at a digital resolution of 0.5 μ/pixel. The resultant whole slide images were acquired as JPEG2000 encoded SVS files and were not further compressed. Digital slides were first viewed using the Oculus Rift DK2 and Virtual Desktop application. After a washout period of 2 weeks, the participants again evaluated the same digital slides on a 27-inch in-plane switching (IPS) 5K display (HP ZR27q) with a resolution of 4096 × 2160 pixels, or a 24-inch IPS display (HP ZR2440w) with a resolution of 1920 × 200, both with a landscape orientation.

Three, board-certified, attending pathologists with subspecialty expertise in cytopathology, breast/gynecologic, and gastrointestinal pathology were recruited for review of the aforementioned digital slides. Each of the three participants had prior experience in viewing digital pathology slides using traditional WSI modalities. The pathologists were first trained to navigate whole slide images in our VR environment [Figure 2]. The primary endpoints assessed included diagnostic concordance and time to “read” (interpret) digital slides using both modalities. Diagnostic concordance was defined as (1) accurate classification of cases as either benign or malignant and (2) rendering a specific pathologic diagnosis (i.e., metastatic carcinoma). Time to interpret cases was recorded using a stopwatch. The pathologists were also asked to rate image quality (on a scale of 1–10), ease of navigation (on a scale of 1–10), and their diagnostic confidence (on a scale of 1–10) for both modalities. Their answers were recorded using a customized data capture sheet created with Google Forms. All primary and secondary endpoints were collected on a per slide/case basis, and the average values for each pathologist were tabulated as “overall" values.

Figure 2.

Virtual reality-based digital slide navigation using Oculus Rift Development Kit 2, virtual desktop, and a web-based whole slide imaging browser

RESULTS

Technically, digital images were able to be viewed in a VR environment wearing the Oculus Rift DK2. All input methods to navigate slides worked satisfactorily. However, the Leap Motion method resulted in the most fatigue after prolonged hand gestures. All three reviewers confirmed that digital pathology slides were easily viewable in a VR environment using the Oculus Rift DK2. There was 90% diagnostic concordance between the traditional method of reviewing WSI on a flat computer monitor and the VR method. Table 1 summarizes the recorded endpoints for each of the reviewing pathologists. The variation in time to interpret digital pathology slides did reach statistical significance when tested using an unpaired t-test (two-tailed P = 0.0009). Variations in image quality and ease of navigation also reached statistical significance when tested using unpaired t-tests (two-tailed P = 0.0001 and 0.027, respectively). The variations in diagnostic confidence, however, did not reach statistical significance (two-tailed P = 0.15). There were no significant side effects (e.g., motion sickness) associated with the use of the Oculus Rift; however, the participants did note that the VR headset did feel heavy after prolonged usage.

Table 1.

Summary statistics of endpoints for each of the reviewing pathologists

CONCLUSION

This proof of concept study represents the first attempt to examine pathology whole slide images in a VR environment. It is important to note, however, that the aim of this initial study was on proof-of-concept and feasibility as opposed to a comprehensive VR validation study. As such, the small sample size of only twenty cases likely limited our ability to discern meaningful differences between VR and traditional modalities. Nonetheless, our findings demonstrate that digital pathology slides can be successfully interpreted using VR. VR technology today has become readily accessible so that it no longer requires the use of very expensive, complicated, or proprietary hardware or software. Therefore, pathology departments currently using WSI can readily leverage VR technologies, such as the Oculus Rift, without significant investment. A widely available, commercial version of the Oculus Rift is expected in 2016.[27] In order for digital slides to be viewed with the Oculus Rift DK2, we had to integrate a WSI viewer into a virtual environment, map the WSI viewer navigation controls into this environment, and utilize the virtual environment to create a novel 3D interaction pattern for navigating digital slides. Navigation was possible using standard input devices (e.g., keyboard, mouse) and natural senses (e.g., hand gestures). We utilized the well-known “aspects of VR” classification scheme to determine which software to employ in this study. This classification includes six major factors of VR: Interaction, immersion, simulation, visualization, real space, and autonomous agents.[28]

To create a truly immersive and engaging VR experience, certain factors must be taken into consideration. A high frame rate and low HMD latency are perhaps the most essential factors in the creation of a truly immersive VR experience. The frame rate is the frequency at which a device displays continuous images for human visual systems to perceive. While the majority of current media is run at 30–60 frames per second, VR media requires higher frame rates to ensure smooth movements and prevent motion sickness. The DK2 makes virtual objects feel almost tangible by supposedly maintaining a frame rate of up to 75 frames per second. Unfortunately, the frame rate was varied significantly as users navigated the digital slides, resulting a noticeable “judder.” Latency, in the context of VR, refers to the total time between movement of the user's head and the updated image being displayed on the screen. Thus, a lower latency results in a more immersive and comfortable user experience. Low latency head tracking is part of what sets the Rift apart from other VR technologies.

We uncovered several limitations when using VR technology for digital slide navigation. First, the image quality on the Oculus Rift DK2 was limited by the resolution of the organic light-emitting diode display panel in the headset, which only has a resolution of 960 × 1080 per eye. In addition, there are hardly any medical applications available for use with VR. While the VR environment we developed provides pathologists with a wide FOV because the digital images themselves were not acquired in 3D, the VR environment only created a virtual projection of the 2D whole slide image floating in a simulated 360° environment. This may explain why the Oculus Rift did not significantly enhance the navigation of digital slides. Moreover, navigation may have been negatively impacted by the input devices (mouse and keyboard) and pathologists were confined to use for the comparative study. More intuitive input methods (e.g., wearable gloves) may enhance slide navigation. Oculus is currently developing a pair of lightweight, wireless, handheld “motion controllers” that may perform better than traditional input devices.[29,30] In addition, our participants did not experience any serious negative side effects largely because they did not spend much time immersed in the VR environment. Given the well-documented side effects associated with VR immersion (e.g., nausea, eye-strain, and seizures), the long-term usage of VR in clinical practice is an area that deserves further investigation.[31]

Digital pathology is becoming more relevant in today's practice of pathology. Unfortunately, many pathologists have encountered frustration when attempting to review digital pathology slides because of navigation difficulties encountered with certain input devices or viewing limitations with mobile computing devices such as laptops. Hence, the availability of an immersive, private, and more natural means of digital slide review would be very helpful. A VR headset such as Oculus Rift could allow pathologists to more easily perform telepathology consultations in a secure environment. VR also has a great potential use in pathology teaching, especially if combined with 3D images (e.g., 3D models of gross pathology specimens). Our group is investigating the methods for generating 3D digital images of tissue blocks that may be benefited from stereoscopic 3D visualization in a VR environment.

Financial Support and Sponsorship

Nil.

Conflicts of Interest

There are no conflicts of interest.