Immersive Virtual Reality for Visualization of Abdominal CT

Abstract

Immersive virtual environments use a stereoscopic head-mounted display and data glove to create high fidelity virtual experiences in which users can interact with three-dimensional models and perceive relationships at their true scale. This stands in stark contrast to traditional PACS-based infrastructure in which images are viewed as stacks of two-dimensional slices, or, at best, disembodied renderings. Although there has substantial innovation in immersive virtual environments for entertainment and consumer media, these technologies have not been widely applied in clinical applications. Here, we consider potential applications of immersive virtual environments for ventral hernia patients with abdominal computed tomography imaging data. Nearly a half million ventral hernias occur in the United States each year, and hernia repair is the most commonly performed general surgery operation worldwide. A significant problem in these conditions is communicating the urgency, degree of severity, and impact of a hernia (and potential repair) on patient quality of life. Hernias are defined by ruptures in the abdominal wall (i.e., the absence of healthy tissues) rather than a growth (e.g., cancer); therefore, understanding a hernia necessitates understanding the entire abdomen. Our environment allows surgeons and patients to view body scans at scale and interact with these virtual models using a data glove. This visualization and interaction allows users to perceive the relationship between physical structures and medical imaging data. The system provides close integration of PACS-based CT data with immersive virtual environments and creates opportunities to study and optimize interfaces for patient communication, operative planning, and medical education.

INTRODUCTION

Modern medical imaging techniques produce large data sets that are difficult to visualize and understand by both medical professionals and patients. There are several difficulties that these large data sets present. First, the amount of data means that viewing axial sections becomes problematic, simply because of the large number of slices that must be navigated. The navigation modalities for these slices, such as sliders and mouse wheels, become increasingly cumbersome as the number of slices increases. Secondly, as the complexity of medical image data grows, restricting the viewing modality to a traditional two-dimensional view may be sub-optimal. Such data is inherently three-dimensional (3D), and exploring it as a 3D quantity, through displays that offer stereoscopic depth perception, may give better insights and comprehension into the overall imaging data.

Immersive virtual reality (IVR) may offer a solution to the issues described above. In this paper, we describe IVRs presented through head-mounted displays (HMDs), stereoscopic display devices that allow the wearer to perceive a three-dimensional virtual environment as though present in it. Immersive virtual reality facilitates the investigations of situations that are difficult to study in the real world, whether for reasons of cost or complexity. It can provide visualization modalities that render complex data more comprehensible and interaction modalities that make large amounts of data more accessible. In this paper, we discuss IVR that uses an HMD to view the virtual world, and articulated data gloves to interact with it.

In this paper, we consider visualization of ventral hernias. Ventral hernias occur in up to 28% of patients undergoing abdominal operations — even in optimal conditions [Pans et al. 1998, Trimbos et al. 1992]. Repair of these hernias is fraught with failure; recurrence rates ranging from 24–43% [Luijendijk et al. 2000]. Recurrence of previously repaired VHs increases costs and morbidity to patients and can sometimes require multiple repairs. In some patients, repair of their end-stage VH may produce a worse outcome than a non-operative strategy [Aguilar et al. 2010]. Communication is a significant problem for clinicians and patients with ventral hernias.

In care planning, it is essential that the patient understand the urgency, degree of severity, and impact of a hernia (and potential repair) on patient quality of life. Hernias are defined by ruptures in the abdominal wall (i.e., the absence of healthy tissues) rather than a growth (e.g., cancer); therefore, understanding a hernia necessitates understanding the entire abdomen. Our proposed environment allows surgeons and patients to view body scans at scale and interact with these virtual models using a data glove.

IVRs have, of course, been used to visualize medical imaging data before. Some systems use a single imaging modality or semi-immersive environments [Zhang et al. 2001; Lo et al. 2007; Gallo 2010]. Using the hands to manipulate the 3D data has also been recognized as having value [Gallo 2010; Gallo et al. 2008]. In particular, Indhumathi et al. [2009] use both an HMD and a high-fidelity data glove to manipulate medical imaging data. The novelty of the present system is coupling of the manual interface with the IVR to achieve improved understanding of large imaging datasets.

METHODS

Data and Processing

Our abdominal segmentation method has been evaluated in separate work [Xu et al. 2012]. Briefly, retrospective, clinically acquired CT data on three male patients and one female patient with suspected VHs (three with confirmed hernia) were acquired in anonymous form under institutional review board supervision from the clinical PACS in DICOM format. Volumes were approximately 512×512×157 voxels with a resolution of 0.89×0.89×3 mm. A low threshold (200 HU) is used to identify the whole bone skeleton. Then, a high threshold (800 HU) is used to divide the skeleton into different components based on relative shape and position information. The skin and abdominal wall are segmented using level set techniques. The outer surfaces of each object (i.e., pelvis, femurs, spinal column, ribs, skin) were tessellated and exported for visualization. Surface colors and transparency were authored for each object using Mayavi, a 3D scientific visualization tool. These models were then exported into the virtual environment.

Materials and Apparatus

Our immersive virtual environment emulates a free-walking space (approximately 8m × 7m × 4m). The virtual environment is viewed through a full color stereo Nvis (Reston, VA) Nvisor SX60 HMD with 1280×1024 pixels per eye, a nominal field of view of 60° diagonally, and a frame rate of 60Hz. An InterSense IS-900 precision motion tracker updates the user’s rotational movements around all three axes, supplemented by optical tracking by four cameras of two infrared LEDs on the HMD to provide position and orientation information. The virtual environment is rendered using Vizard (Worldviz, Santa Barbara, CA). A wireless Cyberglove II data glove (Cyberglove Systems, San Jose, CA) for the right hand is used to track the fingers, enabling gesture control in our system. The data glove is instrumented with 22 high-accuracy sensors and can track the finger movements accurately. The global position and orientation of the hand is tracked with an eight camera Vicon (Los Angeles, CA) MX-F40 optical tracking system.

Mechanisms for Viewing the 3d Model the CT Slices in Immersive Virtual Environment

In the virtual environment, registered computed tomography (CT) images and the 3D model were superimposed. The CT images were made semi-transparent to make the 3D model visible and to give a volumetric effect to the ensemble. A user can interact with a given model by touching (with the index finger of the virtual hand) a CT slice to expose it. When exposed, the CT slice “pops” outside the model and becomes opaque for optimal viewing. The image can be made to disappear by brushing the image with the palm of the hand or by selecting another slice. An accordion-like affect can be achieved by running the virtual hand through the model.

Since the CT slices are along three axes, it can be difficult to select an individual slice from a complete grid. Therefore each axial direction is toggled by making a fist with one’s hand. The state transition was controlled by human making a fist gesture of right hand and releasing immediately. When the fist-gesture was captured by the data glove, the system transitioned to the next state immediately. The initial state had no CT images in it; this process is shown in Figure 1. In the state with the images in the scene, a user can select the image they want to observe as mentioned in the previous paragraph.

Figure 1.

The four states of the model

Volume Rendering in the Immersive Virtual Environment

In scientific visualization and computer graphics, volume rendering is a set of techniques used to display a 2D projection of a 3D discretely sampled data set. Here the 3D data set is a group of 2D slice images acquired by a CT scanner. These are acquired in a regular pattern as one slice approximately every millimeter and have a regular number of image pixels in a regular pattern. To render a 2D projection of the 3D data set, first we define a camera in space relative to the volume. The opacity and color of every voxel are also defined. Direct volume rendering is a computationally intensive task; here we achieve the volume render effect by rendering closely spaced slices with user-controlled transparency. An Xbox Controller (Microsoft, Inc., Redmond, WA) is used to adjust the mapping between intensity value (HU units) and transparency (“Alpha Histogram”), baseline brightness, baseline transparency (“alpha boost”), and to enable whole-volume intensity normalization. These controls are shown in a simulated heads-up display in the upper right field of view.

RESULTS

The abdominal model contains 600893 triangles. The CT images were generated from axial (plane = left/right – front/back), coronal (plane = left/right – head/toe), and sagittal (plane = front/back – head/toe) sections through the data at every millimeter. There were 157 CT scan images in the axial direction, 458 in the sagittal direction, and 458 in the coronal direction. The number of pixels in each dimension is: 512 (left-right) × 512 (front-back) × 157 (head-toe). A pixel corresponds to a unit volume of size 0.8945 mm (left-right) × 0.8945 mm (front-back) × 3 mm (head-toe). For the slice selection interaction, ten CT slices were used in each direction.

We demonstrated the system to two surgeons using an abdominal model containing a hernia; figure 2 illustrates the system in use. Remarks from the surgeons indicate that the system was immersive, easy-to-use, and conveyed a sense of scale and anatomy that is difficult to achieve through a desktop display (figure 3). They saw immediate uses for the system in both patient education and pre-operative planning.

Figure 2.

Illustrations of the proposed system in use. (a) Surgeon using the system. (b) Virtual hand interacting with 3D abdominal model. (c) Navigation of axial slices. (d) Navigation of sagittal slices.

Figure 3.

Illustration of user interaction with the abdominal model

The volume rendering controls enable real time interaction with the rendering properties so that users can adjust the display to reveal different parts of the anatomy from the abdominal wall to the skin, muscles, organs, and bones (as shown in figure 4). These dynamic changes are in addition to the user’s ability to walk around the model and peer over/in/through the display. Volume rendering of the CT data provides a different sense of relative anatomical relationship than the renderings of the segmented data (as in figures 1–3).

Figure 4.

Volume rendering for abdominal wall CT scan

DISCUSSION

We have demonstrated a system capable of visualizing large medical imaging datasets that offers opportunities for improved understanding of such datasets. This work is a demonstration of feasibility and establishes pilot platform on which to evaluate utility of IVR in medical imaging visualization and establish the relative merits of the IVR capability. Volumetric segmentations can be viewed and interacted with alongside traditional 2D slice rendering. This combined experience preserves/creates a perception of scale which may be non-intuitive given more traditional rendering mechanisms. The image pre-processing can be fully automated and triggered based on PACS status, so that IVR could be feasible within a clinical workflow.

Readily available flat screen virtual reality (e.g., Xbox Connect, Microsoft, Redmond, VA) and low cost IVR/HMD (e.g., <$50, Ilixco, Sacramento, CA) provide fascinating opportunities for integrating IVR technology with medical imaging systems. Important questions remain as to the relative utility of full scale (as presented, walk-around IVR) versus seated IVR versus flat screen virtual reality. Moreover, substantial work remains in optimizing intuitive interfaces for self-navigation and for guided navigation (e.g., shared experiences between patients and clinicians). Ongoing efforts are characterizing the utility of IVR for medical education and patient communication. In conclusion, integration of IVR technologies for data interaction with existing PACS infrastructures promises to be a fruitful area of exploration.