A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy

Abstract

This article presents a virtual surgical environment whose purpose is to assist the surgeon in preparation for individual cases. The system constructs interactive anatomical models from patient-specific, multi-modal preoperative image data, and incorporates new methods for visually and haptically rendering the volumetric data. Evaluation of the system’s ability to replicate temporal bone dissections for tympanomastoidectomy, using intraoperative video of the same patients as guides, showed strong correlations between virtual and intraoperative anatomy. The result is a portable and cost-effective tool that may prove highly beneficial for the purposes of surgical planning and rehearsal.

1. Introduction

Given the current availability of high-resolution three-dimensional medical imaging, surgeons commonly have access to multimodal anatomic data prior to undertaking a surgical procedure. Imaging studies such as computed tomography (CT) and magnetic resonance imaging (MRI) can offer accurate information on tissue composition and geometry, and are often used together given their complementary strengths. However, even after structures are identified on imaging, the surgeon must be able synthesize these data into a conceptual model that can predict what will be encountered intraoperatively. To achieve this, the surgeon often needs to take a number of steps in preparing for surgery:

1. Mentally co-register volumetric data from different modalities, so that the studies can be combined effectively to take advantage of the best aspects of each.

2. Formulate an integrated 3-D representation of the patient from the studies, so that anatomic relationships are understood from a variety of potential viewpoints.

3. Create a mental image that predicts how surgical manipulation and removal of tissues will affect subsequent access and exposure.

Accomplishing these steps can be a challenge, especially if the studies are examined as sequential two-dimensional slices, as is the current practice. Despite the use of multiplanar reconstructions, critical spatial relationships need to be inferred rather than seen directly as occurs in actual surgery. We have developed a virtual surgical environment intended to facilitate these steps, and optimize the benefits of available imaging.

Our goal has been to create an environment that can relatively quickly incorporate routine clinical studies, enabling real-time interactive preoperative assessment. Our approach thus far has focused on procedures involving the resection of cholesteatomas (skin cysts) from the middle ear and mastoid, collectively known as tympanomastoidectomy. Such procedures involve the removal of portions of the temporal bone to gain access to these cysts, which are commonly associated with chronic ear infections. The ability to experiment with varied approaches may prove beneficial to the outcome of the procedure.

Traditional imaging of the temporal bone prior to tympanomastoidectomy involves the use of high-resolution CT. This demonstrates bone and air quite well, and therefore shows other adjacent structures by the absence of either of these. It does not, however, differentiate between various types of soft tissue, such as scar or fluid, from the more surgically relevant cholesteatoma. Recently, diffusion-weighted MRI sequences have been used successfully to aid in the identification and localization of cholesteatoma [5]. We obtained MR imaging on a series of patients with chronic ear infections, and integrated its specific identification of cholesteatoma into the bony framework from the CT images.

A number of virtual environments for simulation of temporal bone surgery have been developed [2]. Wiet et al. describe dissection of a virtual temporal bone derived from CT images of a cadaveric specimen in an early work demonstrating a surgery simulator incorporating both visual and haptic feedback [12]. Agus et al. present a similar surgical training system where volumetric object models directly derived from CT and MRI data can be dissected [1]. They simulate a virtual surgical drill using a realistic, physically based model for haptic force feedback and bone tissue removal. Morris et al. present a comprehensive simulation environment for both training and performance evaluation of bone surgery [7]. They discuss techniques for automated evaluation and feedback, allowing the possibility of using the simulator for surgical skills assessment. In a recent study, Tolsdorff and colleagues have proposed the use of individual patient models derived from CT image data for virtual mastoid surgery [11], though the study was conducted using a training-oriented simulator [8], and only allowed for the import of bone tissue.

The majority of surgical simulation work to date has focused on surgical education, training, and assessment using standardized libraries of anatomic models. In contrast, the system described here is intended as a step towards surgical rehearsal, with which a surgeon can prepare for an upcoming case by practicing dissections on a virtual representation of the patient’s specific anatomy. To do this, it makes direct use of multi-modal preoperative imaging data with minimal need for preprocessing. It also incorporates new and efficient methods to render multiple volumetric data sets visually and haptically to enable interaction with the virtual anatomy in a manner familiar to surgeons.

2. Mathods & Materials

Preoperative image data from a growing library of 8 patients, each of whom was a candidate for tympanomastoidectomy, was collected for evaluation. Imaging for each patient consisted of a clinical CT scan of the temporal bone (Figure 1a) and two MR images: a T2-weighted FIESTA sequence and a diffusion-weighted PROPELLER sequence (Figure 1b). Conventional CT and MR imaging cannot easily identify cholesteatoma within the temporal bone, but diffusion-weighted MR imaging shows potential as the modality of choice for this purpose (Figure 1c). These images contain complementary information, and are used collectively to create a virtual model of the patient’s anatomy.

Figure 1.

A preoperative axial clinical CT scan of the temporal bone (a). Soft tissue is seen filling the middle ear (arrow). The corresponding slice of the MR PROPELLER sequence shows a hyperintense region indicative of cholesteatoma (b). A close-up of the cholesteatoma registered and superimposed on the CT (c).

2.1. Data Processing

Registration of the temporal bone CT and the MRI sequences was performed using Amira 5.3 (Visage Imaging Inc., San Diego, CA). The DICOM images were imported into Amira and registered in three dimensions using the automated registration tool with a mutual information metric and the conjugate gradient optimization algorithm.

Anatomic structures of interest were extracted from the different imaging datasets using Amira’s computer-assisted segmentation tools. The CT images were used to segment the semicircular canals, facial nerve and ossicles. The FIESTA image sequence was used to segment the carotid artery and sigmoid sinus, and the PROPELLER image sequence was used to segment cholesteatoma. Segmentations were exported both as label volumes and as Wavefront object meshes to be used in our virtual surgical environment.

Data processing, including segmentation of all vital structures and smoothing of the resulting models, took approximately two hours for each patient. Not all processing steps are necessary in every case though: a clinical CT can be used directly by our virtual surgical environment for visualization and dissection of bone without any preprocessing.

2.2. Volume Rendering

Our system uses three-dimensional preoperative image data as the principal representation of the virtual patient. As this model consists primarily of volumetric data, it is natural to adopt a real-time volume rendering approach for data visualization within the simulation. We have developed a multi-modal volume rendering method based on a GPU-accelerated ray casting approach [9] that is capable of simultaneously displaying the different forms of data supplied to the virtual surgical environment.

We simultaneously display a clinical CT volume (with both isosurface and direct volume rendering), a segmentation volume, and a dissection mask volume in a single rendering pass. Each data volume is represented as a separate 3D texture on the graphics processor, but is co-located in texture coordinate space with the primary (CT) volume. This allows the rendering algorithm to perform ray casting in lock step through all data volumes simultaneously. A ray is cast for each pixel in the rendered image from the viewpoint in virtual space through the volume data, accumulating color and opacity information that determines its final appearance. A shader program samples all volumes along the ray at a regular spatial interval, taking advantage of the highly parallel computational architecture of modern GPUs to achieve interactive visualization.

The preoperative clinical CT serves as the primary volume, and is represented at its native resolution as a 16-bit single-channel texture in video memory. The rendering algorithm traverses the ray until it first encounters a sample value greater than the isosurface value. Several interval bisection steps [3] are then performed to refine the ray-isosurface intersection point, and the surface is shaded with configurable material properties. If the surface is partially or fully transparent, the ray is continued, accumulating color and opacity through the interior of the volume in a front-to-back composition using a pre-integrated color transfer function [9], until the exiting isosurface is found. Thus, the appearance of the primary volume is controlled by an isosurface value (in CT Hounsfield Units) with material properties and a user-defined transfer function that maps Hounsfield Units to optical properties, both of which can be specified interactively. In our data, we have found that an isosurface provides a good indication of the tissue-air boundary, and direct volume rendering generates a realistic visualization of the bone tissue (Figure 3c).

Figure 3.

Temporal bone anatomy, including the sigmoid sinus, semicircular canals, facial nerve, ossicles, carotid artery, and a cholesteatoma lesion rendered as polygonal meshes (a), the larger structures rendered as volumetric isosurfaces with a semi-transparent bone surface in front (b), and direct volume rendering of the bone in a full composite, ready for surgery (c).

A segmentation or label field for an anatomical structure consists of a binary volume which indicates inclusion of contained voxels into the structure. We apply a preprocessing step that combines all non-overlapping label volumes into a two-channel, 8-bit luminance-alpha texture to be rendered volumetrically (Figure 2a, 2b). The alpha channel contains a union of the segmentations, and is low-pass filtered to facilitate isosurface rendering without stepping artifacts [4]. The luminance channel contains a unique index for each constituent structure that maps to a set of material properties, and a dilation operation is performed on the final image to ensure that texture sampling retrieves the correct index. The ray-casting algorithm samples this volume in lock step with the primary volume, and if the sample is found to lie within a segmented structure, material properties are retrieved from an auxiliary texture to perform shading (Figure 3b).

Figure 2.

A slice of a combined and dilated segmentation volume showing the cholesteatoma, cochlea and semi-circular canals, and carotid artery in different colors over the CT scan (a), and the low-pass filtered union of the structures for volumetric isosurface rendering (b). Note the cholesteatoma in (a) is a result of dilation from an adjacent slice, and is absent in (b). The black area in a close-up of a mask volume shows the smooth edge of a region removed by a spherical burr (c).

Finally, the mask volume, which controls visibility of the model, is an 8-bit, single-channel representation that can represent smooth surfaces to a sub-voxel resolution (Figure 2c). During ray casting, both the primary and segmentation volumes are modulated by the mask value. Thus, editing of the volume data can be accomplished by attenuation or zeroing of the mask volume.

In addition to the volumetric data, our virtual surgical environment uses polygonal models to represent surgical instruments and certain segmented structures. Anatomical structures with details near to or finer than the native voxel resolution of the data may be better represented as polygonal meshes (Figure 3a). Examples of these structures include the facial nerve and the ossicles. However, the rendering of polygonal meshes is not affected by changes in the mask volume described earlier, and thus any structure that permits dissection should be represented as part of the segmentation volume.

2.3. Haptic Rendering

The primary representation of the patient’s anatomy in our virtual surgical environment is in volumetric form, and thus a method for haptic interaction with volume data is required. McNeely et al. proposed a popular algorithm for haptic rendering of volumetric geometry [6], and several variants have been described for use in surgical simulators [1,7,8]. However, these methods do not prevent tool-tissue interpenetrations in the simulation environment, and can suffer from the virtual instrument passing completely through a thin layer of tissue (Figure 4a), especially when using commercially available haptic devices with lower force output and limited stiffness capabilities.

Figure 4.

With a point sampling algorithm, the contact force can be in the wrong direction when the instrument is pushed into a thin object (a). A proxy-based algorithm can constrain an interaction point to the surface (b). We combine these algorithms to prevent the virtual instrument from popping through thin layers of bone (c).

Proxy-based rendering algorithms constrain the virtual instrument to the surface of the geometry, preventing pop-through problems (Figure 4b). Salisbury & Tarr have described an algorithm for proxy-based haptic rendering of implicit surfaces [10] that can readily be adapted to render isosurface geometry embedded within volumetric data. A volume can be treated as a discrete sampling of a scalar field, and can be sampled at arbitrary positions through interpolation. Rather than evaluating an analytical gradient, the surface normal can be estimated using a central difference. The primary limitation here is that a tool can only be modeled as a single point for interaction with the volume.

A virtual surgical drill allows dissection of the anatomy in our virtual surgical environment. We model the spherical burr of the drill by extending the proxy-based, point interaction algorithm to incorporate elements of the method described by [7]. The burr is discretized into individual points in three-dimensional space that occupy the volume of the sphere. During interaction, the volume is sampled at these points, and any point found to lie within the surface exerts a fixed amount of force toward the center of the burr. In addition, we treat the center of the burr as a proxy point, so that if the haptic device moves in a way that this point penetrates into an object, the proxy-based algorithm constrains the center of the virtual instrument to lie on the surface of the object. A virtual spring exerts additional force proportional to the displacement between the device position and the proxy point. The end result is that superficial interaction forces are computed primarily from point sampling, whereas the virtual spring force dominates during haptic exploration involving deep tool-tissue interpenetration (Figure 4c).

Tissue resection is modeled by attenuating or removing voxels from the mask volume in a manner similar to that described in [8]. When the virtual drill is on, an anti-aliased voxelization [4] of the spherical burr is subtracted from the mask volume, preserving a smooth, accurate cut surface (Figure 2c). This technique allows modeling of tissue dissection at a sub-voxel resolution, and prevents the jagged or stepped appearance that normally results from voxel-resolution modification of volume data.

3. Results & Conclusions

Images from the use of our virtual surgical environment on data from two selected patients are shown in Figure 5. We have been able to replicate salient anatomic detail in the virtual environment as compared to the video images taken during actual tympanomastoidectomy. The geometry from the CT dataset yields a subjectively accurate representation of the bony contours seen during surgery. Similarly, the cholesteatoma volume derived from PROPELLER MR imaging is accurately placed within the bone, and presents a realistic representation of what the otologic surgeon will encounter in the patient. By rendering the bone transparent, other segmented vital structures can be seen in their familiar relative locations. Our preliminary subjective experience suggests that our virtual surgical environment can offer an accurate and interactive representation of patient-specific anatomy.

Figure 5.

An intraoperative video capture during right tympanomastoidectomy (a) and the corresponding image from a virtual dissection (b) using that patient’s preoperative imaging data. The cholesteatoma (white) was automatically segmented from MR imaging, and has been exposed following removal of overlying bone. Images (c) and (d) demonstrate similar images from a different patient during left tympanomastoidectomy. The size and extent of cholesteatoma in each case was accurately predicted and superimposed onto the bone.

Our system represents a step towards the use of a virtual environment to prepare for tympanomastoid surgery. It enables the relatively rapid integration of multi-modal imaging datasets, direct volume rendering, and a means of manipulating preoperative clinical data in a surgically relevant manner. We anticipate that the methods described can be generalized to a variety of surgical procedures. Clearly tools such as this require objective validation to ensure that they can benefit a surgeon in preparing for an operative case. We intend to carry out such studies in the future as our system becomes further refined. We are encouraged by the assumption that the more a surgeon is familiar with working in and around specific anatomy, the more he or she is likely to be effective. Offering surgeons such an opportunity holds great potential.