Developing a virtual reality simulation system for preoperative planning of thoracoscopic thoracic surgery

Abstract

Background

Video-assisted thoracoscopic surgery (VATS) has become a standard approach for the treatment of lung cancer. However, its minimally invasive nature limits the field of view and reduces tactile feedback. These limitations make it vital that surgeons thoroughly familiarize themselves with the patient’s anatomy preoperatively. We have developed a virtual reality (VR) surgical navigation system using head-mounted displays (HMD). The aim of this study was to investigate the potential utility of this VR simulation system in both preoperative planning and intraoperative assistance, including support during thoracoscopic sublobar resection.

Methods

Three-dimensional (3D) polygon data derived from preoperative computed tomography data was loaded into BananaVision software developed at Colorado State University and displayed on an HMD. An interactive 3D reconstruction image was created, in which all the pulmonary structures could be individually imaged. Preoperative resection simulations were performed with patient-individualized reconstructed 3D images.

Results

The 3D anatomic structure of pulmonary vessels and a clear vision into the space between the lesion and adjacent tissues were successfully appreciated during preoperative simulation. Surgeons could easily evaluate the real patient’s anatomy in preoperative simulations to improve the accuracy and safety of actual surgery. The VR software and HMD allowed surgeons to visualize and interact with real patient data in true 3D providing a unique perspective.

Conclusions

This initial experience suggests that a VR simulation with HMD facilitated preoperative simulation. Routine imaging modalities combined with VR systems could substantially improve preoperative planning and contribute to the safety and accuracy of anatomic resection.

Introduction

Surgery remains the standard of care for primary malignant lesions of the lung. Conventional thoracotomy has been gradually supplanted by minimally invasive surgery (MIS) over recent years, including video-assisted thoracoscopic surgery (VATS) and robotic-assisted thoracoscopic surgery (RATS), following continued improvements in surgical technologies and techniques. VATS or RATS are being applied to more complex procedures, including anatomic segmentectomy. Lack of stereoscopic vision and/or reduced tactile feedback during MIS may lead to unexpected intraoperative or postoperative complications, such as bleeding, pneumonia, atelectasis, and loss of pulmonary function (1). Therefore, accurate delineation of pulmonary anatomy and obtaining sufficient surgical margins are critical for successful sublobar resection.

Preoperative surgical planning with interactive three-dimensional (3D) computed tomography (CT) reconstruction is a useful method to enhance the surgeon’s knowledge of patient’s anatomy including identification of anatomic variants. Intraoperative 3D navigation is feasible and could contribute to safety and accuracy of anatomic resection (2,3). Imaging software programs that reconstruct two-dimensional (2D) CT data for 3D visualization of pulmonary vessels and bronchi have shown advantages for planning accurate sublobar resection (3,4). However, 3D CT provides only static simulation; dynamic simulation, reflecting the intraoperative deformation of lung, has not been developed. Several recent reports have described the innovative use of virtual reality (VR) simulation as a surgical training tool (5,6).

VR navigation with head mounted displays (HMD) is an immersive experience whereby the user's entire field of vision is replaced with a digital image. The images displayed to each eye are varied to create a stereoscopic effect. Advanced volume rendering provides more realistic representations, transforming medical images into a powerful communication tool.

The BananaVision software is a networked, multiuser VR tool for VR HMDs developed by researchers from Colorado State University. This platform allows users to better visualize both human anatomical learning models as well as medical images for the creation of instructional content and clinical evaluation. Further to the educational advantages of VR platforms in general, the networking capabilities allow multiple clinicians to interact in real-time with other specialists inside a common virtual space. These capabilities together provide a more meaningful way to interact with scientific and medical imaging data.

In this study, we aimed to develop a novel VR with HMD simulation system that generates virtual dynamic images based on patient-specific CT data to evaluate its utility for the surgical planning of lung segmentectomy. We present the following article in accordance with the MDAR reporting checklist (available at http://dx.doi.org/10.21037/jtd-20-2197).

Methods

This study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Research Ethics Board of Hokkaido University Hospital, Japan (018-0396) as a single-center, phase I, feasibility study and informed consent was taken from all the patients.

Patients underwent preoperative contrast-enhanced CT scan after intravenous administration of iodinated contrast medium (300 mgI/kg) using mechanical injector (Dual Shot GX; Nemoto Kyorindo, Tokyo, Japan) into the upper extremity over 15 seconds followed by a 30 mL saline flush. Contrast-enhanced CT scans were obtained using an Area Detector CT (ADCT; Aquilion ONE; Canon, Tokyo, Japan) and saved as DICOM files on a server client-type workstation (Zio Station 2; AMIN, Tokyo, Japan). The 3D image data were extracted in the DICOM file format from the Zio station 2. DICOM data of the lung, vessels [pulmonary artery (PA) and pulmonary vein (PV)], bronchi, and tumor were separately saved.

The DICOM dataset was imported into Fiji (https://fiji.sc/) for automated centerline detection of the PA, PV, and bronchi. The raw output was then imported into BananaVision using a dedicated macro plugin.

The BananaVision platform was operated on a Hewlett Packard Elite Desk workstation. The workstation was equipped with an Intel i7-8700 CPU, 32GB RAM, NVIDIA RTX2080 GPU, and Windows 10 Pro. The HMD used was Samsung Odyssey+. The BananaVision platform renders case-specific virtual models following importation of the processed CT data output.

Preoperative review of the resection plan was performed using the patient-specific 3D images. The original preoperative chest CT scan demonstrated a left upper lobe tumor; left upper lingula-sparing segmentectomy was planned. Representative images of the 3D model demonstrate the relationship between the tumor and pulmonary anatomy (Figure 1). The surgeon wears the HMD to display the virtual reality image. A hand device allows the surgeon to interact with the virtual objects. In preparing for VATS sublobar resection, critical segmental vessels and bronchi associated with the target segment were identified (Figure 2) (Video 1).

Figure 1.

A virtual dynamic image is generated from patient-specific 3D-CT data. The original preoperative chest CT scan demonstrating a left upper lobe tumor is shown (top row). Representative images of the 3D model (bottom row) demonstrate the anatomic relationship between the tumor and pulmonary anatomy.

Figure 2.

Preoperative simulation of lingula-sparing left upper segmentectomy. The surgeon wears a head-mounted display to visualize the augmented reality model. A hand device allows the surgeon to interact with the virtual objects.

Video 1.

Download video file ^{(8.2MB, mp4)}

Preoperative review using BananaVision. Functionality within the BananaVision platform allows real time adjustment of density/intensity thresholds with free manipulation of the 3D model while simultaneously tracking the surgeon's position.

Results

Operations were performed with the patient in lateral decubitus position under general anesthesia with one-lung ventilation. Two ports and a utility incision were made without rib spreading. The operation was performed entirely via thoracoscopic visualization. Representative images of the 3D model demonstrate high-resolution anatomical details of sub-subsegmental pulmonary artery branches. Intraoperative VATS images confirm the same vessel orientation as in the 3D model, prior to and after vessel ligation (Figure 3). The lingula-sparing left upper segmentectomy presented here was performed for resection of a 20mm nodular lesion in left segment 1+2. Based on preparation with the VR HMD, we readily identified variant PA vessel anatomy. Intraoperative ICG fluorescence was used to facilitate demarcation of the intersegmental border, as previously described (7). In brief, near infrared thoracoscopy (OPAL1, KARL STORTZ, Germany) was used to guide electrocautery marking of the resected segment following ligation of the relevant PA branches and injection of ICG (3 mg/kg). Lung parenchyma was divided using electrocautery or endoscopic staples along the marked intersegmental border (Video 2).

Figure 3.

Correlation of preoperative reconstruction and intraoperative findings. Representative images of the 3D model demonstrate high-resolution anatomical details of sub-subsegmental pulmonary artery branches (A1+2ci, A1+2cii) (top row). Intraoperative VATS images confirm the same vessel orientation as in the 3D model, prior to and after vessel ligation (bottom row).

Video 2.

Download video file ^{(22.1MB, mp4)}

The lingula-sparing left upper segmentectomy presented here was performed for resection of a nodular lesion of left segment 1+2. The lesion was 20 mm in diameter. Based on preparation with the VR HMD, we quickly identified the variant PA vessel anatomy. We used ICG fluorescence to facilitate demarcation of the intersegmental border.

Discussion

This is the first report to describe preoperative planning prior to anatomic pulmonary resection using an HMD-based VR system. In this preliminary study, we generated patient-specific high-resolution 3D lung models that could be freely manipulated using the BananaVision software platform for display on VR HMDs.

We identified several advantages of the VR HMD setup compared to conventional preoperative imaging review. Preoperative surgical review using 3D reconstructions has already shown great value for thoracic surgery (2-4). In the present study, we relied on the BananaVision platform to obtain VR 3D images for visualization of pulmonary anatomy preoperatively, whereby the surgeon could interact with the reconstruction more easily. Our results suggest an optimal pulmonary resection plan derived from preoperative VR 3D simulation could be directly applied to the operating room, providing surgeons confidence in their VATS approach. Surgeons could view the lung model from any potential orientation relative to the thoracoscope and instruments, facilitating preoperative planning. Anomalous pulmonary vessels could be easily identified, allowing surgeons to plan ahead an approach to managing these structures. Developments in computer sciences and their integration into the medical field have provided new opportunities for preoperative planning prior to complex procedures. This will prove to be a benefit both for the smooth technical performance of complex procedures and improved safety for the patient.

In terms of simulation training, preoperative review of patients’ individual anatomy, tumor size, and tumor location could be beneficial for assessment of surgical resectability during multidisciplinary case discussions. The VR 3D reconstructions can also be important educational tools for trainees. Navigation and review of the 3D models do not require the same interpolation as standard cross-sectional imaging; these models could therefore help trainees review basic anatomy as well as prepare more effectively for complex cases. The multi-user functionality of BananaVision allows both the trainee and attending surgeon to manipulate the same model cooperatively in a shared space, enabling interactive discussions. Simulation-based training can teach technical aspects of procedures and reduce the learning curve, since the surgical trainee can practice technical skills in a highly representative and time-efficient manner to achieve surgical capability (8). With simulation-based training, the trainee can adapt to the variety of procedures and intraoperative complications. Training using VR simulators has been shown to improve performance in the operating room (8).

At this time, BananaVision has been applied as a preoperative simulation tool. However, it may also have utility for real-time intraoperative navigation. If automated registration of endoscopic images to 3D reconstructions was integrated into the BananaVision platform, surgeons could theoretically perform real-time surgical resection with a dynamic BananaVision overlay.

Previously, our group developed an AR thoracoscopic surgical navigation system that accounts for the thoracoscope position to aid in confirmation of tumor location and surgical margin decision-making during MIS (9). This system was composed of a cone beam CT (CBCT), a conventional VATS platform, an optical tracking system, and a custom 3D visualization dashboard that linked these components together (GTx Eyes) (9). The CBCT-based 3D model is displayed from the perspective of the tracked thoracoscope and can be overlaid onto the endoscopic image to permit visualization of otherwise obscured structures. Our future goal is to integrate the previous GTx Eyes platform with the VR HMD system described here. This hypothetical combined system would provide surgeons with improved augmented guidance to enhance surgeon performance during complex thoracoscopic procedures.

An ideal multimodal imaging system would put critical anatomic information right where it is needed in the operative room. This level of integration is only possible though the surgical cockpit in RATS. However, the risk for informational overload must be balanced. It is vital that critical anatomy is displayed to the operator in a manner that is relevant (i.e., appropriately oriented) and timely (i.e., during critical dissection or prior to no-return decisions like stapling of bronchovascular structures). We believe a surgical approach combining a VR HMD system and a robotic platform could limit intraoperative and postoperative complications to improve general surgical performance.

Conclusions

In conclusion, this initial experience demonstrates the feasibility of preoperative planning using VR simulation with HMD. Routine preoperative imaging combined with VR could facilitate preoperative planning to improve the safety and accuracy of anatomic resection. Further studies are warranted to investigate the role of intraoperative VR guidance, especially during complex sublobar resections.

Supplementary

The article’s supplementary files as

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Research Ethics Board of Hokkaido University Hospital, Japan (018-0396) as a single-center, phase I, feasibility study and informed consent was taken from all the patients.