A multiparametric magnetic resonance imaging-based virtual reality surgical navigation tool for robotic-assisted radical prostatectomy

Abstract

Objective

Increased computational power and improved visualization hardware have generated more opportunities for virtual reality (VR) applications in healthcare. In this study, we test the feasibility of a VR-assisted surgical navigation system for robotic-assisted radical prostatectomy.

Material and methods

The prostate, all magnetic resonance imaging (MRI) visible tumors, and important anatomic structures like the neurovascular bundles, seminal vesicles, bladder, and rectum were contoured on a multiparametric MRI using an in-house segmentation software. Three-dimensional (3-D) VR models were rendered and evaluated in a side room of the operating room. While interacting with the VR platform, a real-time stereo video capture of the in situ prostate was obtained to render a second 3-D model. The MRI-based model was then overlaid on the real-time model by using an automated alignment algorithm.

Results

Ten patients were included in this study. All MRI-based VR models were examined by surgeons immediately prior to surgery and at important steps where visualization of the tumors and their proximity to surrounding anatomic structures were critical. This was mainly during the preparation of the prostatic pedicles, neurovascular plexus, the apex, and bladder neck. All participants found the system useful, especially for tumors with locally aggressive growth patterns. For small and centrally located tumors, the system was not considered beneficial due to lack of integration into the robotic console. A fully integrated system with real-time overlays within the robotic stereo viewer was found to be the ideal scenario.

Conclusion

We deployed a preliminary VR-assisted surgical navigation tool for robotic-assisted radical prostatectomies.

Introduction

With 164,690 estimated new cases and 29,430 estimated deaths, prostate cancer remains the most common male malignancy and the second most common cause of cancer-related deaths in the United States.^[1] Although many different treatment modalities have been proven to be effective in treating localized prostate cancer, radical prostatectomy-the complete removal of the prostate and seminal vesicles-is still the most common approach preferred by patients.^[2] Recently, laparoscopic operation techniques have gained in popularity with robot-assisted radical prostatectomy (RARP) becoming the most common technique for prostate cancer surgeries.^[3,4] Although a clear benefit could not be proven when it was compared to open surgery in terms of oncological and functional outcomes, its 3-D stereoscopic view and 360 degrees of freedom of movement of the instruments offer major advantages, especially during reconstructive steps of the surgery. Recently, major advancements in virtual reality (VR) have occurred. The technology was postulated in the 1950s and since then has been deployed mainly by the film and entertainment industry but mostly remained a small niche market. This also held true for biomedical applications where VR-systems were mainly utilized in surgical training and psychotherapy, e.g., in the treatment of phobias, post-traumatic stress disorder, and pain.^[5–7] However, with further developments of more advanced graphic processing units (GPU) and low-persistence organic light-emitting diodes, the technology has experienced a renaissance and become more applicable outside the entertainment industry. In this explorative feasibility study, we aim to investigate the use of next-generation VR technology in conjunction with novel computer vision algorithms as an aid for surgical navigation during RARP.^[8]

Material and methods

Patient population

This study was compliant with the Health Insurance Portability and Accountability Act. Since no additional interventions were performed, the institutional review board approval was waived. Patients with intermediate to high-risk prostate cancer were thoroughly informed about the risks and benefits of all the available treatment options of organ-confined prostate cancer at our institution. Overall, 10 patients who opted for robotic-assisted radical prostatectomy were included in this proof-of-concept study. All the patients were intermediate or high-risk patients, according to the National Comprehensive Cancer Network criteria. Patient demographics and tumor characteristics are summarized in Table 1. Two patients (patients 5 and 8) opted for a neoadjuvant Enzalutamide trial offered at our institution. In this trial, patients receive six months of androgen deprivation by a combination of Goserelin (10.8 mg for 12 weeks) and Enzalutamide (160 mg/d) with subsequent RARP (NCT02430480).

Table 1.

Patient demographics and clinical information prior to surgery

Patient	Age (years)	PSA (ng/mL)	Biopsy Gleason	Core involvement	Comments
1	51	7.56	4+3	12/12 systematic	Only systematic biopsies
2	56	9.24	3+4	5/6 targeted
3	54	19.19	4+3	2/8 targeted
4	52	5.54	3+4	2/2 targeted and 2/12 systematic
5	73	0.02	4+4	4/4 targeted and 2/12 systematic	S/P Neoadjuvant Enzalutamide
6	55	10.86	3+4	4/4 targeted and 3/12 systematic
7	72	16.03	4+4	3/6 targeted and 3/12 systematic
8	73	<0.02	4+3	4/4 targeted and 0/12 systematic	S/P Neoadjuvant Enzalutamide
9	62	10.74	4+4	2/8 targeted and 0/12 systematic
10	63	6.15	3+4	2/3 targeted and 1/12 systematic	Biopsies performed outside our institution

MRI technique

All patients who are treated for localized prostate cancer routinely undergo multiparametric MRI of the prostate on a 3T magnet (Achieva, Philips Medical Systems, Best, The Netherlands). For staging scans, we always utilize combined 16-channel surface coil (SENSE, Philips Medical Systems, Best, The Netherlands) and an endorectal coil (BPX-30, Medrad, Pittsburgh, Pennsylvania) inflated with 45 ml of perfluorocarbon (3 mol/L Fluorinert^™) during the MRI acquisitions. Our scans routinely include triplanar T2-weighted turbo spin echo, diffusion weighted imaging, apparent diffusion coefficient maps, and dynamic contrast-enhanced MRI. Our imaging protocol is summarized in Supplemental Table 1.

MRI evaluation and reporting

All exams were interpreted by one highly experienced radiologist (B.T., 10 years of experience in prostate cancer imaging). All MRI scans were evaluated in accordance with Prostate-Imaging Reporting and Data System version 2 (PI-RADSv2).^[9] Up to four lesions were reported and an assessment category based on the 5-tier PI-RADSv2 system was assigned to each lesion. Furthermore, imaging features that indicate an aggressive growth pattern, such as extraprostatic extension and seminal vesicle invasion, were reported in all detected lesions using a standardized in-house method.^[10] These imaging features correlate with the presence of adverse pathologies in patients undergoing radical prostatectomy and can potentially influence decision-making during surgery. For instance, the presence of a breach of the prostate capsule could prompt the surgeon to perform a wide excision at this location to decrease the risk of residual cancer and improve oncological outcomes. Table 2 summarizes important imaging parameters retrieved from the radiology report of each patient.

Table 2.

Imaging features from multiparametric MRI scans obtained prior to surgery

Patient	Prostate volume (mL)	Number of lesions	PI-RADSv2 category	Lesion(s) size (mm)	Additional MRI features related with local tumor staging
1	23	1	5	19	Broad capsular base and capsular irregularity, possible extraprostatic extension
2	40	2	4, 3	18, 24	None
3	54	3	4, 4, 3	7, 5, 6	None
4	19	1	4	13	None
5	46	2	5, 4	17, 6	Broad capsular base and capsular irregularity, possible extraprostatic extension (after treatment)
6	25	1	4	18	None
7	41	2	5, 5	16, 18	Broad capsular base and capsular irregularity, possible extraprostatic extension
8	73	2	5, 5	30, 30	Possible extraprostatic extension and bladder wall involvement (after treatment)
9	39	4	5, 4, 4, 3	16, 12, 10, 4	None
10	45	1	4	13	Slight capsular bulge, no extraprostatic extension

MRI-transrectal ultrasound (TRUS) fusion-guided targeted prostate biopsy

To establish the prostate cancer diagnosis, patients underwent transrectal MRI-TRUS fusion-guided biopsy after their routine MRIs as previously described.^[11] For patients who had undergone prior biopsies outside our institution, the decision to repeat a biopsy was individually made, based on the patient’s preference, and after a multi-disciplinary discussion. The images were then processed by the radiologist prior to the day of the biopsy. The prostate boundaries were segmented on the axial T2-weighted MRI and the lesions detected on MRI were labeled for targeted biopsies (DynaCAD, Invivo, Best, The Netherlands). The UroNav MRI-TRUS fusion-biopsy system (Invivo, Best, The Netherlands) was used to obtain targeted and systematic biopsies in one session. Each region of interest was sampled in both the axial and sagittal planes. The systematic biopsy was performed using an extended sextant template with 12 cores.

Prostatectomy procedure

All patients underwent RARP performed by a single surgeon with 20 years of experience (P.A.P). The da Vinci^® robotic system (Intuitive Surgical, Sunnyvale, CA, USA) was used in all patients. The decision to perform nerve-sparing was shared, and based on clinical parameters (serum PSA, clinical stage) and multiparametric MRI findings. The multiparametric MRIs were routinely incorporated in surgery planning while informed consent was being obtained from the patient, immediately prior to surgery. In prostate regions with a potential breach of the capsule, a wide excision was performed to mitigate the risk of residual disease. In our experiment, virtual-reality assisted surgery planning was performed in addition to the routine workflow.

Multiparametric MRI segmentation and model rendering

All the processing steps were done 1–2 days prior to surgery. Axial, coronal, and sagittal T2-weighted MRIs were exported as Digital Imaging and Communications in Medicine files using the Carestream client Picture Archiving and Communication System software (Carestream, Rochester, New York, USA). All anatomic structures in the field of view including the whole prostate, MRI-detected tumors, urethra, neurovascular bundles, seminal vesicles, bladder, and rectum were manually segmented and saved as separate files in VOI format using in-house segmentation software (Figure 1).^[12] We created 3-D mesh models using novel computer vision algorithms to perform shape reconstruction, shape modeling and registration, and generate data files, which were then used to render 3-D models using the Unreal Engine version 4 software.^[8,13–16] The 3D models were reconstructed by the segmentation process by first fitting a hybrid shape model to the segmented contours, and then using the model parameters to render a 3-D mesh of desired resolution.^[8,14] All structures were color-coded and visualized either as 3-D wireframes or full 3-D mesh surfaces. After the first two experiments, the surgeon suggested that we may render the prostate, urethra, tumors, seminal vesicles, and neurovascular bundles as 3-D mesh surfaces (without transparency) and the remaining structures as 3-D wireframes, since this approach was considered the most optimal method for surgery planning (Figure 2). The application also allowed us to record short-stereo laparoscopic video clips from the surgery, use the stereo images to perform 3-D reconstruction of the scene, align it to the 3-D models reconstructed from preoperative MRI data, and visualize them together (Figure 3 and 4).^[8,16]

Figure 1.

Manual segmentation of the prostate boundary and critical anatomic structures around the prostate on T2-weighted images using in-house segmentation software. All the following images were derived from the same experiment

Figure 2.

Three-dimensional models based on manual segmentations. Both images represent the same model from two different perspectives. The tumor detected on MRI shows aggressive features with potential extraprostatic extension and invasion of the left neurovascular bundles

MRI: magnetic resonance imaging

Figure 3.

In situ view of the surgeon on the anterior aspect of the prostate during robotic-assisted radical prostatectomy. The stereo cameras of the da Vinci surgical system are used to record 3D stereo videos for rendering the 3D models of the in vivo prostate

Figure 4.

Superimposed 3D model of in vivo prostate on the MRI-derived 3D model using an automatic alignment algorithm from three different perspectives

VR: virtual reality; MRI: magnetic resonance imaging

Experimental setup and performance

A recording system was connected to the stereo Digital Visual Interface output ports of the da Vinci surgical system using Startech^® Universal Serial Bus (USB) 3.0 video capture devices, which allowed us to record short uncompressed, synchronized stereo videos at 60 frames per second. Two different machines were used in developing this application, we will call these the (i) imaging platform, which featured a Intel^® quad-core i7 4710HQ with 24 gigabytes of random-access memory (RAM) and (ii) a visualization platform, which featured a quad-core i7 6700K with 32 GB of RAM and an NVIDIA^® GTX 980 GPU. The imaging platform was used for capturing images, 3-D stereo scene reconstruction, shape fitting, and alignment. The visualization platform was strictly used for running the VR application.

Another computer was used for processing the images to generate mesh models, which has Intel^®’s I7–7700 central processing unit processor, a 64-bit quad-core high-end microprocessor. Our system utilizes novel computer vision techniques to fuse preoperative data and intraoperative data for live VR visualization during the surgery. Pre-segmented MRI data was used to fit hybrid geometric models (Extended Superquadrics and Radial Basis functions) and stereo reconstructions of the operative field was performed from the left and right imagery of the da Vinci^® surgical system. We then registered the preoperative mesh models with the intraoperative reconstruction and presented it to the surgeons in VR for assistance in decision-making. Surgeons were given an option to correct and fine-tune the alignment between our data models. Further details of these processes are detailed in a prior publication.^[14]

Histopathologic evaluation

Prostatectomy specimens were fixed in formalin for 2–24 hours at room temperature, embedded in paraffin, and sliced in axial 6 mm sections using a customized 3-D-printed mold. Final hematoxylin and eosin sections were 5 μm in thickness and were evaluated by a single genitourinary pathologist (M.J.M.) (>20 years of experience) blinded to the MRI results. The tumor burden was assessed by the TNM classification system after applying the 2014 International Society of Urological Pathology modifications of the Gleason grading system.^[17]

Oncological and functional outcomes

Although this is merely a feasibility study without statistical powering, we gathered surgical and follow-up data to create a preliminary impression of the patients’ outcomes. Since RARP is not an organ-sparing approach, the impact of preoperative imaging on the surgical technique is limited. Nevertheless, in some instances, imaging information can impact decision-making during surgery. In patients with a locally aggressive growth pattern visible on a multiparametric MRI, surgeons can undertake a more radical approach by performing a wider excision around the tumor. This has the effect of leaving more tissue on the tumor and reducing the risk of positive surgical margins (Figure 5). A VR-assisted navigation system based on multiparametric MRI could therefore aid the surgeon to correctly identify these spots and improve surgical planning and decision-making during the operation, which may lead to potential improvements in the oncological outcomes. Surgical, histopathological, and functional outcomes at 12 months after prostatectomy were evaluated.

Figure 5.

Posterior aspect of the prostatectomy specimen. Based on the findings on MRI, we decided to perform a wide excision in the area of the tumor, since extraprostatic extension and potential infiltration of the neurovascular bundles were suspected

Results

The VR method was applied without complications during all 10 planned surgery procedures. Mean blood loss during the prostatectomies was 318 ml (median: 263 ml, range: 50–600 ml). Surgical and histopathology outcomes are summarized in Table 3. Oncological and functional outcomes are summarized in Table 4. Among the five patients with MRI findings that were suspicious for extraprostatic extension on preoperative staging prostate MRI, only one patient had positive margins at surgical pathology evaluation and experienced biochemical recurrence after 12 months of follow-up. This patient initially had a locally aggressive disease with high PSA, two large PI-RADSv2 category 5 lesions that were confirmed to include Gleason 4+4 with tertiary Gleason pattern 5, and extraprostatic extension on final histopathology. The rest of the 9 patients have remained recurrence-free to date. All except one patient were continent (0–1 pads) after surgery. This patient had a Gleason 4+3 prostate cancer with a 19 mm PIRADSv2 category 5 lesion and extraprostatic extension seen at the preoperative prostate MRI. A wide excision was mandatory to ensure optimal cancer control. Surgical pathology revealed locally aggressive disease with Gleason 4+4, tertiary Gleason pattern 5, and seminal vesicle invasion in this patient, however the patient is still recurrence-free at 12 months of follow-up.

Table 3.

Surgical and pathological outcomes

Patient	Blood loss (mL)	Prostate weight (g)	TNM	Gleason score at surgical pathology
1	400	29.5	T3bN0Rx	4+4 (tertiary 5)
2	200	54	T2cN0R0	3+4
3	225	64	T2cN0R0	4+4 (tertiary 5)
4	200	32.6	T2aN0R0	3+4
5	50	32.7	T0N0R0	No cancer, treatment effect
6	500	43.6	T2cN0R0	3+4
7	500	41.5	T3aN0R1	4+4 (tertiary 5)
8	200	55	T0N0R0	No cancer, treatment effect
9	300	46.5	T2cN0R0	4+4
10	600	57.5	T2cN0R0	3+4

Table 4.

Oncological and functional outcomes after surgery

Patient	Postoperative PSA (ng/mL)	Post-op follow-up (months)	IPSS	QoL	SHIM score	Continence (number of pads)
1	<0.02	16	4	1	1	3
2	<0.02	4	13	2	0	1
3	<0.02	15	1	1	16	0
4	<0.02	14	10	3	10	0–1
5	<0.02	16	7.5	3	13	1
6	<0.02	12	4	1	22	0
7	0.14	12	N/A	N/A	N/A	1
8	<0.02	13	2	1	1	0
9	<0.02	N/A	N/A	N/A	N/A	N/A
10	<0.02	9	3	6	6	0

IPSS: International Prostate Symptom Score; QoL: Quality of Life; SHIM: Sexual Health Inventory for Men

Discussion

Currently, surgeons are required to analyze patient scans preoperatively and cognitively and fuse all imaging information into a 3-D picture. In this study, we present our first experience with a VR-assisted surgical navigation system during robotic-assisted prostatectomies. The current study is a proof-of-concept study and is the first step toward developing a real-time surgical navigation system. First, we exported the imaging data and segmented the anatomic structures manually. The 3-D VR models were then rendered via a gaming-based 3-D framework. The models were loaded into a commercially available VR head-mounted display and could be interacted with by using handheld controllers. This process took place in a side room of the operating room close to the da Vinci console. Our surgeon decided when to withdraw from the console and when to use the VR system. This typically occurred at critical steps of the procedure such as bladder neck preparation, apical resection, dissection of the prostate pedicles, and especially during nerve-sparing. In a next step, we extracted 3-D stereo video recordings and rendered an in vivo prostate 3-D VR model, which was automatically superimposed with the MRI model by using a dedicated tool based on an alignment algorithm. Our staff had the impression that the alignment was accurate although this could not be determined objectively without a measurable ground truth.

The VR system was applied during 10 RARPs so far. Every case was discussed intensively with the surgical team during and after the experiment. The system was particularly useful in cases with locally advanced prostate cancer. In these patients, the system could improve decision-making and the dissection strategy to prevent residual cancer and potentially decrease recurrence rates. In patients with a less advanced stage of the disease, the system might not be as useful since the scans can also be evaluated preoperatively, with tumors not approaching the capsule or adjacent structures, the prostatectomy is performed in a standardized manner.

This study only encompasses a small number of patients and was designed to test the feasibility of the preliminary VR system and explore its potential and possible limitations. We did not apply any statistical powering for oncological and functional outcomes. Nevertheless, we presented all follow-up data we could gather to this point for the sake of completeness. Although our surgeon interacted regularly with the models during the procedure, no real intervention was done since all prostatectomies were done adhering to our standard workflow. The results had to be interpreted in this context.

There are still several challenges that need to be addressed before the full real-time application of this technique. First, the in vivo imaging data segmentation is currently done manually one day prior to the surgery. Second, model rendering of the in vivo prostate and the alignment need to be processed faster to enable real-time interaction during the surgery. Third, the final goal is to have a fully implemented system that can render 3-D models with real-time superimposition and transmit them to the da Vinci stereo viewer independent of an external head-mounted display.

Figure 6 illustrates the first step into this direction. Nevertheless, although the 3-D models could be transferred into the da Vinci^® system and the surgeon was able to see them through the da Vinci^® stereo viewer, interaction with the models through the proprietary controlling system was not possible. This had to be done by another person on a laptop or by using the handheld controllers of the Oculus Rift^® platform.

Figure 6.

First successful transfer of the MRI and in vivo 3D models into the da Vinci stereo viewer in another patient. The surgeon could see both models and the in situ view simultaneously but was currently not able to use the da Vinci control systems to interact with the model

In conclusion, our study demonstrates the feasibility of interactive 3-D visualization of prostate MRI data during in vivo RARP. Future goals following this research include the use of the robot’s stereo image viewer instead of the head-mounted display and advancement of the MRI-stereo image alignment to facilitate direct intraoperative use of MRI.

Supplementary Materials

Supplemental Table 1.

MRI acquisition parameters of our institution

Endorectal coil	with ERC			Without ERC
Parameters	T2W TSE	DWIa	DCE	T2W TSE	DWIb	DCE
Field of view (mm)	140×140	140×140	262×262	180×180	140×140	262×262
Acquisition matrix	304×234	76×78	188×96	320×216	64×62	176×66
Repetition time (ms)	4434	6987	3.7	3686	7218	3.7
Echo time (ms)	120	52	2.3	120	47	2.3
Flip angle (degrees)	90	90	8.5	90	90	8.5
Section thickness (mm)	3	3	3	3	3	3
Image reconstruction matrix (pixels)	512×512	256×256	256×256	512×512	256×256	256×256
Reconstruction voxel imaging resolution (mm/pixel)	0.27×0.27×3	0.55×0.55×2.73	1.02×1.02×3	0.35×0.35×3	1.09×1.09×3	1.02×1.02×3
Time for acquisition (min:s)	2:48	3:50	5:16	4:48	6:08	5:16

ERC: endorectal coil; TSE: turbo spin echo; DWI: diffusion weighted imaging; DCE: dynamic contrast-enhanced imaging

^aFor b=2000 s/mm²;

^bFor b=1500 s/mm²