Neurosurgical Virtual Reality Simulation for Brain Tumor Using High-definition Computer Graphics: A Review of the Literature

Abstract

Simulation and planning of surgery using a virtual reality model is becoming common with advances in computer technology. In this study, we conducted a literature search to find trends in virtual simulation of surgery for brain tumors. A MEDLINE search for “neurosurgery AND (simulation OR virtual reality)” retrieved a total of 1,298 articles published in the past 10 years. After eliminating studies designed solely for education and training purposes, 28 articles about the clinical application remained. The finding that the vast majority of the articles were about education and training rather than clinical applications suggests that several issues need be addressed for clinical application of surgical simulation. In addition, 10 of the 28 articles were from Japanese groups. In general, the 28 articles demonstrated clinical benefits of virtual surgical simulation. Simulation was particularly useful in better understanding complicated spatial relations of anatomical landmarks and in examining surgical approaches. In some studies, Virtual reality models were used on either surgical navigation system or augmented reality technology, which projects virtual reality images onto the operating field. Reported problems were difficulties in standardized, objective evaluation of surgical simulation systems; inability to respond to tissue deformation caused by surgical maneuvers; absence of the system functionality to reflect features of tissue (e.g., hardness and adhesion); and many problems with image processing. The amount of description about image processing tended to be insufficient, indicating that the level of evidence, risk of bias, precision, and reproducibility need to be addressed for further advances and ultimately for full clinical application.

Introduction

Within 10 years since Computed tomography (CT) and magnetic resonance imaging (MRI) became clinically applied, brain three-dimensional (3D) image reconstructed by medical image were reported in the late 1980s.¹⁾ In the 1990s, Many papers about neurosurgical virtual reality (VR) simulation using 3D images were published.^2–6) VR technology and computer graphics technology have drastically advanced with the rapid increases in computer processing speed over the past 10 years. Application of VR technology in medical care is becoming prominent in both research and clinical settings, and also occupies an important position commercially. In neurosurgery, the introduction of three-dimensional fusion imaging, in which multiple types of medical imaging datasets are fused, has boosted the development of virtual three-dimensional surgery enabling interactive evaluation of surgical procedures (e.g., craniotomy), as well as easy acquisition of detailed anatomical information in the expected surgical field. This is revolutionary because the information that used to be in clinicians’ minds can now be visualized reproducibly, and further, is shared with other individuals. Surgical simulation is noninvasive and repeatable, and thus, is eagerly anticipated in education and training as well as in clinical application. However, the combination of multiple types of advanced complex image processing technologies for virtual surgery tends to be a source of confusion for neurosurgeons, and also tends to make procedures overly complex. Furthermore, objective and quantitative aspects need to be addressed in evaluations of simulations. This study reviewed recent trends in virtual surgery for brain tumors.

Methods

Articles published during an approximately 10-year period (January 2006 and May 2016) were searched in PubMed (https://www.ncbi.nlm.nih.gov/pubmed) using the following keyword combination: neurosurgery AND (simulation OR virtual reality). A total of 1,298 articles were initially found. Articles with solely educational and training objectives and those not about a brain tumor were eliminated. Review articles were also eliminated, leaving original articles, technical notes, and case reports describing neurosurgical simulation of brain tumor in clinical medicine. The criteria of surgical simulation in this study were as follows: use of three-dimensional medical images; image processing for surgical simulation or planning, but not for diagnosis; use of interactive functions (excluding “move”, “rotate” and “zoom”) of a VR model. We described the research trends by using key articles as follows.

Results and Discussion

Among 1,298 hits from the search with the keyword combination (neurosurgery AND (simulation OR virtual reality)), 28 described the clinical application of simulation of neurosurgery for brain tumor (Table 1). Of these articles, 19 were original articles and 8 were technical notes/case reports. It is generally not easy to clearly distinguish surgical simulation from surgical planning. In this study, the surgical simulation was defined as the use of three-dimensional images that reflected procedures with surgical devices, and movement and changes in tissues, while the surgical planning was defined as therapy strategy-making activities based on image findings, in particular with use of preoperative three-dimensional medical imaging. In accordance with these definitions, selected articles were about surgery planning, rather than surgical simulation. Many of the 1,298 articles described the development of surgical simulators for educational and training purposes, but only 28 were about clinical application, suggesting that certain challenges remain to be tackled in the clinical application of surgical simulation. One of them is difficulty in clinical evaluation: A direct connections between the use of surgical simulation and clinical outcome is difficult to prove. Furthermore, objective and quantitative assessment of surgical simulation is challenging. Such difficulties are likely attributable to the nature of surgical simulation and planning, namely, visualization of a surgeon’s mind, and the inability of current image processing technology to visualize pieces of information that are not depicted by medical imaging (e.g., detailed and complicated surgical findings and maneuvers). Indeed, for this review it was hard to select articles in a systematic and well-defined manner to obtain highly reliable knowledge. This hindered identification, selection, and review of reference articles, as well as systematic integration of characteristics and results of selected articles. Nevertheless, research trends in several areas were identified as described below.

Table 1.

Article summary

No	Author	Year	Diagnosis	Number of patients	Analysis software	Summary
1	Ya Li, et al.12)	2015	Suprasellar arachnoid cyst	36	3D Slicer	The feasibility of virtual ensoscopy as a rapid, low-cost, and interactive modality for surgical planning of suprasellar arachnoid cysts.
2	Akihiro Inoue, et al.13)	2015	Pituitary adenoma	99	3D Advantage Workstation Volume Share 4	The 3D image for endoscopic endonasal transphenoidal surgery enabled to obtain a pertinent orientation compared with 2D images.
3	Leila Besharati Tabrizi, et al.24)	2015	Various brain tumors	10	MRIcro	The AR system is accurate and reliable for the intraoperative projection of images to the head.
4	Satoshi Takahashi, et al.31)	2014	Cavernous malformation	2	Ziostation	The fusion model offers the ability to simulate surgical strategy and to utilize brain gyri and sulci as surgical landmarks.
5	Keisuke Maruyama, et al.28)	2014	Brain tumor and others	12	Amira	Using an iPad to handle the result of preneurosurgical simulation was useful because it could easily be handled anywhere.
6	Masanori Yoshino, et al.11)	2013	Cerebellopontine angle meningioma	8	Avizo	The 3D image could estimate the the main feeder and an appropriate route for resection of it.
7	Makoto Oishi, et al.22)	2013	Skull base tumor	23	Zed-View	The VR simulation with a haptic device and the printed model provided a realistic environment for presurgical simulation.
8	Axel T Stadie, et al.23)	2013	Brain tumor and others	241	Dextroscope	The 3D images improved anatomic understanding, refined surgical strategies, and improved intraoperative confidence.
9	Tang Hai-liang, et al.29)	2012	Meningioma	30	Dextroscope	The VR planning could give more anatomic information about meningioma and its surrounding structures.
10	Taichi Kin, et al.10)	2012	Brainstem cavernous malformation	10	Avizo	The 3D images could depict the developmental venous anomaly associated with brainstem cavernous malformation better than 2D images.
11	Yan Zhao, et al.20)	2012	Glioma	20	VectorVision Sky navigation system	The arcuate fiber data could be reliably integrated into a neuronavigation system.
12	Shou-sen Wang, et al.18)	2012	Sellar tumor	66	Dextroscope	The 3D image was useful for individualized preoperative planning for surgery in the sellar region.
13	Ayguel Mert, et al.7)	2012	Various brain tumors	77	StealthViz	The 3D brain surface visualization is clinically reliable for preoperative planning and improves intraoperative orientation with navigation.
14	Makoto Oishi, et al.21)	2011	Skull base tumor	32	Real INTAGE	The intraoperative déjà-vu effect of the simulation increased the confidence of the surgeon in the planned surgical procedures.
15	Taichi Kin, et al.9)	2011	Brain tumor and others	10	Avizo	The fusion model represented an improved visualization method for preoperative virtual simulation for neuroendoscopic surgery.
16	Makoto Oishi, et al.14)	2011	Skull base tumor	20	Real INTAGE	The 3D image could facilitate determination of the most appropriate approach and precise surgical procedures for skull base tumors.
17	Axel T. Stadie, et al.27)	2011	Various brain tumors	48	Dextroscope	In cases of neuronavigation failure, the VR surgery planning serves as an effective backup system to perform a minimally invasive operation.
18	Mario Giordano, et al.34)	2011	Pineal tumor	33	CBYON Suite	The architecture of the pineal veins and their anatomical relationship could be depicted with accuracy by 3D neuronavigation software.
19	Yohei Bamba, et al.15)	2011	Spinal lipoma	19	iPlan	The preoperative planning for complicated spinal repairs using iPlan.
20	Shi-Xin Gu, et al.32)	2011	Brain tumor	25	Dextroscope	The VR technology could accurately simulate the anatomical feature of the temporal bridging veins for individual operations.
21	Paolo Ferroli, et al.8)	2010	Brain tumor and others	100	Dextroscope	The VR could facilitate the understanding of the surgical anatomy and effectively simulate in stereoscopic neurosurgery.
22	Tian-ming Qiu, et al.30)	2010	Glioma	45	Dextroscope	The 3D stereoscopic visualization with tractography enhanced the operators to well understand the anatomic information of intra-axial tumor.
23	David Low, et al.19)	2010	Meningioma	8	Dextroscope	The stereoscopic 3D imaging visualized and gave better spatial appreciation of vein. The AR technology enabled for image guidance.
24	Florian Schulze, et al.16)	2010	pituitary adenoma	12	STEPS	Intra-operative virtual endoscopy provided additional anatomical information to the surgeon.
25	Kathleen Joy Khu, et al.33)	2009	parasagittal and falcine meningioma	16	Dextroscope	The number, size and disposition of the cortical veins in relation to meningiomas were determined using VR technology.
26	Axel Thomas Stadie, et al.17)	2008	brain tumor, others	106	Dextroscope	3D VR models of a patient allowed quick and easy understanding of complex intracranial lesions.
27	S.M.Anil, et al.25)	2007	4th ventricular ependymoma	1	Dextroscope	The integration of 3D imaging with stereoscopic vision made understanding the complex anatomy easier and improved surgical decision making.
28	Eduardo E. Lovo, et al.26)	2007	Brain tumor and others	8	MRIcro	The anatomic definition of the 3D MRI and the AR technique is sufficient to make a proper coregistration in the operative setting.

2D: two-dimensional, 3D: three-dimensional, AR: augmented reality, MRI: magnetic resonance imaging, VR: virtual reality.

Image processing

Many articles did not provide detailed information about image processing. This is a problem as the precision and validity of VR modeling, which generates source materials for surgical simulation, were unavailable. Descriptions of methods were often vague, and this may hinder evaluation of the precision and validity of surgical simulation.

Original image data:

Three-dimensional images acquired from multiple sequences or multimodal fusion were used for surgical simulation in most articles. Image fusion was necessary because a series of surgical maneuvers, from craniotomy and approach making to microscopic observation, need to be simulated.^7,8) Also, it achieves better differentiation of organs.⁹⁾ Selection of original image data is crucial.^10,11) For example, artifacts such as flow void sometimes prevents clear model construction.¹²⁾ However, computed tomography angiography is reported to be sufficient for understanding the area surrounding the anatomically complex sphenoid sinus.¹³⁾ The level of spatial resolution of three-dimensional fused images depends on the resolution of original image data,¹⁴⁾ and so thin-slice high-resolution images should be selected for three-dimensional image fusion. Mert et al. reported that 79% of blood vessels (0.5–1.0 mm) were depicted in medical images of 512 × 512 pixels with 1-mm slice thickness.⁷⁾

Registration:

Image registration, which is a process of transforming different sets of data into one coordinate system, is necessary for integrating different medical image datasets for three-dimensional visualization. Some articles specified that normalized mutual information were used for registration;^9–11) however, many articles simply mentioned “auto-registration” or did not give any information about the method used for registration. Misalignment of the same organs among images of different modalities¹⁰⁾ as well as manual registration due to poor results of auto-registration¹⁵⁾ were reported, suggesting that the accuracy of registration remains to be improved.

Segmentation:

Image segmentation extracts regions of interest. The techniques used in the selected articles varied widely, depending on the software used and on the institutions where studies were conducted.^9,16–18) Automatic segmentation, such as the region-growing method, enables easy extraction of tissue images, but not for micro tissues.¹⁹⁾ Thus, automatic, semi-automatic, and manual methods were combined to extract both large tissues (automatically) and micro tissues (manually).^{15–17,19,20)} Using different combinations of original data according to the tissues of interest has become increasingly popular in recent years.^9,11,18,21)

Rendering:

Conventional medical three-dimensional images were often made by a volume rendering method. This method has been employed in the past for constructing fused three-dimensional images. However, the visibility of overlaid images rendered by the volume method tends to drop when many image datasets are used for image fusion for surgical simulation. This increases the number of parameters that need to be set by users. Further, virtual surgery requires clear visualization of micro-tissues and mesh editing of changes in the brain shape. Surface rendering methods have been introduced in recent years to achieve these needs. Indeed, 7 of 28 selected articles used a surface rendering method. Surface rendering relies on polygonal modeling, and is the main method used for generating the computer graphics used in computer games and movies. Thus, a considerable number of image analysis software packages, allowing various types of image processing, are available. Some articles used both surface rendering and volume rendering technologies.^9,16)

Analysis software:

Many different analysis software packages were applied to surgical simulation (Table 1), and their functions and usability levels varied enormously. Their prices, too, ranged, from free to very expensive. Additionally, much of the software was limited to research purposes only.

VR model construction time:

The time required for constructing VR models largely depends on the types of models to be constructed and is an important factor for clinical applications. As exemplified in the study of preoperative simulation for a suprasellar arachnoid cyst, a virtual endoscopy model requires a short period of time for its construction, and thus, was useful,¹²⁾ suggesting that disease- or purpose-specific modeling may be a valid approach. The time spent for image preparation varied, from approximately 10 min to 6 h among articles selected in this study.^{9,12,14,15,17,22)} Stadie et al. prepared VR models of 208 patients and found that average VR model preparation time decreased as the number of cases increased. For example, the time needed to prepare one model was reduced from 2 h to 1 h when the number of cases was increased to 100, indicating that constructing VR models is feasible as a part of clinical daily work.^17,23)

Manipulations in simulation:

Many of the articles reviewed in this study employed preoperative three-dimensional image fusion to integrate the available information into a surgery plan. Thus, they were three-dimensional image-based planning, rather than simulation. However, 14 articles used a function that enabled the removal of bones during simulation. Despite this, only a few articles reported modification in the brain shape,^11,22) and only one article, about virtual septostomy, demonstrated a simulation that reflected other surgical maneuvers.⁹⁾ Taken together, it is challenging at present to create virtual spaces that can reflect surgical maneuvers. Manipulations, such as moves and modification of tissues, place a considerable load on maintaining physical consistency and increase the computational load. This is a problem to address in the future.

Integration with other devices or techniques:

Virtual reality is all about the creation of a virtual world that users can interact with. On the other hand, augmented reality (AR) is the blending of virtual reality and the real world. Integration of VR models with various hardware devices, rather than a simple display of VR models on the screen, was also tested in surgical simulation. Many of those cases were integration with either a surgical navigation system^7,16) or AR technology, which projects VR images onto the operating field.^19,24–26) These integrations were used for planning in the surgical theater and for referring to intraoperative findings. Mert et al. reported 3D visualization of brain surface anatomy when added to standard navigation, is useful for intraoperative orientation.⁷⁾ A study comparing errors in the incision position for craniotomy found comparable results between use of a VR model and use of a surgical navigation system.²⁷⁾ Maruyama et al. used tablet computer devices to view three-dimensional VR models in an operating room.²⁸⁾ Additionally, a three-dimensional printed model and a VR model²²⁾ and an autostereoscopic display technology²³⁾ were combined for preoperative examination.

Clinical assessment

Clinical assessments of surgical simulation and planning were dominantly qualitative, and quantitative assessments were made by questionnaire or based on pre-established research questions and hypotheses about diagnosis and technical precision. Because of the vague descriptions of image processing, discussed earlier in this article, the clarity and reproducibility of each method was difficult to examine. When methods and purposes are unclear, it is also hard to assess their bias risks. Nevertheless, the clinical benefit is discussed below, with reference to key studies in the literature.

Qualitative benefit:

Many articles generally reported clinical benefit from virtual simulation of surgery. VR modeling and particularly three-dimensional image fusion is useful in examining surgery because it allows simultaneous viewing of multiple sequences of different modalities.¹¹⁾ Complex spatial relationships between a tumor and surrounding tissues can be observed in detail by use of a VR model,^22,29) and orientation of lesions in the floor of fourth ventricle, which is difficult to inspect from a sectional image,¹⁰⁾ was significantly better in three-dimensional images than in two-dimensional images.⁷⁾ VR modeling is especially advantageous in spatial depiction for microsurgery,²²⁾ including patient-specific anatomical information different from normal anatomy, such as tumor invasion and damage to normal tissues.¹⁷⁾ As well, VR models can provide the same field of operation,¹¹⁾ and can thus support selection of a surgical approach¹⁰⁾ and contribute to improvement of the precision of surgical planning and to reduction of surgical invasiveness.¹⁷⁾ VR modeling integrated with functional imaging, such as tractography, improves safety of surgery,^20,30) and is useful in tumorectomy adjacent to the eloquent area because the relationship of white matter fibers to the cerebral gyrus and sulcus can be viewed.³¹⁾ Regarding virtual surgical maneuvers, use of VR was helpful in preoperative examination of septostomy maneuvers,⁹⁾ and trial-and-error bone removal, which can be done in simulation, is beneficial in skull base surgery.¹⁴⁾ Additionally, déjà vu experience during actual surgery, attributed to three-dimensional simulation, was thought to be useful.²¹⁾

Quantitative benefit:

Several approaches were employed to assess surgical simulation in a quantitative manner. In the field of diagnostic imaging, Yoshino et al. reported that three-dimensional fused images were more useful than two-dimensional images in understanding the region of the most vascularized attachment of the cerebellopontine angle meningioma.¹¹⁾ Kin et al. reported that the rate of diagnosis of developmental venous anomaly complicated with brainstem cavernous malformation was improved by use of three-dimensional fused images created from carefully selected raw image sets.¹⁰⁾ Regarding visibility, Kin et al. listed the anatomical landmarks necessary for neuroendoscopic surgery in each patient and found that 97.5% of such structures were successfully depicted in three-dimensional images.⁹⁾ Ye Li et al. demonstrated that VR models depicted the anatomical landmarks necessary for the surgery of a suprasellar arachnoid cyst.¹²⁾ Some articles examined visibility from an anatomical perspective.³²⁾ In particular, Joy et al. reported that the number and diameter of the veins around a meningioma were comparable with those in the control group.³³⁾

Quantitative performance of surgical simulation or planning was also examined. Oishi et al. conducted a questionnaire survey and reported that a 3DCG approach was much more useful than a two-dimensional approach in 44% of cases of surgical dissection and strategizing, and facilitated surgery proceeding according to plan in many cases.²²⁾ Oishi et al. also examined surgery plans made with either two-dimensional images or three-dimensional fused images, and found that two-dimensional images were not enough to make a concrete surgery plan in 4 out of 21 cases, while three-dimensional images enabled accurate selection of a surgical approach.¹⁴⁾ Maruyama et al. focused on usability and employed a three-dimensional viewer on an iPad tablet to test how many repetitions were required to execute the required procedure within 30 s, as performed by 12 students and 6 neurosurgery residents.²⁸⁾

No articles discussed surgical simulation in relation to outcomes of surgery.

Tasks:

Oishi et al. reported that there were cases of uncompleted surgery due to bleeding and of incomplete removal of a very hard tumor, which was not improved by the use of VR models. This suggests that surgical simulation cannot reflect deformation and moves of tissues attributed to surgical manipulation.¹⁴⁾ Along the same lines, Wang et al. assessed surgical simulation using three-dimensional fused images, and found that 9 of 11 surgeons required additional medical images, indicating that VR modeling alone is not a sufficient tool in surgical simulation.¹⁸⁾ Giordano et al. examined veins around a pineal tumor on three-dimensional images in 33 cases, finding that the resolution of VR models was fine enough to depict only medium-sized veins, suggesting problems in the resolution of the original image data and in three-dimensional image processing.³⁴⁾

Future Prospects

Clinical applications of surgical simulation through integration with tools that aid diagnosis and surgery (e.g., flow analysis data and intraoperative indocyanine green video angiography), application to noninvasive therapy (e.g., stereotaxic radiosurgery), and fusion with the real space by augmented- and mixed-reality technologies are ongoing. The processes of three-dimensional image fusion have not yet been standardized, although this indicated the potential of case-specific image processing, possibly leading to personalized medicine. Related technologies, such as surgical navigation, fusion with surgical microscopy, haptic technologies, and three-dimensional printing, were combined with surgical simulation in clinical cases in several studies, and further advances are expected. It is probable that good use of VR and computer graphics technologies will enhance three-dimensional image fusion and related technologies.

Conclusion

Neurosurgical virtual reality simulation for brain tumor was useful in better understanding complicated spatial relations of anatomical landmarks and in examining surgical approaches. The problems were difficulties in standardized, objective evaluation of surgical simulation systems, inability to respond to tissue deformation caused by surgical maneuvers, and absence of the system functionality to reflect features of tissue such as hardness and adhesion. The description about VR image processing tended to be insufficient, indicating that the level of evidence, risk of bias, precision, and reproducibility need to be addressed for further advances and ultimately for full clinical application.