Combining intraoperative ultrasound brain shift correction and augmented reality visualizations: a pilot study of eight cases

Abstract.

We present our work investigating the feasibility of combining intraoperative ultrasound for brain shift correction and augmented reality (AR) visualization for intraoperative interpretation of patient-specific models in image-guided neurosurgery (IGNS) of brain tumors. We combine two imaging technologies for image-guided brain tumor neurosurgery. Throughout surgical interventions, AR was used to assess different surgical strategies using three-dimensional (3-D) patient-specific models of the patient’s cortex, vasculature, and lesion. Ultrasound imaging was acquired intraoperatively, and preoperative images and models were registered to the intraoperative data. The quality and reliability of the AR views were evaluated with both qualitative and quantitative metrics. A pilot study of eight patients demonstrates the feasible combination of these two technologies and their complementary features. In each case, the AR visualizations enabled the surgeon to accurately visualize the anatomy and pathology of interest for an extended period of the intervention. Inaccuracies associated with misregistration, brain shift, and AR were improved in all cases. These results demonstrate the potential of combining ultrasound-based registration with AR to become a useful tool for neurosurgeons to improve intraoperative patient-specific planning by improving the understanding of complex 3-D medical imaging data and prolonging the reliable use of IGNS.

1. Introduction

Each year thousands of Canadians undergo neurosurgery for resections of lesions in close proximity to areas of the brain that are critical to movement, vision, sensation, or language. There is strong support in the literature demonstrating significantly increased survival benefit with complete resection of primary and secondary brain tumors,¹ creating competing constraints that must be balanced during surgery for each patient: “achieving maximal resection of the lesions while causing minimal neurological deficit.”

Since the introduction of the first intraoperative frameless stereotactic navigation device by Roberts et al.² in 1986, image-guided neurosurgery (IGNS), or “neuronavigation,” has become an essential tool for many neurosurgical procedures due to its ability to minimize surgical trauma by allowing for the precise localization of surgical targets. For many of these interventions, preoperative planning is done on these IGNS systems that provide the surgeon with tools to visualize, interpret, and navigate through patient-specific volumes of anatomical, vascular, and functional information while investigating their interrelationships. Over the past 30 years, the growth of this technology has enabled application to increasingly complicated interventions, including the surgical treatment of malignant tumors, neurovascular disorders, epilepsy, and deep brain stimulation. The integration of preoperative image information into a comprehensive patient-specific model enables surgeons to preoperatively evaluate the risks involved and defines the most appropriate surgical strategy. Perhaps more importantly, such systems enable surgery of previously inoperable cases by facilitating safe surgical corridors through IGNS-identified noncritical areas.

For intraoperative use, IGNS systems must relate the physical location of a patient with the preoperative models by means of a transformation that relates the two through a patient-to-image mapping (Fig. 1). By tracking the patient and a set of specialized surgical tools, this mapping allows a surgeon to point to a specific location on the patient and see the corresponding anatomy in the preoperative images and the patient-specific models. However, throughout the intervention, hardware movement, an imperfect patient–image mapping, and movement of brain tissue during surgery invalidate the patient-to-image mapping.³ These sources of inaccuracy, collectively described as “brain shift,” reduce the effectiveness of using preoperative patient-specific models intraoperatively. Unsurprisingly, most surgeons use IGNS systems to plan an approach to a surgical target but understandably no longer rely on the system throughout the entirety of an operation when accuracy is compromised and medical image interpretation is encumbered. Recent advances in IGNS technology have resulted in intraoperative imaging and registration techniques^4,5 to help update preoperative images and maintain accuracy. Advances in visualization have introduced augmented reality (AR) techniques^6–9 at different time points and for different tasks to help improve the understanding and visualization of complex medical imaging data and models and to help with intraoperative planning. We present a pilot study of eight cases combining the use of intraoperative ultrasound (iUS), for brain shift correction, and intraoperative AR visualization with traditional IGNS tools to improve intraoperative accuracy and interpretation of patient-specific neurosurgical models in the context of IGNS of tumors. While other groups have investigated iUS and AR independently, there are very few reports^10–14 of using both technologies to overcome the visualization issues related with iUS and the accuracy issues related to AR. The goal of this pilot study is to investigate the feasibility of combining iUS-based brain shift correction and AR visualizations to improve both the accuracy and interpretation of complex intraoperative data. Our work aims to improve on some of the limitations of the previous work by being a prospective instead of retrospective clinical pilot study,¹² being focused on evaluation in clinical scenarios as opposed to phantoms or animal cadavers,¹¹ having high-quality MRI images for segmentations instead of from difficult to interpret US images,^12,14 using patient-specific data instead of atlas-based data for greater registration accuracy,¹⁴ and finally, using a fast US-MRI registration that allows for an efficient workflow incorporating AR in the operating room (OR) that consumes less time and provides more information than previous reports.^12,13

Fig. 1.

The patient’s head is positioned, immobilized, and a tracked reference frame is attached. The patient’s preoperative images and physical space are registered by using eight corresponding landmarks on the head and face to create a correspondence between the two spaces.

1.1. iUS in Neurosurgery

Intraoperative imaging has seen a wide range of use in neurosurgery over the last two decades. Its main benefit constitutes the ability to visualize the up-to-date anatomy of a patient during an intervention. iUS has been proposed and used as an alternative to intraoperative MRI due to its ease of use, low-cost, and widespread availability.⁵ iUS is relatively inexpensive and noninvasive and does not require many changes to the operating room or surgical procedures. However, its main challenges are associated with relating information to preoperative images, which are generally of a different modality. The alignment of iUS to MRI images is a challenging task due to the widely different nature and quality of the two modalities. While voxel intensity of both modalities is directly dependent on tissue type, US has an additional dependence on probe orientation and depth that can lead to intensity nonuniformity due to the presence of acoustic impedance transitions. Preoperative MR images allow for identification of tissue types, anatomical structures, and a variety of pathologies, such as cancerous tumors. iUS images are generally limited to displaying lesion tissue with an associated uncertainty regarding its boundary, along with a few coarsely depicted structure boundaries. Early reports using iUS in neurosurgery, such as Bucholz and Greco,¹⁵ show success with this technique using brightness mode (B-mode) information. B-mode US has been used to obtain anatomical information^4,5,16 while Doppler US yields flow information for cerebral vasculature.^17,18 The interested reader is directed to Ref. 3 for a history and overview of iUS in neurosurgery in the context of brain shift correction.

1.2. Augmented Reality in Neurosurgery

AR visualizations have become increasingly popular in medical research to help understand and visualize complex medical imaging data. AR is defined as “the merging of virtual objects with the real world (i.e., the surgical field of view).”¹⁹ The motivation for these visualizations comes from the desire to merge preoperative images, models, and plans with the real physical space of the patient in a comprehensible fashion. These augmented views have been proposed to better understand the topology and interrelationships of structures of interest that are not directly visible in the surgical field of view. AR has been explored for neurosurgery in the context of skull base surgery,²⁰ transsphenoidal neurosurgery (i.e., for pituitary tumors),²¹ microscope-assisted neurosurgery,²² endoscopic neurosurgery,^23,24 neurovascular surgery,^25–27 and primary brain tumor resection planning.^28,29 This list is far from comprehensive so the interested reader is referred to Meola et al.¹³ and Kersten-Oertel et al.¹⁹ for detailed reviews of the use of AR in IGNS. In all recently published studies, AR visualizations are described as an enhancement to the minimally invasiveness of a procedure through more tailored, patient-specific approaches. A recent study by Kersten-Oertel et al.³⁰ evaluating the benefit of AR to specific neurosurgical tasks has demonstrated a major pitfall of these types of visualization is the lack of accurate overlay throughout an intervention making them useful only at early parts of an intervention. Recent literature has tried to address this issue through interactive overlay realignment³¹ or through manipulation of visualization parameters^32,33 with some success. In this work, we aim to address this major issue with iUS imaging.

2. Materials and Methods

2.1. Ethics

The Montreal Neurological Institute and Hospital (MNI/H) Ethics Board approved the study, and all patients signed informed consent prior to data collection.

2.2. System Description

All data were collected and analyzed on a custom-built prototype IGNS system, the Intraoperative Brain Imaging System (IBIS).³⁴ This system has previously been described in Refs. 25, 34, and 35 for use with iUS and AR as independent technologies. The Linux workstation is equipped with an Intel Core i7-3820 @ 360 GHz $\times 8$ processor with 32-GB RAM, a GeForce GTX 670 graphics card, and Conexant cx23800 video capture card. Tracking is performed using a Polaris N4 infrared optical system (Northern Digital, Waterloo, Canada). The Polaris infrared camera uses stereo triangulation to locate the passive reflective spheres on both the reference and pointing tools with an accuracy of 0.5 mm.³⁶ The US scanner, an HDI 5000 (ATL/Philips, Bothell, Washington) equipped with a 2D P7-4 MHz phased array transducer, enables intraoperative imaging during the surgical intervention. Video capture of the live surgical scene is achieved with a Sony HDR XR150 camera. Both the camera and US system transmit images using an S-video cable to the Linux workstation at $30\text{frames}/\mathrm{s}$. The camera and US transducer probe are outfitted with a spatial tracking device with attached passive reflective spheres (Traxtal Technologies Inc., Toronto, Canada) and are tracked in the surgical environment. Figure 2 shows the main components of the iUS-AR IGNS system.

Fig. 2.

The different components in an iUS-AR IGNS intervention and their relationship with the surgical and neuronavigation setup. Once the US and live video images are captured from the external devices, they are imported into the neuronavigation system, and all US and AR visualizations are displayed on the neuronavigation monitor (adapted with permission from Ref. 34).

2.3. Patient-Specific Neurosurgical Models

The patient-specific neurosurgical models refer to all preoperative data—images, surfaces, and segmented anatomical structures—for an individual patient. All patients involved in this study followed a basic tumor imaging protocol at the MNI/H with a gadolinium-enhanced T1-weighted MRI obtained on a 1.5-T MRI scanner (Ingenia Phillips Medical Systems). All images were processed in a custom image processing pipeline as follows:³⁷ first, the MRI is denoised, after estimating the standard deviation of the MRI Rician noise.³⁸ Next, intensity nonuniformity correction and normalization are done by estimating the nonuniformity field,³⁹ followed by histogram matching with a reference image to normalize the intensities [Fig. 3(a)]. Within this pipeline, the FACE method⁴⁰ is used to obtain a three-dimensional (3-D) model of the patient’s cortex [Fig. 3(b)]. After processing, the tumor is manually segmented using ITK-Snap,⁴¹ and a vessel model is created using a combination of a semiautomatic intensity thresholding segmentation, also in ITK-Snap and a Frangi Vesselness filter⁴² [Figs. 3(c) and 3(d)]. The processing is done on a local computing cluster at the MNI, and the combined time for the processing pipeline and segmentations is on the order of 2 h. A model of the skin surface was also generated using ray-tracing from the processed images in IBIS using a transfer function to control the transparency of the volume so all segmented structures can be viewed. The processed images and patient-specific models are then imported into IBIS [Fig. 3(e)].

Fig. 3.

Flowchart showing the preoperative steps for creating a patient-specific model and for iUS probe calibration and camera calibration for AR. (a) Preoperative MRI image after denoising, intensity nonuniformity correction, and intensity normalization. (b) The cortical surface is extracted from the MRI image using the FACE algorithm.⁴⁰ (c) Vessels are extracted using an ITK-Snap thresholding segmentation and a Frangi Vesselness filter.⁴² (d) Tumor is manually segmented using ITK-Snap. (e) All preoperative models are combined into a patient-specific model that is imported into the IGNS system. (f) Calibration is performed by serial imaging of a checkboard pattern with an attached tracker in different positions in the tracked camera’s field of view allowing for simultaneous extraction of the intrinsic calibration matrix ($I$) and extrinsic calibration matrix ($T_E$). $T_M$, $T_V$, and $T_E$are the different transformation matrices used to determine $T_E$.³⁴ (g) Tracked US calibration is performed using an N-wire calibration phantom and a custom IGNS calibration plugin allowing for aligning of the virtual N-shaped wires with the intersecting US images.³⁴

2.4. Tracked Camera Calibration and Creating the Augmented Reality View

To create AR visualizations from images captured by a tracked camera, prior calibration of the camera-tracker apparatus must be performed. The intrinsic and extrinsic calibration parameters are determined simultaneously. We determine the intrinsic calibration of the camera using a printed checkerboard pattern fixed on a flat surface with a rigidly attached tracker tool using the method described in Ref. 34. The different components and transformation matrix relationships are shown in Fig. 3(f). Multiple images are taken while displacing the pattern in the camera’s field of view. The intrinsic calibration matrix, $I$, is obtained through automatic detection of the checkerboard corners and feeding the coordinates and tracked 3-D position through an implementation of Zhang’s method.⁴³ This also creates a mapping between the space of the calibration grid and the optical center of the tracked camera $(T_V)$. The extrinsic calibration matrix ($T_E$) is estimated by minimizing the standard deviation of grid points transformed by the right side of the following equation:

$$T_G={(T_M{T}_E{T}_V)}^{-1},$$ (1)

where $T_M$ represents the rigid transformation matrix between the tracking reference and tracked camera and $T_G$ is the transformation between the checkerboard tool and attached tracker. For a more detailed discussion of this procedure, the interested reader is directed to Drouin et al.⁴⁴ The calibration error is measured using a leave-one-out cross-validation procedure with the calibration images to obtain the reprojection error. This is an estimate of the reprojection error that is expected to be obtained if the patient is perfectly registered with the system in the OR; however, this error is compounded with other registration errors that can lead to larger discrepancies. For the cases in this pilot study, the average calibration error was on the order of 0.89 mm (range 0.60 to 1.12 mm). Once the camera has been calibrated and is being tracked, the AR view is created by merging virtual objects, such as the segmented tumor, segmented blood vessels, segmented cortex, and iUS images, with the live view captured from the video camera. To create a perception such that the tumor and other virtual objects appear under the visible surface of the patient, edges are extracted and retained from the live camera view. Furthermore, the transparency of the live image is selectively modulated such that the image is more transparent around the tumor and opaque elsewhere [Fig. 4(d)]. For more details on these visualization procedures, the reader is directed to Refs. 33 and 45.

Fig. 4.

Flowchart of the intraoperative workflow and how surgical tasks are related to IGNS tasks. (a) Patient-to-image registration. After the patient’s head is immobilized, a tracking reference is attached to the clamp, and eight facial landmarks are chosen that correspond to identical landmarks on the preoperative images to create a mapping between the two spaces. (b) AR visualization on the skull is being qualitatively assessed by comparing the tumor contour as defined by the preoperative guidance images and the overlay of the augmented image. (c) A series of US images are acquired once the craniotomy has been performed on the dura and then reconstructed and registered with the preoperative MRI images using the gradient orientation alignment algorithm. (d) AR visualization on the cortex showing the location of the tumor (green) and a vessel of interest (blue). (e) The AR accuracy is quantitatively evaluated by having the surgeon choose an identifiable landmark on the physical patient, recording the coordinates, and then choosing the corresponding landmark on the augmented image, recording the coordinates and measuring the two-dimensional (2-D) distance between the coordinates.

2.5. Tracked US Probe Calibration

When using US guidance for neurosurgery, a correspondence between the physical location of the images and the physical space of the patient must be established. The accuracy of these procedures is closely related to that of device tracking, which is on the order of 0.5 to 1.0 mm for optical tracking systems³⁶ but is often categorized separately since specific phantoms are needed to perform the calibration. Among the various calibration techniques, the N-wire phantoms have been most widely accepted in the literature⁴⁶ due to their robustness, simplicity, and ability to be used by inexperienced users and, therefore, is the technique employed here. Before each case, the US probe was calibrated using a custom-built N-Wire phantom and calibration plugin following the guidelines described in Ref. 46. The US probe calibration is performed by rigidly attaching a tracker to the US probe and filling the phantom containing the N-wires with water. The entire phantom is then registered to an identical virtual model in the IGNS system using fiducial markers on the phantom followed by imaging the N-wire patterns at different US positions and probe depth settings [Fig. 3(g)]. The intersection of the N-wire patterns with the US image slice defines the 3-D position of a point in the US image, and three or more patterns together define the calibration transform for the registered phantom. Within IBIS is a custom manual calibration plugin that allows users to manually identify the intersection points of the wires within a sequence of US images, and the calibration is automatically recomputed after each interaction.³⁴ Following the manual calibration, the calibration is compared with five other N-wire images, and the accuracy is reported as the root mean square of the difference between world coordinates and transformed US image coordinates of the intersection point of the N-wire and the US image plane. The accuracy for each of the cases in this study was on the order of 1.0 mm, which is consistent with reported and accepted values in the literature.⁴⁷

2.6. US-MRI Registration

MR-US registration techniques to correct for brain shift have recently been developed, based on gradient orientation alignment, to reduce the effect of the nonhomogeneous intensity response found in iUS images.⁴⁸ Once an iUS acquisition has been performed, the collected slices are reconstructed into a 3-D volume, resliced in the axial, coronal, and sagittal views, and overlaid on the existing preoperative images. The current simple volume reconstruction works with a raster scan strategy; for every voxel within the volume to be reconstructed, all pixels of all iUS images are evaluated in terms of distance to the reconstructed volume. If a pixel is within this user-specified distance (e.g., between 1 and 3 mm), the intensity of the voxel is increased by the intensity of the iUS pixel, modulated by a Gaussian weighting function. The registration algorithm is based on gradient orientation alignment⁴⁸ focused on maximizing overlap of gradients with minimal uncertainty of the orientation estimates (i.e., locations with high gradient magnitude) within the set of images. This can be described mathematically as

$$T^{*}=\arg {\max }_T\sum _{x\in \mathrm{\Omega }}\cos {(\mathrm{\Delta }\theta )}^2,$$ (2)

where $T*$ is the transformation being determined, $\mathrm{\Omega }$ is the overlap domain, and $\mathrm{\Delta }\theta$ is the inner angle between the fixed image gradient, $\nabla I_f$, and the transformed moving image gradient, $J^T·\nabla I_m$

$$\mathrm{\Delta }\theta =\angle (\nabla I_f,J^T·\nabla I_m).$$ (3)

The registration is characterized by three major components: (1) a local similarity metric based on gradient orientation alignment [$\cos {(\mathrm{\Delta }\theta )}^2$], (2) a multiscale selection strategy that identifies locations of interest with gradient orientations of low uncertainty, and (3) a computationally efficient technique for computing gradient orientations of the transformed moving images.⁴⁸ The registration pipeline consists of two stages. During the initial, preprocessing stage, the image derivatives are computed and areas of low uncertainty gradient orientations are identified. The second stage consists of an optimization strategy that maximizes the average value of the local similarity metric evaluated on the locations of interest using a covariance matrix adaptation evolution strategy.⁴⁹ For an in-depth discussion and more details on this procedure, the interested reader is directed to Ref. 48.

This specific framework was chosen for use in the pilot study due to its validation with clinical US-MRI data.⁴⁸ This registration framework has been shown to provide substantially improved robustness and computational performance in the context of IGNS motivated by the fact that gradient orientations are considered to characterize the underlying anatomical boundaries found in MRI images and are more robust to the effect of nonhomogeneous intensity response found in US images. For this pilot study, only rigid registration transformations were investigated.

Both the volume reconstruction and registration techniques are incorporated into IBIS using a graphics processing unit implementation that allows for high-speed results (on the order of seconds) for reconstruction and rigid registration. This process is briefly summarized in Fig. 4(c).

2.7. Operating Room Procedure

All image processing for each patient-specific model is done prior to the surgical case and imported into the IBIS IGNS console. Once the patient has been brought into the operating room and anesthetized, the patient-to-image registration for IBIS is done simultaneously with the commercial IGNS system, a Medtronic StealthStation (Dublin, Leinster, Republic of Ireland), using eight corresponding anatomical landmark pairs⁵⁰ [Fig. 4(a)]. The quality of this registration is evaluated by the clinical team in the OR. Quantitatively, the fiducial landmark error must be below a certain threshold of 5.0 mm but is often much lower. Qualitatively, the neurosurgeon evaluates the appearance of tracked probe position on the skin of the patient to ensure an appropriate quality of registration has been achieved. AR is then used at different time points during the intervention, and accuracy is qualitatively assessed on the scalp to verify skin incision, on the craniotomy [Fig. 4(b)] to verify craniotomy extent, and on the cortex throughout resection [Fig. 4(d)]. Once the dura has been exposed, tracked intraoperative US images are acquired and used to reregister the preoperative images to the patient. The accuracy of the alignment of the AR view on the cortex is then re-evaluated based on these updated views using both qualitative and quantitative criteria. Comments from the surgeon and visual inspection are used as qualitative criteria, while the pixel misalignment error⁵¹ and target registration error (TRE) of a set of landmark US-MRI landmark pairs are used as quantitative criteria. For each patient, a set of five corresponding landmarks was chosen on both preoperative MRI and iUS volumes to calculate the TRE—as the Euclidian distance between pairs of landmarks—before and after registration. Landmarks were chosen in areas of hyperechoic–hypoechoic transition (US) near tumor boundaries, ventricles, and sulci when the corresponding well-defined features on the MRI were also identifiable. Pixel misalignment error, as the name suggests, is calculated as the distance in pixels between where the augmented virtual model is displayed on the live view and where its true location is in the live view as determined through identification of a single pair of corresponding landmarks identified by the surgeon [Fig. 4(e)]. The precision of the pixel misalignment error measurements relies on the distance between the camera and patient, but caution was taken to ensure the AR camera was always at the edge of the sterile field to ensure that this distance remained relatively constant between cases and calibration. An example calculation is shown in Fig. 5. It is converted to a distance in mm based on the parameters defined through the camera calibration that determine the pixel dimensions.⁵¹

Fig. 5.

The surgeon was asked to place the tip of the tracked pointer at the closest edge of the visible tumor in the surgical field ($T_{\mathrm{L}}$) and to also select the corresponding location on the preoperative images/virtual model ($T_{\mathrm{V}}$). The AR view was initialized and the pixel misalignment error, measured as the 2-D distance between $T_{\mathrm{L}}$ and $T_{\mathrm{V}}$ on the camera image by multiplying the number of pixels with the pixel size (as determined from the camera calibration) between the two points of interest before registration (${\delta }_1$) and after registration (${\delta }_2$).

2.8. Study Design

Two neurosurgeons at the MNI/H were involved in this study. Neither surgeon had prior experience with the AR system before this study but both have been involved with work related to development of the custom neuronavigation system as well as its use and development for intraoperative US brain shift correction. Neither participant was trained with interpreting AR images before the study other than explanation of what the AR images represented and how they would be displayed in the OR.

3. Results

We present the results of our experience in eight iUS-AR IGNS cases. Relevant patient information, including age, sex, and tumor type, is summarized in Table 1.

Table 1.

Summary of patient information.

Patient	Sex	Age	Tumor type	Lobe
1	F	56	Meningioma	L-O/P
2	M	49	Glioma	L-F/T
3	F	72	Metastases	L-O/P
4	M	63	Glioma	R-F
5	F	77	Meningioma	R-F
6	M	24	Glioma	L-F
7	F	62	Glioma	L-O/P
8	F	55	Metastases	R-F

Note: F, frontal; O, occipital; P, parietal; L, left; and R, right.

3.1. Quantitative Results

For all but one case, the iUS-MRI registration improved to under 3 mm, and the pixel misalignment error was on the order of 1 to 3 mm. The average improvement was 68%. In case 2, the camera calibration data were corrupted so the pixel misalignment error could not be calculated. Table 2 summarizes all iUS-MRI registration and pixel misalignment errors. The second column represents the registration misalignment from the initial patient-to-image landmark registration. Columns 3 and 4 represent the registration errors between US and MRI volumes before and after registration, respectively. The final two columns pertain to the virtual model-to-video registration (pixel misalignment error) before and after US-MRI registration.

Table 2.

Summary of registration and pixel misalignment errors.

Patient	Patient-to-image registration (mm)		Pre iUS-MRI registration TRE (mm)	Post iUS-MRI registration TRE (mm)	Prereg pixel misalignment error (mm)	Postreg pixel misalignment error (mm)
	IBIS	Medtronic
1	3.23	3.07	$6.85\pm 3.14$	$1.88\pm 0.93$	N/Aa	N/Aa
2	2.88	3.22	$5.33\pm 1.67$	$2.97\pm 1.54$	5.39	1.19
3	3.96	3.54	$6.31\pm 1.51$	$2.18\pm 1.06$	6.46	1.06
4	4.20	3.66	$7.22\pm 2.31$	$2.34\pm 1.27$	6.88	1.80
5	2.77	3.12	$7.89\pm 1.62$	$2.77\pm 1.34$	7.20	2.35
6	2.33	3.20	$3.58\pm 1.82$	$2.25\pm 1.57$	3.57	1.32
7	4.35	2.98	$6.61\pm 3.39$	$3.48\pm 2.88$	5.55	3.27
8	3.85	3.15	$5.80\pm 2.27$	$1.56\pm 1.02$	4.32	1.22

^aFor this case, the camera calibration data were corrupted, and we were unable to extract the necessary parameters to measure the misalignment error. The pre- to postregistration improvement of mean TRE were statistically significant in all cases (group $t$-test, $p<0.05$).

3.2. Qualitative Results

Qualitative comments from the surgeon reflected largely the benefit of not having to look at or interpret US information but still being able to have an accurately overlaid virtual model that could be augmented to verify the surgical plan and to visualize the surgical target. Their main concerns were the limitation of camera maneuverability due to the size of tracking volume, difficulty in comparing AR visualization with preoperative navigation images, and a learning curve associated with the technology. Table 3 summarizes the different tasks where AR was used throughout these interventions and how surgeons considered it to be useful. Figure 6 is an image summary of four illustrative cases showing a surgical, preregistration AR, and postregistration AR to qualitatively show the improvement of the iUS registration on the AR visualizations.

Table 3.

Summary of AR tasks, benefits, and concerns, as per surgeons using the technology.

Task	Use of AR	Benefits	Concerns
Preoperative planning	• Show location of tumor and other anatomy of interest after patient positioning and registration	• Share surgical plan with other assisting physicians	• Limitation of extent of camera maneuverability due to tracking volume
Craniotomy	• Visualize tumor location in relation to drawn craniotomy borders below bone	• Assess if there is a loss of accuracy from skin landmark registration	• Difficult to verify with navigation images at the same time
		•Minimize craniotomy size. Added comfort in verification that virtual tumor location is within drawn boundaries before removing bone
Cortex pre-iUS Registration	• AR during this context is primarily for research purposes and to determine loss of accuracy from beginning of surgery and initial skin landmark registration	• Despite being used for research, surgeons still found the visualizations useful to have a understanding of the directions of the deviations from initial registration	• Difficulty in understanding data first time during use
			•Limitation of extent of camera maneuverability due to tracking volume
Cortex post-iUS Registration	•Intraoperative planning	• Surgeons found it helpful to compare their physical interpretation of tumor borders with the virtual borders	• Limitation of extent of camera maneuverability due to tracking volume
	•Tumor and vessel identification	• Surgeons commented on the benefit of seeing a virtual model of vessels when they were in close proximity of the tumor and deep to the area of resection
	•Assessment of AR/IGNS accuracy	• AR helped confirm their surgical plan

Fig. 6.

Four illustrative examples of the qualitative improvement of AR for tumor visualization. The left column is the surgical view, the middle column is the initial AR view before iUS registration, and the right column is the iUS brain shift corrected AR view. (a), (b), (c), and (d) are cases 1, 3, 6, and 7, respectively.

4. Discussion and Conclusions

In this study, we successfully combined two important technologies in the context of IGNS of brain tumors. The combination contributed to a high level of accuracy during AR visualizations and obviated the need to directly interpret the iUS images throughout the intervention. The fact that pre-iUS registration error was greater than the registration error following initial patient-to-image registration highlights the gradual loss of accuracy throughout the intervention.⁵² In each case, we improved the patient–image misalignment by registration with iUS data. This resulted in several advantages that included more accurate intraoperative navigation and more reliable AR visualizations as shown qualitatively and quantitatively with the improved TRE measurements and pixel misalignment error measurements, respectively.

The improved accuracy of the system was evaluated by two metrics: the TRE within a series of target points to assess the registration quality and the pixel misalignment error for the improved AR quality. A limitation of measuring the accuracy of AR overlays stems from the lack of a standardized and universal metric, in which the error in AR can be quantified. Some authors use pixel misalignment error,⁵¹ while others use pixel reprojection error,²⁶ and other complex metrics are also described.^6,53 The pixel misalignment error has the implicit assumption that the registration with iUS creates a perfectly aligned image. This assumption is inevitably violated, and thus pixel misalignment error is not a perfect measure of accuracy and is only an indication of relative error between the two AR views rather than an absolute error for either view. Despite this limitation, it was deemed here to be the most appropriate for quantitative evaluation of the AR images. Another consideration on the accuracy of the registration procedure is the effect of heart rate and blood pulsation during iUS acquisition. This detail was not considered in this pilot project and will be investigated in future work. This work is intended to serve as a pilot study to assess the feasibility of the combination of US and AR technologies to improve on each of their shortcomings. For this reason, an US-MRI registration algorithm that has been reported in the literature to work well with this type of clinical data was chosen, as well as an AR evaluation metric that was considered the most appropriate in evaluating the quality of virtual overlay for the data presented in this study. Future work will require more extensive validation against other US-MR registration frameworks to draw stronger conclusions about the quality of accuracy improvement, as well as other AR evaluation metrics to better describe the quality of virtual overlay improvement. Finally, registration errors on the order of 1.0 mm are ideally desired for neuronavigation-assisted tasks; however, this level of accuracy is rarely achievable, and a registration error on the order of 2.0 to 3.0 mm is sufficient to perform the intended tasks appropriately for this pilot study. In future work, with the use of nonlinear registration, we hope to further improve the level of registration accuracy to be closer to ideal conditions.

In this study, AR views were acquired with the use of an external camera to capture images of the surgical scene and render the AR view on the computer workstation. This strategy was employed since one of the surgeons involved in the project does not generally use a microscope while performing tumor resections. Augmenting microscope images is also compatible within IBIS and may facilitate integration of AR in the operating room for navigation.³⁴ While the justification for AR in some of cases presented here may not be great due to the tumor’s proximity to the cortex, the potential of AR in more complicated scenarios should not be understated. For smaller tumors located much deeper within the brain or for tumors near eloquent brain areas, having the ability to see below the surface with accurate visualizations offered using AR creates the possibility of tailoring resection corridors to minimize the invasiveness of the surgery. This benefit can be accomplished only if we are able to maintain a high level of patient-to-image registration accuracy throughout the procedure. Combining iUS registration and AR with more accurate tumor segmentations, such as the process described in Ref. 54, would assist a surgeon in resecting as much tumorous tissue as possible with minimal resection of healthy tissue without having to rely solely on a mental map of the patient’s anatomy and the surgeon’s ability to discriminate tissue types.

It is clear from the qualitative comments from the surgeons involved in this work that there is a learning curve associated with AR in the context of IGNS. In the first several cases, the AR was employed simply as a tool to verify positions of anatomy of interest and to assess the accuracy of the AR image alignment with the tracked preoperative images. As the surgeons became comfortable with the system, the length of time AR was used increased and the number of tasks where it was deemed useful also increased. The surgeons commented on the usefulness of AR to assess and minimize the extent of the craniotomy and to assess the location of the anatomy of interest (i.e., tumors and vessels) once the cortex has been exposed. Additionally, the surgeons commented on the benefit of using AR to share the surgical plan with assisting residents and physicians by being able to show the vessels and tumor location before making an incision. The surgeons also commented on the fact that having a colored AR image for assessing the anatomy was more enjoyable than a grayscale US. The primary concern of the surgeons using the system was the limitation of camera maneuverability due to the size of tracking volume, which also led to difficulty in comparing AR visualization with preoperative navigation images. However, over the course of using the system, the comfort with the system quickly grew over the first few cases and the usefulness and amount of time and information requested increased. With continued use, surgeons found the information increasingly useful as they incorporated it into their intraoperative planning suggesting that with reliable accuracy and training this technology could provide a benefit to improve the minimal invasiveness of surgery and to help with patient model interpretation.

In conclusion, this pilot study highlights the feasibility of combining iUS registration and AR visualization in the context of IGNS for tumor resections and some of the advantages it can have. While many authors have investigated these techniques separately for brain tumor neurosurgery, few have looked at the benefits of combining these two technologies. Our pilot study in eight surgical cases suggests that the combined use of these technologies has the potential to improve on traditional IGNS systems. By adding improved visualization of the anatomy and pathology of interest, while simultaneously correcting for patient–image misalignment, extended reliable use of IGNS throughout the intervention can be maintained that will hopefully lead to more efficient and minimally invasive surgical intervention. In addition, with accurate AR visualizations, the neurosurgeon is not required to interpret the iUS images, which can be confusing to a nonexpert. With continued development and integration of the two techniques, the proposed iUS-AR system has potential for improving tasks, such as tailoring craniotomies, planning resection corridors, and localizing tumor tissue while simultaneously correcting for brain shift.

Biographies

Ian J. Gerard is a PhD candidate of biomedical engineering at McGill University. His current research involves improving the accuracy of neuronavigation tools for image-guided neurosurgery (IGNS) of brain tumors with focus on intraoperative imaging for brain shift management and enhanced visualization techniques for understanding complex medical imaging data.

Marta Kersten-Oertel is an assistant professor at Concordia University specializing in medical image visualization and IGNS. Her current research involves the development of an augmented reality (AR) neuronavigation system and the combination of advanced visualization techniques with psychophysics to improve the understanding of complex medical imaging data.

Simon Drouin is a PhD candidate in biomedical engineering at McGill University. His current research involves user interaction in AR with a focus on depth perception and interpretation of visual cues in augmented environments.

Jeffery A. Hall is a neurosurgeon and an assistant professor of neurology and neurosurgery at McGill University’s Montreal Neurological Institute and Hospital (MNI/H), specializing in the surgical treatment of epilepsy and cancer. His current research, in collaboration with other MNI clinician–scientists, includes developing noninvasive means of delineating epileptic foci, intraoperative imaging in combination with neuronavigation systems, and the application of image-guided neuronavigation to epilepsy surgery.

Kevin Petrecca is a neurosurgeon, an assistant professor of neurology and neurosurgery at McGill University, and the head of neurosurgery at the Montreal Neurological Hospital, specializing in neurosurgical oncology. His research at the MNI/H Brain Tumour Research Centre focuses on understanding fundamental molecular mechanisms that regulate cell motility with a focus on malignant glial cell invasion.

Dante De Nigris was formerly a PhD student in the Department of Electrical and Computer Engineering, McGill University. His research interests focus on analyzing and developing techniques for multimodal image registration, specifically on similarity metrics for challenging multimodal image registration contexts.

Daniel A. Di Giovanni is a PhD student in an integrated program in neuroscience at McGill University. His research interests focus on the analysis of functional MRI data in the context of brain tumors.

Tal Arbel is an associate professor in the Department of Electrical and Computer Engineering and a member of the McGill Centre for Intelligent Machines. Her research goals focus on the development of modern probabilistic techniques in computer vision and their application to problems in the medical imaging domain.

D. Louis Collins is a professor in neurology and neurosurgery, biomedical engineering and an associate member of the Center for Intelligent Machines, McGill University. His laboratory develops and uses computerized image processing techniques, such as nonlinear image registration and model-based segmentation to automatically identify structures within the brain. His other research focuses on applying these techniques to IGNS to provide surgeons with computerized tools to assist in interpreting complex medical imaging data.

Disclosures

All authors declare they have no conflicts of interest.