Augmented reality guidance in cerebrovascular surgery using microscopic video enhancement

Abstract

Cerebrovascular surgery treats vessel abnormalities in the brain and spinal cord, including arteriovenous malformations (AVMs) and aneurysms. These procedures often involve clipping the vessels feeding blood to these abnormalities, making accurate classification of blood vessel types (feeding versus draining) important during surgery. Previous work to guide the intraoperative identification of the vessels included augmented reality (AR) using pre-operative images, injected dyes, and Doppler ultrasound, but each with their drawbacks. The authors propose and demonstrate a novel technique to help differentiate vessels by enhancing short videos of a few seconds from the surgical microscope using motion magnification and spectral analysis, and constructing AR views that fuse the analysis results as intuitive colourmaps and the surgical microscopic view. They demonstrated the proposed technique retrospectively with two real cerebrovascular surgical cases: one AVM and one aneurysm. The results showed that the proposed technique can help characterise different vessel types (feeding and draining the abnormality), which agree with those identified by the operating surgeon.

1. Introduction

Cerebrovascular surgery involves the treatment of several types of vascular diseases and vessel abnormalities in the brain and spinal cord. These procedures have a complication rate of 30.9%, with a 30-day mortality rate of 5.4% [1]. Two abnormalities most commonly addressed by this type of procedure are arteriovenous malformations (AVMs) and aneurysms. AVMs are improper collections of blood vessels in the brain, with a risk of haemorrhage of 2–4% every year [2]. Rather than following the typical pattern of blood flowing from arteries to capillaries and then to veins, in an AVM blood flows directly from an artery to a vein through a fistula [3], which is made up of abnormal vessels that are hybrids between true arteries and veins, known as arterialised veins [4]. These vessels allow for high-flow blood transport by bypassing the capillaries, which induces arterial hypotension in arterialised veins [3]. Furthermore, these feeding and draining vessels often have weakened walls and therefore may lead to leak or rupture. In many cases, AVM treatment is recommended to protect against haemorrhage, as this may lead to stroke, permanent disability, or death. The three major treatment options for patients with AVM include radiation, embolisation, and surgery. One of the major steps in these procedures is to clip the feeding artery to arrest blood flow. Therefore, a detailed understanding of the sites of arterial inflow and venous drainage is important for clinical evaluation and management of AVMs so that the correct vessels may be clipped and destroyed [5]. An aneurysm, on the other hand, occurring in ∼2% of the population [6], is a bulge in the wall of a blood vessel due to the wall's weakness. As this bulge grows in size, the chances of a rupture, stroke or death increase. To prevent rupture, treatments such as embolisation or surgery (to block or close the effected vessel) are employed. Surgical treatment involves performing a craniotomy and clipping the aneurysm.

For vascular neurosurgeries, it is crucial to identify the feeding and draining blood vessels surrounding the pathology, as clipping the wrong vascular branch can result in severe complications (e.g. haemorrhage). However, it is almost impossible to differentiate the feeders and drainers by visual inspection of the surgical scene [7] (see Fig. 1), and neuronavigation technology is necessary to ensure the safety and quality of the intervention. To solve this, several techniques have been proposed. One routinely used approach employs injected dyes, such as indocyanine green or fluorescein [8, 9] to help capture the blood flow of the vasculature conveniently under the surgical microscope used in the procedure. Although the method can offer great visual results, one major disadvantage is its reliance on intra-arterial injection, which requires additional work and can be potentially risky. Furthermore, even with a short period of visibility (∼10 s), the dyes cannot be reinjected for imaging frequently since they stay in the patient's system for more than 15 min. Alternatively, Doppler ultrasound has been used in the clinic to help differentiate the feeding and draining of blood vessels, but it requires direct contact with the tissue. This contact can potentially exert undesired external pressure on the weakened arterial wall. Finally, augmented reality (AR) [10, 11] that combines the 3D surgical scene with processed pre-operative imaging data (e.g. angiography) has also been demonstrated and was shown to be instrumental to plan craniotomy and identification of vascular branches. It offers intuitive visualisation, but the current approach needs meticulous object tracking and camera calibration and fails to offer intraoperative haemodynamic information [11].

Fig. 1.

Averaged signal in the region of interest shown on the left image is plotted as a function of time before and after motion magnification

In this Letter, we propose a novel method in an attempt to combine the benefits of previous methods while mitigating their drawbacks to guide cerebral vascular surgeries. It aims to obtain intraoperative haemodynamic information by magnifying subtle biological signals that are almost invisible to the naked eye from videos of only a few seconds with a surgical microscope, which is already part of the workflow during the intervention. Previously, different video enhancement methods have been used to recover rhythmic biological signals, including in endoscopic surgery [12–14]. Finally, the information is processed with metrics related to power spectrum analysis, and the results are fused as intuitive colourmaps with the microscopic view as an AR scene without the need for 3D object tracking or camera calibration. To demonstrate the technique, we employed video footages from two neurovascular surgical cases (one to correct an AVM and one to correct an aneurysm) and conducted the analysis and AR-construction retrospectively. Our preliminary results showed that the proposed technique can help characterise the draining and feeding blood vessels for both cases with intuitive visualisation.

2. Materials and methods

2.1. Data acquisition

Upon informed consent, short video sequences from a surgical microscope used during two cerebral vascular surgeries (one with AVM and the other with an aneurysm) were acquired retrospectively. For each case, a short video segment of the exposed surgical site was acquired without moving the camera or the surgical tools. More specifically, we curated 5 and 10 s of the sequences from the aneurysm and AVM cases, respectively. The surgeon performing the procedures confirmed which vessels were bringing blood towards and away from the abnormal region from the videos.

2.2. Video enhancement

To reveal blood flow patterns that are not visible to the naked eye, we used two video magnification methods, introduced in [15, 16], together, as shown in Fig. 1. In broad strokes, these methods both enhance the temporal information in three steps. First, the images in the video stream are decomposed, on a pixel-by-pixel basis, into a range of spatial frequency components that encode the dynamic information, such as motion or colour changes at different rates. Next, the target frequency component is multiplied by a desired factor, α. A value of α>1 is chosen if magnification is desired, or α<1 if the goal is attenuation. Finally, the enhanced video is reconstituted by recombining all of the frequency components. When comparing the two methods, Eulerian video magnification [15] is better at enhancing dynamic colour changes, while the more recent phase-based video processing technique [16] is more adept at motion magnification and attenuation [12]. In this study, video intensity signals from 0.8 to 3 Hz were magnified to capture the full range of potential patient heart rates, as the pulse of a diseased patient may vary significantly from the normal of ∼60–100 bpm, or 1–1.67 Hz. Since the Eulerian method [15] simultaneously magnifies the image intensity and motions in videos, to more reliably analyse the signal with consistent pixel location correspondence, the resulting video sequences were first processed using the phase-based video motion attention method [16] to remove motion from the microscope. Then, the image intensity of the video was magnified by a factor of 100 for the selected frequency range with the Eulerian method [15].

2.3. Signal processing

The enhanced video is able to visually reveal the rhythmic colour changes due to the blood pulsations within the vessels. However, such visual cues are difficult to interpret for differentiating the feeders and drainers, so further processing is required. To analyse the enhanced videos, we converted the RGB signals to grey-scale for more in-depth investigation. Generally, it is believed that arteries have a stronger and more well-defined pulsation than their venous counterparts, especially at a frequency corresponding to the patient's heart rate. Therefore, we transformed the enhanced periodic image intensity changes into the frequency domain, and conducted power spectral analysis to better characterise the blood vessel pulsations.

To better grasp the pulsation behaviour of arteries and veins, and confirm the previous assumption, we first obtained the power spectra from the signals averaged from ROIs within the blood vessels of interest in surgery for both clinical cases. The mean value of these signals was subtracted to remove effects caused by differences under lighting conditions. Then, to help better interpret the power spectra at each location within the blood vessels, we employed the metrics of spectral power and spectral entropy [17]. More specifically, spectral power, S, measures the total power of all frequency components shown in the power spectrum $P({\omega }_i)$, defined as

$$S=\sum _{i=1}^{n}P({\omega }_i).$$

This metric is used to quantify the pulsation strength. Spectral entropy (SE) borrows the definition of entropy from information theory and is defined as

$$\mathrm{SE}=-\sum _{i=1}^{n}P({\omega }_i){\log }_2(P({\omega }_i)).$$

It is used to characterise the complexity of the frequency components in a signal. This approach has been previously employed to help characterise other biological signals for the same purpose [17]. Later, to construct the AR view, the metrics S and SE were computed in a pixel-by-pixel manner within the segmented blood vessel masks to save computational time.

2.4. AR view construction

To offer a more intuitive visualisation during surgery, the colourmaps resulting from power spectral analysis were fused with the original microscope scene. To reduce cognitive burden, the relevant information was only augmented in the blood vessels of interest visible within a binary mask M, segmented from the video frame. To naturally blend the resulting colourmaps with the textures and shadings of the surgical scene, the final AR view $I_{\mathrm{AR}}$ was created using the following equation:

$$I_{\mathrm{AR}}=\alpha M\cdot I_{\mathrm{map}}\cdot I_{\mathrm{video}}+\left(1-M\right)\cdot I_{\mathrm{video}},$$

where $I_{\mathrm{map}}$ and $I_{\mathrm{video}}$ are the colourmap and video frame before enhancement, respectively, and $\alpha$ is a scaling factor selected by the user to change the brightness of the data fusion region.

3. Results

To demonstrate the impacts of video enhancements, temporal signal changes within a fragment of the blood vessel are shown in Fig. 1 for the AVM case using the demotioned video sequences. Note that after video enhancement, the averaged signal change within the region of interest (ROI) is much more evident.

To learn about the blood pulsation characteristics of arteries and veins, power spectra of the averaged signals within different blood vessel segments were plotted in Figs. 2 and 3 for the AVM and aneurysm cases, respectively. In both figures, the surgical targets (AVM and aneurysm) are identified; the ROIs of the draining vessels are colour-coded in yellow while the feeders are in white. From the analysis results (Figs. 2d and 3e), it can then be observed that the feeding arteries have a higher power spectral density peak, signalling a stronger pulsation than the draining veins. In addition, the signals from the veins contain a more equal distribution of frequency components’ magnitudes than the arteries, further confirming our assumption of less well-defined pulsatility in drainers. Despite the differences in the power spectra curves across the ROIs, their principal pulsation rates remain nearly the same (1.00 Hz or 60 bpm for the AVM case and 1.41 Hz or 84.7 bpm for the aneurysm case).

Fig. 2.

Averaged signal behaviours of different vascular branches using the power spectrum for the AVM case

a A video frame of the surgical scene with the AVM site identified

b ROIs used for analysis (white = feeder, yellow = drainer)

c 3D-rendered rotational angiography showing AVM and corresponding ROIs

d Power spectrum of the mean signal inside each ROI

Fig. 3.

Averaged signal behaviours of different vascular branches using power spectrum for the aneurysm case

a A video frame of the surgical scene with the aneurysm location identified

b ROIs used for analysis (white = feeder, yellow = drainer)

c Power spectrum of the mean signal inside each ROI

To allow more detailed analysis and to facilitate the surgical decisions, the corresponding colourmaps were generated for spectral power and spectral entropy on a pixel-by-pixel basis for the regions that contain the blood vessels of interest. The results are depicted in Figs. 4b and c and 5b and c for the AVM and aneurysm cases, respectively. In addition, the reconstructed AR views were also shown in Figs. 4d and e and 5d and e for the two cases. Again, it can be observed that generally speaking, the feeding arteries contain higher spectral power and lower spectral entropy than the draining veins. In other words, the arteries have stronger and more rhythmic blood pulsations than the draining blood vessels.

Fig. 4.

Pixel-wise power spectral analysis results and AR view construction for the AVM case

a A video frame of the surgical scene

b Colour-coded spectral entropy map within the blood vessels

c Colour-coded signal power map within the blood vessels

d, e AR views blending the video frame and colour-coded spectral entropy (b) and signal power maps (c), respectively

Fig. 5.

Pixel-wise power spectral analysis results and AR view construction for the aneurysm case

a A video frame of the surgical scene

b Colour-coded spectral entropy map within the blood vessels

c Colour-coded signal power map within the blood vessels

d, e AR views blending the video frame and colour-coded spectral entropy (b) and signal power maps (c), respectively

4. Discussion

Compared with previous neuronavigation techniques [10, 11], the proposed AR technique addresses a shortcoming of previous AR guidance systems by using real-time monitoring and haemodynamic data to provide additional information to distinguish between vessel types. In contrast to systems using angiography with injected dyes, this system has the capability to provide information continuously. Additionally, it does not require an injection of any contrast agent. Compared with Doppler ultrasound, this method avoids direct contact with the tissue of interest, effectively preventing the external force that may damage the weakened blood vessel walls.

To demonstrate our approach, we used two real vascular neurosurgical cases, one for AVM and the other for an aneurysm, representing two of the most common neurovascular interventions. We used power spectral analysis (i.e. spectral power and entropy) to help characterise the pulsation patterns of different types of blood vessels in each case. In general, the arteries exhibit higher spectral power and lower entropy, implying stronger and more rhythmic pulsation patterns. However, in the aneurysm case, region ROI2 (Fig. 3b) showed lower spectral power (Fig. 5c) but similar spectral entropy to the feeding arteries (Fig. 5b). This could be explained by the close proximity of the vessel segment to the feeding artery (ROI4 in Fig. 3b). We will investigate other metrics and techniques to better differentiate the two different types of the vasculature.

Although the case study demonstrates promising results and offers insights into the characteristics of the feeding and draining vessels, the results need to be further confirmed with a larger population. We are currently actively recruiting patients for more in-depth investigation, and will also conduct animal studies to better characterise the method. At the moment, we used manually segmented binary masks to construct the AR view. This is still feasible in the operating room as it can be achieved with a few strokes with suitable segmentation software [18]. However, in the future, as with more surgical cases, we will leverage the power of deep learning [19] to automatically identify the blood vessels from the microscopic view in real-time. For the proposed technique, we used video data from the surgical microscope, which is usually fixed during each phase of the operation. As only 5–10 s of video footage is needed, it is feasible for the clinician to not induce motion by touching the patient, and re-processing can be easily made when tissue shift occurs or the microscope is re-positioned.

Instead of the more commonly used simple α-blending technique to combine the colourmaps and the surgical scene within the blood vessel regions, we used the multiplication of two images. Based on the preliminary experiences of the clinicians involved with this project, this allows a more natural fusion of the colourmap with the textures of the tissues and does not weaken the colour representation of the colourmaps compared with simple α blending. This is crucial for clinical decisions. In addition, the user can adjust the brightness of the fusion region to adapt to different lighting conditions. Finally, based on the feedback from the clinicians involved in this study, our method provides sufficient and intuitive visualisation to help differentiate the feeders and drainers. In the future, we would like to conduct thorough user studies to determine the perceived usefulness, ease of use, and visualisation strategies to further improve our technique.

5. Conclusion

We proposed a new contactless technique to help characterise the draining and feeding blood vessels during vascular neurosurgeries using video sequences of several seconds obtained from the surgical microscope and demonstrated the application in AVM and aneurysm surgeries as a retrospective case study. By magnifying dynamic events that are normally invisible, and augmenting the analytical results with the microscopic view, the technique holds great promise to provide guidance in neurovascular surgeries with minimum interruption of the normal workflow.

6. Funding and declaration of interests

This work was supported by funding from the Canadian Institutes of Health Research, the Canadian Foundation for Innovation, the Natural Sciences and Engineering Research Council of Canada, the Ontario Research Foundation, and a BrainsCAN Postdoctoral Fellowship for YX.

7. Conflict of interest

None declared.