Predicted Microscopic Cortical Brain Images for Optimal Craniotomy Positioning and Visualization

Abstract

During a craniotomy, the skull is opened to allow surgeons to have access to the brain and perform the procedure. The position and size of this opening are chosen in a way to avoid critical structures, such as vessels, and facilitate the access to tumors. Planning the operation is done based on pre-operative images and does not account for intra-operative surgical events. We present a novel image-guided neurosurgical system to optimize the craniotomy opening. Using physics-based modeling we define a cortical deformation map that estimates the displacement field at candidate craniotomy locations. This deformation map is coupled with an image analogy algorithm that produces realistic synthetic images that can be used to predict both the geometry and the appearance of the brain surface before opening the skull. These images account for cortical vessel deformations that may occur after opening the skull and is rendered in a way that increases the surgeon’s understanding and assimilation. Our method was tested retrospectively on patients data showing good results and demonstrating the feasibility of practical use of our system.

1. Introduction and Background

A craniotomy is the surgical removal of part of the bone from the skull to expose the brain, for example to provide access for tumor resection. Surgeons often use well-established image-guidance systems where a patient’s head is registered with a pre-operative MRI scan (Fraser et al. (2009), Bucholz and McDurmont (2009)) to plan the location and size of the craniotomy before beginning surgery. Those systems help surgeons navigate to prepare the patient and locate the tumor. Using the registered pre-operative MRI scans, the surgeons draw a target on the patient’s skin (see Figure 1-d), then opens the skin and cuts the bone to open the skull. The target size and location is chosen to provide optimal access to the tumor while avoiding vessels and other critical structures.. However, several factors, including opening the skull, head position, changes in osmotic pressure and loss of cerebral spinal fluid, can cause the brain to shift upon opening. Thus the planned craniotomy may not provide optimal or adequate exposure, increasing patient risk and surgical complexity.

Figure 1.

Components of our approach: (a) intra-operativelly, surgeons chose a candidate opening position p_c on patient’s head P; the point is represented by a green circle (b) a pre-operative model $M_{\mathbb{S},\mathbb{P},\mathbb{V}}$ is built from MRI scans. The chosen opening position is located on the pre-operative model after a rigid registration; (c) a microscopic image $\mathcal{I}$ is synthesized before opening the skull, taking into account possible brain deformations via function $\mathcal{D}$. The output displacement field is represented with arrows; (d) Surgeons can optimize the craniotomy opening position p_c and size r_c from predicted images in (c) before starting the procedure. Our method can be plugged to the actual method (framed in pink) already used in the opertatin room.

Related Work:

A large amount of work has been dedicated to understanding and compensating for intra-operative brain shift using various types of intra-operative imaging techniques (Morin et al. (2017), Bayer et al. (2017), Miga et al. (2016), Sun et al. (2014), Kuhnt et al. (2012) Reinertsen et al. (2014) Ji et al. (2008) Rivaz and Collins (2015)). The objective of these methods is to compensate for brain shift by updating the pre-operative scans using intra-operative data after dura-opening. Several planning systems have been proposed to anticipate brain shift based on brain anatomy, head positioning, craniotomy size, and other surgical variables Miga (2016). In the context of trajectory planning for Deep Brain Stimulation, Essert et al. (2011) proposed a geometric method based on a database of brain shift cases to build a template of possible defomations. Bilger et al. Bilger et al. (2012) proposed a method to estimate a brain shift risk map using a simulation that gathers statistics on the displacement of anatomical landmarks from different clinical studies. This risk map can be visualized pre-operatively using a color-coded scheme based on vessels proximity. This method was later improved by Hamzé et al. (2015) using more advanced models.

In contrast to estimating brain deformation for pre-operative planning or intra-operative guidance, using brain shift to optimize the location and size of the craniotomy is a little studied problem. Optimizing the craniotomy has been addressed in retrospective studies for evaluating insertion trajectory accuracy (Chen and Nakaji (2012)) and burr hole placement (Rai et al. (2019)). Although focused on burr holes, these studies, undertaken with more than 50 patients, highlight the importance of choosing the optimal entry point and size.

Motivation and Contribution:

Recently, surgeons tend to favor smaller craniotomies to minimize patient exposure. However, accessing tumors while avoiding blood vessels, sulcal folds and other critical structures are the main goal for when choosing the craniotomy position and size. We propose a unique system for optimizing craniotomy opening placement and size. Because it is very difficult (if not impossible) to accurately estimate patient-specific brain shift pre-operatively (Frisken et al. (2019)), we introduce a cortical brain deformation map that estimates possible brain deformation. This maps considers the location, the patient’s head position w.r.t gravity and a user-defined deformation amplitude. For each candidate, a miscrocopic image is generated that synthesizes brain surface appearance using an advanced image analogy technique. To the best of our knowledge, no similar methods have been proposed.

2. Method

2.1. Problem Formulation and Approach overview

Our approach, illustrated in Figure 1, involves a composition function Θ that predicts brain surface deformation and appearance before skull opening at chosen head positions.

Let $M_{\mathbb{S},\mathbb{P},\mathbb{V}}$ be the 3D surface geometry of the brain accounting for the skull, the parenchyma and the vessels (denoted by their respective subscript) derived from pre-opertative MRI scans, and let P be the 3D representation of the actual patient’s head in the operating room. We assume that P and $M_{\mathbb{S}}$ can be registered before starting the procedure. This registration can be done with well-established image-guided neurosurgical systems (Fraser et al. (2009), Bucholz and McDurmont (2009)) Hung Hsieh et al. (2017) and is outside of the scope of our work. Using this registration, surgeons can chose a candidate opening position p_c and size r_c of the craniotomy opening (see Figure 1-d). Our system provide the surgeon with an intuitive tool to chose the best candidate {p_c, r_c}.

The composition Θ aims at predicting an image $\mathcal{I}$ that accounts for possible brain deformation while providing a visually understandable output. The deformation is representedd as a displacement field computed using the function $\mathcal{D}$. $\mathcal{D}$ is computed from pre-defined brain physical properties (gravity, brain elasticity and viscosity), the position of the craniotomy p_c, a user-defined deformation parameter γ and the pre-operative brain surface $M_{\mathbb{S},\mathbb{P},\mathbb{V}}$. $\mathcal{D}$ can be used to transform the deformation from 3D space into an in-plane stress map in 2D space. This projection produces an annotated image that draws the boundaries of the different structures of a cortical brain surface, namely the parenchyma, the vessels and the skull. Using an the analog image pair ($\mathcal{J}$, ${\mathcal{J}}_{\text{label}}$) (see Figures 3-a and 3-b), chosen randomly from a pre-defined database of microscope images of cortical surfaces, the labeled regions are composited, pixel-per-pixel, to generate a the final image $\mathcal{I}$. The composition function as

$$\mathcal{I}={\Theta }_{\mathit{\text{analogy}}}\left(\mathcal{D}\left(M_{\mathbb{P},\mathbb{V}},p,\gamma \right),\mathcal{J},{\mathcal{J}}_{\text{label}},\right)$$ (1)

A new image $\mathcal{I}$ is predicted for each candidate {p_c, r_c} to allow surgeons to chose the optimal one. The composition function Θ is a parametric function that takes into account intra-operative events, i.e.: patient’s head position $M_{\mathbb{S}}$, the amount of deformation γ.

Figure 3.

Image analogy: given a source image $\mathcal{J}$ in (a) and its annotated images ${\mathcal{J}}_{\text{label}}$ in (b) we want to synthesize a target image $\mathcal{J}$ in (d) following its annotation image ${\mathcal{I}}_{\text{label}}$ in (c). Yellow represents the background, red the vessels and green represents the parenchyma.

2.2. FEM-based Brain Shift Model

Initial brain shift is caused by several factors, including cerebro-spinal fluid (CSF) loss and gravitational positioning (Elias et al. (2007)). The amount of CSF loss is hard to anticipate, even with sophisticated models (Hamzé et al. (2015)). The most accurate ways to compensate for brain shift use intra-operative imaging after skull opening (Luo et al. (2019)). However, in our case such information cannot be used since we aim to estimate brain deformation before opening the skull. Instead we rely on a parametric physics-based modelling to predict the geometry of the cortical surface subject to brain shift.

In our simulation we use a non-linear geometric finite element model with a linear constitutive law (Felippa and Haugen (2005)) where the volume of the brain parenchyma is meshed as a set of hexahedral elements. Following the work from Bilger et al. (2011), we model the pressure created by the CSF on the brain as external forces applied on the surface of the parenchyma $M_{\mathbb{P}}$. These external forces are integrated into the global brain motion formulation following:

$$\underset{\text{Brain stiffness}}{\underbrace{Ku}}=\underset{\text{CSF loss pressure}}{\underbrace{{\iint }_{{\text{M}}_{\text{P}}}dgh(m)\partial M_{\mathbb{P}}}}+\underset{\text{Skull contact}}{\underbrace{H^T\lambda }}+\underset{\text{Gravity}}{\underbrace{\text{g}}}$$ (2)

where d is the density of CSF (≈ 1007 kg/m³), g is the norm of gravity and h is the distance between a point m on the surface and the fluid level. This force is computed on each element of the brain mesh that corresponds to the immersed surface. The stiffness matrix K encodes the elasticity of the parenchyma and is built following textbook parameters (Elias et al. (2007)), g is the gravity force while H^Tλ gathers the contacts between the brain and the inner part of the skull. The matrix H contains the known constraints directions and is unknown and has to be computed. λ are Lagrange multipliers containing the constraint force intensities. A linear complementary system is obtained, and is solved using a Gauss-Seidel algorithm (Cotin et al. (2005)).

Equation 2 the computation of brain deformations, which consists of updating the positions vector u, based on the position of the head w.r.t gravity and the amount of CSF loss. While the relative head position can be deduced from the registration step, the CSF loss quantity is unpredictable. We propose to estimate brain shift based on a model of maximal shift (which represents the worst scenario). The amount of CSF loss can be manually adjusted by surgeons in a spectrum of values introduced by Elias et al. (2007) that actually depict the leaked volume. For simplicity, this parameter, which we denote γ, can take values between 0% and 100%, where 0% means no brain shift and 100% means half the CSF has leaked out.

2.3. Cortical Deformation Map

The deformation map $\mathcal{D}$ aims at updating the geometry of the cortical surface M at a selected position p_c w.r.t to the CSF amount γ. This step consists of creating an in-plane displacement map (in 2D) from the volumetric deformation (in 3D) estimated from equation 2. This consists of projecting the surface displacement $\delta M_{(\mathbb{P},\mathbb{V})}$ that is linearly mapped to the displacement field δu computed from equation 2. This projection is done following a projection matrix Π that simulate the surgical microscope camera. Its intrinsic parameters are defined according to pre-defined focal length f_cam and image size w × h, while its position is set to be at a distance d_cam from the candidate position. These parameters values are taken from a textbook and do not account for lens distortions.

$$\mathcal{D}\left(M_{(\mathbb{P},\mathbb{V})},p_c,\gamma \right)=\underset{i=1}{\overset{N_p}{\cup }}\Pi \cdot \mathcal{T}\left(M_{\mathbb{P},\mathbb{V}}(i),\delta u^{\gamma }\right)$$ (3)

where $M_{\mathbb{P},\mathbb{V}}(i)$ corresponds to the i^th triangle of either the parenchyma or the vessels surface around the entry point p_c, with $i\in N_{p_c}$; $\mathcal{T}$ is a linear mapping function that maps the surface $M_{\mathbb{P},\mathbb{V}}$ to the volume u following their barycentric coordinates; and δu^γ is the displacement field computed from Eq. 2 with an amount γ of CSF loss. The opening size r_c is not included in the computation of the displacement field and does not have a physical impact. In practice, the function $\mathcal{D}$ will produce an image of size w × h that encodes the displacement field around an entry point. An example of the output stress map for different regions of the brain is illustrated in Figure 2.

Figure 2.

Output of the cortical brain deformation map: an in-plane deformation map is computed depending on the position of the opening position p_c, the amount of CSF loss γ and head gravitational positioning.

2.4. Sub-cortical Deformation

In order to propagate the surface deformation to tumors and other sub-cortical structures, we use a linear geometrical barycentric mapping function. We restrict the impact of the cortical vessel deformations to the immediate underlying structures. Formally speaking, if we denote the vector of vertices representing a 3D tumor by $M_{\mathbb{T}}$, we can express each vertex $M_{\mathbb{T}}(i)$ using barycentric coordinates of facet vertices u, such that

$$M_{\mathbb{T}}(i)=\sum _{j=1}^3{\varphi }_j\left(x_i,y_i,z_i\right)u_j$$ (4)

where ϕ(x, y, z) = a + bx + cy with (a, b, c) being the barycentric coordinates of the triangle composed of nodal points u_j, with 1 ≤ j ≤ 3. This mapping is computed at rest and remains valid during the deformation.

2.5. Microscopic Image Synthesis via Analogy

Synthesizing the final image $\mathcal{I}$ using analogy requires to have three input images: a source image $\mathcal{J}$, its corresponding labeled image ${\mathcal{J}}_{\text{label}}$ and a labeled target image ${\mathcal{I}}_{\text{label}}$. The labeled images represent a semantic annotation of the brain surface seen through a microscope. They are annotated with three classes: parenchyma, vessels and the skull. ${\mathcal{J}}_{\text{label}}$ is annotated manually while ${\mathcal{I}}_{\text{label}}$ is annotated automatically from the output image of the deformation map $\mathcal{D}$, where ${\mathcal{J}}_{\mathit{\text{label}}}(j)=\{0,1,2\}$ if the j^th pixel belongs to the background, the parenchyma $M_{\mathbb{P}}$ or the vessels $M_{\mathbb{V}}$ respectively. The opening size r_c is defined by the surgeon and permits the delimitation of the background, and can take the form a circle or an ellipsoid.

We used the patch-based image analogy technique proposed by Li and Wand (2016). This technique relies on convolutional neural networks to transfer a texture from $\mathcal{J}$ to $\mathcal{I}$ constrained by their semantic labels ${\mathcal{J}}_{\text{label}}$ and ${\mathcal{I}}_{\text{label}}$ respectively. This method is inspired by the former work on neural style transfer introduced in Gatys et al. (2016) where an equilibrium between a content error and a style error is optimized. Because we are only interested in transferring the source style without keeping the target content, the optimization scheme (describe in details by Li and Wand (2016)) is reduced to minimize the style error solely. A example is illustrated in Figure 3

3. Results

We tested our method retrospectively on 4 patients dataset. These consisted of T1 contrasted MRI scans and microscopic images of the actual craniotomy. The cortical vessels, the parenchyma and the skull were segmented using 3D Slicer (Kikinis et al. (2014)) (see first column of Fig. 4). The brain parenchyma was meshed using CGal (The CGAL Project (2020)) in order to obtain a hexahedral volume of 9400 elements. We used the framework Sofa (Faure et al. (2012)) to simulate the deformations. Brain stiffness was set to 12000 Pa. The virtual camera was set with a focal f_cam = 500 at a distance d_cam = 25 cm of the brain surface. We first tested our method on 3 cases to obtain a predicted output image. For each case we used a different amount of CSF loss γ with the values 55%, 40% and 15% for case 1, case 2, and case 3 respectively. We also used and a different style image $\mathcal{J}$ for each case. Qualitative results are illustrated in Figure 4. We measured the dice coefficient between the generated image and the actual image with the values 85%, 77% and 92% for case 1, case 2 and case 3 respectively.

Figure 4.

Retrospective results on 4 patients. First column: pre-operative MRI scans with the region of interest framed in red around the cortical vessels. Second column: The output of the cortical deformation map is in the bottom right corner. Third column: Predicted microscopic image with random style transfer. Fourth column: ground truth microscopic image acquired intra-operatively. Row 1, row 2, row 3 and row 4 correspond to case 1, case 2, case 3 and case 4 respectively.

We further tested our method on case 4 with a maximal brain shift (γ = 95%). In this particular case the patient had previously undergone a carniotomy, so an estimate of the opening was possible from the pre-operative MRI scans. Using this opening, we generated an expected image that represents the position of the vessels without any brain shift w.r.t the segmented opening. We used our method with a maximal brain shift prediction and measured an average of 5.62 mm difference between the expected and predicted image. This error is used to rectify the opening position on patient’s skin and well as it’s size. Figure 5 gives an example of how our tool could be used to plan an optimal craniotomy (case 4). Surgeons could move the proposed craniotomy location around on the surface of the head and adjust its size while visualizing the predicted microscope view through the proposed craniotomy.

Figure 5.

Rectification of a craniotomy opening (case 4). From the pre-operative planing, the surgeon expects to see (a), the predicting output with brain shift permits to visualize (b), in order to obtain (c) surgeon can update the position and radius of the opening following the rectified target (d). The actual craniotomy view is shown in (d). The underlying tumor is represented by blue dashed lines.

4. Discussion

Usability in the Opertating Room:

In current clinical practice, neurosurgeons use registration software (e.g., BrainLab Fraser et al. (2009), Medtronic Bucholz and McDurmont (2009)) to register preoperative MRI to the patient before performing the craniotomy. With the help of an optical pointer, surgeons can then select a point on the patient’s head (shaved) and visualize the point in the preoperative images. The deformation model is a “maximal brain shift model” that takes the worst case scenario to build the stress map. This parameter can also be adjusted by the surgeon, but a maximal value will ensure the optimal exposure. Our method would simply extend the current practice by displaying a predicted synthetic microscopic image of the exposed brain with an optimized craniotomy opening at the selected point on the head. Our solution can thus easily be integrated into current clinical practice and can alse serve to position not only the craniotomy but also patient’s head.

Deep Brain Shift:

Although brain shift impacts underlying structures, the goal of our solution is to plan the optimal craniotomy given a maximal expected shift of the cortical surface (see Figure 5). While brain shift at the depth of the tumor is important for ensuring more complete resections, shift at the surface at the beginning of surgery impacts “access” to the tumor; if the brain has shifted and the craniotomy is too small or badly placed, the neurosurgeon may not have safe access to the tumor due the presence of cortical blood vessels or proximity to functional cortex. Our method integrates a rigid-to-deformable mapping (See Section 2.4) that permits to propagate cortical brain shift to underlying structures making it suitable for both cortical and sub-cortical deformation.

Visualization:

We proposed a neural style transfer method that can transfer 3 regions of a typical craniotomy, the parenchyma, the vessels and the background. While these 3 labels are enough, one can add the brain sulci as an additional label that can help surgeon’s correpond MRI scans with the synthetize image. Technically speaking, this amounts to update the label image ${\mathcal{J}}_{\text{label}}$ so that it accounts for an additional label that depicts the sulcis and segment the brain folds from the MRI scans.

5. Conclusion

We have addressed the little studied problem of optimizing a craniotomy opening for tumor resection and proposed a novel and versatile solution. We have introduced a parametric, physics-based deformation map to predict possible brain shift based on the amount of CSF loss and the gravitational head position, and used image generation to predict the view of the craniotomy that the surgeon will see through the microscope. Surgeons can use this approach to adjust the planned craniotomy based on predicted brain deformation images. Our retrospective experiments on patients data show good results in estimating brain deformation as well as predicting cortical microscopic images. Although our method does not provide a solution to compensation for brain shift, we strongly believe that our interactive tool can help surgeons in reducing patients’ brain exposure without disrupting the actual clinical routines.

Future work will integrates our tool into an end-to-end system capable of performing the pre- to intra-operative registration. This integration will permit us to include registration errors in the estimation of the deformation map since those errors can be as significant as the actual brain shift Frisken et al. (2019). In addition we plan to test our method on more date and to test it clinically to measure surgeons’ feedback.

Table 1.

Quantitative measurements on 4 patients.

	Case 1	Case 2	Case 3	Case 4
CSF Loss Amount (%)	55	40	15	95
Avg. Cortical Shift (mm)	3.21	2.32	1.23	5.62
Dice Coefficient (%)	85	77	92	81