Multiresolution image registration for multimodal brain images and fusion for better neurosurgical planning

Abstract

Background

Imaging modalities in medicine gives complementary information. Inadequacy in clinical information made single imaging modality insufficient. There is a need for computer-based system that permits rapid acquisition of digital medical images and performs multi-modality registration, segmentation and three-dimensional planning of minimally invasive neurosurgical procedures. In this regard proposed article presents multimodal brain image registration and fusion for better neurosurgical planning.

Methods

In proposed work brain data is acquired from Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) modalities. CT and MRI images are pre-processed and given for image registration. BSpline deformable registration and multiresolution image registration is performed on the CT and MRI sequence. CT is fixed image and MRI is moving image for registration. Later end result is fusion of CT and registered MRI sequences.

Results

BSpline deformable registration is performed on the slices gave promising results but on the sequences noise have been introduced in the resultant image because of multimodal and multiresolution input images. Then multiresolution registration technique is performed on the CT and MRI sequence of the brain which gave promising results.

Conclusion

The end resultant fused images are validated by the radiologists and mutual information measure is used to validate registration results. It is found that CT and MRI sequence with more number of slices gave promising results. Few cases with deformation during misregistrations recorded with low mutual information of about 0.3 and which is not acceptable and few recorded with 0.6 and above mutual information during registration gives promising results.

Background

At a glance commentary

Scientific background on the subject

Biomedical Images deals with different modalities involving different physics. Each modalities gives different information, which is complementary to each. In the proposed research, CT and MRI of the brain is acquired of the same patient and try to fuse both CT and MRI sequences. CT images provides better characterization of bone structure and MRI gives promising characterization of soft tissue.

What this study adds to the field

Proposed study involves image registration before fusion of CT and MRI data. Where image registration process bring both the data into same geometrical space. Fusion of both provides combined complementary information of CT and MRI respectively. Resultant fused sequence and volume rendering of the fused sequence helps in pre neurosurgical planning. Localization of the space occupying lesion and point to incision on the skull can be identified before surgery.

The clinical diagnosis and treatment of patients suffering from brain abnormalities usually require exhaustive exploration of biomedical images. However, images of a single modality do not provide a set of information. The inadequacy of clinical information makes a biomedical image of single modality insufficient for use in clinical interpretation and diagnosis of disease. In general, information acquired from images resulting from different modalities is complementary in nature. Fusion of multimodal images can be achieved after registration of two images. Registration process takes care of two images getting into on single alignment considering any one of the image as a reference. Considering the medical images, geometric deformation referred to the soft tissues behaviour during the interventional procedure or due to the over a period of time changes takes place in the tissue. Image registration has become more important in recent days. Image registration enables to integrate different images into one representation such that the complementary information can be accessed more easily and accurately. Multimodal images of the same person or of different persons generally differ by local geometric differences, and to map such images into one coordinate system, non-rigid or elastic transformations are required. Fused image data can improve medical diagnosis, surgery planning and simulation as well as intraoperative navigation.

Materials and methods

In proposed methodology as shown in Fig. 1, multimodal medical images, such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) are used. CT and MRI data of the brain are taken under the guidance and approval of Institutional Ethical Committee (IEC) Kasturba Hospital, Manipal. Eight CT and MRI data sets (each set belongs to same subject) have been collected and same used in the proposed methodology. Among eight scan sets, one scan set has 206 CT images with voxel size 0.46 × 0.46 × 1 mm and 220 MRI images with voxel size of 1 × 1 × 1 mm. Remaining scans have CT images (34–39 images in each case) and MRI images (23–25 images in each case) varying voxel size between 0.45 × 0.45 × 4.95 ∼ 0.49 × 0.49 × 6.31 mm, respectively. All the data involved in proposed methodology are used in Analyze format (with the extension.hdr and.img).

Fig. 1.

Proposed methodology.

CT and MRI data are pre-processed and saved into 8 bit analyze format from 16 bit representation. It is done because most the time open source toolkits like Insight segmentation and registration Toolkit (ITK) version 4.4.2 supports 8-bit data. Initially a single slice is used for deformable registration using different techniques and achieved promising result using BSpline deformable registration method. In proposed article it shows BSpline technique outcome on 3D sequence of CT and MRI data and further how 3D sequence of CT and MRI is used with multi resolution registration with rigid registration framework.

Image registration

Image registration is the process of determining the spatial transform that maps points from one image to homologous points on an object in the second image. The components of the registration framework and their interconnections are shown in Fig. 2. For n fusion of two dataset, image registration is the first step to be executed. Registration process makes sure that both the images to be fused are in same geometry.

Fig. 2.

The basic components of the registration framework. Courtesy: ITK User Guide.

Basic components of registration framework consists of two input images, metric, optimizer, transform and interpolator as shown in Fig. 2. Basic registration process has two input images. One image is considered as fixed image and other as moving image. Fixed image is the target image where moving image will get the geometry of fixed image. Always registration is treated as optimization problem while aligning two images into one geometry. In process it involves transformation to map point on moving image onto fixed image. Intensities on moving images are evaluated by interpolator and metric measures how well both the images are aligned to each other. Overall it is an iterative process, stops by the optimizer. There are many optimizer, metrics and different transformations, interpolators to implement during registration process. Always it depends on the type of the images used in the methodology. Images from different modalities give complementary information based on principles of physics. In proposed methodology we are using CT and MRI data of the human brain. Always CT gives better characterization of the rigid structures like bone and MRI provides better characterization of the soft tissues. When images are registered correctly, then corresponding anatomical regions overlap. Registration classification is based on modality, subject, objects transformations and many more [1]. Proposed methodology tends to use multimodal intra-patient registration and further classified based on transformation. Rigid transformation and non-rigid (deformable) transformation is used in proposed methodology. It is described briefly registration setup without thorough factors.

BSpline deformable registration

In proposed methodology BSpline deformable registration have used following components in the basic frame work Mattes Mutual Information Image to Image Metric, Regular Step Gradient Descent Optimizer/Limited memory Broyden–Fletcher–Goldfarb–Shanno (LBFGSB) Optimizer, BSpline Transformation and Linear Interpolator. Mattes mutual information metric is better suitable when multimodal images are involved in the registration process. Other metric like similarity metric can be used in mono-modal images. Metric measure plays a very important role in validating registration method. Regular Step Gradient Descent Optimizer fails to converge when maximum step is greater than 0.625 with maximum iterations 100. LBFGSB is a nonlinear algorithm. Results section below shows the outcome of BSpline deformable registration using both optimizers, respectively.

Rigid registration

It involves only translation and rotation during registration. In multimodal registration intensities are not relatively linear between two images. Based on the information of the pixels, similarity metric can be used, but in the case of multimodality similarity measures fails as relationship is not linear. Mutual information metric is used again in rigid registration. Rigid registration of images based on mutual information has been applied in a large number of papers [2]. Translation transformation is implemented with linear interpolator. Data set used in proposed methodology involves 3D sequence (set of slices) Multi-Resolution Registration (MRR) is performed using above mentioned components in registration framework. MRR is performed where images have fewer pixels by determining spatial mapping then at finer scale registration is initialized. Until finest possible scale is reached this process will be repeated. Pair of image pyramid is used at 3 levels defined by the user to down sample the both input images.

Fusion

Fusion of multimodal images is a process of combining data from two different modalities. Combining two images can be done by various methods. Before fusion basic stage involved is image registration. Once the images from different modalities are transformed into one geometrical alignment then fusion can be achieved easily. Since CT and MRI gives complementary information, combining both information into one new image gives the quality information together. CT bone window and registered MRI sequences of human brain is fused in proposed methodology. Outcome of fusion is shown in result section below.

Results

In the proposed methodology CT and MRI sequence is used in Analyze format. It consists of stack of images from 1 to many as shown in Fig. 3. CT sequence is considered as a fixed image and T2 FLAIR MRI sequence as moving image.

Fig. 3.

CT as fixed image (left) and MRI as moving image (Right). Courtesy: Kasturba Hospital.

Input sequences are shown as in Fig. 3. Fig. 3(A) shows first slice of the input sequences and Fig. 3(B) shows 34th and 33rd slice of the input sequences respectively. Both CT and MRI have 100 slices in respective sequence (this sequence is resampled from the data set having 200 and more images to 100 images). Fig. 4 shows BSpline deformable registration applied on CT and MRI slice, showing good result visually. Fig. 4(B) is the MRI image (moving image) gets transformed into the geometry of CT image. Fig. 4(A) results in registered image as seen in Fig. 4(C).

Fig. 4.

(A) CT brain as fixed image; (B) MRI brain as moving image; (C) Resultant MRI images. Courtesy: Kasturba Hospital.

Since medical image used in proposed methodology have a sequence of images, it is not recommended to process each and every images individually. So data sets are transformed into Analyze format and whole sequence is given as input for image registration. By looking the results shown in Fig. 4, 3D sequence (Analyze format) is given as input for the same method used. Fig. 5 shows the results of the same inputs used in Fig. 4. Because of the unsatisfactory results in proposed method, it is tried again by changing optimizer from Regular Step Gradient Descent Optimizer to LBFGSB Optimizer. Fig. 5 left shows the results of BSpline deformable registration using Regular Step Gradient Descent Optimizer. Fig. 5 right shows the results of BSpline deformable registration using LBFGSB Optimizer. Results shown in Fig. 5 are of different slices in the same sequence. Both the techniques with different optimizer produced noise. The reason behind generating noisy image is interpolation. Interpolator estimates the grey value of the resulting point. Linear interpolator is the popular techniques used in many articles. Pixel intensity is defined as the weighted combination of the intensities of its neighbours. The weights are linearly dependent on the distance between the point and its neighbours [2]. Some points of the new image have no match in the input images hence creates black holes like noises during registrations as shown in Fig. 5(A) and (B). By results shown in Fig. 5 it is learnt that, number of slices in two input sequences should be same and corresponding slices should match its anatomical structures respectively. Fig. 3(B) shows slice number 34 in CT sequence match with slice number 33 in MRI sequence. As mentioned before in BSpline implementation registration happened between slice numbered 1 to 1, 30 to 30, respectively. When anatomical structures differs in corresponding slices new image generated will be misregistered with noise as shown in Fig. 5. To overcome the issues faced, it was recommended in the article [3] to use multiresolution registration technique.

Fig. 5.

BSpline registration results using Regular Step Gradient Descent Optimizer (Left) and LBFGSB Optimizer (Right). Courtesy: Kasturba Hospital.

Eight cases have been used to perform multiresolution registration technique and feedback is taken from the radiologists, which will be discussed in the nest section. CT and MRI data sets collected from the PACS. MRI T2 flair brain sequence collected has 23 to 25 slices in a sequence and CT has 34 to 36 slices, respectively. For MRR, input data is in Analyze format. For the purpose of presenting data in this article, copy of it has been taken in the form of JPEG. Originally MRR is performed on Analyze format. In the previous section snapshot of the user interface window has been presented in this article for better understanding of Analyze format. In MRR if the number of slices in fixed image and moving image is not equal then in a new image generated as a resultant of MRR will have the same number slices as fixed image sequence. If fixed image has more number of slices then intermediate slices will be interpolated in the moving image data. So that registered image have the same number slices as fixed image. The two pyramids used in the techniques smoothens the images and subsamples the input images then registration initializes at each level. Present method 3 levels are defined. Fig. 6 shows CT, MRI and MMR results. In Fig. 6 slice numbered 10 of CT data corresponds to slice numbered 13 of MRI data considering anatomical structure of both modalities, Slice 14 is next consecutive slice of MRI data which should corresponds to slice 11 of CT data.

Fig. 6.

First row CT image as fixed image (left), MR images as moving image (Middle and right). Second row shows registered MRI images, slice 14 is interpolated between slice 13 and slice 15 which corresponds to Slice 10 and 11 in first row.

Fig. 6 shows the images interpolated in between the corresponding slices. As mentioned earlier CT and MRI sequences not have same number of slices, henceforth interpolation of intermediate slices takes place. In Fig. 7 CT have 206 slices and MRI have 220 slices. Since number of slices are large in Fig. 7, interpolation of slices between corresponding slices doesn't take place, instead few slices from MRI at the initial level will be removed automatically depending on the object information extracted from the corresponding slices, due to which shape based interpolation takes place in MRR [3].

Fig. 7.

First row CT image as fixed (left) slice 97, MR images as moving (Middle and right) slice 121 and 122. Second row shows registered MRI images, slice 97 and 98 corresponding to slice 121 and 122 in first row.

Resultant sequence from multiresolution registration is fused with CT data. Fig. 8, Fig. 9 show the fusion of CT bone window slices and corresponding registered MRI slices. Results shown in Fig. 8, CT and registered MRI have 206 slices. In Fig. 9, both CT and registered MRI have 35 slices. Fusion results shows how well the anatomical structures from CT data and MRI data are overlapped.

Fig. 8.

First row shows CT slice (left), Registered MRI slice (Middle) and Fused CT with registered MRI (Right) for the slice number 61. And second row shows the same for slice number 129. Total number of slices in this case are 206.

Fig. 9.

First row shows CT slice (left), registered MRI slice (Middle) and fused CT with registered MRI (Right) for the slice number 10. And second row shows the same for slice number 18. Total number of slices in this case are 35.

Discussion

In the proposed methodology BSpline registration and MRR is used. MRR gives promising results by looking at the data in previous section. Hence MRR results and fusion results are taken further to validate using metric measure and feedback from the radiologists. Eight CT and MRI human brain data sets are used during the study. One data set consists of 206 CT slices and 220 MRI slices, all other remaining data sets have CT slices varying between numbers 32–36 and MRI slices varying between numbers 23–25. Fig. 10 shows checker board for three cases among 8 with their respective metric value. Mutual Information (MI) metric [2] is used in proposed methodology where,

$$\begin{array}{c}0<=\mathrm{M<=1}\text{,}\\ \mathrm{Where}\mathrm{X},\mathrm{Y}\mathrm{and}\mathrm{Z}\mathrm{are}\dim \mathrm{ensions}\\ \mathrm{MI}=0,\mathrm{when}\mathrm{X}=\mathrm{Y}=\mathrm{Z}\\ \mathrm{MI}=1,\mathrm{when}\mathrm{X},\mathrm{Y},\mathrm{Z}\mathrm{are}\mathrm{independent}\mathrm{of}\mathrm{each}\mathrm{other}\text{.}\end{array}$$

Fig. 10.

Checker board after registration. Metric value of case 1 is 0.603, case 2 is 0.604 and case 3 is 0.402.

In Fig. 10 case 1 and case 2 have metric 0.603 and 0.604 respectively, which gives promising results. In case 3 there is a mis-registration with metric 0.402. Mutual information and joint entropy are computed for the overlapping parts of the images and the measures are therefore sensitive to the size and the contents of overlap [4]. Low value for the joint entropy is found due to complete mis-registration. These have high values when overlapping parts have anatomical structure and low value when they have only background. Transformations will be penalized by the entropies to balance the measure which decreases the MI measure in the resultant image. Misregistrations will result in a decrease in the MI measure [4]. For the case 1 shown in Fig. 10, MI of most of the intermediate slices where anatomical structures overlay measures to 1 and slices containing only background measures 0.11 and 0. Metric value of the MRR resultant sequence has MI measure of 0.60 and above which can be acceptable. Fig. 11 shows graphical representation of MI recorded for all the eight cases.

Fig. 11.

Mutual information plot of results using eight datasets.

Four radiologists gave feedback on the resultants of registration and fusion. Two or Three cases among eight cases are randomly picked by each radiologist to give feedback. Where a) registration results, b) loss during registration, c) acceptancy of deformation, if any, d) fusion of CT and registered MRI and e) usefulness of overall work, is evaluated by radiologists for marking between 1 and 5. Where 1 was the least and 5 was the maximum marking evaluated. Fig. 12 shows average marks given by individual radiologist for respective cases and Fig. 13 shows overall feedback of the radiologists. Radiologists' comments on the results are given below.

• •Registration and fusion of intra cranial and skull bones is good but facial bones and sinuses are not good.
• •CT pixels fall over grey and white matter of MRI in fused image.
• •Fusion images are good but there is loss of information in the facial planes
• •Alternative slice has good and poor resolutions, respectively.

Fig. 12.

Feedback from the radiologists on different cases selected randomly.

Fig. 13.

Overall feedback from the radiologists (out of 5).

Conclusion

Neurosurgical planning and image guided neurosurgery require the visualization of multimodal data obtained from various functional and structural image modalities, such as MRI, CT, functional MRI, so on and visualized using 2D slices and in some cases using a 3D volume rendering along with the functional imaging results. Visualizing the activation region effectively by still preserving sufficient surrounding brain regions for context is exceedingly important to neurologists and surgeons [5]. Better neurosurgical planning can be achieved by multimodal medical image registration and fusion.

In proposed methodology initially BSpline deformation registration gave visually good results for 2D slices and then applying same on 3D sequence noise is generated in the resultant which is not at all acceptable. MRR technique gives promising results on multimodal such as CT and MRI sequences. Eight CT and MRI data sets are used to validate MRR till date. And also radiologists feedback is taken based on quality of the resultant images, acceptancy of registered and fused resultant images and overall usefulness. Radiologists' points of view on the results are also mentioned in this article.

It is observed by the feedback and by metric measurement that CT and MRI is taken at different time period and data collected has very less number of slices with varying number of corresponding slices respectively. Only one data set has slices around 200 and above. Further it is recommended to use data set with more number of slices, so that interpolation of intermediate slices in the resultant image during MRR can be avoided, which in turn reduces noise and gives better results. Future work will be validating the proposed method on more cases and to 3D visualization of the fused volume for better neurosurgical planning. Resultant data can be used by navigation tools and virtual surgery systems for improved neurosurgical planning and simulations.

Conflicts of interest

No conflicts of interest.