3D Biological/Biomedical Image Registration with enhanced Feature Extraction and Outlier Detection

Abstract

In various applications, such as computer vision, medical imaging and robotics, three-dimensional (3D) image registration is a significant step. It enables the alignment of various datasets into a single coordinate system, consequently providing a consistent perspective that allows further analysis. By precisely aligning images we can compare, analyze, and combine data collected in different situations. This paper presents a novel approach for 3D or z-stack microscopy and medical image registration, utilizing a combination of conventional and deep learning techniques for feature extraction and adaptive likelihood-based methods for outlier detection. The proposed method uses the Scale-invariant Feature Transform (SIFT) and the Residual Network (ResNet50) deep neural learning network to extract effective features and obtain precise and exhaustive representations of image contents. The registration approach also employs the adaptive Maximum Likelihood Estimation SAmple Consensus (MLESAC) method that optimizes outlier detection and increases noise and distortion resistance to improve the efficacy of these combined extracted features. This integrated approach demonstrates robustness, flexibility, and adaptability across a variety of imaging modalities, enabling the registration of complex images with higher precision. Experimental results show that the proposed algorithm outperforms state-of-the-art image registration methods, including conventional SIFT, SIFT with Random Sample Consensus (RANSAC), and Oriented FAST and Rotated BRIEF (ORB) methods, as well as registration software packages such as bUnwrapJ and TurboReg, in terms of Mutual Information (MI), Phase Congruency-Based (PCB) metrics, and Gradiant-based metrics (GBM), using 3D MRI and 3D serial sections of multiplex microscopy images.

1. INTRODUCTION

Image registration, an important process in image processing, is essential for the analysis and interpretation of images acquired from diverse sources [19]. Image registration encompasses several steps, including feature detection, feature matching, mapping function design, and image transformation. Image registration has been used across various fields, including computer vision [29], pattern recognition [20], and machine learning [17]. In medical imaging, it provides a mechanism for comparing and integrating modalities such as MRI, CT, and PET, as well as temporal sequences, thus enhancing patient health analysis [8, 35]. Despite its significant potential, image registration confronts considerable challenges particularly in medical and microscopy images, such as Electron Microscopy (EM) [42], fluorescent microscopy [43], and multiplex imaging [37]. These challenges arise due to human anatomy diversity [31], variability in biological samples [15], non-linear tissue deformation introduced during mechanical processing steps [7], and issues related to image quality, such as noise and resolution discrepancies [26]. Yet, the importance of image registration, particularly in 3D registration of serial-section microscopic images, is undeniably significant [41]. Its application extends to neuroscience for mapping neural circuits [14], histopathology for identifying cellular abnormalities [38], and developmental biology for studying morphological changes during growth [6].

In this study, we propose an innovative image registration algorithm for 3D medical imaging like brain MRIs and z-stack (3D) microscopy images. Our methodology combines two well-regarded feature extraction methods: Scale-Invariant Feature Transform (SIFT) [28], and Residual Network with 50 layers (ResNet50) [18], for more effective and comprehensive image registration. SIFT excels at keypoint detection across scales and orientations, while ResNet50 complements SIFT by learning high-level features, proving advantageous for complex data such as medical and biological images. This fusion results in a robust feature vector capable of handling a wide array of image complexities. We present an adaptive outlier rejection technique by considering the Maximum Likelihood Estimation SAmple Consensus (MLESAC) [49] concept for controlling potential outliers during registration. MLESAC and adaptive thresholding have been utilized in other contexts, but to our knowledge, this is the first study to apply them for outlier rejection of features extracted by both SIFT and ResNet50 in the context of 3D image registration. We validate our algorithm on 3D MRI and 3D serial sections of multiplex microscopy images and compare its performance with several commonly used registration methods in terms of Mutual Information (MI) [36], Gradient-Based Metric (GBM) [50], and Phase Congruency-based Metric (PCB) [23]. The results show promising performance and potential for future advancements in medical and biological image registration.

Our contributions to this paper can be listed as follows:

• We present a novel feature extraction method that employs both SIFT and deep learning techniques (via ResNet50) to accomplish effective image representation. Although both of these techniques have been used independently, their integration in this manner represents a new approach to feature extraction in the context of 3D medical image registration.

• We introduce an adaptive technique for rejecting outliers that modifies the MLESAC method. This novel application of established methodologies constitutes a contribution to the field.

• We validate the proposed methods by applying them to highly complex 3D multiplex microscopy images and brain MRI images.

2. RELATED STUDIES

In image registration, diverse methods have been proposed as a result of several research efforts. For instance, Fan et al. [11] used a deep learning approach to predict deformations. However, a common limitation of deep learning-based models, including this one, is their heavy dependence on pre-existing registration images. In particular, these models require large amounts of annotated images for training, which can often be challenging to obtain or generate, thus becoming a significant constraint on their applicability in various contexts. On the other hand, Nan et al. [33] introduced a morphology-based method for brain image registration that does not rely on deep learning for feature extraction. Zhou et al. [51] proposed an unsupervised deep learning method for serial electron microscopy image registration, prioritizing speed and precision at the expense of adaptability and robustness. Our proposed methods, on the other hand, combine the strengths of SIFT and ResNet50 feature extraction with an advanced outlier rejection method to ensure robustness when applied to a variety of imaging modalities. Kajihara et al. [21] developed a method for 3D reconstruction from serial section images, focusing on resolving structural issues post-feature extraction. Our approach focuses on enhancing feature point quality and matching with a robust feature extraction mechanism and an innovative outlier rejection process to detect and address potential structural problems in advance.

3. METHODS AND DATASETS

This study presents a registration algorithm comprising five core stages: image preprocessing, feature extraction, feature matching, outlier rejection, and image transformation. The proposed method, whose flowchart is demonstrated in Fig. 1, is designed to optimize image registration performance in terms of accuracy, adaptability, and robustness against noise and distortion. The following sections provide a detailed implementation of these stages and discuss the datasets used for evaluating the effectiveness of our image registration approach.

Figure 1:

Flowchart illustrating the stages of the proposed image registration method.

3.1. Preprocessing

The preprocessing stage is essential because it lays the foundation for more precise feature extraction and outlier rejection, resulting in more precise image registration. The preprocessing step includes adjusting brightness and contrast, denoising, and preparing images for the ResNet50 model.

3.1.1. Brightness and Contrast Adjustment.

This process involves a linear transformation applied to an image. Given an image $I$, the outcome is an adjusted image $I^{\prime }$ as follows:

$$I^{\prime }=\alpha \cdot I+\beta ,$$ (1)

where, $\alpha$ is a scaling, and $\beta$ is an additive factors. The $\alpha$parameter modifies the image’s contrast, while the $\beta$ modifies the image’s brightness. By adjusting these two parameters, we can effectively modify the image’s contrast and brightness.

For our datasets, we set $\alpha$ and $\beta$ as follows: for 3D serial-section microscopic images 1, we chose $\alpha =2.0$ and $\beta =60$ to enhance image details; for images 2, $\alpha =1.0$ and $\beta =20$ was found suitable given their inherent contrast and brightness; for CUMC12, we used $\alpha =1.0$ and $\beta =0.0$, as these images were already well contrasted. These choices, made after visual inspection, aimed to highlight image details and align image intensities across datasets

3.1.2. Image Denoising.

We used the Non-Local Means algorithm [5] for denoising due to its ability to preserve key details while reducing noise. It averages similar image fragments. The weight parameter controls the denoising level. Higher weights result in more denoising but reduced detail. Thus, we carefully adjusted the weight values to balance noise reduction and detail preservation.

3.2. Feature Extraction

In our proposed method, we leverage both deep learning models and conventional feature extraction algorithms to generate a robust and exhaustive feature set. This process involves three steps. Initially, we considered the ResNet50[18] model to extract high-level image features. SIFT[28], a conventional yet effective algorithm, is then applied to both the preprocessed images and the ResNet50-generated feature maps. This dual application of SIFT enhances and expands the overall set of features. These two sets of features are then concatenated to form our final, comprehensive feature set.

3.2.1. ResNet50.

Convolutional Neural Networks (CNNs) [27], a type of deep learning model, excel at extracting hierarchical features from images. Each network layer corresponds to an abstractness level. In this study, we use ResNet50, a well-known CNN model, for initial feature extraction. ResNet50, which is based on the principle of learning residual functions, reduces the problem of vanishing gradients and enables the training of deeper networks. ResNet50 is utilized to recognize complex image structures and patterns using high-level features. This is the initial step of our composite feature extraction method, paving the way for the deployment of the SIFT algorithm.

In our study, we used the pre-trained ResNet50, initially trained on ImageNet [10], to extract image features. We fine-tuned the ResNet50 model using a custom dataset created specifically for our task with serial section images. This dataset consists of transformed versions of images from the 3D serial-section microscopic images 1 and 2, which are further detailed in the 3.6 section of this paper Each image underwent random affine [25], rigid [9], rotated [1], and translated [47] transformations, expanding the dataset to 10,000 unique images. This variety allowed the model to adapt to the complexity and variations of real-world images. The model’s last modification was replacing the final fully connected layer with a regression head for predicting pixel intensity values. During the fine-tuning, the Mean Squared Error (MSE) loss was used as the loss function, defined as:

$$L\left(y,y^{\prime }\right)=\frac{1}{N}\sum {\left(y_i-y_i^{\prime }\right)}^2,$$ (2)

where $y_i$ and $y_i^{\prime }$ represent the true and predicted pixel intensity values, respectively, and $N$ is the total number of pixel intensities. Using Stochastic Gradient Descent (SGD) [4] with a learning rate of 0.001 and a momentum of 0.9, the model parameters were optimized. Two epochs were used to fine-tune the model, ensuring that it had learned to represent our specific images while retaining the ability to extract high-level features.

Utilized as a feature extractor for serial section images, our refined ResNet50 model processes the input through multiple layers, effectively extracting hierarchical features. The final layer, a regression head, corresponds to the input dimensions, producing an image with high-level features and reduced noise. In this configuration, the ResNet50 model $R$ processes an input preprocessed image $I^{\prime }$, producing an output image $O$:

$$O=R\left(I^{\prime }\right),$$ (3)

here, $O$ is a feature map of the same size as the input image (224 × 224 × 3). Each convolutional layer can be defined as a function of the input preprocessed image $I^{\prime }$ or the output from the previous layer. Hence, the output from the $kth$ layer (or the $kth$ feature map) can be denoted as $F_k$:

$$F_k=\left\{\begin{array}{ll}{C}_k\left(I^{\prime }\right),& \text{if}k=1\left(\text{input}\text{layer}\right)\\ C_k\left(F_{k-1}\right),& \text{if}k>1\left(\text{for}\text{other}\text{layers}\right)\end{array},\right.$$ (4)

here, $C_k$ is defined as a combination of convolution operation *, activation function $f$ like Rectified Linear Unit (ReLU) [24], possible pooling $p$, and normalization $n$. Therefore:

$$C_k=\left\{\begin{array}{ll}n\left(p\left(f\left(W_k*I^{\prime }+b_k\right)\right)\right),& \text{if}k=1\\ n\left(p\left(f\left(W_k*F_{k-1}+b_k\right)\right)\right),& \text{if}k>1\end{array}\right.,$$ (5)

where, $W_k$ is the weight matrix of the $kth$ layer, and $b_k$ is the bias of the $kth$ layer. We denote the ResNet50 model $R$ as a function that maps our preprocessed input image $I^{\prime }$ to a high-level feature map $O$. This mapping is achieved through a sequential operation of $L$ layers in the model:

$$O=R\left(I^{\prime }\right)=C_L\circ C_{L-1}\circ \dots \circ C_1\left(I^{\prime }\right),$$ (6)

Here, $C_n\left(I^{\prime }\right)$ represents the operation performed by the $n$-th layer of the model on the input $I^{\prime }$, and $L$ denotes the total number of layers in the model. The symbol ∘ denotes function composition, indicating that the output of one layer is fed as the input into the next, in a sequence from $C_1$ through $C_L$. In other words, the image $I^{\prime }$ is processed through the layers sequentially, with each layer’s output becoming the subsequent layer’s input.

We adjusted the final layer’s stride parameters in ResNet50 to maintain the spatial dimensions from input to output. Instead of the default stride of 2, we used a stride of 1, resulting in a high-resolution feature map without reducing feature extraction strength. This modified ResNet50 yields an output of the same size as the input, capturing key high-level features. Then, we incorporate these features into our hybrid pipeline and later add SIFT features to complete it.

3.2.2. SIFT.

SIFT, a robust and dependable algorithm, is recognized for identifying and describing local image features. It operates by computing distinct keypoints and their descriptors in an image, identified by locating the maxima and minima of a Difference of Gaussians (DoG) across scales. Each key point’s descriptor, a unique identifier that captures local image gradient directions, is a 128-dimensional vector resistant to variations in illumination, rotation, and scale. The process can be described with the following steps:

• Scale-space Extrema Detection

• Key Point Localization

• Orientation Assignment

• Key Point Descriptor

The following describes the mathematical representation of the descriptor computation of a keypoint: Let $I(x,y)$ be an image and $G(x,y,\sigma )$ be a variable-scale Gaussian. $L(x,y,\sigma )$, the scale-space of an image is produced by the convolution of $I(x,y)$ with $G(x,y,\sigma )$:

$$L\left(x,y,\sigma \right)=G\left(x,y,\sigma \right)*I\left(x,y\right),$$ (7)

where * denotes the convolution operation. Then the DoG $D(x,y,\sigma )$ is computed from the difference of two nearby scales separated by a constant multiplicative factor $k$:

$$D\left(x,y,\sigma \right)=L\left(x,y,k\sigma \right)-L\left(x,y,\sigma \right),$$ (8)

where, this function provides a close approximation to the scale-normalized Laplacian of Gaussian, which has been demonstrated to be valuable for blob detection [30] in 2D images.

SIFT Feature Extraction on the Original Image:

In this step, we explicitly apply the SIFT algorithm to the input images that have been preprocessed. After preprocessing, the image quality is improved and the noise is reduced. Executing SIFT on these images provides significant spatial and structural features within the images. This term describes:

$${\mathit{\text{SIFT}}}_o=\mathit{\text{SIFT}}\left(I^{\prime }\right),$$ (9)

where $SIF{T}_o$ are the features extracted from the original preprocessed image $I^{\prime }$.

SIFT Feature Extraction on ResNet50 Features:

In the next step, we apply the SIFT algorithm to the high-level feature maps that our fine-tuned ResNet50 model has generated. By applying SIFT to the feature maps, it is possible to not only capture low-level image details but also high-level semantic features. This step provides us with a second, complementary set of SIFT features that improve the robustness and discriminatory capacity of our overall feature set:

$$SIF{T}_r=\mathit{\text{SIFT}}\left(O\right),$$ (10)

where ${\mathit{\text{SIFT}}}_r$ are the features extracted from the ResNet feature map $O$. By relying on SIFT in both the original preprocessed images and the ResNet50 feature maps, we are able to capture a wide range of features. These two feature sets represent different aspects of the images: ${\mathit{\text{SIFT}}}_o$ features are more closely associated with raw image data, capturing local intensity patterns and gradients, whereas ${\mathit{\text{SIFT}}}_r$ features are derived from high-level feature maps and incorporate more abstract, contextually rich information about the images.

Features Concatenation:

In the final step of our feature extraction procedure, the ${\mathit{\text{SIFT}}}_o$ and $SIF{T}_r$ features are concatenated. This yields a comprehensive set of features that incorporates both high-level semantic information and local image details, providing a solid foundation for subsequent image matching. The procedure of concatenation can be described as:

$${\mathit{\text{SIFT}}}_f={\mathit{\text{SIFT}}}_o\parallel {\mathit{\text{SIFT}}}_r,$$ (11)

where $SIF{T}_f$ denotes the final concatenated feature set. Using the capabilities of deep learning models and traditional feature extraction techniques, this combined feature set improves the performance and reliability of our subsequent image matching and transformation stages.

3.3. Feature Matching

Feature matching crucially determines image correspondence for geometric alignment in image registration. Our method utilizes two strategies: a Fast Library for Approximate Nearest Neighbors (FLANN)-based matcher [32] and a Brute-Force matcher, each suited to specific situations.

3.3.1. FLANN Matcher.

The FLANN matcher uses tree-based techniques like a k-dimensional tree (KD-Tree) [3] or K-Means tree [34] for efficient feature correspondence. It quickly locates the nearest neighbors for a given feature. Essentially, for each feature $f1$ in $SIF{T}_{f1}$ from the preprocessed image $I^{\prime }1$ and $SIF{T}_{f2}$ from $I^{\prime }2$, the FLANN matcher finds a corresponding feature $f2$ in $SIF{T}_{f2}$ minimizing the Euclidean distance in the feature space.

$$f2=\underset{f^{\prime }\in {\text{SIFT}}_{f2}}{\text{argmin}}{‖f1-f^{\prime }‖}_2.$$ (12)

This process is repeated for all features in $SIF{T}_{f1}$, generating a set of matches $M_{\text{Flann}}$.

3.3.2. Brute-Force Matcher.

FLANN is a highly efficient method, particularly for large datasets, but it is approximative and may miss the finest matches on occasion. A brute-force matcher can be a feasible alternative in such situations. The brute-force matcher evaluates all possible matches and consistently identifies the best match, despite the fact that it can be computationally expensive. The brute-force matching process can be described as follows: For each feature $f1$ in $SIF{T}_{f1}$, the brute-force matcher computes the Euclidean distance to all features $f^{\prime }$ in $SIF{T}_{f2}$, and selects the one with the minimum distance as the match like Equation 12 to yield a set of matches $M_{BF}$.

3.4. Outlier Rejection

Outlier rejection is crucial to ensuring the quality of the matched features after identifying feature matches. In this study, MLESAC and adaptive thresholding is utilized to reject outliers.

3.4.1. Adaptive Thresholding.

At this step, our aim is to consider an adaptive threshold to divide ideal and poor matches. This adaptive threshold is computed as a multiple of the median distance of the feature matches. If $M$ is the set of distances of all feature matches and $m_d$ is the median distance, the adaptive threshold $T_a$ is given by:

$$T_a=m_d\times \alpha ,$$ (13)

where $\alpha$ is a constant factor. In this scenario, the value of $\alpha$ is set to 5 based on the trial and error and it depends on the characteristics of the dataset.

3.4.2. MLESAC for Outlier Rejection.

MLESAC is a probabilistic method for robust parameter estimation that is widely employed due to its proficiency with outliers. It operates under the assumption that the data consists of inliers, or data whose distribution can be explained by a given set of model parameters, and outliers, or data that do not fit the model. In the context of image registration, the term model refers to a homography matrix [16] that describes the geometric transformation between two images, as will be described in Section 3.5. In our case, we use MLESAC to estimate the homography matrix and identify the inlier matches. We denote $M$ as the set of matches and $m$ as an individual match, with the source point as $m_{src}$ and destination point as $m_{dst}$. MLESAC operates iteratively. In each iteration, it randomly selects a subset of the matches (a minimum of 4), calculates the homography matrix $H_{\mathit{\text{temp}}}$ [16] using these matches, and then computes the error for all matches against $H_{\mathit{\text{temp}}}$. The error $E_i$ for each match $m_i$ is calculated as follows:

$$E_i={∥m_{i_{\text{dst}}}-H_{\text{temp}}\left(m_{i_{\text{src}}}\right)∥}_2.$$ (14)

The likelihoods for each match are computed using a Gaussian error function, with $\sigma$ as the standard deviation, after the errors have been calculated. If $E_i$ is the error for match $m_i$, then $L_i$ is calculated as follows:

$$L_i=\text{exp}\left(-\frac{E_i^2}{2{\sigma }^2}\right).$$ (15)

After that, inliers are recognized as matches with error less than the adaptive threshold, and outliers are those with error greater than the adaptive threshold. The total likelihood $L_{\mathit{\text{total}}}$ for this iteration is the sum of the likelihoods of all matches:

$$L_{\text{total}}=\sum _{i\in M}{L}_i.$$ (16)

The iteration with the highest total likelihood yields the best model and a set of inlier matches. The final homography matrix $H_{\mathit{\text{best}}}$ corresponds to this model.

3.5. Image Transformation

Having the estimated and validated homography matrix that was computed in Section 3.4, the next step is to apply this transformation to our images, particularly the moving image. This method, known as image warping, enables us to align the moving image with the fixed image using the homography matrix. The transformation is applied using the following equation:

$$I_m^{\prime }=H_{\mathit{\text{best}}}*I_m,$$ (17)

where $I_m^{\prime }$ is the transformed or “warped” moving image, $I_m$ is the original moving image, and $H_{\mathit{\text{best}}}$ is the homography matrix estimated in 3.4. In this case, the homography matrix $H_{\mathit{\text{best}}}$ is defined as a 3 × 3 matrix, and it transforms a point in coordinates $(x,y)$ in the moving image to a new point $\left(x^{\prime },y^{\prime }\right)$ in the warped image using the following homogeneous coordinates equation:

$$\left[\begin{array}{l}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right]=H_{\mathit{\text{best}}}*\left[\begin{array}{l}x\\ y\\ 1\end{array}\right],$$ (18)

where, in expanded form, this can be written as:

$$\begin{gathered} x^{\prime }=\frac{h_{11}*x+h_{12}*y+h_{13}}{h_{31}*x+h_{32}*y+h_{33}} \\ {y}^{\prime }=\frac{h_{21}*x+h_{22}*y+h_{23}}{h_{31}*x+h_{32}*y+h_{33}}, \end{gathered}$$ (19)

where $h_{ij}$ denotes the elements of the homography matrix $H_{\mathit{\text{best}}}$. The denominators ensure that the transformation is normalized. With the conclusion of this image transformation step, the moving image has been effectively aligned with the fixed image.

3.6. Dataset

We evaluate the efficacy of our proposed image registration method using three real datasets as well as simulated datasets. The datasets comprise a variety of imaging conditions, each presenting its own challenges and characteristics.

3.6.1. Real Datasets.

• CUMC12: The CUMC12 dataset consists of 18 images acquired at Columbia University Medical Center using a 1.5 T GE scanner [22].

• 3D serial-section microscopic images 1: This dataset includes 10 multichannel fluorescent images at 100× magnification from the University of Connecticut Advanced Light Microscopy Facility. The rat amygdala samples underwent fixation, embedding, and ultra-thin sectioning. The resultant images are in TIF format.

• 3D serial-section microscopic images 2: This second dataset is similar but has images at 10× magnification, stored in PNG format. Both sets are captured using the same Andor microscope, illustrating consistency in image acquisition despite differences in magnification and storage format.

3.6.2. Synthesized datasets.

We created synthetic datasets from the real ones to evaluate our model’s registration capabilities under diverse image transformation scenarios. One image was randomly selected from each real dataset and altered to emulate real-world misalignment scenarios from various factors like camera angle alterations (rotation), distance variations (translation), and intricate distortions [25] and rigid transformations. Fig. 2 visually showcases these diverse transformations.

Figure 2:

Synthetic images illustrating various transformations applied to real 3D serial-section microscopic and MRI images.

3.7. Code Availability

We have made the code we developed and used in this study publicly accessible. You can find it at https://github.com/NabaviLab/Biological-Biomedical-Image-Registration. The repository includes all scripts and relevant files necessary to reproduce the findings we present in this paper.

4. RESULTS AND DISCUSSION

We evaluate our method’s performance on the selected datasets and compare it to state-of-the-art methods. Results start with visual and quantitative analysis of synthetic data. We then present visual and quantitative results from real-world applications. This allows us to demonstrate our method’s efficacy across various imaging modalities and conditions.

4.1. Evaluation metrics

We evaluate the performance of our model using standard image alignment quality assessment metrics described bellow. The assessment involves contrasting the registered image with the fixed image based on the evaluation parameters. This comparison allows us to evaluate the efficacy of the model in image registration tasks

MI: MI measures the statistical dependency between the pixel intensities of two images and is defined as follows:

$$I\left(A;B\right)=\sum _{a\in A}\sum _{b\in B}p\left(a,b\right)\text{log}\left(\frac{p\left(a,b\right)}{p\left(a\right)p\left(b\right)}\right),$$ (20)

where, $I(A;B)$ represents the mutual information between images $A$ and $B$. The two summations ${\sum }_{a\in A}{\sum }_{b\in B}$ denote the sum over all possible intensity values $a$ in image $A$ and $b$ in image $B$. $p(a,b)$ is the joint Probability Density Function (PDF) of the intensity values $a$ and $b$ in images $A$ and $B$, respectively. $p\left(a\right)$ and $p\left(b\right)$ represent the marginal PDFs of the intensity values $a$ and $b$ in images $A$ and $B$, respectively. $\text{log}\left(\frac{p(a,b)}{p\left(a\right)p\left(b\right)}\right)$ is the logarithm of the ratio of the joint PDF to the product of the marginal PDFs, which is a measure of how much the intensities in the two images depend on each other. Higher MI scores indicate better alignment.

GBM: GBM evaluates image match quality based on gradient orientations in images $A$ and $B$. Higher GBM values, computed using Equation 21, signify better image matching.

$$GBM\left(A,B\right)=\frac{1}{N}\sum _{i=1}^N\text{cos}\left({\theta }_{A,i}-{\theta }_{B,i}\right),$$ (21)

where $N$ is the total number of pixels in each image, and ${\theta }_{A,i}$ and ${\theta }_{B,i}$ are the gradient orientations at pixel $i$ in image $A$ and $B$, respectively.

PCB:PCB quantifies the degree to which the Fourier components of an image are in phase. A common approach to calculate phase congruency is based on the sum of the phase deviations divided by the sum of the amplitudes.

$$PCB\left(A,B\right)=\frac{\sum \left|F\left(A\right)-F\left(B\right)\right|}{\sum \left|F\left(A\right)\right|+\left|F\left(B\right)\right|},$$ (22)

here, $F\left(A\right)$ and $F\left(B\right)$ represent the Fourier Transform of images [13] A and B. Higher PCB values indicate a better alignment of features in the two images.

4.2. Baseline methods

We compared our proposed model to three benchmark algorithms: SIFT, SIFT with Random Sample Consensus (RANSAC) [12], and Oriented FAST and Rotated BRIEF (ORB) [39]. The comparison with SIFT underscores the importance of the MLESAC-based outlier rejection in our framework, while the inclusion of SIFT with RANSAC further highlights this point. ORB, noted for its efficiency but lacking explicit outlier rejection, provides another comparison metric. We also benchmark against two widely-used software packages: bUnwarpJ [2, 48] and TurboReg [40, 44–46], whose wide-spread use attests to their robustness. This comparison evaluates the performance of our method, reveals its real-world applicability, and highlights its potential utility across datasets.

4.3. Results

In this Results section, we evaluate the performance of our proposed method against those of the established baseline models. We analyze its effectiveness using both synthesized and real-world data, highlighting its registration performance, outlier rejection, and versatility across various imaging contexts.

4.3.1. Results on Synthesized data.

Fig. 3 (A) displays the results of the registration of synthetic rigid images for the CUMC12 dataset. Using the known transformation applied to the fixed image to generate the synthetic moving image provides a metric for evaluating the registration accuracy. The results demonstrate that our proposed method is highly effective in successfully managing the applied transformation. In contrast, the SIFT method, which lacks an algorithm for rejecting outliers, and ORB fail to register the images, resulting in distortions. Despite their attempts to align the images, SIFT with RANSAC do not achieve a complete and successful registration, demonstrating their inability to manage registration based on the known transformation.

Figure 3:

Comparative visualization of registration results on CUMC12 synthetic dataset and real dataset with various methods.

Table 1 demonstrates image registration performance for synthetic images. The proposed method’s consistently high MI scores across transformations and datasets suggests reliable performance. With a MI score of 3.91 for affine transformation, it excels in the Stack serial-section microscopy images 2 dataset. High GBM scores across various transformations and datasets underscore its effectiveness in capturing structural details. High PCB scores, especially a score of 0.92 for translated transformation in the Stack serial-section microscopy images 2 dataset, show that the proposed method also possesses excellent alignment precision. Thus, the proposed method outweighs traditional algorithms and other software packages in all evaluation metrics, proving its superior image registration capability.

Table 1:

Quantitative comparison of registration results for synthetic data

		SIFT			SIFT with RANSAC			ORB			bUnwrapJ			TurboReg			Proposed Method
Dataset	Transformation/Evaluation	MI	GBM	PCB	MI	GBM	PCB	MI	GBM	PCB	MI	GBM	PCB	MI	GBM	PCB	MI	GBM	PCB
Stack serial-section microscopic images 1	Affine	0.19	0.02	0.47	0.18	0.02	0.45	0.30	0.02	0.37	1.24	0.14	0.88	0.09	0.07	0.36	1.76	0.87	0.97
	Rigid	0.10	0.10	0.37	0.14	0.07	0.36	0.4	0.02	0.12	0.10	0.10	0.37	0.14	0.07	0.36	1.18	0.84	0.87
	Rotated	0.04	0.05	0.34	0.25	0.08	0.39	0.4	0.01	0.45	0.04	0.05	0.34	0.25	0.08	0.39	1.01	0.79	0.86
	Translated	1.15	0.15	0.83	0.09	0.07	0.36	0.4	0.02	0.12	1.15	0.15	0.83	0.09	0.07	0.36	1.16	0.82	0.90
Stack serial-section microscopic images 2	Affine	0.28	0.01	0.57	0.28	0.01	0.57	0.29	0.01	0.54	0.35	0.01	0.57	0.32	0.07	0.59	3.91	0.97	0.92
	Rigid	0.34	0.02	0.45	0.33	0.01	0.58	0.30	0.02	0.44	0.33	0.03	0.43	0.30	0.01	0.58	1.93	0.74	0.80
	Rotated	0.32	0.01	0.57	0.30	0.09	0.59	0.40	0.01	0.47	0.32	0.01	0.57	0.30	0.09	0.59	1.04	0.82	0.85
	Translated	0.20	0.05	0.57	0.11	0.01	0.56	0.24	0.03	0.50	0.83	0.05	0.91	0.11	0.01	0.56	2.04	0.97	0.92
CUMC12	Affine	0.10	0.42	0.56	0.10	0.42	0.56	0.12	0.44	0.58	1.54	0.95	0.56	0.24	0.38	0.51	2.35	0.96	0.52
	Rigid	0.09	0.48	0.62	0.12	0.41	0.57	0.14	0.48	0.58	0.94	0.72	0.55	0.26	0.39	0.51	1.53	0.94	0.57
	Rotated	0.50	0.50	0.58	0.55	0.49	0.57	0.60	0.61	0.50	0.89	0.70	0.54	0.60	0.49	0.58	1.54	0.97	0.56
	Translated	0.17	0.36	0.50	0.14	0.36	0.50	0.15	0.20	0.50	0.71	0.68	0.56	0.26	0.39	0.52	2.35	0.96	0.56

4.3.2. Results on real data.

We tested our model on stack-serial sections of microscopic datasets to evaluate its performance with real data. Fig. 3 (B) presents registration results between a random pair of Stack Microscopy 2 images. The SIFT registration algorithm, although useful, exhibits issues with distortion, primarily due to its sensitivity to transformations and noise. SIFT with RANSAC, however, improves upon these results, underscoring RANSAC’s role in outlier rejection. However, despite mitigating distortion, it struggles to correctly align images. ORB failed across trials in image registration, producing a highly distorted result due to its challenges with scale and rotation changes. In contrast, our proposed method offers the most accurate results, proficiently handling various transformations due to its comprehensive approach, advanced features, and robust matching algorithms. Thus, it effectively resolves distortion issues and ensures precise alignment under challenging conditions.

Table 2 shows a comparison of our method with existing algorithms and software packages, based on a random pair from each dataset. Across datasets, our method consistently performs better, as evidenced by higher MI scores: 0.45 in Stack serial-section microscopic images 1, surpassing SIFT with RANSAC (0.04), ORB (0.04), and TurboReg (0.11); and 1.61 in CUMC12. It also excels in the GBM metric, crucial for capturing structural nuances, achieving a score of 0.51 in the Stack serial-section microscopic images 2 dataset, and 0.97 in the CUMC12 dataset. Furthermore, it demonstrates superior alignment precision, indicated by higher PCB scores across datasets: 0.50 in Stack serial-section microscopic images 1, 0.61 in 2, and 0.64 in CUMC12. Thus, the proposed method outperforms other methods across all evaluation metrics, emphasizing its potential for image registration in random pair scenarios.

Table 2:

Quantitative comparison of registration results for random pair images

	Stack serial-section microscopic images 1			Stack serial-section microscopic images 2			CUMC12
	MI	GBM	PCB	MI	GBM	PCB	MI	GBM	PCB
SIFT	0.01	0.24	0.44	0.21	0.01	0.45	0.10	0.53	0.60
SIFT with RANSAC	0.04	0.11	0.33	0.24	0.01	0.47	0.15	0.45	0.59
ORB	0.04	0.12	0.33	0.21	0.02	0.47	0.13	0.41	0.55
bUnwrapJ	0.04	0.05	0.31	0.36	0.42	0.55	0.96	0.75	0.59
TurboReg	0.11	0.07	0.32	0.32	0.04	0.60	0.29	0.41	0.54
Proposed Method	0.45	0.48	0.50	0.31	0.51	0.61	1.61	0.97	0.64

Image registration, crucial in serial image processing, organizes images in a sequence. It treats the first image as a fixed image, and the next image adjusts to align with it. After aligning, the registered image becomes the new fixed image, and the process repeats with the following images. To evaluate its effectiveness, we compared our method against algorithms using the first serial section microscopy image.

Table 3 compares the performance of various image registration algorithms on nine pairs of serial section microscopy images 1. SIFT fails to analyze the fourth pair, demonstrating a limitation in sequential tasks. SIFT with RANSAC processes all pairs, but with modest scores. ORB fails from the second pair onwards, highlighting its limitations with complex transformations. bUnwrapJ and TurboReg process all pairs, but their MI, GBM, and PCB scores are lower than our method, reflecting limited capture of key details. Our method detected every pair and consistently outperformed others, effectively handling transformations and noise levels, demonstrating its robustness for sequential image registration tasks in serial section microscopy images.

Table 3:

Sequential registration performance comparison of the proposed method with other state-of-the-art methods across nine pairs of serial section microscopy images 1

	SIFT			SIFT with RANSAC			ORB			bUnwrapJ			TurboReg			Proposed Method
	MI	GBM	PCB	MI	GBM	PCB	MI	GBM	PCB	MI	GBM	PCB	MI	GBM	PCB	MI	GBM	PCB
Pair 1	0.03	0.13	0.35	0.03	0.32	0.35	0.04	0.68	0.46	0.05	0.13	0.32	0.05	0.04	0.33	0.47	0.50	0.49
Pair 2	0.11	0.18	0.41	0.17	0.14	0.37	Failed	Failed	Failed	0.08	0.13	0.30	0.10	0.05	0.38	0.49	0.58	0.50
Pair 3	0.04	0.39	0.49	0.05	0.11	0.34	Failed	Failed	Failed	0.10	0.18	0.30	0.47	0.49	0.60	0.64	0.59	0.53
Pair 4	Failed	Failed	Failed	0.04	0.14	0.38	Failed	Failed	Failed	0.33	0.08	0.41	0.43	0.58	0.67	0.80	0.65	0.57
Pair 5	Failed	Failed	Failed	0.06	0.16	0.36	Failed	Failed	Failed	0.10	0.05	0.29	0.22	0.68	0.75	1.51	0.97	0.88
Pair 6	Failed	Failed	Failed	0.06	0.26	0.46	Failed	Failed	Failed	0.05	0.32	0.29	0.34	0.79	0.82	0.34	0.43	0.43
Pair 7	Failed	Failed	Failed	0.34	0.65	0.72	Failed	Failed	Failed	0.05	0.27	0.33	0.19	0.75	0.79	0.54	0.59	0.56
Pair 8	Failed	Failed	Failed	0.02	0.74	0.81	Failed	Failed	Failed	0.05	0.25	0.34	0.07	0.67	0.73	0.63	0.64	0.61
Pair 9	Failed	Failed	Failed	0.01	0.78	0.81	Failed	Failed	Failed	0.10	0.28	0.34	0.29	0.72	0.75	0.78	0.68	0.61

Fig. 4 has three sections: input images, feature matching results, and registration outputs. The input section presents sequential microscopic images. The feature matching section highlights the accuracy of our method (Fig. 4 (c)), demonstrating effective outlier rejection rather than quantity of matches. The registered images section shows variable success with different algorithms and soft ware. SIFT and SIFT with RANSAC manage distortion but struggle with alignment. ORB fails with severe distortion. While bUnwrapJ handles distortion, it lacks precise alignment, and TurboReg, despite adequate alignment, shows noticeable distortion. In contrast our method (Fig. 4 (l)) controls distortion and ensures precise alignment, outperforming all other methods. This verifies our method’s robustness for serial section microscopic image registration.

Figure 4:

Sequential registration of all 10 image stacks using our proposed method, showcasing its effectiveness in aligning complex structures

Fig. 5 shows sequential registration results for 10 image stacks using our method. The consistency across registered images and the clear common areas show our algorithm’s precision and efficacy. The maintenance of intricate structures amidst diverse image content validates our method’s robustness.

Figure 5:

Sequential registration of all 10 image stacks using our proposed method.

We evaluated the average running times of the discussed algorithms on our datasets, using a Dell Optiplex system with Windows OS and 16GB RAM. For SIFT, it took about 3 seconds per pair on 3D serial-section microscopic images 1, 2 seconds on 3D serial-section microscopic images 2, and less than 1 second on CUMC12. RANSAC with SIFT achieved similar times, though slightly increased for 3D serial-section microscopic images 1. ORB was fastest, taking less than 1 second on all datasets. The software bUnwrapJ and TurboReg took 5 seconds and 2 seconds, respectively across all datasets. Our proposed method was marginally slower, 6 seconds on 3D serial-section microscopic images 1,5 seconds on 3D serial-section microscopic images 2, and 3 seconds on CUMC12. Despite this, our method’s performance was competitive and often yielded better registration accuracy than baseline methods.

5. CONCLUSION

In this study, we presented an image registration technique that, according to a number of image quality metrics, performs effectively when compared to current state-of-the-art algorithms and software packages in various scenarios. Despite the challenges of distortions and transformations, our method has demonstrated consistency and effectiveness in registering images with high reliability and visual quality. These findings suggest that our approach has potential for addressing complex image registration tasks. Its performance on intricate 3D multiplex microscopy and 3D MRI images further substantiates this potential. Although our method has demonstrated considerable potential, further investigation and improvement may be required. Improving our algorithm’s ability to handle severe distortions and transformations would expand its applicability to more complex scenarios. In addition, it would be advantageous to test the performance of the method on a wider variety of biological and medical image types, such as histopathology slides, cellular microscopy, and various medical imaging modalities.

CCS CONCEPTS.

• Computing methodologies → Artificial intelligence; Computer vision; Matching.