Region-based image registration for remote sensing imagery

Abstract

We propose an automatic region-based registration method for remote sensing imagery. In this method, we aim to register two images by matching region properties to address possible errors caused by local feature estimators. We apply automated image segmentation to identify the regions and calculate regional Fourier descriptors and standardized regional intensity descriptors for each region. We define a joint matching cost, as a linear combination of Euclidean distances, to establish and extract correspondences between regions. The segmentation technique utilizes kernel density estimators for edge localization, followed by morphological reconstruction and the watershed transform. We evaluated the registration performance of our method on synthetic and real datasets. We measured the registration accuracy by calculating the root-mean-squared error (RMSE) between the estimated transformation and the ground truth transformation. The results obtained using the joint intensity-Fourier descriptor were compared to the results obtained using Harris, Minimum eigenvalue, Features accelerated segment test (FAST), speeded-up robust features (SURF), binary robust invariant scalable keypoints (BRISK) and KAZE keypoint descriptors. The joint intensity-Fourier descriptor yielded average RMSE of 0.446 ± 0.359 pixels and 1.152 ± 0.488 pixels on two satellite imagery datasets consisting of 35 image pairs in total. These results indicate the capacity of the proposed technique for high accuracy. Our method also produces a lower registration error than the compared feature-based methods.

1. Introduction

Image registration has found wide applicability in various application domains including computer vision, remote sensing, biomedicine, and security. For example, image registration can be used in target recognition applications for matching real-time images with a target. In the biomedical imaging domain, image registration can be used to study the growth of tumors. Also, disease diagnosis can be carried out by matching two images from different modalities. In remote sensing, it can be used for monitoring climatic change and land-use using satellite images. Also in the field of computer vision and pattern recognition, it can be used for matching stereo images for object/shape recognition (Brown, 1992; Zitova and Flusser, 2003). Most registration techniques are classified into intensity-based or feature-based categories.

Feature-based registration methods rely on locally invariant and distinctive image features for registration. The major stages in feature-based algorithms are Feature Detection, Feature Extraction, Feature Matching and Estimation of Geometric Transform (Fig. 1). Feature detection deals with the detection of salient image features, such as corners, intersections, blobs, edges, and regions (Tuytelaars and Mikolajczyk, 2008). For the purpose of finding point correspondences, such features are usually required to be locally invariant to geometric transformations, e.g., changes in viewpoint, rotation, translation and scale invariant, and also to be invariant to photometric changes, for example, noise and occlusion. Some known feature detectors in the literature are Harris corner detector (Harris and Stephens, 1988), corner detection by minimum eigenvalues method (Shi and Tomasi, 1994), Features from accelerated segment test (FAST) (Muja and Lowe, 2009) corner detector, Speeded-Up Robust Feature (SURF) (Bay et al., 2006), Binary Robust Invariant Scalable Keypoint (BRISK) (Leutenegger et al., 2011) and KAZE by Alcantarilla et al. (2012).

Fig. 1.

Stages in Feature-based registration.

The FAST algorithm by Muja and Lowe (2009, 2012) primarily detects a corner at a pixel $p$ using threshold tests over the Bresenham circle of radius 3. In BRISK, points of interest are detected in both image and scale dimensions using a saliency criterion. The effectiveness of this technique is boosted by detecting keypoints in octave and intraoctave layers of the scale space pyramid. The location and scale of each keypoint are obtained in the continuous domain via quadratic function fitting (Leutenegger et al., 2011). The SURF algorithm by Bay et al. (2006) uses the FAST Hessian detector for feature detection. Here the scale space is analyzed by up-scaling instead of constantly reducing the original image. In order to localize interest points in image and scale spaces, non maximum suppression is applied on a 3 × 3 × 3 neighborhood. The maxima of the determinant of the Hessian matrix are then interpolated in scale and image space with the method proposed by Brown and Lowe (2002). Feature extraction is typically done on regions centered around detected features. This step aims at measuring features that can distinguish the points of interest such as object corners, edges, lines and intersections. FAST features are extracted by forming feature vectors using pixel intensities from the 16 pixel circle. The orientations of pixels are determined by assigning either a positive or negative sign to the pixels depending on whether they are greater than the center or not (Muja and Lowe, 2009, 2012). SURF features are extracted in two steps. First a reproducible orientation is fixed based on information from a circular region around the interest point. Rotation invariance is achieved by computing the Haar Wavelets response in the $x$ and $y$ directions. The descriptor component defines a square region aligned to the selected orientation from which the SURF descriptor will be extracted (Bay et al., 2006). Feature matching finds point correspondences between the reference and the input image. The FAST approach minimizes the sum of squared difference (SSD) of the feature vector between the feature points. The $O\left(N^2\right)$ cost of testing every feature against every other feature is achieved by sorting $F_I$ by the mean value of feature vectors $\bar{f}$ (Muja and Lowe, 2009). In the BRISK method, two feature descriptors are matched by computing their Hamming distances as discussed in BRIEF (Calonder et al., 2010). The dissimilarity measure between them is determined by the difference in the number of bits in the two descriptors (Leutenegger et al., 2011).

Okorie and Makrogiannis (2017) validated the accuracy of feature-based registration methods on remotely sensed imagery, by calculating pixel discrepancy error between the ground truth transformation and the estimated geometric transform obtained for each of the descriptors mentioned above. Their results show that feature-based registration techniques have the capacity of subpixel accuracy. Feature-based techniques were also applied to remote sensing datasets for image registration in Dare and Dowman (2001), Xiong and Zhang (2009), Makrogiannis and Bourbakis (2010), Huo et al. (2012), Ye and Shan (2014), Gong et al. (2014) and Wang et al. (2015).

Despite their good performance, feature-based techniques make use of mostly low or mid-level image entities, therefore, they lack higher-level visual representation that may prove advantageous for difficult matching problems. In the literature, few methods have been proposed that aim to identify, represent and match regions corresponding to objects or their meaningful parts (Li et al., 1995; Liang et al., 1997; Dare and Dowman, 2001; Rendas and Barral, 2008; Goncalves et al., 2011a,b; Troglio et al., 2012; Kybic and Borovec, 2014; Kybic et al., 2015; Han et al., 2017). An early technique by Liang et al. (1997) employed multiscale segmentation followed by Powell direction set optimization and applied this method to biomedical images. This method required manual selection of regions by the user in the matching stage. Dare and Dowman (2001) employed global shape features to match control points. Rendas and Barral (2008) utilize the MDL criterion between segmented images to find an optimal match for linear registration, however they do not propose a segmentation method and region shape is not explicitly used. Goncalves et al. (2011b) apply histogram processing to identify modes and flat regions of intensity distribution and calculate connected components to form regions. This method represents each region by global shape characteristics and calculates a pairwise cost function over the object pairs. The rotation and translation parameters are sequentially estimated by identifying outliers in their distribution. However, this method does not address scale differences. In Goncalves et al. (2011a), the authors used PCA for multisensor fusion, multi-level Otsu thresholding for segmentation, calculated SIFT features and used a statistical approach for matching, similar to their previous work. In this method, multi-level Otsu thresholding is susceptible to noise and may fail for non-Gaussian distributed pixel intensities. In Troglio et al. (2012) ellipsoidal features were extracted using image processing operations and watershed, and the Hough transform was utilized to match craters and rocks of compact shape for planetary image registration. An object-based registration method for very high resolution VHR images was proposed by Han et al. (2017) that aimed to correct local residual misalignments produced by standard registration methods.

In this work, we transition from pixel-centric to region-based registration for remotely sensed imagery. Our hypothesis is that if we can obtain similar regions in both reference and input images, irrespective of geometric or photometric changes, then these regions can be matched to register the two images. We propose a complete method for region delineation, matching, and geometric transformation. We apply automated image segmentation to identify the regions. In contrast to previous works, we extract and combine object-based intensity and shape information to represent the image content. We introduce regional Fourier descriptors and standardized regional intensity descriptors for each region. We define a joint matching cost, as a linear combination of Euclidean distances, to establish and extract correspondences between regions. To evaluate the performance of this method we compare its registration accuracy to feature-based methods.

In Section 2, we introduce our method including the segmentation stage, the regional shape and intensity descriptors, and our approach to calculating joint region similarity scores. We validate the registration accuracy for real satellite imagery data and compare our method with contemporary feature-based registration in Section 3. We discuss our main findings in Section 4 and we conclude this work in Section 5.

2. Region-based registration

We propose a method, in which we match regions instead of local image features for registration. The main stages of this algorithm are displayed in Fig. 2. The algorithm is divided according to the major components of feature-based registration.

Fig. 2.

Flow chart for region based registration.

2.1. Region delineation and feature detection

In this stage, we delineate the image scene into regions and identify the corresponding features for matching. We apply a histogram matching technique to reduce the effect of multisensor variations on the segmentation. Then we calculate a probabilistic map of the edge location and strength using Parzen kernel density estimation. We apply morphological reconstruction to reduce the false minima and delineate the corresponding regions by applying watershed transform to the reconstructed edge map. The regions obtained after watershed segmentation are then represented by their respective centroids. We briefly describe the steps as follows.

Intensity histogram matching.

This is an intensity mapping technique that aims to transform the histogram of an image to match the histogram of a reference image. The purpose is to produce a similar distribution of intensities in the pair of images that we will register. This pre-processing stage was necessary due to the variation of intensity in the pairs of images in our dataset, which could be as a result of the difference in sensors used in the acquisition process. An example of this process is shown in Fig. 3. We observe in Fig. 3 that the histogram of the input image (middle column) has been transformed (last column) so that the intensity distribution is similar to the reference image (first column).

Fig. 3.

Histogram matching example. (Top, from left to right) The reference image, the input image and an output image of histogram matching. (Bottom, from left to right) Corresponding histograms.

Parzen kernel density estimation probabilistic edge detection.

In our approach, we calculate a measure of edge likelihood from the probability density estimate of the intensity at the median in the neighborhood of each pixel. Our premise is that high probability estimates correspond to uniform distribution models occurring within a region, while low probability values indicate the presence of an edge (Economou et al., 2001; Makrogiannis et al., 2005; Makrogiannis and Bourbakis, 2010). The Parzen Kernel Density estimator (Parzen, 1962) is a non-parametric kernel density estimator given by

$$f_n(x)={\int }_{-\infty }^{\infty }\frac{1}{s}K\left(\frac{x-y}{s}\right)d{F}_n(y)\simeq \frac{1}{n\cdot pbw}\sum _{j=1}^{pk{l}^2}K\left(\frac{x-X_j}{pbw}\right)$$ (1)

where $X_j$ corresponds to the intensity values on a $pkl\times pkl$ window. $pkl$ is the kernel length and $pbw$ is the bandwidth, which determines the density of the edges. Our choice for $K\left(x\right)$ is the Gaussian kernel, defined by

$$K\left(x\right)=\frac{1}{2\pi }{e}^{-\frac{x^2}{2}}.$$ (2)

H-minima transform.

The edge maps from the preceding step also include false edges introduced by noise. In order to reduce over-segmentation, we applied the $h$-minima transform. The $h$-minima transform (Soille, 2013) is a morphological image reconstruction algorithm which suppresses all regional minima with depths less than or equal to some given threshold $h$. Fig. 5 shows the result of applying $h$-minima transform with height $h=2$ to the Parzen edge maps in Fig. 4. As seen in the result, the spurious edges created by noise have been smoothed.

Fig. 5.

Example of segmentation after $h(=2)$ minima transform. (From left to right) Edge estimation by applying $h$ minima transform and the result of watershed segmentation.

Fig. 4.

Example of segmentation using Parzen Kernel density estimation. (From left to right) Original image corrupted by Gaussian noise with zero mean and standard deviation $\sigma =5$, intensity of the highlighted portion in the original image, edge regularization using Parzen kernel density estimation, and the result of watershed segmentation.

Watershed segmentation.

The next step in our region detection algorithm is to apply watershed segmentation to the reconstructed edge map. Watershed segmentation is a morphological image segmentation algorithm. The principle follows from the construction of dams (watershed lines) between flooded catchment basins corresponding to regional minima, to prevent water from moving from one catchment basin to another. The watershed lines demarcate one region from the other (See Fig. 6). In our experiments, we used this algorithm to delineate regions on real satellite imagery.

Fig. 6.

Watershed labels for synthetic images.

2.2. Feature extraction

A region feature describes some properties that distinguishes it from other regions. We represent the region features of an image in an $N\times L$ matrix, where $N$ is the number of regions and $L$ is the length of each feature vector. We propose to use joint shape and intensity descriptors. We represent the shape by Fourier Descriptors and we also calculate region-based intensity features.

2.2.1. Regional Fourier Descriptor (RFD)

Fourier descriptors represent the shape of a 2D region by taking the Fourier transform of the boundary, after converting each boundary point $(x,y)$ to a complex number $z=x+iy$. The original boundary can be reconstructed by taking the inverse Fourier transform. The boundary may be approximated by applying the inverse transform to selected Fourier coefficients. Here we utilize $2^{3+k},k=0,1,2,\dots$ coefficients for shape boundary Fourier description. We enabled translation invariance by subtracting the region centroid from the boundary coordinates (Zhang et al., 2002), i.e.,

$$z\left(t\right)=\left(x\left(t\right)-x_c\right)+j\left(y\left(t\right)-y_c\right)$$ (3)

where $x_c=\frac{1}{N}{\sum }_{t=0}^{N-1}x(t)$; $y_c=\frac{1}{N}{\sum }_{t=0}^{N-1}y(t)$. The term used to describe the shape boundary signature in Eq. (3) in Zhang et al. (2002) is complex coordinate function.

We also considered the centroid distance function,

$$r(t)=|z(t)|$$ (4)

as another shape boundary signature, calculated by the magnitude of the complex number $z$. The advantage of using the complex coordinate function is that it is computationally efficient and can be used for shape boundary reconstruction. Notwithstanding, in our experiments, the centroid distance function yielded better results.

Let $u(t)$ represent the shape signature, $t=0,1,2,\dots ,M-1$. We apply 1D Fourier transform to $\mathbf{u}$ to calculate the frequency domain representation of the shape signature

$$\omega (\tau )=\sum _{t=0}^{M-1}u(t)e^{-j2\pi \frac{\tau }{M}t},\tau =0,1,2,\dots ,M-1$$ (5)

where $u$ is either $r$ or $z$ above. The DC Fourier coefficient, corresponds to $\tau =0$ and is given by

$$\omega (0)=\sum _{t=0}^{M-1}u(t)$$ (6)

Next we calculate the magnitude $|\omega (\tau )|$, $\tau =0,1,\dots ,M-1$, of the Fourier transform (5). We scale by dividing the magnitude by the DC coefficient. Due to the possibility of the DC being zero, especially when using the complex coordinate function on a symmetric boundary, we scale according to the following formula

$${\omega }_s(\tau )=\left\{\begin{array}{ll}\omega \left(\tau \right)/|\omega (0)|,& \text{if}|\omega (0)|\ne 0\\ \omega \left(\tau \right),& \text{if}|\omega (0)|=0\end{array}\right.$$ (7)

We shift the Fourier coefficients to yield $\gamma \left(\tau \right)={\omega }_s(\tau -M/2)$ so that ${\omega }_s\left(0\right)=\gamma (M/2)$, and compute the magnitude

$$r\left(\tau \right)=|\gamma (\tau -M/2)|.$$ (8)

We locate the median index, $m$ of the centered coefficients, and extract $L=NFC$ number of Fourier coefficients around the center or median location

$$FD={\left[r\left(m-\frac{L}{2}-1\right),\dots ,r(m-1),r\left(m\right),r(m+1),\dots ,r\left(m+\frac{L}{2}\right)\right]}^T$$ (9)

$L=2^{3+k},k=0,1,2,\dots K,L\le M$. Eq. (9) yields our Fourier descriptor that is scale, rotation, and translation invariant. We denote the regional Fourier descriptor of centroid distances by RFD.CD.

2.2.2. Standardized Regional Intensity Descriptor (SRID)

Intensity features are vectors of length $L=(BS{)}^2$, whose components consist of the pixel intensity values obtained from an $\left(BS\right)\times \left(BS\right)$ block neighborhood around the centroid. We used 11 × 11 block size in our experiments.

We calculate the regional intensity features for each of the input and reference images. Next, we concatenate the resulting $M\times L$ and $N\times L$ matrices of input and reference features, to obtain $(M+N)\times L$ matrix. Then the concatenated features are scaled by calculating the $z$ score of each component $x_{i,j}$, such that

$$z_{ij}=\frac{x_{ij}-{\mu }_j}{{\sigma }_j},i=1,2,\dots ,(M+N),j=1,2,\dots ,L.$$ (10)

where ${\mu }_j$ and ${\sigma }_j$ are the mean and standard deviation of the jth column. We utilize the standardized scores in the matching stage.

2.2.3. Standardized Global Shape Descriptor (SGSD)

As a reference method for experimental comparisons, we also calculate conventional global shape features (Yang et al., 2008). SGSD employs the major and minor principal axes lengths, principal axes ratio, eccentricity, and solidity, $s=\left[{\lambda }_{\mathit{major}},{\lambda }_{\mathit{minor}},r,e,\varsigma \right]$ that we describe next:

The major principal axis length of a region ${\lambda }_{\mathit{major}}$ is a scalar that specifies the length, in pixels, of the major axis ellipse having the same shape moment, up to second order, as the region.

The minor principal axis length of a region ${\lambda }_{\mathit{minor}}$ is a scalar that specifies the length, in pixels, of the minor axis of an ellipse having the same shape moment, up to second order, as the region.

The principal axes ratio $r$ is the ratio of the major to the minor principal axes, $r={\lambda }_{\mathit{major}}/{\lambda }_{\mathit{minor}}$.

The eccentricity $e$ of a region is the ratio of the distance between the foci to the major principal axis of the ellipse having the same second moment as the region, i.e., $e=\frac{{∥f_1-f_2∥}_2}{{\lambda }_{\mathit{major}}}$ where $f_1,f_2$ are the foci of an ellipse with the same shape moments, up to second moments, as the region.

The solidity $s$ of a region is the area of the region to the area of its Area of shape convex hull, $\varsigma =\frac{\text{Area}\text{of}\text{shape}}{\text{Area}\text{of}\text{convex}\text{hull}\text{of}\text{the}\text{shape}}$.

Shape features are scaled to standardized $z$–scores using the same strategy as described in the standardization of intensity features. We then create joint region descriptors by combining two or more standardized features described above.

2.3. Feature matching

Feature matching is the process of establishing correspondences between the regional descriptors. The similarity between features is measured by a cost function. We describe our feature matching steps next.

2.3.1. Calculation of joint score matrix

A score (similarity) matrix is an $M\times N$ matrix that consists of numerical measures of similarities between a pair of regions from the input and reference images, which are represented by their respective feature vectors.

We calculate pairwise similarity using a joint cost function that weighs the contribution of each feature according to its range. Let $\mathbf{u}={\left[u_1,\dots ,u_L\right]}^{\mathrm{\top }}$ and $\mathbf{v}={\left[v_1,\dots ,v_L\right]}^{\mathrm{\top }}$ be vectors in ${\mathbb{R}}^L$ representing scaled input and reference features, respectively.

Joint cost function.

We define a joint cost function that is a weighted combination of Euclidean score matrices

$$D_J(\mathbf{u},\mathbf{v})=\sum _{i=1}^p{w}_i{\left[\sum _{k=1}^{L_i}\left(u_k^2-v_k^2\right)\right]}^{\frac{1}{2}}$$ (11)

where $p$ is the number of descriptors, $w_i$ is the descriptor weight given by $w_i=\frac{1}{L_i\cdot \sqrt{{\left(b_k-a_k\right)}^2}},$ $\left[a_k,b_k\right]$ defines the range of the scaled $i$ th descriptor under consideration and $L_i$ is the descriptor’s dimensionality. The three descriptor types are SRID, RFD and SGSD.

To compute the scores in Eq. (11), we first compute the score matrices for each descriptor type and then combine them to obtain a joint score matrix. The weights $w_i$ balance the contribution of each descriptor type to prevent possible bias.

2.3.2. Cost minimization

In this stage, we perform a nearest neighbor search on the joint score matrix. The minimum cost for each row of the score matrix is obtained and indexed as (row, col). The row is the index of the reference feature, whereas col is the index of the input feature. Given a reference feature, indexed row, the minimum score in a row describes the closest match with index col among all the input features.

Next, we obtain the second lowest costs for each row to reject unambiguous matches, as discussed in the next subsection. The minimum scores undergo double thresholding to identify the best matches. First, we use the weak matching threshold, ${\tau }_w$ to remove the weak matches. The second threshold is the ambiguity threshold ${\tau }_a$. The ratio of the minimum to the second smallest score is calculated and compared to ${\tau }_a$. The minimum scores whose ratio is less than ${\tau }_a$ are considered unambiguous scores and are kept. Finally, we extract the pairs of points that correspond to the unambiguous scores. These matches are used for estimating the geometric transform.

2.4. Estimation of geometric transform

In this section, we discuss the method of fitting an affine geometric transform on unambiguous matched points. There are different methods for fitting a geometric transform on 2D to 2D correspondences, but in our case, we used the maximum likelihood estimation sampling consensus (MLESAC) (Torr and Zisserman, 2000; Hartley and Zisserman, 2003).

Given a 2D to 2D correspondence ${\mathbf{x}}_i\leftrightarrow {\mathbf{x}}_i^{\prime }$ we wish to obtain an affine geometric transform $\mathrm{H}$ such that

$${\mathbf{x}}_i^{\prime }=\mathrm{H}{\mathbf{x}}_i$$ (12)

where $\mathrm{H},{\mathbf{x}}_i$ and ${\mathbf{x}}_i^{\prime }$ are given by

$$\mathrm{H}=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ 0& 0& 1\end{array}\right];{\mathbf{x}}_i=\left[\begin{array}{c}{x}_i\\ y_i\\ 1\end{array}\right];{\mathbf{x}}_i^{\prime }=\left[\begin{array}{c}{x}_i^{\prime }\\ y_i^{\prime }\\ 1\end{array}\right].$$ (13)

Because ${\mathbf{x}}_i$ and $\mathrm{H}{\mathbf{x}}_i^{\prime }$ are collinear, we have ${\mathbf{x}}_i\times \mathrm{H}{\mathbf{x}}_i^{\prime }=0$. We define ${\mathbf{h}}^1={[h_{11},h_{12},h_{13}]}^{\mathrm{\top }},{\mathbf{h}}^2={[h_{21},h_{22},h_{23}]}^{\mathrm{\top }},{\mathbf{h}}^3=[0,0,1{]}^{\mathrm{\top }}$ so that

$$\mathrm{H}{\mathbf{x}}_i=\left(\begin{array}{l}{\mathbf{h}}^{1T}{\mathbf{x}}_i\\ {\mathbf{h}}^{2T}{\mathbf{x}}_i\\ {\mathbf{h}}^{3T}{\mathbf{x}}_i\end{array}\right).$$

Then the cross product can be written as

$${\mathbf{x}}_i\times \mathrm{H}{\mathbf{x}}_i=\left(\begin{array}{c}{y}_i^{\prime }{{\mathbf{h}}^3}^{\top }{\mathbf{x}}_i-{\mathbf{h}}^{2^{\top }}{\mathbf{x}}_i\\ {\mathbf{h}}^{1^{\top }}{\mathbf{x}}_i-x_i^{\prime }{\mathbf{h}}^{3^{\top }}{\mathbf{x}}_i\\ x_i^{\prime }{{\mathbf{h}}^2}^{\top }{\mathbf{x}}_i-y_i^{\prime }{{\mathbf{h}}^1}^{\top }{\mathbf{x}}_i\end{array}\right).$$

This yields a system of three equations in the entries of $\mathrm{H}$,

$$\left[\begin{array}{ccc}{\mathbf{0}}^{\top }& -{\mathbf{x}}_i^{\top }& y_i^{\prime }{\mathbf{x}}_i^{\top }\\ {\mathbf{x}}_i^{\top }& {\mathbf{0}}^{\top }& -x_i^{\prime }{\mathbf{x}}_i^{\top }\\ -y_i^{\prime }{\mathbf{x}}_i^{\top }& x_i^{\prime }{\mathbf{x}}_i^{\top }& {\mathbf{0}}^{\top }\end{array}\right]\left(\begin{array}{l}{\mathbf{h}}^1\hfill \\ {\mathbf{h}}^2\hfill \\ {\mathbf{h}}^3\hfill \end{array}\right)=\mathbf{0}.$$ (14)

Eq. (14) has the form

$${\mathrm{A}}_i\mathbf{h}=0;\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathbf{h}={\left({\mathbf{h}}^1,{\mathbf{h}}^2,{\mathbf{h}}^3\right)}^{\mathrm{\top }}$$ (15)

where $\mathbf{h}$ is a 9-vector containing entries of $\mathrm{H}$, and ${\mathrm{A}}_i$ is a 3 × 9 matrix.

The third row of ${\mathrm{A}}_i$ is a linear combination of row 1 and row 2 by the relation $R_3=x_i^{\prime }{R}_1+y_i^{\prime }{R}_2$. So the rank of ${\mathrm{A}}_i$ is 2 and we may solve for $\mathbf{h}$ using the equation

$$\left[\begin{array}{ccc}{\mathbf{0}}^{\mathrm{\top }}& -{\mathbf{x}}_i^{\mathrm{\top }}& y_i^{\prime }{\mathbf{x}}_i^{\mathrm{\top }}\\ {\mathbf{x}}_i^{\mathrm{\top }}& {\mathbf{0}}^{\mathrm{\top }}& -x_i^{\prime }{\mathbf{x}}_i^{\mathrm{\top }}\end{array}\right]\left(\begin{array}{l}{\mathbf{h}}^1\\ {\mathbf{h}}^2\\ {\mathbf{h}}^3\end{array}\right)=\mathbf{0}$$ (16)

The affine transform matrix has six degrees of freedom, corresponding to $h_{11},h_{12},h_{13},h_{21},h_{22},h_{23}$. Each correspondence ${\mathbf{x}}_i\leftrightarrow {\mathbf{x}}_i^{\prime }$ gives rise to two linearly independent equations in the entries of $\mathrm{H}$. Hence $\mathrm{H}$ can be computed from three correspondences. The system $(\mathbf{A}\mathbf{h}=\mathbf{0})$ to solve becomes

$$\left[\begin{array}{ccccccc}0& 0& 0& -x_1& -y_1& -1& y_2^{\prime }\\ x_1& y_1& 1& 0& 0& 0& -x_i^{\prime }\\ 0& 0& 0& -x_2& -y_2& -1& y_2^{\prime }\\ x_2& y_2& 1& 0& 0& 0& -x_2^{\prime }\\ 0& 0& 0& -x_3& -y_3& -1& y_3^{\prime }\\ x_3& y_3& 1& 0& 0& 0& -x_3^{\prime }\end{array}\right]\left(\begin{array}{c}{h}_{11}\\ h_{12}\\ h_{13}\\ h_{21}\\ h_{22}\\ h_{23}\\ 1\end{array}\right)=\mathbf{0}$$ (17)

Since the rank of $\mathbf{A}$ is 6, it has a one-dimensional null space which provides a solution of $\mathbf{h}$ subject to $|h|=1.\mathbf{h}$ can be solved by singular value decomposition of $\mathbf{A},\mathbf{A}=U\mathrm{\Sigma }{V}^{\mathrm{\top }}$, and corresponds to the unit singular vector corresponding to the smallest singular value of $\mathbf{A}$.

The MLESAC estimator randomly selects three correspondences ${\mathbf{x}}_i\leftrightarrow {\mathbf{x}}_i^{\prime },i=1,2,3$ at a time, from the set of matched points, estimates the affine matrix $\stackrel{\mathbf{ˆ}}{\mathbf{H}}$ and the corrected correspondences, $\left\{{\stackrel{\mathbf{ˆ}}{\mathbf{x}}}_i\leftrightarrow {\stackrel{\mathbf{ˆ}}{\mathbf{x}}}_i^{\prime }\right\}$, that minimize

$$D=\sum _{i}d{\left({\mathbf{x}}_i,{\widehat{\mathbf{x}}}_i\right)}^2+d{\left({\mathbf{x}}_i^{\prime },{\widehat{\mathbf{x}}}_i^{\prime }\right)}^2$$ (18)

with ${\stackrel{\mathbf{ˆ}}{\mathbf{x}}}_i^{\prime }=\stackrel{\mathbf{ˆ}}{\mathbf{H}}{\stackrel{\mathbf{ˆ}}{\mathbf{x}}}_i$. To obtain minimization, the algorithm compares $D$ from Eq. (18) against a distance threshold $ϵ$. The corrected correspondences $\left\{{\stackrel{\mathbf{ˆ}}{\mathbf{x}}}_i\leftrightarrow {\stackrel{\mathbf{ˆ}}{\mathbf{x}}}_i^{\prime }\right\}$, that satisfy $D<ϵ$ are called inliers.

3. Experiments and results

In this section, we present registration results on satellite images using the described region based method. We describe the datasets and how we generated the ground truth. We outline the experiments and the validation technique. We report results from our method and compare them to results we obtained using feature-based methods, i.e., Harris, minimum eigenvalues, FAST, SURF, BRISK and KAZE.

3.1. Datasets

In this work, we employed two datasets of satellite imagery. These datasets present challenges related to the physical sensors, contrast and noise properties, and resolution limitations.

UCSB dataset.

This dataset contains 19 pairs of satellite images obtained from the UCSB Image Processing and Vision Research Lab (IPVRL-UCSB, 0000) dataset. The image sizes range from 250 × 250 to 600 × 600. 18 out of the 19 pairs were acquired using Landsat TM sensors of different bands (0, 3, 4, 5, 7, 8). The remaining pair was obtained with an AVIRIS-Band-19 sensor. The images in each pair have the same size. The images in the UCSB dataset include viewpoint, temporal, and spectral variations. The pairs of images in the UCSB dataset are shown in Fig. 7. The images in the first pair of the UCSB dataset are images from a Mojave Desert sequence taken with an optical sensor. This image pair consists of single sensor images and the size of each image in the pair is 512 by 512. The eleventh pair comprises of multispectral-optical aerial images of a mountainous area acquired by thematic mapper (TM) simulator. These test images (size: 512 by 512) were cut out from different bands of raw images of size 1000 by 766 after translation and rotation. This pair corresponds to the zeroth and eighth bands from a 12 bands set.

Fig. 7.

The first real dataset (source: UCSB). Consists of 19 pairs of satellite images obtained with Landsat sensors of different bands.

USGS dataset.

Our USGS test dataset was obtained from the online database of the U.S Geological Survey that contains Landsat imagery in Geographic Tagged (GeoTiff) format. We selected images of the Delaware–Maryland–Virginia Peninsula in the United States, which were acquired at eight different time points between the years 2002 and 2011. They were acquired using Landsat TM sensors of bands 4 and 5. Each band of the data is stored as a grayscale, uncompressed, 8 -bit string of unsigned integers. The image matrix sizes range from 7191 × 7951 pixels to 7191 × 8061 pixels with a spatial resolution of 30m. The entire dataset is characterized by multi-temporal, multi-spectral and viewpoint variations, because of the different bands and timepoints, which made it suitable for testing registration performance. For the purpose of registration, we chose as reference the Landsat Band 5 image acquired in 2007, and rotated and translated all other Landsat 4 and 5 images, in addition to the existing viewpoint misalignments. This step generated 16 pairs of images in the USGS to register as shown in Fig. 8. The resulting distributions of ground truth transformation parameters over all the USGS image pairs are 13.003° ± 0.002 for the rotation, 869.40 ± 96.698 for the horizontal translation and −846.958 ± 20.174 for the vertical translation.

Fig. 8.

The second dataset (source: USGS). The image acquired on 2007, band 5 denoted by blue background is used as reference.

3.2. Validation

In order to validate the results of automated registration, we computed the root-mean-squared-error (RMSE), the geodesic distance (GD) between the points obtained from the the ground-truth, $T$ and estimated, $\hat{T}$ transformations on the reference domain. We computed ground truth transformations for each image pair in both datasets by registering the image pairs using about 40 manually selected pairs of control points from both images.

Root Mean Squared Error (RMSE).

Let $\omega =[u,v,1{]}^{\mathrm{\top }}=T[x,y,1{]}^{\mathrm{\top }}$ and $\hat{\omega }=[\hat{u},\hat{v},1{]}^{\mathrm{\top }}=\hat{T}[x,y,1{]}^{\mathrm{\top }}$ be 2D homogeneous coordinates, where $1\le x\le M$ and $1\le y\le N$. Let $\Omega =\{\omega =(u,v):0<u\le M\mathrm{a}\mathrm{n}\mathrm{d}0<v\le N\}$ be the set of points $\omega$ in the spatial domain of the reference image. Then the RMSE between the estimated geometric transformation $\hat{T}$, and the ground truth transformation $T$ is given as

$$RMSE(T,\widehat{T})={\left(\frac{1}{\left|\Omega \right|}\sum _{\omega \in \Omega }\Vert \omega -\widehat{\omega }{\Vert }_2^2\right)}^{\frac{1}{2}}$$ (19)

where $|\Omega |$ denotes the cardinality of the set $\Omega$, and $\parallel \omega -\hat{\omega }{\parallel }_2=\sqrt{(u-\hat{u}{)}^2+(v-\hat{v}{)}^2}$.

Geodesic Distance (GD).

The geodesic distance between two points $(p,\hat{p})$ in $\Omega \times \mathbb{R}$ is the shortest path along the topographic surface (Peyre, 2019) produced by the reference image. Let $\gamma (t)=(u(t),v(t),R(u(t),v(t)\left)\right)$, $t\in [0,1]$ be a parametric 3D curve. The length of the curve is given by

$$L(\gamma (t)):={\int }_0^{1}d\gamma ={\int }_0^{1}W(\gamma (t))∥{\gamma }^{\prime }(t)∥\mathrm{d}t$$ (20)

where $W$ is a weighted length metric. A geodesic curve is the curve that minimizes the length $L$ in (20), $\hat{\gamma }=arg{\mathrm{m}\mathrm{i}\mathrm{n}}_{\gamma }L(\gamma (t))$. Then the geodesic distance $GD$ between $(p,\hat{p})$ is given by

$$GD\left(p,\hat{p}\right)=L\left(\hat{\gamma }\left(t\right)\right).$$ (21)

We utilized the fast marching method by Sethian (1996) to reduce the computational time. We calculated the average geodesic distance over the pairs of test points on the image topographic surface. This is a distance measure that considers the intensity differences between points, therefore it may provide further insight into the effect of misalignment.

3.3. Results

UCSB dataset results.

In these experiments, we tested our automated registration method individual shape and intensity descriptors, or their combinations. We also tested several feature-based techniques that have been published in the literature, that is Harris, the minimum eigenvalue denoted by MINEIGEN, FAST, SURF, BRISK and KAZE. We used grid search to tune the parameters for all compared methods. In our method, we tuned the following parameters: parzen kernel length, parzen bandwidth, h-minima height, match threshold, and ambiguity threshold. We report the results from these experiments in Tables 1, 3 and Fig. 9.

Table 1.

Summary of root mean squared errors before and after registration of UCSB data for validation of registration accuracy of feature-based and region-based techniques.

Descriptor	RMSE before registration		RMSE after registration
	Mean ± Stdev	Median	Mean ± Stdev	Median
HARRIS			0.684 ± 0.441	0.551
MINEIGEN			0.68 ± 0.41	0.634
FAST			0.695 ± 0.416	0.715
SURF			0.742 ± 0.408	0.747
BRISK			0.98 ± 0.658	0.8
KAZE	107.315 ± 66.442	98.9	12.768 ± 51.077	0.556
SRID			0.596 ± 0.402	0.599
RFD.CD			53.348 ± 96.742	1.386
SGSD			71.187 ± 84.283	18.713
SRID.RFD.CD			0.446 ± 0.359	0.267
SRID.SGSD			0.522 ± 0.349	0.509

Table 3.

Summary of point correspondences for the UCSB dataset. Comparison of registration results feature-based and region-based techniques. TVP — Total valid points, UM — unambiguous matches, INL – inliers.

Descriptor	TVP		UM		Ratio (INL/UM)
	Mean ± Stdev	Median	Mean ± Stdev	Median	Mean ± Stdev	Median
HARRIS	2509.8 ± 1883.4	2226	465.5 ± 338	396	0.665 ± 0.205	0.661
MINEIGEN	5572.3 ± 3204.1	4538	1030.1 ± 569.3	800	0.603 ± 0.217	0.6647
FAST	2507.2 ± 2048.4	1756	529.2 ± 498.6	349	0.645 ± 0.197	0.655
SURF	1695.3 ± 1089.1	1563	377.8 ± 236.8	382	0.518 ± 0.221	0.484
BRISK	4114.8 ± 3477.2	3279	753.8 ± 669.2	585	0.598 ± 0.193	0.604
KAZE	7743.9 ± 3772.2	7644	1587.4 ± 750.5	1653	0.584 ± 0.239	0.609
SRID	2161.2 ± 1318.5	1689	268.5 ± 234.8	196	0.43 ± 0.27	0.353
RFD.CD	1590.3 ± 1357.7	968	346.7 ± 312.6	203	0.037 ± 0.038	0.024
SGSD	2269.9 ± 1376.2	1777	501.4 ± 307.8	411	0.034 ± 0.044	0.016
SRID.RFD.CD	1516.9 ± 1298.9	926	170.5 ± 176.5	92	0.293 ± 0.18	0.286
SRID.SGSD	2161.2 ± 1318.5	1689	318.6 ± 251.2	261	0.317 ± 0.179	0.291

Fig. 9.

Comparison of RMSE values produced by feature-based and region-based techniques for each image pair of the UCSB dataset.

In Table 1 we present a summary of root mean square error (RMSE) values of registration of the 19 pairs of images in this dataset, by calculating the average, standard deviation and median of RMSE, before and after registration. The RMSE before registration expresses the average pixel discrepancy, that is quite extensive as we can also observe in Fig. 8. We observe that our joint regional Fourier and intensity descriptor denoted by SRID.RFD.CD yields the lowest average RMSE values among all methods at 0.446 ± 0.359 pixels and the lowest median at 0.267 pixels. Furthermore, the other joint regional shape and intensity descriptor, SRID.SGSD, produced more accurate registration on average than the feature-based techniques with average RMSE values 0.522 ± 0.349 pixels.

Fig. 9 displays the bar graph of the individual RMSE values for each image pair using feature-based methods and our method with the joint shape and intensity descriptors. This offers a more detailed visualization of the performance of each technique and shows that SRID.RFD.CD achieves subpixel accuracy in terms of RMSE for 18 out of the 19 pairs.

Fig. 10 displays the bar graph of the geodesic distances produced by feature-based and region-based methods on the UCSB dataset. We report a summary of these results in Table 2 using the mean, standard deviation and median. From the summary, we observe that SRID.SGSD scored the lowest average geodesic distance at 0.453 ± 0.537 pixels, followed by SRID.RFD.CD method at 0.56 ± 0.681. These results show that our region-based methods yield more accurate registration than feature-based methods.

Fig. 10.

Comparison of geodesic distance values (based on 100 randomly sampled points) produced by feature-based and region-based techniques for each image pair of the UCSB dataset.

Table 2.

Summary of geodesic distance values before and after registration of UCSB data for validation of registration accuracy of feature-based and region-based techniques.

Descriptor	GD before registration		GD after registration
	Mean ± Stdev	Median	Mean ± Stdev	Median
HARRIS			0.629 ± 0.632	0.468
MINEIGEN			0.628 ± 0.555	0.689
FAST			0.638 ± 0.571	0.64
SURF			0.764 ± 0.674	0.702
BRISK			0.692 ± 0.588	0.655
KAZE	71.039 ± 61.459	53.217	11.511 ± 46.871	0.498
SRID			2.096 ± 3.619	0.909
RFD.CD			149.983 ± 82.999	160.631
SGSD			118.38 ± 89.559	118.647
SRID.RFD.CD			0.56 ± 0.681	0.558
SRID.SGSD			0.453 ± 0.537	0.242

In Table 3, the total valid points (TVP) represent the sum of points in both reference and input images, for which features are extracted. Unambiguous matches (UM) refer to the total number of correspondences obtained after applying the matching thresholds outlined in Section 2.3 and detailed in Muja and Lowe (2009, 2012) and Lowe (2004). Inliers (INL) correspond to the number of correspondences obtained after estimation of the geometric transform by MLESAC. We also report the ratio of inliers to unambiguous matches (INL/UM). The region-based methods, in general, produce fewer unambiguous matches and lower INL/UM ratios than their feature-based counterparts.

Finally, we display the registered image pairs produced by SRID. RFD.CD in Fig. 11.

Fig. 11.

Registered image pairs from the UCSB dataset using SRID.RFD.CD.

USGS dataset results.

We performed similar experiments to evaluate the registration accuracy and other performance measures for each stage. We also compared our region-based techniques with feature-based techniques. We report the registration accuracy results in Table 4 and Fig. 14, and feature matching performance measures in Table 6. The results show that SRID.RFD.CD produced very good performance with 1.152 ± 0.488 pixels RMSE estimate and 1.03 pixels median over all 16 image pairs. Among the other methods, MINEIGEN, SRID and SRID.SGSD produce good results, but most of the feature-based methods produced high errors. In Table 6 we observe that our joint regional intensity and shape techniques produce a sufficient ratio of inliers for estimating the geometric transformation. Fig. 12 shows the individual RMSE values for each method and each image pair. Finally, we display the registered image pairs produced by SRID.RFD.CD in Fig. 14.

Table 4.

Summary of root mean squared errors before and after registration of USGS data for validation of registration accuracy of feature-based and region-based techniques.

Descriptor	RMSE before registration		RMSE after registration
	Mean ± Stdev	Median	Mean ± Stdev	Median
HARRIS			256.468 ± 988.988	1.018
MINEIGEN			1.143 ± 0.364	1.060
FAST			2341.732 ± 7168.596	1.514
SURF			1361.969 ± 5265.975	1.977
BRISK			456.078 ± 1760.07	1.507
KAZE	692.538 ± 17.493	691.6	3.18 ± 6.461	1.392
SRID			1.24 ± 0.719	1.140
RFD.CD			2129.787 ± 746.366	2324.1
SGSD			2187.098 ± 496.543	2264.4
SRID.RFD.CD			1.152 ± 0.488	1.027
SRID.SGSD			1.206 ± 0.393	1.277

Fig. 14.

Registered image pairs from the USGS dataset using SRID.RFD.CD.

Table 6.

Summary of point correspondences for the USGS dataset.

Descriptor	TVP		UM		Ratio (INL/UM)
	Mean ± Stdev	Median	Mean ± Stdev	Median	Mean ± Stdev	Median
HARRIS	9548 ± 2308	9460.5	1338.8 ± 540.6	1278.5	0.361 ± 0.188	0.351
MINEIGEN	56 705.8 ± 3074.3	57 248.5	8034.8 ± 2903	7549.5	0.319 ± 0.184	0.285
FAST	8582.4 ± 3022.1	7367.5	900.5 ± 758.4	761.5	0.276 ± 0.189	0.290
SURF	5595.1 ± 1936.7	5125.5	857.6 ± 605.6	837.5	0.303 ± 0.165	0.274
BRISK	12 549.1 ± 4786.9	10 769.5	1322.1 ± 1103.1	1161	0.284 ± 0.188	0.285
KAZE	56 778.7 ± 8997.9	58 165	6011.1 ± 1850	6009	0.190 ± 0.111	0.181
SRID	22 367.9 ± 10 875.6	19 958.5	653.8 ± 724.2	589.5	0.305 ± 0.163	0.255
RFD.CD	3918.8 ± 7790.5	1907.5	710.2 ± 1279.6	392	0.024 ± 0.056	0.010
SGSD	22 367.9 ± 10 875.6	19 958.5	4015.2 ± 1291.7	4250	0.001 ± 0.001	0.001
SRID.RFD.CD	17 527 ± 11 078.9	10 887.5	566.8 ± 355.8	405	0.159 ± 0.191	0.113
SRID.SGSD	22 367.9 ± 10 875.6	19 958.5	4015.2 ± 1291.7	4250	0.246 ± 0.158	0.201

Fig. 12.

Comparison of RMSE values produced by feature-based and region-based techniques for each image pair of the USGS dataset.

Fig. 13 and Table 5 contain the validation results for the USGS dataset using the geodesic distance. SRID.RFD.CD scored the lowest mean geodesic distances followed by SRID.SGSD at 0.711 ± 0.354 and 1.12 ± 0.449 respectively. These results confirm that our region based registration descriptors are capable of producing more accurate registration in a complicated dataset than their feature based counterparts.

Fig. 13.

Comparison of geodesic distance values (based on 100 randomly sampled points) produced by feature-based and region-based techniques for each image pair of the USGS dataset.

Table 5.

Summary of geodesic distance values before and after registration of USGS data for validation of registration accuracy of feature-based and region-based techniques.

Descriptor	GD before registration		GD after registration
	Mean ± Stdev	Median	Mean ± Stdev	Median
HARRIS			145.329 ± 577.183	1.081
MINEIGEN			1.148 ± 0.403	1.206
FAST			308.623 ± 863.958	1.266
SURF			171.41 ± 678.032	1.708
BRISK			160.39 ± 635.371	1.612
KAZE	364.476 ± 29.368	365.375	1.957 ± 2.343	1.444
SRID			175.282 ± 693.726	1.336
RFD.CD			2378.024 ± 756.813	2336.817
SGSD			2472.603 ± 632.975	2392.214
SRID.RFD.CD			0.711 ± 0.354	0.702
SRID.SGSD			1.12 ± 0.449	1.17

4. Discussion

UCSB dataset discussion.

In Tables 1 and 2 we observe that region-based registration techniques with joint shape and intensity descriptors produce the lowest registration errors (SRID.RFD.CD, and SRID.SGSD). In Fig. 11 we can visually confirm the accuracy of registration produced by SRID.RFD.CD. However, the RMSE and geodesic distance values increase significantly when we utilize just the shape descriptors (RFD.CD and SGSD). SRID alone performs very well but still not as well as the joint descriptors. This indicates that intensity and shape features complement each other and their joint use improves the feature matching performance. The feature-based methods of HARRIS, MINEIGEN, FAST and SURF, produce good results but still less accurate than SRID.RFD.CD. A desirable property of RFD is that it represents the complete shape information at an accuracy level that is controlled by the number of Fourier coefficients. The introduction of regional intensity information produces a more accurate representation. We observe that in Table 3 the mean geodesic distance of SRID.RFD.CD is higher than that of SRID.SGSD, while the reverse relationship is observed in Table 1 for the RMSE. This is because the geodesic distance utilizes the intensity (or the gradient) of the reference image, in addition to the spatial coordinates. Overall, the RMSE and geodesic distance validation results are consistent.

To fine tune the algorithm parameters, we performed grid search over the parameter space of the threshold $h$ in the h-minima transform, Parzen kernel length ($pkl$) and Parzen bandwidth ($pbw$) for edge estimation, the number of Fourier coefficients (NFC) for extraction of shape-boundary Fourier descriptors, matching threshold $\left({\tau }_m\right)$ and ambiguity threshold $\left({\tau }_a\right)$ for feature matching, over all pairs of images. Greater values of $h$ remove more local minima. The values of $h=\{3,5,7\}$ were able to reduce over-segmentation caused by intensity variations. Furthermore the Parzen kernel length $pkl$ is related to the number of samples of density estimation and the scale of detection, and the Parzen bandwidth $pbw$ controls the sensitivity to intensity contrast. The following values produced good results for the UCSB dataset $pkl=\{5,7,9\}$, $pbw\in [42,45]$. The numbers of Fourier coefficients NFC that produced effective shape representation were $NFC=\{8,16,32\}$. The amount of unambiguous matches depends on the magnitude of the ambiguity threshold ${\tau }_a$. Due to the closeness of the first and second minimum matches for each row in the score matrix, it was ideal to choose ${\tau }_a\in [0.9,1]$. The range of the threshold for selecting strong matches ${\tau }_m$ is $(0,1]$, but for our real dataset experiments, our results were obtained for values in the interval [0.5, 0.9].

In Table 3, the total valid point (TVP) values are influenced by the number of Fourier coefficients (NFC) and the block size of the intensity features. When extracting joint Fourier and Intensity features, we first reject regions with fewer coefficients than the parameter NFC, and secondly, regions whose intensity blocks exceeded the boundary of the image. Finally, the regions that satisfy both conditions are returned as valid regions for matching. The centroids of the valid regions are the valid points. The sum of valid points for both images is referred to as total valid points (TVP).

When reviewing the INL/UM ratios, we observed that the region based descriptors (with the ratio being in the range 0.293 ± 0.18, for joint-intensity-shape descriptor) produced fewer inliers than feature-based descriptors (ratio is in the interval 0.598 ± 0.193, for BRISK). Hence feature-based descriptors may be able to yield more reproducible matches prior to the estimation of the geometric transform. Our joint descriptors SRID.RFD.CD and SRID.SGSD still produce a sufficient number of inliers for estimating an accurate geometric transformation.

USGS dataset discussion.

The RMSE results in Tables 4 and 5 indicate that SRID.RFD.CD produces very good accuracy followed by MINEIGEN, SRID.SGSD, SRID and KAZE. Because this dataset presents challenges due to the significant increase in the image matrix size and detection of more feature points and fine details, the performance of other FBR techniques declines in comparison with the UCSB dataset. The individual RMSE and geodesic distance values in Figs. 12 and 13 indicate the higher level of difficulty of this dataset for some image pairs such as P1, P3, P13, and P15. Nevertheless, SRID.RFD.CD was able to produce good results. To assess the accuracy of registration we show in Fig. 15 registration results by SRID.RFD.CD that are zoomed-in on coastline and inland regions of USGS image pairs. It is obvious that these images contain fine detail that needs to be preserved in order to find accurate point correspondences. We also note the temporal and multispectral variations that further complicate the registration task.

Fig. 15.

Examples of misalignments and registration results on USGS data.

Our grid search parameter tuning strategy produced the following ranges suitable for the USGS data: $h=\{3,5,7,9,11\}$, $pkl=\{3,5\}$, $pbw\in \left[25,43\right],NFC=\{8,16,32\}$, ${\tau }_m\in [0.1,1]$, ${\tau }_a\in [0.9,1]$. We note that the parameter intervals are similar to those of the UCSB data except for the $pbw$ interval. The $pbw$ level is related to the level of detail to be preserved, therefore a lower value corresponds to increased sensitivity to spectral values that is a desirable algorithm property in this dataset.

The point correspondence summary in Table 6 shows that the region-based techniques detect enough keypoints for matching. We note that the registration accuracy of region-based technique is related to the INL/UM ratio. This relation implies that if a sufficient number of inliers is detected, then the registration result is expected to be accurate. Regarding the stability of solutions, we note that a sufficient number of inliers will support the convergence to good solutions and avoid the local minima. The SRID.RFD.CD method produces in general fewer inliers than some feature-based techniques such as MINEIGEN and KAZE, therefore SRID.RFD.CD is more sensitive to outliers. This result implies that there is potential for further improving the robustness of this descriptor and discriminant capability.

Regarding the implementation, we ran the experiments on Windows platform with Intel Xeon Silver 4114 2.2 GHz CPU and 64 GB RAM. Our region-based methods were implemented in Matlab, but the compared approaches were developed in C++ using the OpenCV library. Therefore the computation times of these groups of techniques are not directly comparable. Qualitative comparisons indicate that while the calculation of RFD descriptors increases the computational complexity of region-based approach, this is comparable to the computational complexity of multi-scale analysis methods such as KAZE.

Qualitative comparisons.

Next, we qualitatively compare methods with published region based methods, such as automatic image registration through segmentation methods by Goncalves et al. (2011b,a), contour based approach by Li et al. (1995) and the methods by Dare and Dowman (2001) and Han et al. (2017).

HAIRIS by Goncalves et al. (2011b), uses mode delineation for segmentation, while the method published in Goncalves et al. (2011a) uses the Otsu thresholding approach for segmentation. These segmentation methods are straightforward and are often affected by noise. In our method, the use of kernel density estimation for edge detection followed by watershed segmentation is advantageous in the sense that it is robust to noise and produces closed regions. Methods like HAIRIS, require a separate pre-filtering process/algorithm for noise reduction, which may reduce the accuracy of registration, because of the blurring of edges and/or boundaries of objects. Our method handles the effect of noise implicitly using the parameters of parzen kernel density edge estimation and the h-minima transform. HAIRIS was modeled to address only additive noise, hence the possibility of failing in presence of other type of noise. Our experiments using a simulated additive noise on synthetic images, showed that by adjusting the parameter of h-minima transform, the effect of noise is minimized.

Dare and Dowman (2001)’s method for automatic feature-based registration of SAR and SPOT images involves an initial manual selection of control points, approximate registration using patch matching, before finding the final accurate registration using edge matching. The possible downsides of this method and how our method is able to address them are enumerated as follows; first, the use of manual control points may be a laborious task especially in a large dataset, such as the UCSB and USGS datasets having a total of 35 pairs of images. Even choosing a minimum of four pairs of points for each pair of images, will amount to 4 × 35 × 2, i.e., 280 control points in our experiments. Secondly, in the approximate registration using patch extraction and matching, thresholding, homogeneous and region growing segmentation algorithms were used for patch extraction. One of the drawbacks of these segmentation approaches is that they are task specific. But our use of watershed segmentation using Parzen kernel density estimation, and accompanied by the h-minima morphological image transformation enable us to estimate range of parameter values based on the physical characteristics such as intensity contrast, spatial and spectral resolution. This helps to produce stable results in multiple sensor and multispectral image registration.

Joint feature extraction proposed in our work has the capability of producing distinctive regions that would yield optimal matches. The feature descriptor, whose components are area, perimeter length and width, and the height of minimum bounding rectangle used in the method proposed by Dare and Dowman (2001), or the area, axis ratio, perimeter and fractal dimension, as used in HAIRIS, may not work as well as the joint feature descriptors SRID.RFD.CD and SRID.SGSD would, because global shape features alone are less discriminant than joint intensity and shape, as we observed from the results of the SGSD method.

Han et al. (2017)’s method makes the assumption that images to be registered have been moderately aligned. Thus could imply that it may not work effectively for extensive misalignments. This also suggests that the user has manually or interactively generated a priori information on the geometric variations between the reference and input images to be registered. On the other hand, our method may address extensive geometric (rotation and translation) misalignments that are occurring in UCSB dataset, for example.

Another advantage of our method is that the segmentation stage produces closed regions, but the contour-based approach by Li et al. (1995) produced both closed and open regions/contours which may reduce the effectiveness of feature extraction and matching. This may lead to difficulties in finding correspondences between regions.

5. Conclusion

In this work, we introduced a segmentation-based registration method for remote sensing imagery. We proposed region-based descriptors and a joint matching cost function for image registration. The region-based descriptors are joint standardized regional intensity and regional Fourier descriptors obtained using centroid distance shape signature (SRID.RFD.CD), and joint standardized region intensity and global shape descriptor (SRID.SGSD). Our joint matching cost function is a weighted combination of Euclidean distances.

In the image segmentation stage, we employ non-parametric kernel density edge estimation and morphological reconstruction before applying the watershed transform to reduce over-segmentation. We also apply intensity histogram matching, in the pre-processing stage, to address sensor variations between two the pairs of images.

We evaluated the performance of our method on two datasets of satellite images resulting in 35 image pairs in total. We also tested the performance of recent feature-based registration techniques on the same data to accommodate comparisons. We validated the registration accuracy by calculating the root mean squared error (RMSE) between ground truth transformations, obtained by fitting affine transformation on control points selected manually, and estimated geometric transformations, obtained by fitting an affine transformation on correspondences established by our methods, using maximum likelihood estimation sampling consensus (MLESAC).

Our techniques produced very good results over these datasets and in comparison to the feature-based techniques. Our joint descriptor denoted by SRID.RFD.CD produced very good average registration accuracy as it achieved the smallest registration error among the compared techniques. These results support the applicability of the proposed region-based technique to automated registration of satellite images.