vEMstitch: an algorithm for fully automatic image stitching of volume electron microscopy

Abstract

Background

As software and hardware have developed, so has the scale of research into volume electron microscopy (vEM), leading to ever-increasing resolution. Usually, data collection is followed by image stitching: the same area is subjected to high-resolution imaging with a certain overlap, and then the images are stitched together to achieve ultrastructure with large scale and high resolution simultaneously. However, there is currently no perfect method for image stitching, especially when the global feature distribution of the sample is uneven and the feature points of the overlap area cannot be matched accurately, which results in ghosting of the fusion area.

Results

We have developed a novel algorithm called vEMstitch to solve these problems, aiming for seamless and clear stitching of high-resolution images. In vEMstitch, the image transformation model is constructed as a combination of global rigid and local elastic transformation using weighted pixel displacement fields. Specific local geometric constraints and feature reextraction strategies are incorporated to ensure that the transformation model accurately and completely reflects the characteristics of biological distortions. To demonstrate the applicability of vEMstitch, we conducted thorough testing on simulated datasets involving different transformation combinations, consistently showing promising performance. Furthermore, in real data sample experiments, vEMstitch successfully gives clear ultrastructure in the stitching region, reaffirming the effectiveness of the algorithm.

Conclusions

vEMstitch serves as a valuable tool for large-field and high-resolution image stitching. The clear stitched regions facilitate better visualization and identification in vEM analysis. The source code is available at https://github.com/HeracleBT/vEMstitch.

Introduction

The continuous development of volume electron microscopy (vEM) in recent decades has led to considerable developments in life science, enabling rich structural information to be observed and captured at nanometer, micrometer, or even millimeter scale, such as in cells and tissues. The quest to understand biological complexity across scales drives the rapid development of vEM. As biologists realize that complexity across scales is present in every organism, the vEM techniques expand from the original neural connectomics research to different domains throughout life sciences [1–5]. Recently, many related infrastructure projects [6, 7] and the international vEM community [8–10] have emerged to facilitate science delivery and collaboration among vEM experts. However, although the ultimate goal in vEM is to achieve both large scale and high resolution, these are irreconcilable contradictions in the process of technological development. Usually when the magnification is relatively high, multiple images are taken in the target area; by moving the electron beam, each image is acquired with a certain overlap area (usually 10%), and then the complete high-resolution image of the target region is obtained via image stitching [8, 11–13], as a fundamental process in the overall vEM pipeline shown in Fig. 1A. Unfortunately, cut marks, folds, deformations, and breakage during sample preparation pose challenges and difficulties in imaging data processing [12, 14, 15], and if the overlap areas happen to contain such artifacts, then accurate image stitching is almost impossible using only rigid transformations.

Figure 1:

(A) The general pipeline of vEM. (B) Two common phenomena in feature-based stitching.

Currently, there are 2 types of image stitching in common use: (i) calculating the translation relationships among overlapping regions by using phase correlations in Fourier space [16–20] or (ii) calculating the geometric relationships among overlapping regions based on feature-based matching of the overlapping images [21–24]. The advantage of using phase correlations in Fourier space is rapid calculations, but if the images have obvious rotation or nonrigid deformation, then ideal seamless stitching is not possible with this method. Therefore, using phase correlations alone cannot provide large scale and high resolution in vEM-based microscopy, such as serial section scanning or transmission electron microscopy and serial block-face scanning electron microscopy [8, 13, 25, 26].

Feature-based approaches use distinctive local features [27–30] within images to identify matching patterns in adjacent tiles and then establish the corresponding relationships among features to estimate the transformation parameters. Unlike Fourier-based methods, which rely on relationships among global frequency components, feature points (FPs) generally reflect more local properties of images, and this makes such methods more suitable and flexible for complex deformations. In the feature-based architecture, the predetermined transformation model and corresponding methods for extracting FPs are constantly being optimized [31–36]. An accurate transformation model captures composite transformation or deformations efficiently and reduces the parameter search space. Also, accurately matched FPs enhance the parameter estimation process, so a well-extracted set of FPs is key for accurate stitching by feature-based methods. However, biological samples have complex and changeable ultrastructure, and when its global feature distribution is inhomogeneous or the local information is overconcentrated and the structure information in other regions is sparse, then feature matching is ineffective, resulting in incorrect matching or even being unable to find the FPs (as shown in Fig. 1B). In this case, simple feature matching cannot achieve an accurate seamless mosaic, thereby affecting the registration among the stitched serial images in the next step of vEM image processing.

Herein, we present a feature-based stitching pipeline named vEMstitch, which is designed specifically for vEM images with uneven feature distribution and local deformation. vEMstitch aims to stitch input images seamlessly while correcting local distortion. Our approach uses a combined global-rigid–local-elastic model to depict complex transformations in vEM images. To estimate global transformation, we use local geometrical properties and random sample consensus (RANSAC) [37] to establish accurate corresponding FPs and calculate rigid parameters. In addressing local elastic transformations, we implement feature reextraction to achieve a balanced distribution of FPs and thin-plate spline (TPS) functions to fit local distortion. Finally, the composite models are built as smoothed pixel displacement fields to generate seamless mosaics (Fig. 2). To validate the efficiency and robustness of vEMstitch, we generate 3 simulated datasets of different transformation combinations to quantify the accuracy. Furthermore, our experiments on real-world data also yield promising results. In summary, vEMstitch is superior to existing open-source stitching tools in generating seamless mosaics.

Figure 2:

(A) Pipeline of proposed method. The input images are processed sequentially in 3 main modules: global-rigid estimation, local elastic correction, and image blending. (B) Pipeline of local elastic correction. The core is the feature reextraction strategy to balance the distribution of feature points (FPs).

Results

Figure 2A illustrates the workflow of vEMstitch. When provided with a biological section composed of several tiles, vEMstitch first determines the stitching order and performs pairwise stitching. For each input image pair, scale-invariant feature transform (SIFT) [27] FPs are extracted and matched based on a simple distance criterion. Assuming that translation dominates the tile transformation, a local outlier filtering strategy is employed to remove obvious mismatches by estimating the trend of local displacement. Considering the smoothness of biological structures, even in the presence of distortion, geometric consistency constraints are applied during the RANSAC process to preserve the relative positions of FPs. This 2-stage filtering strategy ensures the accuracy of FP correspondences, after which global rigid parameters are estimated. Next, vEMstitch uses a feature distribution adjustment and feature reextraction module to enhance the extraction and matching of FPs around edges of biological contours, as shown in Fig. 2B. An evenly distributed FP set better depicts the overall changes in structures at both the global and local levels. Subsequently, taking rigidly transformed FPs, vEMstitch adopts TPS functions to fit the residual displacements between FPs, which correspond to local distortions. To smooth the transition between global rigid transformation and local distortion correction, a pixel displacement field is established by reweighting the regions surrounding the TPS control points. Finally, a clear and seamless mosaic is generated by fusing the transformed images.

To evaluate the proposed method quantitatively and visually, we applied it to 3 simulated datasets constructed with different transformation combinations and also some real-world datasets. The evaluations included comparing the proposed method with 3 open-source stitching tools: the Fourier-based stitching tools Fiji [38, 39] and MIST [19] and the feature-based stitching method TrakEM2 [23]. These open-source tools are available as ImageJ/Fiji plugins.

Simulation experiments

To quantify the similarity among overlapping regions and the discrepancy among rigid parameters, we developed a simulation flow to construct datasets of different transformations (shown in Fig. 3A). We selected 100 raw images of size from the Circuit Reconstruction from Electron Microscopy Images challenge [40], and then to simulate mechanical displacement, we applied random rigid transformations to the raw image with parameters (a rotation angle of , 1], and a translation from [0.5, 1] times the height or width of the raw image, respectively) and obtained (). We then generated pixel displacement matrices to mimic biological deformation by iteratively adding Gaussian smoothed random displacement, and the sampling grids were applied on () to generate deformed images (). Finally, 4 tiles were cut with an overlap rate of 10% to generate the final image grid. The total process of data simulation can be summarized as follows:

Figure 3:

Overview of simulation experiments. (A) Workflow for generating simulated data. A random rigid transformation is applied to simulate the rigid movement of samples, and an optional random elastic transformation is applied on the overlapping regions to simulate local distortion. (B) The runtime of the compared methods for images of different sizes. (C) The number of matching FPs with the increasing tile size. (D) The runtime corresponds to different sizes of matching FPs. (E) Visual examples of compared methods in 3 simulation experiments.

(1)

(2)

where is the ith randomly generated displacement field with elements in the range of , Gauss() is a 2-dimensional (2D) Gaussian filter operator that is applied separately on the 2 channels of , determines the magnitude of the displacement, and controls the smoothness of the deformation. In our experiments, we used = 0.08 size, = 2 size, and .

We conducted 3 simulation experiments (i.e., experiments I, II, and III), each of which had its own transformation combination. In experiment I, we applied only random rigid transformation (only translation, without rotation) to the raw images to simulate the translation of samples. In experiment II, we applied random rigid transformation (translation and rotation) to the raw images to simulate the rigid movement of samples. In experiment III, we applied both random rigid transformation (translation and rotation) and random elastic transformation to the raw images to simulate composite image deformation.

Results for simulated data

We quantify the stitching accuracy of the final mosaic using the following metrics. To evaluate the similarity between 2 overlapping regions, we use normalized cross-correlation (NCC) and structural similarity (SSIM) [41], defined as follows:

(3)

(4)

where and are the average values of images x and y, and are the standard deviations, is the covariance, and and are constants. To measure the differences between the ground-truth rigid parameters and the estimated ones, we use the translation error and rotation error of each tile.

We summarize the average metrics of the simulated datasets in Table 1 and choose 3 representative examples as shown in Fig. 3E. To the right of the stitching results are some partial enlargements of the overlapping regions with the corresponding NCC and SSIM values.

Table 1:

Average stitching metrics on simulation datasets

		Metric (P value)
		Overlapping region		Transformation parameters
Method		NCC	SSIM	Translation error	Rotation error
I	Fiji	0.989 0.007	0.938 0.039	0.202 0.046	−
		()	()	()
	Mist	0.994 0.002	0.959 0.014	0.305 0.088	−
		()	()	()
	TrakEM2	0.969 0.0951	0.921 0.220	1.115 4.590	0.022 0.211
		()	()		(0.0142)
	Ours	0.999 0.001	0.989 0.003	0.010 0.003	0
II	Fiji	0.529 0.136	0.228 0.113	7.118 1.411	−
	Mist	0.267 0.199	0.180 0.148	113.989 105.244	−
	TrakEM2	0.983 0.078	0.951 0.171	1.636 4.223	0.012 0.013
		(0.0372)	(0.0245)	(0.0747)	()
	Ours	0.990 0.003	0.999 9e-05	0.875 0.196	0
III	Fiji	0.456 0.125	0.189 0.088	8.699 2.016	−
	Mist	0.211 0.172	0.134 0.121	108.984 101.742	−
	TrakEM2	0.808 0.144	0.690 0.158	5.467 4.053	0.012 0.011
		()	()	(0.1343)	()
	Ours	0.998 0.001	0.957 0.018	5.135 1.758	0.006 0.003

In experiment I, which involved only translation deviation, all the stitching tools including both the Fourier-based and feature-based methods achieved nearly perfect metrics. This is supported by the visual representations in Fig. 3E (I), where the overlapping regions in all the stitching mosaics are the same as the ground truth. These results show that both Fourier-based and feature-based methods can perform well on simple translation estimation. In experiment II with rigid transformation, Table 1 shows clearly that feature-based methods (ours and TrakEM2) outperform Fourier-based ones (Fiji and MIST) across all quantitative metrics. Fourier-based methods struggle with rotation estimation and exhibit unacceptable translation errors in cases with complex transformation. By contrast, feature-based TrakEM2 presents better metrics and the rotation estimation error is acceptable for stitching, but the translation error is not sufficiently stable. As shown in Fig. 3E (II), TrakEM2’s stitching results produce obvious folding when subjected to larger translation errors. By contrast, our method consistently performs the best and shows robustness. A key contribution to this is that we use local geometry information to establish more accurate feature matching. In experiment III involving both rigid displacement and elastic distortion, Table 1 shows the superior performance of feature-based methods (ours and TrakEM2) over Fourier-based ones (Fiji and MIST) in all metrics. Fourier-based methods cannot handle complex composite transformations, and their estimated translation errors are unacceptable in such cases. Feature-based TrakEM2 and our method both have larger parameter errors than those in experiment II. This is attributed to the challenge of local distortion, which can cause correct FP pairs to be falsely rejected in the RANSAC process. However, our approach uses all filtered FP pairs to model deformation effectively by using a combination of global-rigid and local elastic techniques. The highest NCC and SSIM values of our method in Table 1 further indicate its effectiveness in local distortion correction. Moreover, Fig. 3E (III) shows the representative stitching results from the comparison methods, which exhibit obvious ghosting, whereas our method produces a clear and seamless mosaic.

Runtime analysis

Execution time is a major consideration for the large volume. Following the simulation experiment III, we cropped tiles from the CREMI dataset, with sizes set to 512, 1,024, 1,536, and 2,048, respectively. For fairness and simplicity, all methods were compared on 2 Intel Core i9-10900X cores. Table 2 summarizes the average runtimes for cases where tiles have different sizes. Phase-correlation approaches definitely run faster than feature-based ones, because they only consider translation parameters, and fast Fourier transform (FFT) speeds up the calculation. Compared to the feature-based architecture in TrakEM2, our method runs faster due to the C++ implementation and efficient elastic correction. The block-based correction approach used in TrakEM2 consumes a lot of time. Table 2 also offers the number of our estimated matching FPs. Larger tile sizes have more FPs, which leads to longer stitching time. Further, we focus on the tile size of 1,536. Figure 3D shows the relationship between runtime and the number of matching FPs. As the number of feature points increases, the running time increases roughly at a rate close to the square. Fortunately, the section stitching process can be divided into intrarow and interrow stitching. The number of matching points in each stitching process is acceptable.

Table 2:

Average runtime and matching FPs on simulation datasets

Tile size (pixel)	Runtime (s)					Number of matching FPs (ours)
	Fiji	Mist	TrakEM2	Ours (Python)	Ours (C++)
512	0.409	2.659	7.131	2.793	0.586	102.0
1,024	1.246	3.091	51.228	11.211	3.412	597.51
1,536	2.467	3.829	269.994	24.006	11.481	791.01
2,048	4.804	4.858	940.564	97.315	54.232	2,208.57

Segmentation analysis

A clear and seamless 2D mosaic helps to enhance the quality of subsequent processes, particularly accurate structure segmentation. We trained a state-of-the-art segmentation network [42] using the CREMI dataset and applied it to the 2D stitched results from compared methods. Table 3 averages the Dice coefficients and Hausdorff distances of segmentation maps across 3 simulation experiments. The 2 metrics are defined as follows:

Table 3:

Average segmentation metrics on simulation datasets

	Dice (P value)			Hausdorff (P value)
	Fiji	TrakEM2	Ours	Fiji	TrakEM2	Ours
I	0.7103 0.134	0.6800 0.199	0.7209 0.132	4.9277 4.928	5.1959 1.030	5.1430 1.045
	(0.4517)	(0.0209)		()	(0.0001)
II	0.5600 0.165	0.6666 0.172	0.7208 0.133	5.3926 0.787	5.2904 0.830	5.1431 1.043
	()	(0.0007)		(0.0096)	(0.0004)
III	0.4250 0.184	0.4588 0.201	0.7092 0.123	5.6698 0.754	5.7142 0.790	5.1648 0.767
	()	()		()	()

(5)

(6)

(7)

where are segmentation maps of images x and y respectively, and is a distance measurement between 2 points. The Dice coefficient focuses on the number of overlapping elements, while the Hausdorff distance emphasizes the difference in segmentation boundaries.

Segmentation accuracy tends to decrease as simulated transformation models become more complex, especially with the phase-correlation approach Fiji. However, vEMstitch maintains more consistent Dice and Hausdorff metrics regardless of changes in transformation models. The similar numerical results across 3 experiments demonstrate the stable stitching performance and broad applicability of vEMstitch. Moreover, Fig. 4 illustrates the impact of inaccurate stitching on segmentation maps. Fiji often produces blurred mosaics that do not consider rotation and elastic deformation. The blurred images make it difficult for the segmentation network to recognize structural edges, resulting in blank regions as indicated by both red and blue arrows. TrakEM2, which combines the global rigid transformation and local correction, possibly produces false deformation due to insufficient feature-matching constraints. As indicated by a red arrow, TrakEM2 generates an incorrect redundant structure, causing unnecessary segmentation lines in the map.

Figure 4:

Segmentation examples for simulated experiment III.

Results for real-world data

To illustrate further the feasibility and effectiveness of our method, we show 3 examples of micrograph stitching (a common application of vEM imaging) provided by the Center for Biological Imaging, Institute of Biophysics, Chinese Academy of Sciences. The 3 examples of mussels settling in the deep sea were imaged by scanning electron microscopy after resin embedding at normal temperature and frozen ultrathin slicing. The accelerating voltage of the microscope was set to 2 kV, and the detector was one involving concentric backscattered electrons. Figure 5 shows the different mussel sections and the corresponding stitched results of Fiji, MIST, TrakEM2, and our method. In real-world sections, the overlapping rate is ca. 10% and translation transformation plays the main role, but some rotation and local distortion are inevitable. Therefore, the Fourier-based Fiji and MIST and the feature-based TrakEM2 all exhibit obvious ghosting or folding and more severely in some areas with less valid information, such as the cell membrane. Because limited pixel information prevents accurate parameter estimation, vEMstitch enriches the image by feature reextraction and achieves the best stitching results, as the enlargements indicate. Meanwhile, in some areas with rich information, the Fourier-based Fiji and MIST exhibit blurred overlapping regions, and the feature-based TrakEM2 exhibits insufficient displacement estimation and obvious visible boundaries. Excessive FPs make mismatches more difficult to filter, so vEMstitch uses local geometry properties to reduce the number of candidate points and establish more accurate point matching. Generally, the NCC values of selected enlargements of Fiji, MIST, and TrakEM2 are mostly between 0.6 and 0.8 or even lower than 0.1. Neglecting the effects of local deformation, the existing stitching tools are inadequate for clear and seamless mosaic generation of large-field serial section electron microscopy images.

Figure 5:

Real-world mussel section data. The first column shows the input microscopy images.

Also, in Table 4, we assess the stitching quality using the 3 aforementioned real datasets, each of which comprised 200 sets of tiles. Because of the absence of ground-truth transformation, we mainly evaluate the overlapping regions of adjacent tiles using the NCC and SSIM metrics. In the experiments, it is evident that feature-based methods consistently outperform Fourier-based approaches such as Fiji and MIST. Actually, regions with NCC values below 0.7 often exhibit obvious ghosting, and the lower NCC values produced by Fourier-based methods indirectly indicate the presence of nonlinear distortion in the tiles. By contrast, our pipeline consistently gives the highest NCC values across all datasets, thereby emphasizing its robust distortion-correcting capabilities that ensure superior performance in seamless stitching.

Table 4:

Average stitching metrics on real datasets

		NCC	SSIM
Sample I	Fiji	0.52 0.16	0.02 0.02
	Mist	0.47 0.22	0.02 0.02
	TrakEM2	0.70 0.19	0.08 0.12
	Ours	0.94 0.04	0.25 0.11
Sample II	Fiji	0.79 0.08	0.13 0.05
	Mist	0.79 0.13	0.16 0.08
	TrakEM2	0.89 0.05	0.26 0.10
	Ours	0.97 0.02	0.50 0.10
Sample III	Fiji	0.69 0.07	0.11 0.06
	Mist	0.71 0.08	0.15 0.07
	TrakEM2	0.89 0.05	0.24 0.09
	Ours	0.97 0.01	0.52 0.10

Analysis of modules

Proposed herein is a feature-based stitching architecture for large-field vEM images, and we focus mainly on 2 characteristics of FPs: matching accuracy and spatial distribution.

Visual example for ablation study

First, we emphasized that both the elastic correction and local enhancement strategies are developed to address different stitching problems, which are perhaps encountered in practice. In Fig. 6A, we provide an example to illustrate the results from different stages according to linear blending of the stepwise aligned tiles. Without outlier filtering strategies, the feature-matching process cannot provide accurate point correspondences, resulting in a severe “tile separation” phenomenon. Specifically, loose distance matching (ratio = 0.6) is highly likely to generate false correspondences, making the stitching mosaic unacceptable. In the sole global rigid transformation, the overall biological shape is restored and there is no particularly obvious misalignment. However, as the enlarged region boxed by red shows, a slight local distortion remains to be corrected, leaving blurred content. Around the edge of the organism, the misalignment of structural contours is also enlarged by a blue box. TPS-based elastic correction follows closely after the global rigid estimation and does not affect the position arrangement of tiles. The corrected deformation filed helps to produce a clear mosaic in overlapping regions. Unfortunately, both global rigid and local elastic estimation depend on the initially extracted FPs. The edge structures enlarged by blue boxes lack sufficient matched point pairs and therefore are still misaligned after elastic correction. The local enhancement strategy aims to detect these regions and expand the number of extracted FPs. The last mosaic shows the stitched result under the complete pipeline. Regions with sufficient FPs are not changed, as the red box shows, while the regions with inadequate FPs are enhanced to generate correct feature correspondences, as enlarged by the blue box.

Figure 6:

Analysis of modules. (A) Stepwise stitching results of a real-world mussel section data. (B) Feature-matching results. Above are curves of precision and feature number with increasing threshold ratio, and below is a typical visual example. (C) Distribution of FPs. Above is the point distribution without feature reextraction, and below is the point distribution with feature reextraction.

Module for feature matching

Accurate parameter estimation is dominated by the precision of feature matching. A commonly used and straightforward criterion in natural image processing involves comparing the nearest-neighbor and second nearest-neighbor distances and filtering when this ratio is lower than a given threshold [27]. In fact, the ratio is high for features that are not distinctive, and lowering the threshold ratio enhances the accuracy of feature matching but often results in fewer retained feature pairs. In Fig. 6B, we show how the precision and the number of acquired feature pairs change as the threshold ratio is increased, using the simulated datasets I, II, and III. Experiments I and II involve only rigid transformations (translation and rotation), and straightforward feature matching is effective for handling these cases while retaining hundreds of FP pairs. It performs well with a threshold ratio of 0.6, and the local outlier filtering and local geometry constraints proposed herein merely elevate the acceptable threshold ratio to ca. 0.8. Nevertheless, in experiment III with interference from local distortions, the description of features is less accurate than in experiments I and II, and thus the matching accuracy fails to improve as the threshold ratio is decreased. This reveals the limitations of straightforward feature matching, which reaches a bottleneck. The local outlier filtering strategy enhances the matching accuracy by ca. 10%, and local geometry constraints achieve an accuracy exceeding 80%, deemed acceptable for subsequent processes. Note that these 2 proposed strategies are based on initial matching results with an accuracy higher than 60%, and they cannot be applied to outcomes with lower accuracy or unmatched points. For better illustration, we also show a visual example in Fig. 6B. The local outlier filtering strategy is effective at removing most points not located in the overlapping regions. Furthermore, local geometry constraints help to identify false matches by checking whether lines drawn between corresponding points intersect. As indicated by the red arrows in the third image, false FP matching occurs when both points lie on the edges of a spot. This situation leads easily to misjudgment and interferes with the subsequent local distortion correction.

Module for feature reextraction

Feature-based algorithms rely heavily on the extracted FPs, particularly when addressing local distortion correction. In the context of vEM images, the inherent characteristics of specimens can lead to certain regions having few or even no FPs after the matching process. Figure 6C shows this phenomenon clearly, typically in the edge area of the organism. Within the complex and diverse internal structures of the organism, an abundance of points can be found, providing adequate information for distinct FP extraction. However, near the organism’s edges, the structures are often simple yet occasionally blurred, leading to insufficient FPs. These FPs are always lacking in distinction and may subsequently be filtered during the matching process. To illustrate this further, we show a distribution of points using a 50-pixel-wide counting box below the image. The feature reextraction strategy is designed to extract more key points on the edge region or linear structure, by edge enhancement and image fusion. Meanwhile, the reextraction strategy acts on only the selected regions, not the whole overlapping area. Consequently, compared to feature matching at the global stage, the matching process in the reextraction stage is considerably simpler and results in a larger number of accurate FP pairs, as shown in Fig. 6C. To provide quantitative insights, Table 5 gives the average number of FPs from the real datasets, with the data again organized using a 50-pixel-wide counting box. Note that in Table 5, “ratio” is the threshold ratio in the matching process, “avg_fea_num” is the average number of raw FP pairs, “avg_fea_area” is the average number in regions with few FPs, and “avg_enhanced_area” is the average number in the area after applying the reextraction strategy. As can be seen, the reextraction strategy increases the number of FP pairs substantially.

Table 5:

Average number of FPs in counting boxes

Ratio	avg_fea_num	avg_fea_area	avg_enhanced_area
0.2	3.41	1.33	2.26
0.3	15.11	3.27	8.71
0.4	41.64	3.75	14.33
0.5	87.06	3.83	19.44
0.6	147.96	3.88	26.82

Note that the proposed strategies were designed specifically for biological vEM data. The sections have a relatively fixed overlap rate, making local outlier filtering particularly effective because most matched points are located within belt-shaped regions. Meanwhile, because we are dealing with sections imaged by the same device and parameters, as opposed to natural images, there is no need to consider parallax or view-related issues. This simplifies the feature-matching process significantly, allowing us to confidently adopt rigid transformations as the best and only choice for global transformation. Furthermore, the distortions present in these sections are typically small, localized, unrelated to the structure, and sometimes even difficult to notice visually. Our proposed geometry constraints are aimed at preventing folding and severe local changes, meeting the inherent requirements. Besides, the unbalanced FP distribution is a direct consequence of the high-resolution biological structures of the specimens. These areas are concentrated primarily along the edges of the organism, thereby driving the development of our image edge enhancement strategy.

Robustness analysis

Robustness to noisy inputs

In real-world scenarios, it is usual to encounter electron microscopy images with varying degrees of background noise, which can interfere with the alignment of overlapping regions. Table 6 summarizes the average stitching metrics on 3 noise-affected simulation datasets, corresponding to the datasets in the “Simulation Experiments” section. During this simulation, we kept images and clean and added different levels of additional noise (Gaussian noise) to images and . Interestingly, with increasing noise level, the translation and rotation errors do not change significantly, which indicates the robustness of SIFT in noisy images and highlights the broad adaptability in rigid transformation estimation. Moreover, the consistently high values of NCC (all exceeding 0.9) show that our method remains effective even when working with noisy inputs.

Table 6:

Average stitching metrics on noise-affected simulation datasets

	Noise ()	0.0	5.0	10.0	15.0	20.0
I	NCC	0.999 0.001	0.988 0.003	0.958 0.010	0.915 0.017	0.865 0.022
	SSIM	0.989 0.003	0.841 0.035	0.662 0.053	0.525 0.056	0.425 0.053
	Translation error	0.010 0.003	0.015 0.006	0.026 0.010	0.037 0.016	0.054 0.028
	Rotation error	0	0	0	0	0
II	NCC	0.990 0.003	0.990 0.002	0.978 0.006	0.954 0.013	0.923 0.021
	SSIM	0.999 0.000	0.863 0.033	0.703 0.052	0.578 0.058	0.482 0.058
	Translation error	0.875 0.196	0.876 0.199	0.877 0.195	0.878 0.197	0.884 0.198
	Rotation error	0	0	0	0	0
III	NCC	0.998 0.001	0.993 0.002	0.977 0.006	0.953 0.013	0.923 0.021
	SSIM	0.957 0.018	0.836 0.041	0.682 0.054	0.561 0.059	0.467 0.058
	Translation error	5.135 1.758	5.123 1.767	5.111 1.749	5.159 1.787	5.265 1.889
	Rotation error	0.006 0.003	0.006 0.003	0.006 0.004	0.006 0.004	0.006 0.004

Robustness to deformation

Excessive local distortion definitely poses challenges for feature-based architecture. To assess the robustness of our method under different levels of simulated distortion, we conducted tests with varying iteration number N for displacement accumulation, as discussed in the “Simulation Experiments” section. To ensure that the deformation increased with increasing N, we fixed the random-number seeds for the displacement-field generation and then added them repeatedly. Figure 7A presents the NCC and SSIM values for our method both with and without local distortion correction as the iteration number N increases. The yellow curves represent the cases where only rigid transformations are considered, resulting in low NCC values of less than 0.7 and extremely low SSIM values. This observation shows that the obvious local distortion exists in the overlapping regions. By contrast, the blue curves represent our method, which consistently yields high NCC values exceeding 0.9, indicating the success of our local distortion correction. Figure 7B provides illustrative examples for , and 8, and fortunately micrographs exhibiting local deformations similar to those for and 4 are more common in practical applications.

Figure 7:

Overlapping regions evaluated for different deformation levels: (A) NCC and SSIM values for our method with or without local distortion correction and (B) examples for different deformation levels, showing local distortion becoming more obvious as N increases.

2D stitching vs. 3D alignment

Both 2D stitching and 3-dimensional (3D) alignment are fundamental to the vEM pipeline, and they both aim to identify reliable patterns within 2 images. In the feature-based alignment, 3D alignment shares the same architecture with 2D stitching, including transformation model determination, feature pattern extraction, feature correspondence construction, and final transformation parameter estimation [43–45]. Meanwhile, the combination of global rigid/affine registration and local elastic correction is also preferred for both processes. In this perspective, vEMStitch has a potential to be extended to 3D alignment. However, different imaging properties (isotropic vs. anisotropic) result in significantly different algorithm implementations. For example, in 2D stitching, pairwise stitching is generally sufficient because of the almost identical overlapping regions. By contrast, the pairwise alignment in 3D is always the intermediate result and requires further overall refinement. The anisotropic imaging makes the alignment errors easily accumulate along the z-axis. Regarding local elastic correction, 2D stitching has a simple metric to measure correction effectiveness, ensuring that the warped overlapping regions match completely. Typically, 2D stitching only warps 1 image during pairwise stitching, naturally preserving the structural continuity. On the other hand, 3D alignment faces the challenge of measuring the axial continuity along the cross-section direction, especially when the z-axis resolution is extremely lower than the planar ones. The 3D alignment considers several neighboring sections to estimate the rational distortions under necessary deformation constraints [46–48]. Unlike the “plug-and-play” approach of multiple transformations in 2D stitching, the local correction in 3D alignment must follow global transformations across the all sections.

It is worth emphasizing that seamless 2D stitching provides strong support for accurate 3D alignment. In the typical vEM pipeline, 3D alignment is performed on the stitched section stack. As previously discussed, local elastic correction relies heavily on extracted feature points. Blurred areas cannot provide accurate feature description or even cannot generate feature points. Feature points located on misaligned regions still participate in the feature-matching process, highly likely remaining in the 3D alignment. Compared to 2D stitching, 3D alignment lacks the reference of biological continuity, making it difficult to eliminate small misalignments in 2D sections. On the contrary, a clear 2D mosaic provides sufficient and reliable feature points, leaving adequate biological structure information for further processing. vEMstitch proposed in this article has a proven ability to perform reliable 2D stitching.

Conclusion

vEMstitch is an accurate and robust 2D grid-based image stitching tool that is effective at handling both simple rigid displacement and complex local distortion. Its demonstrated accuracy and visual performance make vEMstitch more applicable for large-field and high-resolution images with possible composite deformations, as in vEM applications. Although vEMstitch emphasizes local distortion correction, the use of local geometric information also makes global-rigid estimation more accurate and robust. In terms of stitching accuracy and seamlessness, vEMstitch has broader applicability and stronger deformation modeling ability compared to existing open-source tools.

Despite the effectiveness of the proposed pipeline, vEMstitch has several limitations that should be noted. It was designed for grid images with a given scanning path, and the stitching order is determined in advance, so currently vEMstitch cannot process unordered data. Also, its feature-based stitching architecture means that vEMstitch has difficulty in handling hundreds or thousands of tiles for 1 section. The current version of vEMstitch offers a simple section-based parallelism while ensuring the generation of accurate and seamless mosaics. However, the global-rigid estimation module of vEMstitch is independent of local distortion correction and could be used on a standalone basis; the simulation experiments reported in the “Simulation Experiments” section also indicated the superiority of vEMstitch in rigid parameter estimation, and users could obtain less accurate mosaics rapidly and robustly by bypassing the subsequent modules.

Methods

As shown in Fig. 2, vEMstitch comprises the following 3 main components: (i) geometry consistency constraints and RANSAC-based global-rigid estimation, (ii) FP distribution balance and local elastic correction based on TPS interpolation [49], and (iii) combined global and local transformation and final mosaic generation.

Global-rigid estimation

Abundant FPs are effective for capturing the main characteristics of images, and accurate point-set matching helps with reliable identification of similar regions in input images. For 2 input images, we extract SIFT FPs and then match them coarsely by means of the closest 2 points ratio. Large-field images of biological specimens usually have massive and locally intensive small structures, resulting in many false matches. To acquire accurate corresponding points, we remove local outliers to reduce the number of candidate points, and then we use RANSAC based on geometry consistency constraints to determine the correct matching. Finally, the closed-form rigid parameters are calculated from the above filtered FP set.

Filtering of local outliers

Effective deformation estimation is based on accurate corresponding FP pairs. Compared to global transformation, local elastic deformation considering all matching points is more sensitive to outliers. The local outlier filtering involves 2 main stages: (i) filter the outliers coarsely on a per-image basis to reduce the number of candidate points and (ii) filter finely by the distance between matching points.

In the first stage, we consider the local distribution of FPs. A reliable point should not be isolated locally, and multiple adjacent points are better for depicting changes in a small region. In elastic transformation in particular, a locally isolated point of false matching easily results in discontinuous estimated pixel displacement. We define the average axial distance between source point i and each adjacent point in a window as , and the filtering threshold is

(8)

where point j is in the window of point i and is the number of adjacent points. Experimentally, we set the to 3.

In the second stage, we study the relative distances between matching points. The displacement or deformation of an organism is locally smooth, indicating that the corresponding points of adjacent FPs are usually located closely. Similar to the first stage, we define the Euclidean distance between source point i and the corresponding one as , and the filtering threshold is

(9)

where point j is in the window of point i and is the number of adjacent points. Compared to point coordinates in the first stage, the distances between valid matching points should present a more concentrated distribution. Here, we set the to 1.5.

RANSAC based on local geometric constraints

Biological deformations are usually small and smooth, making it almost impossible to produce folding in small areas. Therefore, regarding FPs, the lines between matching points cannot cross in a local region. Furthermore, from a geometrical perspective, a pair of matching points should have the same geometric location in the neighborhood. To use the above geometric property simply and effectively, we propose a strategy combined with RANSAC for maintaining the relative positions of points. As shown in Fig. 8, given any 3 matched pairs, we determine the position of 1 point relative to the others by simple vector products.

Figure 8:

Schematic of determining relative position. Local geometry constraints aim to preserve the relative spatial positions of matched points in RANSAC.

The position judgment is then embedded into RANSAC to obtain a well-matched point set by the following procedure.

1. Input the matching FP sets of the 2 images and normalize them.

2. Randomly select 4 point pairs and calculate their respective vector products , the signs of which are used to determine whether the position relationships of the raw and corresponding points are consistent. If any point does not satisfy the condition, skip this iteration.

3. Compute the closed-form homography parameters using good points, and calculate the projection error for each point. The inner points are defined as those for which the corresponding projection error is smaller than a given threshold .

4. Calculate steps 2 and 3 circularly until the given number of random sampling (in practice, is enough). Finally, return the well-matched sets with the most inner points.

Estimation of closed-form rigid parameters

Having acquired well-matched points , we calculate closed-form rigid parameters using singular value decomposition (SVD) as follows:

(10)

where is the SVD matrix, R is the rotation matrix, t is the translation vector, and are the respective centroids of .

Local deformation correction

Local deformation estimation relies heavily on the extracted FPs and cannot work on regions without control points. Meanwhile, more abundant feature information leads to more accurate local distortion correction. However, in larger biological specimens of tissues or organs, the FPs are usually relatively sparse or even missing, and to tackle this problem, we propose feature reextraction to enrich the extracted points and ensure that they are distributed well on all the overlapping regions. TPS functions based on the above points are then estimated to generate local pixel displacement fields, and the complete pipeline is shown in Fig. 2B.

Image enhancement and feature reextraction

It may be difficult to extract features from raw images for various reasons, so an effective image enhancement method is necessary. First, we split an overlapping region into some small areas and check the number of FPs to determine where to reextract features. We then use the Laplacian of Gaussian (LoG) to extract edge information, but the LoG image may be discontinuous, which is a considerable obstacle for later feature extraction. Therefore, we smooth the results using a simple weighted fusion strategy to obtain enhanced images:

(11)

where denotes the Gaussian blurring operation and denotes the LoG operator.

The enhanced images are dominated by edge information, so we apply the corner-detection algorithm FAST [50] to the enhanced region to obtain the key points, and we use the ORB algorithm to assign orientations for each key point. After that, the fine point-set matching is the same as in the “RANSAC Based on Local Geometric Constraints” section.

Computation of displacement fields

TPS interpolation is a 2D nonrigid transformation method that is commonly used in biological deformation estimation. Here, we use matching features as the control points in TPS interpolation to compute local pixel displacement fields. Although local outliers are specifically processed in the “Filtering of Local Outliers” section, the other matching points may contain incorrect relationships, and the TPS algorithm takes all input points as effective control ones. Considering that each coefficient corresponds to the bending force provided by the respective control point, the differences between coefficients may not be significant assuming small nonlinear distortions. For improved robustness and stability, we use multiple iterations to further eliminate the outliers [34] as follows.

1. Solve the TPS linear equation to obtain the weights of each function component. The optimal solution of TPS is expressed as

(12)

where is the estimated displacement of pixel at , is a radial basis function, and are coefficients of components.

2. Collect the number of outliers as defined by the 3-sigma criterion, and reject them if the number is larger than the given threshold.

3. Recalculate using the updated parameters to obtain the outliers, and repeat the above process.

4. Obtain a good set of control points to solve the TPS linear equation.

Image blending

To extend local deformation smoothly from overlapping to nonoverlapping regions, we propose a fusion strategy that effectively combines the global-rigid and local elastic transformations. A seamless mosaic is then generated by linearly weighting the overlapping regions.

Generation of smooth pixel displacement field

To transition the local deformation field smoothly to the global-rigid transformation, we design a smooth pixel displacement field. The weight of the pixel displacement field is used to adjust the smoothness of the deformation; ranges between 0 and 1 so that the deformation function of the overlapping region decreases gradually to zero. As given by Equation 13, if the distance between the current position and the boundary of the overlapping region is greater than , then is 1, and if it is less than , then is 0:

(13)

where is the maximum distance of the pixel point from the boundary of the local deformation field. By default, the are set to 0 and the maximum displacement of nonlinear distortions, respectively.

This keeps the values of the deformation field inside the local deformation field constant and eliminates those outside the local deformation field, while the values in the transition region are obtained by weighted averaging. This achieves a smooth combination of global-rigid transformation and local elastic correction, that is,

(14)

Generation of blending masks

To obtain a linear weighting mask, we begin by smoothing the deformed image masks and linearly so that the mask value of the overlap region decreases gradually from 1 to 0. The overlap of the final blending masks and is set to . Finally, we use the smoothed mask to weight-blend the images and processed by the pixel shift field to obtain the final stitched image :

(15)

where m represents the middle position of the overlapping region; u and d are the lower and upper ends of the overlapping region, respectively; and the radius , with the parameter set to 0.15, and

(16)

Abbreviations

FP: feature point; LoG: Laplacian of Gaussian; NCC: normalized cross-correlation; RANSAC: random sample consensus; SIFT: scale-invariant feature transform; SSIM: structural similarity; SVD: singular value decomposition; TPS: thin-plate spline.

Supplementary Material

giae076_Reviewer_1_Report_Original_Submission

Yuelong Wu -- 4/16/2024 Reviewed

giae076_Reviewer_1_Report_Revision_1

Yuelong Wu -- 7/8/2024 Reviewed

giae076_Reviewer_1_Report_Revision_2

Yuelong Wu -- 8/13/2024 Reviewed

giae076_Reviewer_2_Report_Original_Submission

Erik C. Johnson, Ph.D -- 4/27/2024 Reviewed

giae076_Reviewer_2_Report_Revision_1

Erik C. Johnson, Ph.D. -- 7/13/2024 Reviewed

Author Contributions

B.H. developed the method, constructed the experiments, wrote and reviewed the manuscript. R.H. and F.Z. identified research goals and aims and provided research direction. F.S. explained the characteristics of the problem and provided technical support. Y.Z. wrote the abstract and introduction. Z.Z. wrote the draft of the Methods section. Y.C. implemented and wrote the simulation experiments.

Funding

This work was supported by the National Natural Science Foundation of China projects grant (62072280, 32371248, 61932018, 62072441, 31730023 and 31521002) and the Natural Science Foundation of Shandong Province (ZR2023YQ057).

Data Availability

The simulation data are from the Circuit Reconstruction from Electron Microscopy Images challenge [40]. Examples of simulation data and real-world data, presented in the main text, are provided in [51]. Supporting data, including real-world EM sections of mussels settling in the deep sea, are available via the GigaScience repository, GigaDB [52].

Availability of Source Code and Requirements

• Project name: vEMstitch

• Project home page: https://github.com/HeracleBT/vEMstitch

• Operating system(s): Platform independent

• Programming language: Python/C++

• Other requirements: Python version:

  https://github.com/HeracleBT/vEMstitch/blob/main/environment.yaml

• Other requirements: C++ version: OpenCV

• License: GNU GPL.

• Bio.tools: biotools:vemstitch

• RRID: SCR_025642

Competing Interests

The authors declare they have no competing interests.