Improvements on marker-free images alignment for electron tomography

Graphical abstract

Highlights

• •More robust so that it can be applied to a wider range of tilt series.
• •More accurate so that the 3D reconstructions are more faithful to the acquired data and has less alignment artifacts.
• •We have also extended the deformation model to account for elastic deformations.

Abstract

Electron tomography is a technique to obtain three-dimensional structural information of samples. However, the technique is limited by shifts occurring during acquisition that need to be corrected before the reconstruction process. In 2009, we proposed an approach for post-acquisition alignment of tilt series images. This approach was marker-free, based on patch tracking and integrated in free software. Here, we present improvements to the method to make it more reliable, stable and accurate. In addition, we modified the image formation model underlying the alignment procedure to include different deformations occurring during acquisition. We propose a new way to correct these computed deformations to obtain reconstructions with reduced artifacts. The new approach has demonstrated to improve the quality of the final 3D reconstruction, giving access to better defined structures for different transmission electron tomography methods: resin embedded STEM-tomography and cryo-TEM tomography. The method is freely available in TomoJ software.

1. Introduction

Electron tomography is a technique that produces structural information of samples at a resolution between 20Åand 5Å(Kovtun et al., Sep. 2018, Mattei et al., 2018). This structural information helps scientists to determine the precise organization and relationship between the different components of samples. In biology, it has helped in understanding subcellular organelles thereby becoming a cutting-edge tool in structural biology (Donohoe et al., 2006, Pfeffer and Mahamid, 2018). In material science, the use of electron tomography is more recent, but it has already facilitated the studies going from carbon nanotubes doped with nitrogen (Florea et al., 2012) to dislocations in minerals or semiconductors (Midgley and Dunin-Borkowski, 2009).

Electron tomography relies on the tomographic reconstruction of a sample from projection images acquired at different orientations by using a transmission electron microscope. The most popular and simplest data collection geometry is the single-axis tilt series, although other geometries are also possible like dual-axis (Mastronarde, 1997), multiple-axes (Messaoudi et al., 2006, Hata et al., 2011), or conical tilt (Zampighi et al., 2005). Since single-axis tilt series is the most common and simplest method, in this paper we will focus on this technique. In single-axis tilt series, the sample is tilted around an axis to provide its different projection views. This simple approach presents some inconveniences such as the limitation of tilt range, due to the increment of the sample section to be crossed by the electrons at high tilt; mechanical instabilities during data collection producing shifted images; or deformations of images, due to small magnification change or radiation damage. Because the reconstruction algorithms are based on the existence of a single and well defined axis, the relative projection directions for each image must be established before starting the three-dimensional reconstruction process. This calls for the determination of the common tilt axis on all images, which requires the correction of shifts and/or deformations occurring during the acquisition.

Software for acquisition of single-axis tilt series is improving concerning the tracking of the sample during tilt series acquisition. However, it does not absolutely compensate for all shifts in the sample. It also tracks and corrects for changes in the defocus, but cannot compensate for magnification changes or deformations occurring in samples sensible to radiation damage. So post-acquisition alignment is mandatory to get images in the correct geometric system for reconstruction.

With biological samples, alignment of the tilt series is often performed by using gold beads serving as fiducial markers. This technique is quite effective, but since markers are not always visible or trackable, or because they are not available, it is not always applicable. In addition, the use of markers needs the intervention of user, during image processing, to manually indicate the position of fiducials on images making impossible automatic processing of the whole data set. Some papers proposed automatic detection of fiducials (Amat et al., Mar. 2008, Mastronarde and Held, 2017, Han et al., Mar. 2018). Alternatively, image corners can be used as fiducial markers (Brandt et al., 2001), or the image information content itself can be used through cross-correlation (Frank and McEwen, 1992). Electron microscopy of single particles has proved that algorithms based on image information such as cross-correlation provide the best results when a reference volume pre-exists. However, since the objects to be reconstructed in electron tomography are unique, reference volumes cannot be generated. Consequently, post-acquisition alignment requires algorithms based on information exclusively contained in images from the tilt series. Three main possibilities using cross-correlation to align tilt series exist. The first one is to align the first image with the second, the second with the third, and so on. This serial approach compares similar images, although it has the drawback of serious potential drifts due to error propagation as it reduces the alignment to multiple two-images problems, instead of considering the tilt series as a whole. This results in inaccurate estimations of the tilt axis and shift parameters for each image. The second approach computes a rough reconstruction of the tomogram and realigns the tilt series with respect to reprojections to iteratively refine the computed reconstruction. This approach, frequently used in single particle analysis (Frank, 2006), is highly time consuming and limited for tomography because of the small number of projections available to compute the first volume. The third option is to use cross-correlation to track features within the tilt series. Features are characteristic points in the images (due to change of contrast, borders, corners, etc.) that can be tracked in neighboring images. The feature tracking allows defining markers that are visible in sub-sequences of the whole tilt series.

Despite the difficulties presented for the second alignment option, some implementations are based on this strategy. This is the case of (Chen et al., 2019), where after an initial coarse alignment based on cross-correction comparison and common line methods, an initial tomogram is reconstructed. From this first volume, the darkest voxels are detected as the 3D landmarks for posterior refinement, tending to select fiducials (if present) for this purpose. Then, a landmark coordinates refinement and an image transformation optimization is computed through an iterative alignment procedure. Another fiducialess alignment implementation that relies on the second alignment option is the one presented in (Noble et al., Nov 2015) where, a first prealignment of the tilt-series is performed to subsequently reconstruct an initial volume. From this initial volume an iterative refinement process of the geometrical model and the image transformations is performed. Other implementations, as (Mastronarde and Held, 2017), relies on the third alignment option where, after an initial coarse alignment, fiducial markers are detected at the central slices for the posterior tracking of its position through the series. From this track both the angle of the tilt axis and the rotations and size changes in the images can be solved and adjusted. This implementation also allows for the fiducialless alignment of the series although the authors claim that it is somewhat less effective. In spite of all these advances, the problem of automatic tilt series alignment is still open as there are still tilt series that fail to be automatically aligned.

In 2009, we proposed an approach, based on the third option, whose improvements are discussed hereafter (Sorzano et al., 2009). This approach was as follows. First, shifts were roughly corrected by cross-correlation. Rotations and deformations were estimated using affine transforms. This was a pre-alignment required for further steps. Second, a grid was applied on each image to place seeds in regular positions. As the grid was applied on unaligned images, the seeds were not located on the same features. Third, a patch was extracted and, using the pre-alignment parameters, the corresponding patch in previous and next images was found by cross-correlation. At this step, while the correlation score between patches was high enough, the process continued to produce a landmark chain. Fourth, chains having a length smaller than a threshold defined by user were discarded. Fifth, a 3D model of landmarks was created iteratively along with detection of tilt axis orientation, in-plane shifts and rotations (incorrectly detected landmark chains were automatically identified and removed).

In this article we introduce several improvements to the previously described procedure:

• 1.We have speeded-up the prealignment of micrographs by a factor 100 using a multiscale image alignment algorithm (see Sections 2.1.1, 3.2.1).
• 2.We have enhanced the generation of seeds producing fewer and better seeds (see Sections 2.1.2, 3.2.2).
• 3.We have introduced an algorithm specially suited to spherical image features (like gold beads, but not limited to these; see Sections 2.1.2, 3.2.3).
• 4.We have made the landmark chain generation more robust by considering the reversibility of the landmark matching (see Section 2.1.3).
• 5.We have improved the fusion of landmark chains producing longer chains and of better quality (see Sections 2.1.4, 3.2.4).
• 6.We have extended the deformation model to include sample shrinkage and shearing effects as well as differences in image magnification (see Section 2.2.1).
• 7.We have included a better procedure to spot incorrectly identified projections of the 3D landmarks or incorrect parts of the landmark chains (see Section 2.2.2).
• 8.We have modified the 3D reconstruction algorithm to account for the sample out-of-plane deformations. Alternatively, we also provide two faster 2D transformations of the experimental images that allow partial correction of the out-of-plane effects. The three alternatives are shown in Section 2.3, and their results in Sections 3.1, 3.2.5.

All these improvements have resulted in a more robust and accurate algorithm. At the geometric level it has an advantage over the automatic alignments based on the refinement of a coarse 3D reconstruction, namely those methods do not consider sample deformations or variable magnifications along the data acquisition. On the other side, our method has the disadvantage that we take the tilt angles as fixed, known values (an interesting extension of the method proposed in this paper would refine the tilt angles, too). We show the performance of the method on CryoEM and resin-embedded STEM data.

2. Methods

In this section, the improvements for each step of the workflow of the algorithm published in Sorzano et al. (2009) are detailed. They have been included in the ImageJ (Abramoff et al., 2004, Schneider et al., 2012) plugin TomoJ (Messaoudi et al., 2007) and are freely available from http://www.cmib.fr/en/download/softwares/TomoJ.html. These modifications affect the method used to generate landmark chains (Section 2.1), the deformation model (Section 2.2), and the 3D reconstruction algorithm that compensates for these deformations (Section 2.3).

2.1. Feature tracking to produce landmarks chains

The first part that have been improved is the one producing landmarks chains from the tilt series. It can be decomposed in four steps: 1) pre-alignment to help future tracking, 2) detection of points of interest, 3) creation of landmark chains, and 4) selection of high quality chains.

The execution time and memory of this part of the algorithm does not depend too much on the size of images. The only part in which the full size images are used is in the detection of the seeds. The main parameters having an effect on the computation time are: 1) the number of seeds (linear), 2) the patch size (quadratic), and 3) the length of chains (linear).

2.1.1. Pre-alignment procedure simplification

In order to reduce the computation time, the entire procedure, including affine transform, can often be replaced by a simplified two-steps multiscale translation determination by cross-correlation. In this new procedure, full size images are downsampled by a factor 2 leading to an improvement of the Signal-to-Noise Ratio (SNR) as expected for an averaging filter. Then, an alignment by cross-correlation is serially performed. In addition, because the dimension of images is reduced, this leads to a gain in the computation time. The user can chose to repeat this step or to change the downsampling factor, but, as this step aims to correct for the largest translations, it is not really required. Following this first step, refinement is done by computing the cross-correlation on the center of the images at full resolution (without downsampling). Using only this central part prevents inaccuracies due to the apparition, at high tilt angle, of features in the field of view which are absent on images recorded at angles close to 0°. Section 3.2.1 shows the results obtained by this improvement.

Our implementation of this step in TomoJ allows the application of a stretching of the more tilted image in the image pair so that a more faithful comparison can be performed. In our experience, this stretching is only necessary for acquisitions with large gaps in the tilt angle. Otherwise, we have not observed a significant difference between the results of the cross correlation with and without stretching.

2.1.2. Seeds detection

Our algorithm introduced in 2009 was based on an initialization process in which the seeds were regularly placed on the intersections of a grid whose size was defined by user. However, this does not consider any information coming from images and, as a consequence many seeds were located on places with no feature to be tracked reducing the efficiency of the next steps of the workflow. Therefore, in order to optimize the initialization process, the local extrema (minima or maxima depending on image contrast) are now used as seeds. Because these extrema correspond to regions having high contrast, such as recognizable features in images or gold beads frequently used as fiducial markers, this provides an additional value to the efficiency of the determination of chains. The detailed procedure implemented consists in band-pass filtering images to remove noise, followed by the generation of a coordinates map with the position of all local minima (or maxima). This map is computed by determining if each pixel value corresponds to a local minimum within a radius given by user. For each minimum, its score is computed as the mean square difference between this pixel and its neighbors within the radius previously used. The best N seeds are kept, being N a number selected by the user. Although the seed radius and N parameters are user settings, default values are offered for both. Section 2.1.2 shows the results obtained by this algorithmic improvement.

In addition, an optimized procedure for spherical feature detection is available. As spherical 3D features will approximately project on images as 2D Gaussians, for each detected minimum, two 1D Gaussian curves are fitted to the sum of pixel values on X and Y directions respectively. This fit is done over the radius used to determine the minimum. The centers of the Gaussian curves help determining the true center of the feature. Based on the regression coefficient used as a fit score, this approach can also be used to remove non-spherical features using a score threshold defined by the user. This optimization in the selection of spherical features directly leads to a gain in computation time for the tracking part by removing uninteresting seeds (see Section 3.2.3).

2.1.3. Generation of landmark chains

The robust determination of landmark chains is a major issue to guarantee the final accuracy of the alignment process. In our previous work, the seeds were followed by using local patches and cross-correlation and multiple refinements of the chains, in forward and backward direction, were used to validate the chains. However, as proposed by Díez et al. (Apr 2010), this can be done in a more efficient way by checking if the matching of each landmark is reversible. This means that: given a patch located in position ${\mathbf{p}}_i$, in image i, and its corresponding matching position ${\mathbf{p}}_{i+1}$, in image $i+1$, the matching position of ${\mathbf{p}}_{i+1}$ in image $i,{\widehat{\mathbf{p}}}_i$ should be at the same position than ${\mathbf{p}}_i$. Ideally the distance $d({\mathbf{p}}_i,{\widehat{\mathbf{p}}}_i)$ should be zero. However, this ideal case is unreachable in a large number of cases, but moderate values still give usable matches. A maximum distance of 2 pixels is considered good enough to validate the matching as correct. Since the number of cross-correlations needed in this process is lower than that required for the procedure proposed in 2009, this procedure results in a reduction of the computation time.

In case of spherical features, an additional optimization of the landmark chains generation is possible by applying symmetry to the patch used as reference during tracking. This helps the correct tracking of spherical features when other highly contrasted structures (such as cell membranes in stained biological samples) appear in a same patch by forcing a circular shape without need of a priori knowledge about its size. This symmetrization is done by averaging 12 rotated images on 360° with 30° steps.

2.1.4. Selection of high-quality chains

The correlation score between patches used during the generation of landmark chains is highly dependent on the matching of the patches, but also on the surface occupied by features in the patch and the SNR of the images. This score is used as the criterion to stop the tracking process when it is lower than a threshold defined by the user (TomoJ provides a default value that normally does not need to be changed). However, because of the variable contribution of the three enumerated factors, it is difficult for any user to propose a correct threshold a priori. This difficulty results in an unpredictable computation time and in a number of chains highly dependent on the user input. In order to make the process more robust and reproducible, the user threshold is replaced by the length of the landmark chains. Thus, all seeds are tracked on a given number of images, corresponding to the selected length of the landmark chains, even if the correlation score is low. Each chain receives a score corresponding to the lowest correlation value obtained during tracking. Only chains with the highest scores are kept. The user may change the default number of chains to keep, with the possibility to fix a minimum correlation value. This minimum correlation value guarantees that chains with very low correlation values, corresponding to inaccurate tracking, will not deteriorate the convergence of the alignment in the next step. Note that the default values provided for all these parameters normally suffice for a standard alignment.

In addition, as some of the local minima can fall on a same structure in the sample, many landmarks chains can share a common segment, which leads to an over-representation of some regions of the images. To prevent this, a fusion of landmark chains is done when the landmarks chains share landmarks within a radius of less than two pixels on at least three images. The fusion is done by averaging the position of landmarks. This procedure has the advantage to produce longer chains than those defined by user, allowing a significant reduction in the number of images on which seeds might be tracked. This new approach makes, in a robust way, the selection of chains more independent of arbitrary and difficult to define parameters. Section 3.2.4 shows the results related to this part of the algorithm.

2.2. 3D Deformation model and estimation of its parameters

The improved landmark chains are then introduced into an alignment algorithm that will create a 3D model with the 3D coordinates of each landmarks. This 3D model was improved by adding sample deformations, and variable magnifications.

In pursuit of computational efficiency, those processes related to computing and updating reprojection errors, landmarks 3D position, and parameters of deformation are multithreaded at the level of individual image (i) or individual landmark chain (j).

2.2.1. Alignment with normalized distances

Let ${\mathbf{r}}_j$ be the coordinate of a 3D landmark, ${\mathbf{p}}_{\mathit{ij}}$ the coordinate of its projection onto image i. Let $V_j$ be the set of all images in which the j-th landmark is seen, and $V_i$ the set of all landmarks seen in image i. We assume that the relationship between ${\mathbf{r}}_j$ and ${\mathbf{p}}_{\mathit{ij}}$ is

$${\mathbf{p}}_{\mathit{ij}}=A_i{\mathbf{r}}_j+{\mathbf{d}}_i$$

where $A_i$ is the projection matrix accounting for the tilting around the tilt axis and a posterior in-plane rotation, and ${\mathbf{d}}_i$ is a 2D vector accounting for an in-plane shift of the i-th image. $A_i$ is computed as

$$A_i={\mathit{HR}}_{{\psi }_i}{R}_{{\theta }_i,{\mathbf{u}}_{\mathit{axis}}}{D}_i$$ (1)

where H is the matrix that projects 3D coordinates into its $X,Y$ components, $R_{{\psi }_i}$ is a rotation matrix of ${\psi }_i$ degrees around the Z axis (that is, the beam axis), $D_i$ is any 3D deformation matrix with which we can encode change of scale (with a diagonal matrix) or shearing along any arbitrary axis (a unitary matrix with all its eigenvalues equal to 1 and a degenerate eigenspace) or any other arbitrary 3D transformation, and $R_{{\theta }_i,{\mathbf{u}}_{\mathit{axis}}}$ is a rotation matrix of ${\theta }_i$ degrees around the tilt axis ${\mathbf{u}}_{\mathit{axis}}$. The tilt axis is described by the two Euler angles $\alpha$ and $\beta$ yielding the vector representation

$${\mathbf{u}}_{\mathit{axis}}=(\sin \beta \cos \alpha ,\sin \beta \sin \alpha ,\cos \beta )$$ (2)

Let us also refer to the product $R_{{\psi }_i}{R}_{{\theta }_i,{\mathbf{u}}_{\mathit{axis}}}$ as $R_i$. In the Electron Tomography field, it is traditional to model $S_i$ (Mastronarde, 2006) as

$$D_i=\left(\begin{array}{ccc}(m_i+s_i)\cos \left({\delta }_i\right)\hfill & 0\hfill & 0\hfill \\ (m_i+s_i)\sin \left({\delta }_i\right)\hfill & m_i\hfill & 0\hfill \\ 0\hfill & 0\hfill & m_i{t}_i\hfill \end{array}\right)$$

although we prefer the model

$$D_i=\left(\begin{array}{ccc}{m}_i{s}_i\cos \left({\delta }_i\right)\hfill & 0\hfill & 0\hfill \\ m_i{s}_i\sin \left({\delta }_i\right)\hfill & m_i\hfill & 0\hfill \\ 0\hfill & 0\hfill & m_i{t}_i\hfill \end{array}\right)$$ (3)

Both deformation models can span the same subspace of deformation matrices. In this model, $m_i$ is a global change of magnification/compression, $t_i$ is a vertical compression of the specimen, $s_i$ is a compression along X, and ${\delta }_i$ represents a shearing operation. In Mastronarde (2006)[Section 3] it is correctly argued that the thinning parameter $t_i$ cannot be uniquely determined because there is an interplay between this parameter and the tilt angle. In our implementation we do not suffer from this lack of uniqueness because we take the tilt angle as a fixed, known parameter. However, if in the future, the tilt angle is also optimized, then some regularization of the thinning parameter, the tilt angle, and the x-axis stretch may be required to guarantee that there is not a significant cross-talk between all these parameters.

The goal is to minimize the reprojection error

$$\begin{array}{ccc}\hfill E& \hfill =\hfill & {\sum }_i{\displaystyle \sum _{j\in V_i}}\Vert {\mathbf{p}}_{\mathit{ij}}-(A_i{\mathbf{r}}_j+{\mathbf{d}}_i){\Vert }_{W_j}^2\hfill \\ \hfill & \hfill =\hfill & {\sum }_i{\displaystyle \sum _{j\in V_i}}{\mathbf{p}}_{\mathit{ij}}^t{W}_j{\mathbf{p}}_{\mathit{ij}}-2{\mathbf{p}}_{\mathit{ij}}^t{W}_j(A_i{\mathbf{r}}_j+{\mathbf{d}}_i)+{\mathbf{r}}_j^t{A}_i^t{W}_j{A}_i{\mathbf{r}}_j+{\mathbf{d}}_i^t{W}_j{\mathbf{d}}_i+2{\mathbf{d}}_i^t{W}_j{A}_i{\mathbf{r}}_j\hfill \end{array}$$

where $W_j=I+\lambda S_j^{-1},S_j$ is the sample covariance of the residuals of the j-th landmark calculated as

$$\begin{array}{c}{\mathbf{e}}_{\mathit{ij}}={\mathbf{p}}_{\mathit{ij}}-(A_i{\mathbf{r}}_j+{\mathbf{d}}_i)\hfill \\ {\mathbf{e}}_{·j}=\frac{1}{N_j}{\displaystyle \sum _{i\in V_j}}{\mathbf{e}}_{\mathit{ij}}\hfill \\ S_j=\frac{1}{N_j-1}{\displaystyle \sum _{i\in V_j}}\left({\mathbf{e}}_{\mathit{ij}}-{\mathbf{e}}_{·j}\right){\left({\mathbf{e}}_{\mathit{ij}}-{\mathbf{e}}_{·j}\right)}^t\hfill \end{array}$$

and $\lambda$ is a regularization parameter that balances the importance between the Euclidean and Mahalanobis distance (we have observed that a weight different from zero helps to estabilize the alignment, especially by avoiding jumps in the zero degree image). In fact, this goal function is a generalization of our previous one, since the previous corresponds to the particular case $\lambda =0$. Note that $W_j$ is a symmetric matrix ($W_j^t=W_j$), and that at every iteration of our algorithm we only have an estimate of the matrix $W_j^{\left(k\right)}$ given by the current estimates of the parameter $A_i^{\left(k\right)}$ and ${\mathbf{d}}_i^{\left(k\right)}$.

In order to reduce the cross-talking between alignment parameters, we constrain the minimization so that

$$\begin{array}{c}\hfill \frac{1}{I}{\displaystyle \sum _i}{m}_i=\frac{1}{I}{\displaystyle \sum _i}{s}_i=\frac{1}{I}{\displaystyle \sum _i}{t}_i=1\\ \hfill \frac{1}{I}{\displaystyle \sum _i}{\delta }_i=0\end{array}$$

which is simply stating that the mean of each of the deformation parameters should be around 1 (no scaling, $m_i,t_i,s_i$) or 0 (no shearing, ${\delta }_i$).

For performing the minimization of E with respect to the different parameters, we differentiate E with respect to each parameter and equate to 0. This gives us updates for all the model parameters (different deformations of the volume, image shifts and rotations, 3D landmarks, and position of the tilt axis). The details of these updates can be seen in the Suppl. Material 1.

2.2.2. Evaluation of the landmarks quality and removal of outliers

We can also use the normalized distance introduced in this paper to evaluate the quality of the landmarks used for alignment. For doing so, we can analyze the residuals, ${\mathbf{e}}_{\mathit{ij}}$, which we have seen that play a central role along the alignment procedure, and identify those landmarks with larger residuals according the metric introduced in this paper. We can do this in two different ways:

• 1.
  Identification of large isolated residuals: We may identify those landmarks for which at least one of their residuals in at least one of the projections is larger than a given threshold:

  $$z_1=\Vert {\mathbf{e}}_{\mathit{ij}}{\Vert }_{W_j}>{∊}_1$$
• 2.
  Identification of large average residuals: We may identify those landmarks for which, on average, their residuals are larger than a given threshold:

  $$z_2=\frac{1}{N_j}{\displaystyle \sum _{i\in V_j}}\Vert {\mathbf{e}}_{\mathit{ij}}{\Vert }_{W_j}>{∊}_2$$

These thresholds may be either set by the user or estimated by some suitable procedure like calculating a high percentile (90–95%) of the distribution of the measured feature ($\Vert {\mathbf{e}}_{\mathit{ij}}{\Vert }_{W_j}$ or $\frac{1}{N_j}{\sum }_{i\in V_j}\Vert {\mathbf{e}}_{\mathit{ij}}{\Vert }_{W_j}$) over all landmarks. Alternatively, we here propose an automatic way of removing landmarks based on Mahalanobis distance. Let us define for landmark i a vector formed by its residuals ${\mathbf{z}}_i=\left(\begin{array}{c}{z}_{i1}\hfill \\ z_{i2}\hfill \end{array}\right)$. We define the covariance of all residual vectors as

$$S_Z=\frac{1}{N-1}{\displaystyle \sum _{i=1}^N}({\mathbf{z}}_i-\overline{\mathbf{z}}){({\mathbf{z}}_i-\overline{\mathbf{z}})}^T$$

where N is the number of landmarks and $\overline{\mathbf{z}}$ is the mean of all residuals. Then, for every landmark we compute its Mahalanbobis norm with respect to the distribution mean as

$$\Vert \mathbf{z}\Vert =\sqrt{{({\mathbf{z}}_i-\overline{\mathbf{z}})}^T{S}_Z^{-1}({\mathbf{z}}_i-\overline{\mathbf{z}})}$$

Those landmarks whose norm is larger than a threshold are identified as outliers. In the multivariate statistics literature it is normally accepted to use the threshold $\sqrt{p+3\sqrt{2p}}$, where p is the number of variables of the $\mathbf{z}$ vectors (in our case, $p=2$).

2.3. 3D Reconstruction and compensation of the deformations

Once the alignment has been estimated, we want to produce a reconstruction that takes into account the deformation in such a way that the reconstructed volume is undeformed. In the following we show three different methods with increasing degree of accuracy.

The 3D reconstruction algorithm of TomoJ has been implemented using OpenCL so that it can benefit from hardware architectures employing GPUs or multiple CPUs. The parallelization of the 3D reconstruction process poses some problems to a straightforward memory allocation of the projections depending on the selected method to perform the reconstruction. These difficulties are explained below.

2.3.1. Method 1: Linear mapping of the acquired images

Following the classical approach in the spirit of Mastronarde (2006), we may try to find a linear transformation that converts points in the underformed projection into points in the deformed (acquired) one. Consider a point ${\mathbf{r}}^{\prime }$ in the underformed volume $V^{\prime }$. In the undeformed projection it would be projected at the coordinate

$${\mathbf{s}}^{\prime }={\mathit{HR}}_{{\theta }_i}{\mathbf{r}}^{\prime }$$

while in the deformed projection it would be projected at the coordinate

$$\mathbf{s}={\mathit{HR}}_i{D}_i{R}_{\alpha }^{-1}{\mathbf{r}}^{\prime }+{\mathbf{d}}_i$$

We may now look for an affine transformation of one coordinate into the other

$$\begin{array}{ccc}\hfill \mathbf{s}& \hfill =\hfill & G_i({\mathbf{s}}^{\prime }+{\mathbf{d}}_i^{\prime })\hfill \\ \hfill {\mathit{HR}}_i{D}_i{R}_{\alpha }^{-1}{\mathbf{r}}^{\prime }+{\mathbf{d}}_i& \hfill =\hfill & G_i{\mathit{HR}}_{{\theta }_i}{\mathbf{r}}^{\prime }+G_i{\mathbf{d}}_i^{\prime }\hfill \end{array}$$

which implies

$$\begin{array}{ccc}\hfill {\mathbf{d}}_i& \hfill =\hfill & G_i{\mathbf{d}}_i^{\prime }\hfill \\ \hfill {\mathit{HR}}_i{D}_i{R}_{\alpha }^{-1}& \hfill =\hfill & G_i{\mathit{HR}}_{{\theta }_i}\hfill \end{array}$$

If we multiply the second equation by $H^t$ from the right

$${\mathit{HR}}_i{D}_i{R}_{\alpha }^{-1}{H}^t=G_i{\mathit{HR}}_{{\theta }_i}{H}^t$$

From this last equation we can solve for $G_i$

$$G_i={\mathit{HR}}_i{D}_i{R}_{\alpha }^{-1}{H}^t{\left({\mathit{HR}}_{{\theta }_i}{H}^t\right)}^{-1}$$ (4)

You may compare this latter equation with Eq. 6. It can be seen that the latter is a 2D version of the one in Method 2. Once we have the matrix $G_i$ we can easily calculate the affine displacement

$${\mathbf{d}}_i^{\prime }=G_i^{-1}{\mathbf{d}}_i$$

Finally, the undistorted projection image would be computed as

$$I_i^{\prime }\left({\tilde{\mathbf{s}}}^{\prime }\right)\approx I\left(G_i\right({\mathbf{s}}^{\prime }+{\mathbf{d}}_i^{\prime }\left)\right)$$ (5)

2.3.2. Method 2: Non-linear mapping of the acquired images

For reconstruction, it is convenient to have the tilt axis aligned with the vertical axis and with all shifts and deformations corrected. For this, we need to generate the images

$$I_i^{\prime }\left({\tilde{\mathbf{s}}}^{\prime }\right)={\int }_{-\infty }^{\infty }{V}^{\prime }\left({\tilde{R}}_{{\theta }_i}^{-1}{\tilde{H}}_{z^{\prime }}^t{\tilde{\mathbf{s}}}^{\prime }\right){\mathit{dz}}^{\prime }$$

where we have made used of the matrix

$${\tilde{H}}_z^t=\left(\begin{array}{ccc}1\hfill & 0\hfill & 0\hfill \\ 0\hfill & 1\hfill & 0\hfill \\ 0\hfill & 0\hfill & z\hfill \\ 0\hfill & 0\hfill & 1\hfill \end{array}\right)$$

and used homogeneous coordinates (with tilde) to refer to all positions in space (as a reminder, homogeneous coordinates for a 2D or 3D location are not unique, but it suffices to think of them as $\tilde{\mathbf{s}}={(x,y,1)}^t$ or $\tilde{\mathbf{r}}={(x,y,z,1)}^t$). Among all ${\tilde{\mathbf{r}}}^{\prime }$ points projecting onto a given coordinate ${\tilde{\mathbf{s}}}^{\prime }$ only one of them lies in the XY plane, ${\tilde{\mathbf{r}}}_0^{\prime }$

$${\tilde{\mathbf{r}}}_0^{\prime }={\tilde{R}}_{{\theta }_i}^{-1}{\tilde{H}}_{z_0^{\prime }}^t{\tilde{\mathbf{s}}}^{\prime }$$

Let ${\mathbf{r}}_{{\theta }_i,1}^t,{\mathbf{r}}_{{\theta }_i,2}^t$ and ${\mathbf{r}}_{{\theta }_i,3}^t$ be the first, second and third rows of the $R_{{\theta }_i}$ matrix. Then, it can easily be shown that (note that we have abandoned the homogeneous coordinates)

$${\mathbf{r}}_0^{\prime }=R_{{\theta }_i}^{-1}{H}_{z_0^{\prime }}^t{\mathbf{s}}^{\prime }=x^{\prime }{\mathbf{r}}_{{\theta }_i,1}+y^{\prime }{\mathbf{r}}_{{\theta }_i,2}+z_0^{\prime }{\mathbf{r}}_{{\theta }_i,3}$$

Since ${\mathbf{r}}_0^{\prime }$ lies in the XY plane, it must be

$${\mathbf{r}}_{0,z}^{\prime }=0\Rightarrow z_0^{\prime }=-\frac{x^{\prime }{R}_{{\theta }_i,13}+y^{\prime }{R}_{{\theta }_i,23}}{R_{{\theta }_i,33}}$$

In the experimental (deformed) image, this coordinate is projected at the location

$$\tilde{\mathbf{s}}={\tilde{H}}_0{\tilde{T}}_i{\tilde{R}}_i{\tilde{D}}_i{\tilde{R}}_{\alpha }^{-1}{\tilde{\mathbf{r}}}_0^{\prime }={\tilde{H}}_0{\tilde{T}}_i{\tilde{R}}_i{\tilde{D}}_i{\tilde{R}}_{\alpha }^{-1}{\tilde{R}}_{{\theta }_i}^{-1}{\tilde{H}}_{z_0^{\prime }}^t{\tilde{\mathbf{s}}}^{\prime }$$ (6)

Note that this relationship is valid only for the ${\mathbf{r}}^{\prime }$ points in the XY plane and that the relationship is non-linear because the matrix ${\tilde{H}}_{z_0^{\prime }}^t$ depends on the coordinate ${\mathbf{s}}^{\prime }$.

We can relate the ideal images needed for reconstruction with the acquired images as

$$I_i^{\prime }\left({\tilde{\mathbf{s}}}^{\prime }\right)\approx I_i\left({\tilde{H}}_0{\tilde{T}}_i{\tilde{R}}_i{\tilde{D}}_i{\tilde{R}}_{\alpha }^{-1}{\tilde{R}}_{{\theta }_i}^{-1}{\tilde{H}}_{z_0^{\prime }}^t{\tilde{\mathbf{s}}}^{\prime }\right)$$ (7)

Note that because of the 3D deformation $D_i$, this latter equation gives only an approximation to the true image $I_i^{\prime }$. The larger the deformation and the sample thickness, the worse the approximation is. However, the approximation becomes exact for infinitely thin samples. Instead of creating “undeformed” images, a more exact alternative would be to construct a 3D reconstruction algorithm whose projector explicitly takes into account the deformation. Finally, note that the output image in Eq. 7 (Method 2) is different from the output image in Eq. 5 (Method 1). The reason is that in Method 1, the affine transformation does not depend on ${\mathbf{s}}^{\prime }$ (making the transformation affine), while in Method 2 it does (making the transformation non-linear).

In the following paragraph we show that this way of generating an “undeformed” projection (Eq. 7) generalizes the current approach to projection alignment of a tilt series. In the case that there is no deformation (the assumption of the standard tomographic approach), the expression above simplifies to

$$\begin{array}{ccc}\hfill I_i^{\prime }\left({\tilde{\mathbf{s}}}^{\prime }\right)& \hfill =\hfill & I_i\left({\tilde{H}}_0{\tilde{T}}_i{\tilde{R}}_i{\tilde{R}}_{\alpha }^{-1}{\tilde{R}}_{{\theta }_i}^{-1}{\tilde{H}}_{z_0^{\prime }}^t{\tilde{\mathbf{s}}}^{\prime }\right)\hfill \\ \hfill & \hfill =\hfill & I_i\left({\tilde{H}}_0{\tilde{T}}_i\right({\tilde{R}}_{{\psi }_i}{\tilde{R}}_{{\theta }_i,{\mathbf{u}}_{\mathit{axis}}}\left){\tilde{R}}_{\alpha }^{-1}{\tilde{R}}_{{\theta }_i}^{-1}{\tilde{H}}_{z_0^{\prime }}^t{\tilde{\mathbf{s}}}^{\prime }\right)\hfill \\ \hfill & \hfill =\hfill & I_i\left({\tilde{H}}_0{\tilde{T}}_i{\tilde{R}}_{{\psi }_i}\right({\tilde{R}}_{-\alpha }{\tilde{R}}_{90-\beta }{\tilde{R}}_{{\theta }_i}{\tilde{R}}_{-(90-\beta )}{\tilde{R}}_{\alpha }\left){\tilde{R}}_{\alpha }^{-1}{\tilde{R}}_{{\theta }_i}^{-1}{\tilde{H}}_{z_0^{\prime }}^t{\tilde{\mathbf{s}}}^{\prime }\right)\hfill \\ \hfill & \hfill =\hfill & I_i\left({\tilde{H}}_0{\tilde{T}}_i{\tilde{R}}_{{\psi }_i-\alpha }{\tilde{R}}_{90-\beta }{\tilde{R}}_{{\theta }_i}{\tilde{R}}_{-(90-\beta )}{\tilde{R}}_{{\theta }_i}^{-1}{\tilde{H}}_{z_0^{\prime }}^t{\tilde{\mathbf{s}}}^{\prime }\right)\hfill \end{array}$$

where $\alpha$ and $\beta$ are the two angles defining the tilt axis (Eq. (2)), ${\tilde{R}}_{\alpha }$ is a rotation around $Z,{\tilde{R}}_{90-\beta }$ is a rotation around X and ${\tilde{R}}_{{\theta }_i}$ is a rotation around Y. If the tilt axis is perfectly perpendicular to the electron beam, then $\beta =90$ and the expression above further simplifies to

$$\begin{array}{ccc}\hfill I_i^{\prime }\left({\tilde{\mathbf{s}}}^{\prime }\right)& \hfill =\hfill & I_i\left({\tilde{H}}_0{\tilde{T}}_i{\tilde{R}}_{{\psi }_i-\alpha }{\tilde{H}}_{z_0^{\prime }}^t{\tilde{\mathbf{s}}}^{\prime }\right)\hfill \end{array}$$

Note that the translation and rotations are only performed in the XY plane, so using the ${\tilde{H}}_{z_0^{\prime }}^t$ or ${\tilde{H}}_0^t$ gives the same result. Consequently,

$$\begin{array}{ccc}\hfill I_i^{\prime }\left({\tilde{\mathbf{s}}}^{\prime }\right)& \hfill =\hfill & I_i\left({\tilde{H}}_0{\tilde{T}}_i{\tilde{R}}_{{\psi }_i-\alpha }{\tilde{H}}_0^t{\tilde{\mathbf{s}}}^{\prime }\right)\hfill \end{array}$$

That is the standard way of aligning projection images of a tilt series, that is, simply correcting for shift of each one of the images in the tilt series (${\tilde{T}}_i$) and its the angle with which the tilt axis is seen in that image (${\tilde{R}}_{{\psi }_i-\alpha }$).

2.3.3. Method 3: 3D Projector integrated in the 3D reconstruction algorithm

Each one of the images acquired responds to a model that can be described as (Sorzano et al., 2014, Sorzano et al., 2015)

$$I_i\left(\tilde{\mathbf{s}}\right)={\int }_{-\infty }^{\infty }V\left({\left({\tilde{T}}_i{\tilde{R}}_i{\tilde{D}}_i\right)}^{-1}{\tilde{H}}_z^t\tilde{\mathbf{s}}\right)\mathit{dz}\text{.}$$

For the sake of clarity let us specify each one of the matrices involved:

$$\begin{array}{ccc}\hfill {\tilde{T}}_i& \hfill =\hfill & \left(\begin{array}{cccc}1\hfill & 0\hfill & 0\hfill & d_{\mathit{ix}}\hfill \\ 0\hfill & 1\hfill & 0\hfill & d_{\mathit{iy}}\hfill \\ 0\hfill & 0\hfill & 1\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 1\hfill \end{array}\right)\hfill \\ \hfill {\tilde{R}}_i& \hfill =\hfill & {\tilde{R}}_{{\psi }_i}{\tilde{R}}_{{\theta }_i,{\mathbf{u}}_{\mathit{axis}}}=\left(\begin{array}{cccc}\cos \left({\psi }_i\right)\hfill & \sin \left({\psi }_i\right)\hfill & 0\hfill & 0\hfill \\ -\sin \left({\psi }_i\right)\hfill & \cos \left({\psi }_i\right)\hfill & 0\hfill & 0\hfill \\ 0\hfill & 0\hfill & 1\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 1\hfill \end{array}\right)\left(\begin{array}{cc}{R}_{{\theta }_i,{\mathbf{u}}_{\mathit{axis}}}\hfill & \mathbf{0}\hfill \\ {\mathbf{0}}^t\hfill & 1\hfill \end{array}\right)\hfill \\ \hfill {\tilde{D}}_i& \hfill =\hfill & \left(\begin{array}{cc}{D}_i\hfill & \mathbf{0}\hfill \\ {\mathbf{0}}^t\hfill & 1\hfill \end{array}\right)(\mathrm{SeeEq}.3)\hfill \end{array}$$

As in Mastronarde (2006), with this reconstruction method we are correcting for anisotropic changes in the specimen. Thus, the relation between the experimental images as the independent projection of each XZ plane from the specimen is broken.

We are interested in producing a tilt series whose tilt axis is not arbitrary but it is aligned with the Y axis. Let us call $V^{\prime }$ the corresponding volume. The relationship between $V^{\prime }$ and V is given by

$$V\left(\tilde{\mathbf{r}}\right)=V^{\prime }\left({\tilde{R}}_{\alpha }\tilde{\mathbf{r}}\right)$$

where $\alpha$ is the rotation angle defining the orientation of the tilt axis orientation (see Section 2.2.1) and ${\tilde{R}}_{\alpha }$ is a Z-rotation matrix with this angle. In our convention, angles are measured from the Y axis (i.e., the Y axis has $\alpha =0$) and clockwise angles are positive.

Thus, we may rewrite the acquired image in terms of the vertically aligned volume as

$$\begin{array}{c}\hfill I_i\left(\tilde{\mathbf{s}}\right)={\int }_{-\infty }^{\infty }{V}^{\prime }\left({\tilde{R}}_{\alpha }{\left({\tilde{T}}_i{\tilde{R}}_i{\tilde{D}}_i\right)}^{-1}{\tilde{H}}_z^t\tilde{\mathbf{s}}\right)\mathit{dz}={\int }_{-\infty }^{\infty }{V}^{\prime }\left({\left({\tilde{T}}_i{\tilde{R}}_i{\tilde{D}}_i{\tilde{R}}_{\alpha }^{-1}\right)}^{-1}{\tilde{H}}_z^t\tilde{\mathbf{s}}\right)\mathit{dz}\end{array}$$

Note that all points in the volume $V^{\prime }$ of the form

$${\tilde{\mathbf{r}}}_z^{\prime }={\left({\tilde{T}}_i{\tilde{R}}_i{\tilde{D}}_i{\tilde{R}}_{\alpha }^{-1}\right)}^{-1}{\tilde{H}}_z^t\tilde{\mathbf{s}}$$

project onto the point $\tilde{\mathbf{s}}$ of the experimentally observed image. Conversely, given a point ${\tilde{\mathbf{r}}}^{\prime }$ of the volume $V^{\prime }$, it is projected on the experimental image at the location

$$\tilde{\mathbf{s}}={\tilde{H}}_0{\tilde{T}}_i{\tilde{R}}_i{\tilde{D}}_i{\tilde{R}}_{\alpha }^{-1}{\tilde{\mathbf{r}}}^{\prime }$$ (8)

This need for a modified projector operator was already identified by Mastronarde (2006)[Section 5] and it has been implemented in IMOD as the Z-factors option. Our derivation here is more compact thanks to the use of the matrix notation, but the effect is totally equivalent.

Including Eq. 8 into the reconstruction process breaks the possibility to optimize parallel beam reconstruction by working on each XZ plane independently. This drawback is particularly crucial with optimization on GPU where memory is often not sufficient to hold the entire reconstruction. The reconstruction of part of the reconstruction and then joining them is problematic in the junctions, because some of the information to use in the process is not attainable. To solve this problem, it is possible to reconstruct overlapping sub-reconstruction and then copy only the parts where all information is present. Using Eq. 8 we can determine the change for volume coordinates of each point as $\widetilde{{\mathbf{r}}^{\prime }}={\tilde{D}}_i{\tilde{R}}_{\alpha }^{-1}\tilde{\mathbf{r}}$.

Knowing this, it is then easy to determine the range of the correctly reconstructed pixels from the corners coordinates of the sub-reconstruction.

$$r_{\mathit{ymin}}={\displaystyle \underset{{\theta }_i}{\max }}\max r_{{\theta }_i}^{\prime }(x_{\mathit{min}},y_{\mathit{start}}),r_{{\theta }_i}^{\prime }(x_{\mathit{max}},y_{\mathit{start}})$$

and

$$r_{\mathit{ymax}}={\displaystyle \underset{{\theta }_i}{\min }}\min r_{{\theta }_i}^{\prime }(x_{\mathit{min}},y_{\mathit{end}}),r_{{\theta }_i}^{\prime }(x_{\mathit{max}},y_{\mathit{end}})$$

Finally, to determine the start of next sub-reconstruction, we use the same approach by taking

$$r_{\mathit{nextstart}}={\displaystyle \underset{{\theta }_i}{\min }}\min r_{{\theta }_i}^{\prime }(x_{\mathit{min}},y_{\mathit{max}}),r_{{\theta }_i}^{\prime }(x_{\mathit{max}},y_{\mathit{max}})$$

Special care with these limits should be taken when $\alpha$ is higher than $45°$ or lower than $-45°$, and in these cases we must use $\alpha \pm 90$°, in order to have the correct image corners.

3. Results

We first show the ability of the algorithm to recover deformed tilt series with a phantom test. This test will also help to understand the artifacts induced by the different correction methods. We then illustrate each one of the new improvements on a standard dataset (Experimental Example 1) widely used in ours and others’ publications. Finally, we show the application of our method to the alignment of datasets in the absence of fiducial markers (Experimental Examples 2 and 3).

3.1. Phantom test

Phantom data was used to validate the new model used for alignment and prove its correctness. First, a phantom volume was created as white dots randomly distributed in a dark background. The projection of the points were computed using Eq. (1) with variations of shifts, rotation, and deformation parameters following sine functions. The maximum values were 50 pixels for shift, $2°$ for in-plane rotation (${\psi }_i$), $0.02$ for magnification ($m_i$), $0.05$ for shrinkage ($t_i$), $0.1$ for scale on X axis ($s_i$) and $0.5°$ for shearing angle (${\delta }_i$). The reader is referred to the Section 2.2.1 for a detailed explanation of these parameters. The coordinates on the projection images were stored and then used as input to the procedure to determine all parameters for alignment and deformation. The parameters of deformation found were very similar to the true ones (Supplementary Material 2). However, to quantitatively assess the effect of the small differences in the deformation parameters, we calculated the warping index between the true transformations applied to the phantom at tilt $i,D_i$, and the estimated ones, ${\widehat{D}}_i$, (Sorzano et al., 2005)

$$\mathit{WI}=\frac{1}{N_{\mathit{voxels}}{N}_{\mathit{tilts}}}{\displaystyle \sum _{j=0}^{N_{\mathit{voxels}}}}{\displaystyle \sum _{i=0}^{N_{\mathit{tilts}}}}\parallel {\mathbf{r}}_j-{\widehat{D_i}}^{-1}\left(D_i\right({\mathbf{r}}_j\left)\right){\parallel }^2$$

where $N_{\mathit{voxels}}$ and $N_{\mathit{tilts}}$ are the number of voxels in the phantom and the number of tilts images created respectively. ${\mathbf{r}}_j$ is the $j^{\mathit{th}}$ voxel coordinate in the phantoms. On the phantom data used in Fig. 1 the warping index was 0.24 voxels. This difference is in the subvoxel accuracy, proving the accuracy of the algorithm.

Fig. 1.

Comparison of the reconstructions of the phantom with the different method of correction of deformation. Each image corresponds to the sum of XZ planes to better see the artifacts: (A) no correction of deformations, (B) correction using 2D affine transforms, (C) correction using 2D non-linear mapping, (D) optimal reconstruction if no deformation exists, (E) correction using 3D projector integrated in reconstruction, (F) correction using 3D projector integrated in reconstruction with deformations computed from landmarks. White arrows point to features. Near the central plane (thin arrows), the feature are well reconstructed with all methods of correction of deformation. The features above (dotted arrows) or far (large arrows) from central plane display correction of the in-plane deformation, but show triangular shapes unless the 3D projector is employed.

In Section 2.3, we discussed three different ways of correcting the estimated deformations. To show their effect, three different reconstructions were performed using the true deformations parameters. As can be seen in Fig. 1, the points on the central plane are perfectly reconstructed with the different methods of correction of deformations (thin white arrows). The points that are not on the central plane are perfectly reconstructed only in the case of applying alignment with the 3D projector during the reconstruction process (Method 3; Fig. 1 E and F). With the other two methods (B and C), the points are reconstructed as triangles (dotted or large white arrows).

3.2. Experimental example 1

In Section 2.1 we have introduced several modifications for the detection and tracking of landmarks. In this section we test these ideas along with the analysis of the different ways of estimating (Section 2.2) and correcting (Section 2.3) for the deformation. To evaluate these ideas, several tests were performed on the TomoJ tutorial data (Pyrodictium abyssii) cell strain TAG11 (Rieger et al., 1995), embedded in vitreous ice layer on holey carbon grid (Sorzano et al., 2009).

3.2.1. Multiscale prealignment with cross-correlation

The total computing time was reduced from 388 s. in our previous method to 4 s. in the method introduced in this paper using a laptop computer equipped with an Intel i7-4700MQ processor and 32 GB RAM. This implies a speed-up factor of two orders of magnitude. We measured the quality of the new prealignment by evaluating the number of landmark chains and their alignment score. The alignment score of the previous algorithm (Sorzano et al., 2009) and the new one are very similar, 1.37 and 1.38, respectively. The alignment score is the average error in pixels between the reprojection of the 3D landmarks and their experimentally observed positions. However, the number of landmark chains dropped from 2,684 to 2,127. Although there is a drop of about 20% in the number of chains, there is no drop in the quality of the detected chains, and the execution time has been reduced by a factor 100.

3.2.2. Seed generation

We evaluated the ability of the new methods to generate seeds using the ratio of final retained chains versus the number of initial chains (Table 1). We see that in our previous method the success rate of a seed was about 5.8%, that was the probability of a grid point hitting a feature that could be tracked along the tilt series. We see that the new method is much more successful resulting in a larger number of final chains and higher success probability. The fact that adding more seeds per image in the new method reduces the success rate can be explained by the fact that high quality seeds are chosen first, and adding more seeds per image results in a degradation of the quality of the added seeds.

Table 1.

Comparison of the method of seeds determination. For our previous and current methods. The table shows the number of seeds per image, the number of final chains, the success probability of a seed to generate a final chain, and the alignment score in pixels.

Grid (Sorzano et al., 2009)
Seeds/image	Final chains	Success (%)	Alignment score
400	2,127	5.8	1.47
225	1,196	5.8	1.48
100	528	5.8	1.33
49	265	5.9	1.66
25	98	4.3	1.21
Local minima (new method)
Seeds/image	Final chains	Success (%)	Alignment score
400	3,194	8.6	1.46
225	3,074	14.7	1.48
100	2,643	28.4	1.51
49	1,573	34.1	1.36
25	867	36.1	1.13

3.2.3. Spherical feature optimization

The spherical feature optimization plays a role at two levels. First, the detection of seeds by filtering with Gaussian function. The effect of this step is to reduce the number of seeds to track in the subsequent algorithm. The activation of this option with a threshold of fit of 0.5 gives 5,792 seeds instead of the 18,592 found after the detection of local minima. The number of chains found dropped from 7,708 to 3,176 (1,638 and 943 after fusion respectively) and the final alignment score is very similar: 1.26 and 1.20 without and with gold bead optimization, respectively. The main gain is the computation of only 31% of the potential seeds at the expense of computing the Gaussian fitting. This is confirmed by the total time to compute alignment (detection and alignment) that drops from 10 min and 22 s to 4 min and 46 s.

The second role of spherical feature optimization is in the tracking of the seeds. The effect of the symmetrization during tracking resulted in a lower number of chains found, 1,992 (166 after fusion), compared to the tracking without symmetrization, 3,176 (943 after fusion). First, the fusion is more efficient with the symmetrization with only 8% of the chains remaining instead of 29%. Second, after fusion the length of landmark chains improves from an average length of 35 to an average length of 47. The number of tracked chains highlights the fact that the symmetrization limits the selected features to spherical ones. The high level of fusion proves that few distinct features are found by the procedure. Moreover, the interest of the procedure for spherical features is confirmed by the increase of average length of landmark chains as long chains help the future alignment. Adding this symmetrization step allows a new drop in the total computation time to 2 min and 55 s.

As can be observed in Fig. 2 the algorithm found many spherical features of different sizes. Some of them are gold beads present on sample as expected. However, many are just particles present in ice or even proteins of the archaeon.

Fig. 2.

Visualization of landmarks chains found with the gold bead-like optimization. A) The landmarks found are displayed with green circles on the projection image near $0°$. B) zoom on a region with gold bead. C) zoom on a region with no gold bead, but sufficiently spherically symmetric, distinctive features.

3.2.4. Tracking of landmark chains

In Section 2.1.4, we have proposed a cross-validation procedure to track a seed along the different tilt series images. The use of this procedure increased the number of chains from 3,036 to 3,701 (increase of 21%) while keeping similar alignment score (1.48 and 1.54 respectively).

The use of fixed sized chains proves to be efficient. A first experiment was done with the correlation threshold approach with a minimum size of 15 and threshold of 0.9. It produced 3,702 chains with an average length of 52 and correlation scores ranging from 0.90 to 0.96. For the fixed size approach, chains of size 31, the algorithm produced twice the number of chains (7,708) with correlation scores ranging from 0.5 (minimum accepted score) to 0.96. After alignment, the score obtained slightly improved (1.09 for fixed size and 1.54 for the old approach).

The fusion of landmarks reduced the number of chains from 7,708 to 1,638 chains. The fused chains had an average length of 39 images. The chain fusion on the alignment caused a slight increase of the alignment score from 1.09 to 1.25.

3.2.5. New reconstruction algorithm with deformation compensation

The new reconstruction algorithm was first compared with the previous version. The reconstructions, Fig. 3, show an important improvement. As can be seen, the gold beads pointed by white arrows display a higher circularity and no banana shape.

Fig. 3.

Comparison of reconstructions obtained with different alignment models. The first column corresponds to alignment using our old alignment model (Sorzano et al., 2009), the second column displays the result of the new model with only shifts and in-plane rotation (this experiment shows the improvements not due to the 3D deformation model), and third column shows the result of the new model with deformation correction. The top row corresponds to the central XY plane. The second and third rows correspond to the XZ plane number 49 and 315, respectively. White arrows points to features with visible change between algorithms. Black arrows points the plastic of holey grid where change in shape is visible. The last column corresponds to the comparison of a feature with visible differences between the three reconstructions. The scale bar corresponds to 500 nm.

Next, we explored the effect of the three different deformation compensation algorithms. The three compensation schemes were tested. As Fig. 4 shows, visible changes between the three methods are present. In the case of application using 3D projector during reconstruction (Method 3) the gold beads pointed by white arrows display less reconstruction artifacts.

Fig. 4.

Comparison of the methods of application of alignment with deformations. The top row corresponds to the central XY plane. The second and thirds row correspond to the XZ plane number 49 and 315, respectively. The first column corresponds to method 1 with linear mapping. The second column corresponds to method 2 with non-linear mapping. The third column corresponds to method 3 with application during reconstruction process. White arrows point to features with visible change between algorithms. The last column corresponds to the comparison of a feature with visible differences between the three methods of application of alignment with deformations.Scale bar corresponds to 500 nm.

3.3. Experimental example 2

The method was applied on an acquisition of Trypanosoma brucei cilia embedded in Epon resin. The electron tomography was performed using a JEM 2200FS microscope (JEOL, Tokyo, Japan) operated at 200 keV in STEM bright field mode. The tilt series ranged from −75.62° to  + 77.62° with a Saxton scheme (Saxton, 1978) for a total number of 97 images. The images, of a size of $2,048\times 2,048$ pixels, were recorded at nominal magnification of 100,000X with a pixel size of 1.46 nm. at specimen level. Images were binned by a factor 2 before processing. The algorithm was applied using the following parameters. For prealignment, the two pass cross-correlation approach was applied on a region of interest (ROI) centered on image center. First pass was with a size of $720\times 720$ pixels followed by a factor 2 binning. The second pass was performed on a $360\times 360$ pixels size ROI with no additional binning. The generation of landmarks was done using local minima detection (radius 4 pixels) on band-pass filtered images (with radius in Fourier images 0 and 256 pixels). 1,800 seeds were detected on each image (on the 95% central part). The tracking length was 24 images using a $11\times 11$ pixels size for the patch with a minimum allowed correlation of 0.5. After tracking, 111,963 chains were found. After fusion of these chains, 24,575 remained.

Two independent alignments were produced from the fused chains. The first was with only shift and rotation, the second with all the deformations defined in Eq. 3. With only shifts and rotation the final score was 1.99 (average residuals 1.20, isolated residuals 2.79) with 18,083 chains remaining. With deformations, the final alignment score was 1.81 (average residuals 1.09, isolated residuals 2.53) with 21,243 chains remaining. Reconstructions were done using SART algorithm with 20 iterations and a relaxation coefficient of 0.1. As can be seen in Fig. 5, the two reconstructions of the cilia seems at first similar proving that the alignment procedure gives good alignment. However, close examination of the reconstructions shows clear differences such as microtubules under the membrane in the bottom left corner of the central XY plane. Without correction of deformations some part of the volume are very good while others are blurred. With correction of deformations, the reconstruction is well defined in all areas.

Fig. 5.

Comparison of the reconstructions of Trypanosoma Brucei cilia obtained with or without deformation. A and C: Central slice in XY and XZ direction respectively from the reconstruction with deformation correction, B and D: Central slice in XY and XZ direction respectively from the reconstruction without correction of deformations. a, b, c, d, e zoomed-in region corresponding to the areas delimited by white squares on central slices from reconstruction with deformation correction. a’, b’, c’, d’, e’ zoomed-in region from reconstruction without correction of deformations corresponding to the same areas as a, b, c, d, e respectively (delimited by white squares on central slices). White arrows points to details improved in the reconstruction with deformation correction. The scale bar corresponds to 200 nm.

3.4. Experimental example 3

The method was applied on an acquisition of T5 bacteriophages. Bacteriophages T5st(0), a heat-stable T5 mutant, were produced in E.coli Fsu$\beta$+ and purified on a CsCl gradient as described in Boulanger et al. (1996). They were dialysed against 10 mM Tris–HCl pH 7.5, 100 mM NaCl, 1 mM MgCl2, 1 mM CaCl2, and stored at 4° C at a DNA concentration of 7 mg/ml. 3–4 $\mu l$ of the phage suspension were deposited onto a glow-discharged cryoEM grid (Quantifoil R2/2). The grid was blotted with a filter paper for 4 s, and plunged into liquid, using a Vitrobot Mark IV operated at room temperature and 100% relative humidity. Tilt series were collected at 300 kV on a Titan Krios electron microscope equipped with Volta phase plates, a GATAN GIF Quantum post-column energy filter, and a K2 Summit direct electron detector, at a pixel size of 2.185 Å, using the Serial EM software (Mastronarde, 2005). A dose-symmetric recording scheme was used, from a starting angle of $0°$ in a $\pm 60°$ angular range, with an angular increment of $2°$. The electron dose was set to $1.2e/$Ų for individual images, corresponding to a total dose of $73.2e$/Ų. Images were binned by a factor 2 before processing.

The algorithm was applied using the following parameters. For prealignment, the two-step cross-correlation correction was applied. First step was performed with a size of $1,024\times 1,024$ pixels followed by a factor 2 binning. The second step was performed on a $512\times 512$ pixels size ROI with no additional binning. The generation of seeds was accomplished with a local minima detection (radius 15 pixels) on band-pass filtered images (with radius in Fourier images 2 and 125 pixels) arising 2,000 seeds detected on each image (on the 90% central part). The tracking length was 12 images using a $41\times 41$ pixels size patches with a minimum correlation allowed of 0.3. The algorithm identified 787 chains found after fusion. The alignment with all deformations was computed with this new chains resulting in a score of 3.33 (average residual 2.21, isolated residual 4.48) with 444 chains remaining. The reconstruction performed was a SART with 600 iterations and a relaxation coefficient of 0.01.

As can be seen in Fig. 6A, the viruses were well defined (the zoomed views are filtered using a 3D Gaussian blur of radius 1). Around half of them were intact and still contain their DNA inside (Fig. 6, B and C) while the other half was empty with broken capside. Interestingly, three of the empty viruses seemed to have some DNA chains still connected to the tail. The tails were well defined (Fig. 6, H and I) presenting a helicoidal periodic structure clearly identifiable (Fig. 6, I’) with a pitch of 43.6Åand a twist of 38.66° (in Effantin et al. (2006) the values described were 43.8Åand 39.4° respectively). The tail end is capped with the central fiber and lateral fibers were connected (Fig. 6, E and F). Some of the tails have lost the capping by the central fiber, corresponding to empty viruses, even if no link to this state can be made at the moment (Fig. 6, G). The only gold bead present in this sample showed no deformation (Fig. 6, J and J’).

Fig. 6.

Reconstruction of bacteriophage T5. A) plane from the reconstruction. B) to D) zoom in virus either filled with DNA (B and C) or empty except for a brand of DNA going through the tail (D, white arrow). E) to G) zoom in tail extremity. When viruses are filled with DNA the tail fibers are clearly identifiable in E) and F): central fiber is pointed with a white solid arrow, and lateral with a dotted arrow. When viruses are empty, the central fiber is lost and the tail is open. H) zoom on tail, I) average of a tail, and I’) its Fourier transform: spots correspond to the helical structure and the pitch angle (green lines) presents a value of 38°. J) and J’) gold bead view in XZ plane. Scale bar corresponds to 200 nm.

To give a glimpse into the execution time required to perform the different parts of the algorithm, we illustrate these times with the data from this example. The algorithm was executed in a laptop with 4 i7 CPU cores. The prealignment of the tilt series and the generation of the seeds takes between 15 to 30 s, depending on the different choices (whether to include all the parameters (shifts, rotations, magnification, shrinkage, scaleX and shear) or just a subset of them). Then, each iteration of chain evaluation, calculation of the 3D landmarks, and outlier removal takes between 2-7 s depending on the number of landmark chains (in our example it ranged from 500–800). The 3D reconstruction time of a volume with size 1,854$\times$1,919$\times$600 voxels is very quick (less than 1-2 min in CPU) with Method 1 (Section 2.3.1); while it takes 40 min with Method 2 (Section 2.3.2) in CPU or 5 min in GPU; and 50 min with Method 3 (Section 2.3.3) in CPU or 6 min in GPU.

4. Conclusions

We have shown with phantom data that the new methods are capable to properly estimate the underlying deformations. The correction of these deformations was shown to be better when done during 3D reconstruction than when applied on 2D projection images. This can be explained by the fact that the two methods correcting deformations by modifying the 2D projections (Methods 1 and 2) consider that all points on the images come from the same plane (see Method 2 derivation). However, these two corrections provide fast approximations to the more expensive 3D full correction. The parameters used to create the phantom data are larger than the ones usually found on experimental data, thus, this artifact is much less visible in experimental data, although it is present.

The differences between the affine transform and the nonlinear transform are small, even if in theory the nonlinear version should be better. The main drawback of the affine transformation (Method 1) is that it is correct only in the central plane, but not in the other planes of the reconstruction. On the other side, the nonlinear correction (Method 3) is computationally more demanding as the step to create the aligned image is more complex.

On experimental data, the improvement of seed detection and tracking helps to produce better chains with easier parameterization of the algorithm. The alignment with deformations proved to be worthy even on cryo-tomography data. The algorithm permits a detailed analysis of the different parameters in the deformations. The correction of the deformation proved to be more effective with Method 3. The method proved effective in aligning cryo-tomography data without the need of gold beads.

Throughout the manuscript we have referred to parameters and thresholds that can be selected by the user. In our TomoJ implementation we have provided default parameters that normally do not need to be fine tuned. However, they are still available so that the user can have better control in difficult cases.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Supplementary data

The following are the Supplementary data to this article:

Additional file 1 – Demonstration of model parameters determination.

The file contains the mathematical explanations to determine all parameters of the model with deformation.

Additional file 2 – Excel file containing all parameters of shifts, rotations and deformations applied to phantom data and the parameters found by the algorithm.

First page is the original parameters, second page corresponds to computed parameters and third page display graphs to compare the parameters.