Deformed alignment of super-resolution images for semi-flexible structures

Abstract

Due to low labeling efficiency and structural heterogeneity in fluorescence-based single-molecule localization microscopy (SMLM), image alignment and quantitative analysis is often required to make accurate conclusions on the spatial relationships between proteins. Cryo-electron microscopy (EM) image alignment procedures have been applied to average structures taken with super-resolution microscopy. However, unlike cryo-EM, the much larger cellular structures analyzed by super-resolution microscopy are often heterogeneous, resulting in misalignment. And the light-microscopy image library is much smaller, which makes classification challenging. To overcome these two challenges, we developed a method to deform semi-flexible ring-shaped structures and then align the 3D structures without classification. These algorithms can register semi-flexible structures with an accuracy of several nanometers in short computation time and with greatly reduced memory requirements. We demonstrated our methods by aligning experimental Stochastic Optical Reconstruction Microscopy (STORM) images of ciliary distal appendages and simulated structures. Symmetries, dimensions, and locations of protein complexes in 3D are revealed by the alignment and averaging for heterogeneous, tilted, and under-labeled structures.

Introduction

In the past decade, the development of localization-based super-resolution microscopy has brought the light microscopy to nanometer scales. Imaging beyond the diffraction limit addressed structural and functional biomedical questions of subcellular organelles that could not be resolved by conventional light microcopy. For example, the in situ dissection of macromolecular protein complexes includes ciliary transition zone [1–3], neuronal synapses [4], nuclear pore complex [5, 6], focal adhesion complex [7], clathrin-coated pits [8], centrosome [9, 10], and the escort complex at viral budding sites [11].

To obtain optical super-resolution images, the target proteins or DNA/RNA sequences are fluorescently labeled by organic dyes or fluorescent protein tags. However, the targets are usually under labeled. The cause of the low labeling efficiency includes low affinity antibodies, dye quenching, immature fluorescent proteins, and less-than-one dye to antibody conjugation ratios, etc. In certain cases, it might not be possible to achieve full labeling experimentally. Consequently, image alignment and averaging are often required to make accurate conclusions about the biological system of interest. Template-free cryo-EM image alignment procedures have been applied and adapted to average structures taken with super-resolution microscopy in 2D [12–14], and in 3D [15].

Unlike cryo-EM, the much larger cellular structures analyzed by super-resolution microscopy are often not completely rigid and homogenous. For instance, the diameter of the rings of ciliary distal appendages varies from 369 to 494 nm [1]. And the shape for the rings are often elliptical due to the flexibility of cilia. In general, many cellular organelles larger than 200 nm are semi-flexible. These semi-flexible structures keep the common symmetry and angular arrangement, but have heterogeneous size and shape, either because of elastic deformation or because of variations in molecular composition. A wide range of cellular structures have such characteristics, including centrioles [9], ciliary transition zones, ciliary distal appendages [1–3], and pre- and post-synapse density [4]. In this case, direct alignment and averaging will lose structural details. Cryo-EM particle alignment deals with structural heterogeneity using classification and class-averaging [16–18]. However, super-resolution structure analysis often has much fewer images to start with for the following two reasons. First, there are only a few target organelles in a cell. For instance, usually one embryonic fibroblast cell only develops one primary cilium [1], and each cell only has two centrioles [9]. Second, different from EM, super-resolution light microscopes detect fluorescence labels instead of electron density. Many structures seriously under-labeled cannot be used for the direct alignment and averaging. The small number of usable super-resolution images is often not enough to meet the minimal amount required for classification. Consequently, heterogeneity in semi-flexible structures will cause misalignment in direct rigid-registration used in existing algorithms for super-resolution image alignment [12–15], leading to substantial degradation of the resolution of the aligned images.

To address these problems, we take structural flexibility as one degree of freedom for image registration. We designed algorithms to first deform the semi-flexible structures to fairly uniform shape and size, and then to align all deformed structures based on cross correlations. As characterized by Fourier Ring Correlation (FRC) analysis, we have shown that our deformed alignment algorithm achieved better aligned image resolution for semi-flexible structures compared to the state-of-the-art rigid registration algorithm [12]. Although here we specifically developed our deformed algorithm for the ciliary transition zone and distal appendages [1], this procedure could be generalized for many other subcellular structures such as the nuclear pore complex and the centriole.

Methods

Overview of the workflow

We designed two algorithms to align 2D and 3D semi-flexible ring structures, respectively. The 2D deformed alignment algorithm is suitable for heterogeneous ring-shaped structures in which the heterogeneity is mainly caused by the flexibility of the structure, for example, transition zone and centrioles. The algorithm first deforms individual structures by circulating the structures, and then aligns the images. The 3D alignment algorithm was designed for randomly oriented flat structures in 3D. The random orientation causes the heterogeneity in the x-y projection. This 3D algorithm rotates the structures in 3D, and aligns the projection on the x-y plane, without structure deformation. Both deformed alignment algorithm and 3D alignment algorithm share the same alignment metric, the cross correlation with the reference in the frequency domain. And the initial reference is just the average image of the structures to be aligned. After iterations of alignment, all the images in the frequency domain are inverse Fourier transformed to the space domain. The final alignment information is used to deform, translate and rotate the coordinates in the molecule lists of the super resolution images. Both deformed alignment algorithm and 3D alignment algorithm achieved sub-pixel precision and fast alignment for linear computing time. On top of the single-color alignment algorithms, we facilitated multicolor alignment by applying the 3D alignment parameters for the reference channel to other imaged channels.

We aligned experimental data, single-color STORM images, to validate and demonstrate the utility of our deformed 2D alignment algorithm, and 3D algorithm. We used simulated structures to further test these algorithms’ capabilities to handle various image resolution, labeling efficiency, and structural complexity. In addition, we validated our two-color alignment algorithm using simulated two-color images. Because the quality of the alignment performance is reflected in the resolution of the average image of aligned super-resolution structures we employed the FRC resolution [19] to quantify the performance of different alignment methods. All software codes and sample data in this manuscript are available at https://github.com/Huanglab-ucsf/Deformed-Alignment.

STORM image acquisition

We have described the technical details of the STORM image acquisition and reconstruction methods in our previous work on ciliary transition zone [1]. The same instrument and software were used to acquire the STORM data for algorithm testing in this work. Briefly, the photoswitchable dye pair Alexa 405/Alexa 647 was used for STORM imaging with a ratio of 0.8 Alexa 647 molecules per antibody. During the imaging process, the 405 nm activation laser was used to activate a small fraction of the Alexa 647 at a time, and individual activated fluorophores were excited with a 642 nm laser. The typical power for the lasers at the back port of the microscope was ~ 1 kW/cm² for the 642nm imaging laser and 0–20 W/cm² for the 405 nm activation laser.

Analysis of STORM raw data was performed in the Insight3 software, which identifies and fits single molecule spots in each camera frame to determine their x, y and z coordinates as well as photon numbers. Sample drift during data acquisition were corrected using imaging correlation analysis. The lateral and the axial localization precision was calculated to be 17 nm (standard deviation).

Coordinate-based image deformation

Taking structural flexibility as one degree of freedom for image registration, we deform the semi-flexible structures before aligning the images (Fig 1A). Compared to pixel-based images, it is much easier to apply complex deformation functions to coordinate-based images. Fortunately, the data format of localization microscopy is a list of coordinates. Each coordinate (x, y, z) for a 3D image or (x, y) for a 2D image locates a fluorophore. First, we used robust fitting to fit the coordinates of individual structures to ellipses described as

$$\mathrm{A}{x}^2+\mathrm{B}{y}^2+Cx+Dy+E=0,$$ [1]

where the algebraic parameters A to E are converted to geometric parameters center (x₀, y₀), long axis a, short axis b, and the rotation angle ϕ.

Fig 1. Demonstration of deformed alignment algorithm on real 2D STORM data.

(A) 2D STORM images of the ciliary distal appendages of mTECs cells with CEP164 labeled. The STORM localizations of individual structure (B) are fitted to an ellipse. The Robust fitting (C, red) is better than the least-square fitting (D, blue). The ellipse is then deformed to a circle. The coordinates in the structure are moved according to the ellipse-to-circle deformation (E). The circularized structure (F) is normalized to a ring with the fixed diameter. The fixed diameter is the average diameter of total 31 original structures. Images (G) and (H) are the average image of the 31 deformed structures, and their alignment result, respectively. As a comparison, Images (I) and (J) are the average image of the 31 original structures and their alignment result, respectively. Plot (K) includes the FRC curves of average image of original images (blue) and the deformed alignment result (red). Plot (L) compares the FRC curves of the alignment results with deformation (blue) and without deformation (red). The workflow (M) describes how the iterative FFT cross-correlation alignment algorithm aligns super-resolution images in 2D. Scale bar, 100 nm.

Robust fitting is an alternative to least squares fitting when data are contaminated with outliers. It is more resistant to outliers in the data because it uses least absolute deviations rather than least squares. Because STORM images are composed of noisy localizations, robust fitting should work better than least square fitting to find a function which closely approximate the localizations. In Fig 1B–1D, we compared the ellipse fittings of a STORM image of ciliary distal appendages with robust fitting and with least square fitting, respectively. Fig 1B is a STORM image of the distal appendages of a cilium. The structure looks like an ellipse with the long axis titling towards northwest. The robust fitting results in an ellipse with the long axis titling towards northwest (red ellipse in Fig 1C), agreeing with the structure. However, the least square fitting gives an ellipse with the long axis slightly titling towards northeast (blue ellipse in Fig 1D), disagreeing with the orientation of the structure. This comparison demonstrated the advantage of robust fitting for STORM data analysis. In this algorithm, coordinates out of 1.5 standard deviations are removed to clean the images. With the geometric parameters obtained from the ellipse robust fitting, we center the structure to (x₀, y₀) and rotate all the coordinates by–ϕ around the center. Finally, we circularize the structure by simply scaling (x, y) with Eqs [2] and [3],

$${\mathrm{x}}_{\mathrm{c}\mathrm{i}\mathrm{r}\mathrm{c}}=\frac{\mathrm{x}}{\mathrm{a}}\times \mathrm{R},$$ [2]

$${\mathrm{y}}_{\mathrm{c}\mathrm{i}\mathrm{r}\mathrm{c}}=\frac{\mathrm{y}}{\mathrm{b}}\times \mathrm{R},$$ [3]

where a is the long axis, b is the short axis, and R is the average radius of all structures to be aligned (Fig 1A).

Fast image alignment using correlation in the Fourier domain

Different from our previously reported alignment framework using coordinate-based correlation, this work employs pixel-based correlation and DFT to accelerate computation speed. The DFT algorithm is a modification of the efficient subpixel image registration algorithm written by Guizar-Sicairos et al [20]. that achieved efficient subpixel image registration by up-sampled DFT cross-correlation. This algorithm registers images using 2D rigid translation. We considered sample rotation and implemented rotational registration (Fig 1M). Briefly, the original algorithm first obtains an initial 2D shift estimate of the cross-correlation peak by fast Fourier transforming (FFT) an image to register to a reference image. Second, it refines the shift estimation by up-sampling the DFT only in a small neighborhood of that estimate by means of a matrix-multiply DFT. We added a loop to rotate the images to align from 0 to 359° by 1° at a step, and then performed the original first step after each rotation step to obtain 360 cross-correlation peaks. The biggest peak provides not only the initial 2D shift estimate but also the optimal angle of rotation. The images of single structure were aligned for ten iterations. To eliminate the bias that can be introduced by a template, we used the average of all the images to be aligned as the reference image for the first iteration. Each image was translated and rotated to the reference image to maximize the cross-correlation between the image to be aligned and the reference image in the Fourier domain. The average of the images aligned in one iteration was then served as the reference image for the next iteration. Our algorithm aligned the images with a precision of 1/100 of a pixel. The up-sampling rate can be adjusted higher without increasing computation time.

We quantify the alignment using normalized root-mean-square error (NRMSE) E between f(x, y) and rotated g(x, y), defined by [4]

$${\mathrm{E}}^2={\min }_{\alpha ,x_0,y_0,\theta }\frac{{\sum }_{x,y}{\left|\alpha \bullet g_{\theta }\left(x-x_0,y-y_0\right)-f\left(x,y\right)\right|}^2}{{\sum }_{x,y}{\left|f\left(x,y\right)\right|}^2},$$ [4]

where summations are taken over all image points (x,y), α is an arbitrary constant, g_θ(x, y) is the image g(x, y) rotated by angle θ₀ as in Eq [5]

$$\left[\begin{array}{c}x\prime \\ y\prime \end{array}\right]=\left[\begin{array}{cc}cos\theta & -sin\theta \\ sin\theta & cos\theta \end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right].$$ [5]

Finding the x₀, y₀, and θ₀ for the minimum NRMSE is equivalent for the maximum cross-correlation r_fg, defined by [6]

$$r_{fg}\left(x_0,y_0,\theta \right)={\sum }_{x,y}f\left(x,y\right)g_{\theta }^{*}\left(x-x_0,y-y_0\right)={\sum }_{u,\upsilon }F(u,\upsilon )G_{\theta }^{*}(u,\upsilon )exp\left[i2\pi (\frac{u{x}_0}{M}+\frac{\upsilon y_0}{N})\right],$$ [6]

where N and M are the image dimensions, (*) denotes complex conjugation, F(μ, ν) and G_θ (μ, ν) are the DFT of f(x, y) and g_θ(x, y), respectively, as given by the relation [7]

$$F\left(u,\upsilon \right)={\sum }_{x,y}\frac{f(x,y)}{\sqrt{MN}}exp[-i2\pi (\frac{ux}{M}+\frac{\upsilon y}{N})].$$ [7]

3D alignment

Based on the 2D alignment algorithm, we developed a 3D alignment algorithm finding the minimum NRMSE (maximum cross-correlation) while rotating and translating the structures in 3D. According to Euler’s rotation theorem, combinations of rotations around any two axes can reproduce the rotation around the third axis. For example, any rotation around y axis is equal to the combination of sequential rotations around z by 90°, around x axis, and around z by -90°. So, the 3D alignment algorithm adds only one for-loop out of the 2D alignment algorithm described above and excludes the deformation step. This for-loop rotates each individual structure around its x axis. The localization coordinates (x, y, z) are rotated by angle ϕ in a preset range. The coordinates are transformed by operation

$$\left[\begin{array}{c}x\prime \\ y\prime \\ z\prime \end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& cos\varphi & -sin\varphi \\ 0& sin\varphi & cos\varphi \end{array}\right]\left[\begin{array}{c}x\\ y\\ z\end{array}\right].$$ [8]

The structures are aligned finding the x₀, y₀, θ₀, and ϕ₀ for minimum NRMSE or maximum cross-correlation r_fg.

Two-color alignment

The two-color alignment includes two steps. We first picked the channel with higher resolution as the reference channel and aligned this channel with the algorithm described above. Second, the alignment parameters for the reference channel were applied to the other channel.

Results

Coordinate-based image deformation and alignment

By deforming the super-resolution images, we correct the heterogeneity in semi-flexible structures which cause misalignment in rigid registration. The deformation improves the resolution and symmetry in the aligning result. With experimental data, we demonstrated this improvement, by comparing the average image of structures aligned by the deformed alignment algorithm (Fig 1H) to the aligning result of the rigid registration algorithm recently developed in our group [12] (Fig 1J). The FRC analysis quantified the resolution improvement made by the coordinate-based image deformation (Fig 1K & 1L). The FRC resolution of the average image of structures aligned with deformation is 41.4 ±1.2 nm, while the FRC resolution for the rigid registration is 45 ± 2.2 nm.

The algorithm was demonstrated with the experimental 2D STORM data of ciliary distal appendages of mouse tracheal epithelial cells (MTECs) with CEP164 labeled by Alexa Fluor 647. The deformed 2D alignment result (Fig 1H) of 31 under-labeled structures with semi-flexible shape and size (Fig 1A) showed clear 9-fold symmetry. The elongated distribution of CEP164 in each distal appendage was also represented in individual subunit of the average image. The elongated distribution of CEP164 in each subunit agrees with the fiber shape of the distal appendage imaged by EM [21]. 30 images can be aligned in a few minutes with a CPU at 2.8 GHz and 24 GB memory (our lab computer). And the computation time is linear to the number of the images to be aligned.

To test the capability of our algorithm to align images with various localization precision, labeling efficiency, and high structural complexity, we simulated three sets of localization images. (1) With the localization precision of 60 nm, we simulated twenty rings, each of which consists of 9 evenly distributed clusters with the diameter of 300 nm (Fig 2A average structure, B single structure). The algorithm was able to resolve 9 clusters clearly by aligning simulated structures with 10 integrations in 363 seconds on a server with 2.8 GHz CPU and 24GB memory (Fig 2C). We used autocorrelation of aligned image as a function of the angle of rotation to quantify how well the symmetry is retrieved by the alignment. The peak at 38° in the autocorrelation function (Fig 2J) agreed well with the 9-fold symmetry in the simulated images. (2) To test the alignment of structures with low labeling efficiency, we simulated 20 9-cluster rings with 5 random clusters labeled at the localization precision of 15 nm (Fig 2D average structure, E single structure). The average image of aligned under-labeled structures recovered clear 9 clusters (Fig 2F). The peak at 40° in the autocorrelation function (Fig 2K) retrieved with the 9-fold symmetry in the simulated images. (3) We also evaluated the alignment accuracy with simulated structures with higher structural complexity. We simulated 20 rings with 18 clusters which have alternative 15° and 25° angular spacing between the neighboring clusters (Fig 2G average structure, H single structure). The simulated labeling efficiency is 67%. The average image of aligned structures successfully represented the complex symmetry (Fig 2I). The peaks at 15°, 25° and 40° in the autocorrelation function (Fig 2L) again perfectly retrieved the complex angular distribution of simulated structures.

Fig 2. 2D alignment of simulated structures with various localization precision, labeling efficiency, and structural complexity.

(A) is the average image of 20 simulated 9-cluster ring structures with the localization precision of 60 nm, and image (B) is one representative of the 20 structures. Image (C) is the 2D alignment result of (A). Image (D) is the average image of 20 simulated 9-cluster rings with 5 cluster labeled at the localization precision of 15 nm, and image (E) is one representative of the 20 structures. Image (F) is the alignment result of (D). Image (G) is the average image of 20 simulated rings with 18 clusters which have alternative 15° and 25° angular spacing between the neighboring clusters at the localization precision of 15 nm, and image (H) is one representative of the 20 structures. Image (I) is the alignment result of (G). (J) plots the autocorrelation of the aligned images (C) as functions of the rotation angle. (K) is a plot of autocorrelation of the aligned images (F). (L) is a plot of autocorrelation of the aligned images (I). Scale bars 100 nm.

3D alignment

We used the algorithm to align 29 experimental 3D STORM images of ciliary distal appendages with long axes varying from 376 to 470 nm (Fig 3B). These in situ structures were randomly oriented in 3D, and their labeling efficiency was about 60–85%. Before alignment, the average structure showed a ring shape but with no information on the angular distribution of the clusters. After 5 iterations of 3D alignment, the average image showed 9 clusters with approximately even angular distribution around the ring.

Fig 3. Demonstration of 3D alignment algorithm on STORM data and simulated structures.

The workflow (A) describes how the iterative cross-correlation alignment algorithm aligns super-resolution images in 3D. The alignment is demonstrated on real 3D STORM data of the ciliary distal appendages of mouse tracheal epithelial cells with CEP164 labeled (B). The average (C) of 29 original individual CEP164 images shows no information about the symmetry of the structure. The average of aligned images (D) shows 9-fold symmetry of the distal appendages labeled by CEP164. (E) FRC analysis of the aligned image in (D). To test the capability of alignment of tilted structures, we used simulated rings with 9 clusters (F). These simulated structures were randomly rotated around x axis within 60°, around z axis within 360°, with a localization precision of 15 nm. Image (G) is the average of 20 original simulated tilted images. Image (H) is the 3D alignment result of (G). Image (I) is the average of 40 simulated structures randomly rotated around x axis within 90°, around z axis within 360°, with a localization precision of 15 nm. (J) is the 3D alignment result of (I). Image (K) is the average of 10 structures simulated localization precision of 60 nm, and randomly rotated around x axis within 30°, around z axis within 360°. (L) is the 3D alignment result of (K). Scale bars: 100 nm. The localization points are colored by their z coordinate values.

To test the capability of alignment of largely tilted structures, we simulated ten rings with 9 clusters. These simulated structures are randomly rotated around the x axis within 60° and around the z axis within 360°, at a localization precision of 15 nm (Fig 3F). The average of simulated ring images before alignment shows no information about the symmetry of the structures (Fig 3G). The 3D alignment result shows 9-fold symmetry (Fig 3H). The algorithm also provides good alignment results for simulated structures with a large tilting angle (90°) around x-axis (Fig 3I and 3J), and for large localization precision (60 nm) in 3D (Fig 3K and 3L). These results indicate that our 3D alignment algorithm can accurately register largely titled and noisy images, and can efficiently extract the common structural pattern among these heterogeneous images.

Two-color alignment

To test the algorithm of two-color alignment, we simulated twenty two-color super-resolution images with 15 nm lateral localization precision and 30 nm axial localization precision (Fig 4A). Each simulated structure is composed of two parallel rings with diameters of 200 and 300 nm, respectively. Each ring has 9 clusters that are evenly distributed. The average number of localizations in each cluster is 50. The angular distributions of the clusters of the smaller rings and the bigger ring are offset by 20°. The two rings are 100 nm distal to each other. Our alignment algorithm successfully aligned the images. The aligned average image shows 9-fold symmetry and the 20° angular offset between the two rings in the top view (Fig 4C), and 100 nm distal distance between the two rings in the side view (Fig 4E).

Fig 4. Two-color 3D alignment of simulated structures.

(A) Projections of simulated two-color super-resolution images on x-y and x-z planes, with 15 nm lateral localization precision and 30 nm axial localization precision. Image (B) and (D) are the average of 20 original simulated structures on x-y and x-z planes, respectively. Image (C) and (E) are the two-color alignment results on x-y and x-z planes, respectively. (F) Simulated two-color super-resolution images randomly titled around x-axis within 30°, with 15 nm lateral localization precision and 30 nm axial localization precision. (G) and (I) are the average of 20 original tilted simulated structures on x-y and x-z planes, respectively. Image (H) and (J) are the 3D two-color alignment results on x-y and x-z planes, respectively. Scale bars: 100 nm.

To validate the algorithm’s capability of aligning two-color images that are tilted in 3D, we randomly rotated the structures used in the case above around x axis in a range of 30°, and around z axis in a range of 360° (Fig 4F). The 3D alignment algorithm again efficiently aligned the titled structures and recovered the 3D arrangement in the average structure (Fig 4H and 4J).

Conclusions

We have demonstrated a deformed 2D alignment algorithm that can accurately align semi-flexible ring structures from both experimental STORM images of ciliary distal appendages and simulated images with high noise and complexity. Information on symmetry and common structural features were efficiently extracted from a few tens of heterogeneous structures. The cross-correlation based alignment algorithm is largely accelerated by DFT and registration in the frequency domain. We also demonstrated that our 3D alignment algorithm can accurately align images of structures tilted in space, using 3D STORM images of ciliary distal appendages, and simulated structures randomly rotated around x axis. For two-color alignment, we simply applied the alignment parameters for the reference channel to the other channel, and achieved sub-pixel alignment of two-color super-resolution images. The two-color algorithm can be easily expanded to multiple color alignment. Use of these algorithms makes accurate registration of super-resolution images within a hundredth of a pixel. Information on the general geometric feature of heterogeneous structures can be extracted with tens of images. All the algorithms including deformed 2D, 3D, and two-color alignments are computationally manageable on a regular desktop computer. The deformed 2D algorithm can be applied to any semi-flexible ring-shaped structures. The 3D and multicolor algorithms will provide substantial advantage for any applications that require aligning and averaging super-resolution images in 3D.

Data Availability

All relevant data and software code are on GitHub at https://github.com/Huanglab-ucsf/Deformed-Alignment.

Funding Statement

This project is supported by the National Institutes of Health (Director’s New Innovator Award DP2OD008479 and R01GM124334 to B.H., R01AR054396 and R01GM095941 to J.F.R., F32GM109714 to G.G.III), and by the NIH Pathway to Independence Award (K99GM126136) and the UCSF Mary Anne Koda-Kimble Seed Award for Innovation to X.S. B.H. is a Chan Zuckerberg Biohub investigator.