Image Resolution in Optical Nanoscopy

Abstract

Resolution in optical nanoscopy depends on the localization uncertainty of single fluorescent labels, the density of labels covering the sample, and the sample’s spatial structure. Currently there is no integral, practical resolution measure that takes all factors into account. Here we introduce such a measure that can be computed directly from the image. We demonstrate its validity and benefits on 2D and 3D localization microscopy images of tubulin and actin filaments. Our approach makes it possible to compare achieved resolutions in images taken with different nanoscopy methods, optimize and rank different emitter localization and labeling strategies, define a stopping criterion for data acquisition, describe image anisotropy and heterogeneity, and, surprisingly, estimate the average number of localizations per emitter. Our findings challenge the current focus on obtaining the best localization precision, but instead show how the best image resolution can be achieved as fast as possible.

The first and foremost law of conventional optical imaging science is that resolution is limited to a value on the order of λ/NA, with λ the wavelength of light and NA the numerical aperture of the imaging lens. Rayleigh and Sparrow captured this law by empirical resolution criteria. These were placed on solid foundations by Abbe and Nyquist, defining resolution as the inverse of the spatial bandwidth of the imaging system. This diffraction limit, however, has been overcome by numerous optical nanoscopy techniques in the last decade, notably STED¹, RESOLFT², the family of localization microscopy techniques such as PALM, STORM, GSDIM, and dSTORM^3–6 and statistical methods as BLINK and SOFI^{7, 8}.

These revolutionary developments raise the question: what is resolution in diffraction-unlimited imaging. The resolving power of the instrument is often coupled to the uncertainty of localizing single emitters, that is, point sources. The closely related two-point resolution can be given a precise meaning in the context of localization microscopy⁹, thus generalizing the Rayleigh criterion of conventional microscopy. These concepts characterize the scale at which an image of a limited number of molecules of interest is resolved when they have all been labeled and successfully imaged. If, however, more or less continuous structures with a very large number of potential labeling sites are imaged, such as actin filaments or organelle membranes, then it is clear that the average density of localized fluorescent labels must also play a role. As early as the first demonstration of localization microscopy for cell imaging³, it was noted that “both parameters - localization precision and the density of rendered molecules - are key to defining performance…”. The effect of labeling density and photoswitching kinetics on resolution has since been investigated experimentally^{10, 11}. Recently an estimation-theoretic resolution concept was presented by Fitzgerald et al.¹² that combines both labeling density and localization uncertainty using an a-priori model of the sample.

We conclude from all prior work that neither the average density of localized molecules needed for random Nyquist sampling nor the localization uncertainty alone are suitable measures to assess the resolution. In addition, the resolution depends on a multitude of other factors such as the link between the label and the structure, the underlying spatial structure of the sample itself, and the extensive data processing required to produce a final super-resolution image comprising e.g. single emitter candidate selection and localization algorithms. Only an integral, image-based resolution measure, not depending on any a-priori information, is ultimately suitable to determine what level of detail can be reliably discerned in any given image.

RESULTS

Here, we propose a new image resolution measure, termed Fourier Image REsolution (FIRE), that can be computed directly from experimental data alone. FIRE is centered on the Fourier Ring Correlation (FRC) or equivalently the spectral signal-to-noise-ratio (SSNR), which is in common use in the field of cryo-electron microscopy to assess single particle tomographic reconstructions of macro-molecular complexes^13–15. For FIRE we divide the set of single emitter localizations that constitute a super-resolution image into two statistically independent sub-sets, yielding two sub-images f₁ (r⃗) and f₂ (r⃗), where r⃗ denotes the spatial coordinates. Subsequent statistical correlation of their Fourier transforms f̂₁ (q⃗) and f̂₂ (q⃗) over the pixels on the perimeter of circles of constant spatial frequency magnitude q = |q⃗| gives the FRC ¹⁴:

$$FRC(q)=\frac{\sum _{\overrightarrow{q}\in \text{circle}}{\widehat{f}}_1(\overrightarrow{q}){\widehat{f}}_2{(\overrightarrow{q})}^{*}}{\sqrt{\sum _{\overrightarrow{q}\in \text{circle}}{\widehat{f}}_1{(\overrightarrow{q})}^2\sum _{\overrightarrow{q}\in \text{circle}}{\widehat{f}}_2{(\overrightarrow{q})}^2}}.$$ (1)

For low spatial frequencies the FRC curve is close to unity and for high spatial frequencies noise dominates the data and the FRC decays to zero. The image resolution is defined as the inverse of the spatial frequency for which the FRC curve drops below a given threshold. We evaluated different threshold criteria used in the field of electron microscopy^{13, 16–18} and found that the fixed threshold equal to 1/7 ≈ 0.143 (ref. 18) is most appropriate for localization microscopy images (Supplementary Fig. 1, Supplementary Methods, sec. 1). Figure 1a illustrates the FIRE concept and the steps needed to compute it. FIRE describes the length scale below which the image lacks signal content. Details in the image smaller than this value are not resolved. Resolution values as given by FIRE will always be larger than those based on localization uncertainty or labeling density alone (Supplementary Fig. 2).

Figure 1. The FIRE principle and trade-off between localization uncertainty and labeling density.

a, All localizations are divided into two halves, the correlation of their Fourier transforms over the perimeter of circles in Fourier space of radius q is calculated, resulting in an FRC curve indicating the decay of the correlation with spatial frequency. The image resolution is the inverse of the spatial frequency for which the FRC curve drops below the threshold 1/7 ≈ 0.143, e.g. a threshold value at q = 0.04 nm⁻¹ is equivalent to 25 nm resolution b, Simulated localization microscopy image of a line pair with mean labeling density in the area occupied by the lines ρ = 2.5 · 10³μm⁻² and localization uncertainty σ = 7.6 nm (line distance 70 nm, cosine squared cross-section as defined in Supplementary Note 2). c, Theory (lines) and simulation data (circles) of constant FIRE value (20, 40, . . ., 100 nm) for line pairs as in b as a function of localization uncertainty and labeling density in the area occupied by the lines. Regions of localization uncertainty limited resolution (blue) and labeling density limited resolution (yellow) are separated by the red line ρσ² = e/6π. d, Localization uncertainty versus image resolution for different fixed total measurement times (1, 5, 10, 20 and 30 min.). Camera frame rates were varied to match the on-times of the emitter. The minima of the curves fall on the red line FIRE = 2πσ that separates the yellow region in which not enough emitters have been localized from the blue region in which the emitters have not been localized precisely enough.

Theoretical considerations and simulations

FIRE allows predictions about the impact of different imaging and sample parameters on the achievable resolution based on the expectation value of the FRC curve, which is given by (Supplementary Fig. 3–4, Supplementary Note 1):

$$\langle FRC(q)\rangle =\frac{\sum _{\overrightarrow{q}\in \text{circle}}(Q+N{|\widehat{\psi }(\overrightarrow{q})|}^2)\text{exp}(-4{\pi }^2{\sigma }^2{q}^2)}{\sum _{\overrightarrow{q}\in \text{circle}}[2+(Q+N{|\widehat{\psi }(\overrightarrow{q})|}^2)\text{exp}(-4{\pi }^2{\sigma }^2{q}^2)]},$$ (2)

where N is the total number of localized emitters, σ is the average localization uncertainty, and ψ̂ (q⃗) denotes the Fourier spectrum of the spatial distribution of the fluorescent emitters. The parameter Q is a measure for spurious correlations due to e.g. repeated photo-activation of the same emitter. Each emitter contributing to the image is localized once for Q = 0 and in general Q/ (1 − exp(−Q)) times on average, provided the emitter activation follows Poisson statistics. Careful analysis of the spatio-temporal correlations in the image and the emitter activation statistics (including effects of photobleaching) can provide a way to estimate Q and correct for its effect on image resolution, as well as to estimate the number of fluorescent labels contributing to the image, as will be discussed below.

Analytical expressions for FIRE can be derived for particular object types often used in resolution definitions, such as line pairs (Supplementary Note 2). FIRE for an image consisting of two parallel lines with a cosine squared cross-section and mean labeling density ρ in the area occupied by the lines is:

$$R=\frac{2\pi \sigma }{\sqrt{W(6\pi \rho {\sigma }^2)}},$$ (3)

where W(x) is the Lambert W-function or Omega-function, which is defined as the inverse of the function x exp(x) ¹⁹. Two regimes can be identified in which either labeling density or localization uncertainty has the most impact on improving resolution. The boundary between these two regimes is found by setting the relative gain in resolution due to a change of either quantity equal to each other. This trade-off occurs at R = 2πσ (Supplementary Note 3), which corresponds to:

$$\rho {\sigma }^2=\frac{e}{6\pi }\approx \mathrm{0.14.}$$ (4)

Figure 1c shows these two regions; ρσ² < e/6π is labeling density limited, ρσ² > e/6π is localization uncertainty limited. The exact boundary between the two regimes depends on the underlying object, so the boundary value for the two-line example serves only as a rule-of-thumb (see Supplementary Note 3). For example, for M parallel lines we obtain a value e/3πM. From this it may be inferred that for any intricate but irregular object structure the trade-off occurs for a value smaller than 0.14.

The same trade-off as above may also manifest itself in the optimization of image resolution given a fixed total acquisition time, as shown in Figure 1d. Suppose that the photon count per localization is improved by increasing the on-times of the emitters while keeping the emitters’ brightness and the number of simultaneously active emitters constant, then this also reduces the total number of labels that can be localized in a given acquisition time. Therefore longer single emitter events yield more accurate localizations but at the expense of a lower recorded emitter density ^{3, 20}. Again, the optimum is R = 2πσ, independent of the object (Supplementary Note 3). Tuning the on-times as described here may be done in the design phase of an experiment by the choice of label or buffer composition.

Resolution build-up during data acquisition and effects of data processing

In order to test and evaluate FIRE, we have imaged tubulin networks in fixed HeLa cells labeled with Alexa 647 with localization microscopy (Fig. 2a). The FIRE value improves with acquisition time (Fig. 2b–f), or equivalently with the density of localized labels. The trade-off point between the localization density and localization uncertainty limited regimes lies at R = 2πσ = 61 nm. Therefore the resolution values for Figure 2b–e are labeling density limited and the trade-off point is just crossed at the end of the data acquisition. The combination of current real-time single molecule fitting algorithms²¹ with continuous monitoring of resolution build-up provides a much needed stopping criterion for localization microscopy data acquisitions. The FIRE concept is also sensitive to differences in localization uncertainty (Fig. 2, g–i). Maximum likelihood estimation (MLE, FIRE = 58±1 nm) is theoretically optimal ²² and slightly better than least squares fitting (LS, FIRE = 60±1 nm) and superior to centroid fitting (CEN, FIRE = 88±2 nm). Since the effect of the parameter Q on the FIRE values for this dataset was found to be negligible, it was not necessary to correct for it.

Figure 2. The effect of density of localizations and data processing on resolution.

a, Localization microscopy image of tubulin labeled with Alexa 647 in a HeLa cell (FIRE = 58±1 nm for the whole image). The image is obtained from 1.4 · 10⁴ frames recorded in 4 min, the localization uncertainty is σ = 9.7 nm after merging nearby localizations in subsequent image frames (Supplementary Methods, sec. 3.3) and the density of localizations is ρ = 6.0 · 10²μm⁻². b–e, Zoomed insets of two crossing lines constructed from fewer time frames showing poorer resolutions (indicated by the distance between the blue arrows). f, Resolution build-up during the entire data acquisition with FIRE = 2πσ plotted in blue, showing a transition from density limited resolution to precision limited resolution close to the end of the data acquisition. At the end of data acquisition we reach ρσ² = 0.06 on the order of the rule-of-thumb value 0.14 for the transition point (see Eq.(4)). g–i, Reconstructions by different localization algorithms showing that MLE (g, FIRE = 58±1 nm) and LS (h, FIRE = 60±1 nm) methods outperform CEN (i, FIRE = 88±2 nm). j, Localization microscopy image of actin cytoskeleton (F-actin) of a fixed HeLa cell labeled with Phalloidin coupled to Alexa 647, after correction for sample drift of ~ 70–100 nm during acquisition. The image is obtained from 5.0 · 10⁴ frames recorded in 8 min, the localization uncertainty (after merging nearby localizations in subsequent image frames) is σ = 8.0 nm and the density of localizations is ρ = 8.2 · 10³μm⁻² (ρσ² = 0.52, 2πσ = 50 nm). k–n, Zoomed insets of reconstructions before (k, m) and after (l, n) drift correction show that more detail is visible after drift correction, which is reflected by a better FIRE value for the entire image after drift correction (54±1 nm compared to 79±2 nm before drift correction).

Sample drift is a common annoyance in optical nanoscopy as motion has to be limited to a few nanometers over typical acquisition times of many minutes. Figure 2j shows an image of the actin cytoskeleton of a fixed HeLa cell labeled with Phalloidin coupled to Alexa 647, which was analyzed for drift without the use of fiducial markers²³. A drift of ~70–100 nm was found with this procedure and corrected for. Computed FIRE values before drift correction (Fig. 2k, m, FIRE = 79±1 nm) are much worse than after drift correction (Fig.2l, n, FIRE = 54±1 nm), in agreement with the apparent detail in the images (arrows in Fig. 2k–m). For this dataset the effect of the parameter Q was also found to be negligible. More experimental data are provided to show the validity of FIRE (Supplementary Fig. 5–7).

Estimation of the number of localizations per emitter

Multiple localizations per emitter due to e.g. repeated photo-activations leads to spurious correlations between the two image halves resulting in overoptimistic FIRE values. This overestimation is particularly problematic for high numbers of localizations per emitter, low localization uncertainties, and low labeling densities (Supplementary Fig. 8–9). The FRC can be corrected for this effect by estimating the spurious correlation parameter Q in Eq. (2). To that end the numerator of the FRC is divided by the weighted average of the function exp (−4π²σ²q²) over the distribution of localization uncertainties. The spurious correlation parameter Q is proportional to the minimum of that curve, which takes the form of a broad plateau if Q ≫ 1 (Supplementary Methods, sec. 1). In order to test this method we have analyzed a two-color image of tubulin labeled with both Alexa 647 and Alexa 750²⁴. Fig. 3a–d show the resulting localization microscopy image (a, b) and the FRC curves without (c) and with (d) correction. The FIRE-values for Alexa 647 and Alexa 750 without correction (25±1 nm and 34±1 nm, respectively) are much lower than the FIRE-value derived from the cross-channel, i.e. when taking the two color images as data halves for the FRC (118±2 nm). This difference is due to spurious correlation effects arising from multiple localizations per emitter, which affect the one color FRC curves but not the cross-channel curve. The FRC curves are much more similar after correction and therefore also the attendant FIRE values (108±1 nm for Alexa 647, 133±2 nm for Alexa 750, 121±2 nm for the cross-channel). The remaining differences in the calculated resolution value for the two channels reflect the differences in labeling density (density of localizations 3.9·10³μm⁻² for Alexa 647, 1.2·10³μm⁻² for Alexa 750) and localization uncertainty (9.2 nm and 12 nm, respectively). The estimated Q-values (Q = 10 for Alexa 647, Q = 18 for Alexa 750) agree qualitatively with the values reported earlier²⁵. The values for the localization uncertainty and for the ratio of the Q-parameters are supported by an analysis of bright isolated clusters of localizations in the image. The values for Q are also consistent with the uncorrected FIRE values. It follows from Eq. (2) that FIRE without spurious correlation correction is equal to $2\pi \sigma /\sqrt{\text{log}(3Q)}$ in the regime where spurious correlations dominate the FRC curve (Supplementary Note 1). Using this formula and the found values for σ and Q we obtain 31 nm for Alexa 647 and 37 nm for Alexa 750, close to the uncorrected FIRE values of 25 nm and 34 nm respectively.

Figure 3. Spurious correlations from two-color localization microscopy image.

a, b, Overview image of a tubulin network (a) labeled with both Alexa 647 (red) and Alexa 750 (green), and inset (b) showing the quality of registration. c, The uncorrected FRC curves for the red and green channel are significantly higher than the cross-channel curve due to spurious correlations from repeated photo-activations of individual emitters, resulting in overly optimistic FIRE values for the red and green channels (25±1 nm and 34±1 nm, respectively, compared to 118±2 nm for the cross-channel). d, FRC curves corrected for spurious correlations are much closer to each other and give rise to similar FIRE values (108±1 nm for Alexa 647, 133±2 nm for Alexa 750, 121±2 nm for the cross-channel). e–g, Scaled FRC numerator curves showing a plateau for intermediate spatial frequencies, which is used to estimate the correction term and the parameter Q that determines the average number of localizations per binding site (Q = 10 for Alexa 647, Q = 18 for Alexa 750, Q = 0.3 for the cross-channel). For this correction (see Supplementary Methods, sec. 1) we used a mean and width of the distribution of localization uncertainties equal to 9.2 and 2.8 nm for Alexa 647, and 12 and 2.0 nm for Alexa 750.

We checked the datasets of Fig. 2a and Fig. 2j for spurious correlations and found Q = 0.28 and Q = 0.33, respectively, leading to corrected FIRE values equal of 62 ± 2 nm and 66 ± 1 nm respectively. This means, that neglecting to correct for spurious correlations gives rise to an underestimation of the FIRE-value of only several nanometers here. These estimated values for Q are smaller than the values for the dataset of Fig. 3 primarily because Q scales with the time of the data acquisition, which is much smaller here (1.4 · 10⁴ frames in 4 min for Fig. 2a and 5.0 · 10⁴ frames in 8 min. for Fig. 2j compared to 1.4 · 10⁵ frames in 39 min. and 3.0 · 10⁴ frames in 25 min. for Alexa 647 and Alexa 750 in Fig. 3). Other reasons may be found in differences in photobleaching behaviour (subpopulation that bleaches after first activation) and preprocessing for candidate selection of single emitter events (false positives). Finally, the labeling density for Fig. 2j is close to 10⁴μm⁻², one to two orders of magnitude more than the other datasets. In the limit of high labeling density the effects of spurious correlations are relatively insignificant compared to the intrinsic image correlations (Supplementary Note 1 and 2). We point out that the correction method appears to be quite sensitive to (the distribution of) the localization uncertainty, and to any residual effects of drift, and must therefore be exercised with care.

The estimation of the average number of localizations per emitter from the spurious correlation parameter Q also enables counting of the actual number of fluorescent labels that contribute to the overall image. Our method does not require a model for the correlations in the spatial distribution of the fluorescent labels, unlike methods based on pair correlation functions that have also been used for this purpose^{26, 27}, and no calibration experiment is needed, as opposed to the cluster kymography analysis²⁸. Deviations from Poisson statistics of the emitter activation due to photobleaching may lead to overestimation and in some cases underestimation (for very small Q) of the number of localizations per emitter (Supplementary Note 1). The same caveat applies to the correlation function based apprach^{26, 27}. Control experiments on sparsely distributed labelled antibodies on a glass surface suggest that photobleaching causes the Q-values for the data in Fig. 3 to overestimate the true number of localizations per emitter with a factor 1.5 for Alexa 647 and 1.7 for Alexa 750, even though the Q-parameter was estimated much more accurately (Supplementary Fig. 10, Supplementary Methods, sec. 3). The measured density of localizations, the estimated Q-parameter, and the photobleaching correction factor then finally lead to an estimated labeling density equal to 5.9·10²μm⁻² for Alexa 647 and 1.1·10²μm⁻² for Alexa 750.

FIRE in 3D, anisotropic and heterogeneous image content

The FIRE concept can be generalized and extended in a number of ways. The first way targets image anisotropy, which may arise from e.g. line-like features in the image or from differences between the axial and lateral resolving power of the microscope in 3D-imaging²⁹. Anisotropic image resolution can be described similar to FRC by correlating the two data halves in Fourier space over a line in 2D (Fourier Line Correlation, FLC) or plane in 3D (Fourier Plane Correlation, FPC) perpendicular to spatial frequency vectors q⃗. The set of spatial frequencies for which the FLC/FPC is above the threshold are resolved in the image. Fig. 4 shows the FPC for a 3D image of a tubulin network labeled with Alexa 647, using the bifocal method³⁰. The filaments are mostly in the xy-plane oriented along the x-direction (Fig. 4e). Therefore, the FPC is highest in the y-direction, orthogonal to the filaments, and worst in the z-direction, reflecting that the axial localization precision is inferior to the lateral one. Supplementary Figure 11 shows the FLC of (a part of) the dataset of Fig. 2j. The region of resolved spatial frequencies is anisotropic and highest in the direction orthogonal to the filaments, in agreement with expectations. Another way in which the FIRE concept may be generalized targets local variations in the density of the sample’s spatial structure. The local image resolution can be obtained by the FIRE value of overlapping sub-image patches (Supplementary Methods, sec. 3). Supplementary Figure 12 shows an example of such a resolution map for the dataset of Fig. 2a.

Figure 4. 3D-FIRE.

a, Orthogonal slices of the Fourier Plane Correlation (FPC) of a 3D localization microscopy image of a tubulin network. b–d, Cross-sections of the q_xq_z-plane, the q_yq_z-plane, and the q_xq_y-plane of the Fourier Plane Correlation, with added resolution threshold contours FPC = 1/7 (black lines). e, Representation of the 3D tubulin network, with the axial coordinate in false color. The FPC clearly shows the anisotropy of image content resulting from the line-like structure of the filaments (highest image resolution perpendicular to the filaments), as well as from the anisotropy in localization uncertainty (lowest resolution in the axial direction).

DISCUSSION

The FRC concept and FIRE are not only applicable to any form of localization microscopy but can be naturally extended to STED, imaging with an extended diffraction limit such as Structured Illumination Microscopy (SIM)³¹ and to conventional diffraction limited confocal and wide-field imaging. It is not only possible to conceptually extend the FIRE method, but this approach also allows the resolution to be measured directly from the experimental data. This stands in contrast to the recently introduced unified resolution concept of Mukamel et al. ³² which provides a rigorous theoretical framework but does not allow for practical measurement of the resolution. The FIRE number is most easily computed from two images of the same scene that only differ in noise content. For those cases the value depends on the signal-to-noise ratio, the spectral image content, and the (effective) optical transfer function. The width of the effective point spread function replaces the role of the localization uncertainty. In the limit of infinitely high signal-to-noise ratio, the FIRE measure reduces to Abbe’s diffraction limit (for the conventional fluorescence imaging modalities) or to the limit proposed by Hell (for STED³³) (Supplementary Note 4). Supplementary Figure 13 shows that the FIRE value matches well with Abbe’s limit in widefield acquisitions of fluorescent beads with a high signal-to-noise ratio. FIRE is also applicable to time lapse recordings in scanning electron microscopy (SEM) in single electron counting mode. Supplementary Figure 14 shows such a resolution build-up over time, as quantified by FIRE. For any extension of the FRC concept and FIRE, care must be taken to prevent dependencies (i.e. spurious correlations) between the images due to e.g. fixed pattern noise or common alignment references. Especially alignment references have been problematic in the past for the application of the FRC concept in the field of single particle electron microscopy³⁴.

In conclusion, we envision that FIRE may be used for characterizing and optimizing fluorescent labeling and data processing strategies in general. Next to the much needed stopping criterion detailed above, FIRE may be used to rate different approaches for enabling faster super-resolution image build-up that deal with high densities of simultaneously active emitters^35–37. Access to the number of molecules inside a multi-molecular complex, e.g. the spliceosome or transcription machinery, without making assumptions about their spatial structure adds a new dimension to the application of optical nanoscopy. Most importantly though, a resolution measure as proposed here is indispensable for advancing the blooming field of optical nanoscopy because it provides a quantitative guide for reliable interpretation of data, thus enabling sound biological conclusions.

METHODS

Methods and any associated references are available in the online version of the paper.