Abstract
Single-molecule localization microscopy techniques transcend the diffraction limit of visible light by localizing isolated emitters sampled stochastically. This time-lapse imaging necessitates long acquisition times, over which sample drift can become large relative to the localization precision. Here, we present an efficient and robust method for estimating drift, using a simple peak-finding algorithm based on mean shifts that is effective for single-molecule localization microscopy in two or three dimensions.
Why it matters
Drift correction is a regular part of the analysis pipeline in super-resolution fluorescence localization microscopy because the microscope stage typically moves farther than the localization precision over the time needed to acquire an image. This work presents a mathematically simple mean shift (MS) algorithm that allows for accurate drift correction with high time resolution from acquired localizations in two or three dimensions.
Introduction
Stochastic super-resolution microscopy techniques, such as STORM (1,2) and PALM (3,4), exploit photoswitching of fluorescent probes to enable imaging of densely labeled samples with resolutions an order of magnitude smaller than the diffraction limit of visible light. Sparsely distributed point spread functions (PSFs) of single emitters are identified in individual image frames, and their centroids are determined according to an appropriate fitting algorithm. The axial position of molecules can be encoded in their PSFs through engineering measures utilizing astigmatism (5,6), multifocal plane imaging (7), or a double helix PSF (8). The final reconstruction is typically a two-dimensional (2D) or three-dimensional (3D) histogram of these single-molecule positions.
Drift due to thermal expansion or mechanical instabilities can degrade image quality over the course of image acquisition, which typically occurs on the timescale of minutes. Drift compensation requires either active stabilization of the microscope (9, 10, 11, 12, 13) or a posteriori computation of the drift curves, either using fiducial markers (14, 15, 16, 17, 18) or the acquired single-molecule localizations (19, 20, 21, 22, 23, 24, 25, 26). In this report, we present a mathematically simple approach to drift correction using a mean shift (MS) algorithm (27, 28, 29) for static single-molecule localization microscopy (SMLM) data sets without fiducial markers, with some advantages over past approaches that use nonlinear least-squares (NLLS) fitting of image-based cross-correlations (19, 20, 21).
Results
A graphical illustration of the MS algorithm as applied to sample 2D localizations is presented in Fig. 1. The localizations all lie in one of two data sets which sample the same uniformly distributed emitters, but with a constant relative shift rshift in space. The first step of the algorithm is to extract pairwise displacements between all localizations across the two data sets. When individual displacements are plotted as points (Fig. 1 b), displacements arising from the same labeled objects (magenta points) cluster around rshift, whereas displacements arising from different objects (green points) distribute randomly over space. The MS algorithm determines the center of the peak of the distribution through iteration (27, 28, 29). At each iteration, all pairs within the radius of consideration are extracted, and the updated shift estimate is the centroid of these pairs. The uniformly distributed background will tend to bias the centroid toward the center of the observation window, whereas the peak moves the mean toward rshift. The observation window is then redrawn around the new mean and the process is repeated until the peak is centered in the observation window. Three iterations of the algorithm are visualized in Fig. 1 c.
Although the emitters of Fig. 1 are distributed uniformly in space, leading to the uniform distribution of the pairs from different emitters, the MS method does not depend on this assumption. In samples in which emitters are organized into structures or randomly clustered, the pairs arising from different emitters are also more likely to be at shorter distances, so that the distribution of green points in Fig. 1 b will also be peaked at rshift. However, in our experience, pairs of localizations from the same emitter are more important for the MS and other drift estimates. We also note that our analysis assumes that emitters that are localized in one data set remain within the field of view in the second data set, and vice versa. This may not always be the case, and could in principle lead to bias in shift estimates, but in practice this is typically a negligible effect. Roughly, the contribution of fluorophores near the edge of the field of view may be biased by up to about the localization precision, and the fraction of fluorophores that are affected is restricted to those that lie within about a localization precision from the edge of the field of view, in the direction of the drift. So, for example, in a 100 μm field of view with localization precision of 15 nm, we would expect this bias to be on the order of a picometer.
To benchmark this MS approach, we evaluated the ability of the algorithm to detect known shifts of simulated data sets of a circular test cell, as summarized in Fig. 2. Shifts were estimated by both the MS algorithm and by NLLS fitting of a Gaussian to the spatial cross-correlation function of the two data sets, as implemented in the supporting software provided with (21). The performance of each algorithm was similar for easy cases that produce a well-defined peak at rshift. An extremely easy case is depicted at the top of Fig. 2 a, in which single molecules are well spaced (surface density = 5/μm2) and their positions are well sampled in both frames (twice per molecule on average). In this case, the shift can be clearly identified by eye, and both algorithms reliably and accurately identify the displacement between frames. The simulation depicted at the bottom of Fig. 2 a represents a much harder case, in which molecules are present at higher surface density (20/μm2) and only 1 in 20 molecules are imaged on average in a given data set. In this case, MS modestly outperforms NLLS fitting, both by locating the peak with improved precision and by more reliably finding the peak overall. These trends hold over simulations conducted over a broad range of molecular densities and localizations per molecule (Fig. 2 b). We also estimated shifts from the overall center of mass of each data set, which yielded precisions more than an order of magnitude worse than both the MS and NLLS methods. Moreover, MS is more computationally efficient than NLLS, largely because fast Fourier transforms are not computed in the MS approach. This improvement in speed is enabled through the use of a particularly efficient algorithm from the R package spatstat (30) to extract pairwise displacements between nearby points (see Supporting methods and materials).
For large displacements, both the MS and NLLS algorithms applied in Fig. 2 require an initial step to identify an approximate starting point for the higher accuracy calculation. Fig. S1 shows the failure rate of each algorithm as a function of the distance of the start point from the true shift. MS robustly identifies the main peak over a broad range of simulation conditions as long as it resides within the initial observation window, so large shifts can be identified simply using a large window in the first iteration. This window is typically 100 nm for experimental localizations and 150 nm for the simulations of Fig. 2. NLLS robustly identifies the main peak when the starting point for the computation falls within the localization precision of the peak of the cross-correlation function. In many practical cases, the peak is much farther from the origin than the localization precision, so a separate method is needed to identify a suitable starting point. Here, this is accomplished using a particularly effective algorithm that identifies the global maximum in a smoothed cross-correlation function, as described in the supporting material of (21). The robustness of the NLLS fitting approach is dependent on the ability of this algorithm to identify a suitable starting point over a broad range of simulation conditions. Note that data sets for which the emitter distribution is highly structured or clustered typically lead to improved performance of the start point identification routine by introducing a broad peak in the cross-correlation function in addition to the sharp peak that represents repeat localizations of the same fluorophore.
This MS approach is applied to SMLM localizations that experience continuous drift by distributing localizations into nonoverlapping temporal bins with equal numbers of frames, and displacement estimates are tabulated between all possible pairs of bins. The number of frames in each temporal bin is an important parameter; short temporal bins have few localizations per molecule, so individual displacements may be estimated imprecisely. Long temporal bins have more localizations per molecule and more precise drift estimates but reduce the time resolution of the drift estimate. A linear least-squares fitting algorithm is then used to generate a trajectory that passes through control points positioned, at times, centered on each temporal bin, as described previously (21), taking advantage of the high redundancy to improve precision of the control points. We have slightly modified this past approach by including weights in the linear least-squares fitting, where weights are determined directly from data using a relation that approximates error in the mean displacement (described in Supporting methods and materials), as demonstrated in simulated data sets (Fig. 2 c). Briefly, errors are reduced when there are more pairs originating from the same molecules (magenta points in Fig. 1) and errors increase when more pairs originate from different molecules within the observation window (green points in Fig. 1). Estimated errors can also act as a proxy for overall reliability of the algorithm. Fig. 2 c also shows that MS reliably finds the desired peak when the estimated error remains smaller than one-quarter of the localization precision. This observation can act as a guide when selecting the number of frames included in temporal bins.
It is tempting to distribute frames into overlapping temporal bins, which in principle could improve time resolution while retaining a sufficient number of localizations to accurately determine displacements. However, we find that drift estimates from overlapping time bins are subject to substantial bias, underestimating the actual displacements accrued over time (Fig. S2 a). This occurs because the same localizations are present in adjacent bins, biasing the result toward rshift = 0. Similar bias can arise even in the absence of overlapping time bins because SMLM data frequently contain time-correlated localizations arising from the finite off rates of fluorescent blinking (PALM/dSTORM) or binding (PAINT). These factors mean that pairs of localizations from the same fluorophore are mostly from time separations that are shorter than the time difference between the bin centers and therefore underestimate the average drift between the bins.
The MS approach is applied to a 2D experimental data set of nuclear pore complexes (NPCs) in Fig. 3. NPC assemblies are labeled with primary and secondary antibodies against Nup210 within the nuclear envelope of intact primary mouse neurons, and Fig. 3, a–d shows reconstructed images at various magnifications. Fig. 3 b is a reconstruction produced without drift correction, in which localizations from single NPCs are smeared over a large area, highlighting the importance of drift correction.
The performance of the MS algorithm was tested on this data set by generating multiple drift trajectories through binning with different temporal resolutions. These trajectories were each applied to the full SMLM data set, and Fourier ring correlation (FRC) (31,32) was used to quantify image resolution (Fig. 3 e). For comparison, we conducted drift corrections using the redundant cross-correlation NLLS approach, as described previously (21). In this case, MS modestly outperforms NLLS fitting, allowing for accurate drift correction with smaller temporal bins and modestly improving the resolution of the reconstructed image. We used this data set to explore possible bias introduced because of temporal correlations of single-fluorophore blinking by running the linear least-squares algorithm and including or excluding adjacent pairs of bins on the data of Fig. 2. We found no significant difference between the two cases (Fig. S2 b), indicating that the impact of this bias is negligible within experimental errors. Additional diagnostics for the MS and NLLS approaches are shown in Fig. S3.
Drift trajectories are shown in Fig. 3 f for temporal bin widths that produce accurate FRC metrics for the MS and NLLS approaches. For MS, a temporal bin slightly larger than the minimum from the FRC curve is used because this produces smaller errors on individual control points. As expected, the drift trajectories follow the same general shape, but the trajectory generated from MS has improved time resolution. In parts of the trajectory, the errors of the control points are smaller than the distance between the trajectories. In these regions, higher time resolution yields improved spatial resolution in the final reconstructed image. Although the differences in the trajectories are significant, their impact is not apparent when viewing reconstructed images of entire nuclei or collections of NPCs, as in Fig. 3, a and c. Differences become more apparent in images of individual pores, in which displacements of several nanometers shift the relative positions of labeled subunits (Fig. 3 d).
The MS algorithm is easily extended to localizations acquired in 3D, in which performance improvements are more evident compared with the established NLLS approach. Because the MS algorithm uses points instead of reconstructed images and fast Fourier transforms, it can be extended into 3D without needing expanded memory resources that limit the practical application of NLLS in 3D. Instead, the 3D application of NLLS drift correction is typically accomplished by generating 2D projections that contain less information than the 3D localizations from which they are produced (21). To see why, consider a pair of emitters that are close together in x-y but far apart in z. Pairs of localizations from this pair of fluorophores will be included in 2D MS drift estimation when using data projected into the x-y plane but excluded from the full 3D drift estimation method by virtue of their large separation in z. We compare the precision and robustness of 3D MS and NLLS on simulated localizations spread over a cylindrical volume in Fig. S4, in which the NLLS correction is performed on projections into the x-y, x-z, and y-z planes, as described in (21). We also directly compare the x-y performance of the full 3D MS method with the 2D MS method performed on data projected into the x-y plane (Fig. S5).
Fig. 4 applies the MS approach to an experimental SMLM data set of labeled B cell receptors on the ventral membrane of B cells imaged using a phase mask in the emission path to localize fluorophores in 3D (8). As was the case for simulated data sets, the differences in the performance of the MS and NLLS fitting methods are more pronounced than in the 2D data set of Fig. 3. Additional diagnostics for the 3D case are shown in Fig. S6.
In summary, a mathematically simple MS algorithm modestly outperforms cross-correlation-based estimates of drift correction in 2D and more significantly improves the time resolution of drift corrections in 3D. The approach is computationally efficient, is robust without sophisticated methods to estimate starting points, and does not require image reconstruction with memory and pixelation limitations. The metric provided to estimate error and predict robustness directly from data provides users with a means to evaluate the quality of a drift correction within an SMLM analysis pipeline. For the example data sets explored, modest improvements in resolution lead to adjustments of localized molecule positions relevant for evaluating the structure of protein complexes in cells.
Author contributions
The method was devised by T.R.S. in consultation with F.J.F. and S.L.V. F.J.F. wrote the majority of the code and performed the majority of analyses in consultation with T.R.S. and S.L.V. F.J.F., T.R.S., and S.L.V. wrote the text. S.K. prepared and imaged nuclear pore complex samples for Fig. 3. R.A.B. prepared and imaged B cell samples for Fig. 4.
Declaration of interests
The authors declare no competing interests.
Acknowledgments
We thank Kathleen Wisser and Sarah Shelby for assistance with experiments.
This work was supported by grants from the U.S. National Science Foundation (MCB1552439) and National Institutes of Health (GM129347 and GM110052).
Editor: Ulrike Endesfelder.
Footnotes
Frank J. Fazekas and Thomas R. Shaw contributed equally to this work.
Supporting material can be found online at https://doi.org/10.1016/j.bpr.2021.100008.
Supporting citations
References (33, 34, 35, 36, 37, 38, 39) appear in the Supporting material.
Supporting material
References
- 1.Rust M.J., Bates M., Zhuang X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM) Nat. Methods. 2006;3:793–795. doi: 10.1038/nmeth929. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Heilemann M., van de Linde S., et al. Sauer M. Subdiffraction-resolution fluorescence imaging with conventional fluorescent probes. Angew. Chem. Int. Ed. Engl. 2008;47:6172–6176. doi: 10.1002/anie.200802376. [DOI] [PubMed] [Google Scholar]
- 3.Betzig E., Patterson G.H., et al. Hess H.F. Imaging intracellular fluorescent proteins at nanometer resolution. Science. 2006;313:1642–1645. doi: 10.1126/science.1127344. [DOI] [PubMed] [Google Scholar]
- 4.Hess S.T., Girirajan T.P.K., Mason M.D. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophys. J. 2006;91:4258–4272. doi: 10.1529/biophysj.106.091116. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Kao H.P., Verkman A.S. Tracking of single fluorescent particles in three dimensions: use of cylindrical optics to encode particle position. Biophys. J. 1994;67:1291–1300. doi: 10.1016/S0006-3495(94)80601-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Holtzer L., Meckel T., Schmidt T. Nanometric three-dimensional tracking of individual quantum dots in cells. Appl. Phys. Lett. 2007;90:053902. [Google Scholar]
- 7.Juette M.F., Gould T.J., et al. Bewersdorf J. Three-dimensional sub-100 nm resolution fluorescence microscopy of thick samples. Nat. Methods. 2008;5:527–529. doi: 10.1038/nmeth.1211. [DOI] [PubMed] [Google Scholar]
- 8.Pavani S.R.P., Thompson M.A., et al. Moerner W.E. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function. Proc. Natl. Acad. Sci. USA. 2009;106:2995–2999. doi: 10.1073/pnas.0900245106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Carter A.R., King G.M., et al. Perkins T.T. Stabilization of an optical microscope to 0.1 nm in three dimensions. Appl. Opt. 2007;46:421–427. doi: 10.1364/ao.46.000421. [DOI] [PubMed] [Google Scholar]
- 10.Grover G., Mohrman W., Piestun R. Real-time adaptive drift correction for super-resolution localization microscopy. Opt. Express. 2015;23:23887–23898. doi: 10.1364/OE.23.023887. [DOI] [PubMed] [Google Scholar]
- 11.Tafteh R., Abraham L., et al. Chou K.C. Real-time 3D stabilization of a super-resolution microscope using an electrically tunable lens. Opt. Express. 2016;24:22959–22970. doi: 10.1364/OE.24.022959. [DOI] [PubMed] [Google Scholar]
- 12.Schmidt P.D., Reichert B.H., et al. Sivasankar S. Method for high frequency tracking and sub-nm sample stabilization in single molecule fluorescence microscopy. Sci. Rep. 2018;8:13912. doi: 10.1038/s41598-018-32012-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Coelho S., Baek J., et al. Gaus K. 3D active stabilization for single-molecule imaging. Nat. Protoc. 2021;16:497–515. doi: 10.1038/s41596-020-00426-9. [DOI] [PubMed] [Google Scholar]
- 14.Huang B., Wang W., et al. Zhuang X. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science. 2008;319:810–813. doi: 10.1126/science.1153529. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Bon P., Bourg N., et al. Lévêque-Fort S. Three-dimensional nanometre localization of nanoparticles to enhance super-resolution microscopy. Nat. Commun. 2015;6:7764. doi: 10.1038/ncomms8764. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Colomb W., Czerski J., et al. Sarkar S.K. Estimation of microscope drift using fluorescent nanodiamonds as fiducial markers. J. Microsc. 2017;266:298–306. doi: 10.1111/jmi.12539. [DOI] [PubMed] [Google Scholar]
- 17.Ma H., Xu J., et al. Liu Y. A simple marker-assisted 3D nanometer drift correction method for superresolution microscopy. Biophys. J. 2017;112:2196–2208. doi: 10.1016/j.bpj.2017.04.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Balinovic A., Albrecht D., Endesfelder U. Spectrally red-shifted fluorescent fiducial markers for optimal drift correction in localization microscopy. J. Phys. D Appl. Phys. 2019;52:204002. [Google Scholar]
- 19.Mlodzianoski M.J., Schreiner J.M., et al. Bewersdorf J. Sample drift correction in 3D fluorescence photoactivation localization microscopy. Opt. Express. 2011;19:15009–15019. doi: 10.1364/OE.19.015009. [DOI] [PubMed] [Google Scholar]
- 20.Geisler C., Hotz T., et al. Egner A. Drift estimation for single marker switching based imaging schemes. Opt. Express. 2012;20:7274–7289. doi: 10.1364/OE.20.007274. [DOI] [PubMed] [Google Scholar]
- 21.Wang Y., Schnitzbauer J., et al. Huang B. Localization events-based sample drift correction for localization microscopy with redundant cross-correlation algorithm. Opt. Express. 2014;22:15982–15991. doi: 10.1364/OE.22.015982. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Elmokadem A., Yu J. Optimal drift correction for superresolution localization microscopy with Bayesian inference. Biophys. J. 2015;109:1772–1780. doi: 10.1016/j.bpj.2015.09.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Schlangen I., Franco J., et al. Rickman C. Marker-less stage drift correction in super-resolution microscopy using the single-cluster PHD filter. IEEE J. Sel. Top. Signal Process. 2016;10:193–202. [Google Scholar]
- 24.Han R., Wang L., et al. Zhang F. Drift correction for single-molecule imaging by molecular constraint field, a distance minimum metric. BMC Biophys. 2015;8:1. doi: 10.1186/s13628-014-0015-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Wester M.J., Schodt D.J., et al. Lidke K.A. Robust, fiducial-free drift correction for super-resolution imaging. bioRxiv. 2021 doi: 10.1101/2021.03.26.437196. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Cnossen J., Cui T.J., et al. Smith C. Drift correction in localization microscopy using entropy minimization. Opt. Express. 2021 doi: 10.1364/OE.426620. [DOI] [PubMed] [Google Scholar]
- 27.Fukunaga K., Hostetler L. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans. Inf. Theory. 1975;21:32–40. [Google Scholar]
- 28.Cheng Y. Mean shift, mode seeking, and clustering. IEEE Trans. Pattern Anal. Mach. Intell. 1995;17:790–799. [Google Scholar]
- 29.Comaniciu D., Meer P. Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 2002;24:603–619. [Google Scholar]
- 30.Baddeley A., Rubak E., Turner R. CRC Press, Taylor & Francis Group; Boca Raton; London; New York: 2016. Spatial Point Patterns: Methodology and Applications with R. [Google Scholar]
- 31.Nieuwenhuizen R.P.J., Lidke K.A., et al. Rieger B. Measuring image resolution in optical nanoscopy. Nat. Methods. 2013;10:557–562. doi: 10.1038/nmeth.2448. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Banterle N., Bui K.H., et al. Beck M. Fourier ring correlation as a resolution criterion for super-resolution microscopy. J. Struct. Biol. 2013;183:363–367. doi: 10.1016/j.jsb.2013.05.004. [DOI] [PubMed] [Google Scholar]
- 33.Hilgenberg L.G.W., Smith M.A. Preparation of dissociated mouse cortical neuron cultures. J. Vis. Exp. 2007;2007:562. doi: 10.3791/562. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Pappas S.S., Liang C.-C., et al. Dauer W.T. TorsinA dysfunction causes persistent neuronal nuclear pore defects. Hum. Mol. Genet. 2018;27:407–420. doi: 10.1093/hmg/ddx405. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Haughton G., Arnold L.W., et al. Mercolino T.J. The CH series of murine B cell lymphomas: neoplastic analogues of Ly-1+ normal B cells. Immunol. Rev. 1986;93:35–51. doi: 10.1111/j.1600-065x.1986.tb01501.x. [DOI] [PubMed] [Google Scholar]
- 36.Stone M.B., Shelby S.A., et al. Veatch S.L. Protein sorting by lipid phase-like domains supports emergent signaling function in B lymphocyte plasma membranes. eLife. 2017;6:e19891. doi: 10.7554/eLife.19891. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Izeddin I., Boulanger J., et al. Sibarita J.B. Wavelet analysis for single molecule localization microscopy. Opt. Express. 2012;20:2081–2095. doi: 10.1364/OE.20.002081. [DOI] [PubMed] [Google Scholar]
- 38.Smith C.S., Joseph N., et al. Lidke K.A. Fast, single-molecule localization that achieves theoretically minimum uncertainty. Nat. Methods. 2010;7:373–375. doi: 10.1038/nmeth.1449. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Li Y., Mund M., et al. Ries J. Real-time 3D single-molecule localization using experimental point spread functions. Nat. Methods. 2018;15:367–369. doi: 10.1038/nmeth.4661. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.