Optimized molecule detection in localization microscopy with selected false positive probability

Abstract

Single-molecule localization microscopy (SMLM) allows imaging beyond the diffraction limit. Detection of molecules is a crucial initial step in SMLM. False positive detections, which are not quantitatively controlled in current methods, are a source of artifacts that affect the entire SMLM analysis pipeline. Furthermore, current methods lack standardization, which hinders reproducibility. Here, we present an optimized molecule detection method which combines probabilistic thresholding with theoretically optimal filtering. The probabilistic thresholding enables control over false positive detections while optimal filtering minimizes false negatives. A theoretically optimal Poisson matched filter is used as a performance benchmark to evaluate existing filtering methods. Overall, our approach allows the detection of molecules in a robust, single-parameter and user-unbiased manner. This will minimize artifacts and enable data reproducibility in SMLM.

The work presents a molecule detection method for single-molecule localization microscopy. By means of probabilistic thresholding with control over false positive detections, the method improves robustness and enhances reproducibility.

Introduction

Single-molecule localization microscopy (SMLM) encompasses powerful techniques that surpass the diffraction limit of light^1,2. These advancements have lead to breakthroughs in fields such as cellular or molecular biology. The process of molecule detection in microscopy images is integral to SMLM methodologies. It is an essential first step for extracting quantitative information from microscopy images and forms a basis for all subsequent analysis³. The detection algorithm aims to identify individual molecules and determine their pixel-based locations, effectively separating molecular signals from background noise. However, the current state of molecule detection in SMLM faces several challenges. The detection process involves numerous methods, each with multiple settings that users manually adjust. This complexity makes the process difficult to execute, reproduce, and interpret without errors or artifacts⁴. Furthermore, the selection of the detection method in localization software is often ad hoc lacking a systematic approach. The detection methods implemented across various localization software differs^5,6. This restricts users to potentially sub-optimal performance and limits the reproducibility of data analysis.

The variety and complexity of current detection methods are substantial. Briefly, these methods include linear filters such as mean filters, Gaussian filters^5,7, and more complex compositions like Laplacian of Gaussian⁸ and difference of Gaussians. Nonlinear filters, such as median filters, and advanced noise suppression techniques like bilateral filtering and nonlocal means, are also employed⁹. Morphological operations, including top-hat filters and h-dome transforms, play a role in some approaches¹⁰. Thresholding techniques range from simple intensity-based thresholds to more sophisticated methods like Otsu’s threshold^5,6,11. Additionally, machine learning approaches, including traditional segmentation methods and emerging deep learning techniques, are becoming increasingly relevant in this field^7,10,12–15. For details see our overview of single-molecule detection methods in Supplementary Note 1.

A significant issue in the current framework for molecule detection is the lack of quantitative control over detection performance, particularly in terms of false positive and false negative detections. Users typically set parameters based on experience and make subjective judgments about detection accuracy, leading to inconsistent results. In SMLM, the negative impact of false positive detections is particularly problematic. False positives can form non-existent artificial structures in the super-resolved image and propagate errors throughout the SMLM processing pipeline, affecting post-processing and quantitative analysis. The variability in detection errors is evident in the outcomes of the SMLM challenge, where different algorithms produced significantly different false positive detections over the same dataset (Supplementary Fig. 9). This variability underscores the need for a more standardized and controlled approach to molecule detection in SMLM. An uncontrolled level of false positive detections acts as noisy data and negatively affects the entire subsequent processing chain (from post-processing steps like drift correction and merging, to quantitative analysis such as clustering, molecule count, quality assessment, particle fusion, and tracking) in unpredictable ways.

To address these challenges, we have developed a user-unbiased detection algorithm that is theoretically optimal (with the highest detection probability), simplifies the process for the user, and quantifies its performance. It is a combination of probabilistic thresholding and optimal filtering. Our approach frames molecule detection within the context of signal detection theory¹⁶, and derives a detection method with the highest theoretical performance that enables control over the level of false positive detections. We combine it with a theoretically optimal Poisson Matched Filter (PMF) using it as performance benchmark to systematically evaluate alternative methods. The benchmark revealed Matched Filter (MF) to be the method of choice. This enables users to ensure optimal and reproducible molecule detection simply by setting a desired false positive probability. Notably, the presented method is readily generalizable to three-dimensional imaging. This approach is well established in astronomy¹⁷ and radar¹⁸, and we now bring its benefits to SMLM. We present an implementation of the method that is fast, robust, and uses standard Python libraries (implementation details are given in Supplementary Note 2).

Results

Algorithm description

Our algorithm is comprised of two main steps: filtering and probabilistic thresholding. In this section, we introduce the Poisson-matched filter to set a performance benchmark for other filtering methods. We then show how to apply the probabilistic thresholding for molecule detection, allowing the user to control molecule detection by selecting the probability of false positive detection. Last, we add background adaptation to our method, where the threshold level is calculated from the local background level. Based on the detection performance and computational efficiency, we propose optimized molecule detection that consists of a matched filter and probabilistic thresholding visualized in Fig. 1.

Fig. 1. Schematics of the detection.

Proposed molecule detection algorithm using matched filter and adaptive probabilistic thresholding where the user sets the false positive probability P_FP to adjust the detection threshold. Scale bar is 1 μm.

Optimal filtering

We start by deriving the Poisson Matched Filter (PMF), a theoretically optimal detection filter (with the highest detection probability) for the Poisson noise model, which is well-suited for shot noise-limited conditions prevalent in microscopy. PMF has already been proposed in X-ray astronomy¹⁹ and optical communication systems²⁰ but has not appeared in the context of fluorescence microscopy. The derivation uses techniques from the signal detection theory^16,21, particularly the Generalized Neyman-Pearson’s Likelihood Ratio Test (GLRT). PMF offers theoretically optimal performance, which means achieving the maximum probability of detection P_D while keeping the probability of false positive P_FP equal to a selected level. By solving the GLRT for a Point Spread Function (PSF) generation model in a Poisson noise (see “Methods” for details), we receive weights representing the PMF filter kernel

$$w_{mn}\left(a,b,\sigma \right)=ln\left(1+\frac{a}{b}{s}_{mn}\left(\sigma \right)\right).$$ 1

The shape of the PMF filter depends on the PSF shape s_mn(σ), its width σ, and the molecule contrast, defined here as the signal-to-background ratio a/b. We used PMF as a benchmark to systematically assess the performance of other filtering methods used in SMLM. Interestingly, the PMF’s shape closely approximates that of other filters across different signal-to-background ratios: matched filter for low contrast a/b, and B-spline wavelets for medium to high contrast a/b, depending on B-spline parameters (see Fig. 2), which places these filters in a common theoretical framework. The width of the optimal PMF filter increases with increasing ratio a/b. This is in agreement with the conclusions of the related work²², which recommends (based on numerical simulations) to use a Gaussian filter with a larger standard deviation than the one of the PSF.

Fig. 2. Comparison of the shape of the Poisson Matched Filter (PMF) with other relevant filters.

a PMF for different signal-to-background ratios a/b shows that the shape is practically identical to that of a Matched Filter (MF) for a/b = 0.1. b Comparison of B-Spline wavelets (BS) and PMF showing clear similarity for mid and high a/b. Markers indicate the values used to create the filtering mask.

Because molecules with medium to high contrast are generally detected with high accuracy regardless of the applied filter, it is central to optimize detection performance for molecules with low contrast. Therefore, the matched filter emerges as the preferable option not only because of its optimal performance but also because of its simplicity. Unlike PMF, which requires three parameter estimates (a, b, σ), the matched filter requires only a single parameter (σ). We explain the relation of the PMF and matched filter for low contrast molecules in Supplementary Note 3.

In specific cases, the Poisson noise model might not be a suitable approximation for microscopy data, for example when the shot noise is combined with significant read-out noise as in the CCD camera. Then, the model needs to be extended to the Poisson–Gaussian noise²³. We performed GLRT in Poisson-Gaussian noise showing that it does not lead to a compact formula (Supplementary Note 4). However, if we prioritize the detection of the weak signal with a low signal-to-background ratio a/b, the GLRT is equivalent to the matched filter. It demonstrates that the matched filter remains a preferable solution even under the mixed Poisson-Gaussian noise conditions. Another case generalizing detection in the Poisson noise model is the case of the sCMOS camera, where each pixel effectively has a known, different variance of the noise. In Supplementary Note 5, we show that the optimal detection filter is an inversely weighted matched filter, sometimes also referred to as a whitened matched filter.

Probabilistic thresholding

In the second step, we implement a probabilistic detection threshold to enable quantitative control over false positive detection errors. This method is well-established in the fields of radar¹⁸, astronomy⁹, and medicine²⁴ and has been previously recommended for microscopy and particle tracking in refs. ^25–27. In our methodology, detection performance is optimized by aligning the algorithm’s sensitivity with a user-defined false positive probability, effectively calibrating the detection threshold. In practice, this means the user only needs to input the desired false positive probability, which serves as not only the sole parameter for setting the detection threshold but also a detection error estimate.

To use the probability of false positive detection as a parameter that sets the detection threshold, we take advantage of the knowledge of the analytical formula of P_FP, where P_FP depends on the mean background level and on the threshold τ. As the background level can be easily estimated from the images, the only parameter controlling the probability P_FP is the decision threshold τ. By inverting the equation, we receive a formula for threshold as a function of P_FP

$$\tau =\sqrt{\widehat{b}\sum _{m,n=-L}^L{w}_{mn}^2}{Q}^{-1}\left(P_{\mathrm{FP}}\right)+\widehat{b}\sum _{m,n=-L}^L{w}_{mn},$$ 2

where $\widehat{b}$ denotes the mean background estimate, w_mn is a filter kernel of the size (2L + 1) × (2L + 1) and Q⁻¹ is the inverse complementary cumulative distribution function of the standard normal distribution. We refer to threshold τ as the probabilistic threshold. Probabilistic thresholding can be used in combination with any linear filter common in SMLM. Probabilistic thresholding for a special case of low input signal in EMCCD camera is discussed in Supplementary Note 6.

Adaptability of thresholding to local background

In typical SMLM images, the background may vary across the field of view both spatially and temporally, changing from frame to frame. The estimation of the background is necessary to set the probabilistic threshold. Any existing background estimation methods can be used for this purpose. From local arithmetic averaging in space, median filtering over time²⁸, to more complex methods utilizing precise knowledge of the background structure²⁹, or estimation based on powerful deep neural networks³⁰. It is reasonable to estimate the background from values in both space and time if they can be considered as locally constant in this neighborhood generally corresponding to 3D space-time averaging. To complement existing methods, we propose an averaging method implemented using filtering (Fig. 3). Background estimation is performed from local values, but the nearest neighborhood (approximately the size of PSF) containing pixels biased by the presence of the signal is omitted. The resulting filter kernel is shaped like a 2D or 3D doughnut (see Fig. 3d). This method is widely used in radar¹⁸ (more details in Supplementary Note 7).

Fig. 3. Molecule detection in images with varying background using adaptive thresholding.

a An example of a simulated image of a varied background with 30 molecules placed across the area. Scale bar is 2 μm. b Detection artifacts displaying an incorrect number of false positive detections when the threshold is not adaptive to varying background and c A solution to this problem by background adaptive thresholding. P_FP is set to 10⁻⁴ in both images. d Kernels displaying reference pixels (yellow) used for local background estimate and guard pixels (blue) preventing errors due signal present.

We found that the thresholding using adaptive background estimation surpasses considered methods that are robust to background variability or methods that eliminate the background (see Fig. 4). Our results indicate that the widely used practice of background removal might not be theoretically justified and might become unnecessary when using background-adaptive thresholding, pointing to an interesting problem for further studies.

Fig. 4. Performance of multiple detection method on simulated data.

Performance evaluation by Receiver Operating Characteristic (ROC) curves showing the probability of detection P_D as a function of false positive probability P_FP for various methods across three different scenarios a Uniform background (blue): The Poisson Matched Filter (PMF) and Matched Filter (MF) consistently achieve the highest P_D for any given P_FP, outperforming other methods. b Varying background (red): The performance difference becomes more pronounced under varying background conditions. The PMF maintains its superior detection capability. c SMLM challenge dataset⁵ (green). The SMLM challenge dataset provides a realistic test scenario, where the PMF and MF continue to outperform other methods.

Generalization for molecule detection in 3D

The presented methodology of molecule detection is general and allows application in 3D. The specific form of implementation depends on the assumed model of signal generation. We distinguish three basic models to outline the detection algorithms. The first one (called here multiplane) is for the case of the full z-stack in all slices of potential molecule z-axis position. An example can be the multifocus microscopy method³¹. The second one (which we call z-parameterized PSF) uses PSF engineering methods that encode the value of the z-axis into a PSF shape. Representatives are for example the astigmatism, double helix, and tetrapod methods. The last considered method is called z-projection detection, where we do not distinguish molecules in the z-axis and the detection is performed on data where the z-component of the data has been marginalized (averaged out). An example is the biplane method³². For the overview of the 3D SMLM methods, we rely on references^33,34.

Detection in multiplane

Similarly to the 2D case, we seek the maximum of the test statistic (i.e., the filtered signal) over all possible integer shifts of the molecule in all dimensions. In 3D, we search in three dimensions in the received signal, which is no longer a 2D image but a 3D datacube. Using the Poisson noise model and the GLRT test we obtain the optimal Poisson-matched filter (see Supplementary Note 14 for details).

$$w_{lmn}=ln\left(1+\frac{a}{b}{s}_{lmn}\right),$$ 3

where s_lmn denotes the 3D PSF. Any analytical model or measured function can be used as s_lmn.

Detection with z-parameterized PSF

For detection with parameterized PSF, we only have 2D measurements in one axial axis in which we search for the corresponding slice of the parameterized PSF. For this task, we can also use a GLRT detection test where one of the search parameters is the z-coordinate of the molecule, where w_mn(l) is the l-th slice of the parametrized detection filter. For the Poisson noise model, the detection filter is

$$w_{mn}\left(l\right)=ln\left(1+\frac{a}{b}{s}_{mn}\left(l\right)\right).$$ 4

Symbol s_mn(l) denotes the l-th slice of parametrized PSF model.

Detection in z-projection

In the z-projection detector, the detection is performed over 3D data where the z-axis component has been averaged out to obtain a 2D model of the signal that is equivalent to the model described in the manuscript, except that now instead of using a 2D PSF function, we use a z-axis averaged 3D PSF

$${\tilde{s}}_{mn}=\frac{1}{N_z}\sum _l{s}_{lmn}.$$ 5

As a proof of concept for the application of detection theory in 3D SMLM, we show an implementation of the 3D detection for the case of multiplane imaging, including a basic performance evaluation (Supplementary Note 8). A comparison of all possible detection filters for all forms of 3D detection is beyond the scope of this work. Here, we demonstrate the general functionality of the method.

Evaluation of performance and comparison to other methods

We evaluated our detection algorithm by comparing its performance against several commonly used detection methods in SMLM that differ only in the type of detection filter and its parameters. Table 1 lists these filters alongside the SMLM software in which they are implemented. Definitions of these methods are provided later in “Reference methods”.

Table 1.

List of detection filters and availability in SMLM software

Filters	Acronyms	References	Software
A trous B-spline wavelet	BS	9,52	ThunderStorm61, SMAP55,
Difference of Arithmetic means	DoA	53,54	ThunderStorm61
Difference of Gaussians	DoG	55–57	ThunderStorm61, SMAP55, QuickPALM56, TrackMate58
Laplacian of Gaussian	LoG	8,58	TrackMate58
Lowered Gaussian	LG	59	DAOSTORM59
Matched Filter	MF	22,25,26	SimpleSTORM26
Net-gradient search	Net-grad	60	Picasso60

To assess the performance, we have selected Receiver Operating Characteristic (ROC) curves³⁵ as a primary performance metric. ROC curves illustrate the relationship between the detection probability P_D on the probability of false positive detection P_FP. The higher the P_D for given P_FP, the better the algorithm’s performance. While related works often use F1/Dice measures or Jaccard index^5,6,22, we chose ROC evaluation due to the asymmetric impact of false positives and false negatives in molecule detection, where false positives have more severe consequences. This approach is consistent with recent community recommendations on metric selection in image processing tasks³⁶.

To quantitatively evaluate the performance of detection methods, it is necessary to use datasets with annotated ground truth detections^5,6. We evaluated detection methods across five different datasets: simulated data with uniform background, simulated data with varying background, two datasets created for the SMLM challenges, and experimental data of static gold beads.

First, we tested the methods using data with constant background, each simulated as a single randomly placed fluorescent molecule on a uniform background with Poisson noise. An example of a generated image is shown in Supplementary Fig. 10. The highest detection probability P_D for any probability of false positive P_FP was given by PMF, supporting the theory. The matched filter performed similarly well. The net-gradient method performed the worst, followed by Difference of Gaussians (DoG) and Difference of Arithmetic means (DoA) filters as depicted in Fig. 4a). For illustration, we also show the performance of the methods on this type of data using Jaccard metric in Supplementary Fig. 11.

Second, we have compared the performance on data with varying background and Poisson Noise. We modeled a parabolic type of background that corresponds to imaging conditions of a non-uniform illumination similar to the effect of vignetting³⁷ as illustrated in Fig. 3. Results are found in Fig. 4b), showing PMF and matched filter as the best performers.

As the next step, we took advantage of the datasets created for the SMLM challenge^5,6, and compared the performance of various filters on the noisy Bundled Tubes Long Sequence (BT-LS) dataset (Fig. 4c) and microtubules, low SNR, low density (MT0_N2_LD) dataset (Supplementary Fig. 12). Results support the theoretical predictions and confirm PMF and MF as the highest performers.

To demonstrate the practical use of the detection algorithm on experimental data, we additionally tested the performance on images of DNA origami-bound gold beads. They are easy to detect under standard visualization conditions due to their high brightness, effectively serving as a reference to annotate ground truth detections. To ensure the homogenous distribution of nanoparticles and reduce agglomeration during the deposition, we used gold nanoparticles that were attached to DNA origami with biotinylated sites and deposited them on BSA-biotin-treated glass coverslips by using neutravidin as a linker. We visualized the samples in two modes. One under standard visualization conditions that result in bright diffraction-limited spots on a low background that served to annotate the ground truth reference. In the second mode, we changed the imaging conditions to make the beads appear less distinct, mimicking how low-contrast molecules might appear in other experiments and used it to test the performance of the detection methods (Fig. 5a). Resulting ROC curves show that PMF and matched filter outperforms others on experimental data (Fig. 5b) as well.

Fig. 5. Performance evaluation of multiple detection filtering methods over experimental dataset.

a Examples of detected gold nanoparticles in images with extra added light source compared to the reference image without interference. Scale bar is 500 nm. b Receiver operating characteristics showing the probability of detections P_D of selected methods as a function of false positive probability P_FP on the experimental dataset. Results show the top performance of the PMF and MF filters.

Computational efficiency

We have designed a computationally efficient implementation of the PMF and probabilistic thresholding. The main demand for computational resources arises from the estimation of unknown parameters in GLRT. We present a series of simplifications and approximations that lead to a computationally efficient implementation in the form of linear filtering. The primary simplification lies in estimating the molecule’s unknown position (m₀, n₀), where we restrict the potential values to integers, resulting in a negligible performance degradation (see Supplementary Fig. 15), but allowing much faster calculations. Additionally, we simplify the estimation of the filtering parameters (a, b, and σ) for the PMF. Fixed values for parameters a, b can be pre-determined through molecule localization over a small set of frames within the SMLM software, without requiring user interaction. This step becomes unnecessary when employing matched filter, as it is independent of a and b. We assume that the last parameter, σ, is known based on the measurement setup.

In terms of computational complexity, our analysis (Supplementary Note 10) shows that the PMF operates as a linear filter, with an asymptotic complexity no worse than that of other common filtering methods. The adaptive background estimation for probabilistic thresholding has similar or not worse complexity as the linear filtering methods, making it computationally efficient for real-time applications. The computational complexity analyses are complemented by examples of specific run times on our regular desktop PC. We present a straightforward implementation of proposed methods using standard Python libraries (details in Supplementary Note 2).

When comparing the numerical complexity in the detection part of the SMLM algorithm to the Deep Learning (DL) methods, the DL segmentation is more numerically challenging than the linear filtering and thresholding presented in our method. Typical U-net-based³⁸ DL segmentation algorithms contain an image encoder that consists of a parallel bank of convolutional filters, i.e., consisting of many linear filtering operations. The computational complexity of current-state-of-the-art DL segmentation models is far exceeding linear filtering. For example, the number of parameters in the segment anything model³⁹ alone is over 600 million. DL models are typically less demanding and faster than the analytical solution if it is a full model that represents the entire chain of SMLM processing operations including the fitting of sub-pixel positions, the most computationally demanding part of the algorithm^40,41. See Supplementary Note 11 for further comparison of our method and deep learning.

In summary, we show that matched filter paired with background-adaptive probabilistic thresholding is the optimal choice for analyzing SMLM data. This resolves issues related to parameter configurations by users and reproducibility of results while allowing to address the impact of false positive detections on subsequent qualitative localization analysis, for example clustering analysis, drift correction, localization dynamics and others. We advocate for adopting matched filtering and adaptive probabilistic thresholding as standard features in SMLM software. Beyond SMLM, our methodology holds potential benefits for other imaging techniques suffering from low SNR.

Methods

Neyman–Pearson’s Likelihood Ratio Test (LRT)

The Neyman-Pearson theorem states that the likelihood ratio test

$$\mathrm{\Lambda }\left(\mathbf{R}\right)=\frac{p\left(\mathbf{R}\mid H_1\right)}{p\left(\mathbf{R}\mid H_0\right)}\stackrel{H_1}{>}\tau$$ 6

offers a theoretically optimal molecule detection performance. The word optimal in this context means achieving the maximum probability of detection P_D while keeping the probability of false positive P_FP equal to a selected level¹⁶. For a list of synonymous terminology, we refer to Supplementary Table 5. The property of maintaining a constant probability of false positive P_FP is often referred to as constant false alarm rate detection, especially in the context of radar signal processing¹⁸.

The symbol of inequality in test (6) means that we reject the null hypothesis H₀ (signal absent) and accept the alternative hypothesis H₁ (signal present) if the inequality is fulfilled. For the opposite inequality, we do not reject the signal-not-present hypothesis H₀. The symbol τ denotes the detection threshold, which is determined by the chosen false positive detection probability P_FP, as presented in “Probabilistic Thresholding and Probability of Detection”.

The derivation of a PMF filter can be shown in an arbitrary number of dimensions, but for simplicity we consider a 2D data space where symbol R is a raw image matrix $\mathbf{R}\in {\mathbb{N}}^{M\times N}$ of the size M × N. Symbol p(R∣H₁) denotes the conditional probability density function (denoted as likelihood function) of obtaining R given the hypothesis signal present. Similarly, likelihood p(R∣H₀) is conditioned by the hypothesis signal absent.

Statistical hypotheses of image formation model

Suppose that we have a PSF signal model in Poisson noise-limited conditions. This model corresponds to the scenario where all other types of noise are negligible, and what remains is the irremovable photon noise arising from the particle nature of the light⁴². At the same time, we assume precise conversion from analog-digital units to photon counts, using a well-calibrated camera gain and baseline offset^26,43. Then, the hypothesis of signal absent H₀ and present H₁ has the following statistical description

$$\begin{array}{c}{H}_0:\mathbf{R}\in {\mathbb{N}}^{M\times N},{\left[\mathbf{R}\right]}_{mn}=r_{mn}~\mathcal{P}\left(b\right),\hfill \\ H_1:\mathbf{R},{\left[\mathbf{R}\right]}_{mn}=r_{mn}~\mathcal{P}\left(a{s}_{m-m_0,n-n_0}\left(\sigma \right)+b\right),\hfill \end{array}$$ 7

where letter $\mathcal{P}\left(b\right)$ denotes the Poisson distribution with the mean b. Symbol  ~ means “distributed as”. The mnth pixel of raw image [R]_mn is denoted by the lowercase symbol r_mn. Other symbols indicate the number of emitted photons a, the mean background level b, and the sampled PSF s_mn(σ). Any PSF model is applicable here, such as the 2D Gaussian model

$$s_{mn}\left(\sigma \right)=\frac{1}{2\pi {\sigma }^2}{\mathrm{e}}^{-\frac{1}{2{\sigma }^2}\left(m^2+n^2\right)},m\in \mathbb{Z},n\in \mathbb{Z},$$ 8

the integrated Gaussian model or others including the measured one. Symbol σ denotes the standard deviation of PSF. Unknown coordinates of the molecule m₀, n₀ are assumed to fit the size of the raw image with a tolerance of K pixels given by the expected size of the PSF.

Parametric likelihood functions

The likelihood functions for both hypotheses (7) are obtained as the joint distribution of independent identically distributed Poisson random variables as

$$p\left(\mathbf{R}\mid H_0,{\mathit{\theta }}_0\right)=\prod _{mn}\frac{b^{r_{mn}}}{r_{mn}!}{\mathrm{e}}^{-b},$$ 9

$$p\left(\mathbf{R}\mid H_1,{\mathit{\theta }}_1\right)={\prod }_{mn}\frac{{\left(a{s}_{m-m_0,n-n_0}\left(\sigma \right)+b\right)}^{r_{mn}}}{r_{mn}!}{\mathrm{e}}^{-\left(a{s}_{m-m_0,n-n_0}\left(\sigma \right)+b\right)},$$ 10

where we have shortened the notation by symbol ${\prod }_{mn}\triangleq {\prod }_{m=0}^{M-1}{\prod }_{n=0}^{N-1}.$ Symbol  ≜ means the equality from the definition. We see that the likelihood functions (9), (10) cannot be evaluated because they depend on the unknown parameters denoted by vectors ${\mathit{\theta }}_0=b,{\mathit{\theta }}_1={\left[a,b,\sigma ,m_0,n_0\right]}^T,$ where ${\left(\star \right)}^T$ is the matrix transposition. Therefore, the likelihoods cannot be directly used in the LRT (6).

Generalized Likelihood Ratio Test (GLRT)

GLRT accepts hypothesis H₁, if the likelihood ratio is greater than the threshold τ

$${\mathrm{\Lambda }}_G\left(\mathbf{R}\right)=\frac{p\left(\mathbf{R}\mid H_1,{\widehat{\mathit{\theta }}}_1\right)}{p\left(\mathbf{R}\mid H_0,{\widehat{\mathit{\theta }}}_0\right)}\stackrel{H_1}{>}\tau$$ 11

where the unknown parameters in θ₀ and θ₁ have been estimated by the maximum likelihood estimation

$${\widehat{\mathit{\theta }}}_i=\underset{{\mathit{\theta }}_i}{\mathrm{argmax}}p\left(\mathbf{R}\mid H_1,{\mathit{\theta }}_i\right),i\in \left\{1,2\right\},$$ 12

where the estimated values are labeled by a hat symbol $\widehat{\star }$. The GLRT is adjusted for the likelihoods (9), (10) to the inequality

$${\mathrm{\Lambda }}_G\left(\mathbf{R}\right)=\prod _{mn}{\left(1+\frac{\widehat{a}}{\widehat{b}}{s}_{m-{\widehat{m}}_0,n-{\widehat{n}}_0}\left(\widehat{\sigma }\right)\right)}^{r_{mn}}{\mathrm{e}}^{-\widehat{a}{s}_{m-{\widehat{m}}_0,n-{\widehat{n}}_0}\left(\widehat{\sigma }\right)}\stackrel{H_1}{>}\tau .$$ 13

The inequality in (13) does not change if we take the logarithm of both sides, leading to

$$ln{\mathrm{\Lambda }}_G\left(\mathbf{R}\right)=\sum _{mn}{r}_{mn}ln\left(1+\frac{\widehat{a}}{\widehat{b}}{s}_{m-{\widehat{m}}_0,n-{\widehat{n}}_0}\left(\widehat{\sigma }\right)\right)-\widehat{a}\stackrel{H_1}{>}ln\tau ,$$ 14

where we assume that PSF is normalized such that ${\sum }_{m,n}{s}_{m-{\widehat{m}}_0,n-{\widehat{n}}_0}\left(\widehat{\sigma }\right)=1.$ Here, symbol ∑_mn means ${\sum }_{mn}\triangleq {\sum }_{m=0}^{M-1}{\sum }_{n=0}^{N-1}$.

Poisson Matched Filter (PMF)

Once the parameters in (14) are estimated, we can further adjust the detection inequality to the form

$$T\left(\mathbf{R}\right)=\sum _{mn}{r}_{mn}{w}_{m-{\widehat{m}}_0,n-{\widehat{n}}_0}\left(\widehat{a},\widehat{b},\widehat{\sigma }\right)\stackrel{H_1}{>}{\tau }^{\prime },$$ 15

where T(R) is called a test statistic. Symbol ${\tau }^{\prime }=ln\tau +\widehat{a}$ is a new decision threshold incorporating estimate $\widehat{a},$ the value of which is determined by the desired probability of false positive P_FP. Left-hand side of the inequality (15) is a weighted arithmetic mean with the weights given by

$$w_{mn}\left(a,b,\sigma \right)=ln\left(1+\frac{a}{b}{s}_{mn}\left(\sigma \right)\right).$$ 16

We call the weights in (16) the filter kernel of the PMF. To simplify the notation, we omit the functional dependency on the parameters and simply write w_mn. We assume that the PMF kernel takes significant non-zero values in the interval [− L, L] × [− L, L] and describe the PMF kernel by matrix $\mathbf{W}\in {\mathbb{R}}^{K\times K},K=2L+1$, where ${\left[\mathbf{W}\right]}_{mn}=w_{mn}.$ We see that the PMF shape is non-linearly dependent on PSF width σ and on the signal-to-background ratio a/b.

Estimation of unknown parameters in GLRT: implementation as a linear filter

To efficiently implement PMF as a linear filter, we present a series of simplifications and approximations.

Estimation of unknown position

The first and fundamental simplification is obtained when we approximate the unknown position of the molecule $m_0,n_0\in \mathbb{R}$ to acquire only integer values $m_0,n_0\in \mathbb{Z},$ where symbols $\mathbb{R}$ and $\mathbb{Z}$ refer to the real and integer numbers. Let us consider for now GLRT (11), where all unknown parameters except of the molecule position m₀, n₀ are known

$${\mathrm{\Lambda }}_G\left(\mathbf{R}\right)=\underset{m_0,n_0}{\max }\frac{p\left(\mathbf{R}\mid H_1,\widehat{a},\widehat{b},\widehat{\sigma },m_0,n_0\right)}{p\left(\mathbf{R}\mid H_0,\widehat{b}\right)}$$ 17

which can be adjusted (using the even symmetry of w_mn) to the form

$$T\left(\mathbf{R}\right)=\underset{m_0,n_0}{\max }\sum _{mn}{r}_{mn}{w}_{m_0-m,n_0-n}=\sum _{mn}{r}_{mn}{w}_{{\widehat{m}}_0-m,{\widehat{n}}_0-n},$$ 18

where

$${\widehat{m}}_0,{\widehat{n}}_0=\underset{m_0,n_0}{\mathrm{argmax}}\sum _{mn}{r}_{mn}{w}_{m_0-m,n_0-n}.$$ 19

Now, if $m_0,n_0\in \mathbb{Z},$ we recognize in (18) a discrete 2D convolution of raw image r_mn and PMF filter w_mn(16). The GLRT then takes the form

$$T\left(\mathbf{R}\right)=\underset{m_0,n_0}{\max }{\left[r**w\right]}_{m_0{n}_0}={\left[r**w\right]}_{{\widehat{m}}_0{\widehat{n}}_0}\stackrel{H_1}{>}{\tau }^{\prime },$$ 20

where symbol ** denotes a discrete 2D convolution. We denote the filtered image by matrix T, where ${\left[\mathbf{T}\right]}_{mn}={\left(r**w\right)}_{mn}.$ The maximum of filtered image ${\left[\mathbf{T}\right]}_{\widehat{m_0}{\widehat{n}}_0}$ is compared to the threshold to decide on the presence of a molecule. Note that the derivation naturally assumes a single-molecule detection problem. For multiple non-overlapping molecule detections, the maximum in (20) needs to be replaced by the local maximum in the neighborhood of the PSF size. Reference⁴⁴ summarizes various efficient ways to implement local maximum search. Note that the non-maximum suppression does not need to be evaluated for all pixels of the image, but only for those that are larger than the detection threshold ${\tau }^{\prime }$. The generalization of the detection problem of multiple overlapping molecules (denoted as crowded field problem) is given in Supplementary Note 12.

Estimation of parameters a, b, and σ

The PMF filter kernel w_mn(16) then depends on the unknown molecule parameters a, b, and PSF parameter σ. Some references^19,25 assume knowledge of the PSF σ parameter or at least its roughly estimated value⁴⁵ given e.g., by the expression $\widehat{\sigma }=0.21{\lambda }_{\mathrm{em}}/\mathrm{NA},$ where λ_em is the emission wavelength and NA is the numerical aperture. The maximum likelihood estimate of a and b with fixed $\widehat{\sigma }$ is a convex optimization problem with effective algorithms available⁴⁶. However, the estimation needs to be evaluated for each pixel in the raw data which is still numerically too intensive.

To implement PMF as a linear filter, we need to fix the values of the unknown parameters a, b. A possible approach takes advantage of the fact that the detection algorithm is part of SMLM software. We use the 0.25 percentile of the true parameter values obtained from the SMLM localization algorithm applied on a randomly selected set of a few frames of the raw data. This approach conveniently does not require user interaction.

Probabilistic thresholding and probability of detection

We take advantage of the knowledge of the analytical closed-form formula for the probability of false positive P_FP. We evaluate two performance metrics (probability of false positive P_FP and probability of detection P_D) characterizing two fundamental error types of False Positive (FP) and False Negative (FN) detection. The probability threshold is then obtained by inverting the analytical expression for P_FP.

Probability of false positive detection

Probability P_FP is the probability that the GLRT (15) decides in favor of H₁ (test statistic T(R) is higher than threshold ${\tau }^{\prime }$) but H₀ is true

$$P_{\mathrm{FP}}\triangleq \mathrm{Pr}\left[\widehat{H}=H_1\mid H_0\right]=\mathrm{Pr}\left[T\left(\mathbf{R}\right)>{\tau }^{\prime }\mid H_0\right],$$ 21

where the symbol $\widehat{H}$ denotes the chosen hypothesis. We determine the performance by analyzing the distribution of the test statistics T(R) on the condition that the molecule position coordinates are integer numbers $m_0,n_0\in \mathbb{Z}.$ Let us re-write the weighted sum of random variables (15) using the fact that PMF is significantly non-zero for m, n ∈ [ − L, L] to get

$$T\left(\mathbf{R}\right)=\sum _{m,n=-L}^L{r}_{m+{\widehat{m}}_0,n+{\widehat{n}}_0}{w}_{m,n}$$ 22

where we have shortened the notation by ${\sum }_{m,n=-L}^L\triangleq {\sum }_{m=-L}^L{\sum }_{n=-L}^L.$ The test statistic (22) is a random variable with a distribution well approximated by a Gaussian distribution due to the central limit theorem. The full statistical description is obtained by expressing the first two moments for the null hypothesis H₀ where $r_{m+{\widehat{m}}_0,n+{\widehat{n}}_0}~\mathcal{P}\left(b\right)$ and so

$$E\left[T\left(\mathbf{R}\right)\mid H_0\right]=E\left[\sum _{m,n=-L}^L{r}_{m+{\widehat{m}}_0,n+{\widehat{n}}_0}{w}_{mn}\mid H_0\right]=b\sum _{m,n=-L}^L{w}_{mn},$$ 23

$$\mathrm{var}\left[T\left(\mathbf{R}\right)\mid H_0\right]=\mathrm{var}\left[\sum _{m,n=-L}^L{r}_{m+{\widehat{m}}_0,n+{\widehat{n}}_0}{w}_{mn}\mid H_0\right]=b\sum _{m,n=-L}^L{w}_{mn}^2,$$ 24

using the linearity of the mean operator E[⋆] and the rule for the variance of the sum of independent random variables. The false positive probability P_FP can be expressed in a closed form using the complementary cumulative distribution function (also denoted as the survival function) of standard normal distribution

$$Q\left(x\right)\triangleq \frac{1}{\sqrt{2\pi }}{\int }_x^{\infty }\exp \left(-\frac{t^2}{2}\right)\mathrm{d}t$$ 25

as

$$P_{\mathrm{FP}}=Q\left(\frac{{\tau }^{\prime }-E\left[T\left(\mathbf{R}\right)\mid H_0\right]}{\sqrt{\mathrm{var}\left[T\left(\mathbf{R}\right)\mid H_0\right]}}\right)=Q\left(\frac{{\tau }^{\prime }-b{\sum }_{m,n=-L}^L{w}_{mn}}{\sqrt{b{\sum }_{m,n=-L}^L{w}_{mn}^2}}\right).$$ 26

Probability of detection

Similarly as for P_FP, the detection probability is given by

$$P_{\mathrm{D}}\triangleq \mathrm{Pr}\left[\widehat{H}=H_1\mid H_1\right]=\mathrm{Pr}\left[T\left(\mathbf{R}\right)>{\tau }^{\prime }\mid H_1\right].$$ 27

We can express P_D in a close form when the integer molecule position is estimated correctly (${\widehat{m}}_0=m_0,{\widehat{n}}_0=n_0$) where the received signal $r_{m+{\widehat{m}}_0,n+{\widehat{n}}_0}~\mathcal{P}\left(a{s}_{mn}+b\right)\mid H_1$ and so the test statistics are

$$E\left[T\left(\mathbf{R}\right)\mid H_1\right]=E\left[{\sum }_{m,n=-L}^L{r}_{m+{\widehat{m}}_0,n+{\widehat{n}}_0}{w}_{mn}\mid H_0\right]=a\sum _{m,n=-L}^L{s}_{mn}{w}_{mn}+b\sum _{m,n=-L}^L{w}_{mn},$$ 28

$$\mathrm{var}\left[T\left(\mathbf{R}\right)\mid H_1\right]=\mathrm{var}\left[\sum _{m,n=-L}^L{r}_{m+{\widehat{m}}_0,n+{\widehat{n}}_0}{w}_{mn}\mid H_0\right]=a\sum _{m,n=-L}^L{s}_{mn}{w}_{mn}^2+b\sum _{m,n=-L}^L{w}_{mn}^2.$$ 29

The detection probability is then

$$P_{\mathrm{D}}=Q\left(\frac{{\tau }^{\prime }-E\left[T\left(\mathbf{R}\right)\mid H_1\right]}{\sqrt{\mathrm{var}\left[T\left(\mathbf{R}\right)\mid H_1\right]}}\right)=Q\left(\frac{{\tau }^{\prime }-a{\sum }_{m,n=-L}^L{s}_{mn}{w}_{mn}-b{\sum }_{m,n=-L}^L{w}_{mn}}{\sqrt{a{\sum }_{m,n=-L}^L{s}_{mn}{w}_{mn}^2+b{\sum }_{m,n=-L}^L{w}_{mn}^2}}\right).$$ 30

Probabilistic threshold controlling the false positive probability

Probability P_FP(26) depends on the mean background level b and the threshold ${\tau }^{\prime }$. After capturing the image, the mean background parameter b can be estimated considering that the quantity of background pixels is usually significantly higher than that of signal pixels (more discussed in “Background Estimation for Adaptive Thresholding”). Then, the only parameter controlling the probability P_FP is the decision threshold ${\tau }^{\prime }$. The Q function (25) is monotonic, so it has an inversion (sometimes called inverse survival function) leading to the expression of the decision threshold as

$${\tau }^{\prime }=\sqrt{\widehat{b}{\sum }_{m,n=-L}^L{w}_{mn}^2}{Q}^{-1}\left(P_{\mathrm{FP}}\right)+\widehat{b}\sum _{m,n=-L}^L{w}_{mn}.$$ 31

We refer to the threshold (31) as the probabilistic threshold to emphasis that the threshold is determined by the selected probability of false positive P_FP. Note that the probabilistic threshold (31) is not conditioned by the PMF filter and can be used with any other linear filter common in SMLM.

Background estimation for adaptive thresholding

We assume a sufficiently sparse fluorescence signal with most of the pixels being the background that allows a maximum likelihood estimation⁴⁷ of the parameters of the Poisson and normal distribution necessary to calculate the probabilistic thresholds (31). Estimating the local background level around the pixel under evaluation by the detection algorithm allows to adapt the threshold to slow-varying background.

Mean estimate

The optimal maximum likelihood estimation of the central moment for both Poisson and Gaussian distributions is given by the arithmetic mean^21,47. Let us denote the local arithmetic mean background estimate as

$${\widehat{b}}_{ij}=\frac{1}{{\sum }_{mn}{w}_{mn}}\underset{\left(m,n\right)\in {\mathcal{K}}_{ij}}{\mathrm{mean}}\left[t_{mn}\right],$$ 32

where t_mn denotes the mnth pixel of the filtered image $t_{mn}={\left[\mathbf{T}\right]}_{mn}={\left(r**w\right)}_{mn}.$ The normalization by the sum of weights ∑_mnw_mn corresponds to (23). The symbol ${\mathcal{K}}_{ij}$ describes the set of indexes centered at position (i, j) corresponding to the reference pixels used in the local arithmetic mean as shown in Fig. 3d). The local mean (32) can be efficiently implemented as a linear filter as

$${\widehat{b}}_{ij}=\frac{1}{{\sum }_{mn}{w}_{mn}}{\left(t**k\right)}_{ij}=\frac{1}{{\sum }_{mn}{w}_{mn}}\sum _{m,n=-L}^L{k}_{mn}\cdot t_{i-m,j-n},$$ 33

where $\mathbf{K}\in {\mathbb{R}}^{K\times K}$ denotes the kernel matrix with k_mn being the mnth value ${\left[\mathbf{K}\right]}_{mn}=k_{mn}$ and K is the kernel size assumed to be an odd number with K = 2L + 1. The kernel is normalized to perform the arithmetic mean so ∑_mnk_mn = 1. For example,

$$\mathbf{K}=\frac{1}{40}\left[\begin{array}{ccccccc}\hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill \end{array}\right]$$ 34

is the square-shaped kernel of size 7 with a guard interval of 3 pixels. In the case where nearby fluorescence frequently occurs so that the signal is often present in the reference pixels, the arithmetic mean (32) is biased and better results can be obtained by local median filtering

$${\widehat{b}}_{ij}=\frac{1}{{\sum }_{mn}{w}_{mn}}\underset{\left(m,n\right)\in {\mathcal{K}}_{ij}}{\mathrm{median}}\left[t_{mn}\right].$$ 35

There are known efficient algorithms for median filtering with linear⁴⁸, and even constant⁴⁹ computational complexity.

Variance estimate

Probabilistic thresholding for Poisson-Gaussian noise (described in Supplementary Note 4) also requires estimation of the second moment, which can be done as local filtering. The variance estimation of the normal distribution can be implemented by linear mean filter as

$${\widehat{\sigma }}_{b,ij}^2=\frac{1}{{\sum }_{mn}{w}_{mn}^2}\underset{\left(m,n\right)\in {\mathcal{K}}_{ij}}{\mathrm{mean}}\left[{\left(t_{mn}-{\widehat{b}}_{mn}\sum _{mn}{w}_{mn}\right)}^2\right].$$ 36

Robust estimation can be performed e.g., by the Inter-Quantile Range estimate as

$${\widehat{\sigma }}_{b,ij}^2=\frac{1}{{\sum }_{mn}{w}_{mn}^2}0.7413\left(\underset{\left(m,n\right)\in {\mathcal{K}}_{ij}}{\mathrm{iqr}}\left[t_{mn}\right]\right),$$ 37

or by median absolute deviation and other alternatives^50,51.

Reference methods

We consider the following filters as reference methods (in alphabetical order): B-Spline (BS) wavelet filter^9,52, DoA means^53,54, DoG^55–57, LoG^8,58, LG⁵⁹, MF^25,26, and the net-gradient search⁶⁰. All methods except the net-grad search are based on linear filtering and for their description it is sufficient to define just the filter kernel. The entire detection procedure for the general method using a linear filter is identical to that of PMF: the test statistic is calculated using Eqs. (15) and (20), the probabilistic decision threshold using (2) in the manuscript, and (14) in Supplementary Note 4, and the background estimation (Background estimation for adaptive thresholding). Selected filters are visualized in Supplementary Note 13.

À trous B-Spline (BS) wavelet filter

Continuous B-Spline basis is given by the recursion

$$B_1\left(x\right)=\left\{\begin{array}{cc}1\hfill & 0\le x<1,\hfill \\ 0\hfill & \mathrm{else,}\hfill \end{array}\right)$$ 38

$$B_q\left(x\right)=\frac{x}{q-1}{B}_{q-1}\left(x\right)+\frac{q-x}{q-1}{B}_{q-1}\left(x-1\right),$$ 39

where $q\in \mathbb{N}$ is the BS order. Scaled, normalized, and continuous BS basis with the scale $s\in \mathbb{N}$ is

$$BS\left(x,q,s\right)=\frac{1}{s}{B}_q\left(\frac{x}{s}+\frac{q}{2}\right).$$ 40

The sampled BS wavelet for the number of samples K = 2⌈qs/2⌉ − 1 is usually called ${\mathbf{k}}_1\in {\mathbb{R}}^L$ so

$${\mathbf{k}}_1\left(q,s\right)=BS\left(i,q,s\right),i\in \left\{-L,-L+1,\dots ,L\right\},$$ 41

where K = 2L + 1. For example ${\mathbf{k}}_1\left(3,2\right)=\left(1/16,1/4,3/8,1/4,1/16\right).$ The à trous vector ${\mathbf{k}}_2\in {\mathbb{R}}^{2K-1}$ denotes the vector k₁ with inserted in between zeros, such that ${\mathbf{k}}_2\left(3,2\right)=\left(1/16,0,1/4,0,3/8,0,1/4,0,1/16\right)$. We construct the filter kernel ${\left[\mathbf{W}\right]}_{mn}=w_{mn}$ of à trous BS wavelet as

$$\mathbf{W}={\mathbf{M}}_1**\left(\mathrm{\Delta }-{\mathbf{M}}_2\right),$$ 42

where ${\mathbf{M}}_1={\mathbf{k}}_1{\mathbf{k}}_1^T\in {\mathbb{R}}^{K\times K}$ and ${\mathbf{M}}_2={\mathbf{k}}_2{\mathbf{k}}_2^T\in {\mathbb{R}}^{\left(2K-1\right)\times \left(2K-1\right)}$ are matrices of outer products and $\mathrm{\Delta }\in {\mathbb{R}}^{\left(2K-1\right)\times \left(2K-1\right)}$ is the zero matrix with a single 1 in the center ${\left[\mathrm{\Delta }\right]}_{mn}=1,$ if m = n = K. 

Difference of Arithmetic means (DoA) filter

The DoA filter is a two-parameter filter

$$w\left(x,y,a_1,a_2\right)=\frac{1}{a_1^2}\mathrm{rec}\left(\frac{x}{a_1},\frac{y}{a_1}\right)-\frac{1}{a_2^2}\mathrm{rec}\left(\frac{x}{a_2},\frac{y}{a_2}\right),$$ 43

where 2D rectangular function is defined as rec(x, y) = 1 for $\mid x\mid \le \frac{1}{2},\mid y\mid \le \frac{1}{2},$ and rec(x, y) = 0 else. Reference⁵³ recommends parameters equal to $a_1=2\sigma +1,a_2=2\left(2\sigma +1\right)$ which we assume here. Symbol σ is standard deviation of PSF.

Difference of Gaussians (DoG) filter

The DoG filter is a two-parameter filter

$$w\left(x,y,{\sigma }_1,{\sigma }_2\right)=\frac{1}{2\pi {\sigma }_1^2}{\mathrm{e}}^{-\frac{1}{2{\sigma }_1^2}\left(x^2+y^2\right)}-\frac{1}{2\pi {\sigma }_2^2}{\mathrm{e}}^{-\frac{1}{2{\sigma }_2^2}\left(x^2+y^2\right)},$$ 44

which can approximate the LoG filter¹² very-well for ${\sigma }_1=0.805{\sigma }_{\mathrm{LoG}},{\sigma }_2=1.288{\sigma }_{\mathrm{LoG}}.$ SMLM literature⁵⁶ recommends values σ₁ = 0.5σ, σ₂ = 2 FWHM = 4.710σ which we assume here.

Laplacian of Gaussian (LoG) filter

The LoG filter also known as Mexican Hat, 2nd Derivative of Gaussian, or Ricker Wavelet, is given by

$$w\left(x,y,{\sigma }_{\mathrm{LoG}}\right)=\left(\frac{2{\sigma }_{\mathrm{LoG}}^2-x^2-y^2}{2\pi {\sigma }_{\mathrm{LoG}}^6}\right){\mathrm{e}}^{-\frac{1}{2{\sigma }_{\mathrm{LoG}}^2}\left(x^2+y^2\right)},$$ 45

where LoG parameter ${\sigma }_{\mathrm{LoG}}$ is assumed to closely approximate Gaussian function which leads to ${\sigma }_{\mathrm{LoG}}=\sqrt{2}\sigma$ with σ being standard deviation of PSF.

Lowered Gaussian (LG) filter

The LG filter has a Gaussian kernel shape (8) from which the total value is subtracted such that the filter has the sum of values equal to zero ∑_m,nw_m,n = 0. The zero-sum property (also found in LoG, DoG, DoA, and BS filters) is advantageous because it implies that a slowly changing background is filtered out.

Matched Filter (MF)

MF has the continuous form of filter kernel equal to s(σ) PSF model, which e.g., for Gaussian PSF model equals to

$$w\left(x,y,\sigma \right)=\frac{1}{2\pi {\sigma }^2}{\mathrm{e}}^{-\frac{1}{2{\sigma }^2}\left(x^2+y^2\right)}$$ 46

with σ being s.t.d. of PSF function.

Net-gradient search

The net-gradient search is a non-linear filtering method that sums raw image gradient vector magnitudes in a rectangle box. We implement the image gradient by Sobel filters

$$\mathbf{G}=\left[\begin{array}{ccc}\hfill 1\hfill & \hfill 0\hfill & \hfill -1\hfill \\ \hfill 2\hfill & \hfill 0\hfill & \hfill -2\hfill \\ \hfill 1\hfill & \hfill 0\hfill & \hfill -1\hfill \end{array}\right],\mathbf{H}=\left[\begin{array}{ccc}\hfill 1\hfill & \hfill 2\hfill & \hfill 1\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill -1\hfill & \hfill -2\hfill & \hfill -1\hfill \end{array}\right]$$ 47

The squared outputs of these filters are summed and then the square root is taken to compute the magnitude of the gradient at each pixel. These gradient magnitudes are then averaged using a 5 × 5 box filter to smooth the final result.

Poisson Matched Filter (PMF)

The PMF filter derived in “Poisson Matched Filter (PMF)” has the continuous form of filter kernel equal to

$$w\left(x,y,a,b,\sigma \right)=ln\left(1+\frac{a}{b}s\left(\sigma \right)\right),$$ 48

where the procedure for selection of parameters a, b, σ is discussed in “Estimation of Unknown Parameters in GLRT: Implementation as a Linear Filter” and s(σ) is the sampled PSF function.

No filter

As a reference method, we also consider the no-filter case, which can be described as a linear filter with discrete weights w_mn = 1 for m = n, and w_mn = 0 else.

Performance evaluation

Simulations with uniform background and Poisson noise

First, the detection methods are tested in a constant background environment where a single randomly placed spot in Poisson noise is detected as illustrated in Supplementary Fig. 10.

In every Monte Carlo run, the spot position is a different real random vector. True Positive (TP) is counted when the detected position is within the tolerated distance of ${\tau }_{\mathrm{TP}}=\sqrt{2}\mathrm{px}$ from the ground true value. True Negative (TN) is counted as a correctly detected background pixel. False positive is a falsely marked detection which is farther than τ_TP. False Negative (FN) is missed detection. By repeating the experiment many times (we ran M_MC = 5000 Monte Carlo trials), we can express ${\widehat{P}}_D=\mathrm{mean}\left[\mathrm{TP}/\mathrm{TP}+\mathrm{FN}\right]$ and ${\widehat{P}}_{FP}=\mathrm{mean}\left[\mathrm{FP}/\mathrm{FP}+\mathrm{TN}\right].$. We select values of low signal-to-background ratio a/b = 0.75 (particularly with a = 750 photons and b = 1000 photons) where the differences between the methods are more pronounced. In cases with medium to high a/b ratios, the differences between the methods are smaller. The size of the raw image is 128 × 128 pixels, s.t.d. of PSF is σ = 1 px and the non-maximum suppression box is a 5 × 5 box. All the considered methods use probabilistic thresholding (introduced in “Probabilistic Thresholding and Probability of Detection”) to guarantee selected P_FP, but we rather use the estimated ${\widehat{P}}_{FP}$ in the plots that are occasionally slightly different from the true values. This is the reason why the plot markers are not perfectly aligned. PMF used the precise values of the unknown parameters a, b, σ.

Simulations with varying background and Poisson noise

We assume a parabolic type of background to model a non-uniform illumination similar to the effect of vignetting³⁷ which is coarsely approximated by parabola (${\cos }^4\left(\alpha \right)\simeq 1-2{\alpha }^2$). We use the parabola model 1/c²(x² + y²) with curvature value c = 10 pixels, which is then mean-shifted to have the same mean as in the case of uniform background (i.e., mean value b = 1000 ph). Supplementary Fig. 16 illustrates such an example signal.

PMF, MF, and no filter methods use local adaptive background estimation. The local adaptive background estimation used local mean implementation (32) in the floating square doughnut-shape kernel (illustrated in Fig. 3d) of the size 13 × 13 pixels with guard interval 5 × 5. 

Performance in bundled tubes long sequence (BT-LS) dataset

We used the publicly available dataset provided by the 2013 SMLM software challenge⁵. This dataset has a very good signal-to-noise ratio therefore, the performance of the most tested algorithms is close to ideal. In order to better observe the differences between the methods in challenging conditions, we added a noisy background of the level of 4000 photons. Type of the noise in BT-LS data corresponds to the Poisson-Gaussian noise, so the probabilistic thresholding uses formula (14) in Supplementary Note 4 with background estimation (32) and (36).

Performance in MT0_N2_LD dataset

Microtubulin dataset MT0_N2_LD publicly provided by the 2016 SMLM software challenge⁶ has a challenging signal-to-noise ratio and low density of emitters.

Performance in experimental datasets

To validate our method using experimental data, we created a reference sample with homogenously distributed single gold nanoparticles and visualized it in two modes. One under standard visualization conditions that result in bright diffraction-limited spots on a low background that are easy to detect and used this measurement as a reference. For the second visualization mode, we added an extra light source from the microscope lamp to increase the background and used it as testing data.

To reduce agglomeration of gold nanoparticles during deposition on a glass coverslip, we used gold nanoparticles that were attached to DNA origami (Supplementary Fig. 14) with biotinylated sites and deposited them on neutravidin-treated glass coverslips, previously passivated with BSA-biotin (Supplementary Fig. 13). A detailed description of the experimental dataset (used materials, nanoparticle functionalization, DNA origami synthesis, origami-particle hybridization, sample preparation, and measurement) is given in Supplementary Note 9.

We created two datasets. One with a medium level of interference light and the other with a strong interference light. Ground truth detections were obtained by evaluating the same ROIs but without interference light. In rare cases, some fluorophores were not present in the reference data, which slightly raised the false positive detection rate. More often, the weaker fluorescence emission in the reference data disappeared completely in the presence of interference or was below the background noise level, which then showed up as a missed detection in the evaluation.

The type of noise in the experimental data was close to Poisson-Gaussian noise, so the detection threshold was set as described in Supplementary Note 4 with background estimation (32) and (36).

Author contributions

M.H. developed the theory, implementation, and simulations, D.R. and K.U. performed and set up measurements, N.H. and S.F. prepared samples and characterized them, V.P. acquired the funding and supervised the work. M.H. and V.P. wrote the manuscript with the contribution of all authors.

Peer review

Peer review information

Nature Communications thanks Koen Martens and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Data availability

All data are available at https://gitlab.com/VladkaPetrakova/petrakova_group/-/tree/main/mirek/publication/molecule_detection, data from the SMLM Challenges used for testing are available at https://bigwww.epfl.ch/smlm-datasets/challenge2D/datasets/Bundled_Tubes_Long_Sequence-GT/ and https://srm.epfl.ch/srm/dataset/challenge-3D-simulation/MT0.N2.LD/index.html.

Code availability

The code is available at https://gitlab.com/VladkaPetrakova/petrakova_group/-/tree/main/mirek/publication/molecule_detection.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-025-55952-5.