Structured illumination microscopy with noise-controlled image reconstructions

Abstract

Super-resolution structured illumination microscopy (SIM) has become a widely used method for biological imaging. Standard reconstruction algorithms, however, are prone to generate noise-specific artefacts that limit their applicability for lower signal-to-noise data. Here, we present a physically realistic noise model that explains the structured noise artefact and that is used to motivate new complementary reconstruction approaches. True Wiener-filtered SIM optimizes contrast given the available signal-to-noise ratio, flat-noise SIM fully overcomes the structured noise artefact while maintaining resolving power. Both methods eliminate ad-hoc user adjustable reconstruction parameters in favour of physical parameters, enhancing objectivity. The new reconstructions point to a trade-off between contrast and a natural noise appearance. This trade-off can be partly overcome by additional notch filtering, but at the expense of a decrease in signal-to-noise ratio. The benefits of the proposed approaches are demonstrated on focal adhesion and tubulin samples in 2D and 3D, and on nano-fabricated fluorescent test patterns.

Introduction

Structured Illumination Microscopy (SIM) is a super-resolution technique that offers two-fold increased spatial resolution along two or three dimensions plus optical sectioning with strongly enhanced contrast compared to conventional widefield fluorescence microscopy using uniform illumination^1,2,3,4,5,6, and is compatible with live cell imaging^7,8,9. Further reduction of resolution below the 100 nm length scale can be achieved by making use of the non-linear fluorescence response^10,11,12,13. Current image reconstruction methods^3,4,14,15,16 depend on ad-hoc tuneable parameters and are susceptible to various types of artefacts^17,18. The effort to distinguish ‘real’ structural resolution improvement from noise-related deconvolution artefacts in SIM has recently inspired much controversy in the field^19,20. The root causes of this confusion are (i) the lack of insight into the mechanism of image formation which requires a reconstruction procedure of considerable mathematical complexity, (ii) the use of ad-hoc parameters with large impact on the final image but with unclear physical meaning, and (iii) the absence of an unambiguous method to separate true signal from noise.

Here, we address these issues first by an in-depth analysis of the propagation of noise through the image reconstruction chain of state-of-the-art linearly filtered SIM. We use this to elucidate the structured noise artefact of SIM, the amplification of noise structures at intermediate length scales at low signal conditions. In particular, the analysis clarifies the role of ad-hoc filtering operations such as regularization. In a next step, the understanding of noise propagation is applied in new SIM reconstruction methods. These methods are designed for optimizing contrast or optimizing a natural noise appearance, and for eliminating ad-hoc reconstruction parameters. The underlying goal is to make the representation of objects in SIM images as objective as possible.

Results

Propagation of noise through the image reconstruction chain of SIM

The goal of the image reconstruction is to provide an estimate $e_k^{\text{rec}}$ (with k = 1,⋯,N the index denoting the different pixels) of the underlying fluorescent object f_k. This estimate depends linearly on the images acquired for the M_r different rotations and M_t different translations of the illumination pattern via a sequence of Fourier (spatial frequency) space manipulations (see Supplementary Note 1). The reconstructed object $e_k^{\text{rec}}$ is a sum of a term e_k that corresponds to the reconstruction in the absence of noise and a perturbation δe_k of that ideal reconstruction due to noise. The Fourier transforms of e_k and δe_k are denoted as ê_j and δê_j, respectively, where the hat indicates the Fourier transform and the index j indicates the Fourier pixel with spatial frequency ${\overrightarrow{q}}_j$.

The signal power and the noise variance in Fourier space ${\widehat{S}}_j\equiv {\left|{\widehat{e}}_j\right|}^2\text{and}{\widehat{N}}_j\equiv \langle {\left|\delta {\widehat{e}}_j\right|}^2\rangle$ are used here to quantify the noise and signal level. The noise variance can be determined empirically by making K repeated acquisitions of a fixed specimen, and computing the variance over the K reconstructions in Fourier space. We have analysed the propagation of noise through the reconstruction chain (full theory in Supplementary Note 2) and found an analytical expression for the signal power and noise variance:

$${\widehat{S}}_j=\frac{{{{\displaystyle \widehat{A}}}_j}^2{{{\displaystyle \widehat{D}}}_j}^2}{{[{\widehat{w}}_j+{\widehat{D}}_j]}^2}|{\widehat{f}}_j{|}^2$$ (1a)

$${\widehat{N}}_j=\frac{{{{\displaystyle \widehat{A}}}_j}^2}{{[{\widehat{w}}_j+{\widehat{D}}_j]}^2}({\widehat{f}}_0{\widehat{V}}_j+N{\sigma }^2{\widehat{D}}_j)$$ (1b)

Here ${\widehat{f}}_j$ is the spatial frequency spectrum of the underlying fluorescent object, ${\widehat{f}}_0\equiv {\displaystyle {\sum }_{k=1}^N{f}_k}$ is the sum fluorescence signal, Â_j is the apodization filter²¹, which has a low-pass character for avoiding halo, negative pixel and edge ringing artefacts, Ŵ_j is the regularization filter, and σ is the root-mean-square camera readout noise. State-of-the-art SIM reconstructions use a regularization filter that is ad-hoc chosen to be independent of spatial frequency (Ŵ_j = w for all ${\overrightarrow{q}}_j$). The functions ${\widehat{D}}_j$ and ${\widehat{V}}_j$ are sums over the M_r rotations and over the set 𝕄 of non-zero Fourier orders of the periodic illumination pattern:

$${\widehat{D}}_j=M_t\sum _{r=1}^{M_r}{{\displaystyle \sum }}^{}_{m\in 𝕄}|{\widehat{a}}_m{|}^2|\widehat{g}({\overrightarrow{q}}_j-{\overrightarrow{k}}_{rm}){|}^2$$ (2a)

$${\widehat{V}}_j=M_t{{\displaystyle \sum }}^{}_{r=1}^{M_r}{{\displaystyle \sum }}^{}_{m,m^{\prime }\in 𝕄}{\widehat{a}}_m{{{\displaystyle \widehat{a}}}_{m^{\prime }}}^{\ast }{\widehat{a}}_{m-m^{\prime }}\widehat{g}({\overrightarrow{q}}_j-{\overrightarrow{k}}_{rm}{)}^{\ast }\widehat{g}({\overrightarrow{q}}_j-{\overrightarrow{k}}_{r{m}^{\prime }})\widehat{g}({\overrightarrow{k}}_{r{m}^{\prime }}-{\overrightarrow{k}}_{rm})$$ (2b)

where $\widehat{g}({\overrightarrow{q}}_j)$ is the microscope’s Optical Transfer Function (OTF), â_m are the Fourier order strengths, and ${\overrightarrow{k}}_{rm}$ are the centre spatial frequencies of the orders. The noise variance is the sum of contributions due to shot noise and due to readout noise. For current sCMOS or EM-CCD cameras the readout noise is zero or very nearly so. For that reason we will henceforth neglect the readout noise term. Our theory predicts that in general the noise variance is not constant across Fourier space, implying that SIM suffers from spatial frequency dependent noise amplification. This is the explanation of the structured noise artefact of SIM.

The noise model also enables the assessment of the expected noise power averaged over a neighbourhood of a pixel k in real space (see Supplementary Note 2). The expected noise power is used to compute the fraction Z_k of the reconstruction that can be attributed to noise in the neighbourhood of pixel k. This noise fraction may be used as a feature confidence map, quantifying the impact of noise enhancement in different sub-regions of the final SIM reconstruction.

In the limit where shot noise is the only noise source we can also use another noise assessment based on a single SIM acquisition, one that is model independent. The detected photo-electrons in each pixel of the set of raw images are randomly assigned to two new datasets according to a binomial distribution. If the number of photo-electrons per pixel is a statistical variable following Poisson statistics then it can be shown that the number of photo-electrons per pixel of the two split datasets also follow Poisson statistics²², and that the two Poisson distributions are independent, implying that the noise in the two split datasets is uncorrelated. By processing the split datasets two noise independent SIM reconstructions (at half the average signal level) can be generated. The noise level in the SIM reconstruction derived from the original full dataset can now be assessed from the correlations between the two reconstructions derived from the split data (see Supplementary Note 2 for proofs).

We have tested the theoretical noise analysis by imaging a fixed sample of GFP-labelled zyxin, a building block of focal adhesions²³, in U2OS cells K = 10 times (experimental details in Methods). The signal level is kept low (see Figure 1a-b) to make the effects of noise amplification better visible. Figure 1c-h show the widefield image and SIM reconstruction for different settings of the regularization parameters w, with clear noise enhancement for small w. Figure 1i-l shows that the noise variance according to our model agrees well with the empirical multi-image noise variance, as well as with the model-independent noise assessment from processing the split datasets, providing two controls that validate our noise model. The noise enhancement at intermediate spatial frequencies for decreasing w is clearly present. The impact of w on the reconstruction, the noise fraction map that quantifies the level of noise enhancement, and the spectral noise variance is shown in Supplementary Movie 1.

Figure 1. Noise propagation in SIM reconstructions.

(a) Example raw image of a SIM acquisition of GFP-zyxin in focal adhesions, and (b) Modulation Contrast to Noise Ratio (MCNR), indicating muted stripe contrast due to low signal levels. (c-d) Widefield and state-of-the-art SIM reconstruction with constant regularization w. (e-f) Insets of the boxed region in (d), showing the widefield image and SIM reconstructions with different regularization parameters w. A noise pattern builds up when w is decreased. (i-l) Measured noise variance 〈N〉 in Fourier space over the 10 independent SIM reconstructions, averaged over rings in Fourier space, indicating that the noise enhancement in SIM at low w is concentrated in a ring in Fourier space. The measured noise variance agrees well with the prediction of the proposed noise model, and with the assessment from the randomly split datasets. (m) The SSNR determined from the 10 independent acquisitions for widefield and SIM (and averaged over rings in Fourier space) agrees well with the estimated SSNR for a single SIM reconstruction based on the proposed noise model, and based on the randomly split datasets. (k) Fourier Ring Correlation (FRC) curve for widefield and SIM. The intersection of the FRC curves with the resolution threshold 1/7 (dashed curve) gives FRC resolution values 239±2 nm (widefield, red arrow) and 149±2 nm (SIM, blue arrow) below and above the widefield diffraction limit λ/2NA = 191 nm. Scale bar (a-d) 5 μm, scale bar (e-h) 1 μm.

Figure 1m shows the spectral Signal-to-Noise Ratio²⁴ $SSNR={\widehat{S}}_j/{\widehat{N}}_j$ (averaged over rings in Fourier space) as a function of spatial frequency. For the low signal levels in the current experiment SIM only provides a gain over widefield imaging in the region around the widefield diffraction limit, for length scales larger than about λ/NA = 382 nm there is no difference in SSNR. This is attributed to the relatively low effective first order modulation in view of the non-zero thickness of the sample. The limited improvement due to the low signal level is also revealed by the experimental Fourier Ring Correlation (FRC)²⁵ curves (Figure 1j), indicating an improvement of FRC-resolution in SIM (149±2 nm) over widefield (239±2 nm) with a factor 1.6, and significantly less than the extended SIM diffraction limit (1/(2NA/λ + 2/p)=103 nm for the estimated pattern pitch p = 444 nm), and the diffraction limit for widefield (λ/2NA = 192 nm), respectively.

Noise-controlled SIM reconstructions

The SSNR according to our noise propagation model appears to be equal to the product of two factors. The first factor $|{\widehat{f}}_j{|}^2/{\widehat{f}}_0$ depends only on the spatial frequency spectrum of the underlying object, the fluorescent labelling density, and the detected fluorescence brightness. The second factor ${{{\displaystyle \widehat{D}}}_j}^2/{\widehat{V}}_j$ depends only on the imaging system (the microscope OTF and the illumination patterns). The SSNR is independent of the regularization filter ŵ_j and of any subsequent apodization filter Â_j. The implication is that several, physically equivalent, choices for these filters can be made, each revealing or emphasizing different aspects of image quality, such as contrast or a natural noise appearance, but rooted in exactly the same underlying information content. It is also clear that these different choices cannot tune signal level and contrast independent from the noise appearance.

The first choice of noise-controlled filtering follows the Wiener principle, which optimizes contrast by making the OTF as close as possible to a target OTF (e.g. the apodization filter), subject to available signal-to-noise ratio. Mathematically, we seek the minimum of the noise averaged quadratic difference between reconstruction and underlying object, low-pass filtered by the target OTF (see Supplementary Note 3). This implies a regularization filter:

$${\widehat{w}}_j=\frac{{\widehat{V}}_j{\widehat{f}}_0}{{\widehat{D}}_j{\left|{\widehat{f}}_j\right|}^2}=\frac{{\widehat{D}}_j}{SSN{R}_j}$$ (3)

that increases with decreasing SSNR, different from state-of-the-art SIM with constant regularization. The key problem in applying true-Wiener filtering is that it requires an estimate of the SSNR from the image itself. The reconstruction $|{\widehat{e}}_j+\delta {\widehat{e}}_j{|}^2={\widehat{S}}_j+2Re\{{\widehat{e}}_j\delta {{{\displaystyle \widehat{e}}}_j}^{\ast }\}+{\widehat{N}}_j$ is a sum of the signal power Ŝ_j, the noise power ${\widehat{N}}_j$, and a signal-noise cross-term. Averaging over rings in spatial frequency space effectively eliminates the signal-noise cross-term, leaving the sum of signal and noise power. The proposed noise model enables a direct computation of the noise variance ${\widehat{N}}_j$, which can subsequently be used to estimate the signal power Ŝ_j and the SSNR.

True-Wiener filtered SIM provides a high-contrast representation of the object, but it suffers, just as state-of-the-art SIM, from an elevated noise level at intermediate spatial frequencies, especially in cases with low photon count or low (effective) modulation of the illumination pattern¹⁸. The current noise analysis points to a second novel reconstruction scheme, one from which this structured noise artefact is eliminated altogether. According to Equation (1b), the spectral noise variance can be made independent of spatial frequency by using a regularization filter:

$${\widehat{w}}_j=\sqrt{{\widehat{V}}_j}-{\widehat{D}}_j$$ (4)

and by discarding the apodization filter altogether (Â_j = 1). This flat-noise SIM reconstruction provides a reconstruction that is independent of the object content and from which all ad-hoc tuneable parameters are eliminated, but instead is fully determined by physical parameters only.

The resulting OTF ${\widehat{g}}_j^{SIM}={\widehat{D}}_j/\sqrt{{\widehat{V}}_j}$ is a direct measure for the transfer in SSNR from the object to the final flat-noise SIM reconstruction. This can result in a relatively low OTF compared to true-Wiener or state-of-the-art reconstructions, implying a reduced contrast. A mitigation may be found in additional notch filtering of the image Fourier orders. This method has been introduced to suppress the 2D-SIM specific honeycomb artefact and/or background arising from out-of-focus layers^14,16. Here, we use it to optimize the image contrast, while keeping the noise spectrum flat. The depth and width of the notch filters per image Fourier order influence the reconstructions functions ${\widehat{D}}_j$ and ${\widehat{V}}_j$ and thereby the flat-noise SIM-OTF (see Supplementary Note 3). These notch filter parameters can thus be tuned to minimize the difference between the flat-noise SIM-OTF and a target OTF such as the apodization filter Â_j. This procedure improves the contrast of flat-noise SIM across all spatial frequencies, without introducing any user-adjustable parameters. A drawback is that the overall SSNR level for the lower spatial frequencies is decreased, i.e. the overall noise level is higher.

We have tested the three proposed noise-controlled reconstructions on the GFP-zyxin datasets of Figure 1. Figure 2a-c show the widefield, and true-Wiener, flat-noise, and notch-filtered SIM reconstructions, Figure 2d-f show the corresponding empirical noise variances. Extended Data Figure 1 shows these reconstructions in combination with the noise fraction maps. True-Wiener SIM corresponds visually to the optimum regularization setting for state-of-the-art SIM, providing a high contrast representation and avoiding over- or under-regularization. The structured noise artefact has disappeared in flat-noise SIM, at the expense, however, of a loss in contrast. Notch-filtered SIM fixes the loss in contrast, but at the expense of an increase in overall noise level. The measured spectral noise variances show a noise ring at intermediate spatial frequencies for true-Wiener SIM that is about an order of magnitude higher than the constant plateau for flat-noise SIM. Further evidence for these assessments is provided by the K noise-independent reconstructions and the average of the K reconstructions for the inset (see Supplementary Movie 2). The regularization parameters as a function of spatial frequency for true-Wiener and flat-noise SIM, shown in Figure 2g, appear to have low-pass and band-pass character, respectively. The experimental SSNR over the K reconstructions (Figure 2h) for true-Wiener and flat-noise SIM are identical and higher than the SSNR for notch-filtered SIM at lower spatial frequencies, and agree reasonably well with the single-image estimates used in the true-Wiener approach. The MTF (Modulation Transfer Function, absolute value of the OTF) for the three reconstructions (Figure 2h) show a reduced contrast for flat-noise SIM compared to true-Wiener and notch-contrast SIM.

Figure 2. Noise controlled SIM reconstructions.

(a) State-of-the-art SIM with regularization w = 5 × 10⁻⁴ (b) True-Wiener SIM, (c) flat-noise SIM, and (d) notch-filtered SIM reconstructions for the inset shown in Fig. 1e-h. The two-line sub-structure is a recognizable true feature, the pronounced small-scale twirls (‘hammer finish pattern’) is an artefact of reconstructed noise component. (e-h) Corresponding measured noise variance in 2D Fourier space over the 10 independent SIM reconstructions. State-of-the-art SIM has a noise ring at medium to high spatial frequencies, True-Wiener SIM has a noise ring at somewhat lower spatial frequencies, flat-noise SIM has a constant noise plateau, notch-filtered SIM has a constant, but elevated noise plateau. (i) Regularization parameter (averaged over rings in Fourier space) as a function of spatial frequency. (j) The SSNR (averaged over rings in Fourier space) for state-of-the-art, true-Wiener and flat-noise SIM is identical, the SSNR for notch-filtered SIM is lower for smaller spatial frequencies. (j) The FRC curves for state-of-the-art, true-Wiener and flat-noise SIM are identical, the FRC curve for notch-filtered SIM is lower for smaller spatial frequencies. (l) The MTF (averaged over rings in Fourier space) of the four reconstructions, indicating a lower contrast for flat-noise SIM compared to the other three reconstructions. Scale bar 1 μm.

Noise-controlled SIM reconstructions provide representations with strongly reduced noise enhancement in case of poor illumination pattern modulation (Extended Data Figure 2), in contrast to state-of-the-art SIM. Similar results are also obtained for different illumination patterns than line patterns (see Supplementary Note 3), in particular for a system that uses a Digital Micro-mirror Device (DMD) for scanning a multi-spot array across the sample²⁶ (Extended Data Figure 3).

Trade-off between contrast, noise and signal-to-noise

Next, we have imaged nano-fabricated test structures (experimental details in Methods). Figure 3a-d shows a widefield image and the three noise-controlled SIM reconstructions of a chirped line structure, demonstrating improved resolution of SIM and the trade-offs between contrast, noise level, and signal-to-noise ratio between the three novel SIM reconstructions. The average and standard deviation along the test object lines (Figure 3e-h) show about a two-fold lower noise level in flat-noise SIM. The modulations as a function of line spacing in relation to the noise level point to an identical SSNR and an image resolution of approximately 160 nm, close to the FRC estimate for the GFP-zyxin sample of Figure 1. Flat-noise SIM does provide better visibility of very high spatial frequencies, close to the image resolution, as demonstrated on 140 nm pitch line patterns in Extended Data Figure 4. The trade-offs are also evident from reconstructions of mCherry-CSYCP3 protein in the synaptonemal complex (Extended Data Figure 5), where the SIM reconstructions resolve the internal two-line structure with line distance of about 200 nm, but differ in modulation, contiguous line structure, and background noise.

Figure 3. Trade-off between noise and contrast in SIM reconstructions.

(a-d) Widefield and the three noise-controlled SIM reconstruction of a chirped nano-patterned structure. (e-h) Mean and standard deviation of the chirped line pattern over the boxed region in a for the four images. True-Wiener SIM has high contrast, at the expense of spatial frequency dependent noise enhancement. Flat-noise SIM shows two times less noise as quantified by the standard deviation of the line response, but with less contrast. Contrast is restored in notch-filtered SIM, but at the expense of a noise enhancement that is constant over all spatial frequencies. Scale bar 2 μm.

Noise-controlled SIM reconstructions in 3D

So far, results have been shown for 2D-SIM reconstructions of single focal slices of image acquisitions on relatively thin objects. The trade-off between contrast and noise enhancement as in 2D-SIM reconstructions is observed in 3D-SIM as well, exacerbated by the degree of optical sectioning along the optical axis. Figure 4, Extended Data Figure 6 and Supplementary Movies 3 to 6 show full 3D-SIM reconstructions and noise fraction maps of a tubulin sample at different signal levels (Supplementary Figure 1). Comparison of state-of-the-art SIM with a fixed regularization parameter (Figure 4b-d) to true-Wiener SIM (Figure 4e) shows that the SSNR and spatial frequency dependent regularization of true-Wiener SIM (Supplementary Figure 2) results in a reasonable reconstruction for all signal levels, avoiding over and under-regularization in all cases, and resulting in a satisfactory MTF (Supplementary Figure 3). Flat-noise SIM (Figure 4e) improves resolution somewhat, indicated by the reduction of the width of the tubulin filaments both laterally and axially, and shows no noise amplification. The degree of optical sectioning, however, is poor compared to true-Wiener SIM, making this representation not so valuable for 3D-SIM. This can be substantially improved by contrast-optimized flat-noise SIM using notch filtering (Figure 4f), albeit at the expense of a small decrease in SSNR (Supplementary Figure 2 and 3). We have observed similar characteristics of the different noise-controlled SIM reconstructions in SIM images of a bead layer dataset (Extended Data Figure 7), of a three-color labelled bovine pulmonary artery endothelial cell (BPAEC) (Extended Data Figure 8 and Supplementary Movies 7), of a two-color labelled C127 cell (Extended Data Figure 9 and Supplementary Movies 8), and of a live HeLa cell dataset at low signal conditions (Extended Data Figure 10, Supplementary Figure 4 and Supplementary Movie 9).

Figure 4. Noise-controlled 3D SIM reconstructions.

Cross-sections of 3D-reconstructions (xy and yz) for four different signal levels (camera exposure indicated) (a) widefield, (b-d) state-of-the-art SIM for low, medium and high regularization, (e) true-Wiener SIM, (f) flat-noise SIM, and (g) notch-filtered SIM. The dashed lines in (a) indicate the location of the xz and xy slices. Scale bar 3 μm.

Deconvolution and denoising

Richardson-Lucy (RL) deconvolution is known to improve the sharpness of point, line and edge features in an image, which, depending on the sample, slightly extends the spatial frequency content above the diffraction limit^27,28. It is also a suitable benchmark for the noise-controlled linear SIM reconstructions considered so far, as RL-deconvolution is also free from ad-hoc user adjustable parameters. RL-deconvolution is appropriate when the noise on the input images is Poisson-distributed. Flat-noise SIM offers this to a good degree, as opposed to other pre-processed inputs. Figure 5 and Supplementary Figure 5 show the results of RL-deconvolution applied to the widefield and the flat-noise SIM reconstructions of the zyxin sample, showing an improvement in resolution in both, which is reflected in slightly better FRC-resolution values (187±2 nm for widefield and 131±3 nm for SIM), but at the expense of noise enhancement, similar to state-of-the-art or true-Wiener SIM. It therefore seems that RL-deconvolution is constrained by the same trade-off between contrast and noise enhancement as linear SIM reconstructions.

Figure 5. Resolution improvement and noise enhancement in deconvolution of GFP-zyxin dataset.

(a) RL-deconvolution of widefield image, (b) DL-deconvolution of widefield image at 40 nm pixel size, (c) DL-deconvolution of widefield image at 65 nm pixel size, (d) RL-deconvolution of (flat-noise) SIM image, (e) H denoised SIM. Arrows indicate features where a significant difference is observed. (f-j) Spectral noise variance (on a logarithmic scale) over the K = 10 independent outcomes of the deconvolution for the different methods, showing different types and degrees of noise enhancement. Note the differences in scale for h in view of the different pixel size. (k) FRC curves of widefield based deconvolutions in comparison to the curve for widefield. The FRC-resolution improves from 239±2 nm to 187±2 nm (RL), 142±3 nm (DL 40 nm), and 123±6 nm (DL 65 nm). (l) FRC curves of SIM based deconvolutions in comparison to the curve for SIM. The FRC-resolution changes from 149±1 nm to 131±3 nm (RL) and 159±0.1 nm (H). Scale bar 1 μm.

We have further applied a recent Deep Learning (DL) based deconvolution method²⁹, which is trained to transform widefield images into images with SIM level resolution, to the GFP-zyxin dataset. Figure 5 and Supplementary Figure 5 show a trade-off between precision and accuracy of this method, pointing to limited robustness for image data not seen by the neural network in the training phase³⁰. Up sampling the input widefield image to 40 nm pixel size leads to a reasonable representation of image features with moderate improvement in resolution (142±3 nm FRC resolution), up sampling to 65 nm pixel size leads to a largely inaccurate outcome, but with high precision, indicated by a large improvement in the overall FRC curve (123±6 nm FRC resolution). Both settings lead to an anisotropic noise enhancement, larger in magnitude than for RL-deconvolution. Figure 5 and Supplementary Figure 5 also show the result of denoising with a Hessian (H) regularization^31,32. This method does indeed reduce the noise level for high spatial frequencies, but unfortunately at the expense of resolution (158.7±0.1 nm FRC resolution) and a varying, anisotropic, noise variance in Fourier space.

Conclusion

In summary, we have proposed three new, complementary image reconstruction methods for SIM. They reduce or fully eliminate ad-hoc user adjustable reconstruction parameters. In general, it is recommended to use the true-Wiener reconstruction as the default reconstruction, as this method seems to give the best overall compromise between contrast, resolution and noise profile. The noise fraction map can be used to quantitatively assess the noise enhancement artefact. Based on the examined datasets we propose that noise enhancement can be characterized as severe, mild or nihil when associated with noise fraction ranges Z_k >0.4, 0.4 ≥ Z_k >0.2, and 0.2 ≥Z_k, respectively. In case of severe noise enhancement it is recommended to either use the flat-noise (2D) or the notch-filtered (3D) reconstruction, specifically for image features on the intermediate length scale of around λ/NA, typically in the range 200-400 nm. As to additional deconvolution it appears that the RL-method applied to (flat-noise) SIM gives the best overall performance in terms of simplicity, resolution and degree of noise enhancement.

The currently proposed reconstructions can also be used for SIM with a reduced number of acquired images³³, and also for other modalities. True-Wiener filtering can be directly generalized to widefield imaging, as well as image scanning and rescan microscopy approaches⁶ and, with an analysis of noise propagation, be extended to lattice light sheet microscopy³⁴ and to tomographic imaging modalities³⁵. In a next step in algorithm development the regularization could be made to depend on spatial frequency as well as on the position in the imaged field³⁶, depending on the spectral and local SNR. In depth knowledge of noise and noise propagation in complex computational imaging techniques can also be used to steer or optimize different regularized approaches³¹, as well as DL based deconvolution, reconstruction or denoising^29,37,38,39 methods. Finally, the use of spectral SNR and the generation of image representations with a flat noise spectrum open up objective methods to assess the relative benefits of any super-resolution or deconvolution method.

Methods

Microscope setups

Images for Figure 1, 2, 3, and 5 and Extended Data Figure 1, 2, 4, and 5 are acquired with a commercial Zeiss Elyra PS1 system, using a 63×/1.40 oil immersion objective and a 1004×1002, 8 μm pixel, Andor iXon3 885 EM-CCD camera. A magnification adapted tube lens is used giving rise to a 79 nm back-projected pixel size. Raw images are acquired for five rotations and five translations of the line illumination pattern.

Images for Figure 4 and Extended Data Figure 6, 7, 8, 9, and 10 are acquired with a commercial DeltaVision OMX V3 Blaze (GE Healthcare) instrument, using a 60x/1.42 PlanApo oil immersion objective (Olympus) and a 2048×2048, 6.5 μm pixel, PCO edge 4.2 sCMOS cameras with a magnification adapted tube lens giving rise to a 82 nm back-projected pixel size. Raw images are acquired for three rotations and five translations of the line illumination pattern.

The raw images used for the reconstructions shown in Extended Data Figure 3 are recorded with a DMD-SIM setup described in detail elsewhere²⁶. In short, multi-spot arrays with a pitch of 10 DMD-pixels (DMD pixelsize 13.68 μm) were created using 488 nm despeckled laser illumination and projected onto the sample via a relay path and a 60×/0.7 air objective (projected DMD-pixel size 137 nm) of an Olympus IX71 microscope and subsequently scanned across the sample. The images were captured on a 2048×2048, 6.5 μm pixel, Hamamatsu Orca Flash 4.0 camera (projected pixel size 108 nm).

Samples

Figure 1, 2 and 5, Extended Data Figure 1, and Supplementary Figure 5 show data of GFP-zyxin expressing U2OS cells. Zyxin is an integral protein in focal adhesions, protein complexes that form a connection between the extracellular matrix through integrin receptors and the actin cytoskeleton, through its interaction with α-actinin⁴⁰ and the stretch sensitive protein p130^cas in which it acts as one of the mechanosensing components in focal adhesions^41,42. Sterile high precision coverslips #1.5H (Marienfeld Superior) were incubated with 10 μg/ml purecol (Advanced Biomatrix) overnight at 4°C and subsequently washed three times with PBS. U2OS cells were grown in DMEM and transfected with GFP-zyxin (a kind gift from Johan de Rooij). The cells were fixed for 20 min in 4% formaldehyde/PBS and mounted on a glass slide in Vectashield antifade mounting medium (Vectorlabs). Cells grown on collagen coated coverslips typically show rod shaped focal adhesions, consisting of parallel linear structures²³, additionally zyxin is observed to be present on actin fibers in distinct patches.

Figure 3 and Extended Data Figure 4 show data of nano-fabricated fluorescent test structures patterned using a previously published method⁴³, that was slightly adopted for higher resolution patterning. In short, a monolayer of 3-[Methoxy(polyethyleneoxy)propyl]trimethoxysilane (ABCR, Germany) was covalently grafted onto ITO-coated no.1 cover glass (Optics Balzers) and locally exposed to a focused electron beam following the defined patterns. After removal from the SEM, the sample was incubated for 45 minutes with a 100nM IgG-Alexa Fluor 488 solution in 1X TE buffer. Samples were then washed with TE buffer and deionized water, followed by drying and transfer to the optical microscope.

Extended Data Figure 5 show data of the mCherry-SYCP3 protein in the synaptonemal complex. Mouse oocytes from mice expressing mCherry-SYCP3⁴⁴ were isolated and spread⁴⁵ on #1.5H high precision coverslips and embedded in prolong gold (Invitrogen). SYCP3 is part of the lateral element of the synaptonemal complex (SC) that is formed during meiosis prophase I. The synaptonemal complex is comprised of two lateral elements that form parallel linear protein assemblies at a distance of ~220 nm apart⁴⁶.

Figure 4, Extended Data Figure 6, and Supplementary Movies 3 to 6 show data of collected of 4% formaldehyde fixed mouse C127 cells grown on #1.5H high precision coverslips, immunostained for microtubules using DM1A mouse monoclonal anti-α-Tubulin primary antibodies (Sigma-Aldrich) and donkey anti-mouse Alexa Fluor 488 secondary antibodies (ThermoFisher), and mounted in Vectashield H-1000 medium (Vector Labs). Extended Data Figure 7 shows data of a monolayer of 100 nm yellow-green Fluosphere beads (ThermoFisher), dried and mounted in glycerol¹⁸. Extended Data Figure 9 and Supplementary Movie 8 show data of a 2% formaldehyde fixed C127 cell immune-labelled with rabbit anti-histone H3K4me3 primary antibodies (Active Motif) and goat-anti rabbit Alexa Fluor 488 secondary antibodies (ThermoFisher), counterstained with DAPI and mounted in Vectashield. Supplementary Movie 9 and Extended Data Figure 10 show live cell data of stably expressing histone H2B-GFP grown in a 35 mm μ Dish with high precision glass bottom (Ibidi) and imaged at 37° and 5% CO₂. A time series was recorded with 2s intervals. For each time point 7 z-sections with z-distance of 0.125 μm were acquired (in total 7x3x5=105 raw images per time point).

Extended Data Figure 2, 3, 8 and Supplementary Movie 7 show data of a bovine pulmonary artery endothelial cell (BPAEC), with mitochondria labelled with MitoTracker Red, actin labeled with Alexa Fluor 488, and DNA labeled with DAPI, and embedded in hardening mounting medium (Thermo Fisher Fluo Cells slide #1).

SIM processing and reconstruction

The data shown in Figure 1, 2, 3, and 5 and Extended Data Figure 1, 2, 4 and 5 pertain to 2D-SIM reconstructions made from a single focal slice of a 3D-SIM acquisition, i.e. made with a three-beam interference illumination pattern. The data shown in Figure 4 and Extended Data Figure 6, 7, 8, 9 and 10 pertain to full 3D-SIM reconstructions. A flow diagram illustrating the different steps in making (2D and 3D) SIM reconstructions is shown in Supplementary Figure 6, and consists of pre-processing steps, illumination pattern estimation and image Fourier order computation steps, and filtering and reconstruction operations.

Pre-processing steps

The set of pre-processing operations starts with a gain and offset calibration for providing image signals that represent the number of detected photo-electrons⁴⁷. The EM-CCD or sCMOS cameras that are used have zero or negligible readout noise so that the image signals follow Poisson statistics to a good approximation. Some effects of fixed pattern noise (pixel-to-pixel variations in offset and gain) are visible in sCMOS based images, but are ignored here for the sake of simplicity. In a future study this could possibly be incorporated by an additional camera calibration step, or by extending the method of ref. 47. Optionally, the images are grouped in sets of five images acquired with the same illumination pattern angle, and registered in an all-to-one manner in order to correct for drift. It turns out that leaving out the step of drift correction does not substantially deteriorate the reconstruction outcomes for the imaged specimens.

The illumination pattern modulation in 3D-SIM can be characterized by the Modulation Contrast to Noise Ratio (MCNR), a quality measure for faithful illumination pattern retrieval, part of the SIMcheck quality control software package¹⁷. The proposed method of computation of the MCNR in ref. 17 involves images acquired at different focal planes, and can therefore not be directly applied to 2D-SIM. To that end we use an alternative way to compute the MCNR that can be computed per pixel/voxel. Starting point is a 1D Fourier Transform (FT) of the five phase step images for each pixel/voxel to the set of photon counts N_j of each pixel (voxel) for the phase steps j = 1,2,…,M_t, resulting in the fit:

$$N_j=A_0+A_1\text{cos}\left(\frac{2\pi j}{M_t}+{\phi }_1\right)+A_2\text{cos}\left(\frac{4\pi j}{M_t}+{\phi }_2\right)$$ (5)

The modulation is taken as twice the root-squared average of the first and second order Fourier coefficients A ₁ and A ₂, and the shot noise level is the square root of the zeroth Fourier coefficient A ₀. This leads to:

$$MCNR=\frac{2\sqrt{{A_1}^2+{A_2}^2}}{\sqrt{A_0}}$$ (6)

The results obtained with the current proposal for computing the MCNR agree well with the results obtained with SIMcheck, although small quantitative differences appear. For example, the lack of averaging Fourier coefficients over focal slices gives a more noisy appearance of the MCNR across the Field Of View (FOV) for low signal acquisitions. The rule-of-thumb for reliable pattern parameter estimation is to have sufficient foreground pixels/voxels with MCNR ≥ 3. The MCNR is actually a SIM reconstruction in itself, which generalizes the original SIM proposal¹ by including the second order Fourier coefficients, and shows some degree of optical sectioning (see Supplementary Figure 1). The peak MCNR averaged over the pattern orientations per focal slice has a maximum as a function of the focal slice (see Supplementary Figure 7). Having a satisfactory MCNR only for a limited range of focus positions may be attributed to not just a limited axial extent of the sample, but also to spherical aberration caused by refractive index mismatch. For typical high-NA immersion microscopes the refractive index of the immersion medium must be controlled at the 10⁻³ level for optimum results¹⁸, and the axial range of images with useful modulation appears to be typically only a few μm. For 2D-SIM processing we take the focal slice with the maximum illumination pattern modulation.

So-called z-wrapping artefacts¹⁸ may arise for datasets with a limited number of focal slices, as e.g. many live cell 3D datasets. The periodic boundary conditions of the FT then perturb the first and final few of the slices of the reconstruction. This can be mitigated by preferably ignoring these, or by only representing the final SIM reconstruction by a Maximum Intensity Projection of the reconstruction stack. Another method to mitigate the impact of z-wrapping artefacts is by adding a number of extra, fictitious, focus layers. The reconstructions for the live cell dataset of Supplementary Movie 9 and Extended Data Figure 10, which is based on just 7 focal slices, are made using 14 extra layers. It is estimated that the required number of extra layers is in the range from 10 to 20. These extra focus layers interpolate linearly between the first and last slice of the focus stack. Further, they are blurred by convolution with a Gaussian kernel to mimic the effect of defocus. The kernel size ranges from one pixel for the layers directly adjacent to the first and last focal slice, to 20 pixels for the layer(s) in the middle of the fictitious additional stack. Finally, artificial shot noise is added for maintaining Poisson statistics.

The next step in pre-processing is to apply a window to the data cube in order to enforce continuity in the periodic boundary conditions assumed in subsequent FTs, i.e. for eliminating streaking along the coordinate axes in the FTs. For the voxel indices j = 1,…,N along any of the three coordinate axes we can define a scaled coordinate r_j = (2j − 1 − N)/(2N), the windowing is applied to the edges defined by 1/2−|r_j |≤b, where we take b in the range 0.1-0.2 along the lateral directions (r = x and r = y) and b = 0 along the axial direction (r = z). Over these boundary region voxels, the window function is taken to be τ_j = sin(π(1−2|r_j|)/(4b))², for the inner voxels τ_j = 1 is taken. The overall window function is the product of the window functions for the three orthogonal coordinate axes. No windowing is applied in the axial direction because it appears to have limited use there. The first and last focal slice typically show no recognizable structure as all object features are drowned by defocus and spherical aberration induced loss of illumination pattern modulation, which implies that the discontinuity arising from the FT periodic boundary conditions in the axial direction is not that harmful. The additional fictitious layers that interpolate between the first and last focal slice, with features that are gradually blurred away, plays the same role for datasets with limited number of focal slices. An additional factor here is that the axial Fourier streak is suppressed anyway by the low-pass filtering step with the 3D-OTF of the microscope, which has the well-known missing cone.

The windowing operation by a simple pointwise multiplication of the image data cube with the window function compromises Poisson statistics of the image signals. This can be overcome by applying the random binomial data splitting approach (see Supplementary Note 2). The image signal for a pixel is written as I = n + ε, where n = round(I). The value of the window function for this pixel τ satisfies 0≤ τ ≤1 and is used as the probability in a binomial probability distribution for each of the integer n photon counts, giving a random total of n′ counts satisfying 0≤ n ′≤n. The remainder ε is reduced by the ratio n′/n. This procedure preserves Poisson statistics across the entire data cube.

The next pre-processing step is up-sampling in order to accommodate the extended cut-off of the SIM OTF (typically by a factor of 2 in the lateral directions, no up-sampling in the axial direction) by zero padding in Fourier space. This operation compromises Poisson statistics, but this can be solved by artificially filling the extra high spatial frequency Fourier pixels, that are created by zero padding, with noise. For each voxel in the up-sampled image with up-sampled image signal n a random variable n′ is generated using the signal n as Poisson rate. The difference n′ − n, the artificially created noise, is Fourier transformed and masked to fill the new Fourier pixels created by zero padding, while keeping the original Fourier pixels obtained from the FT of n unaltered. Inverse FT then gives an up sampled image that follows Poisson statistics.

Illumination pattern parameters and OTF

The illumination pattern parameters (pitch, orientation and phases) are estimated using a 2D-projection of the pre-processed 3D dataset. This projection is the (weighted) sum over all focal slices, where the average MCNR values over each focal slice is taken as weight. This improves SNR by averaging over noise in the individual images and over the 3D-structure of the fluorescent object, but under the assumption that these gains are bigger than possible residual shifts in the illumination patterns between different focal slices, left after possible drift correction. Next, the cross-correlation image matrix for all M_t × M_t image combinations with different pattern phases is computed, zoomed in at regions around integer multiples of the expected Fourier peaks at ${\overrightarrow{q}}_{est}$. The peak in the root mean square of the cross-correlation matrix is used to update the estimate of the pattern spatial frequency vector ${\overrightarrow{q}}_{est}$ (see Supplementary Note 1 for more detail). The precision of peak detection is aided by the zooming capability of the chirp z-transform for evaluating FTs⁴⁹, reaching a relative precision in determining the pitch equal to 6×10^-5 over the K = 10 noise independent acquisitions of the GFP-zyxin dataset and 8×10^-5 over the 15 frames of the live cell histone H2B-GFP dataset. The pattern phases are estimated from the phase of the autocorrelation peaks⁵⁰, reaching a precision of around 1 deg for the GFP-zyxin and for the live cell histone H2B-GFP datasets.

The retrieved values for the pitch, orientations, and phases of the illumination patterns are used to compute the different image Fourier orders (0th, 1st, 2nd) per orientation of the line illumination pattern, and to shift these orders in the lateral direction to the correct location in Fourier space.

The 3D-OTF is obtained from a bead calibration experiment¹⁸, if such data is available. The illumination pattern parameters are used to create laterally shifted copies of the different Fourier orders per orientation of the line illumination pattern. An alternative to the calibration OTF is computation using a vectorial Point Spread Function (PSF) model, taking all effects of high NA and polarization into account (see e.g. ref. 48 and references therein). For 3D datasets, this requires an additional axial separation of the two branches of the 1st Fourier order. The theoretical value k_z = ±n_med (1−cos θ)/λ_ex is used, with λ_ex the excitation wavelength, n_med the medium refractive index, and sin θ = λ_ex/n_medp), where p is the estimated illumination pattern pitch.

Finally, the 1st and 2nd order strengths a ₁ and a ₂ are estimated from the image data itself by requiring consistency across order overlap regions, as the different orders depend on the spatial frequency spectrum of the same underlying fluorescent object (see Supplementary Note 1 for more detail). It turns out that the retrieved order strengths depend on the signal-to-noise ratio and sparsity of the sample, leading to lower estimates for relatively dense samples and/or samples recorded under adverse signal-to-noise conditions. The estimated order strengths should therefore be seen as effective order strengths, not as the true underlying ground truth values. A fixed set of order strengths in the range of values found for the sparse tubulin set of Figure 4 measured at high signal levels (a ₁ = 0.30 and a ₂ = 0.45) is used for all SIM reconstructions.

Reconstruction

The functions ${\widehat{D}}_j$ and ${\widehat{V}}_j$ (defined in the Supplementary Note 2) are computed from the copies of the incoherent (2D or 3D) OTF, shifted in Fourier space, and from the order strengths a ₁ and a ₂. This is sufficient to obtain the regularization filter for flat-noise SIM. For state-of-the-art and true-Wiener SIM we use as apodization filter the trianglex filter Â_j = ${{{\displaystyle \widehat{\Lambda }}}_j}^x$ with ${\widehat{\Lambda }}_j$ the triangular filter (interpolating linearly between 1 at zero spatial frequency to 0 at the extended SIM cutoff) and x = 0.4 a numerical coefficient, because this has also been used in the literature^13,14. The triangular filter with x = 1 gives a visually similar reconstruction as the Lukosz-bound filter²¹, and is more benign for artefacts, such as the structured noise artefact and the z-wrapping artefact that arises for low number of focal slices, than the trianglex-filter with x = 0.4.

An initial pre-Wiener filtered SIM reconstruction is made by low-pass filtering the different image Fourier orders with the corresponding shifted copy of the incoherent OTF, and then adding all contributions weighted with order strengths. Overall Wiener filters ${\widehat{A}}_j/({\widehat{D}}_j+w)$ and $1/({\widehat{D}}_j+{\widehat{w}}_j)=1/\sqrt{{\widehat{V}}_j}$ are applied and a subsequent inverse FT is executed to generate the state-of-the-art and flat-noise SIM reconstructions, respectively.

The pre-Wiener filtered SIM reconstruction is used to make an estimate of the Spectral Signal-to-Noise Ratio (SSNR) needed for the true-Wiener filtered SIM reconstruction. The initial SIM reconstruction $e_k^{\mathit{\text{rec}},i}$ has a spectral power $|{\widehat{e}}_j^{rec,i}{|}^2=|{\widehat{e}}_k^i+\delta {\widehat{e}}_k^i{|}^2=|{\widehat{e}}_k^i{|}^2+2Re\{{\widehat{e}}_k^{i^{*}}\delta {\widehat{e}}_k^i\}+|\delta {\widehat{e}}_k^i{|}^2$. Averaging over rings in spatial frequency space effectively eliminates the signal-noise cross-term. The average $\langle {\widehat{N}}_j^i{\rangle }_{ring}$ of the noise power $|\delta {\widehat{e}}_k^i{|}^2$ is independent of the object spectrum and is proportional to the noise variance function ${\widehat{V}}_j$ (see Equation (1b)). The estimate for the SSNR is then found as:

$$SSN{R}_j\approx \frac{{\langle {\left|{\widehat{e}}_j^{rec,i}\right|}^2\rangle }_{\mathit{\text{ring}}}}{{\langle {\widehat{N}}_j^i\rangle }_{\mathit{\text{ring}}}}-1$$ (7)

which can subsequently be used with Equation (3) to find the regularization filter for the final true-Wiener filtered reconstruction. A minor drawback of ring averaging is that azimuthal variations in SSNR are not accounted for. Averaging schemes based on e.g. Gaussian blurring over regions in spatial frequency space can possibly provide an alternative in case these azimuthal variations become relevant.

The error in the estimated SSNR becomes comparable to the SSNR for the highest spatial frequencies, where there is too little signal (see e.g. Figure 1m). This issue can be solved by extrapolating the regularization filter from low spatial frequencies to high spatial frequencies. The simplest extrapolation is to take the maximum of $\widehat{w}(\overrightarrow{q})$ for spatial frequencies in the region $\langle SSNR(\overrightarrow{q}){\rangle }_{\mathit{\text{ring}}}>SSN{R}_{\mathit{\text{thr}}}$ and use this as a constant regularization in the region $\langle SSNR(\overrightarrow{q}){\rangle }_{\mathit{\text{ring}}}<SSN{R}_{\mathit{\text{thr}}}$. Here, SSNR_thr is a threshold value that can typically be selected from the range between about 1 and 10. A bit more complex is a quadratic extrapolation $\widehat{w}(\overrightarrow{q})=\alpha |\overrightarrow{q}{|}^2,$ where the parameter α is estimated from the spatial frequency region defined by $\langle SSNR(\overrightarrow{q}){\rangle }_{\mathit{\text{ring}}}>SSN{R}_{\mathit{\text{thr}}}$. This corresponds to a regularization function in real space α|▽e|², with ▽ the 2D or 3D-gradient operator²¹. A more general power-law extrapolation $\widehat{w}(\overrightarrow{q})=\alpha |\overrightarrow{q}{|}^{\beta }$ works fine in many cases as well, where now both parameters α and β are estimated from the spatial frequency region defined by $\langle SSNR(\overrightarrow{q}){\rangle }_{\mathit{\text{ring}}}>SSN{R}_{\mathit{\text{thr}}}$. Typical values found for the power-law exponent β are in the range 1.2 < β < 2.7. For the sake of simplicity we have used the quadratic extrapolation scheme with SSNR_thr = 5 for all datasets.

Notch filtering of the different image Fourier orders has been applied to improve the optical sectioning in SIM, mostly for 2D-SIM reconstructions^14,16. The retrieved orders (prior to lateral shifting in Fourier space) are multiplied with filter kernels as defined in Equation (3.21), (3.22), and (3.23) in Supplementary Note 3. The notch depths α ₀, α ₁, α ₂, the lateral notch width △q _‖, and, for 3D-SIM, the axial notch width △q _⊥, appearing in the filters are in principle user-adjustable parameters. Instead of an ad-hoc choice, we fix the parameter values to optimize the contrast of flat-noise SIM by making the flat-noise OTF ${\widehat{g}}_j^{FN-SIM}={\widehat{D}}_j/\sqrt{{\widehat{V}}_j}$ as close as possible to a target OTF, which we take to be equal to the Lukosz-bound apodization function. A suitable OTF error function for this optimization is defined in Equation (3.26) in Supplementary Note 3. Reasonably good results can be obtained for notch widths that scale with the cut-off frequency △q _‖ = 2 ρ NA/λ_ex, and, for 3D-SIM $\Delta q_{\perp }=\rho \left(n_{med}-\sqrt{{n_{med}}^2-N{A}^2}\right)/{\lambda }_{ex}$, where we take the numerical pre-factor ρ = 1.25. For 2D-SIM it is sufficient to apply the notch filtering to the zeroth order only, i.e. α ₁ = α ₂ =0. The remaining non-zero notch depth α ₀ = 1 − 10^−d₀ is determined using Matlab’s fminbnd to find the optimum value of the notch dip exponent d ₀. The procedure converges within about 10 iterations with a precision of around 10^-3. Typically this results in values in the range 1.5 ≤ d ₀ ≤ 3. The final notch filtered images do not depend hugely on the initial choice for the parameter ρ, generally a value ρ > 0.75 will suffice, but small differences can arise between different datasets. The task of optimizing the flat-noise OTF is more complex for 3D-SIM, because the requirement on the axial transfer function comes on top of the requirement on the lateral transfer function, and because the native flat-noise OTF has rather pronounced peaks at the centre spatial frequencies of the contributing orders. It turns out that now applying a notch filter to all contributing orders is necessary, especially as this appears beneficial for diminishing the susceptibility for hexagonal background imprint artefacts. For the sake of simplicity we take α ₀ = α ₁ = α ₂ = 1 − 10^−d_n, and again use Matlab’s fminbnd to find the optimum value of the notch dip exponent d_n. Typically, this results in values in the range 3.5 ≤ d _n ≤ 5.5. It is quite conceivable that more sophisticated designs for the (notch) filters could improve the current results.

Noise assessment

The noise model is validated by the spectral noise variance and SSNR that are obtained from the K = 10 noise independent acquisitions of the GFP-zyxin dataset by computing the unbiased sample variance over the K reconstructions. The spectral noise variance for the widefield reconstruction obtained by summing over the M_r rotations and M_t translations appears to be constant across the spatial frequency spectrum with variations up to several percent (see Figure 1i). The small peak at low spatial frequencies is attributed to residual effects of photo-bleaching, illumination variations, and drift. This small peak gives rise to satellite peaks in the experimental spectral noise variance for the SIM reconstructions at the centre spatial frequencies of the orders, that do not correspond to actual noise enhancement. FRC curves are computed²⁵ for all K(K − 1)/2 pairs of reconstructions, the mean and standard deviation over all these reconstruction pairs are plotted in all FRC-results. FRC-curves need no bleaching correction, as they are independent of overall intensity variations of the two input images. The model independent noise assessment via the random binomial data splitting method is described in Supplementary Note 2.

Noise fraction map

The noise model enables the assessment of the fraction Z_k of the SIM reconstruction that is due to noise, based on the average signal and noise level in a neighbourhood around each pixel k (see Supplementary Note 2). We have implemented the computation of this noise fraction by Gaussian smoothing with a width λ/2NA, which corresponds to about four SIM pixels. For smaller kernel sizes there is insufficient noise averaging, for larger kernel sizes the noise fraction values of the background and of the foreground features is blurred too much. It is expected that 0 ≤ Z_k ≤ 1, but values higher than one can arise due to incomplete averaging of noise in the pixel neighbourhood.

Deconvolution and denoising

Starting point for the 2D Richardson-Lucy (RL) deconvolution is a gain recalibration by fitting a straight line through the mean vs. variance curve obtained from the K noise independent reconstructions. This corrects for any possible changes to intensity level during the reconstruction and so ensures the best approximation to Poisson statistics for the signal at each pixel. The reason for this step is that RL-deconvolution can be framed as an object retrieval procedure, based on maximizing a likelihood function that is determined by Poisson statistics²⁷. Suitable initial estimates of the non-linear iterative algorithm are either the recorded widefield image (RL-widefield) or the flat-noise SIM reconstruction (RL-SIM). The OTF used in the RL deconvolution algorithm is either the incoherent OTF (RL-widefield) or the flat-noise SIM OTF (RL-SIM). The iterative procedure is stopped when the error $({\displaystyle {\sum }_k{(e_k^{n+1}-e_k^n)}^2})/({\displaystyle {\sum }_k{(e_k^n)}^2})$, with $e_k^n$ the n-th estimate, is less than 10⁻⁵. The resulting RL-SIM deconvolution has been compared to the joint RL deconvolution^51,52 for this dataset, which gave visually the same outcome. For that reason joint RL results are not shown.

The publicly available code for deep learning based deconvolution²⁹ has been applied with no modification to up sampled widefield representations. The up sampling has been performed by zero padding in Fourier space and subsequent filling the added zeros with noise components, just as done for the SIM pre-processing. The publicly available code for Hessian SIM³¹ is applied in the denoise mode with the recommended parameter settings.

Image data visualisation

All images are rendered with full dynamic range, i.e with no clipping whatsoever. The 3D live cell dataset of histone H2B-GFP is represented in Supplementary Movie 9 by a Maximum Intensity Projection, in view of the limited number (7) of focal slices.

Extended Data

Extended Data Figure 1. Noise-controlled SIM reconstructions of GFP-zyxin protein in focal adhesions (green) and noise fraction map (magenta) over full FOV.

(a-d) State-of-art SIM (w = 5 × 10⁻⁴), true-Wiener SIM, flat-noise SIM, and notch-filtered SIM reconstructions. Contours of the noise fraction map are added in white with contour level indicated. In all reconstructions the noise fraction is lowest in the foreground features and highest in the background region outside the cell. Overall, flat-noise SIM and true-Wiener SIM offer the lowest, and notch-filtered SIM the highest noise enhancement. Scale bar 5 μm.

Extended Data Figure 2. Multi-colour noise-controlled 2D SIM reconstructions.

(a) Combined widefield, true-Wiener SIM, flat-noise SIM, and notch-filtered SIM reconstructions of a fluorescent test slide of a bovine pulmonary artery endothelial cell (red channel: mitochondria labeled with MitoTracker Red, green channel: actin labelled with Alexa Fluor 488, blue channel: DNA labeled with DAPI). Note that due to embedding in hardening mounting medium, cells are flattened and 3D nuclear morphology is compromised. (b-f) Insets of the DAPI-channel comparing state-of-the-art SIM with clear noise amplification artefact to the noise-controlled SIM reconstructions. The SSNR in the DAPI channel is low in this example case, due to reduced signal intensity and compromised morphology. The low SSNR is properly taken into account by the noise-controlled SIM reconstructions, without introducing artefacts, but not by the state-of-the-art SIM reconstruction. Scale bar (a) 10 μm, scale bar (b-f) 5 μm.

Extended Data Figure 3. Noise propagation in DMD-SIM.

(a) Reconstructions of Alexa Fluor 488 labelled actin filaments in a bovine pulmonary artery endothelial cell with the iterative pattern-illuminated Fourier Ptychography (piFP) algorithm (see Supplementary Note 1) and with a band-pass regularization approach for flat-noise SIM. (b) Comparison of flat-noise SIM to a widefield reconstruction obtained by summing the whole set of acquired images. (c-e) Insets of the boxed region in (a) and (b). Both piFP and flat-noise SIM offer a resolution improvement, but piFP has better contrast than flat-noise SIM. The piFP reconstruction shows corrugated line structures and punctuated features (upper right of insets), similar to the structured noise artefact in state-of-the-art SIM with line illumination patterns, flat-noise SIM shows this to a lesser degree. Scale bar (a,b) 10 μm, scale bar (c-e) 4 μm.

Extended Data Figure 4. Flat-noise SIM provides better visibility of high spatial frequency structures.

(a-d) Widefield, true-Wiener, flat-noise, and notch-filtered SIM reconstructions of a nano-fabricated test structure of lines with 140 nm pitch. The line pattern is just visible in flat-noise, and notch-filtered SIM but overshadowed by the noise pattern with uneven distribution of noise over spatial frequencies in true-Wiener SIM. Scale bar 1 μm.

Extended Data Figure 5. Noise-controlled 2D SIM of synaptonemal complex.

(a-d) Widefield, and true-Wiener, flat-noise and notch-filtered SIM reconstructions of the mCherry-CSYCP3 protein in the synaptonemal complex. (e-h) Line profiles along the lines indicated in (b). The SIM reconstructions reveal the two cable sub-structure with a line distance of around 200 nm, flat-noise SIM has less contrast but shows smoother lines and no background noise structure. Scale bar 3 μm.

Extended Data Figure 6. Cross-sections in xy and yz-planes of 3D-reconstructions of tubulin (green) and noise fraction maps (magenta) for different camera exposure times.

(a) Widefield, (b-d) state-of-the-art SIM for low, medium and high regularization, (e) true-Wiener SIM, (f) flat-noise SIM, and (g) notch-filtered SIM. The dashed lines in (a) indicate the location of the xz and xy slices. Scale bar 3 μm.

Extended Data Figure 7. Widefield and noise-controlled 3D-SIM reconstructions of a 100 nm bead layer sample.

(a,b) Widefield, (c,d) true-Wiener SIM, (e,f) flat-noise SIM, (g,h) notch-filtered SIM. The white box in (c) indicates the insets (b,d,f,h). (i,j) SSNR of the SIM reconstructions without (i) and with (j) notch filtering. The data is averaged over rings in Fourier space and the plot is on a logarithmic scale according to log₁₀(1 + SSNR). The red line indicates the (ring averaged) support of the SIM-OTF, the white line indicates the SSNR=5 region in Fourier space used for the extrapolation of the true-Wiener regularization filter. (k) FRC curves for SIM obtained from 4 repeated acquisitions of the bead layer sample. The FRC resolution is 106.3±0.5 nm, very close to the extended SIM diffraction limit 1/(2NA/λ + 2/p) = 99 nm for the estimated pattern pitch p = 416 nm, consistent with the relatively high signal level (peak pixel intensities above 10⁴ detected photons) and the broad support of SSNR above one in spatial frequency space. Scale bar (a,c,e,g) 3 μm, scale bar (b,d,f,h) 1 μm.

Extended Data Figure 8. Widefield and 3D noise-controlled SIM reconstructions of a bovine pulmonary artery endothelial cell.

(a) Widefield, (b) true-Wiener SIM, (c) flat-noise SIM, (d) notch-filtered SIM (BPAEC, red channel: mitochondria labelled with Alexa Fluor 594, green channel: actin labelled with FITC, blue channel: DNA labelled with DAPI). Scale bar 5 μm.

Extended Data Figure 9. Widefield and 3D noise-controlled SIM reconstructions of a mouse C127 cell.

(a) Widefield, (b) true-Wiener SIM, (c) flat-noise SIM, (d) notch-filtered SIM (magenta channel: DNA labelled with DAPI, green channel: H3K4me3 labelled with Alexa Fluor 488, blue channel: DNA labelled with DAPI). Scale bar 5 μm.

Extended Data Figure 10.

Widefield, state-of-the-art SIM and noise-controlled 3D-SIM reconstructions for one time frame of the 15 time-frame, 7-layer dataset of H2B-GFP histone in a live HeLa cell. Scale bar 6 μm.

Code availability

Matlab code is available at https://github.com/qnano/simnoise. ImageJ code for 2D-SIM is available at https://github.com/fairSIM.

Data availability

Data is available at https://doi.org/10.4121/12942932.