Abstract
Structured illumination microscopy (SIM) provides images of fluorescent objects at an enhanced resolution greater than that of conventional epifluorescence wide-field microscopy. Initially demonstrated in 1999 to enhance the lateral resolution twofold, it has since been extended to enhance axial resolution twofold (2008), applied to live-cell imaging (2009) and combined with myriad other techniques, including interferometric detection (2008), confocal microscopy (2010) and light sheet illumination (2012). Despite these impressive developments, SIM remains, perhaps, the most poorly understood ‘super-resolution’ method. In this article, we provide answers to the 13 questions regarding SIM proposed by Prakash et al. along with answers to a further three questions. After providing a general overview of the technique and its developments, we explain why SIM as normally used is still diffraction-limited. We then highlight the necessity for a non-polynomial, and not just nonlinear, response to the illuminating light in order to make SIM a true, diffraction-unlimited, super-resolution technique. In addition, we present a derivation of a real-space SIM reconstruction approach that can be used to process conventional SIM and image scanning microscopy (ISM) data and extended to process data with quasi-arbitrary illumination patterns. Finally, we provide a simple bibliometric analysis of SIM development over the past two decades and provide a short outlook on potential future work.
This article is part of the Theo Murphy meeting issue ‘Super-resolution structured illumination microscopy (part 2)’.
Keywords: structured illumination, super-resolution, extended-resolution, fluorescence
1. Introduction
Structured illumination microscopy (SIM) is a method in fluorescence microscopy which is capable of providing resolutions better than those that can be achieved using uniform illumination in epifluorescence wide-field microscopy. While first demonstrated, at the turn of the millennium, to enhance only lateral resolution [1–4], subsequent developments allow for the enhancement of axial resolution as well [5,6]. SIM has become a popular technique, particularly for live cell imaging [7–9], when better resolution is required as, unlike super-resolution methods such as single molecule localization microscopy (SMLM) [10–14] and stimulated emission depletion microscopy (STED) [15–17], it does not require high laser powers nor special fluorophores. However, as will be discussed further later, in its normal form it only provides a roughly twofold enhancement of resolution.
It should be noted that, in 1997, an alternative form of structured illumination microscopy was presented by Neil et al. [18]. Here the goal was not to improve resolution, but to add optical sectioning to wide-field microscopy. This form of structured illumination has been commercially successful, with further developments of the technology available from Aurox (Clarity), Andor (Revolution DSD2) and Zeiss (Apotome). For the purposes of this article, SIM will be taken to correspond only to those methods designed to improve resolution (often referred to as SR-SIM, as opposed to the OS-SIM of Neil et al.).
In this article, we aim to provide a general overview of SIM and answer the 13 questions recently proposed by Prakash et al. [19] (§§1–13), along with three more of our own invention (§§14–16). A similar attempt at answering the 13 questions of Prakash et al. has recently been presented by Heintzmann [20]. For some questions, the answers given by Heintzmann are significantly different from those presented here: where this is the case, this will be highlighted in later sections. In §2, we consider the so-called ‘diffraction limit’ and show that SIM as described here does not break it, even though it provides resolutions beyond those achievable with conventional wide-field imaging. Following this, in §3, we consider whether an impressive demonstration of SIM combined with total internal reflection fluorescence (TIRF) microscopy, which achieves a resolution of 84 nm, breaks the diffraction limit. We then look at the conditions required for SIM to break the diffraction limit in §§4 and 5.
In the second part of the manuscript, we show that, despite their prevalence, Fourier methods are not necessarily required to reconstruct images (§6) and study the challenges of using SIM in thick samples (§§7 and 8). We then argue that image scanning microscopy (ISM), while experimentally a different method, is a form of SIM and show how arbitrary illumination patterns can be used (§9). Following this, we study the effects of illumination sparsity in §10. We note that SIM cannot be used to improve the resolution of transmission microscopy (§11) and propose a careful approach to the use of machine learning with super-resolution (§12).
In the last part of the manuscript, we discuss the practicalities of building a SIM system (§13) and consider how to combine SIM with SMLM (§14). We then investigate how the axial resolution can be improved beyond a twofold limit without a nonlinear response (§15), how to combine SIM with light sheet microscopy (§16) and how quickly SIM images can be acquired (§17). Finally, we use a simple bibliometric analysis to track interest in SIM since its inception and conclude with an outlook on future developments (§18).
2. What is super-resolution microscopy and should diffraction-limited linear SIM be classed as ‘super-resolution’?
Super-resolution microscopy refers to a set of techniques that can be used to break ‘the diffraction barrier’, a seemingly unsurmountable goal rewarded with the 2014 Nobel Prize in Chemistry. The trick in this question is deciding exactly what the diffraction barrier is…
There are a number of resolution metrics in use, such as the Rayleigh and Sparrow criteria, that derive from trying to answer the question of how closely two stars can appear while still being distinct in naked-eye astronomical observations [21,22]. These criteria are, however, heuristic and have no fundamental physical basis. Ernst Abbe was the first to propose a physically motivated limit to the resolution of a light microscope by studying diffraction from a grating [23]. He considered using a microscope to image a grating using transmitted light and noted that, in order for the image to have any periodicity, at least two of the diffracted orders from the grating must be captured by the objective. For head-on illumination, i.e. normal to the grating, he noted that three orders (the , 0 and orders) were captured, giving a minimum grating periodicity of:
2.1 |
where is the vacuum wavelength, the refractive index of the sample and the semiaperture angle (i.e. the maximum angle from the objective’s optical axis from which orders can be collected). For convenience, the product is known as the numerical aperture (NA).
Referring to figure 1a (top row), we see that these three orders are located symmetrically about the centre of the back focal plane. By tilting the illumination, we can shift the location of these orders laterally. Given that we only need two orders, not three, for a grating-like structure to be seen, we can imagine that by tilting the illumination such that the zeroth order is right at the edge of the back focal plane, we could increase the order spacing by a factor of two and hence half the grating period (figure 1a, bottom row). This leads to the Abbe equation for oblique illumination:
2.2 |
What about fluorescence imaging, where light from the sample is emitted in all directions? Abbe never studied the case of fluorescence imaging, but it turns out that the right-hand side of Abbe’s equation for oblique illumination is correct for this case as well:
2.3 |
More specifically, this is the lateral resolution, i.e. that within the focal plane. Perpendicular to the focal plane, the axial resolution is given by
2.4 |
Why are these equations correct? To explain this, we will follow an argument similar to that of Gustafsson [24], which in turn builds on arguments by McCutchen [25] and Ewald [26]. First, we must note that we will use the scalar approximation of the electric field, although the argument can be generalized by considering each component of the vectorial electric field separately. We will also work in the far-field, as near-field components decay exponentially quickly and will not be detected by a normal microscope. In addition, fluorescence is an incoherent process, with emission from a source bearing no phase relationship to the stimulating light or emission from another source (we ignore the small probability of stimulated emission). As such, we can limit our consideration to just one source, as the overall response from the sample is the incoherent sum of the responses from all the sources present.
Consider the amplitude of the electric field, , produced by a source radiating monochromatically. The source produces a spherical wave which, in the far field, can be decomposed, via a Fourier transform, into a sum of plane waves with different directions. All these plane waves will have the same wavelength, , and so all the wavevectors, , will have the same length. Hence, as there are no preferred directions and all wavevectors have the same length, the Fourier transform of , (we will use the notation that the Fourier transform of is written as throughout), will have the form of a spherical shell, centered on the origin and with radius (figure 1b). This detected far field is very much not the same as the original emitted field, as the evanescent components have been lost. The consequences of this can most clearly be seen by noting that a true spherical wave would decay smoothly with increasing distance from the origin, while this field has local minima and maxima all over space.
However, our image is not given by the amplitude of the electric field, but the intensity. This can be found by multiplying the amplitude by its complex conjugate, giving . This means that in the Fourier domain, we have
2.5 |
where denotes convolution. Hence, is the autocorrelation of and so will have the form of a ball with twice the radius of the sphere of , (figure 1c).
This is interesting, as it shows that there is a limit to the information carried by the far field, irrespective of the objective lens used: only spatial frequencies up to are present. Converting back to real space, this critical spatial frequency corresponds to a spatial scale of . If we consider that our source may be located in a medium of refractive index , then we can convert to a vacuum wavelength, , and so obtain the far-field limit:
2.6 |
Now, let us look at what happens if we use an objective lens to capture the light from our source. The objective can only capture wavevectors over a certain angular range, meaning that the Fourier domain electric field amplitude distribution has the form of a spherical cap, rather than a full sphere (figure 1d). As such, the autocorrelation of this has the form of a toroidal solid, shown in cross-section in figure 1e.
We can identify that the maximum lateral spatial frequency is given by , while the maximum axial spatial frequency is given by , where is the semiaperture angle. Converting to real space, this gives:
2.7 |
and
2.8 |
as was quoted previously.
Considering not just a single source, but a distribution of sources emitting light given by a distribution , the toroidal solid formed by autocorrelating the spherical cap defines a transfer function, . Given this, the image spectrum, , of the emission spectrum is given by
2.9 |
where denotes point-wise multiplication. In real space
2.10 |
where is known as the point spread function.
Equation (2.9) makes clear that the image spectrum only contains spatial frequencies for which is non-zero. As the form of is based only on the phenomenon of far-field diffraction, this defines a true diffraction limit based on the spatial frequency bandwidth of .
Now we are in a position to consider the mechanism of structured illumination and see how it circumvents this limit. For simplicity, we will restrict our consideration to one dimension and replace with .
The fundamental basis of SIM is the realization that, in our image formation equation, equation (2.10), we are not truly interested in the emission distribution function, , but instead the distribution of sources . These distributions are related by , where is the illumination distribution. In the case of uniform illumination, merely acts as a scalar multiplier of . However, for non-uniform illumination we have an updated image formation equation:
2.11 |
If we consider this image formation equation in the Fourier domain, then we have
2.12 |
This suggests that, while the maximum spatial frequency that can be recorded is limited by the support of , a careful choice of may let us shift the sample spectrum, , around via the convolution such that other spatial frequencies are transmitted.
SIM, in its simplest variant, achieves this by using illumination with two coherent beams, providing a pattern of the form
2.13 |
or, in the Fourier domain,
2.14 |
where denotes the Dirac delta function. Hence, for a given phase, , we have
2.15 |
This shows that we have the usual wide-field component, , overlaid with two shifted copies of the sample spectrum. This alone is insufficient to ‘see’ , but by recording multiple images with different phases of the illumination pattern, these components can be linearly separated and used to reconstruct an image with improved resolution. As there are three components to separate, at least three phase shifts are required.
For the purposes of this article, we shall not discuss the details of the image reconstruction routine, nor how SIM can be used to enhance the resolution in all directions (i.e. both laterally and axially). Instead, the interested reader is directed to the excellent article by Gustafsson et al., which provides detail on both these issues [5]. Further information on the individual steps of reconstruction is presented by Lahrberg et al. [27] (pattern estimation), Wicker [28] (phase estimation) and Smith et al. [29] (noise-appropriate recombination). An overview of the entire reconstruction routine, as well as open-source software, is presented by Lal et al. [30]. ImageJ-compatible software for checking the quality of raw and reconstructed data are provided by SIMcheck [31] and for reconstruction by fairSIM [32].
It is clear from equation (2.15) that higher resolutions are achieved with finer illumination periods. If we illuminate the sample with the same objective lens that we use to capture fluorescence, then the finest illumination we can create is
2.16 |
Combining this with the emission diffraction limit, we see that the total resolution is given by
2.17 |
and so we have
2.18 |
where . Hence, for the case where , we see that SIM provides double the resolution of uniform (epifluorescence) illumination.
While this is beyond the resolution of diffraction-limited imaging with uniform illumination, the resolution is still limited, in some sense, by diffraction. In order to not be diffraction-limited, we would need to be able to create patterns with arbitrarily fine periods. Extensions of SIM that provide this will be discussed in §4. These methods, theoretically, create images that do not have a spatial band-limit. To distinguish SIM, and other techniques, that provide improved resolution but are limited by diffraction in both illumination and emission, rather than just in emission, the term ‘extended resolution’ has been proposed [33–36].
This allows us to classify methods into four distinct groups based on their theoretical resolving power:
-
(i)
Standard resolution—as achieved by a normal wide-field epifluorescence microscope.
-
(ii)
Extended resolution—beyond standard resolution but still limited by the combination of the illumination and emission diffraction limits, as achieved by SIM and ISM (also referred to by some authors as restricted super-resolution).
-
(iii)
Enhanced resolution—beyond extended resolution, including methods with a finite resolution limit, such as photoactivated nonlinear SIM.
-
(iv)
Super resolution—the subset of enhanced resolution methods not limited by diffraction and theoretically capable of infinite resolution, discussed further in §4, such as stimulated emission depletion microscopy and saturated SIM.
Note that some authors, including Heintzmann in his answers to these questions [20], would classify all methods capable of imaging beyond the extended resolution limit as super-resolution methods, but this is not a sufficient condition for ‘breaking’ the diffraction barrier [37]. Furthermore, it is worth highlighting the difference between ‘imaging at the diffraction limit’, i.e. with standard resolution, and ‘diffraction-limited resolution’, which may be at any of the levels apart from diffraction-unlimited super resolution.
3. Should high-NA TIRF-SIM, which can achieve lateral resolutions down to 84 nm, be considered as diffraction limited?
Total internal reflection fluorescence (TIRF) microscopy exploits the nature of total internal reflection to produce a short-range, evanescent illuminating field tethered to a refractive index boundary. Given two materials with different refractive indices, , Snell’s Law states that light incident on the boundary, from the side, at angles equal to or greater than cannot be transmitted and is instead reflected. However, Maxwell’s equations demand continuity at the boundary and so a rapidly decaying evanescent field is created on the side.
For light incident at an angle , the exponential decay length of the evanescent field is given by
3.1 |
and is typically of the order of 100 nm. While this ensures that only a thin section of the sample is illuminated, it does not enhance axial resolution as such as the illumination cannot be scanned over the sample. TIRF-SIM uses two such high-angle beams to (i) reduce background fluorescence, (ii) provide finer patterns than can be created directly in the medium [38].
When imaging a watery sample, for which , the maximum possible numerical aperture of a water-immersion objective is (i.e. ), while oil immersion objectives with numerical apertures of 1.4, 1.49, 1.5 and even 1.7 are available. These objectives can form a pattern right at the coverslip boundary at the limit of their numerical apertures and have the near-field be patterned with the same spatial period, even though there is no way of creating such a fine pattern in water using far-field illumination.
Li et al. used such a 1.7-NA objective with between of 488 nm illumination at 1.58 NA and approximately 510 nm detection and claim a resolution of 84 nm [36]. Using a modified version of equation (2.17), where we allow the numerical apertures of illumination and detection to be different, we can back-calculate the effective detection NA for their claim to be 1.38, consistent with the high end of the range of reported cytosolic refractive indices [39]. The corresponding non-SIM TIRF resolution for this case would then be worse, at 185 nm, while the maximum possible far-field SIM resolution would be 90 nm. Nevertheless, this 84 nm is still diffraction-limited, as the period of the pattern created at the boundary is still limited by diffraction on the side. It is interesting to note that the authors rejected the possibility of collecting super-critical fluorescence (i.e. near-field emission that is converted to propagating light by interacting with the boundary) to further enhance their resolution calculation (i.e. the detection NA is limited to the refractive index of the sample), despite other measurements with the same objective showing that significant super-critical light can be present [40].
It is particularly instructive to compare this answer (yes) with the companion given by Heintzmann: ‘a clear “no”’ [20]. For Heintzmann, taking equation (2.17), all forms of SIM break the emission Abbe limit and hence break the diffraction limit. This equation can be used to define a new limit, that of ‘restricted super resolution’, introduced by Sheppard [41]. For TIRF-SIM, the structured illumination pattern has a finer frequency than that permitted by the refractive index of the sample, and so Heintzmann says that TIRF-SIM breaks this ‘restricted super resolution’ criterion. Hence, the suggestion is that TIRF-SIM is not diffraction-limited even under the considerations of this more strict approach.
However, the fact remains that the fineness of the illumination pattern is limited by diffraction, just in a medium of refractive index rather than . Even in the situation where the sample was mounted on a diamond coverslip (ignoring the deleterious effects of the strong dispersion in such a material) and we had an objective lens with numerical aperture matching that of the refractive index of the material, 2.4, TIRF-SIM of the same sample as Li et al. used could not produce a resolution better than
3.2 |
Hence, this author would class this as extended resolution, not super resolution or even enhanced resolution.
4. Can nonlinear SIM become broadly applicable and live-cell compatible?
Nonlinear SIM breaks the assumption that the fluorescent response from the sample is linearly related to the intensity of the illuminating light (). The nonlinearity adds extra harmonics (i.e. spatial frequency components above that of the linear case) to the effective illumination pattern, further extending the resolution. A patent application (DE19908883A1) describing this idea was submitted by Heintzmann and Cremer in March 1999, barely two months after the publication of the first SIM paper. The simplest case to consider is a normal SIM system in which a further sinusoidal pattern, , is used to control the fluorescence response, e.g. by photoswitching a fraction of the molecules between dark and light states. Now, the camera data acquired have the form:
4.1 |
and in the Fourier domain:
4.2 |
Assuming that and have the same period, , we now have a total of five delta functions, at (see figure 2b middle row). By phase-shifting both and simultaneously, we can unmix these five components with only five images, producing a reconstruction with sample information at three times the conventional limit.
Despite the further resolution increase, even this approach is diffraction-limited, as and are both individually diffraction-limited. Hence, the spatial frequencies available through this method are limited to a certain band. In effect, all we have done is taken the normal SIM equation (equation (2.11)) and replaced with . We could imagine hypothetically extending the scheme even further, with yet another sinusoidal pattern controlling the response of the sample to the light which controls the fluorescence response, but this would correspond to and so, while providing higher resolution (to four times the usual limit), would still be diffraction-limited. Hence, while these methods would provide enhanced resolution beyond extended resolution, they do not belong to the class of diffraction-unlimited super-resolution methods.
In order to achieve true super-resolution imaging with SIM, we need not just a nonlinear response, but a non-polynomial one. This is because a polynomial response can be considered as a (scaled, repeated) convolution of the illumination spectrum with itself and so, as we start with a finite number of harmonics, we must end with a larger, but still finite, number. By contrast, a non-polynomial response would provide an infinite number of harmonics and hence cannot be said to be limited by diffraction (see figure 2a,b bottom rows).
To recap, a uniform illumination pattern provides standard resolution at the diffraction limit. Patterned illumination with a linear fluorescence response to illumination light provides extended resolution up to twice the diffraction limit (assuming excitation and emission light have the same wavelength). Replacing the linear response with a nonlinear, polynomial response provides further enhanced resolution, albeit one that is still bandlimited. Finally, implementing a non-polynomial response allows for diffraction-unlimited super-resolution imaging.
The first non-polynomial SIM method to be proposed and demonstrated involves saturating the fluorescence, such that a linear increase in illumination intensity results in a sub-linear increase in fluorescence [42] (compare the top and bottom rows of figure 2a). Saturation occurs when a significant fraction of the illuminated molecules are in the excited state when further excitation photons are incident, as there are insufficient ground-state molecules that can be excited by the photon flux. So far, the only demonstration of saturated SIM was provided by Gustafsson in 2005 [43], which showed the ability to resolve a close-packed monolayer of 50 nm-diameter fluorescent beads. Here, pulsed illumination was used to provide an , where is the fluorescence excitation cross-section of the fluorophore used, (where is the fluorescence lifetime) and is the pulse duration.
While, theoretically, an infinite number of harmonics were present, the signal-to-noise ratio limited the number of ‘usable’ harmonics to three, i.e. a fivefold resolution enhancement, although five harmonics were present in calibration data. This is a good illustration of the point that, while super-resolution methods may theoretically be capable of providing superior resolution to enhanced-resolution methods, in practice the distinction is often irrelevant.
Given that the average illumination irradiance was only , one may wonder why this method has never been demonstrated on biological samples. This is because considering only the average irradiance is misleading, as the illumination was pulsed. Averaging instead over the pulse duration gives a peak irradiance of , comparable to the levels used for multiphoton microscopy. Photodamage mechanisms are known to be nonlinear with respect to light intensity, such that a doubling in intensity produces more than double the damage. Nevertheless, it is not clear that this irradiance is necessarily too damaging for imaging biological samples, especially as the saturation irradiance of EGFP has been measured to be similar, at [44], and multiphoton microscopy has successfully been used to image many varied biological samples. While multiphoton microscopy uses redder wavelengths of illumination, which is suggested to contribute less to photodamage, standard use of confocal fluorescence microscopy also often operates at illumination irradiances close to the saturation irradiances of similar fluorophores.
Another non-polynomial SIM approach, published in 2012 by Rego et al., involved saturated depletion of fluorescence, in which the photoswitchable nature of the fluorescent protein Dronpa was used to confine fluorescence to thin bands [45]. This followed a 2008 demonstration of the same mechanism by Hirvonen et al., although here the authors were unable to successfully reconstruct a high-resolution image [46]. The idea of saturating the depletion of fluorescence is one example of the RESOLFT concept [37], which was demonstrated in a parallelized, wide-field manner in 2007 [47]. In the work of Rego et al., images with resolutions better than 50 nm were produced despite the required irradiances being of the order of only . Unfortunately, only fixed cells were imaged as one reconstructed frame required 945 s of imaging time.
In order to develop a live-cell compatible method, Li et al. abandoned the use of non-polynomial SIM and instead focused on developing a technique that would work without the need for saturation [36]. To this end, they used the photoswitchable fluorescent protein Skylan-NS for patterned activation, rather than depletion. This enabled the timelapse imaging of live cells at resolutions better than the 84 nm of their high-NA linear SIM approach (see §3), producing a movie of 30 frames at 65 nm resolution. Furthermore, by increasing the activation irradiance to , they began to saturate the transition, creating a non-polynomial method with an extra harmonic above the noise floor. This increased the resolution to 45 nm, but only provided 12 frames at useful signal-to-noise ratios.
Hence, both nonlinear and non-polynomial SIM have been demonstrated to be live-cell compatible. However, while linear SIM can image live samples for hundreds of time points, nonlinear approaches are currently limited to a few tens of time points. In addition, while non-polynomial methods theoretically provide unlimited resolution, signal-to-noise concerns show that they are, in practice, less well-suited to extending resolution than carefully chosen nonlinear methods such as the patterned activation of Li et al. As an example of this, compare the ‘quadratic’ and ‘saturated’ plots in figure 2b. While the saturated case has more harmonics in total, the strengths of those at and are much stronger in the quadratic case. Given a finite photon budget, more photons must be spent to raise these harmonics above the spectrally flat noise floor using saturation. This problem is exacerbated at higher illumination powers as, while the strength of higher harmonics increase, the strength of the first harmonic decreases (see electronic supplementary material, video S1).
Numerical simulations suggest that saturation using continuous wave illumination, for which an illumination of is mapped to an effective excitation of , produces harmonics that decay geometrically beyond the first harmonic, with the strength of the first harmonic decaying with increased saturation strength. For pulsed illumination, a similar pattern of harmonics is seen, with a short pulse length mimicking the effect of a higher intensity. For polynomial illumination response with harmonics (i.e. peaks in the spectrum) and perfect modulation contrast, the strength of the th harmonic is given by
4.3 |
where would correspond to a harmonic as strong as the DC peak (). These strengths can be read off from every other row in Pascal’s triangle (dividing by the central element), with uniform illumination corresponding to the first row.
5. Do you need ‘switching’ of states for nonlinear super-resolution imaging?
So far, all of our discussions of nonlinear SIM and super-resolution imaging have used continuous models. Despite this, we know that light is quantized into photons and fluorescent samples are quantized into fluorophores. As such, all the approaches discussed in §4 feature ‘switching’ of states, but this is also true of normal fluorescence imaging. Hence, switching is not a sufficient condition for nonlinear super-resolution microscopy. While switching of states can provide useful nonlinearities, it alone is not sufficient to encompass all possible nonlinearities (e.g. Rabi oscillations [48]), merely those that have been successfully demonstrated to work with SIM so far. Hence, switching is theoretically not a necessary condition for nonlinear super resolution, but has been used in all methods implemented so far. For more details on alternative super-resolution mechanisms, see Heintzmann’s answer in reference [20].
6. Do high-quality SIM images require reconstruction in Fourier space?
In general, SIM reconstructions can be split into two separate steps: (i) estimating the parameters of the pattern of illumination used (period, orientation and phase) and (ii) using these parameters to separate and recombine information contained within the raw images. In many implementations, both of these steps have been conducted in the Fourier domain, although there is no inherent reason why they must be.
Pattern estimation can potentially be skipped if the system is well calibrated and the sample does not distort the illumination pattern. If the pattern parameters are unknown, they are typically acquired via cross-correlating images with different phases. Assuming the pattern parameters are well known, either via calibration or through a pattern estimation routine that processes the raw data, a typical reconstruction would separate the different information components, shift them to their proper locations in Fourier space, and recombine them to form an image.
Separating the information components is typically thought of as solving the Fourier space matrix equation
6.1 |
where , the mixing matrix, has been calculated for the case of equal phase steps with no global phase offset, for simplicity. However, this mixing matrix acts pixel-wise and so there is no reason to not consider the real space equivalent instead:
6.2 |
This particular mixing matrix can be seen as a discrete Fourier transform along the phase direction, but this is not generally true and is not what is meant by a reconstruction in Fourier space (as this operates along the spatial directions). If we allow our image processing routine to use complex numbers, then we can separate each component by calculating the matrix inverse of and applying it to the raw data.
Shifting each component to the correct location in Fourier space can be achieved in real space, via the Fourier shift theorem, by multiplying with a phase ramp (see electronic supplementary material, video S2). It should, however, be noted that in many reconstructions a Fourier transform and inverse transform of a padded spectrum are used to artificially reduce the pixel size before shifting, as the raw data are not Nyquist-sampled for the higher resolution reconstruction. Staying in real space either requires oversampled raw data or an alternative upscaling technique.
Given shifted components, all that remains is to combine them to form an SIM image. This can be achieved with a direct sum, but most reconstructions choose to weight different parts of the separated components differently depending on their exact location in Fourier space, in an attempt to counteract the effects of the OTF.
As an alternative to this view of separating and recombining components, we can think of the process of structured illumination imaging as akin to amplitude modulation (AM), as used in early radios. Here, the carrier signal is our illumination pattern, the sample structure is our message signal and the recorded data is the product of these two, as is the AM signal. To demodulate an AM signal and recover the message signal, a radio can use a ‘product detector’, which heterodynes (i.e. multiplies) the AM signal with a signal from a local oscillator designed to match the carrier signal exactly. This suggests that a similar approach might be applicable to SIM data.
Consider a two-beam SIM system where, for simplicity, we will only consider one pattern orientation and perfect modulation contrast. In the usual way, the data acquired, , where is used as a phase index, are equal to the product of the illumination and sample, convolved with the PSF, :
6.3 |
6.4 |
As usual, in the Fourier domain:
6.5 |
Let us now consider multiplying our AM signals, , with a ‘local oscillator’, i.e. another sinusoidal pattern with the same phase and period:
6.6 |
In the Fourier domain, our local oscillator takes the form:
6.7 |
Now, let us heterodyne our signals:
6.8 |
such that in the Fourier domain we have:
6.9 |
Calculating this gives:
6.10 |
If we impose the conditions that
6.11 |
then most of these terms cancel when summed, giving:
6.12 |
Hence, the reconstructed image, obtained by multiplying each raw image with our local oscillator and summing all the images, has the same form as a traditional SIM reconstruction if we set . This form also makes clear why a SIM acquisition with only two phases does not reconstruct properly: in this case as desired, but and so we are left with terms of the form that we do not want.
Figure 3 shows an example of this reconstruction process for simulated data. Here, the ground truth sample (figure 3a) was engineered to be periodic, to avoid ‘cross’ artefacts in the spectra (figure 3d–f). In addition, its form makes the Moiré effect of the fringe-like illumination particularly visible (figure 3h).
Considering a three-beam system, for which
6.13 |
and
6.14 |
this approach provides
6.15 |
and so we see that we should set . Indeed, for beams, the correct is one which is spectrally flat, i.e. all delta functions have equal amplitude. This reconstruction approach will be revisited in §9 to consider the case of non-sinusoidal illumination. Similar real-space reconstruction approaches were also considered by Cragg & So early on in the development of SIM [4,49].
Intriguingly, the idea of using a patterned illumination and a local oscillator to enhance the resolution of an imaging system predates the idea of structured illumination microscopy by more than 30 years. In a series of papers, beginning in 1966, Lukosz investigated the possibility of enhancing the resolution of an imaging system by inserting opaque masks into object and image space [50–52]. While the details of the method are quite different to the modern idea of structured illumination microscopy, the underlying idea and the realization that an increase in spatial bandwidth must be accompanied by a decrease in temporal bandwidth were already present.
7. Can SIM be used for deep tissue imaging?
Using SIM for deep tissue imaging is complicated by two issues: (i) sample-induced aberrations and (ii) insufficient stripe contrast caused by out-of-focus and scattered light. We will restrict the discussion here to that of sample-induced aberrations and discuss (iii) in §8. In larger samples, it is common for the local refractive index to not be uniform over the entire sample. This means that light that arrives at or departs from a given location in the sample experiences a path-dependent phase shift with respect to light travelling in a uniform refractive index. These unanticipated phase shifts mean that the light is not properly focused by the microscope system. This is true for both illumination light and fluorescence emission.
Fortunately, it has been shown by Arigovindan et al. that aberrations predominantly affect the fluorescence emission light, as the effect of phase shifts on the illumination pattern can be taken into account in the reconstruction routine [53]. However, this assumes that the variation across the field-of-illumination is small (i.e. the ‘isoplanatic patch’ is large), otherwise the image must be split into blocks over which the pattern parameters are fixed. Correcting for aberrations in the fluorescence emission can be achieved using an ‘adaptive optic’, typically a deformable mirror, which applies a complementary phase shift to each part of the wavefront to cancel the aberrations. Usefully, this cannot only be used to compensate for sample-induced aberrations, but any residual aberrations from the microscope itself. However, as the same correction is applied to every point in the field-of-view there is once again a requirement that the variation across the field is small. This is a limitation of all wide-field adaptive optics systems and is in contrast to point-scanning systems, for which the aberration correction can be updated for every scan position, if necessary [54].
Such adaptive optics SIM systems have been constructed by a number of groups, with most differences being in the way an appropriate wavefront correction is obtained. Using a multiphoton guide star allows for direct wavefront sensing, in which the wavefront is measured using a Shack–Hartmann sensor, but typically requires an expensive femtosecond laser which may cause excess sample damage [55,56]. By contrast, sensorless schemes assume that the wavefront can be decomposed into a sum of orthogonal modes, and simply see which combination of modes maximizes an image quality metric. Recently, Žurauskas et al. exploited the nature of the structured illumination itself to further improve the performance of such a sensorless scheme [57]. These approaches have the benefit of not requiring a guide star, but mean that a large number of images must be acquired to obtain a good correction (at best for modes) [54].
8. How can the fundamental limitation of SIM, i.e. generating sufficient stripe contrast in densely labelled and/or extended biological structures due to out-of-focus light, be addressed?
Even for situations in which the adaptive optics schemes discussed in §7 are not required, obtaining high-quality SIM reconstructions can be challenging in thicker samples due to the out-of-focus light contaminating the image and hence reducing stripe contrast. For single-cell imaging, the out-of-focus light produced by 2-beam or 3-beam SIM is normally insufficient to preclude high-quality reconstructions. However, for tissue imaging, background fluorescence from scattered light and out-of-focus regions of the sample can quickly degrade imaging performance.
In general, there are two approaches to solve this problem: (i) avoid producing out-of-focus light in the first place; (ii) somehow remove the out-of-focus light before it hits the detector. In conventional fluorescence microscopy, (i) is the approach taken by multiphoton and light sheet microscopy, while (ii) is that taken by confocal microscopy. All three approaches can usefully be combined with SIM.
In 2012, Andresen & Pollok et al. created a sinusoidal multiphoton SIM pattern by scanning an array of 32 beamlets perpendicular to the beamlet line, with phase-stepping and reconstruction in the usual manner [58]. However, as each beamlet is incoherent with the others, the fine fringe patterns necessary for a high-resolution enhancement would have too low a modulation contrast in this scheme, precluding a full doubling of resolution. Later, various authors combined multiphoton excitation with some form of image scanning microscopy (discussed further in §9), which uses a focused spot for excitation and a different reconstruction scheme [59]. This has also been combined with adaptive optics to improve the imaging performance at depth further still [60].
Initially, light sheet microscopy was first combined with incoherent OS-SIM to enhance optical sectioning, rather than resolution [61]. Soon after, Planchon et al. used an incoherent superpositions of Bessel beams to provide enhanced resolution as well [62]. However, this still used the reconstruction approach of OS-SIM, resulting in a spatially varying point spread function and a concomitant loss of ability to describe the image formation process linearly. Gao et al. developed this further, using the traditional reconstruction approach of SR-SIM to enhance effective resolution further still and maintain linear image formation [63]. Here, the authors showed that this Bessel light sheet structured illumination approach provided clearer images of live LLC-PK1 cells in anaphase than wide-field SIM, despite the superior theoretical resolution of the latter. In the final development of Bessel light sheet SIM, Chen et al. [64] used a coherent superposition of Bessel beams to create a so-called ‘lattice’ light sheet, featuring improved modulation contrast and a higher duty-cycle. However, one drawback of all these approaches is that only one pattern orientation is used, resulting in an anisotropic resolution enhancement. This is further discussed in §16.
The prototypical example of combining SIM with confocal microscopy is image scanning microscopy, discussed in §9. One of the main drawbacks of this approach is the need to acquire an entire image for each scan position of the confocal spot, meaning that hundreds or thousands of images must be recorded to reconstruct one enhanced resolution image. As such, multifocal structured illumination microscopy (MSIM) was introduced, using multiple spots of illumination in parallel [65]. Following this, it was realized independently by multiple research groups that the reconstruction procedure was amenable to an all-optical implementation [66–68]. A further approach, using a 32-segment confocal detector and multichannel photomultiplier tube has also been introduced in Zeiss’ Airyscan product [69].
In all these approaches, the use of a confocal pinhole (either physical or digital) introduces optical sectioning, making these techniques less sensitive to out-of-focus light than traditional SIM. However, the effective resolution and contrast in non-scattering samples is lower, due to the less effective way in which the illumination brings higher spatial frequencies into the passband. Compared with traditional confocals, apart from the need for smaller pixel pitches to satisfy the Nyquist–Shannon sampling criterion, there is no drawback to using these SIM-like approaches, with benefits not only in resolution but also in signal-to-noise ratio, due to the superconcentration of light [70,71].
9. Should image scanning microscopy be considered a form of SIM and what forms of structured illumination could be used other than stripes?
Image scanning microscopy (ISM), first proposed by Sheppard in 1988 [72] and first demonstrated by Müller and Enderlein in 2010 [73], is experimentally quite different from SIM. In ISM, a tightly focused illumination spot is scanned over the sample, with a camera image of the emission recorded at every scan position. For each position in the spot image, the sample location most likely to have produced that intensity is not, as one might assume, at the location corresponding to the image position, but instead is located between the illumination scan position and image position, where is the Stokes’ shift factor (i.e. for no Stokes’ shift, the most-likely location is halfway between).
In ISM processing, every pixel in each image is relocated in this way, with the corresponding intensity being added to an upsampled pixel grid. At first, this seems very different from the SIM discussed so far. However, the idea that non-uniform illumination can provide higher resolution information is still the same, albeit with a very different form of structured illumination.
Indeed, the real-space reconstruction approach introduced in §6 can be extended to process ISM data as well, albeit not in a way that uses photon reassignment. For the purposes of visualization and computational speed, figure 4 uses multiple focused illumination spots in parallel, as produced by MSIM [65]. As before, each raw image (figure 4h) is multiplied by a pattern that mimics the spatial period and phase of the illumination, but is spectrally flat. These multiplied images (figure 4i) are then summed to produce the final reconstruction (figure 4c). As shown by a comparison of the spectra in figure 4e,f, the processed image contains information at higher spatial frequencies than in the simple sum of the raw data.
Furthermore, the real-space reconstruction approach can be modified to handle quasi-arbitrary illumination patterns. As an example of this, figure 5 shows a workflow for using a projection of the text ‘SIM’ as an illumination pattern. Electronic supplementary material, video S3 shows the steps in the acquisition and reconstruction as data is collected. While not necessarily spectrally optimal, a comparison of the spectra of the raw sum and remultiplied sum, along with linearly deconvolved images, shows that higher resolution information has successfully been extracted. An alternative approach in which the illumination pattern is not known, called ‘blind SIM’, has also been investigated [74–77].
10. How does sparse illumination compare to dense illumination in linear and nonlinear SIM?
As discussed in §8, SIM requires the illumination pattern to have good contrast in order to work well. For thicker samples, where much out-of-focus light contaminates the image of the plane of interest, sparse illumination is helpful. However, for maximum performance at high spatial frequencies, a dense illumination is required. In addition, a greater number of phase shifts must be used.
This can be understood by considering real space and the Fourier domain as reciprocal spaces. Hence, sparse illumination results in a dense Fourier spectrum and vice versa. A dense Fourier spectrum means that more components must be unmixed, and so a greater number of phase shifts must be used.
Furthermore, dense illumination is more readily combined with total internal reflection fluorescence microscopy. To see this, consider an MSIM-like illumination pattern, where the back focal plane of the objective has the illumination shown in figure 6a, with circular polarization. This produces an illumination in the focal plane as shown in figure 6b. If we wished for the illumination to be confined to the TIRF-regime, then we must ensure that only the outer annulus of the back focal plane is illuminated, as shown in figure 6c. This results in the illumination shown in figure 6d. Here, the interference in the pattern causes unwanted illumination, leading to excess photodamage and noise.
Recently, Ingerman et al. considered a related question specifically for dense illuminations: is it better to use two-dimensional patterns or sequentially rotated one-dimensional patterns for linear and nonlinear SIM [78]? Perhaps surprisingly, they found that two-dimensional patterns were better for high resolutions in nonlinear SIM while one-dimensional patterns were superior for linear SIM, although the performance advantage of two-dimensional patterns disappeared when non-photoswitchable background was introduced. In addition, for the nonlinear case, they found that two-dimensional patterns require almost twofold more exposure to off-switching light as one-dimensional patterns, for the same resolution enhancement. This suggests that one-dimensional methods are superior if photodamage from off-switching light is present.
These considerations began more than a decade before appearing in print, but publication was delayed by the deteriorating health and untimely death of Mats Gustafsson in 2011. The version initially submitted in 2008 was revived and modified by Ingerman, London and Heintzmann as a tribute, being finally published in 2019.
11. Can SIM be used to improve the resolution of (Rayleigh scattering) transmission microscopy?
No. As noted in §2, the resolution limit for oblique illumination is already twice as good as that for head-on illumination. A structured illumination set-up can provide such oblique illumination, but there is no further resolution to be gained. For further information, see the excellent article by Wicker & Heintzmann [79].
12. Can we generate ‘true’ super-resolution images from simple instruments enhanced with machine-learning-based algorithms?
Machine learning has proven to be a powerful tool for diverse image processing applications, such as image segmentation, denoising, deconvolution, etc. [80–83]. However, care should be taken to distinguish between approaches that try to extract information from an image and those that try to add information to an image. With good training, it has been possible to develop multiple deep-learning approaches that convert an image, or set of images (such as an SIM acquisition sequence), into a higher resolution image [84–87]. While processing a set of images may fall into the ‘extract’ category, processing a single image in such a way must necessarily fall into the ‘add’ category. As such, machine-learning will never be able to truly replace, rather than augment, a super-resolution technique. Nevertheless, such single-image approaches may be able to improve the effective resolution in a manner similar to deconvolution.
13. Can research-grade super-resolution (SIM) microscopes be built cost-efficiently?
All structured illumination microscopes are relatively cheap—the expense lies in the time required to make them work well! A basic system can be constructed using a rotation mount, piezo translation mount, diffraction grating, polarizer and a standard laser, plus some lenses and mirrors (as in the very first SIM publications [1,2,5]). However, such a system will be too slow for live-cell imaging and can severely limit the throughput of fixed-sample imaging [8,9]. Constructing an SIM system fast enough for live-cell imaging requires a way of generating the sinusoidal illumination pattern without physical rotation of a grating. In general, two methods are common: spatial light modulators (SLMs) and interferometric systems.
Spatial light modulators can, to a first approximation, be thought of as reprogrammable diffraction gratings and used in a similar way. However, instead of shifting and rotating the SLM, the pattern displayed by the SLM is instead updated to match. This arrangement maintains the common-path layout of grating-based systems, minimizing the sensitivity to air currents and temperature fluctuations. However, fast-switching ferroelectric SLMs have a very low diffraction efficiency (often less than ) and so require powerful laser sources to be used. Fast-switching nematic SLMs, with much higher diffraction efficiencies, are now commercially available, albeit at high cost and not yet capable of the frame rates of ferroelectric SLMs.
Interferometric systems normally feature some kind of Michelson/Twyman–Green or Mach–Zehnder interferometer, where pattern orientation is changed by changing optical paths and phase steps are accomplished by changing the length of one arm of the interferometer [88,89]. These approaches are much more photon-efficient than SLM-based systems and, as the pattern period changes with wavelength, rather than being fixed, can be used to ensure the same effective illumination NA over all wavelengths in use. This is the approach taken by the commercial OMX Blaze system [90]. However, such interferometers are extremely sensitive to air currents and temperature fluctuations, as any phase shift in one arm of the interferometer directly couples to a phase shift in the illumination pattern. In an attempt to circumvent this issue, fibre-based interferometeric SIM systems have recently been developed [91,92].
Overall, SLM-based systems are by far the most common, in both research-grade and commercial instruments. In particular, Forth Dimension Displays have cemented themselves as the preferred supplier of ferroelectric SLMs for the SIM community, due to their high pixel count and rapid switching speed. Nevertheless, successful operation of such an SLM is tricky for a newcomer, due to effects from the binary nature of the pixels, interpixel vias, duty-cycle requirements and low photon-efficiency. As such, there has recently been much interest in the development of systems that use a different type of SLM, namely digital micromirror devices (DMDs) [93,94]. Found in many projectors, these devices rely on tilting tiny micromirrors to direct light in one direction or another, acting as a binary amplitude display. While these provide high photon-efficiencies, their tilted-mirror nature means that they act akin to blazed gratings, with a strongly wavelength-dependent distribution of power into different diffraction orders. As such, their use is currently restricted to those laboratories interested in researching how best to make use of DMDs for SIM, rather than following the tried-and-tested approach of ferroelectric SLMs and powerful lasers.
14. How can information about single-molecule detection be best combined with the knowledge of the illumination structure?
This is an area of ongoing research, prompted by the development of MINFLUX [95]. In MINFLUX, an intensity minimum at the centre of a doughnut beam is used to localize single fluorophores by iteratively updating the doughnut position and minimizing the photon flux. While this approach can provide excellent localization precisions (less than ), it is inherently slow as it only probes one molecule at a time. Since the initial publication of the MINFLUX method, there have been a number of publications which combine full-field fringe-like structured illumination with the MINFLUX concept to provide high-precision localizations at faster rates [96–99]. Most recently, Schmidt et al. have shown that these methods only use half the localization information encoded by the structured illumination and demonstrate a further method that, unlike previous approaches, achieves the Cramér–Rao lower bound [100].
15. Can the axial resolution of an SIM microscope be enhanced beyond a twofold improvement?
Enhancing the axial resolution of an SIM system requires accessing the sample from the other side, either to introduce extra wavevectors for illumination, or to capture extra wavevectors of fluorescence. Perhaps the most impressive SIM microscope yet built, the of Shao et al., does both [101]. By using a second opposing high-numerical-aperture objective, a further three beams of illumination are introduced. The same objective is also used to capture fluorescence, with the two objectives being aligned to such a precision that the fluorescence captured by both can be made to interfere via a beamsplitter, despite the approximately coherence length of the light. This provides an almost-isotropic, better-than-100 nm resolution in all three dimensions, providing volumes of objects with no preferred viewing direction.
Despite this impressive performance, is not widely used, due to the challenging alignment and stability requirements for its operation. In addition, the very short coherence length of fluorescence emission (approx. ) means that sample-induced phase differences restrict the technique to imaging thin single cells. Recently, Manton et al. proposed that a lower-numerical-aperture, long-working-distance dipping objective could be used to introduce a single counterpropagating beam and provide much of the axial resolution enhancement of in a much simpler set-up [102]. While such a system is theoretically capable of providing three-dimensional resolutions better than 125 nm, the transfer function at high spatial frequencies is lower than that of an microscope without interferometric detection. As such, we expect that six-beam structured illumination provided by two opposing high-numerical-aperture objectives will be the future method of choice for extended-resolution imaging when isotropic resolution is key.
16. Can light sheet microscopy and SIM, with multiple pattern orientations, be properly combined?
As noted in §8, light sheet and SIM appear to be a perfect match: light sheet microscopy is limited in resolution due to the relatively low numerical aperture objectives used, but generates very little out-of-focus light—the bane of SIM. However, traditional light sheet SIM (LSFM-SIM) suffers from the relatively coarse illumination pattern that can be provided by the excitation objective in LSFM systems and the fact that only one orientation of the pattern is present. As such, while improvements in axial resolution can be made, very little useful enhancement of lateral resolution is possible.
In an attempt to counter these issues, Manton & Rees proposed an LSFM-SIM system featuring three mutually orthogonal objectives [103]. However, the use of 0.8 NA objectives, necessary for reasons of steric hindrance, meant that the maximum possible lateral resolution would still only be around 230 nm (i.e. approximately the theoretical resolution of a 1.1 NA water-dipping objective as used in some other light sheet microscope designs). While they also suggested the use of interferometric excitation through two objectives to improve the resolution to 120 nm, the experimental complexity of such a system means that it is unlikely to ever be realized in practice.
In 2017, Chang et al. demonstrated a three-objective system featuring interferometric excitation through two objectives and counterpropagating light sheets [104]. While the use of counterpropagating sheets produces a fine fringe period, issues with steric hindrance limited the orientations over which patterns could be produced, precluding a reconstruction with isotropic resolution. In addition, the large distance between arms of the interferometer mean that the system is sensitive to temperature-induced phase shifts, as discussed in §13, and sample mounting is made more difficult.
Furthermore, there is a theoretical issue with using interferometric excitation through multiple objectives that has so far been unaddressed. In the traditional SIM approach, we consider the illumination to be described by a discrete sum of purely harmonic functions (i.e. functions which consist of a single spatial frequency). As shown in figure 7a, this condition is satisfied for lattice light sheet microscopy [64] as the pupil pattern consists solely of vertical lines, which project orthographically onto the Ewald sphere (i.e. they still look like straight lines when viewed from above, with the separation between lines being constant—the distribution is a discrete sum of harmonic functions along the pattern direction). However, now consider a system in which a single vertical line in the pupil is used to produce a light sheet with one objective, with another identical pupil and objective opposing (figure 7b). Now, when the Ewald sphere is viewed from above, the distribution is not a discrete sum of harmonic functions along the pattern direction. While the limitations on numerical aperture provided by use of a single objective can be circumvented by using C-shaped illumination profiles in ancillary objective pupils (figure 7c), it appears that the use of counterpropagating sheets is not allowed, at least not by standard reconstruction approaches.
17. How quickly can SIM image a sample?
Fundamentally, the speed of SIM is restricted by the requirement that enough photons are detected for the signal-to-noise ratio to permit a high-quality reconstruction. So far, the fastest SIM acquisitions demonstrated were acquired at 798 Hz and 714 Hz, corresponding to image frame rates of 88.7 Hz and 79 Hz [105,106]. Given that even a single fluorophore can emit at rates in excess of 100 million photons per second, it appears that current speed limits are more practical than fundamental.
In an attempt to reduce the acquisition time for volumetric data by reducing the number of stage movements that must be completed, SIM has been combined with multifocal microscopy [107]. Here, images from multiple depths within the sample are recorded at the same time. While the initial demonstration in 2017, by Abrahamsson et al., used an aberration-free multifocus scheme, the limited speed of the commercial SIM system used prevented fast acquisitions [108]. A more recent demonstration, in 2020, of high-speed acquisition used an aberration-inducing multifocus prism, limiting the depth over which images could be acquired in parallel [109].
While these experiments demonstrate an impressive effort in pushing hardware to its limits, other effort has been expended on improving the reconstruction process. To enable live visualization of a sample with extended resolution, Markwirth et al. developed a software system that can provide GPU-accelerated reconstructions in almost real-time [110]. Many groups, along with Zeiss in their Elyra 7, have adopted ‘interleaved reconstruction’, in which rolling sets of three orientations are used to produce reconstructions, rather than only producing one image from each group of three-orientation data [111]. While this produces smoother videos, it is a form of interpolation and does not truly increase speed [112].
A number of attempts have been made at reducing the number of raw images required for a successful reconstruction, down from nine. Heintzmann was the first to propose that a smaller set of images could be sufficient, identifying that using as few as four images might be possible [113]. Lal et al. have proposed such a reconstruction using just four images, but the illumination patterns used do not average out to a uniform intensity [114]. This means that, for time-lapse imaging, the illumination pattern would ‘burn’ into the sample via spatially dependent photobleaching. Orieux et al. have developed a Bayesian approach to SIM reconstruction and used this to develop a method that uses four images and has no specific constraints on the illumination [115]. Furthermore, Ströhl and Kaminski investigated the possibility of a further reduction to just three images and developed a maximum-likelihood approach to deal with the resulting underdetermined system [116].
18. Outlook
At the time of writing, SIM is more than 22 years old. A PubMed analysis, searching for publications with the phrase ‘structured illumination’ in their title or abstract suggests that more than 100 SIM-related articles have been published every year since 2015 (figure 8a). An analysis of the number of citations, as indexed by Google Scholar, of the top 10 most-cited SIM papers shows that the total number of citations of this cohort now exceeds 1000 per year (figure 8b). Ignoring the unusual peak in 2017/2018, it appears that interest in SIM continues to increase.
Indeed, the estimation of the number of SIM-related papers is likely to be a significant underestimate—SIM as a technique has now matured to the point where it is no longer a ‘shiny new toy’ that must be mentioned in the title or abstract. Nowadays, it suffices to just mention SIM in the main text alongside well-respected techniques such as confocal microscopy.
Perhaps the most important development in the past few years is that of all-optical versions of ISM, providing extended-resolution imaging at high frame rates and without the need for computational reconstruction [66–68,118]. These instruments present the user with a system that effectively works as a familiar confocal or spinning disc microscope, albeit one with double the resolving power. Furthermore, by avoiding computational reconstruction, these approaches avoid the reconstruction artefacts that can often plague SIM [119]. However, in the absence of background fluorescence, at high spatial frequencies traditional SIM still provides superior contrast and is more compatible with TIRF imaging. As such, further investigation into which techniques provide better performance in different samples and imaging scenarios will be welcome, as would an all-optical SIM reconstruction approach.
For longer-term time-lapse imaging, light sheet fluorescence microscopy (LSFM) is now the technique of choice for minimal sample damage where fluorescence contrast is required. While combinations of LSFM and SIM have already been demonstrated (see §§8 and 16), an instrument that achieves maximal performance of both components has yet to be demonstrated.
Further enhancements to the capabilities of SIM for time-lapse imaging are being provided by enhanced reconstruction routines that use various forms of regularization to successfully reconstruct images with lower signal levels [120,121]. In combination with brighter fluorophores, further development of these techniques and other denoising procedures will help provide gentle, extended-resolution imaging of proteins of interest at endogenous expression levels.
As well as enhancing the resolution of images and movies, SIM has also been demonstrated to enhance the resolution of biophysical microscopy methods. While fluorescence lifetime imaging microscopy (FLIM) has been combined with OS-SIM [122] and used correctively with SR-SIM [123], so far no true combination of FLIM and SIM has been achieved. In 2019, Zhanghao et al. reported their method of polarized structured illumination microscopy (pSIM), providing extended-resolution imaging of dipoles in biopolymers such as cytoskeletal networks and -DNA [124]. Most recently, in 2019 Colin-York et al. have combined traction force microscopy (TFM) with 3D-SIM [125], while in 2021 Barbieri et al. combined TFM with TIRF-SIM [126].
Altogether, it seems that there is still much room for further work in SIM, both in terms of technique development and applications. It will be interesting to survey the field again in 20 years’ time and see how many of the outstanding challenges have been solved and which unanticipated developments arose. If the past 20 years are any guide, the answers will be surprising.
Acknowledgements
The author is grateful to Jérôme Boulanger, Reto Fiolka, Rainer Heintzmann, Chris Rowlands, Florian Ströhl and Andrew York for various SIM-related discussions/arguments over the years, and thanks Jérôme Boulanger for the ground-truth image used in figures 3–5. The author acknowledges support from Fitzwilliam College, Cambridge, through a Research Fellowship.
Data accessibility
This article has no additional data.
Authors' contributions
J.M.: investigation, software, visualization, writing original draft, writing review and editing.
Competing interests
I declare I have no competing interests.
Funding
No funding has been received for this article.
References
- 1.Heintzmann R, Cremer CG. 1999. Laterally modulated excitation microscopy: improvement of resolution by using a diffraction grating. In BiOS Europe’98, pp. 185–196. Stockholm, Sweden: SPIE.
- 2.Gustafsson MGL. 2000. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J. Microsc. 198, 82-87. ( 10.1046/j.1365-2818.2000.00710.x) [DOI] [PubMed] [Google Scholar]
- 3.Frohn JT, Knapp HF, Stemmer A. 2000. True optical resolution beyond the Rayleigh limit achieved by standing wave illumination. Proc. Natl Acad. Sci. USA 97, 7232-7236. ( 10.1073/pnas.130181797) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Cragg GE, So PTC. 2000. Lateral resolution enhancement with standing evanescent waves. Opt. Lett. 25, 46-48. ( 10.1364/OL.25.000046) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Gustafsson MGL, Shao L, Carlton PM, Wang CR, Golubovskaya IN, Cande WZ, Agard DA, Sedat JW. 2008. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. Biophys. J. 94, 4957-4970. ( 10.1529/biophysj.107.120345) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Shao L, Isaac B, Uzawa S, Agard DA, Sedat JW, Gustafsson MGL. 2008. I5S: wide-field light microscopy with 100-nm-scale resolution in three dimensions. Biophys. J. 94, 4971-4983. ( 10.1529/biophysj.107.120352) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Hirvonen LM, Wicker K, Mandula O, Heintzmann R. 2009. Structured illumination microscopy of a living cell. Eur. Biophys. J. 38, 807-812. ( 10.1007/s00249-009-0501-6) [DOI] [PubMed] [Google Scholar]
- 8.Kner P, Chhun BB, Griffis ER, Winoto L, Gustafsson MGL. 2009. Super-resolution video microscopy of live cells by structured illumination. Nat. Methods 6, 339-342. ( 10.1038/nmeth.1324) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Shao L, Kner P, Rego EH, Gustafsson MGL. 2011. Super-resolution 3D microscopy of live whole cells using structured illumination. Nat. Methods 8, 1044-1046. ( 10.1038/nmeth.1734) [DOI] [PubMed] [Google Scholar]
- 10.Betzig E, Trautman JK, Harris TD, Weiner JS, Kostelak RL. 1991. Breaking the diffraction barrier: optical microscopy on a nanometric scale. Science 251, 1468-1470. ( 10.1126/science.251.5000.1468) [DOI] [PubMed] [Google Scholar]
- 11.Betzig E, Patterson GH, Sougrat R, Lindwasser OW, Olenych S, Bonifacino JS, Davidson MW, Lippincott-Schwartz J, Hess HF. 2006. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642-1645. ( 10.1126/science.1127344) [DOI] [PubMed] [Google Scholar]
- 12.Hess ST, Girirajan TPK, Mason MD. 2006. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophys. J. 91, 4258-4272. ( 10.1529/biophysj.106.091116) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Rust MJ, Bates M, Zhuang X. 2006. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793-796. ( 10.1038/nmeth929) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Sharonov A, Hochstrasser RM. 2006. Wide-field subdiffraction imaging by accumulated binding of diffusing probes. Proc. Natl Acad. Sci. USA 103, 18 911-18 916. ( 10.1073/pnas.0609643104) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Hell SW, Wichmann J. 1994. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy. Opt. Lett. 19, 780-782. ( 10.1364/OL.19.000780) [DOI] [PubMed] [Google Scholar]
- 16.Klar TA, Hell SW. 1999. Subdiffraction resolution in far-field fluorescence microscopy. Opt. Lett. 24, 954-956. ( 10.1364/OL.24.000954) [DOI] [PubMed] [Google Scholar]
- 17.Klar TA, Jakobs S, Dyba M, Egner A, Hell SW. 2000. Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission. Proc. Natl Acad. Sci. USA 97, 8206-8210. ( 10.1073/pnas.97.15.8206) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Neil M, Juškaitis R, Wilson T. 1997. Method of obtaining optical sectioning by using structured light in a conventional microscope. Opt. Lett. 22, 1905-1907. ( 10.1364/OL.22.001905) [DOI] [PubMed] [Google Scholar]
- 19.Prakash K, Diederich B, Reichelt S, Heintzmann R, Schermelleh L. 2021. Super-resolution structured illumination microscopy: past, present and future. (https://arxiv.org/abs/2102.13649) [DOI] [PMC free article] [PubMed]
- 20.Heintzmann R. 2021. Answers to fundamental questions in superresolution microscopy. Phil. Trans. R. Soc. A 379, 20210105. ( 10.1098/rsta.2021.0105) [DOI] [PubMed] [Google Scholar]
- 21.Rayleigh L. 1879. Investigations in optics, with special reference to the spectroscope. Lond. Edinb. Dublin Philos. Mag. J. Sci. 8, 261-274. ( 10.1080/14786447908639684) [DOI] [Google Scholar]
- 22.Sparrow CM. 1916. On spectroscopic resolving power. Astrophys. J. 44, 76. ( 10.1086/142271) [DOI] [Google Scholar]
- 23.Abbe E. 1873. Beiträge zur theorie des mikroskops und der mikroskopischen wahrnehmung. Arch. Mikrosk. Anat. 9, 413-468. ( 10.1007/BF02956173) [DOI] [Google Scholar]
- 24.Gustafsson MGL, Agard DA, Sedat JW. 1995. Sevenfold improvement of axial resolution in 3D wide-field microscopy using two objective lenses. SPIE Proc. 2412, 147-156. ( 10.1117/12.205334) [DOI] [Google Scholar]
- 25.McCutchen CW. 1964. Generalized aperture and the three-dimensional diffraction image. JOSA 54, 240-244. ( 10.1364/JOSA.54.000240) [DOI] [Google Scholar]
- 26.Ewald PP. 1913. Zur theorie der interferenzen der röntgentstrahlen in kristallen. Phys. Z. 11, 465-472. [Google Scholar]
- 27.Lahrberg M, Singh M, Khare K, Ahluwalia BS. 2018. Accurate estimation of the illumination pattern’s orientation and wavelength in sinusoidal structured illumination microscopy. Appl. Opt. 57, 1019-1025. ( 10.1364/AO.57.001019) [DOI] [PubMed] [Google Scholar]
- 28.Wicker K. 2013. Non-iterative determination of pattern phase in structured illumination microscopy using auto-correlations in Fourier space. Opt. Express 21, 24 692-24 701. ( 10.1364/OE.21.024692) [DOI] [PubMed] [Google Scholar]
- 29.Smith CS, et al. 2021. Structured illumination microscopy with noise-controlled image reconstructions. bioRxiv, p. 2021.03.11.434940. ( 10.1101/2021.03.11.434940) [DOI]
- 30.Lal A, Shan C, Xi P. 2016. Structured illumination microscopy image reconstruction algorithm. IEEE J. Sel. Top. Quantum Electron. 22, 50-63. ( 10.1109/JSTQE.2016.2521542) [DOI] [Google Scholar]
- 31.Ball G, Demmerle J, Kaufmann R, Davis I, Dobbie IM, Schermelleh L. 2015. SIMcheck: a toolbox for successful super-resolution structured illumination microscopy. Sci. Rep. 5, 15915. ( 10.1038/srep15915) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Müller M, Mönkemöller V, Hennig S, Hübner W, Huser T. 2016. Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ. Nat. Commun. 7, ncomms10980. ( 10.1038/ncomms10980) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Gustafsson MGL. 1999. Extended resolution fluorescence microscopy. Curr. Opin Struct. Biol. 9, 627-628. ( 10.1016/S0959-440X(99)00016-0) [DOI] [PubMed] [Google Scholar]
- 34.Chung E, Kim D, So PT. 2006. Extended resolution wide-field optical imaging: objective-launched standing-wave total internal reflection fluorescence microscopy. Opt. Lett. 31, 945-947. ( 10.1364/OL.31.000945) [DOI] [PubMed] [Google Scholar]
- 35.Stemmer A, Beck M, Fiolka R. 2008. Widefield fluorescence microscopy with extended resolution. Histochem. Cell Biol. 130, 807. ( 10.1007/s00418-008-0506-8) [DOI] [PubMed] [Google Scholar]
- 36.Li D, et al. 2015. Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics. Science 349, aab3500. ( 10.1126/science.aab3500) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Hell SW. 2003. Toward fluorescence nanoscopy. Nat. Biotechnol. 21, 1347-1355. ( 10.1038/nbt895) [DOI] [PubMed] [Google Scholar]
- 38.Fiolka R, Beck M, Stemmer A. 2008. Structured illumination in total internal reflection fluorescence microscopy using a spatial light modulator. Opt. Lett. 33, 1629-1631. ( 10.1364/OL.33.001629) [DOI] [PubMed] [Google Scholar]
- 39.Liu PY, et al. 2016. Cell refractive index for cell biology and disease diagnosis: past, present and future. Lab Chip 16, 634-644. ( 10.1039/C5LC01445J) [DOI] [PubMed] [Google Scholar]
- 40.Dasgupta A, Deschamps J, Matti U, Hübner U, Becker J, Strauss S, Jungmann R, Heintzmann R, Ries J. 2021. Direct supercritical angle localization microscopy for nanometer 3D superresolution. Nat. Commun. 12, 1180. ( 10.1038/s41467-021-21333-x) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Sheppard CJR. 2007. Fundamentals of superresolution. Micron 38, 165-169. ( 10.1016/j.micron.2006.07.012) [DOI] [PubMed] [Google Scholar]
- 42.Heintzmann R, Jovin TM, Cremer C. 2002. Saturated patterned excitation microscopy—a concept for optical resolution improvement. JOSA A 19, 1599-1609. ( 10.1364/JOSAA.19.001599) [DOI] [PubMed] [Google Scholar]
- 43.Gustafsson MGL. 2005. Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution. Proc. Natl Acad. Sci. USA 102, 13 081-13 086. ( 10.1073/pnas.0406877102) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Kubitscheck U, Kückmann O, Kues T, Peters R. 2000. Imaging and tracking of single GFP molecules in solution. Biophys. J. 78, 2170-2179. ( 10.1016/S0006-3495(00)76764-6) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Rego EH, Shao L, Macklin JJ, Winoto L, Johansson GA, Kamps-Hughes N, Davidson MW, Gustafsson M. 2012. Nonlinear structured-illumination microscopy with a photoswitchable protein reveals cellular structures at 50-nm resolution. Proc. Natl Acad. Sci. USA 109, E135-E143. ( 10.1073/pnas.1107547108) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Hirvonen L, Mandula O, Wicker K, Heintzmann R. 2008. Structured illumination microscopy using photoswitchable fluorescent proteins. In Three-dimensional and multidimensional microscopy: image acquisition and processing XV, vol. 6861 (eds J-A Conchello, CJ Cogswell, T Wilson, TG Brow), pp. 133–140. Bellingham, WA: SPIE. [DOI] [PubMed]
- 47.Schwentker MA, Bock H, Hofmann M, Jakobs S, Bewersdorf J, Eggeling C, Hell SW. 2007. Wide-field subdiffraction RESOLFT microscopy using fluorescent protein photoswitching. Microsc. Res. Tech. 70, 269-280. ( 10.1002/(ISSN)1097-0029) [DOI] [PubMed] [Google Scholar]
- 48.Heintzmann R, Gustafsson MGL. 2009. Subdiffraction resolution in continuous samples. Nat. Photonics 3, 362-364. ( 10.1038/nphoton.2009.102) [DOI] [Google Scholar]
- 49.So PTC, Kwon H-S, Dong CY. 2001. Resolution enhancement in standing-wave total internal reflection microscopy: a point-spread-function engineering approach. JOSA A 18, 2833-2845. ( 10.1364/JOSAA.18.002833) [DOI] [PubMed] [Google Scholar]
- 50.Lukosz W. 1966. Optical systems with resolving powers exceeding the classical limit*. JOSA 56, 1463-1471. ( 10.1364/JOSA.56.001463) [DOI] [Google Scholar]
- 51.Lukosz W. 1967. Optical systems with resolving powers exceeding the classical limit. II. JOSA 57, 932-941. ( 10.1364/JOSA.57.000932) [DOI] [Google Scholar]
- 52.Bachl A, Lukosz W. 1967. Experiments on superresolution imaging of a reduced object field. JOSA 57, 163-169. ( 10.1364/JOSA.57.000163) [DOI] [Google Scholar]
- 53.Arigovindan M, Sedat JW, Agard DA. 2012. Effect of depth dependent spherical aberrations in 3D structured illumination microscopy. Opt. Express 20, 6527-6541. ( 10.1364/OE.20.006527) [DOI] [PubMed] [Google Scholar]
- 54.Booth MJ. 2014. Adaptive optical microscopy: the ongoing quest for a perfect image. Light: Sci. Appl. 3, e165-e165. ( 10.1038/lsa.2014.46) [DOI] [Google Scholar]
- 55.Turcotte R, Liang Y, Tanimoto M, Zhang Q, Li Z, Koyama M, Betzig E, Ji N. 2019. Dynamic super-resolution structured illumination imaging in the living brain. Proc. Natl Acad. Sci. USA 116, 9586-9591. ( 10.1073/pnas.1819965116) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Lin R, Kner PA. 2021. Structured illumination microscopy with direct wavefront sensing. In Adaptive optics and wavefront control for biological systems VII, vol. 11652 (eds TG Bifano, S Gigan, N Ji), p. 116520A. Bellingham, WA: International Society for Optics and Photonics.
- 57.Z̆urauskas M, Dobbie IM, Parton RM, Phillips MA, Göhler A, Davis I, Booth MJ. 2019. IsoSense: frequency enhanced sensorless adaptive optics through structured illumination. Optica 6, 370-379. ( 10.1364/OPTICA.6.000370) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Andresen V, et al. 2012. High-resolution intravital microscopy. PLoS ONE 7, e50915. ( 10.1371/journal.pone.0050915) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Winter PW, et al. 2014. Two-photon instant structured illumination microscopy improves the depth penetration of super-resolution imaging in thick scattering samples. Optica 1, 181-191. ( 10.1364/OPTICA.1.000181) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Zheng W, et al. 2017. Adaptive optics improves multiphoton super-resolution imaging. Nat. Methods 14, 869-872. ( 10.1038/nmeth.4337) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61.Keller PJ, Schmidt AD, Santella A, Khairy K, Bao Z, Wittbrodt J, Stelzer EHK. 2010. Fast, high-contrast imaging of animal development with scanned light sheet-based structured-illumination microscopy. Nat. Methods 7, 637-642. ( 10.1038/nmeth.1476) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 62.Planchon TA, Gao L, Milkie DE, Davidson MW, Galbraith JA, Galbraith CG, Betzig E. 2011. Rapid three-dimensional isotropic imaging of living cells using Bessel beam plane illumination. Nat. Methods 8, 417-423. ( 10.1038/nmeth.1586) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Gao L, Shao L, Higgins CD, Poulton JS, Peifer M, Davidson MW, Wu X, Goldstein B, Betzig E. 2012. Noninvasive imaging beyond the diffraction limit of 3D dynamics in thickly fluorescent specimens. Cell 151, 1370-1385. ( 10.1016/j.cell.2012.10.008) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Chen B-C, et al. 2014. Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution. Science 346, 1257998. ( 10.1126/science.1257998) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.York AG, Parekh SH, Nogare DD, Fischer RS, Temprine K, Mione M, Chitnis AB, Combs CA, Shroff H. 2012. Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy. Nat. Methods 9, 749-754. ( 10.1038/nmeth.2025) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 66.Roth S, Sheppard CJR, Wicker K, Heintzmann R. 2013. Optical photon reassignment microscopy (OPRA). Opt. Nanoscopy 2, 5. ( 10.1186/2192-2853-2-5) [DOI] [Google Scholar]
- 67.De Luca GMR, et al. 2013. Re-scan confocal microscopy: scanning twice for better resolution. Biomed. 4, 2644. ( 10.1364/BOE.4.002644) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 68.York AG, Chandris P, Nogare DD, Head J, Wawrzusin P, Fischer RS, Chitnis A, Shroff H. 2013. Instant super-resolution imaging in live cells and embryos via analog image processing. Nat. Methods 10, 1122-1126. ( 10.1038/nmeth.2687) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69.Huff J. 2015. The Airyscan detector from ZEISS: confocal imaging with improved signal-to-noise ratio and super-resolution. Nat. Methods 12, i-ii. ( 10.1038/nmeth.f.388) [DOI] [Google Scholar]
- 70.Sheppard CJR, Roth S, Heintzmann R, Castello M, Vicidomini G, Chen R, Chen X, Diaspro A. 2016. Interpretation of the optical transfer function: significance for image scanning microscopy. Opt. Express 24, 27 280-27 287. ( 10.1364/OE.24.027280) [DOI] [PubMed] [Google Scholar]
- 71.Roth S, Sheppard CJR, Heintzmann R. 2016. Superconcentration of light: circumventing the classical limit to achievable irradiance. Opt. Lett. 41, 2109-2112. ( 10.1364/OL.41.002109) [DOI] [PubMed] [Google Scholar]
- 72.Sheppard C. 1988. Super-resolution in confocal imaging. Optik 80, 83-84. [Google Scholar]
- 73.Müller CB, Enderlein J. 2010. Image scanning microscopy. Phys. Rev. Lett. 104, 198101. ( 10.1103/PhysRevLett.104.198101) [DOI] [PubMed] [Google Scholar]
- 74.Mudry E, Belkebir K, Girard J, Savatier J, Le Moal E, Nicoletti C, Allain M, Sentenac A. 2012. Structured illumination microscopy using unknown speckle patterns. Nat. Photonics 6, 312-315. ( 10.1038/nphoton.2012.83) [DOI] [Google Scholar]
- 75.Ayuk R, et al. 2013. Structured illumination fluorescence microscopy with distorted excitations using a filtered blind-SIM algorithm. Opt. Lett. 38, 4723-4726. ( 10.1364/OL.38.004723) [DOI] [PubMed] [Google Scholar]
- 76.Jost A, Tolstik E, Feldmann P, Wicker K, Sentenac A, Heintzmann R. 2015. Optical sectioning and high resolution in single-slice structured illumination microscopy by thick slice blind-SIM reconstruction. PLoS ONE 10, e0132174. ( 10.1371/journal.pone.0132174) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 77.Yeh L-H, Tian L, Waller L. 2017. Structured illumination microscopy with unknown patterns and a statistical prior. Biomed. Opt. Express 8, 695-711. ( 10.1364/BOE.8.000695) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 78.Ingerman EA, London RA, Heintzmann R, Gustafsson MGL. 2019. Signal, noise and resolution in linear and nonlinear structured-illumination microscopy. J. Microsc. 273, 3-25. ( 10.1111/jmi.2019.273.issue-1) [DOI] [PubMed] [Google Scholar]
- 79.Wicker K, Heintzmann R. 2014. Resolving a misconception about structured illumination. Nat. Photonics 8, 342-344. ( 10.1038/nphoton.2014.88) [DOI] [Google Scholar]
- 80.Arganda-Carreras I, Kaynig V, Rueden C, Eliceiri KW, Schindelin J, Cardona A, Sebastian Seung H. 2017. Trainable Weka segmentation: a machine learning tool for microscopy pixel classification. Bioinformatics 33, 2424-2426. ( 10.1093/bioinformatics/btx180) [DOI] [PubMed] [Google Scholar]
- 81.Schmidt U, Weigert M, Broaddus C, Myers G. 2018. Cell detection with star-convex polygons. 11071, 265-273. (https://arxiv.org/abs/1806.03535) [Google Scholar]
- 82.Boulanger J, Kervrann C, Bouthemy P, Elbau P, Sibarita J, Salamero J. 2010. Patch-based nonlocal functional for denoising fluorescence microscopy image sequences. IEEE Trans. Med. Imaging 29, 442-454. ( 10.1109/TMI.2009.2033991) [DOI] [PubMed] [Google Scholar]
- 83.Guo M, et al. 2020. Rapid image deconvolution and multiview fusion for optical microscopy. Nat. Biotechnol. 38, 1337-1346. ( 10.1038/s41587-020-0560-x) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 84.Ling C, Zhang C, Zhang C, Wang M, Meng F, Du L, Du L, Yuan X, Yuan X. 2020. Fast structured illumination microscopy via deep learning. Photonics Res. 8, 1350-1359. ( 10.1364/PRJ.396122) [DOI] [Google Scholar]
- 85.Jin L, Liu B, Zhao F, Hahn S, Dong B, Song R, Elston TC, Xu Y, Hahn KM. 2020. Deep learning enables structured illumination microscopy with low light levels and enhanced speed. Nat. Commun. 11, 1934. ( 10.1038/s41467-020-15784-x) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 86.Qiao C, Li Di, Guo Y, Liu C, Jiang T, Dai Q, Li D. 2021. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat. Methods 18, 194-202. ( 10.1038/s41592-020-01048-5) [DOI] [PubMed] [Google Scholar]
- 87.Shah ZH, Müller M, Wang T-C, Scheidig P, Schneider A, Schüttpelz M, Huser T, Schenck W. 2021. Deep-learning based denoising and reconstruction of super-resolution structured illumination microscopy images. Photonics Res. 9, B168-B181. ( 10.1364/PRJ.416437) [DOI] [Google Scholar]
- 88.Best G, Amberger R, Baddeley D, Ach T, Dithmar S, Heintzmann R, Cremer C. 2011. Structured illumination microscopy of autofluorescent aggregations in human tissue. Micron 42, 330-335. ( 10.1016/j.micron.2010.06.016) [DOI] [PubMed] [Google Scholar]
- 89.Liu W, Liu Q, Zhang Z, Han Y, Kuang C, Xu L, Yang H, Liu X. 2019. Three-dimensional super-resolution imaging of live whole cells using galvanometer-based structured illumination microscopy. Opt. Express 27, 7237-7248. ( 10.1364/OE.27.007237) [DOI] [PubMed] [Google Scholar]
- 90.Dougherty WM, Quarre SC. 2013. Method and system for fast three-dimensional structured-illumination-microscopy imaging. Patent No. US8570650B2. Applied Precision Inc.
- 91.Hinsdale TA, Stallinga S, Rieger B. 2021. High-speed multicolor structured illumination microscopy using a hexagonal single mode fiber array. Biomed. Opt. Express 12, 1181-1194. ( 10.1364/BOE.416546) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 92.Pospíšil J, Pospíšil J, Wiebusch G, Fliegel K, Klíma M, Huser T. 2021. Highly compact and cost-effective 2-beam super-resolution structured illumination microscope based on all-fiber optic components. Opt. Express 29, 11 833-11 844. ( 10.1364/OE.420592) [DOI] [PubMed] [Google Scholar]
- 93.Brown PT, Kruithoff R, Seedorf GJ, Shepherd DP. 2020. Multicolor structured illumination microscopy and quantitative control of polychromatic coherent light with a digital micromirror device. bioRxiv, p. 2020.07.27.223941. ( 10.1101/2020.07.27.223941) [DOI] [PMC free article] [PubMed]
- 94.Sandmeyer A, et al. 2019. DMD-based super-resolution structured illumination microscopy visualizes live cell dynamics at high speed and low cost. bioRxiv 797670. ( 10.1101/797670) [DOI] [Google Scholar]
- 95.Balzarotti F, Eilers Y, Gwosch KC, Gynnå AH, Westphal V, Stefani FD, Elf J, Hell SW. 2017. Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes. Science 355, 606-612. ( 10.1126/science.aak9913) [DOI] [PubMed] [Google Scholar]
- 96.Reymond L, Reymond L, Ziegler J, Knapp C, Wang F-C, Huser T, Ruprecht V, Ruprecht V, Wieser S. 2019. SIMPLE: structured illumination based point localization estimator with enhanced precision. Opt. Express 27, 24 578-24 590. ( 10.1364/OE.27.024578) [DOI] [PubMed] [Google Scholar]
- 97.Gu L, Li Y, Zhang S, Xue Y, Li W, Li D, Xu T, Ji W. 2019. Molecular resolution imaging by repetitive optical selective exposure. Nat. Methods 16, 1114-1118. ( 10.1038/s41592-019-0544-2) [DOI] [PubMed] [Google Scholar]
- 98.Cnossen J, Hinsdale T, Thorsen RØ, Siemons M, Schueder F, Jungmann R, Smith CS, Rieger B, Stallinga S. 2020. Localization microscopy at doubled precision with patterned illumination. Nat. Methods 17, 59-63. ( 10.1038/s41592-019-0657-7) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 99.Jouchet P, Cabriel C, Bourg N, Bardou M, Poüs C, Fort E, Lévêque-Fort S. 2021. Nanometric axial localization of single fluorescent molecules with modulated excitation. Nat. Photonics 15, 297-304. ( 10.1038/s41566-020-00749-9) [DOI] [Google Scholar]
- 100.Schmidt M, Hundahl AC, Flyvbjerg H, Marie R, Mortensen KI. 2021. Camera-based localization microscopy optimized with calibrated structured illumination. Commun. Phys. 4, 1-9. ( 10.1038/s42005-020-00504-0) [DOI] [Google Scholar]
- 101.Shao L, et al. 2008. I5S: wide-field light microscopy with 100-nm-scale resolution in three dimensions. Biophys J. 94, 4971-4983. ( 10.1529/biophysj.107.120352) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 102.Manton JD, Manton JD, Ströhl F, Ströhl F, Fiolka R, Fiolka R, Kaminski CF, Rees EJ. 2020. Concepts for structured illumination microscopy with extended axial resolution through mirrored illumination. Biomed. Opt. Express 11, 2098-2108. ( 10.1364/BOE.382398) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 103.Manton JD, Rees EJ. 2016. TriSPIM: light sheet microscopy with isotropic super-resolution. Opt. Lett. 41, 4170-4173. ( 10.1364/OL.41.004170) [DOI] [PubMed] [Google Scholar]
- 104.Chang B-J, Didier Perez Meza V, Stelzer EHK. 2017. csiLSFM combines light-sheet fluorescence microscopy and coherent structured illumination for a lateral resolution below 100 nm. Proc. Natl Acad. Sci. USA 114, 4869-4874. ( 10.1073/pnas.1609278114) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 105.Guo Y, et al. 2018. Visualizing intracellular organelle and cytoskeletal interactions at nanoscale resolution on millisecond timescales. Cell 175, 1430-1442.e17. ( 10.1016/j.cell.2018.09.057) [DOI] [PubMed] [Google Scholar]
- 106.Song L, Lu-Walther H-W, Förster R, Jost A, Kielhorn M, Zhou J, Heintzmann R. 2016. Fast structured illumination microscopy using rolling shutter cameras. Meas. Sci. Technol. 27, 055401. ( 10.1088/0957-0233/27/5/055401) [DOI] [Google Scholar]
- 107.Prabhat P, Ram S, Ward ES, Ober RJ. 2004. Simultaneous imaging of different focal planes in fluorescence microscopy for the study of cellular dynamics in three dimensions. IEEE Trans. Nanobioscience 3, 237-242. ( 10.1109/TNB.2004.837899) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 108.Abrahamsson S, et al. 2017. Multifocus structured illumination microscopy for fast volumetric super-resolution imaging. Biomed. Opt. Express 8, 4135-4140. ( 10.1364/BOE.8.004135) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 109.Descloux A, et al. 2020. High-speed multiplane structured illumination microscopy of living cells using an image-splitting prism. Nanophotonics 9, 143-148. ( 10.1515/nanoph-2019-0346) [DOI] [Google Scholar]
- 110.Markwirth A, Lachetta M, Mönkemöller V, Heintzmann R, Hübner W, Huser T, Müller M. 2019. Video-rate multi-color structured illumination microscopy with simultaneous real-time reconstruction. Nat. Commun. 10, 4315. ( 10.1038/s41467-019-12165-x) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 111.Ma Y, Li D, Smith ZJ, Li D, Chu K. 2018. Structured illumination microscopy with interleaved reconstruction (SIMILR). J. Biophotonics 11, e201700090. ( 10.1002/jbio.2018.11.issue-2) [DOI] [PubMed] [Google Scholar]
- 112.Boualam A, Rowlands CJ. 2021. Method for assessing the spatiotemporal resolution of structured illumination microscopy (SIM). Biomed. Opt. Express 12, 790-801. ( 10.1364/BOE.403592) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 113.Heintzmann R. 2003. Saturated patterned excitation microscopy with two-dimensional excitation patterns. Micron 34, 283-291. ( 10.1016/S0968-4328(03)00053-2) [DOI] [PubMed] [Google Scholar]
- 114.Lal A, Shan C, Zhao K, Liu W, Huang X, Zong W, Chen L, Xi P. 2018. A frequency domain SIM reconstruction algorithm using reduced number of images. IEEE Trans. Image Process. 27, 4555-4570. ( 10.1109/TIP.2018.2842149) [DOI] [PubMed] [Google Scholar]
- 115.Orieux F, Sepulveda E, Loriette V, Dubertret B, Olivo-Marin J. 2012. Bayesian estimation for optimized structured illumination microscopy. IEEE Trans. Image Process. 21, 601-614. ( 10.1109/TIP.2011.2162741) [DOI] [PubMed] [Google Scholar]
- 116.Ströhl F, Kaminski CF. 2017. Speed limits of structured illumination microscopy. Opt. Lett. 42, 2511-2514. ( 10.1364/OL.42.002511) [DOI] [PubMed] [Google Scholar]
- 117.Schermelleh L, et al. 2008. Subdiffraction multicolor imaging of the nuclear periphery with 3D structured illumination microscopy. Science 320, 1332-1336. ( 10.1126/science.1156947) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 118.Azuma T, Kei T. 2015. Super-resolution spinning-disk confocal microscopy using optical photon reassignment. Opt. Express 23, 15 003-15 011. ( 10.1364/OE.23.015003) [DOI] [PubMed] [Google Scholar]
- 119.Demmerle J, et al. 2017. Strategic and practical guidelines for successful structured illumination microscopy. Nat. Protoc. 12, 988-1010. ( 10.1038/nprot.2017.019) [DOI] [PubMed] [Google Scholar]
- 120.Boulanger J, Pustelnik N, Condat L, Sengmanivong L, Piolot T. 2018. Nonsmooth convex optimization for structured illumination microscopy image reconstruction. Inverse Probl. 34, 095004. ( 10.1088/1361-6420/aaccca) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 121.Huang X, et al. 2018. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy. Nat. Biotechnol. 36, 451-459. ( 10.1038/nbt.4115) [DOI] [PubMed] [Google Scholar]
- 122.Hinsdale T, Olsovsky C, Rico-Jimenez JJ, Maitland KC, Jo JA, Malik BH. 2017. Optically sectioned wide-field fluorescence lifetime imaging microscopy enabled by structured illumination. Biomed. Opt. Express 8, 1455-1465. ( 10.1364/BOE.8.001455) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 123.Görlitz F, Corcoran DS, Garcia Castano EA, Leitinger B, Neil MAA, Dunsby C, French PMW. 2017. Mapping molecular function to biological nanostructure: combining structured illumination microscopy with fluorescence lifetime imaging (). Photonics 4, 40. ( 10.3390/photonics4030040) [DOI] [Google Scholar]
- 124.Zhanghao K, et al. 2019. Super-resolution imaging of fluorescent dipoles via polarized structured illumination microscopy. Nat. Commun. 10, 4694. ( 10.1038/s41467-019-12681-w) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 125.Colin-York H, et al. 2019. Spatiotemporally super-resolved volumetric traction force microscopy. Nano Lett. 19, 4427-4434. ( 10.1021/acs.nanolett.9b01196) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 126.Barbieri L, et al. 2021. Two-dimensional TIRF-SIM—traction force microscopy (2D TIRF-SIM-TFM). Nat. Commun. 12, 2169. ( 10.1038/s41467-021-22377-9) [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
This article has no additional data.