Inverse matrix based phase estimation algorithm for structured illumination microscopy

Abstract

The fast imaging speed and low-intensity requirement of structured illumination microscopy (SIM) have made it one of the most widely used imaging tools in live cell imaging. In order to obtain a high fidelity reconstructed image, a precise estimation of the phase of the illumination pattern is required, especially in those structured illumination based techniques that rely on high-order harmonics to improve the resolution. This can be achieved in one of two fundamental ways. The first is to build a high-end control system capable of shifting a sinusoidal pattern with high precision, while the second is to apply estimation algorithms to determine how patterns shift during post-processing. The latter method is preferred in low-cost super-resolution imaging systems; however, existing algorithms are either time-consuming or fail due to noise and a low modulation depth. In this paper, we introduce additional matrixes into the phase estimation algorithm and propose an inverse matrix based phase estimation method with which analytical solutions of the phases can be determined without iteration. The proposed algorithm was validated via simulation and experiments using a home-made total internal reflection fluorescent SIM system (TIRF-SIM). When tested, the method obtained the true phase even when the modulation depth was low. The source code is now available for download by researchers and others.

1. Introduction

When compared to electron microscopy, optical microscopy is less damaging to live cells; however, conventional optical microscopy is unable to resolve details smaller than 200 nm. Recent developments in fluorescence microscopy have enabled nanometer resolution, and super-resolution microscopy has great potential in imaging live cells. Due to the chemical specificity of proteins, it can also be applied to fluorescence labeling. A variety of fluorescence super-resolution techniques can be employed to uncover interactions between different organelles, and other techniques, such as stimulated emission depletion microscopy (STED) [1,2], stochastic optical reconstruction microscopy (STORM) [3], photo-activated localization microscopy (PALM) [4], and structured illumination microscopy (SIM) [5–7] can support higher resolutions than conventional optical microscopy. Among these, SIM is fast and can be adapted to different kinds of fluorophores. Despite the fact that the resolution enhancement in SIM is limited to a factor of two, in addition to its speed and adaptability, this technique is popular in biomedical research due to its lower phototoxicity and photobleaching effects compared to those of other methods.

Various methods have been proposed to improve the resolution in SIM, such as total internal reflection fluorescent microscopy (TIRF) [8–10] based structured illumination, plasmonic SIM (PSIM) [11,12], 3D-SIM [13–15], and saturated SIM (SSIM) [16]. While SSIM requires a greater intensity than SIM and may cause more photodamage to the cell, photo-switchable proteins can be used to reduce the illumination intensity requirement [17,18].

A key step in the post-processing of images is in determining the modulation frequency and estimating the phase. A high fidelity reconstruction typically requires the correct modulation frequency $k_0$ and phase $\phi$ of each illumination pattern, and errors in the estimated parameters degrade the image quality and introduce artifacts. The phase estimation algorithms are employed to estimate the phase $\phi$ of the fundamental, while the phase of the n-th harmonic in SSIM and non-linear SIM can be expressed as ${\phi }_{n-th}=n\phi$. Consequently, the error in the estimation step is also magnified by a factor of n if the n-th harmonic is applied. For this reason, SSIM and non-linear SIM is extremely sensitive to estimation errors.

Hence, many methods have been proposed to obtain the correct phase of each sinusoidal pattern [19–23]. For example, iterative cross-correlation based algorithms provide a precise phase estimation in most cases; however, they are time-consuming and may get trapped in a local minimum [21], which is the case for most optimization algorithms and limits their application. The auto-correlation based phase estimation algorithm can be used to directly calculate the phase of the illumination pattern without optimization, but its performance is generally poor compared to that of iterative methods when the SNR is poor, and assumes the modulation vector differs from the spatial frequency vector for some periodic samples [20].

In this paper, we propose a new algorithm based on an inverse matrix that improves on existing algorithms. The proposed algorithm solves equation sets describing the phases of the illumination pattern analytically without requiring optimization. In this way, the algorithm avoids being trapped in a local minimum, which is a risk in iteration based algorithms. Compared with the auto-correlation method, this approach performs better in low SNR and low modulation depth conditions. The phase-only correlation method is introduced in this paper to enhance the amplitude of the peak resulting from the modulation of the illumination field. The source code for the inverse matrix based method is available for download [24] (Code 1).

2. Principles and algorithms

Assuming a three-step phase shifting strategy is applied for simplification, the image sequences $D_i$(i = 1, 2, 3) captured for a certain illumination pattern orientation can be expressed as:

$$D_i=\left\{S\cdot \left[1+m\cdot \text{cos}\left(2\pi k_0\cdot r+{\phi }_i\right)\right]\right\}\ast h_{de},$$ (1)

where * refers to convolution, S denotes the sample, m denotes the modulation depth, $k_0$ is the modulation light wave vector, ${\phi }_i$ is the phase of the i-th illumination pattern, and $h_{de}$ is the detection point spread function (PSF).

In the Fourier domain, this can be represented as:

$$\begin{array}{l}\left[\begin{array}{c}{\tilde{D}}_1\left(k\right)\\ {\tilde{D}}_2\left(k\right)\\ {\tilde{D}}_3\left(k\right)\end{array}\right]=M\left(1,1,1;{\phi }_1^{ill},{\phi }_2^{ill},{\phi }_3^{ill}\right)\left[\begin{array}{c}\tilde{S}\left(k\right)\cdot {\tilde{H}}_{de}\left(k\right)\\ \tilde{S}\left(k-k_0\right)\cdot {\tilde{H}}_{de}\left(k\right)\\ \tilde{S}\left(k+k_0\right)\cdot {\tilde{H}}_{de}\left(k\right)\end{array}\right],\\ \text{where}M\left({\mu }_1,{\mu }_2,{\mu }_3;{\alpha }_1,{\alpha }_2,{\alpha }_3\right)=\left[\begin{array}{ccc}{\mu }_1& \frac{m}{2}\cdot e^{i{\alpha }_1}& \frac{m}{2}\cdot e^{-i{\alpha }_1}\\ {\mu }_2& \frac{m}{2}\cdot e^{i{\alpha }_2}& \frac{m}{2}\cdot e^{-i{\alpha }_2}\\ {\mu }_3& \frac{m}{2}\cdot e^{i{\alpha }_3}& \frac{m}{2}\cdot e^{-i{\alpha }_3}\end{array}\right]\end{array}$$ (2)

where the tilde (~) over S and D denotes the corresponding Fourier transform, ${\mu }_i$ is real, ${\phi }_i^{\mathit{\text{ill}}}$ denotes the actual phase of the illumination pattern, ${\tilde{H}}_{de}$ denotes the detection optical transfer function (OTF), and M is the modulation matrix.

In SIM, the phase optimization step in the reconstruction process comes after estimating the modulation frequency. Therefore, the prerequisite for phase estimation is to obtain the correct modulation frequency vector $k_0$. The phase-of-peak method proposed in the past estimates the modulation frequency vector and phases of the illumination pattern using the Fourier transform of the acquired images. This method assumes that the modulation frequency $\left|k_0\right|$ is smaller than the cutoff frequency $k_c$of the OTF. As the zero-frequency of the sample is shifted to $\pm k_0$, a peak appears at the corresponding position in the Fourier transform and the phase-of-peak method regards the phase of this peak as the phase of the illumination pattern. However, this assumption is not always satisfied. To address this, the cross-correlation method was proposed to correct the modulation matrix. Unlike the phase-of-peak method, which estimates the phase based on a single point, the cross-correlation method calculates the correlation between different components. All of the detectable frequency components in the Fourier domain contribute to the “peak” located at $k_0$ as well as the estimated phase. The correlation based method can be applied when $\left|k_0\right|$ is approximately equal to or larger than k_c, as in TIRF-SIM [20,21,25]. In addition to the above, other approaches that enhance the contrast of the peak based on the cross-correlation include the deconvolution method [22].

2.1 Estimation of the modulation frequency vector

In the estimation process, a low-pass filter similar to the Wiener filter is first applied to avoid the high-frequency noise beyond the OTF.

$${\tilde{D}}_i^{\mathit{\text{update}}}=\frac{\mathit{\text{conj}}\left({\tilde{H}}_{de}\right)\cdot {\tilde{D}}_i}{\left|{\tilde{H}}_{de}\right|+{\epsilon }_1},$$ (3)

where conj is the complex conjugation operation and ${\epsilon }_1$ is a positive regularization constant related to the noise that also serves to eliminate the zero denominator.

Despite the fact that the correct phase of the illumination pattern is not known, the high-frequency and baseband components can be separated with the inverse matrix of $M\left(1,1,1;{\gamma }_1,{\gamma }_2,{\gamma }_3\right)$. Here, ${\gamma }_i$can be selected casually in principle as long as ${\gamma }_i\ne {\gamma }_j\left(i\ne j\right)$. The reason the inverse matrix can separate different frequency components is illustrated by the following equation, which was derived by calculating the product on the left side:

$$M^{-1}\left(1,1,1;{\gamma }_1,{\gamma }_2,{\gamma }_3\right)M\left(1,1,1;{\phi }_1^{ill},{\phi }_2^{ill},{\phi }_3^{ill}\right)=\left[\begin{array}{ccc}1& c_0& \mathit{\text{conj}}\left(c_0\right)\\ 0& r_1{e}^{i{\varphi }_1}& r_2{e}^{i{\varphi }_2}\\ 0& r_2{e}^{-i{\varphi }_2}& r_1{e}^{-i{\varphi }_1}\end{array}\right],$$ (4)

where c₀ is a complex number and $r_1,r_2,{\varphi }_1,{\varphi }_2$ are non-negative real numbers. In the simulations and experiments in this study, $M\left(1,1,1;0,\frac{2\pi }{3},\frac{4\pi }{3}\right)$ was selected as it represents an ideal set of phase shifts. Also, for purposes of simplification, r₁>r₂ was assumed. The separated results are:

$$\left[\begin{array}{c}{\tilde{R}}_1\left(k\right)\\ {\tilde{R}}_2\left(k\right)\\ {\tilde{R}}_3\left(k\right)\end{array}\right]=M^{-1}\left(1,1,1;{\gamma }_1,{\gamma }_2,{\gamma }_3\right)\left[\begin{array}{c}{\tilde{D}}_1\left(k\right)\\ {\tilde{D}}_2\left(k\right)\\ {\tilde{D}}_3\left(k\right)\end{array}\right]=\left[\begin{array}{ccc}1& c_0& \mathit{\text{conj}}\left(c_0\right)\\ 0& r_1{e}^{i{\varphi }_1}& r_2{e}^{i{\varphi }_2}\\ 0& r_2{e}^{-i{\varphi }_2}& r_1{e}^{-i{\varphi }_1}\end{array}\right]\left[\begin{array}{c}\tilde{S}\left(k\right)\cdot {\tilde{H}}_{de}\left(k\right)\\ \tilde{S}\left(k-k_0\right)\cdot {\tilde{H}}_{de}\left(k\right)\\ \tilde{S}\left(k+k_0\right)\cdot {\tilde{H}}_{de}\left(k\right)\end{array}\right],$$ (5)

where ${\tilde{R}}_i$ represents the separated result. Thus, ${\tilde{R}}_2$ can be expressed as:

$${\tilde{R}}_2\left(k\right)=r_1{e}^{i{\varphi }_1}\tilde{S}\left(k-k_0\right)\cdot {\tilde{H}}_{de}\left(k\right)+r_2{e}^{i{\varphi }_2}\tilde{S}\left(k+k_0\right)\cdot {\tilde{H}}_{de}\left(k\right),$$ (6)

where the first term on the right side of the equation is dominant as r₁>r₂ and the second term is a redundant component caused by the incorrect phase. Hence, ${\tilde{R}}_2$ can be considered an estimate of a particular high-frequency component.

At the conclusion of the above steps, the noise has been reduced and the different frequency components have been roughly separated. Next, the modulation frequency vector is estimated via the phase-only correlation, which yields:

$$C\left(k\prime \right)=\frac{\tilde{W}\left(k\right)}{\left|\tilde{W}\left(k\right)\right|+{\epsilon }_2}\otimes \frac{{\tilde{R}}_2\left(k\right)}{\left|{\tilde{R}}_2\left(k\right)\right|+{\epsilon }_2}={\displaystyle \int \mathit{\text{conj}}\left(\frac{\tilde{W}\left(k\right)}{\left|\tilde{W}\left(k\right)\right|+{\epsilon }_2}\right)\cdot \frac{{\tilde{R}}_2\left(k+k\prime \right)}{\left|{\tilde{R}}_2\left(k+k\prime \right)\right|+{\epsilon }_2}d}k,$$ (7)

where $\otimes$ denotes correlation, $\tilde{W}$ is the Fourier transform of the widefield, and ${\epsilon }_2$ is a positive constant to avoid a zero denominator. The Fourier theorem can be applied to calculate the cross-correlation and estimate $k_0$to sub-pixel accuracy. In the modulation frequency estimation, $\tilde{W}$can be replaced by ${\displaystyle {\sum }_i}{\tilde{D}}_i$ or ${\tilde{R}}_1$. This is because this estimation only requires the peak located at $k_0$ (or $-k_0$). The corresponding correlation result yields:

$$\begin{array}{l}C\left(k_0\right)=\left[\frac{{\tilde{R}}_1\left(k\right)}{\left|{\tilde{R}}_1\left(k\right)\right|+{\epsilon }_2}\otimes \frac{{\tilde{R}}_2\left(k\right)}{\left|{\tilde{R}}_2\left(k\right)\right|+{\epsilon }_2}\right]\left(k_0\right)={\displaystyle \int \mathit{\text{conj}}\left(\frac{{\tilde{R}}_1\left(k\right)}{\left|{\tilde{R}}_1\left(k\right)\right|+{\epsilon }_2}\right)\cdot \frac{{\tilde{R}}_2\left(k+k_0\right)}{\left|{\tilde{R}}_2\left(k+k_0\right)\right|+{\epsilon }_2}d}k\\ ={\displaystyle \int \frac{\mathit{\text{conj}}\left[\tilde{S}\left(k\right)+c_0\tilde{S}\left(k-k_0\right)+\mathit{\text{conj}}(c_0)\tilde{S}\left(k+k_0\right)\right]}{\left|{\tilde{R}}_1\left(k\right)\right|+{\epsilon }_2}\cdot \frac{\left[r_1{e}^{i{\varphi }_1}\tilde{S}\left(k\right)+r_2{e}^{i{\varphi }_2}\tilde{S}\left(k+2k_0\right)\right]}{\left|{\tilde{R}}_2\left(k+k_0\right)\right|+{\epsilon }_2}}\\ \cdot \mathit{\text{conj}}\left[{\tilde{H}}_{de}\left(k\right)\right]\cdot {\tilde{H}}_{de}\left(k+k_0\right)dk\\ \approx {\displaystyle \int \frac{\mathit{\text{conj}}\left[\tilde{S}\left(k\right)\right]}{\left|{\tilde{R}}_1\left(k\right)\right|+{\epsilon }_2}\cdot \frac{r_1{e}^{i{\varphi }_1}\tilde{S}\left(k\right)}{\left|{\tilde{R}}_2\left(k+k_0\right)\right|+{\epsilon }_2}\cdot \mathit{\text{conj}}\left[{\tilde{H}}_{de}\left(k\right)\right]\cdot {\tilde{H}}_{de}\left(k+k_0\right)d}k\end{array}$$ (8)

The above approximation holds as $\tilde{S}\left(k\right)$, $\tilde{S}\left(k\pm k_0\right)$, and $\tilde{S}\left(k+2k_0\right)$ are uncorrelated.

A set of data was acquired using an homebuilt system [26], and the results after applying the existing and phase-only algorithms are shown in Fig. 1. The corresponding reconstruction results are shown later in this paper. The peak intensity, which was located at $\pm k_0$, is shown in Fig. 1(e). Even though a high-pass mask was applied to the auto-correlation results, it was still difficult to determine the modulation frequency as the amplitude at $k_0$ served as a local maximum and there was a large amount of background noise. When the deconvolution method was used to enhance the signal located at $k_0$, a high-pass mask was still required in order to detect the correct modulation frequency. In contrast, when the phase-only algorithm was used, the intensity became a global maximum, which obviated the need for a mask, as implied in Fig. 1(e). The signal amplitude located at $-k_0$ was significantly enhanced, which demonstrates the potential of the phase-only estimation algorithm compared to existing methods. The phase-only correlation algorithm may increase the sensitivity and improve the performance of the cross-correlation method, which can be used to optimize the phase based on the conjugated peak [21].

Fig. 1.

Comparison of different modulation frequency estimation methods. The peak inside the green boxes in (a)–(d) is located at $k_0$, while the peak inside the blue box is located at $-k_0$. The intensity of (a)–(d) was normalized by the corresponding maximum in each image. The correlation results were obtained when the phase shift was set as $2\pi /3$. Because the actual phase shift differed from the desired $2\pi /3$, the redundant component appeared in the cross-correlation and phase-only correlation results. A high-pass mask was applied in (a) to exclude components whose spatial frequency was smaller than $0.8k_c$. (e) The peak intensities are located at $k_0$(green square) and $-k_0$(blue circle). Abbreviations: AC, auto-correlation; CC, cross-correlation; CCwD, cross-correlation with deconvolution; PoC, phase-only correlation.

2.2 Estimation of the phases

Once the modulation frequency has been determined, the phase of $C\left(k\prime \right)$ can also be determined. As this phase is a function of the phase of the illumination pattern ${\psi }_{\mathit{\text{peak}}}=f\left({\phi }_1^{ill},{\phi }_2^{ill},{\phi }_3^{ill}\right)$, it is possible to compute the phases analytically. With this in mind, an inverse matrix based algorithm is proposed, the principle of which is as follows.

To mitigate the influence of the noise, the phase ${\psi }_{\mathit{\text{peak}}}$ was calculated as follows:

$$\begin{array}{l}{\psi }_{\mathit{\text{peak}}}=\text{arg}\left\{\left[\tilde{W}\otimes {\tilde{R}}_2\right]\left(k_0\right)\right\}=\text{arg}\left\{{\displaystyle \int \mathit{\text{conj}}\left[\tilde{W}\left(k\right)\right]\cdot {\tilde{R}}_2\left(k+k_0\right)d}k\right\}\\ =\text{arg}\left\{{\displaystyle \int \mathit{\text{conj}}\left[\tilde{S}\left(k\right)\cdot \tilde{H}\left(k\right)\right]\cdot \left[r_1{e}^{i{\varphi }_1}\tilde{S}\left(k\right)+r_2{e}^{i{\varphi }_2}\tilde{S}\left(k+2k_0\right)\right]\cdot {\tilde{H}}_{de}\left(k+k_0\right)d}k\right\},\\ \approx \text{arg}\left\{r_1{e}^{i{\varphi }_1}{\displaystyle \int \mathit{\text{conj}}\left[\tilde{S}\left(k\right)\right]\cdot \tilde{S}\left(k\right)\cdot \mathit{\text{conj}}\left[\tilde{H}\left(k\right)\right]\cdot {\tilde{H}}_{de}\left(k+k_0\right)d}k\right\}\approx {\varphi }_1\end{array}$$ (9)

where $\text{arg}(\cdot )$ is the argument of a complex number. The first approximation holds because $\tilde{S}\left(k\right)$ and $\tilde{S}\left(k+2k_0\right)$ are considered to be uncorrelated; thus, the signal generated by $r_2{e}^{i{\varphi }_2}\tilde{S}\left(k+2k_0\right)$ is much smaller (typically 2–3 orders of magnitude) than the other terms in Eq. (9), and the second approximation is an assumption that appears in all correlation based algorithms [20,21]. This equation also holds when $\tilde{W}$ is replaced by ${\displaystyle {\sum }_i}{\tilde{D}}_i$, as $\tilde{S}\left(k\right)$ will be dominant in. ${\displaystyle {\sum }_i}{\tilde{D}}_i$ And $\tilde{S}\left(k\right)$, $\tilde{S}\left(k\pm k_0\right)$ are also uncorrelated.

To determine how ${\phi }_i^{\mathit{\text{ill}}}$ acts on ${\varphi }_1$, the product of the matrix (the inverse is calculated by the adjugate matrix method) is considered:

$$\begin{array}{l}{M}^{-1}\left(1,1,1;{\gamma }_1,{\gamma }_2,{\gamma }_3\right)M\left(1,1,1;{\phi }_1^{ill},{\phi }_2^{ill},{\phi }_3^{ill}\right)={\left[\begin{array}{ccc}1& \frac{m}{2}\cdot e^{i{\gamma }_1}& \frac{m}{2}\cdot e^{-i{\gamma }_1}\\ 1& \frac{m}{2}\cdot e^{i{\gamma }_2}& \frac{m}{2}\cdot e^{-i{\gamma }_2}\\ 1& \frac{m}{2}\cdot e^{i{\gamma }_3}& \frac{m}{2}\cdot e^{-i{\gamma }_3}\end{array}\right]}^{-1}\left[\begin{array}{ccc}1& \frac{m}{2}\cdot e^{i{\phi }_1^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_1^{\mathit{\text{ill}}}}\\ 1& \frac{m}{2}\cdot e^{i{\phi }_2^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_2^{\mathit{\text{ill}}}}\\ 1& \frac{m}{2}\cdot e^{i{\phi }_3^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_3^{\mathit{\text{ill}}}}\end{array}\right]\\ =\left[\begin{array}{ccc}{x}_1& x_2& 1-x_1-x_2\\ b_1{e}^{i{\beta }_1}& b_2{e}^{i{\beta }_2}& b_3{e}^{i{\beta }_3}\\ b_1{e}^{-i{\beta }_1}& b_2{e}^{-i{\beta }_2}& b_3{e}^{-i{\beta }_3}\end{array}\right]\left[\begin{array}{ccc}1& \frac{m}{2}\cdot e^{i{\phi }_1^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_1^{\mathit{\text{ill}}}}\\ 1& \frac{m}{2}\cdot e^{i{\phi }_2^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_2^{\mathit{\text{ill}}}}\\ 1& \frac{m}{2}\cdot e^{i{\phi }_3^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_3^{\mathit{\text{ill}}}}\end{array}\right]=\left[\begin{array}{ccc}1& c_0& \mathit{\text{conj}}\left(c_0\right)\\ 0& r_1{e}^{i{\varphi }_1}& r_2{e}^{i{\varphi }_2}\\ 0& r_2{e}^{-i{\varphi }_2}& r_1{e}^{-i{\varphi }_1}\end{array}\right]\end{array}$$ (10)

where $x_1,x_2$ are real numbers and $b_i,{\beta }_i(i=1,2,3)$are non-negative real numbers. As $b_i,{\beta }_i(i=1,2,3)$ are part of the inverse matrix, they all depend on the chosen phases ${\gamma }_i$. From Eq. (10):

$$\frac{m}{2}\left(b_1{e}^{i{\phi }_1^{ill}+i{\beta }_1}+b_2{e}^{i{\phi }_2^{ill}+i{\beta }_2}+b_3{e}^{i{\phi }_3^{ill}+i{\beta }_3}\right)=r_1{e}^{i{\varphi }_1}\approx r_1{e}^{i{\psi }_{peak}}.$$ (11)

As the ratios of the imaginary and real parts in the right and left sides of Eq. (11) are equal, we have:

$$\begin{array}{l}\left[b_1\text{sin}\left({\phi }_1^{ill}+{\beta }_1\right)+b_2\text{sin}\left({\phi }_2^{ill}+{\beta }_2\right)+b_3\text{sin}\left({\phi }_3^{ill}+{\beta }_3\right)\right]\text{cos}\left({\psi }_{peak}\right)\\ =\left[b_1\text{cos}\left({\phi }_1^{ill}+{\beta }_1\right)+b_2\text{cos}\left({\phi }_2^{ill}+{\beta }_2\right)+b_3\text{cos}\left({\phi }_3^{ill}+{\beta }_3\right)\right]\text{sin}\left({\psi }_{peak}\right).\end{array}$$ (12)

The simplified expression is:

$$\begin{array}{l}{b}_1\text{sin}\left({\theta }_1-{\phi }_1^{ill}\right)+b_2\text{sin}\left({\theta }_2-{\phi }_2^{ill}\right)+b_3\text{sin}\left({\theta }_3-{\phi }_3^{ill}\right)=0,\\ \text{where}{\theta }_i={\psi }_{peak}-{\beta }_i.\end{array}$$ (13)

Note that $b_i,{\beta }_i(i=1,2,3)$ originate from the inverse matrix that depends on the chosen phases ${\gamma }_i$, and ${\psi }_{\mathit{\text{peak}}}$ can be determined in the cross-correlation. Thus, the remaining unknown variables represent the phases of interest. Also, note that the sign of the imaginary and real parts of Eq. (11) should be the same, as this is useful in the inverse matrix based phase estimation method explained below. As ${\phi }_i^{\mathit{\text{ill}}}(i=1,2,3)$ are the only unknowns, theoretically, the above equations can be solved via three independent equations. While there will be multiple solutions in Eq. (13) for ${\phi }_i^{\mathit{\text{ill}}}\in \left[0,2\pi \right)$, unique solutions that match the correct phases can be found using Eq. (11). As the constraints associated with Eq. (11) are more robust than those for Eq. (13), such as the sign issue mentioned earlier, solutions in this paper that satisfy Eq. (13) but do not hold for Eq. (11) are considered redundant.

However, as it difficult to solve trigonometric equations involving three variables, Eq. (13) is solved by introducing additional independent equations to avoid applying the Pythagorean trigonometric identity of sine and cosine. For example, consider the following set of equations: $2\text{sin}A+3\text{sin}B=\sqrt{3},2\text{cos}A+\text{cos}B=2,\text{sin}B=0$. If the third equation is not known, solving the first two equations will result in the additional answer $\left(A,B\right)=\left(\frac{\pi }{3},0\right)$, despite the fact that the other answers were also obtained. In contrast, if all three equations are known, it is straightforward to solve them as the Pythagorean identity need only be applied once.

The classical inverse (demodulation) matrix, which is $M^{-1}\left(1,1,1;{\gamma }_1,{\gamma }_2,{\gamma }_3\right)$, has only three independent arguments because m can be considered as a weighting factor of the modulation matrix, and therefore does not influence the phase of Eq. (11). Hence, the inverse matrix $M^{-1}\left(1,1,1;{\gamma }_1,{\gamma }_2,{\gamma }_3\right)$ has three “degrees of freedom.” In this way, three independent equations can be obtained that rely on only ${\gamma }_i$.

To obtain four or more independent equations, the inverse matrix can be changed to $M^{-1}\left({\sigma }_1,{\sigma }_2,{\sigma }_3;{\gamma }_1,{\gamma }_2,{\gamma }_3\right)$, where${\sigma }_i$ is positive and real. The product of this inverse matrix and the modulation matrix $M\left(1,1,1;{\phi }_1^{\mathit{\text{ill}}},{\phi }_2^{\mathit{\text{ill}}},{\phi }_3^{\mathit{\text{ill}}}\right)$ is:

$$\begin{array}{l}{M}^{-1}\left({\sigma }_1,{\sigma }_2,{\sigma }_3;{\gamma }_1,{\gamma }_2,{\gamma }_3\right)M\left(1,1,1;{\phi }_1^{ill},{\phi }_2^{ill},{\phi }_3^{ill}\right)\\ ={\left[\begin{array}{ccc}{\sigma }_1& \frac{m}{2}\cdot e^{i{\gamma }_1}& \frac{m}{2}\cdot e^{-i{\gamma }_1}\\ {\sigma }_2& \frac{m}{2}\cdot e^{i{\gamma }_2}& \frac{m}{2}\cdot e^{-i{\gamma }_2}\\ {\sigma }_3& \frac{m}{2}\cdot e^{i{\gamma }_3}& \frac{m}{2}\cdot e^{-i{\gamma }_3}\end{array}\right]}^{-1}\left[\begin{array}{ccc}1& \frac{m}{2}\cdot e^{i{\phi }_1^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_1^{\mathit{\text{ill}}}}\\ 1& \frac{m}{2}\cdot e^{i{\phi }_2^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_2^{\mathit{\text{ill}}}}\\ 1& \frac{m}{2}\cdot e^{i{\phi }_3^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_3^{\mathit{\text{ill}}}}\end{array}\right]\\ =\left[\begin{array}{ccc}{x}_1& x_2& x_3\\ b_1{e}^{i{\beta }_1}& b_2{e}^{i{\beta }_2}& b_3{e}^{i{\beta }_3}\\ b_1{e}^{-i{\beta }_1}& b_2{e}^{-i{\beta }_2}& b_3{e}^{-i{\beta }_3}\end{array}\right]\left[\begin{array}{ccc}1& \frac{m}{2}\cdot e^{i{\phi }_1^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_1^{\mathit{\text{ill}}}}\\ 1& \frac{m}{2}\cdot e^{i{\phi }_2^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_2^{\mathit{\text{ill}}}}\\ 1& \frac{m}{2}\cdot e^{i{\phi }_3^{\mathit{\text{ill}}}}& \frac{m}{2}\cdot e^{-i{\phi }_3^{\mathit{\text{ill}}}}\end{array}\right]=\left[\begin{array}{ccc}{w}_0& c_0& \mathit{\text{conj}}\left(c_0\right)\\ w_C& r_1{e}^{i{\varphi }_1}& r_2{e}^{i{\varphi }_2}\\ \mathit{\text{conj}}(w_C)& r_2{e}^{-i{\varphi }_2}& r_1{e}^{-i{\varphi }_1}\end{array}\right],\end{array}$$ (14)

where $w_C$ is complex and is determined by the inverse matrix, and $x_3$ and $w_0$ are real positive numbers. Now, ${\tilde{R}}_2\left(k\right)$ becomes:

$${\tilde{R}}_2\left(k\right)=w_C\tilde{S}\left(k\right)\cdot {\tilde{H}}_{de}\left(k\right)+r_1{e}^{i{\varphi }_1}\tilde{S}\left(k-k_0\right)\cdot {\tilde{H}}_{de}\left(k\right)+r_2{e}^{i{\varphi }_2}\tilde{S}\left(k+k_0\right)\cdot {\tilde{H}}_{de}\left(k\right).$$ (15)

To obtain a better approximation between ${\psi }_{\mathit{\text{peak}}}$ and ${\varphi }_1$, ${\tilde{R}}_2\left(k\right)$ can be updated before the phase-only correlation and estimation of ${\psi }_{\mathit{\text{peak}}}$ by eliminating the widefield component:

$${\tilde{R}}_2^{\mathit{\text{updated}}}={\tilde{R}}_2-w_C\cdot \frac{1}{n}\cdot {\displaystyle \sum _{i=1}^n{\tilde{D}}_i}.$$ (16)

Once ${\tilde{R}}_2\left(k\right)$ has been updated, Eqs. (8) and (13) can be applied to solve Eq. (13). Since there are up to six independent arguments in the inverse matrix $M^{-1}\left({\sigma }_1,{\sigma }_2,{\sigma }_3;{\gamma }_1,{\gamma }_2,{\gamma }_3\right)$, it is feasible to obtain six independent equations in the form given by Eq. (13). In addition, the equation set can be considered to be a linear equation set that is consistent with the Pythagorean theorem. In this case, no redundant answer is obtained as the constraints are sufficiently robust. However, it may be computationally expensive and time-consuming to manipulate six equations because this requires calculating additional correlations in order to obtain the phase ${\psi }_{\mathit{\text{peak}}}$. Here, four independent equations are used to convert the original problem into two trigonometric equations with only two unknown angles (phases). To eliminate the redundant answers from the answer set, a cost function is introduced based on Eq. (11):

$$g\left({\phi }_1^{\mathit{\text{ill}}},{\phi }_2^{\mathit{\text{ill}}},{\phi }_3^{\mathit{\text{ill}}}\right)={\displaystyle \sum _t\left|\frac{b_1{e}^{i{\phi }_1^{ill}+i{\beta }_1}+b_2{e}^{i{\phi }_2^{ill}+i{\beta }_2}+b_3{e}^{i{\phi }_3^{ill}+i{\beta }_3}}{\left|b_1{e}^{i{\phi }_1^{\mathit{\text{ill}}}+i{\beta }_1}+b_2{e}^{i{\phi }_2^{\mathit{\text{ill}}}+i{\beta }_2}+b_3{e}^{i{\phi }_3^{\mathit{\text{ill}}}+i{\beta }_3}\right|}-e^{i{\psi }_{peak}}\right|},$$ (17)

where the subscript t denotes the t-th inverse matrix $M_t^{-1}$, and the other parameters with the exception of ${\phi }_i^{\mathit{\text{ill}}}$ depend on the inverse matrix, or t. The correct phases are then obtained by minimizing g over the obtained answer sets.

Due to the noise in the captured image, it is likely that the sine and cosine values of the phase ${\phi }_s^{\mathit{\text{ill}}}$(s = 1/2/3), which was replaced by the above substitution, does not obey the Pythagorean theorem. For this reason, the phase ${\phi }_s^{\mathit{\text{ill}}}$ was estimated via the following equation:

$${\phi }_s^{\mathit{\text{ill}}}={\text{sin}}^{-1}\left[\frac{{\left(\text{sin}{\phi }_s^{ill}\right)}_{est}}{\sqrt{{\left(\text{sin}{\phi }_s^{\mathit{\text{ill}}}\right)}_{est}^2+{\left(\text{cos}{\phi }_s^{\mathit{\text{ill}}}\right)}_{est}^2}}\right]$$ (18)

where the subscript “est” denotes the estimated values of $\text{sin}{\phi }_s^{\mathit{\text{ill}}}$ and $\text{cos}{\phi }_s^{\mathit{\text{ill}}}$, which were calculated from the other two phases.

3. Simulations and experiments

In order to verify the above algorithm, a simulation was conducted in which different numbers of photons were emitted from a particular object, and the resulting phase was estimated using the proposed inverse matrix based algorithm, the cross-correlation and the auto-correlation algorithm. The object in question was a Siemens star (see the ground truth discussion below) and the spatial frequency of the illumination pattern was $1.1k_c$, which is larger than the detection cutoff frequency and feasible due to the Stokes shift of the fluorescence light or the surface plasmonic wave. The designated phase shift was $2\pi /3$ and the uniformly distributed phase error was in the range [-0.3, 0.3].

The inverse matrixes employed in the simulations and experiments in this paper were $M_1^{-1}\left(1.1,1.1,0.9;0,\frac{2}{3}\pi ,\frac{4}{3}\pi \right)$, $M_2^{-1}\left(1.2,0.9,0.8;0,\frac{1}{3}\pi ,\frac{2}{3}\pi \right)$, $M_3^{-1}\left(1.15,0.85,1.3;0,\frac{4}{7}\pi ,\frac{10}{7}\pi \right)$, and $M_4^{-1}\left(0.8,1.2,1.3;0,\frac{4}{5}\pi ,\frac{8}{5}\pi \right)$ as these parameters were the most promising among the six different sets evaluated.

The average phase errors of the inverse matrix based, cross-correlation, and auto-correlation algorithms for different modulation depths are shown in Fig. 2. The program used to implement these algorithms was written in MATLAB and executed on a personal computer with an Intel Core i5-4200H 2.8 GHz processor. The run-times of the inverse matrix based, cross-correlation, and auto-correlation methods were about 0.27, 0.33, and 0.18 s, respectively, when estimating three phases for an image size of 513 × 513 pixels. In an ideal system, the performance the three algorithms is quite similar, as shown in Fig. 2(a), while the auto-correlation method consumes less computation time. Thus, in this case, the auto-correlation method is a good choice for reconstruction.

Fig. 2.

Simulations of the proposed inverse matrix based phase estimation algorithm versus the auto-correlation and cross-correlation algorithms. The period of the illumination pattern was 195 nm and the minimum detectable period in this simulation was 214 nm. As the radial pattern consisted of certain frequencies, the auto-correlation algorithm failed to retrieve the correct phase under low modulation depth conditions even though the number of photons was large. The value of m in the lower left of each subfigure demotes the modulation depth. In this simulation, Poisson noise was introduced, the minimum detectable number of photons was 10, and 24 phases were simulated when calculating the mean phase error.

However, when the modulation depth is lower than expected, the inverse matrix and iterative cross-correlation based algorithms are better than the auto-correlation method. In this case, the execution time of the inverse matrix based phase estimation method was 20% shorter than that of the iterative method.

If the non-linear effect is utilized, 2N + 1 images are required to cover the N-th harmonic [16–18]. In this case, the images can be divided into small groups of three images each (there may be duplicate images in the different subgroups). Once the true phases of the illumination patterns have been estimated, the modulation depth estimation step can be improved because it is also based on the correlation result in the correlation-based estimation algorithms [21,25].

The parameters that are required for SIM reconstruction are obtained in the above step. The image process was simulated using a standard object that was the same as that in Fig. 2, in which the maximum phase error was 0.6 rad. As shown in Fig. 3(f), there was no redundant component as the phases of the illumination pattern were the correct ones as per the proposed method. The modulation depth in Figs. 3(a)–(f) was 0.6. Apart from the estimated phases, the other aspects of the reconstruction method remained the same and the reconstruction result shown in Fig. 3(c) is similar to the ground truth image illustrated in Fig. 3(a). However, the stripes in Fig. 3(b) appear broken as a result of the existing redundant component.

Fig. 3.

(a) Simulation of the ground truth, (b) uncorrected reconstruction, and (c) phase corrected result using the inverse matrix based algorithm. The Fourier transform of the reconstructed images (b) and (c) are shown in (d). The green background pattern in (d)–(f) is the Fourier transform of the ground truth sample. (e) and (f) show the corresponding areas in (d), and the white dashed square in (e) highlights part of the redundant Fourier component that appears due to the error in the phase. The reconstruction results using (g) the auto-correlation, (h) cross-correlation, and (i) inverse matrix based method employed the same set of data that was also applied in Fig. 2 (m = 0.3, number of photons = 3162, three images were used).

To demonstrate the reliability of the inverse matrix based algorithm, the image process was simulated with a low modulation depth, after which the image was reconstructed using the above three methods. The results are shown in Figs. 3(g)–(i). When the modulation depth was m = 0.3, one set of phases (the second pattern orientation) estimated using the auto-correlation method was completely incorrect. Thus, the corresponding reconstruction result, which is shown in Fig. 3(g), is worse than the results of the other two phase estimation methods.

A home-made TIRF-SIM system [26] was constructed to verify the inverse matrix based phase estimation algorithm. In the experiment, 100 nm diameter yellow-green fluorescent beads (excitation wavelength: 505 nm/emission wavelength: 515 nm, Model F8803, Life Technologies) were imaged and the results are shown in Fig. 4. The images were captured using a high numerical aperture TIRF objective (100 × / 1.49 TIRF, Nikon) and the results were reconstructed using the inverse matrix based phase estimation algorithm. All aspects of the code used for the simulation remained unchanged, except for the phase estimation method. The data used to obtain Fig. 1 is from the same data set as that in Fig. 4. As shown in the figure, the redundant component that appeared in Fig. 1 disappeared in this figure. This indicates that the phases estimated by the inverse matrix algorithm were the correct phases. It can be seen in the image and subfigures in Fig. 4(c) that the beads are less distorted than those in Fig. 4(b). The contrast in Fig. 4(c) is also better. The brightness of each bead in the phase corrected results are similar, which is highly consistent with the prior knowledge, while the brightness fluctuates in the results without phase correction.

Fig. 4.

Experimental results: (a) the filtered widefield results, (b) the reconstructed results without correction, and (c) the results with phase correction via the inverse matrix based method (note the brightness was increased by 40% to improve the visibility). (d) The phase-only correlation results of the reconstructed image. Lower left: without correction, and upper right: with correction. The intensity profile along the cutline indicated by the arrows in (a)–(c) is shown in (e) using the results without brightness enhancement.

Microtubule was also imaged and reconstructed via the three phase estimation algorithm. The sample material was Alexa Fluor 647 (excitation wavelength: 650 nm / emission wavelength: 665 nm). The sample preparation and staining procedures were as follows.

COS-7 (African green monkey kidney fibroblast-like cell line) cells were purchased from Boster Biological Technology Co., Ltd., Wuhan, China, and cultured in Dulbecco's Modified Eagle's Medium (DMEM; Thermo Fisher Scientific, Inc.). All media were supplemented with 10% (v/v) fetal bovine serum (FBS; Thermo Fisher Scientific, Inc.), and were kept at 37°C.

COS-7 cells were seeded in Nunc Glass Bottom Dishes (Φ 12 mm, Thermo Fisher Scientific, Inc.) at a density of 1.5-2.0 × 104 per well in growth medium (150 µL).

Antibodies were labeled using the manufacturers’ protocols. In brief, 10 µl of 1 M aqueous NaHCO3 (pH 8.0) was added to 50–100 µg of antibody to a final antibody concentration of ~2 µM. Alexa Fluor 647 N-hydroxysuccinimidyl ester (Invitrogen) were dissolved in dimethyl sulfoxide and added to antibody solutions at a final concentration of ~4 µM to achieve the desired dye to protein ratios. Labeled antibodies were purified by gel filtration using NAP-5 columns (GE Healthcare).

The immunostaining procedure consisted of fixation for 10 min with 3% paraformaldehyde and 0.1% glutaraldehyde in PBS, washing with PBS, reduction for 5 min with 0.1% sodium borohydride in PBS to reduce background fluorescence, washing with PBS, blocking for 30 min with 3% bovine serum albumin and 0.25% (v/v) Triton X-100 in PBS (blocking buffer), staining for 45 min with rat primary antibody to tubulin (Abcam) diluted in blocking buffer to a concentration of 10 µg ml−1, washing with PBS, incubation for 45 min with secondary antibodies (~1–2 dyes per antibody; Jackson ImmunoResearch Laboratories) at a concentration of ~2.5 µg ml−1 in blocking buffer, washing with PBS, fixation after labeling for 10 min with 3% paraformaldehyde and 0.1% glutaraldehyde in PBS, and finally washing with PBS.

As shown in Fig. 5(e), reconstructed results using the cross-correlation and inverse matrix based phase estimation methods were better than those using the auto-correlation method due to the accuracy of the estimated phases.

Fig. 5.

Filtered widefield result (a), SIM results with the auto-correlation (b), cross-correlation (c), and inverse matrix (d) based phase estimation methods. The profile along the line segment in each subfigure is shown in (e). The profile was normalized by the corresponding maximum along each line segment. COS-7 cells were used in this experiment. Abbreviation: WF, widefield; AC, auto-correlation method; CC, cross-correlation method; IM, inverse matrix based method.

4. Conclusions

In this paper, a phase-only correlation method was introduced to optimize the modulation spatial frequency vector $k_0$. As the estimation of the modulation frequency is the first step in the reconstruction process, the correct modulation frequency vector $k_0$ is the basis of all subsequent steps. This phase-only correlation method outperforms previous methods and significantly enhances the peak signal located at $k_0$.

By analyzing the inverse matrix and the outcomes of the separation steps, an inverse matrix based phase estimation algorithm was developed. Unlike the cross-correlation phase optimization algorithm, which uses iteration when determining the correct phase, the inverse matrix based algorithm does not require iteration and cannot be trapped in a local minimum. Thus, by way of comparison, the inverse matrix algorithm was demonstrated to be a strong competitor to the auto-correlation algorithm. While the performance of the inverse matrix method is similar to the cross-correlation method, the proposed method is faster, and the two methods are both better than the auto-correlation method when the modulation depth is low.

Although the inverse matrix based phase estimation method has the potential to be widely used, more work is required. While a correct set of phases is essential for high quality reconstruction, it should be noted that the quality of the reconstruction is not solely dependent on the phases as the SNR of the image is also important. Thus, the use of an improved noise reduction algorithm would improve the reliability of the reconstruction results and reduce the phase estimation errors. In addition, as the parameters of the inverse matrix $M^{-1}\left({\sigma }_1,{\sigma }_2,{\sigma }_3;{\gamma }_1,{\gamma }_2,{\gamma }_3\right)$ affect the errors of the phase estimation algorithm, the selection of an appropriate set of parameters has the potential to reduce the average phase error in the inverse matrix based method. These topics can be explored in future research.

Funding

National Basic Research Program of China (973Program) (2015CB352003); National Natural Science Foundation of China (NSFC) (61735017, 61335003, 61427818, 61827825); Natural Science Foundation of Zhejiang province (LR16F050001); and the Fundamental Research Funds for the Central Universities (2017FZA5004).

Disclosures

The authors declare that there are no conflicts of interest related to this article.