Speckle-structured illumination for 3D phase and fluorescence computational microscopy

Abstract

High-content biological microscopy targets high-resolution imaging across large fields-of-view, often achieved by computational imaging approaches. Previously, we demonstrated 2D multimodal high-content microscopy via structured illumination microscopy (SIM) with resolution $>2\times$ the diffraction limit, using speckle illumination from Scotch tape. In this work, we extend the method to 3D by leveraging the fact that the speckle illumination is in fact a 3D structured pattern. We use both a coherent and an incoherent imaging model to develop algorithms for joint retrieval of the 3D super-resolved fluorescent and complex-field distributions of the sample. Our reconstructed images resolve features beyond the physical diffraction-limit set by the system’s objective and demonstrate 3D multimodal imaging with $\sim 0.6\times 0.6\times 6\mu$m³ resolution over a volume of $\sim 314\times 500\times 24\mu$m³.

1. Introduction

High-content optical microscopy is a driving force for large-scale biological study in fields such as drug discovery and systems biology. With fast imaging speeds over large fields-of-view (FOV) and high spatial resolutions [1–8], one can visualize rare cell phenotypes and dynamics. The traditional solution for 2D high-content microscopy is to mechanically scan samples through the limited FOV of a high-NA (i.e. high resolution) imaging objective and then digitally stitch the images together. However, this scheme is limited in imaging speed due to the large-distance translations of the sample, as well as the need for auto-refocusing at each position [9]. These issues are further compounded when extending this high-content imaging strategy to 3D.

Recently, computational imaging has demonstrated efficient strategies for high-content 2D microscopy. In contrast with slide scanning, these strategies often employ a low-NA imaging objective to acquire low-resolution (large-FOV) measurements, then use computational techniques like synthetic aperture [10–12] and super-resolution (SR) [13–18] to digitally reconstruct a high-resolution image. This eliminates the requirement for large-distance mechanical scanning in high-content imaging, which results in faster acquisition and more cost-effective optical setups, while also relaxing the sample’s auto-refocusing requirements due to the low-NA objective’s longer depth-of-field (DOF) [19–36]. Examples of such approaches include lensless microscopy [19–21] and Fourier ptychography [22–28] for coherent absorption and quantitative phase imaging. For incoherent fluorescent imaging, micro-lenslet arrays [29–32], Talbot plane scanning [33–35], diffuse media [36], or meta-surfaces [37] have also been demonstrated. Among these examples, 3D high-content imaging capability has only been demonstrated in the coherent imaging context (quantitative phase and absorption) by Fourier ptychography [25, 27].

Our previous work demonstrated multimodal coherent (quantitative phase) and incoherent (fluorescence) imaging for high-content 2D microscopy [38]. Multimodal imaging is important for biological studies requiring cross-correlative analysis [39–43]. Structured illumination microscopy (SIM) [10, 16, 17, 44] with speckle illumination [36, 45–53] was used to encode 2D SR quantitative phase and fluorescence. However, because propagating speckle contains 3D features, it also encodes 3D information. Considering speckle patterns as random interference of multiple angled plane waves, the scattered light from interactions with the sample carries 3D phase (coherent) information, similar to the case of non-random angled illumination in diffraction tomography [54–57] and 3D Fourier ptychography [25, 27]. Simultaneously, the fluorescent (incoherent) light excited by the 3D speckle pattern encodes 3D SR fluorescence information as in the case of 3D SIM [58]. Combining these, we propose a method for 3D SR quantitative phase and fluorescence microscopy using speckle illumination.

Experimentally, we position a Scotch tape patterning element just before the sample, mounted on a translation stage to generate a translating speckle field that illuminates the sample (Fig. 1). Because the speckle grain size is smaller than the PSF of the low-NA imaging objective (which provides large-FOV), the coherent scattered light from the speckle-sample interaction encodes 3D SR quantitative phase information. In addition to lateral scanning of the Scotch tape, axial sample scanning is necessary to efficiently capture 3D SR fluorescence information. Nonlinear optimization methods based on the 3D coherent beam propagation model [25, 59–61] and the 3D incoherent imaging model [58] were formulated to reconstruct the 3D speckle field and imaging system aberrations, which are subsequently used to reconstruct the sample’s 3D SR quantitative phase and fluorescence distributions. Since the Scotch tape is directly before the sample, the illumination NA is not limited by the objective lens, allowing for $>2\times$ lateral resolution gain across the entire FOV. This framework enables us to achieve 3D imaging at sub-micron lateral resolution and micron axial resolution across a half-millimeter FOV.

Fig. 1.

3D multimodal structured illumination microscopy (SIM) with laterally translating Scotch tape as the patterning element. The coherent arm (Sensor-C1 and Sensor-C2) simultaneously captures images with different defocus at the laser illumination wavelength (λ_ex = 532 nm), used for both 3D phase retrieval and speckle trajectory calibration. The incoherent (fluorescence) arm (Sensor-F) captures low-resolution raw fluorescence acquisitions at the emission wavelength (λ_em = 605 nm) for 3D fluorescence super-resolution reconstruction. OBJ: objective, DM: dichroic mirror, SF: spectral filter, ND-F: neutral-density filter.

2. Theory

We start from the concept of 3D coherent and incoherent transfer functions (TFs), using the Born (weak scattering) assumption [54], to analyze the information encoding process. We then lay out our 3D coherent and incoherent imaging models and derive the corresponding inverse problems to extract SR quantitative phase and fluorescence from the measurements.

First, we introduce linear space-invariant relationships between raw measurements and 3D coherent scattering and incoherent fluorescence [54, 58, 62, 63], by invoking the Born (weak scattering) approximation [54]. These relationships enable us to define TFs for the coherent and incoherent imaging processes. The supports of these TFs in 3D Fourier space determine how much spatial frequency content of the sample can be passed through the system (i.e. the 3D diffraction-limited resolution).

In a coherent imaging system with on-axis plane-wave illumination, the TF describes the relationship between the sample’s scattering potential and the measured 3D scattered field, taking the shape of a spherical cap in 3D Fourier space (Fig. 2(a)). In an incoherent imaging system, the TF is the autocorrelation of the coherent system’s TF [63], relating the sample’s fluorescence distribution to the 3D measured intensity. It takes the shape of a torus (Fig. 2(b)). The spatial frequency bandwidth of these TFs are summarized in Table 1, where the lateral resolution of the system is proportional to the lateral bandwidth of the TF. The incoherent TF has 2× greater lateral bandwidth than the coherent TF. Axial bandwidth generally depends on the lateral spatial frequency, so axial resolution is specified in terms of the best-case. Note that the axial bandwidth of the coherent TF is zero, which means there is zero axial resolution for coherent imaging; hence the poor depth sectioning ability in 3D holographic imaging [41, 56, 64].

Fig. 2.

3D coherent and incoherent transfer function (TF) analysis of the SIM imaging process. The 3D (a) coherent and (b) incoherent TFs of the detection system are auto correlated with the 3D Fourier support of the (c) illumination speckle field and (d)illumination intensity, respectively, resulting in the effective Fourier support of 3D (e) coherent and (f) incoherent SIM. In (e) and (f), we display decomposition of the auto-correlation in two steps: ① tracing the illumination support in one orientation and ② replicating this trace in the azimuthal direction.

Table 1.

Summary of spatial frequency bandwidths

	Lateral bandwidth	Axial bandwidth
Coherent TF	$\frac{2\mathrm{N}{\mathrm{A}}_{\mathrm{d}\mathrm{e}\mathrm{t}}}{{\lambda }_{\mathrm{e}\mathrm{x}}}$	0
Incoherent TF	$\frac{4\mathrm{N}{\mathrm{A}}_{\mathrm{d}\mathrm{e}\mathrm{t}}}{{\lambda }_{\mathrm{e}\mathrm{m}}}$	$\frac{2\left(1-\mathrm{c}\mathrm{o}\mathrm{s} {\theta }_{\mathrm{d}\mathrm{e}\mathrm{t}}\right)}{{\lambda }_{\mathrm{e}\mathrm{m}}}$
Illum. field	$\frac{2\mathrm{N}{\mathrm{A}}_{\mathrm{i}\mathrm{l}\mathrm{l}\mathrm{u}\mathrm{m}}}{{\lambda }_{\mathrm{e}\mathrm{x}}}$	0
Illum. intensity	$\frac{4\mathrm{N}{\mathrm{A}}_{\mathrm{i}\mathrm{l}\mathrm{l}\mathrm{u}\mathrm{m}}}{{\lambda }_{\mathrm{e}\mathrm{x}}}$	$\frac{2\left(1-\mathrm{c}\mathrm{o}\mathrm{s} {\theta }_{\mathrm{i}\mathrm{l}\mathrm{l}\mathrm{u}\mathrm{m}}\right)}{{\lambda }_{\mathrm{e}\mathrm{x}}}$
3D coherent SIM	$\frac{2\mathrm{N}{\mathrm{A}}_{\mathrm{d}\mathrm{e}\mathrm{t}}+2\mathrm{N}{\mathrm{A}}_{\mathrm{i}\mathrm{l}\mathrm{l}\mathrm{u}\mathrm{m}}}{{\lambda }_{\mathrm{e}\mathrm{x}}}$	$\frac{1-\mathrm{c}\mathrm{o}\mathrm{s} {\theta }_{\mathrm{d}\mathrm{e}\mathrm{t}}}{{\lambda }_{\mathrm{e}\mathrm{x}}}+\frac{1-\mathrm{c}\mathrm{o}\mathrm{s} {\theta }_{\mathrm{i}\mathrm{l}\mathrm{l}\mathrm{u}\mathrm{m}}}{{\lambda }_{\mathrm{e}\mathrm{x}}}$
3D incoherent SIM	$\frac{4\mathrm{N}{\mathrm{A}}_{\mathrm{d}\mathrm{e}\mathrm{t}}}{{\lambda }_{\mathrm{e}\mathrm{m}}}+\frac{4\mathrm{N}{\mathrm{A}}_{\mathrm{i}\mathrm{l}\mathrm{l}\mathrm{u}\mathrm{m}}}{{\lambda }_{\mathrm{e}\mathrm{x}}}$	$2\left(\frac{1-\mathrm{c}\mathrm{o}\mathrm{s} {\theta }_{\mathrm{d}\mathrm{e}\mathrm{t}}}{{\lambda }_{\mathrm{e}\mathrm{m}}}+\frac{1-\mathrm{c}\mathrm{o}\mathrm{s} {\theta }_{\mathrm{i}\mathrm{l}\mathrm{l}\mathrm{u}\mathrm{m}}}{{\lambda }_{\mathrm{e}\mathrm{x}}}\right)$

${\text{NA}}_{\det },{\text{NA}}_{\mathrm{illum}}$: the numerical aperture ($\sin \theta$) of the detection and illumination system,

${\theta }_{\text{det}},{\theta }_{\text{illum}}$: the maximal detectable and illuminating half angle of light,

${\lambda }_{\text{ex}},{\lambda }_{\text{em}}$: the wavelength of the excitation and emission light.

SIM enhances resolution by creating beat patterns. When a 3D structured pattern modulates the sample, the sample’s sub-diffraction features create lower-frequency beat patterns which can be directly measured and used to reconstruct a SR image of the sample via post-processing [17, 58]. This process is generally applicable to both coherent and incoherent imaging [40–43], enabling 3D SR multimodal imaging. Mathematically, a modulation between the sample contrast and the illumination pattern in real space can be interpreted as a convolution in Fourier space. This convolution result is then passed through the 3D TF defined in Fig. 2(a,b). The effective support of information going into the measurements can be estimated by conducting cross-correlations between the 3D TFs and the Fourier content of the illumination patterns, as shown in Fig. 2(c,e) and 2(d,f) for coherent and incoherent systems, respectively. The lateral and axial spatial frequency bandwidth of both illumination and 3D SIM Fourier supports for coherent and incoherent imaging are summarized in Table 1. Assuming approximately equal excitation and emission wavelengths, the achievable lateral resolution gain of 3D SIM (ratio between lateral bandwidths of 3D SIM and 3D TF) is $\left({\text{NA}}_{\det }+{\text{NA}}_{\mathrm{illum}}\right)/{\text{NA}}_{\det }$ for both coherent and incoherent imaging. Axially, coherent SIM builds up the spatial frequency bandwidth in the axial direction, and incoherent SIM can achieve axial resolution gain with a factor of $\left(2-\cos {\theta }_{\text{det}}-\cos {\theta }_{\text{illum}}\right)/\left(1-\cos {\theta }_{\text{det}}\right)$.

In this work, because the Scotch tape does not pass through an objective, it is able to create high-resolution speckle illumination such that ${\text{NA}}_{\mathrm{illu}}>{\text{NA}}_{\det }$, enabling $>2\times$ lateral resolution gain without sacrificing FOV [38]. From the TF analysis, we also see that information beyond diffraction-limit in the axial dimension is obtainable. The next sections outline our computational scheme for 3D SR phase and fluorescence reconstruction. To provide higher-quality reconstructions and more robust operation, our algorithm jointly estimates the illumination speckle field, system pupil function (aberrations), the sample’s 3D transmittance function, and the sample’s 3D fluorescence distribution.

2.1. 3D super-resolution phase imaging

We adopt a multi-slice coherent scattering model to describe the 3D multiple-scattering process [25, 59–61] and solve for 3D SR quantitative phase. Our system captures intensity at two focus planes, $z_{c1}$ and $z_{c2}$, for every speckle-scanned point [38]. With these measurements and the multi-slice model, we are able to reconstruct the sample’s 3D SR complex-field and the scattered field inside the 3D sample, which is used in the fluorescence inverse problem.

2.1.1. Forward model for 3D coherent imaging

Figure 3(a) illustrates the 3D multi-slice coherent imaging model. Plane-wave illumination of the Scotch tape, positioned at the $\mathcal{l}$-th scanned point, ${\mathbf{r}}_{\mathcal{l}}$, creates speckle field $p_c\left(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}}\right)$, where $\mathbf{r}=\left(x,y\right)$ is the lateral spatial coordinate. This speckle field propagates a distance $\mathrm{\Delta }{s}_{\mathcal{l}}$ to the sample. The field interacting with the first layer of the sample is described as:

$$f_{\mathcal{l},1}\left(\mathbf{r}\right)=\mathcal{C}\{p_c(\mathbf{r}-{\mathbf{r}}_{\mathcal{l}})\otimes h_{\mathrm{\Delta }{s}_{\mathcal{l}},{\lambda }_{\text{ex}}}(\mathbf{r})\},$$ (1)

where $h_{z,\lambda }\left(\mathbf{r}\right)={\mathcal{F}}^{-1}\left\{\text{exp}\left(i2\pi z\sqrt{n_0^2/{\lambda }^2-\Vert \mathbf{u}{\Vert }_2^2}\right)\right\}$ is the angular spectrum propagation kernel inside a homogeneous media with refractive index n₀[65] $\mathbf{u}=\left(u_x,u_y\right)$ is the spatial frequency coordinate, and $\mathcal{C}\{\cdot \}$ is a cropping operator that selects the part of the speckle field that illuminates the sample. To model scattering and propagation inside the sample, the multi-slice model treats the 3D sample as multiple slices of complex transmittance function, $t_m\left(\mathbf{r}\right)$ ($m=1,\cdots ,M$), where m is the slice index number. As the field propagates through each slice, it first multiplies with the 2D transmittance function at that slice, then propagates to the next slice. The spacing between slices is modeled as uniform media of thickness $\mathrm{\Delta }{z}_m$. Hence, at each layer we have:

$$\begin{array}{l}{g}_{\mathcal{l},m}\left(\mathbf{r}\right)=f_{\mathcal{l},m}\left(\mathbf{r}\right)\cdot t_m\left(\mathbf{r}\right),m=1,\cdots ,M,\\ f_{\mathcal{l},m+1}\left(\mathbf{r}\right)=g_{\mathcal{l},m}\left(\mathbf{r}\right)\otimes h_{\mathrm{\Delta }{z}_m,{\lambda }_{\text{ex}}}\left(\mathbf{r}\right),m=1,\cdots ,M-1.\end{array}$$ (2)

Fig. 3.

3D multi-slice model: (a) coherent and (b) incoherent imaging models for the interaction between the sample and the speckle field as light propagates through the sample.

After passing through all the slices, the output scattered field, $g_{\mathcal{l},M}\left(\mathbf{r}\right)$, propagates to the focal plane to form $G_{\mathcal{l}}\left(\mathbf{r}\right)=g_{\mathcal{l},M}\left(\mathbf{r}\right)\otimes h_{\mathrm{\Delta }{z}_{M,\mathcal{l}},{\lambda }_{\text{ex}}}\left(\mathbf{r}\right)$ and gets imaged onto the sensor (with defocus z), forming our measured intensity:

$$I_{c,\mathcal{l}z}\left(\mathbf{r}\right)={\left|G_{\mathcal{l}}\left(\mathbf{r}\right)\otimes h_c\left(\mathbf{r}\right)\otimes h_{z,{\lambda }_{\text{ex}}}\left(\mathbf{r}\right)\right|}^2,\mathcal{l}=1,\cdots ,N_{\text{img}},z=z_{c1},z_{c2},$$ (3)

where $h_c\left(\mathbf{r}\right)$ is the system’s coherent point spread function (PSF). The measured intensity subscripts c and $\mathcal{l}$ denote indices for the coherent imaging channel and acquisition number, respectively. N_img is the total number of translations of the Scotch tape. Note that all the spacing distances, $\mathrm{\Delta }{z}_m$, are independent of the axial scanned position, $\mathrm{\Delta }{s}_{\mathcal{l}}$, except for the distance to the focal plane, which is $\mathrm{\Delta }{z}_{M,\mathcal{l}}=\mathrm{\Delta }{s}_{\mathcal{l}}+z_0$, where z₀ is the distance from the last layer of the sample to the focal plane (before axial scanning). As the sample is scanned, we account for this shift by propagating extra distance back to the focal plane.

2.1.2. Inverse problem for 3D coherent imaging

We take the intensity measurements from both coherent cameras, $\left\{I_{c,\mathcal{l}z}\left(\mathbf{r}\right)\right|z=z_{c1},z_{c2}\}$, and the scanning trajectory, ${\mathbf{r}}_{\mathcal{l}}$ (calculated via standard rigid-body 2D registration [38, 66]), as inputs to jointly estimate the sample’s 3D SR transmittance function, $t_1\left(\mathbf{r}\right),\cdots ,t_M\left(\mathbf{r}\right)$, as well as the illumination complex-field, $p_c\left(\mathbf{r}\right)$, and the system’s coherent PSF, $h_c\left(\mathbf{r}\right)$, including aberrations.

Based on the forward model in the previous section, we formulate the inverse problem as:

$$\begin{array}{l}\underset{t_1,\cdots ,t_M,p_c,h_c}{\text{minimize}}{e}_c\left(t_1,\cdots t_M,p_c,h_c\right)={\displaystyle \sum }_{\mathcal{l},z}{e}_{c,\mathcal{l}z}\left(t_1,\cdots ,t_M,p_c,h_c\right)\\ \text{where}{e}_{c,\mathcal{l}z}\left(t_1,\cdots ,t_M,p_c,h_c\right)={\displaystyle \sum }_{\mathbf{r}}{\left|\sqrt{I_{c,\mathcal{l}z}\left(\mathbf{r}\right)}-\left|G_{\mathcal{l}}\left(\mathbf{r}\right)\otimes h_c\left(\mathbf{r}\right)\otimes h_{z,{\lambda }_{\text{ex}}}\left(\mathbf{r}\right)\right|\right|}^2.\end{array}$$ (4)

Here we adopt an amplitude-based cost function, e_c, which minimizes the difference between the measured and estimated coherent amplitude in the presence of noise [67]. In order to solve this optimization problem, we use a sequential gradient descent algorithm [67, 68]. The gradient based on each single measurement is calculated and used to update the sample’s transmittance function, illumination speckle field, and coherent PSF. A whole iteration of variable updates is complete after running through all the measurements. In Appendix A, we provide a detailed derivation of the gradients and in Appendix B we lay out our reconstruction algorithm.

2.2. 3D super-resolution fluorescence imaging

Reconstruction of 3D SR images for the fluorescence channel involves an incoherent multi-slice forward model (Fig. 3(b)) and a joint inverse problem solver. The coherent result provides a good starting estimate of the 3D speckle intensity throughout the sample, which, together with the fluorescent channel’s raw data, is used to reconstruct the sample’s 3D SR fluorescence distribution and the system’s aberrations at the emission wavelength, ${\lambda }_{\text{em}}$.

2.2.1. Forward model for 3D fluorescence imaging

The 3D fluorescence distribution is also modeled by multiple slices of 2D distributions, $o_m\left(\mathbf{r}\right)$ ($m=1,\cdots ,M$), as shown in Fig. 3(b). Each layer is illuminated by the m-th layer’s excitation intensity, ${\left|f_{\mathcal{l},m}\left(\mathbf{r}\right)\right|}^2$, for Scotch tape position ${\mathbf{r}}_{\mathcal{l}}$. The excited fluorescent light is mapped onto the sensor through 2D convolutions with the incoherent PSF at different defocus distances, $z_{m,\mathcal{l}}$. The sum of contributions from different layers form the measured fluorescence intensity:

$$I_{f,\mathcal{l}}\left(\mathbf{r}\right)={\displaystyle \sum }_{m=1}^M\left[o_m\left(\mathbf{r}\right)\cdot {|f_{\mathcal{l},m}\left(\mathbf{r}\right)|}^2\right]\otimes {|h_{f,z_{m,\mathcal{l}}}\left(\mathbf{r}\right)|}^2,\mathcal{l}=1,\cdots ,N_{\text{img}},$$ (5)

where $h_{f,z_{m,\mathcal{l}}}\left(\mathbf{r}\right)$ is the coherent PSF at defocus distance $z_{m,\mathcal{l}}$, which could be further decomposed into $h_{f,z_{m,\mathcal{l}}}\left(\mathbf{r}\right)=h_f\left(\mathbf{r}\right)\otimes h_{z_{m,\mathcal{l}},{\lambda }_{\text{em}}}\left(\mathbf{r}\right)$, where $h_f\left(\mathbf{r}\right)$ is the in-focus system’s coherent PSF at ${\lambda }_{\text{em}}$. The incoherent PSF is the intensity of the coherent PSF at ${\lambda }_{\text{em}}$. The subscript f denotes the fluorescence channel and the defocus distance, $z_{m,\mathcal{l}}$, depends on the axial scanning position, $\mathrm{\Delta }{s}_{\mathcal{l}}$.

2.2.2. Inverse problem for 3D fluorescence imaging

The fluorescence inverse problem takes as input the raw fluorescence intensity measurements, $I_{f,\mathcal{l}}\left(\mathbf{r}\right)$, the registered scanning trajectory, ${\mathbf{r}}_{\mathcal{l}}$, and the 3D estimates from the coherent model, in order to estimate the sample’s 3D SR fluorescence distribution and aberrations at the emission wavelength. We also refine the speckle field estimate using the fluorescence measurements.

Based on the incoherent forward model, our 3D SR fluorescence inverse problem is:

$$\begin{array}{l}\underset{o_1,\cdots ,o_M,p_c,h_f}{\text{minimize}}{e}_f\left(o_1,\cdots o_M,p_c,h_f\right)={\displaystyle \sum }_{\mathcal{l}}{e}_{f,\mathcal{l}}\left(o_1,\cdots ,o_M,p_c,h_f\right)\\ \text{where}{e}_{f,\mathcal{l}}\left(o_1,\cdots ,o_M,p_c,h_f\right)={\displaystyle \sum }_{\mathbf{r}}{\left|I_{f,\mathcal{l}}\left(\mathbf{r}\right)-{\displaystyle \sum }_{m=1}^M\left[o_m\left(\mathbf{r}\right)\cdot {|f_{\mathcal{l},m}\left(\mathbf{r}\right)|}^2\right]\otimes {|h_{f,z_{m,\mathcal{l}}}\left(\mathbf{r}\right)|}^2\right|}^2,\end{array}$$ (6)

where e_f is the cost function. Similar to the coherent inverse problem, we adopt a sequential gradient descent algorithm for estimation of each unknown variable. The detailed derivation of gradients and algorithm implementation are summarized in Appendix A and B, respectively.

3. Experimental results

Figure 1 shows the experimental setup. A green laser beam (BeamQ, 532 nm, 200 mW) is collimated through a single lens and illuminates the layered Scotch tape element, creating a speckle pattern at the sample. The number of layers of Scotch tape sets the degree of scattering; we use 16 layers here. The layered Scotch tape and the sample are mounted on a 3-axis closed-loop piezo-stage (Thorlabs, MAX311D) and a 1-axis open-loop piezo-stage (Thorlabs, NFL5DP20), respectively, to enable lateral speckle scanning and axial sample scanning. The separation between the tape and the sample is approximately 1 mm, which is the minimal distance we can achieve for high-angle and high-power illumination without physically touching the sample. The transmitted diffracted and fluorescent light from the sample then travels through the subsequent 4f system formed by the objective lens (Nikon, CFI Achro 20×, NA=0.4) and a tube lens. The coherent and fluorescent light have different wavelengths and are optically separated by a dichroic mirror (Thorlabs, DMLP550R), after which the fluorescence is further spectrally filtered before being imaged onto Sensor-F (PCO.edge 5.5). The coherent light is ND-filtered and then split by a beam-splitter onto two sensors (FLIR, BFS-U3-200S6M-C). Sensor-C1 is in focus, while Sensor-C2 is defocused by 3 mm, enabling efficient phase retrieval across a broad swath of spatial frequencies, according to the phase transfer function [69].

Successful reconstruction relies on appropriate choices for the scanning range and step size [38]. Generally, the translation step size should be 2-3× smaller than the targeted resolution and the total translation range should be larger than the diffraction-limited spot size of the original system. Our system has detection NA of 0.4 and targeted resolution of 500 nm, so a 36 × 36 Cartesian scanning path with a step size of 180 nm is appropriate for 2D SR reconstruction. For coherent imaging, since there is zero axial bandwidth in the coherent TF (Fig. 2(a)), the sample’s complete diffraction information is projected axially and encoded in the measurement. This enables SR reconstruction of the sample’s 3D quantitative phase from just the translating speckle. Incoherent imaging, however, has optical sectioning due to its torus-shaped TF (Fig. 2(b)); hence, fluorescent light that is outside the DOF of the objective will have weak contrast. In order to reconstruct 3D fluorescence with high fidelity, we add axial scanning to our acquisition scheme [58].

A direct combination of lateral xy-scanning of the speckle and axial z-scaning of the sample will result in $36\times 36\times N_z$ measurements for both channels, where N_z is the number of axial scan positions. Fortunately, there is a high-degree of redundancy in this data. As previously stated, the 3D coherent information does not require axial scanning, and the speckle pattern measured from the coherent channel is used to initialize the fluorescent reconstruction. Thus, only minor refinements are needed for faithful fluorescent reconstruction.

To save acquisition time, we use an interleaving scanning scheme, alternating between axial sample scanning and lateral speckle scanning (Fig. 1). We laterally scan the speckle pattern through $36\times 36xy$ positions, while incrementing the z position for each patch of 12 × 12 positions. The 36 × 36 Cartesian speckle scanning path is divided into 9 blocks of 12 × 12 sub-scanning paths. Each sub-scanning path is associated with a z-scan position. This means the distance from incident speckle field to sample is

$$\mathrm{\Delta }{s}_{\mathcal{l}}=\left(n-1\right)s,\text{for}\mathcal{l}={12}^2\left(n-1\right)+1,\cdots ,{12}^{2}n,\text{where}n=1,\cdots ,9,$$ (7)

where s is the axial step size and n is the index for different z planes. We set the fifth z-scan position as the middle of the sample. The total scanning range is roughly the thickness of the sample and the step size is at least 2× smaller than the Nyquist-limited axial resolution of the fluorescence microscope. This interleaving measurement scheme enables high quality coherent and fluorescent 3D SR reconstructions.

3.1. 3D super-resolution demonstration

With a 0.4 NA objective, our system’s native lateral resolution is 1.33 μm for coherent imaging and 760 nm for fluorescence (Table 1). The intrinsic DOF is infinite for coherent imaging and 7.3 μm for fluorescence imaging. In order to characterize the resolution capability of our method, we begin by imaging a sample with features below both diffraction limits - a mono-layer of fluorescent polystyrene microspheres with diameter 700 nm. We use a z-scan step size of 1 μm across8 μm range, fully covering the thickness of the sample. 15 axial layers are assigned to the transmittance function, separated by 1.7 μm based on Nyquist sampling of the expected axial resolution for our 3D reconstruction, resulting an overall reconstructed axial range that spans the sum of the axial scanning range and the two axial ranges of the effective 3D PSF.

Figure 4 shows that our 3D reconstructions (400 × 400 × 15 voxels with voxel size of $0.096\times 0.096\times 1.7\mu$m³ ) clearly resolve the sub-diffraction individual microspheres and demonstrate better sectioning ability in both coherent and fluorescent channels compared to standard widefield imaging (without deconvolution). In the reconstruction, the average lateral peak-to-peak distance of these microspheres is around 670 nm, which is smaller than the nominal size of each microsphere. This is likely due to vertical staggered stacking of the microspheres. Given that our lateral resolution is at least 670 nm, we do break the lateral diffraction limit for both coherent and fluorescent channels, and the coherent channel achieves $>2\times$ lateral resolution improvement. Axially, we demonstrate 6 μm resolution for both channels, which is beyond the axial diffraction limit for both channels. The coherent channel improves the axial resolution from no sectioning ability to 6 μm. Given lateral resolution of 670 nm in the coherent channel, we can deduce the illumination NA of this speckle to be >0.4, which suggests the speckle intensity grain size is smaller than 670 nm.

Fig. 4.

3D multimodal (fluorescence and phase) SIM reconstruction compared to widefield fluorescence and coherent intensity images for 700 nm fluorescent microspheres. Resolution beyond the system’s diffraction limit is achieved in both the (a) coherent and (b) fluorescent arms.

3.2. 3D large-FOV multimodal demonstration

Next we use the same setup to demonstrate 3D multimodal imaging for our full sensor area (FOV ∼314μm × 500μm). As shown previously, our method achieves ∼0.6×0.6×6μm³ resolution for both fluorescence and phase imaging over an axial range of ∼24 μm. This corresponds to ∼14 Mega-voxels of information. Our experiments are only a prototype; this technique is scalable to the Gigavoxel range with a higher-throughput objective and higher illumination NA.

Figure 5 shows the full-sensor 3D quantitative phase and fluorescence reconstructions ($5200\times 3280\times 15$ voxels with voxel size of $0.096\times 0.096\times 1.7\mu$m³) of a multi-size sample (mixed 2 μm, 4 μm fluorescent, and 3 μm non-fluorescent polystyrene microspheres). We adopt the same z-scan step size and number of slices as in Fig. 4. Zoom-ins on 2 regions of interest (ROIs) display 4 axial layers for each. The arrows highlight 2μm fluorescent microspheres, which defocus more quickly than the larger ones. The locations of the fluorescent microspheres match well in both channels. However, there are some locations in the fluorescence reconstruction where 4 μm microspheres collapse because the immersing media is dissolving the beads over time.

Fig. 5.

Reconstructed 3D multimodal (fluorescence and phase) large-FOV for mixed 2 μm, 4μm fluorescent and 3 μm non-fluorescent polystyrene microspheres. Zoom-ins for two ROIs show fluorescence and phase at different depths.

Finally, we demonstrate our technique on human colorectal adenocarcinoma (HT-29) cells fluorescently tagged with AlexaFluor phalloidin ($5200\times 3280\times 18$ voxels with voxel size of $0.096\times 0.096\times 1.7\mu$m³), which labels F-actin filaments (sample preparation details in Appendix C). We use a z-scan step size of 1.6 μm across a 12.8 μm range and reconstruct 19 axial layers, separated by 1.7 μm. Figure 6 shows the full-sensor 3D quantitative phase and fluorescence reconstructions, with zoom-ins on 2 ROIs. The sample’s morphological features, as visualized with quantitative phase, match well with the F-actin visualization of the fluorescent channel. This is expected since F-actin filaments are generally known to encapsulate the cell body.

Fig. 6.

Reconstructed 3D multimodal (fluorescence and phase) large-FOV imaging for HT-29 cells (See Visualization 1 ^{(14.4MB, avi)} and Visualization 2 ^{(8.9MB, avi)} ). Zoom-ins for two ROIs show fluorescence and phase at different depths. The blue arrows in two ROIs indicate two-layer cell clusters that come in and out of focus. The orange arrows indicate intracellular components, including higher-phase-contrast lipid vesicles at z = −5.1 μm, nucleolus at z = 0, as well as the cell nucleus and cell-cell membrane contacts.

4. Discussion

Unlike traditional 3D SIM or 3D quantitative phase methods which use expensive spatial-light-modulators (SLMs) [70, 71] or galvonemeter/MEMs mirrors [57, 72, 73], our technique is relatively simple and inexpensive. Layered Scotch tape efficiently creates speckle patterns with ${\text{NA}}_{\mathrm{illum}}>0.4$, which is hard to achieve with traditional patterning approaches to high-content imaging (e.g. lenslet array or grating masks [29–35]). Furthermore, the random structured illumination conveniently multiplexes both phase and fluorescence information into the system’s aperture, enabling us to achieve multimodal 3D SR.

One limitation of our technique is that the fluorescent reconstruction relies on the recovered 3D speckle from the coherent imaging channel, so mismatch between the two channels can result in artifacts that degrade resolution. Indeed, the SR gain we achieve experimentally in the fluorescent channel does not match that achieved in the coherent channel. We attribute this mainly to mismatch in axial alignment between the coherent and fluorescent cameras, since the long DOF of the objective made it difficult to axially align the cameras to within the axial resolution limit of the high-resolution speckle pattern. In addition, our 3D coherent reconstruction suffers from coherent noise due to system instabilities during the acquisition process. Specifically, 3D phase information is encoded into the speckle-like (high dynamic range) features within the measurements, which are affected by Poisson noise. These factors reduce performance in both the 3D phase and fluorescence reconstructions.

Another limitation is the relatively long acquisition time - $\sim 1200$ translations of the Scotch tape results in $\sim 180$ seconds (without hardware optimization). The number of acquisitions could potentially be reduced with further investigation of the redundancy in the data, which would also reduce computational processing time for the reconstruction, which currently takes ∼6 hours on a NVIDIA, TITAN Xp GPU with MATLAB, for each $40\times 40\mu$m² patch. Cloud computing could also parallelize the reconstruction by patches.

5. Conclusion

We have presented a 3D SIM multimodal (phase and fluorescence) technique using Scotch tape as the patterning element. The Scotch tape efficiently generates high-resolution 3D speckle patterns over a large volume, which multiplexes 3D super-resolution phase and fluorescence information into our low-NA imaging system. A computational optimization algorithm based on 3D coherent and incoherent imaging models is developed to both solve the inverse problem and self-calibrate the unknown 3D random speckle illumination and the system’s aberrations. The result is 3D sub-diffraction fluorescence reconstruction and 3D sub-diffraction phase reconstruction with $>2\times$ lateral resolution enhancement. The method is potentially scalable for Gigavoxel imaging.

Appendix A: Gradient derivation

A.1. Vectorial notation

In order to derive the gradient to solve for multivariate optimization problem in Eq. (4) and (6), it is more convenient to represent our 3D coherent and fluorescent model in the linear algebra vectorial notation in the following sections.

According to Eq. (1) and (2), we are able to re-express the multi-slice scattering model using the vectorial formulation into

$$\begin{array}{l}{\mathbf{f}}_{\mathcal{l},1}={\mathbf{H}}_{\mathrm{\Delta }{s}_{\mathcal{l}},{\lambda }_{\text{ex}}}\mathbf{Q}{\mathbf{S}}_{\mathcal{l}}{\mathbf{p}}_c,\\ {\mathbf{g}}_{\mathcal{l},m}=\text{diag}\left({\mathbf{f}}_{\mathcal{l},m}\right){\mathbf{t}}_m,m=1,\cdots ,M,\\ {\mathbf{f}}_{\mathcal{l},m+1}={\mathbf{H}}_{\mathrm{\Delta }{z}_m,{\lambda }_{\text{ex}}}{\mathbf{g}}_{\mathcal{l},m},m=1,\cdots ,M-1,\\ {\mathbf{G}}_{\mathcal{l}}={\mathbf{H}}_{\mathrm{\Delta }{z}_{M,\mathcal{l}},{\lambda }_{\text{ex}}}{\mathbf{g}}_{\mathcal{l},M},\end{array}$$ (8)

where the boldface symbols are the vectorial representation of the 2D variables in non-boldface form in the original model except for the cropping operator Q, the shifting operator ${\mathbf{S}}_{\mathcal{l}}$ that shifts the speckle pattern with ${\mathbf{r}}_{\mathcal{l}}$ amount, and the defocus convolution operation expressed as

$${\mathbf{H}}_{z,\lambda }={\mathbf{F}}^{-1}\text{diag}\left({\tilde{\mathbf{h}}}_{z,\lambda }\right)\mathbf{F},$$ (9)

where F and ${\mathbf{F}}^{-1}$ are Fourier and inverse Fourier transform operator, respectively, ${\tilde{\mathbf{h}}}_{z,\lambda }$ is the vectorized coherent TF for propagation distance z and wavelength λ. With all these equations defined in vectorial form, we rewrite our coherent and fluorescence intensity as

$$\begin{array}{l}{\mathbf{I}}_{c,\mathcal{l}z}={\left|{\mathbf{H}}_c{\mathbf{H}}_{z,{\lambda }_{\text{ex}}}{\mathbf{G}}_{\mathcal{l}}\right|}^2,\\ {\mathbf{I}}_{f,\mathcal{l}}={\displaystyle \sum }_{m=1}^M{\mathbf{K}}_{z_{m,\mathcal{l}}}\text{diag}(|{\mathbf{f}}_{\mathcal{l},m}{|}^2){\mathbf{o}}_m,\end{array}$$ (10)

where ${\mathbf{H}}_c$ is also a convolution operation as expressed in Eq. (9) with the TF vector, ${\tilde{\mathbf{h}}}_{z,\lambda }$, replaced by the pupil vector ${\tilde{\mathbf{h}}}_c$, and

$${\mathbf{K}}_{z_{m,\mathcal{l}}}={\mathbf{F}}^{-1}\text{diag}\left(\mathbf{F}{|{\mathbf{F}}^{-1}\text{diag}\left({\tilde{\mathbf{h}}}_{z_{m,\mathcal{l}},{\lambda }_{\text{em}}}\right){\tilde{\mathbf{h}}}_f|}^2\right)\mathbf{F}$$ (11)

is the convolution operation with the incoherent TF at $z_{m,\mathcal{l}}$.

Next we use this vectorial model to represent the coherent and fluorescent cost functions for a single intensity measurement as

$$\begin{array}{l}{e}_{c,\mathcal{l}z}\left({\mathbf{t}}_1,\cdots ,{\mathbf{t}}_M,{\mathbf{p}}_c,{\tilde{\mathbf{h}}}_c\right)={\mathbf{e}}_{c,\mathcal{l}z}^T{\mathbf{e}}_{c,\mathcal{l}z}=\sqrt{{\mathbf{I}}_{c,\mathcal{l}z}}-{\Vert {\mathbf{H}}_c{\mathbf{H}}_{z,{\lambda }_{\text{ex}}}{\mathbf{G}}_{\mathcal{l}}|\Vert }_2^2,\\ e_{f,\mathcal{l}}\left({\mathbf{o}}_1,\cdots ,{\mathbf{o}}_M,{\mathbf{p}}_c,{\tilde{\mathbf{h}}}_f\right)={\mathbf{e}}_{f,\mathcal{l}}^T{\mathbf{e}}_{f,\mathcal{l}}=\begin{array}{c}{\Vert {\mathbf{I}}_{f,\mathcal{l}}-{\displaystyle \sum }_{m=1}^M{\mathbf{K}}_{z_{m,\mathcal{l}}}\text{diag}({|{\mathbf{f}}_{\mathcal{l},m}|}^2){\mathbf{o}}_m\Vert }_2^2,\end{array}\end{array}$$ (12)

where ${\mathbf{e}}_{c,\mathcal{l}z}=\sqrt{{\mathbf{I}}_{c,\mathcal{l}z}}-|{\mathbf{H}}_c{\mathbf{H}}_{z,{\lambda }_{\text{ex}}}{\mathbf{G}}_{\mathcal{l}}|$ and ${\mathbf{e}}_{f,\mathcal{l}}={\mathbf{I}}_{f,\mathcal{l}}-{\displaystyle \sum }_{m=1}^M{\mathbf{K}}_{z_{m,\mathcal{l}}}\text{diag}({|{\mathbf{f}}_{\mathcal{l},m}|}^2){\mathbf{o}}_m$ are the coherent and fluorescent cost vectors, respectively.

A.2. Gradient derivation

The following derivation is based on CR calculus and is similar to the derivation introduced by our previous work [38, 67].

A.2.1. Gradient derivation for 3D coherent imaging

To optimize Eq. (4) for ${\mathbf{t}}_1,\cdots ,{\mathbf{t}}_M$, ${\mathbf{p}}_c$, ${\tilde{\mathbf{h}}}_c$, we need to take the derivative of the coherent cost function with respect to them. We first express the gradients of all the transmittance function vectors, ${\mathbf{t}}_1,\cdots ,{\mathbf{t}}_M$ as

$${\nabla }_{{\mathbf{t}}_m}{e}_{c,\mathcal{l}z}={\left(\frac{\partial e_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{\mathcal{l},m}}\cdot \frac{\partial {\mathbf{g}}_{\mathcal{l},m}}{\partial {\mathbf{t}}_m}\right)}^{†}=\text{diag}\left(\overline{{\mathbf{f}}_{\mathcal{l},m}}\right){\left(\frac{\partial e_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{\mathcal{l},m}}\right)}^{†}=\text{diag}\left(\overline{{\mathbf{f}}_{\mathcal{l},m}}\right){\mathbf{v}}_{\mathcal{l},m},$$ (13)

where

$$\begin{array}{l}{\mathbf{v}}_{\mathcal{l},M}={\left(\frac{\partial e_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{\mathcal{l},M}}\right)}^{†}=-{\mathbf{H}}_{\mathrm{\Delta }{z}_{M,\mathcal{l}},{\lambda }_{\text{ex}}}^{†}{\mathbf{H}}_{z,{\lambda }_{\text{ex}}}^{†}{\mathbf{H}}_c^{†}\text{diag}\left(\frac{{\mathbf{H}}_c{\mathbf{H}}_{z,{\lambda }_{\mathrm{ex}}}{\mathbf{G}}_{\mathcal{l}}}{\left|{\mathbf{H}}_c{\mathbf{H}}_{z,{\lambda }_{\mathrm{ex}}}{\mathbf{G}}_{\mathcal{l}}\right|}\right){\mathbf{e}}_{c,\mathcal{l}z}\\ {\mathbf{v}}_{\mathcal{l},m}={\left(\frac{\partial e_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{\mathcal{l},m+1}}\cdot \frac{\partial {\mathbf{g}}_{\mathcal{l},m+1}}{\partial {\mathbf{g}}_{\mathcal{l},m}}\right)}^{†}={\mathbf{H}}_{\mathrm{\Delta }{z}_m,{\lambda }_{\text{ex}}}^{†}\text{diag}\left(\overline{{\mathbf{t}}_{m+1}}\right){\mathbf{v}}_{\mathcal{l},m+1},m=1,\cdots ,M-1,\end{array}$$ (14)

are auxiliary vectors for intermediate gradient derivation steps, $†$ is the Hermitian operation, and $\overline{\cdot }$ is the complex conjugate operation. With these auxiliary vectors, it is relatively simple to derive the gradient of the speckle field vector, ${\mathbf{p}}_c$, as

$${\nabla }_{{\mathbf{p}}_c}{e}_{c,\mathcal{l}z}={\left(\frac{\partial e_{c,\mathcal{l}z}}{\partial {\mathbf{g}}_{\mathcal{l},1}}\cdot \frac{\partial {\mathbf{g}}_{\mathcal{l},1}}{\partial {\mathbf{p}}_c}\right)}^{†}={\mathbf{S}}_{\mathcal{l}}^{†}{\mathbf{Q}}^{†}{\mathbf{H}}_{\mathrm{\Delta }{s}_{\mathcal{l}},{\lambda }_{\text{ex}}}^{†}\text{diag}\left(\overline{{\mathbf{t}}_1}\right){\mathbf{v}}_{\mathcal{l},1}.$$ (15)

As for the gradient of the pupil function, ${\tilde{\mathbf{h}}}_c$, we have

$${\nabla }_{{\tilde{\mathbf{h}}}_c}{e}_{c,\mathcal{l}z}={\left(\frac{\partial e_{c,\mathcal{l}z}}{\partial {\tilde{\mathbf{h}}}_c}\right)}^{†}=-\text{diag}\left(\overline{\mathbf{F}{\mathbf{G}}_{\mathcal{l}}}\right)\text{diag}\left(\overline{{\tilde{\mathbf{h}}}_{z,{\lambda }_{\text{ex}}}}\right)\mathbf{F}\text{diag}\left(\frac{{\mathbf{H}}_c{\mathbf{H}}_{z,{\lambda }_{\text{ex}}}{\mathbf{G}}_{\mathcal{l}}}{\left|{\mathbf{H}}_c{\mathbf{H}}_{z,{\lambda }_{\text{ex}}}{\mathbf{G}}_{\mathcal{l}}\right|}\right){\mathbf{e}}_{c,\mathcal{l}z}$$ (16)

A.2.2. Gradient derivation for 3D fluorescence imaging

To optimize Eq. (6) for ${\mathbf{o}}_1,\cdots ,{\mathbf{o}}_M$, ${\mathbf{p}}_c$, ${\tilde{\mathbf{h}}}_f$, we need to take the derivative of the fluorescent cost function with respect to each. First, we express the gradient for the fluorescence distribution vectors from different layers, ${\mathbf{o}}_1,\cdots ,{\mathbf{o}}_M$ as

$${\nabla }_{{\mathbf{o}}_m}{e}_{f,\mathcal{l}}={\left(\frac{\partial e_{f,\mathcal{l}}}{\partial {\mathbf{o}}_m}\right)}^{†}=-2\text{diag}({|{\mathbf{f}}_{\mathcal{l},m}|}^2){\mathbf{K}}_{z_{m,\mathcal{l}}}^{†}{\mathbf{e}}_{f,\mathcal{l}},m=1,\cdots ,M$$ (17)

Then, we would like to derive the gradient of the speckle field, ${\mathbf{p}}_c$, as

$${\nabla }_{{\mathbf{p}}_c}{e}_{f,\mathcal{l}}={\displaystyle \sum }_{m=1}^M{\left(\frac{\partial e_{f,\mathcal{l}}}{\partial {\mathbf{f}}_{\mathcal{l},m}}\cdot \frac{\partial {\mathbf{f}}_{\mathcal{l},m}}{\partial {\mathbf{p}}_c}\right)}^{†}=-2{\displaystyle \sum }_{m=1}^M{\left(\frac{\partial {\mathbf{f}}_{\mathcal{l},m}}{\partial {\mathbf{p}}_c}\right)}^{†}\text{diag}\left({\mathbf{f}}_{\mathcal{l},m}\right)\text{diag}\left({\mathbf{o}}_m\right){\mathbf{K}}_{z_{m,\mathcal{l}}}^{†}{\mathbf{e}}_{f,\mathcal{l}},$$ (18)

where

$$\begin{array}{l}{\left(\frac{\partial {\mathbf{f}}_{\mathcal{l},m}}{\partial {\mathbf{p}}_c}\right)}^{†}={\left(\frac{\partial {\mathbf{f}}_{\mathcal{l},m}}{\partial {\mathbf{g}}_{\mathcal{l},m-1}}\cdot \frac{\partial {\mathbf{g}}_{\mathcal{l},m-1}}{\partial {\mathbf{g}}_{\mathcal{l},m-2}}\dots \frac{\partial {\mathbf{g}}_{\mathcal{l},2}}{\partial {\mathbf{g}}_{\mathcal{l},1}}\frac{\partial {\mathbf{g}}_{\mathcal{l},1}}{\partial {\mathbf{p}}_c}\right)}^{†}\\ ={\mathbf{S}}_{\mathcal{l}}^{†}{\mathbf{Q}}^{†}{\mathbf{H}}_{\mathrm{\Delta }s,{\lambda }_{\text{ex}}}^{†}\left[\text{diag}\left(\overline{{\mathbf{t}}_1}\right){\mathbf{H}}_{\mathrm{\Delta }{z}_1,{\lambda }_{\text{ex}}}^{†}\right]\cdots \left[\text{diag}\left(\overline{{\mathbf{t}}_{m-1}}\right){\mathbf{H}}_{\mathrm{\Delta }{z}_{m-1},{\lambda }_{\text{ex}}}^{†}\right].\end{array}$$ (19)

As for the gradient of the pupil function at the fluorescent wavelength, ${\tilde{\mathbf{h}}}_f$, we can express as

$$\begin{array}{l}{\nabla }_{{\tilde{\mathbf{h}}}_f}{e}_{f,\mathcal{l}}=-2{\displaystyle \sum }_{m=1}^M\text{diag}(\overline{{\tilde{\mathbf{h}}}_{z_{m,\mathcal{l}},{\lambda }_{\text{em}}}})\mathbf{F}\text{diag}({\mathbf{F}}^{-1}\text{diag}({\tilde{\mathbf{h}}}_{z_{m,\mathcal{l}},{\lambda }_{\text{em}}}){\tilde{\mathbf{h}}}_f)\\ {\mathbf{F}}^{-1}\text{diag}(\overline{\mathbf{F}\text{diag}({\left|{\mathbf{f}}_{\mathcal{l},m}\right|}^2){\mathbf{o}}_m})\mathbf{F}{\mathbf{e}}_{f,\mathcal{l}}\end{array}$$ (20)

Appendix B: Reconstruction algorithm

B.1. Initialization of the variables

Since we use a gradient-based algorithm to solve, we must initialize each output variable, ideally as close as possible to the solution, based on prior knowledge.

For 3D coherent reconstructions, the targeted variables are transmittance function, $t_m\left(\mathbf{r}\right)$, incident speckle field, $p_c\left(\mathbf{r}\right)$, and pupil function, ${\tilde{h}}_c\left(\mathbf{u}\right)$. We have no prior knowledge of the transmittance function or pupil function, so we set $t_m\left(\mathbf{r}\right)=1$ for $m=1,\cdots ,M$ and ${\tilde{h}}_c\left(\mathbf{u}\right)$ to be a circle function with radius defined by ${\text{NA}}_{\det }/{\lambda }_{\text{ex}}$. This initializes with a completely transparent sample andnon-aberrated system. If the sample is mostly transparent, the amplitude of our incident speckle field is the overlay of all the in-focus shifted coherent intensities:

$$p_c^{\text{initial}}\left(\mathbf{r}\right)={\displaystyle \sum }_{\mathcal{l}=1}^{N_{\text{img}}}{I}_{c,\mathcal{l},z=0}\left(\mathbf{r}+{\mathbf{r}}_{\mathcal{l}}\right)/N_{\text{img}}.$$ (21)

For 3D fluorescence reconstruction, the targeted variables are sample fluorescence distribution, $o_m\left(\mathbf{r}\right)$, incident field, $p_c\left(\mathbf{r}\right)$, and pupil function at the emission wavelength, ${\tilde{h}}_f\left(\mathbf{u}\right)$. We have no prior knowledge of the system’s aberrations, so we set ${\tilde{h}}_f\left(\mathbf{u}\right)$ to be a circle function with radius defined by ${\text{NA}}_{\det }/{\lambda }_{\text{em}}$. For the incident speckle field, we use the estimated speckle field from the coherent reconstruction as our initialization. The key to a successful 3D fluorescence reconstruction with this dataset is an initialization of the sample’s 3D fluorescence distribution using the correlation-based SIM solver [53, 74–78] that gives us an approximate result to start with. We adapt the correlation-based solver in our case for rough 3D SR fluorescence estimation. The basic idea is that we use the knowledge of illumination speckle intensity from the coherent reconstruction to compute the correlation between the speckle intensity and our fluorescence measurement. This correlation is stronger when the speckle intensity lines up with the fluorescent light generated by this excitation in the measurement. Each layer of the estimated speckle intensity gates out out-of-focus fluorescent light in the measurement, so we could get a rough estimate of the 3D fluorescent sample. Mathematically, we express this correlation as

$$o_m^{\text{initial}}\left(\mathbf{r}\right)={\displaystyle \sum }_{n=1}^9{\langle \left(I_{f,\ell }(r)-{\langle I_{f,\ell }(r)\rangle }_{\ell (n)}\right)\left({|f_{m,\ell }(r)|}^2-{\langle {|f_{m,\ell }(r)|}^2\rangle }_{\ell (n)}\right)\rangle }_{\ell (n)},$$ (22)

where ${\langle \cdot \rangle }_{\mathcal{l}\left(n\right)}$ is the averaging operation over $\mathcal{l}$ index of fluorescence images with the same z-scan position (at the n-th layer) in the set defined $\mathcal{l}\left(n\right)=\left\{{12}^2\left(n-1\right)+1,\cdots ,{12}^{2}n\right\}$.

To understand why this correlation gives a good estimation of the 3D fluorescent sample, we go through a more detailed derivation with a short-hand notation Δ to denote the operation $\mathrm{\Delta }{a}_{\mathcal{l}}\left(\mathbf{r}\right)=a_{\mathcal{l}}\left(\mathbf{r}\right)-{\langle a_{\mathcal{l}}\left(\mathbf{r}\right)\rangle }_{\mathcal{l}\left(n\right)}$. Then, we examine one component of Eq. (22):

$$\begin{array}{l}{\langle \mathrm{\Delta }{|f_{m,\mathcal{l}}(\mathbf{r})|}^2\mathrm{\Delta }{I}_{f,\mathcal{l}}(\mathbf{r})\rangle }_{\mathcal{l}(n)}={\displaystyle \sum }_{m\prime =1}^M\int o_{m\prime }(\mathbf{r}\prime ){\langle \mathrm{\Delta }{|f_{m,\mathcal{l}}(\mathbf{r})|}^2\mathrm{\Delta }{|f_{m\prime ,\mathcal{l}}(\mathbf{r}\prime )|}^2\rangle }_{\mathcal{l}(n)}{h}_{f,z_{m\prime ,\mathcal{l}}}(\mathbf{r}-\mathbf{r}\prime )d\mathbf{r}\prime \\ \approx {\displaystyle \sum }_{m\prime =1}^M\int o_{m\prime }(\mathbf{r}\prime ){\langle \mathrm{\Delta }{|f_{m,\mathcal{l}}(\mathbf{r})|}^2\rangle }_{\mathcal{l}(n)}^2{\delta }_{m,m\prime }\delta (\mathbf{r}-\mathbf{r}\prime )h_{f,z_{m\prime ,\mathcal{l}}}(\mathbf{r}-\mathbf{r}\prime )d\mathbf{r}\prime \\ \propto {\langle \mathrm{\Delta }{|f_{m,\mathcal{l}}(\mathbf{r})|}^2\rangle }_{\mathcal{l}(n)}^2{o}_m(\mathbf{r}),\end{array}$$ (23)

where we assume the speckle intensity is completely uncorrelated spatially in 3D, which is an approximation because the speckle has finite grain size depending on the illumination NA. Under this assumption, this correlation is almost the 3D fluorescence distribution with an extra modulation factor. Hence, it serves well as a initialization for our 3D fluorescence distribution.

B.2. Reconstruction algorithm

With all the initializations, Algorithm 1 and Algorithm 2 summarized by the following pseudo-code:

Algorithm 1.

3D coherent imaging reconstruction

Require: $I_{c,\mathcal{l}z}$, ${\mathbf{r}}_{\mathcal{l}}$, $\mathcal{l}=1,\dots ,N_{\text{img}}$
1:	initialize $t_1^{\left(1,0\right)},\cdots ,t_M^{\left(1,0\right)}$, $p_c^{\left(1,0\right)}$, ${\tilde{h}}_c^{\left(1,0\right)}$; normalize $I_{c,\mathcal{l}z}$
2:	for $k=1:K_c$ do
3:	Sequential gradient descent
4:	for $j=1:(N_{\text{img}}\cdot N_z)$ do
5:	$z=z_{\text{mod}(j,2)}$; $\mathcal{l}=\text{mod}(j,N_{\text{img}})$
6:	if $j<N_{\text{img}}\cdot N_z$ then
7:	for $m=1:M$ do
8:	$t_m^{(k,j)}=t_m^{\left(k,j-1\right)}-{\nabla }_{t_m}{e}_{c,\mathcal{l}z}(t_1^{(k,j-1)},\cdots ,t_M^{\left(k,j-1\right)},p_c^{\left(k,j-1\right)},{\tilde{h}}_c^{\left(k,j-1\right)})/4\max {\left(\left|p_c^{\left(k,j-1\right)}\right|\right)}^2$
9:	end for
10:	$p_c^{\left(k,j\right)}=p_c^{\left(k,j-1\right)}\left(\mathbf{r}\right)-{\nabla }_{p_c}{e}_{c,\mathcal{l}z}(t_1^{\left(k,j-1\right)},\cdots ,t_M^{\left(k,j-1\right)},$
	$p_c^{\left(k,j-1\right)},{\tilde{h}}_c^{\left(k,j-1\right)})/\max {\left(|t_1^{\left(k,j-1\right)}|,\cdots ,|t_M^{\left(k,j-1\right)}|\right)}^2$
11:	$\xi =\mathcal{F}\{G_{\mathcal{l}}^{\left(k,j-1\right)}\}$
12:	${\tilde{h}}_c^{\left(k,j\right)}={\tilde{h}}_c^{\left(k,j-1\right)}-{\nabla }_{{\tilde{h}}_c}{e}_{c,\mathcal{l}z}(t_1^{\left(k,j-1\right)},\cdots ,t_M^{\left(k,j-1\right)},p_c^{\left(k,j-1\right)},{\tilde{h}}_c^{\left(k,j-1\right)})\cdot |\xi |/\left[\max (|\xi |\right)\cdot ({|\xi |}^2+\delta )]$, where δ is chosen to be small
13:	else
14:	Do the same update but save to $t_1^{\left(k+1,0\right)},\cdots ,t_M^{\left(k+1,0\right)},p_c^{\left(k+1,0\right)},{\tilde{h}}_c^{\left(k+1,0\right)}$
15:	end if
16:	end for
17:	Filter $t_1^{\left(k+1,0\right)},\cdots ,t_M^{\left(k+1,0\right)}$ with Gaussian filter to damp down high-frequency artifacts
18:	end for

Algorithm 2.

3D fluorescence imaging reconstruction

Require: $I_{f,\mathcal{l}}$, ${\mathbf{r}}_{\mathcal{l}}$, $f_{1,\mathcal{l}},\cdots ,f_{M,\mathcal{l}}\mathcal{l}=1,\dots ,N_{\text{img}}$
1:	initialize $o_1^{\left(1,0\right)},\cdots ,o_M^{\left(1,0\right)}$, $p_c^{\left(1,0\right)}$, ${\tilde{h}}_f^{\left(1,0\right)}$; normalize $I_{f,\mathcal{l}}$
2:	for $k=1:K_f$ do
3:	Sequential gradient descent
4:	for $\mathcal{l}=1:N_{\text{img}}$ do
5:	if $j<N_{\text{img}}$ then
6:	for $m=1:M$ do
7:	$o_m^{\left(k,j\right)}=o_m^{\left(k,j-1\right)}-{\nabla }_{o_m}{e}_{f,\mathcal{l}}(o_1^{\left(k,j-1\right)},\cdots ,o_M^{\left(k,j-1\right)},$
	$p_c^{\left(k,j-1\right)},{\tilde{h}}_f^{\left(k,j-1\right)})/\max {\left(|f_{1,\mathcal{l}}^{\left(k,j-1\right)}|,\cdots ,|f_{M,\mathcal{l}}^{\left(k,j-1\right)}|\right)}^4$
8:	end for
9:	$p_c^{\left(k,j\right)}=p_c^{\left(k,j-1\right)}\left(\mathbf{r}\right)-{\nabla }_{p_c}{e}_{f,\mathcal{l}}(o_1^{\left(k,j-1\right)},\cdots ,o_M^{\left(k,j-1\right)},$
	$p_c^{\left(k,j-1\right)},{\tilde{h}}_f^{\left(k,j-1\right)})/\max {\left(|t_1^{\left(k,j-1\right)}|,\cdots ,|t_M^{\left(k,j-1\right)}|\right)}^2$
10:	${\tilde{h}}_f^{\left(k,j\right)}={\tilde{h}}_f^{\left(k,j-1\right)}-{\nabla }_{{\tilde{h}}_f}{e}_{f,\mathcal{l}}(o_1^{\left(k,j-1\right)},\cdots ,o_M^{\left(k,j-1\right)},$
	$p_c^{\left(k,j-1\right)},{\tilde{h}}_f^{\left(k,j-1\right)})/\max \left(|\mathcal{F}\{o^{\left(k,j-1\right)}\cdot {|f_{\mathcal{l}}^{\left(k,j-1\right)}|}^2\}|\right)$
11:	else
12:	Do the same update but save to $o_1^{\left(k+1,0\right)},\cdots ,o_M^{\left(k+1,0\right)},p_c^{\left(k+1,0\right)},{\tilde{h}}_f^{\left(k+1,0\right)}$
13:	end if
14:	end for
15:	end for

3D coherent reconstruction takes about 40 iterations, while the 3D fluorescence reconstruction takes around 25 iterations to reach convergence.

Appendix C: Sample preparation

The sample shown in Fig. 4 is a monolayer of 700 nm diameter polystyrene microspheres (Thermofischer, R700), prepared by placing microsphere dilutions (60 uL stock-solution/500 uL isopropyl alcohol) onto #1.5 coverslips and then allowing to air dry. Water is subsequently placed on the coverslip to reduce the index-mismatch of the microspheres to the air. An adhesive spacer followed by another #1.5 coverslip was placed on top of the original coverslip to assure a uniform sample layer for imaging.

The sample used in Fig. 5 is a mixture of 2 μm (Thermofischer, F8826) and 4 μm (Thermofischer, F8858) fluorescently-tagged (${\lambda }_{\text{em}}=605$ nm) and 3 μm non-fluorescent (Sigma-Aldrich, LB30) polystyrene microspheres. We follow a similar procedure as before, except that the dilution is composed of 60 uL stock solution of each type of microspheres and 500 uL isopropyl alcohol. Since the microspheres are larger in size, we adopt high-index oil ($n_m\left(\lambda \right)=1.52$ at λ = 532 nm) for sample immersion.

Figure 6 uses a sample of HT-29 cells grown in DMEM with 10% FBS, trypsonized with 1× trypsin, passaged twice a week into 100mm dishes at 1/5, 1/6, 1/8 dilutions and stored in a 37C 5% CO₂ incubator. For imaging, HT-29 cells were grown on glass coverslips (12mm diameter, No. 1 thickness; Carolina Biological Supply Co.) and fixed with 3% paraformaldehyde for 20min. Fixed cells were blocked and permeabilized in phosphate buffered saline (PBS; Corning Cellgro) with 5% donkey serum (D9663, Sigma-Aldrich), 0.3% Triton X-100 (Fisher Scientific) for 30 minutes. Cells were incubated with Alexa Fluor 546 Phalloidin (A22283, ThermoFisher Scientific) for 1 hour, washed 3 times with PBS, and mounted onto a second glass coverslip (24×50mm, No. 1.5 thickness; Fisher Scientific) and immobilized with sealant (Cytoseal 60; Thermo Scientific).

Funding

STROBE: A National Science Foundation Science & Technology Center (DMR 1548924); Gordon and Betty Moore Foundation’s Data-Driven Discovery Initiative (GBMF4562); Chan Zuckerberg Biohub; Ruth L. Kirschstein National Research Service Award (F32GM129966).

Disclosures

The authors declare that there are no conflicts of interest related to this article.