Combining deep learning approaches and point spread function engineering for simultaneous 3D position and 3D orientation measurements of fluorescent single molecules

Abstract

Point Spread Function (PSF) engineering is an effective method to increase the sensitivity of single-molecule fluorescence images to specific parameters. Classical phase mask optimization approaches have enabled the creation of new PSFs that can achieve, for example, localization precision of a few nanometers axially over a capture range of several microns with bright emitters. However, for complex high-dimensional optimization problems, classical approaches are difficult to implement and can be very time-consuming for computation. The advent of deep learning methods and their application to single-molecule imaging has provided a way to solve these problems. Here, we propose to combine PSF engineering and deep learning approaches to obtain both an optimized phase mask and a neural network structure to obtain the 3D position and 3D orientation of fixed fluorescent molecules. Our approach allows us to obtain an axial localization precision around 30 nanometers, as well as an orientation precision around 5 degrees for orientations and positions over a one micron depth range for a signal-to-noise ratio consistent with what is typical in single-molecule cellular imaging experiments.

1. Introduction

Early work in single-molecule detection [1] and imaging [2] led to the emergence of single-molecule localization microscopy beyond the diffraction limit (SMLM) [3]. However, to handle dense labels and achieve true super-resolution, localization must be combined with some form of active control of the emitters to maintain sparsity and prevent overlap. Blinking and/or switching [4] of fluorophores provides this, as in (F)-PALM [5,6], (d) STORM [7,8] or (DNA)-PAINT [9,10], and an alternate inclusive designation for all these processes is single-molecule active control microscopy (SMACM) [11]. Initially limited to the 2D observation of single molecules, many efforts have been made in the past decades to increase the amount of information that can be extracted from a fluorescent image of a molecule, given a better understanding of their arrangement and environment at the nanoscale. While some recent works are focused on the analysis of the number of photons emitted by the molecules to extract spectral or 3D information [12-17], the most commonly used methods are based on the analysis of the shape of the emission pattern, also called the point spread function (PSF). Among these methods, we focus on PSF engineering approaches that are based on the introduction of controlled aberrations in the fluorescence signal to increase the sensitivity of the PSF to certain parameters.

The fluorescence signal of a single molecule can be modeled by the emission of a dielectric dipole. In SMLM, the dipoles are often considered to be freely rotating resulting in an isotropic emission pattern. In this case, the spatial distribution of the emitted electric field can be easily estimated in the different planes of the imaging system, and the impact of introducing phase modulation into the optical system can be easily evaluated. Thus, different phase masks have been designed to extract the 3D position [18-20] or the spectral information [20]. Optimization methods using Fisher information have also been proposed to obtain ultimate axial performances as in the case of the tetrapod PSF [21], which provides optimal 3D information over an axial range of several microns. However, in some cases, the molecule may have reduced mobility, which has a significant impact on the resulting PSF and can therefore create measurement biases of several hundred nanometers [22-24]. To counter these biases, one solution is to simultaneously measure the 3D position of the molecules as well as their orientations. Several works have already been proposed to answer this problem [25-31] and an optimized phase mask has even been proposed to extract this information [32]. Yet, the high dimensionality of the problem brings new constraints for classical optimization methods: it is difficult to optimize the phase mask over the whole range of different parameters and there can be time-consuming computation to extract these parameters from the PSF. Recently, thanks to the democratization and efficiency of new graphical processing units (GPUs), new approaches to SMLM imaging using deep learning methods have emerged. Among other things, deep learning has enabled solution of complex problems such as phase mask optimization [33], structural background estimation [34] or extraction of the position and/or orientation of fluorescent molecules [33,35-37].

Here, we propose a novel optimization method that combines deep learning and phase mask optimization to simultaneously obtain the 3D orientation and position of immobilized fluorescent molecules from single images. Our optimization approaches simultaneously train a phase mask and convolutional layers that extract the 3D position and 3D orientation from the engineered dipole pattern emission. The entire approach is trained on simulated data that matches the expected experimental conditions from a two-polarization pyramidal mirror-based microscope and the sample. Finally, we obtain on the one hand an optimized phase mask applicable in our detection setup and on the other hand a fitting network that returns 3D orientation and 3D positions from each experimentally acquired PSF.

2. Experimental setup

Fig. 1 depicts our 4f experimental PSF engineering apparatus used to simultaneously obtain the 3D orientation and the 3D position of immobilized single molecules. Since a fluorescent molecule can be well approximated by an electric dipole emitter, its orientation is defined by its unitary dipole moment vector $\overrightarrow{\mu }$. As shown in Fig. 1a, the 3D components of this moment vector can be transformed into components depending on the polar angle $\theta$ and the azimuthal angle $\varphi$ by the following relationship: $[{\mu }_x,{\mu }_y,{\mu }_z]=[cos(\varphi )sin(\theta ),sin(\varphi )sin(\theta ),cos(\theta )]$. The polar angle represents the out of plane orientation while the azimuthal angle corresponds to the xy projected in-plane angle. In addition, we choose to define the axial position of the molecule z_em as the relative distance of the focal plane from the position of the molecule. In order to obtain both the molecule’s position and orientation simultaneously, we use a pyramidal geometry experimental setup similar to optical configurations described in previous single-molecule orientation and position studies [27,28]. This setup is depicted in Fig. 1b and allows us to install a phase modulation in both crossed polarization channels using a single spatial light modulator with minimal photon losses, and this is explained in further detail below.

Fig. 1. Single-molecule orientation and experimental setup.

(a) A single molecule Is approximated as a fluorescent dipole with a vector moment $\overrightarrow{\mu }$ defined by Its vector components. These components are constructed from the polar and azimuthal angles $\theta$ and $\varphi$. (b) PSF engineering microscope based on the pyramidal geometrical configuration. CF: Cleanup Filter, RA: Right Angle, NF: Notch Filter, BP: Band Pass, TL: Tube Lens, PBS: Polarizing Beam Splitter, LC-SLM: Liquid Crystal Spatial Light Modulator. (c) Reflection of both polarized channels on the pyramidal mirror aligns the two polarizations on the LC-SLM. (d) Simulation of the intensity in the Back Focal Plane for molecules at different orientations. Scale bar = 2 mm (e) Phase modulation orientation and symmetry of the optical path relative to the LC-SLM plane for both polarization channels.

The sample is first excited with a laser emitting at 561 nm (Coherent, Sapphire 561-100-CW) with circular polarization induced by the presence of a quarter wave plate (Thorlabs, WPQ10M-546). The beam diameter is then increased by a factor of 10 using a telescope before being focused on the BFP of a 100 × 1.4 NA oil objective (Olympus, 1-U2B836), which allows for widefield excitation. The emitted fluorescence signal is collected by the same objective and is processed with the pyramidal setup. After passing through the different emission filters (Dichroic filter: Semrock, Di03-R561-t3-25 × 36, Notch filter: Chroma, ZET561NF, Band pass filter: Semrock, FF01-630/92-25) that remove background fluorescence signals and residual pump light, the fluorescence signal is split into two detection paths for the two orthogonal polarizations $x$ and $y$ (or s and p) using a Polarizing Beam Splitter cube (PBS) (B. Halle, PTW 10). The 2 beams are then directed toward a custom-built pyramid-shaped mirror that reflects the incident light upward, as shown in Fig. 1c. The reflection angles of each face of the pyramidal mirror are such that the two reflected beams are superimposed in the conjugate plane of the BFP (this plane is also called the Fourier plane), which is ensured by the propagation through the 150 mm focal length doublet lens f₁ just before PBS. The phase modulation of the light in the Fourier plane is achieved by introducing a reflective liquid crystal spatial light modulator (LC-SLM, Boulder Non-linear) which also redirects the light downward toward the pyramidal mirror and yielding propagation of each beam parallel to the optical table. The LC-SLM is composed of a 512 × 512 pixels matrix whose individual size is 15 × 15 μm² that can be controlled individually to modulate the phase of the incident light only along one polarization direction. Importantly, our pyramidal mirror configuration allows us to bypass this experimental limitation and use the same modulator for both detection channels. Indeed, due to the upward reflection, the y-polarization is rotated while the x-polarization is not affected. Thus, both polarizations are oriented in the same direction on the phase modulator. Subsequently, each beam passes through a 100 mm doublet lens and a telescope, yielding real-space images on the camera.

Thanks to vectorial diffraction theory [38], we can efficiently compute the emitted electric field components of a dipole in the BFP and in the image plane. Fig. 1d shows several examples of the light intensity distribution in the BFP for the two polarized detection channels for molecules at four different orientations. A Fourier transform of the electric field in the BFP after the phase modulation gives the final electric field components in the camera image plane, and thus the resulting PSF. Our calculations must also consider the additional reflection introduced in the y-polarized channel relative to the x-polarized channel due to the presence of the PBS in our setup. This additional reflection creates a mirror symmetry relative to the y-axis for the x-polarized channel, as shown in Fig. 1e. Moreover, as the direction of propagation of the two channels are orthogonal, the phase modulation applied for one of the detection channels will then be rotated by 90 degrees with respect to the other detection channel. This property is also included in our realistic experimental simulations of the PSFs. Finally, the two polarized channels are imaged side by side on a Si sCMOS camera (Hamamatsu, Orca Fusion BT) after a total magnification of 50 induced by the lenses and the objective of the setup. The corresponding optical pixel size referred to the sample plane is 130 nm on the detector.

3. Phase mask optimization using deep learning

The LC-SLM provides great flexibility for PSF engineering approaches due to the large number of controllable pixels that can spatially modulate the phase of the incident light. Using a SLM combined with a pyramidal geometry experimental setup, previous studies have simultaneously extracted the orientation and 3D position of immobilized fluorescent probes, notably with the use of established phase masks such as the Double-Helix phase mask or the Vortex phase mask [27-29]. More recently, a more complex phase mask has been developed by Wu et al. using a pixelwise phase mask optimization approach to simultaneously obtain the orientation and 3D position of fluorescent molecules with different degrees of mobility with very high precision [32]. Due to the complexity, high dimensionality, and computational cost of the optimization problem, the authors chose to optimize the phase mask using the Fisher information only in orientation space. They chose to fix other critical parameters such as the emitter’s position, the focus position (set at the coverslip in this study), and the signal to noise ratio. By choosing these constraints, the combined Fisher information/pixelwise optimization approach provides an optimized phase mask after a reasonable computation time. In a different pixelwise phase mask optimization approach, deep learning approaches were used to generate a mask that can extract the 3D position of molecules in a densely labeled sample [33]. This approach offers the possibility to easily introduce several experimental conditions such as the labeling density, multiple signal-to-noise ratios, and different emitter 3D position during the optimization.

In general, deep learning methods for single-molecule localization have become more popular in recent years, especially due to their ability to recognize structures in images with high precision and accuracy. The information of interest can thus be extracted 100 times faster than with standard iterative fitting methods. Due to the fast computational speeds now available, phase mask optimization with deep learning approaches can be performed over a huge range of configurations rather than a fixed set of emitter parameters. Deep learning mask generation methods should then encode precise and accurate information over a large range of experimental emitter configurations, which is not the case for standard fixed emitter optimization approaches. Therefore, in this work we combined PSF engineering and deep learning methods to generate a phase mask that encodes precise 3D position and orientation information in single PSF images. The PSFs resulting from our optimized phase mask are thus optimized in position and orientation space to incorporate high sensitivity to all these parameters but are also designed to be efficiently analyzed by the convolutional neural network.

The first neural network we will describe, called the Phase Mask Optimization NET (PMO-NET), is schematically described in Fig. 2a. The PMO-NET is trained on simulated data generated over a large sample space of molecular coordinates in a controlled fashion, which is not easily available with experimental data. As illustrated in the first panel of Fig. 2a (i), the inputs used for the PMO-NET training correspond to the different axial positions z_em and orientation parameters [${\mu }_x,{\mu }_y,{\mu }_z$] of an imaged fluorescent molecule. The axial coordinates, defined as the relative distance between the molecule and the focal plane for a molecule fixed at the coverslip, are generated by displacing the focal plane over a range from −500 nm to 500 nm with an axial step size of 50 nm. Due to the symmetric nature of the dipole moment when emitted fluorescence is detected, the different orientations are included by considering a half sphere defined by the azimuthal angles $\varphi$ between −90 and 90 degrees and the polar angles $\theta$ between 0 and 180 degrees, with a step size of 5 degrees and 10 degrees respectively. The final training list of input parameters corresponds to 17,199 possible combinations. This parameter list is then transferred to our GPU (Nvidia, RTX-A6000) and randomly shuffled before starting training.

Fig. 2. Phase Mask Optimization NET.

(a) Training workflow of the pixelwise phase mask optimization using a convolutional neural network. (i) Input: List of orientations and axial positions used during the neural network training. (ii) BFP layer: non-trainable. Uniform random distribution of $x$ and $y$ values associated with the axial position and orientation selected yields the BFP electric field for both detection channels, with 2 exemplary BFP cases shown. Scale bar = 2 mm. Same scale for all images. (iii) Trainable physical layer used as a phase mask in the calculation. The trainable mask layer at an early training point is shown here. (iv) PSF computation from the BFP electric field after the phase mask layer. Signal and approximated noise are applied here, viewed as non-trainable information. Resulting shown PSF for the two example cases. Scale bar = 1 μm. Same scale for all images. (v) Trainable layers based on the Incept-Res-Net architecture. (vi) Outputs resulting from (v) with the same shape as the inputs and xy position. (b) Resulting optimized phase mask. (c) Simulated normalized PSFs obtained with the phase mask in (b), for molecules considered at different axial positions and orientations. Scale bar = 1 μm. Same scale for all images. For more examples, see Supplementary Figures 2 and 3. (d) Violin diagrams of the Cramer–Rao Lower Bounds (CRLB) for the 3D position ($x$, $y$, and $z$) and the azimuthal and polar angles ($\varphi$ and $\theta$). Results are computed considering 2500 photons collected and a background per pixel per channel of 25 photons. Inset shows that any orientations close to the optical axis ($\theta <{10}^{{}_{°}}$) will have low $\varphi$ precision.

A sub-population (batch size = 16) of this parameter list is injected in the first layer of the PMO-NET called the BFP layer (Fig. 2a (ii)). In a first step, the axial positions and orientations of the subpopulation are combined with uniformly distributed lateral positions $x$ and $y$ in the range of −300 to 300 nm. The $x$ and $y$ coordinates are saved and will be used later in the loss function. These 3D orientations and positions are used to compute the electric field distribution in the Fourier plane. The simulations include the actual experimental parameters such as the index mismatch (1.52 for oil and glass optics into aqueous sample media with index 1.33) and the microscope configuration (see Table S1 for all simulated parameters). Note that the BFP layer is not composed of trainable weights and is only used to compute the electric field of the molecules in the BFP. The first trainable layer of our PMO-NET is a matrix of 512 × 512 trainable weights, which matches the size of LC-SLM. This matrix is then used during the training as a phase mask applied to the previously calculated BFP electric field. An example of the different weight values in each SLM/pixel obtained after an early degree of training is shown in panel (iii). Next, the phase modulated BFP electric fields are passed through a sequence of non-trainable layers to calculate the PSFs (panel (iv)). The electric fields in the image plane are computed by applying a Fourier transform on the electric fields in the BFP after phase modulation. The corresponding intensities are calculated, normalized, and cropped around 20 × 20 pixel regions from the center of the images to keep only the signal of interest. This ensures that the final optimized phase mask will constrain the photon distribution of the PSF within the 20 × 20 pixel region; the PSF shape will not spread beyond this cropped region and no precious photon signal will be lost. The signal emitted for each molecule as well as the background per pixel and per channel is selected from a uniform distribution over a range of 1500 to 7000 photons and 12.5 to 150 photons per pixel and per channel, respectively. This integrated signal is then impressed upon the cropped images and an approximate Poisson noise similar to the one used in [33] is applied to the image to obtain the final simulated PSFs as shown in panel (iv). These PSFs are then fed into a set of trainable layers (panel (v)) based on the Incept-ResNet architecture [39]. The detailed Incept-ResNet architecture is shown in Supplementary Figure 1. The last dense layer provides the output of the network, which is the list of the six predicted parameters. The output shape is the same shape as the input, with the addition of the predicted lateral positions $x$ and $y$ (panel vi). The predicted parameters and the ground-truth labeled parameters, normalized by the range for the positions (500 for $z$, 300 for $x$ and $y$), are used to compute the loss function based on Mean Squared Error (MSE) as shown below.

$$\begin{gathered} \text{Loss}=\frac{1}{6\mathit{K}}\sum _{i=0}^{\mathit{K}}(x_i-x_i^p{)}^2+(y_i-y_i^p{)}^2 \\ +(z_i-z_i^p{)}^2+{\left({\mu }_{x,i}-{\mu }_{x,i}^p\right)}^2 \\ +{\left({\mu }_{y,i}-{\mu }_{y,i}^p\right)}^2+{\left({\mu }_{z,i}-{\mu }_{z,i}^p\right)}^2 \end{gathered}$$ (1)

where the superscript p represents the predicted value while the absence of this superscript represents the ground truth value. K is the total number of molecules in the batch and $i$ denotes the specific molecule that is considered. The value of each trainable weight is adjusted with the classical ADAM optimizer. Training is repeated on all the batches for about 100 epochs before convergence, which is around 6 hours (see Table S2 for all PMO-NET training parameters). Note that we use the components of the moment vector instead of the polar and azimuthal angles independently in the loss function in order to better take into account the coupling of these angles in the error of the molecule’s orientation. For orientations close to the optical axis, $\varphi$ is not sensitive or meaningful, and using the moment vector components in the Loss function takes this into account. The code for this entire process is realized in Tensorflow, a deep learning Python package [40].

We call the resulting optimized phase mask the Arrowhead Phase mask (Fig. 2b). Several resulting PSFs for molecules at different orientation and position are shown in Fig. 2c, and more examples are provided in Supplementary Figure 2 and in Supplementary Figure 3. We can see that the optimized PSF is mainly composed of 3 spots whose distance depends on the axial position of the molecule. We also notice that the intensity distribution between these 3 spots depends largely on the axial component of the orientation (out-of-plane) of the molecule while the intensity distribution between the different channels reflects largely the projection of the orientation in the xy plane (in-plane). Note that this behavior can also be observed in previous works [26-28]. In previous work, a phase mask optimized only on the 3D position shows a smaller footprint of the PSF at the middle of the axial range, with a symmetric variation of the shape around this value [21]. Interestingly, the footprint of the Arrowhead PSF, which includes 3D position and orientation, is smaller at the edge of the axial range. We believe that this effect arises from removing the possible ambiguity in the different PSF shapes since the optimization is performed over a large range of parameters in five-dimensional space. The absence of ambiguity of the PSF shape for different orientation and position is illustrated in the supplementary Figure 4 and supplementary Figure 5 where the accuracy map for each retrieved parameter as a function of the polar and azimuthal angles at different axial positions is depicted. Since the accuracy maps are roughly flat and do not show any pattern with high bias values, we conclude that our procedure is able to retrieve the true position and orientation for many different simulated molecules.

In order to assess the performance of the Arrowhead phase mask, we calculated the Cramer–Rao Lower bound (CRLB) for each of the different parameter values with realistic experimental signal and background levels of 2500 emitted signal photons and 25 background photons per pixel per channel. The CRLB represents the theoretical best achievable precision given a PSF model and specific signal and background levels. The results are represented as a violin diagram in Fig. 2d, where the red plots represent the CRLB for the positions and the green plots correspond to the orientation angles. We notice at first that the distributions are quite uniform. The median value of the CRLB for the lateral positions is around 5 nm versus 12 nanometers axially. The theoretical precision on the polar angle $\theta$ is very uniform and shows a median value around 2 degrees, while the median theoretical precision for the azimuthal angle $\varphi$ is around 3 degrees. Note that the presence of the tail in the theoretical precision distribution for the angle $\varphi$ (denoted by the *) corresponds to molecules whose orientation is almost coincident with the z axis. When the molecules are oriented vertically, the azimuthal angle no longer has physical meaning, so the azimuthal CRLB in the region will be high. Results from completely out-of-plane molecules ($\theta =0$ degree) are not shown in this diagram. The high precision obtained from our CRLB calculations validates our theoretical approach and the whole optimization concept. Additional averaged curves showing the behavior of the CRLBs as a function of the axial position are also presented in Supplementary Figure 6.

4. Single-molecule orientation and position measurements

Experimentally, the different physical components of a real microscope will generate optical aberrations that impact the experimental PSF, which will displace the experimental model from the theoretical model. Without including these effects, our localization precision and accuracy results will severely degrade. These aberrations are usually obtained using a phase retrieval method to compute a new theoretical model that matches the experimental reality. To measure the optical aberrations present in our optical system, we imaged immobilized fluorescent nanobeads in water at different focus positions. As shown in Fig. 3a, the total axial scan is performed over a range of 2 μm symmetrically around the coverslip with a 50 nm axial step between each focus position. To reduce the impact of the Poisson noise fluctuations, each imaged field of view is acquired 50 times and averaged in the phase retrieval algorithm. The axial scanning is performed precisely by using a nanometric precision stage (Physik Instrumente, P-545.3C8S) that moves the sample up and down. As the fluorescent bead is composed of many randomly oriented dipoles, the corresponding PSF can be calculated as the superposition of dipole orientations along the $x$, $y$ and $z$ axes (Fig. 3b). A Gaussian blur is also added to the computed PSF to simulate the fact that the bead is not a finite size object but has a large diameter compared to a single molecule, as was effectively done in previous studies [41,42]. Fig. 3c shows the theoretical PSF calculated at different focal positions in the ideal case of the Arrowhead phase mask without aberrations (Fig. 3d). The comparison with the experimental PSFs shown in Fig. 3e shows clear differences between theory and the experiment. We used a custom developed phase retrieval algorithm adapted from [43] to retrieve the actual phase modulation present in our experimental setup. The details of the algorithm are presented in Supplementary Figure 7. Due to the presence of two optical detection channels, the phase retrieval algorithm is applied independently to each polarized channel, resulting in two different phase modulations that must be added to the phase from the Arrowhead mask. The resulting aberrated phase masks for each detection channel are shown in Fig. 3f. The simulated PSFs obtained from the two retrieved phase masks (Fig. 3g) now present quite strong similarity with the experimental PSFs, confirming the veracity of the results obtained by the phase retrieval process.

Fig. 3. Phase retrieval on the optimized PSF.

(a) Axial stack acquisition on a 2 μm range of fluorescent 40 nm diameter nanobeads in water and fixed to the coverslip. Each stack is composed of 50 frames and the axial step size is 50 nm. (b) A fluorescent nanobead can be approximated as a superposition of dipole orientated along the 3 axes. (c) Theoretical PSFs obtained with the Arrow-head phase mask for a fluorescent nanobead at different focus position. Scale bar = 1 μm. (d) Theoretical phase modulation in each polarization channel without aberrations. (e) Experimental PSFs obtained from a fluorescent nanobead at different focus position. Scale bar = 1 μm. (f) Phase retrieved modulation from (e) in both polarized channels. The modulation corresponds to the superposition of modulation introduced by the Arrow-head phase mask and the aberrations coming from the different components in the microscope. (g) Simulated PSFs computed from the phase retrieved modulation shown in (f). Scale bar = 1 μm.

The last part of the problem is to determine the parameters which are implied by a specific measured PSF. To do this, we introduce a second neural network, which we call the Analysis-NET, to extract the 3D orientation and position of molecules from experimental PSFs without bias from optical aberrations. Unlike for PMO-NET, the shape of the PSF at several orientations and positions can be computed before network training since the phase modulation in both detection channel is known. This will reduce the number of calculation steps, and thus, the training time. Moreover, we can determine the theoretical precision limits of our optimized PSFs by extracting the CRLBs from the approximated gradients and PSF models for all parameters. The CRLB values can be introduced in the loss function as penalizer for low signal-to-noise ratio PSFs, and this improves the performance of the Analysis-NET as demonstrated in Ref. [35] (see Eq. (2)). Note that this operation is not possible in the PMO-NET training because the PSF model varies with each modification of the phase mask in that case. This results in a modification of the CRLB during each training step as well. As neural network training seeks to minimize the loss function, using a loss function with CRLB in the denominator in Eq. (2) would lead to very high values of CRLBs, which would greatly reduce the impact of the emitter position during PMO-NET training. Thus, the CRLB was not considered during PMO-NET training. The workflow of the Analysis-NET training is shown in Fig. 4a. Beginning with the training, the inputs shown correspond to the Probability Density Function (PDF), which represents the normalized spatial distribution of the signal emitted by the molecule, and the gradients of this distribution for each orientation and position of the molecule. The gradient and the PDF dimension correspond to a region of 26 × 26 pixels. The region size is slightly larger than that used in the PMO-NET training to also account for coarse localization and cropping errors. These inputs are computed for uniformly distributed positions over an axial range of −1000 to 1000 nm and a lateral range of −500 to 500 nm, and with uniformly distributed orientations over a range of 0 to 180 degrees for both the out-of-plane and the in-plane angles. Each PDF and gradient are labeled with the corresponding x, y, z positions and dipole moment vector components [${\mu }_x,{\mu }_y,{\mu }_z$].

Fig. 4. Analysis NET and funnel optimization.

(a) Analysis NET training workflow. The Analysis NET is trained on simulated data that considers the aberrations present in the microscope. The inputs correspond to the Probability Density Function (PDF) of simulated molecules at different 3D positions and orientations, and the corresponding approximated gradients for the positions and orientations. The PDF and gradients are labeled with the ground truth values of orientations and positions. Random signal and background are randomly generated and applied to the PDF and gradients to compute the PSFs and the Cramer Rao Lower Bounds. The signal is uniformly distributed over a range of 1500 to 7000 photons while the background per pixel per channel is uniformly distributed over a range of 10 to 150 photons. PSFs are injected into the trainable Incept-ResNet layers. Scale bar = 1 μm. The outputs of the last dense layer correspond to a list of predicted positions and orientations. The prediction, labels and the CRLB are injected in a custom loss function build to penalize the training on low signal-to-noise data. (b) Funnel optimization principle. Multiple NET are initialized and sequentially trained on few epochs for multiple iteration. After each iteration, the NETs are evaluated and the NETs with the higher Loss values are removed from the training. The operation is repeated until only 1 is NET left.

Similar to the PMO-NET training, the signal emitted for each molecule as well as the background per pixel and per channel are uniformly distributed over a range of 1500 to 7000 photons and 10 to 150 photons, respectively. The signal and background values, and our approximate Poisson noise model are applied to the PDF to calculate the PSF. The signal and background values are also applied to both the PDF and the gradient to compute the CRLB. The PSFs are injected into the Incept-Res-Net architecture, which is composed of many trainable weights that are optimized during training. As for the PMO-NET, the last dense layer of the Analysis-NET provides the outputs of the network, which is the list of the predicted orientation and position. These outputs, combined with the CRLB and the ground-truth labels, are injected into the custom loss function defined as:

$$\begin{gathered} Loss=\frac{1}{\mathit{K}}\sum _{i=0}^{\mathit{K}}\frac{(x_i-x_i^p{)}^2}{{\sigma }_{x_i}}+\frac{(y_i-y_i^p{)}^2}{{\sigma }_{y_i}} \\ +\frac{(z_i-z_i^p{)}^2}{{\sigma }_{z_i}}+{\left({\mu }_{x,i}-{\mu }_{x,i}^p\right)}^2 \\ +{\left({\mu }_{y,i}-{\mu }_{y,i}^p\right)}^2+{\left({\mu }_{z,i}-{\mu }_{z,i}^p\right)}^2 \end{gathered}$$ (2)

where ${\sigma }_{x,y,z_i}$ corresponds to the CRLB of the spatial coordinates x,y and z for the ith PSF. The other variables of the loss function are already defined in the previous PMO-NET loss function (Eq. (1)). After convergence, the performance of the neural network is evaluated from simulated PSFs not used during training.

Interestingly, we found that the performance of the NET was very variable. The same Analysis-NET trained on multiple separate occasions each yielded different performance, as shown in Supplementary Figure 8. The disparity in training performance is certainly due to the presence of multiple local minima in the high-dimensional space of the non-convex loss function. In order to obtain a reliable and hopefully near-optimal performance for the Analysis-NET, we have developed a custom optimization scheme which we describe in Fig. 4b. The principle of this new optimization is based on the sequential training of several Analysis-NETs, which are evaluated at different times during training. As the training time increases, we remove poorly performing networks in finite intervals. The percentage of retained networks also decreases with increased training time until two fully trained and high performing networks are obtained. Out of these two, the network with the best performance is then used for further analysis. As this process is reminiscent of a funnel, we call this approach Funnel Optimization. Due to our GPU limitations, multiple networks cannot be trained at the same time. Therefore, each network is trained on a few epochs and stored for future comparison. After comparison of the loss function results, retained networks are reloaded in their previous states and the funneling optimization continues. We start the funnel optimization with 40 analysis NETs that we train sequentially over 2 epochs as shown in Fig. 4b. After comparing the loss values of each NET, we keep the 30 NETs with the lowest loss values. These NETs are then trained over 8 more epochs. We then keep the 15 NETs corresponding to the lowest loss values, and we restart the training for 20 more epochs. The 6 best NETs are then trained over 40 more epochs, then the two best NETs are trained over 80 more epochs. Finally, the NET with the lowest loss value is the one we will keep for performance analysis and experimental data fitting. After the total 150 epochs, the final network training generally converges. Although funnel optimization is more time consuming than the standard training approach, it requires only 16 hours of training and provides optimal and repeatable performance on multiple runs. The comparison of the performance coming from our Funnel Optimization with the performance coming from the classic ADAM optimization alone is presented in Supplementary Figure 8. A comparison of the loss value as a function of the epoch number for the PMO-NET and the Analysis-NET is shown in Supplementary Figure 9.

To validate our new optimal PSF and analytical CNN, we evaluate the precision and accuracy of 3D positions and orientations from experimental measurements of single fluorescent DCDHF-N-6 molecules immobilized in a polymer composed of 1% poly(methyl methacrylate) (PMMA, Sigma Aldrich) in toluene (Thermo Scientific). The fluorescent molecules are diluted to a final concentration of 10 pM before being spin coated onto a HR coverslip (Corning, cat no 2850-22), resulting in a thin layer (a few dozens of nm thick) of immobilized single molecules in the polymer [27]. In addition, a droplet of water is added on top of the polymer sample to reproduce the index mismatch oil/water used in our simulation. We first coarsely detect single molecules from the raw data images. To accomplish this on our complex PSF shape, we first acquired an image of the observed field of view without applying phase modulation on the SLM. This same field of view is then be imaged with the optimized phase mask applied on the SLM. We use the open aperture images to coarsely estimate the positions of the single emitter using approaches described previously. The 26 × 26 cropped ROI on the Arrowhead PSF shapes is centered around these coarse position estimates. The coarse position of the same molecule’s PSF in the other polarized channel is determined by applying a previously estimated affine transformation on the coarsely localized pixel. As illustrated in Fig. 5a, we image the same field of view (FOV) at different axial positions by moving the sample with a nanometer precision stage to obtain a good axial sampling and to estimate the accuracy of the axial measurement provided by our Analysis-NET. The axial step size of 100 nm serves as a reference for the axial accuracy estimation. At a given axial position, the FOV is acquired for 100 frames and the localization precision of the 3D position and orientation is obtained from the standard deviation of the measured distribution for each detected molecule. As shown in Fig. 5b and 5c, we consider in our analysis the molecules with emitted photon numbers between 2000 and 7000 photons, with a median value of 3900 photons, and a background contribution between 10 and 60 photons per pixel per channel (median value = 40), similar to the signal and noise used during training. Finally, the precisions and accuracies discussed below have been obtained on 3D positions and orientations which are distributed as shown in Fig. 5d and 5e. Note, however, a greater density of in-plane molecules ($\theta ={90}^{{}_{°}}$) along the x-axis or y-axis of polarization ($\varphi =-{90}^{{}_{°}}$, 0° or 90°). Since the excitation is in the widefield configuration, the polarization component of the excitation is mainly in-plane, which allows us to excite the molecules at $\theta ={90}^{{}_{°}}$ more efficiently. Moreover, the emission of molecules oriented along the $x$ axis ($\varphi =0^{°}$) or the $y$ axis ($\varphi ={90}^{{}_{°}}$) will be collected primarily by one of the channels, which make them brighter and easier to detect, which leads to a higher population of analyzed molecules at these azimuthal angles. If in the future better sampling of z-oriented dipoles is desired, stronger pumping of such molecules can be achieved with, for example, annular illumination.

Fig. 5. Precision and accuracy obtained on single molecule immobilized at the coverslip.

(a) Workflow of the experiment used to estimate the localization precision, orientation precision and accuracy metrics from immobilizedsingle molecule images. (b) Detected photons distribution of localized single molecules. (c) Background per pixel per channel of localized molecules. (d) 3D scatter plot of the retrieved orientation. (e) Axial position distribution measured on the different localized molecules. (f) Estimated lateral localization precision obtained with the Analysis NET (blue) and corresponding CRLB (gray). (g) Estimated axial localization precision obtained with the Analysis-NET (blue) and corresponding CRLB (gray). (h) Estimated axial accuracy of the obtained with the Analysis-NET as a function of the estimated axial position. (i) Estimated orientation precision obtained with the Analysis-NET (dark purple) and corresponding CRLB (light purple). (k) Experimental workflow used to extract the orientation accuracy by comparing results from the Analysis-NET/optimized PSF and from Maximum Likelihood Estimation (MLE)/vortex PSF. (i) Experimental orientation accuracy. Please see Supplementary Figure 11 for superior accuracy performance when MLE is performed on the optimized Arrowhead mask.

Next, we compare the measured experimental precisions to the corresponding CRLB values and assess accuracy. The localization precisions for the lateral and axial positions are presented in Fig. 5f and 5g. As shown in Fig. 5f, the median value of the experimental lateral precision corresponding to the average value between the localization precision along $x$ and $y$, is 11 nm while the corresponding median CRLB value is about 5.6 nm, resulting in a ratio of 1.9. Regarding the axial precision, the distributions show an experimental median value of 33.6 nm and a CRLB median value of 13.8 nm, which corresponds to a ratio of 2.4. The estimation of the axial accuracy of the Analysis NET presented in Fig. 5h has been obtained by comparing the measured axial median value of each individual molecule retrieved during the stage-driven axial scanning. The accuracy as a function of the estimated axial distribution has a median value of −6 nm meaning that the results are not biased along the whole axial range. To quantify the orientation precision shown in Fig. 5i, we chose to compute the mean angular standard deviation as a metric like in [36]. The mean angular standard deviation ${\sigma }_{\delta }$ has the advantage of taking into account errors on both azimuthal and polar angles such as:

$${\sigma }_{\delta }=2arcsin(\sqrt{\frac{sin(\langle \theta \rangle ){\sigma }_{\theta }{\sigma }_{\varphi }}{\pi }})$$ (3)

Where $\langle \theta \rangle$ is measured mean polar angle and ${\sigma }_{\theta }$ and ${\sigma }_{\varphi }$ are respectively the error on the polar angle and the error on the azimuthal angle. The distribution shows a median value for the experimental orientation precision of 4.9° versus 3.5° for the expected orientation precision obtained with the CRLB calculation. The precision on the measured azimuthal angle $\varphi$ and the polar angle $\theta$ are shown in the Supplementary Figure 10. Finally, we estimate the orientation accuracy of the Analysis NET by comparing the orientation of individual molecules obtained sequentially with the Arrowhead phase mask and the vortex phase mask. Since the vortex phase mask has been used in two different previous works to obtain the orientation of single molecules, we consider that the recovered orientation can be used as an approximation to the ground truth orientation. The orientations measured with the vortex phase mask are obtained using the standard MLE fitting approach [30]. The accuracy of the orientation is obtained from the scalar product of the average orientation vector $\overrightarrow{O}{}_{AH}$ obtained with the Arrowhead phase mask and the average vector $\overrightarrow{O}{}_V$ obtained with the vortex phase mask. As illustrated in Fig. 5k, the relative angle $\alpha$ between these 2 vectors is then extracted using the arccosine of the scalar product as:

$$\alpha =co{s}^{-1}(\overrightarrow{O}{}_{AH}.\overrightarrow{O}{}_V)$$ (4)

The distribution of the retrieved accuracy value is shown in Fig. 5l. Once again, the resulting histogram of $\alpha$ presents a median value around 11 degrees, reflecting the good accuracy of the Analysis-NET. Since our method return the absolute difference, we do not expect a histogram symmetric around zero, and our sensitivity about the accuracy values is driven by the amplitude of the angle $\alpha$. Note that we performed a similar operation on the optimized experimental PSFs by comparing the distributions obtained by analysis-NET and by MLE fitting on the Arrowhead mask model. The results are shown in Supplementary Figure 11. With this comparison, we obtain an accuracy of 6 degrees, which differs from our 11 degrees accuracy estimation we made using the vortex PSF. We explain this difference by the fact that some aberrations (such as rotation of the PSF due to misalignments) are not properly corrected in the polarized vortex PSF. Therefore, the ground truth value estimated by the polarized vortex MLE fit may be slightly biased. In addition to the good precision and the good accuracy of our method over the different parameters of interest, the Analysis-NET returns the positions and orientation of 146,504 PSFs after almost 11 min (669 sec) of calculation, which correspond to 4 ms per molecule to obtain the 3D orientation and the 3D position.

5. Discussion

We have demonstrated the effectiveness of our deep learning approach for the simultaneous measurement of the 3D orientation and position of fluorescent molecules. Unlike classical phase mask optimization methods, deep learning approaches allow us to efficiently optimize the phase mask over a large set of the different parameters considered, even when these parameters belong to different spaces. The obtained Arrowhead phase mask was first evaluated theoretically using CRLB calculations over the entire axial range and for all different orientations. The consistency of the theoretical results validates this phase mask optimization principle. Moreover, the simultaneous learning of the phase mask itself and the CNN of the Incept-ResNet architecture allowed us to obtain a PSF whose spatial properties are consistent with the type of experimental configuration chosen. Thus, after evaluating the aberrations present in our optical device, we train the Analysis-NET based on the same architecture to extract the 3D orientation and position from the experimental data. The performance evaluation of the Analysis-NET shows experimental precisions approximately 2 times higher than the theoretical values obtained by the CRLBs. Although a loss of precision is observed compared to standard PSF MLE fitting approaches, the speed of extraction of the different parameters from the Analysis-NET is about 4 ms per PSF, which will be a major advantage in the context of imaging complex cellular structures where millions of locations are considered. In addition, we have shown that the accuracy of the different parameters obtained with the Analysis-NET are not biased over the entire range of parameters. Therefore, it would be reasonable to use the results obtained by the Analysis-NET as a starting point for further fitting by MLE to refine the localization precision without very high computation times.

As for all deep learning approaches, some concerns could be the impact of untrained situations that may appear experimentally such as single molecules with a high mobility, registration error or PSF overlapping. In an initial consideration of these different parameters with simulated data, we evaluated the precision of the localization and the accuracy of the results returned by the previously used Analysis-NET. The results, shown in Supplementary Figures 12 to 14, do not show excessive accuracy bias, and appear to maintain correct localization accuracy for the signal and noise considered. Note that, however, a non-uniform background with spatial structures (PSF overlapping, sample nonuniformities, excitation speckle…) leads to much larger biases of the order several hundred of nanometers axially for example (Supplementary Figure 15). The experimental PSFs with this type of non-uniform background must then be filtered before analysis to obtain a high-quality super-resolved image, perhaps with the BGnet architecture [34]. Finally, the optimization principles described here can be generalized and applied to other parameters such as spectral properties or molecule rotational mobility. For example, the rotational mobility could be obtained by a simple modification of the current code in order to return the second vector moment instead of the first moment vector, and by introducing a rotational mobility model such as the well-known cone model [44-46]. We look forward to use of the Arrowhead mask for cellular imaging in future measurements. As a possible extension to this work, the emitter could be placed some distance into the low index material where the forbidden light is not collected by a high NA objective. The proposed workflow is a general approach to optimizing phase masks for simultaneous measurement of the 3D position and orientation of an immobilized single molecule using the described pyramid microscope. An alternative experimental setup could be considered, such as using an independent SLM for each polarization channel, which would provide a greater degree of freedom and allow the use of two different masks, one for each detection channel.

Data availability

Data will be made available on request.