Three-dimensional vectorial holography based on machine learning inverse design

3D vectorial holography reconstructs an arbitrary 3D vectorial field–carrying wavefront using machine learning inverse design.

Abstract

The three-dimensional (3D) vectorial nature of electromagnetic waves of light has not only played a fundamental role in science but also driven disruptive applications in optical display, microscopy, and manipulation. However, conventional optical holography can address only the amplitude and phase information of an optical beam, leaving the 3D vectorial feature of light completely inaccessible. We demonstrate 3D vectorial holography where an arbitrary 3D vectorial field distribution on a wavefront can be precisely reconstructed using the machine learning inverse design based on multilayer perceptron artificial neural networks. This 3D vectorial holography allows the lensless reconstruction of a 3D vectorial holographic image with an ultrawide viewing angle of 94° and a high diffraction efficiency of 78%, necessary for floating displays. The results provide an artificial intelligence–enabled holographic paradigm for harnessing the vectorial nature of light, enabling new machine learning strategies for holographic 3D vectorial fields multiplexing in display and encryption.

INTRODUCTION

Since its invention by Gabor (1), optical holography, which allows the reconstruction of both the amplitude and phase information of a three-dimensional (3D) image of an object, has propelled many advanced technologies including optical display (2–5), data storage (6, 7), optical trapping (8), holographic fabrication (9), pattern recognition (10), artificial neural networks (11), and all-optical machine learning (12). The 3D vectorial nature of light is fundamentally important to the understanding of the light-matter interaction (13–15). It should also play a substantial role in holographic optical trapping (16), high-resolution fabrication (9) and imaging (17–19), and high-capacity data storage (20). However, the 3D vectorial information of a holographic image has been completely inaccessible, primarily because of the negligence of the vectorial nature of light in conventional digital holography (21). Mathematically, embedding the vectorial information in a digital hologram requires an inverse solution to a complex 3D vectorial field distribution; however, such a complexity has remained a challenging endeavor to 3D vectorial holography.

Machine learning inverse design has revolutionized on-demand design of structures and devices including functional proteins in biology (22), complex materials in chemical physics (23), bandgap structures in solid-state physics (24), and photonic structures with previously unattainable functionalities and performance (25). As one of the most successful machine learning methods, artificial neural networks inspired by the brain architecture have been demonstrated to be an accurate and time-efficient approach to solving complex inverse problems in functional nanostructure design (26), machine learning microscopy and imaging (27), detection of quantum states (28), and all-optical machine learning (12).

Here, we demonstrate 3D vectorial holography through the use of machine learning inverse design based on multilayer perceptron artificial neural networks (MANN) for the time-efficient and accurate reconstruction of a 3D vectorial holographic image. The principle of 3D vectorial holography is depicted in Fig. 1, where a vectorial hologram with two digital functions of a phase hologram and a 2D vector field distribution is designed to reconstruct a 3D vectorial holographic image (kangaroo) (Fig. 1A). Hence, each pixel in the vectorial holographic image can be represented by a 3D vectorial field. To create an arbitrary 3D vectorial field with high purity, we train the MANN to statistically learn the relationship between a given 3D vectorial field in the image space and the 2D vector field distribution in the hologram plane (Fig. 1B). Notably, the longitudinal (E_z) component of a 3D vectorial field is achieved by a large-angle Fourier transform (FT) holographic lens embedded in the digital phase hologram (fig. S1). As a result, the wavefront leaving the vectorial hologram features a broad angular spectrum and transforms onto the sphere weighted by a vectorial distribution. This vectorially weighted Ewald sphere plays a key role in the lensless reconstruction of a 3D vectorial holographic image for floating display with an ultrawide viewing angle up to 94°.

Fig. 1. Principle of 3D vectorial holography based on the machine learning inverse design using the MANN.

(A) Schematic of floating display of a 3D vectorial holographic image based on a vectorial hologram, which contains a digital phase hologram and a digital 2D vector field derived from the MANN. (B) Schematic illustration of the generation of an arbitrary 3D vectorial field based on the MANN.

RESULTS

Inverse design of an arbitrary 3D vectorial field

The physical mechanism of the MANN for reconstructing a 3D vectorial field at a given position in the image plane is illustrated in Fig. 2A. The key is to determine the physical relationship between a given 3D vectorial field in the image plane and a 2D vector field distribution in the hologram plane. For simplicity, the 3D vectorial field is represented by E(α, β) in a spherical coordinate system, where α and β are the azimuthal and polar angles, respectively. The vectorial diffraction theory is generalized to evaluate the electric field components (E_x, E_y, E_z) of a 3D vectorial field by transforming a 2D vector field distribution into the Ewald sphere under the Helmholtz condition for 3D uniform imaging, which is a necessity for 3D holography (see note S1) (29). In this context, our machine learning inverse design algorithm reveals that the three orthogonal components (E_x, E_y, E_z) of a 3D vectorial field can be independently correlated to different azimuthal and radial spatial components of a 2D vector field in the hologram plane (Fig. 2B). The resultant correlation coefficients are close to unity in the transverse components and more than 0.9 in the longitudinal component for the holographic lens with numerical aperture (NA) larger than 0.8 (fig. S2).

Fig. 2. Machine learning inverse design of an arbitrary 3D vectorial field using the MANN.

(A) Schematic illustration of how a 2D vector field in the hologram plane is transformed to a 3D vectorial field in the image plane through a vectorially weighted Ewald sphere. Inset shows the definition of a 3D vectorial field in a spherical coordinate system. (B) The azimuthal spatial components modulated by a π-phase step with an orientation along the horizontal and vertical directions are used to independently manipulate the transverse electric field components E_x and E_y, respectively. A radial spatial component is used for the manipulation of the longitudinal electric field component E_z. The insets show the corresponding intensity distributions of these azimuthal and radial spatial components in the image plane. (C) Schematic of the four 3D vectorial fields derived from the MANN. The insets represent the MANN-derived 2D vector field distributions. (D) Experimental characterization of the three electric field components (E_x, E_y, E_z) of 3D vectorial fields through two-photon fluorescence imaging of the gold nanorods with an orientation along x, y, and z directions, respectively.

To achieve the inverse design of an arbitrary 3D vectorial field, we trained the MANN to produce a 2D vector field through synthesizing the weighted amplitude and phase of the derived azimuthal and radial spatial components (fig. S3). In our MANN model (see Materials and Methods), an input layer is a six-element array consisting of the amplitude and phase of the three orthogonal components of a target 3D vectorial field. Four hidden perceptron layers with 1000 neurons in each layer are used to learn the complex nonlinear relationship between the input and output data. The output layer represents another six-element array containing the amplitude and phase information of the derived azimuthal and radial spatial components. After an appropriate training, the MANN is capable of making the precise prediction for the blind testing dataset (fig. S4), exhibiting a small loss function with the mean square error of only 0.0021 (fig. S5). Moreover, the MANN could instantly (2.42 ms) predict incident 2D vector fields used for the generation of any desired 3D vectorial field targets (fig. S3) without conducting any forward diffraction calculation, which usually takes 8.68 s on the same computer.

Experimental verification of 3D vectorial fields

Hence, the generation of different 3D vectorial fields using the MANN was experimentally demonstrated (Fig. 2, C and D). A spatial light modulator (SLM) with a split screen (fig. S6) was used to prepare the MANN-derived 2D vector fields (see Materials and Methods) (fig. S7). To experimentally verify the three orthogonal components of a 3D vectorial field, two-photon fluorescence imaging of anisotropic single gold nanorods was carried out (fig. S8) (20, 30). The orientation of nanorods was determined through vectorial imaging using the azimuthal and radial polarization (see Materials and Methods and fig. S9), and thereafter, three nanorods with orientations along x, y, and z directions were selected as our localized nanoprobes to independently image the three orthogonal components of E_x, E_y, and E_z, respectively. Consequently, four arbitrarily selected 3D vectorial fields of E(0, π/4), E(0, 0), E(π/2, π/4), and E(π/4, π/2) with high purity in a 3D focal volume were experimentally produced (Fig. 2D), showing a good agreement with the simulation results in fig. S10. In addition, the high-purity 3D vectorial field was numerically characterized (fig. S11) and experimentally confirmed (fig. S12) for an off-axis position resulting from a blazed grating in the hologram plane.

3D vectorial holography

Applying the pure 3D vectorial field in the inverse design of a vectorial hologram, we demonstrate 3D vectorial holography, which allows the reconstruction of different 3D vectorial field distributions on a holographic image. The experimental approach of 3D vectorial holography is illustrated in Fig. 3A, where a vectorial hologram consisting of a MANN-derived 2D vector field and a digital phase hologram were used to reconstruct a 3D vectorial holographic image. To maintain the property of a 3D vectorial field on adjacent pixels in the holographic image, we sampled the target image by a 2D Dirac comb function with sampling constants (X, Y) larger than the intensity distribution of the 3D vectorial field in the image plane. Notably, the reconstructed electric field distribution in the image plane can be expressed as a convolution between a holographic image and a 3D vectorial field; hence, the holographic transformation of a 3D vectorial field into a 3D vectorial holographic image was handled outside the MANN. Consequently, 3D vectorial holographic images with different 3D vectorial field distributions were experimentally reconstructed, where their associated 3D vectorial fields of E(0, π/4), E(π/4, π/4), E(π/2, π/4), and E(3π/4, π/4) were verified on selected pixels from the reconstructed images, respectively (Fig. 3B and fig. S13). The simultaneous manipulation of the phase and vector distribution of a vectorial hologram through a split-screen SLM opens the possibility of holographic multiplexing of 3D vectorial fields (see Materials and Methods). Hence, multiplexing different 3D vectorial field distributions on a holographic image was demonstrated, wherein the four 3D vectorial field distributions (Fig. 3C) on different parts of the reconstructed image were experimentally verified through fluorescence imaging of localized nanoprobes (Fig. 3D).

Fig. 3. Experimental demonstration of 3D vectorial holography.

(A) Experimental approach of 3D vectorial holography based on a vectorial hologram, which consists of a MANN-derived 2D vector field and a digital phase hologram, respectively. (B) Experimental characterization of a 3D vectorial holographic image captured by a charge-coupled device (CCD). The insets show the two-photon fluorescence images of the three orthogonal components of a single pixel randomly selected from the reconstructed 3D vectorial holographic image. (C) Schematic illustration of the simultaneous generation of four 3D vectorial field distributions on a holographic image. (D) The experimentally reconstructed 3D vectorial field–multiplexed holographic image on a CCD, wherein the pseudocolors consistent with the ones in (C) were used to highlight the four different 3D vectorial fields. The insets of (D) show the experimentally characterized two-photon fluorescence images of the three orthogonal components of four pixels randomly selected from different sections of the multiplexed holographic image.

To reduce the pixel size imposed by an SLM, we use high-resolution 3D direct laser writing of a vectorial hologram for the lensless reconstruction of a 3D vectorial holographic image (Fig. 4). The two digital functions of a vectorial hologram including a 2D vector field distribution and a digital phase hologram were both implemented using laser-printed phase patterns in a transparent photoresist. In this context, a pair of phase patterns was printed to modulate two orthogonal circular polarization states, and their coherent superposition was used to generate the MANN-derived 2D vectorial fields, with the digitalization resolution identical to that of the printed digital phase hologram (Fig. 4A). In our experiment, phase patterns were fabricated with a resolution of 500 nm by 500 nm and a large size of 2 mm by 2 mm, respectively (inset of Fig. 4A). This high-resolution laser printing was experimentally verified by fabricating a binary diffraction grating, which results in the large-angle beam deflection by 52° at a wavelength of 808 nm (fig. S14).

Fig. 4. 3D direct laser writing of a vectorial hologram for the lensless reconstruction of 3D vectorial field–carrying and vectorial field–multiplexing holographic images.

(A) Optical setup for the lensless reconstruction of a 3D vectorial holographic image, wherein a pair of laser-printed phase patterns were used to modulate two orthogonal circular polarization for the generation of the MANN-derived 2D vector fields, which are further directed to a digital phase hologram. The insets present the optical images of the printed high-resolution phase patterns, each of which is with a size of 2 mm by 2 mm. The red arrows label out the laser beam propagation directions. BS, beam splitter; LP, linear polarizer; HWP, half-wave plate; QWP, quarter-wave plate; PBS, polarizing beam splitter. (B) Experimental characterization of the lensless reconstruction of a 3D vectorial holographic image. The insets show the two-photon fluorescence images of the three orthogonal components of a randomly selected pixel in the reconstructed vectorial holographic image. Scale bar, 1 μm (inset). (C) Experimental verification of the divisibility property of a vectorial hologram based on different sections of a laser-printed vectorial hologram. The insets show the optical images of the laser-printed sectional vectorial holograms and their experimentally reconstructed holographic images, respectively. (D) Schematic illustration of the 3D vectorial mapping of eight vector-field distributions in a designed spiral shape as a function of azimuthal angle α. (E) Experimentally reconstructed 3D vectorial field–multiplexed holographic image designed in (D) on a CCD, wherein eight enlarged pixels with pseudocolors were used to highlight the different 3D vectorial fields (insets). Scale bar, 1 μm (inset).

On the basis of the high-resolution vectorial hologram, we demonstrate the lensless reconstruction of 3D vectorial field–carrying and vectorial field–multiplexing holographic images. To achieve the lensless reconstruction, we embedded a large-angle FT holographic lens with an NA of 0.8, capable of creating a 3D diffraction-limited focus in the image plane, in the digital phase hologram (fig. S15). As a result, the lensless reconstruction of a 3D vectorial field distribution of E(π/4, π/4) on a holographic image was achieved, exhibiting an ultrawide viewing angle of 94° and a high diffraction efficiency of 78% (Fig. 4B). This ultrawide viewing angle suggests that the reconstructed 3D vectorial holographic image is floating on the vectorial hologram, with the image size that can be linearly scaled by the hologram size (see note S2 and fig. S16). In addition, our demonstrated vectorial hologram is divisible, and hence, sectional vectorial holograms are capable of reconstructing the same-size holographic image but with a reduced diffraction efficiency and 3D vectorial field purity (Fig. 4C and fig. S17). Consequently, the lensless reconstruction of a 3D vectorial field–multiplexing holographic image was experimentally obtained, wherein eight different 3D vectorial field distributions (Fig. 4D) were simultaneously reconstructed on the holographic image (Fig. 4E).

DISCUSSION

Our demonstration provides an artificial intelligence–enabled holographic paradigm shift for harnessing the previously inaccessible 3D vectorial nature of light in holography. We demonstrate that the generated 3D vectorial fields are capable of addressing emerging 3D metasurfaces (31) for the display of 3D vectorial field–encrypted optical information through the excitation of hybrid transverse and longitudinal resonant modes in a 3D oriented silicon nanoantenna array (fig. S18). Notably, our demonstrated vectorial hologram with two digital functions of a 2D vector field distribution and a phase hologram allows the reconstruction of a 3D vectorial wavefront, providing unprecedented wavefront manipulation capabilities by extending from 2D polarization states (32, 33) to arbitrary 3D vectorial fields. Unlike the direct optimization of 3D polarization states with low on-axis polarization purity based on the trial-and-error method in microscopy (20, 34), the trained MANN can be applied to instantly and accurately generate any desired 3D vectorial fields (Figs. 2C, 3C, and 4D and fig. S3) without conducting any further calculation. Moreover, unlike the conventional methods that allow the evaluation of only a few vector beams (20, 32–34), the vectorial diffraction theory has been generalized to be able to evaluate any given 2D vector field distribution. In our experiment, an interferometer was used to generate any desired 2D vector beam before focusing on an arbitrary 3D vectorial field. Compared to the q-plate approach (35), it has a strong advantage of the independent and dynamic control over both polarization and phase (see Materials and Methods), opening the possibility of on-demand 3D vectorial holography. In addition, it allows the complete access to any desired vector beam and therefore is beyond the q-plate approach, which only allows the generation of vector vortex beams on higher-order Poincaré spheres.

Extending digital holography to the 3D vectorial domain is crucial to record and reconstruct 3D vectorial wavefronts. This extension has witnessed notable interests in recent years in such applications as polarization holography for data storage (36), Stokes holography (37), and polarization digital holography (38). 3D direct laser writing of a vectorial hologram in a transparent photoresist thin film is a scalable technology, and its merging with 3D vectorial holography may open up an unprecedented opportunity for ultrahigh-capacity holographic devices and systems with high diffraction efficiency. The holographic multiplexing of randomly distributed 3D vectorial fields exhibits 3D vectorial vortices/singularities (fig. S19), which can provide a new degree of freedom in Möbius strips (13), phase singularities (14), and optical vortex knots (15). The demonstrated 3D vectorial holography may open avenues to widespread applications such as holographic trap display (5) and multidimensional data storage (39), machine learning microscopy, and imaging systems (27). In particular, our demonstrated 3D vectorial holography with MANN may provide a useful tool to precisely design and tailor on-demand 3D trapping force inside a single optical trap or among multiple traps (16).

MATERIALS AND METHODS

MANN model

The general architecture of the MANN consists of an input layer with six input elements, an output layer with six output elements, and four more hidden layers with hidden neurons. For this problem, we established a small MANN with only four hidden layers with 1000 neurons in each hidden layer. The input vector is a six-element array containing the amplitude (E_0x, E_0y, E_0z) and phase (ϕ_x, ϕ_y, ϕ_z) of any desired 3D vectorial field (fig. S3). The input data were normalized (minimal to maximal scaling) before applying to the training model. The output is another six-element array, containing the amplitude (E_ax, E_ay, E_r) and phase (ϕ_ax, ϕ_ay, ϕ_r) of the azimuthal and radial spatial components, respectively. A rectified linear unit was used as the activation function between each layer, due to the fact that the rectified linear unit is suitable for the input value in the range of [0, 1]. We used the uniform distribution to initialize all weights for neurons in each layer. The mean square error was selected as the loss function due to a typical regression problem. To find the optimal weights for the MANN model, we used Adam as the optimizer.

We set the batch size as 5000 and split the data into three subdatasets: training (576,000 data samples), validation (144,000 data samples), and testing (144,000 data samples). The test dataset was not used for training the model, which means that the trained model is blind to the test data. Notably, the MANN is capable of estimating the real value for the blind test data. The hyperparameters are not heavily tuned, which suggests that the better performance is possible. For all test session, we used five-folder validation method to evaluate the result. All test samples were randomly partitioned into five equal-sized subdatasets. In each time, we used a single subdataset as the testing dataset and other four subdatasets as the training datasets. This process was repeated five times, and the average results were reported in the paper (figs. S4 and S5).

Generation of a 2D vector field

We show that a 2D vector field with spatially variant polarization can be generated by coherently superposing two orthogonal circular polarizations with phase modulation. The optical setup for the generation of a 2D vector field is given in Fig. 4A and fig. S6 based on a pair of high-resolution laser-printed phase patterns and a split-screen SLM, respectively. In fig. S6, a laser beam passing through a linear polarizer (LP) had a vertical polarization state, which was further changed to a linear polarization with an orientation of 45° through a half-wave plate. This linearly polarized beam was split into horizontally and vertically polarized components with an equal amplitude through a polarizing beam splitter. The horizontally and vertically polarized components were phase modulated by an SLM (Hamamatsu X13138-02) with a split screen. Hence, two different phase modulations (Φ_L and Φ_R) were successfully implemented in the horizontal and vertical polarization components, respectively. Thereafter, the two orthogonal linear polarization components were coherently superposed by a beam splitter and then converted to orthonormal circular polarization components via a quarter-wave plate, with an electric field vector of $\mathbf{E}=\frac{1}{\sqrt{2}}(e^{i{\mathrm{\Phi }}_{\mathrm{L}}}{\mathbf{e}}_{\mathrm{L}}+e^{i{\mathrm{\Phi }}_{\mathrm{R}}}{\mathbf{e}}_{\mathrm{R}})$in the circular polarization basis, where Φ_L and Φ_R denote the phase modulation imprinted by the left and right half screen of the SLM, respectively, and e_L and e_R represent the left- and right-handed circular vector bases, respectively. Hence, through transforming the coordinates into the Cartesian coordinates, an arbitrary 2D vector field of $\mathbf{E}=e^{i\mathrm{\Psi }}\left[\begin{array}{c}\text{cos}\mathrm{\Theta }\\ \text{sin}\mathrm{\Theta }\end{array}\right]$ can be obtained, where Ψ = (Φ_L + Φ_R)/2 and Θ = (Φ_L − Φ_R)/2. Therefore, the phase Ψ and polarization Θ information of an optical beam can be simultaneously modulated.

Two-photon fluorescence imaging of single gold nanorods

Because of the fact that single fluorescent gold nanorods emit an anisotropic radiative pattern of an oscillating electric dipole, the transverse and longitudinal orientation of single nanorods could be explicitly determined by fluorescence imaging based on an azimuthal and radial polarization states, respectively (fig. S9). The two-lobe shape of the scanned image under the azimuthal polarization is highly dependent on the orientation of single gold nanorods, laying the basis of determining the transverse orientation of gold nanorods. On the other hand, a circular shape of the scanned image under the radial polarization, while disappearing under the azimuthal polarization, could be used to determine the nanorods oriented along the longitudinal axis, due to the fact that the azimuthal polarization is depolarization-free in nature. Consequently, three gold nanorods labeled as “1,” “2,” and “3” were selected as our localized nanoprobes to experimentally characterize the electric field components of E_x, E_y, and E_z, respectively.

Numerical simulation of optical responses of 3D oriented nanoantennas excited by 3D vectorial fields

We used a PythonToolkit (40) derived from the Dyadic Green Method to calculate different physical quantities, including near- and far-field patterns, scattering cross section, and back focal plane images through interacting with nanostructures with arbitrary shapes. To properly consider and evaluate the excitation of 3D vectorial fields on nanostructures, we incorporated the generalized vectorial Debye diffraction theory, capable of creating arbitrary 3D vectorial fields, into the python toolkit for full-field electrodynamical simulations of nanostructures. Hence, the scattering spectral and back focal plane images of a 3D oriented nanoantenna under the excitation of different 3D vectorial fields were numerically simulated in fig. S18.

Preparation of single gold nanorods for fluorescence imaging

Gold nanorods with an extinction peak at the wavelength of 790 nm and an average aspect ratio of 4 ± 1 were prepared following the wet chemical methods (20, 36, 39). To prepare single gold nanorods for nanoprobing the 3D vectorial components, the gold nanorod solution was diluted with an optical density of 0.25 and drop-casted on the polyvinyl alcohol–coated cover glass. This optical density was optimized to ensure the sample containing sparsely distributed single gold nanorods without aggregation while maintaining sufficient single nanorods with a 3D random orientation. To excite the surface plasmon–enhanced two-photon luminescence without photon thermal reshaping single gold nanorods, we used an incident power of 200 μW at the back focal plane of a high-NA lens (Figs. 2 and 3) and at the sample surface of a vectorial hologram (Fig. 4) in our experiment, respectively.

Preparation of photoresist sample for 3D direct laser writing

Our vectorial hologram with eight-level depth-controlled phase modulation was experimentally printed using the Nanoscribe. Specifically, a pair of phase patterns used for the generation of the MANN-derived 2D vector fields and a digital phase hologram embedded with a large-angle FT holographic lens with an NA of 0.8 was printed in an IP-Dip photoresist (IP-Dip, Nanoscribe). The printed phase patterns feature 4000 pixel by 4000 pixel and a physical size of 2 mm by 2 mm. In our experiment, the IP-Dip with the low proximity effect was drop-casted onto a cover glass substrate.

SUPPLEMENTARY MATERIALS

Supplementary material for this article is available at http://advances.sciencemag.org/cgi/content/full/6/16/eaaz4261/DC1