Holographic Sampling Display Based on Metagratings

Summary

Glasses-free three-dimensional (3D) display is considered as a potential disruptive technology for display. The issue of visual fatigue, mainly caused by the inaccurate phase reconstruction in terms of image crosstalk, as well as vergence and accommodation conflict, is the critical obstacle that hinders the real applications of glasses-free 3D display. Here we propose a glasses-free 3D display by adopting metagratings for the pixelated phase modulation to form converged viewpoints. When the viewpoints are closely arranged, the holographic sampling 3D display can approximate a continuous light field. We demonstrate a video rate full-color 3D display prototype without visual fatigue under an LED white light illumination. The metagratings-based holographic sampling 3D display has a thin form factor and is compatible with traditional flat panel and thus has the potential to be used in portable electronics, window display, exhibition display, 3D TV, as well as tabletop display.

Graphical Abstract

Highlights

• •Metagratings are designed pixel by pixel to form converged viewpoints in 3D display
• •Holographic sampling 3D display reconstruct discrete light field
• •Video rate full-color display is reconstructed with a thin form factor
• •Vergence and accommodation conflict is eliminated by single eye accommodation

Optical Imaging; Optical Physics; Wave Optics; Metamaterials

Introduction

The world by nature is three-dimensional (3D). Ideally, a manmade 3D display device should be able to reconstruct the entire information of the light field of a 3D object, independent of the observation method. Holography produces perfect 3D images by reconstituting the amplitude and the phase information of a scene, which can be viewed at a wide angle (Figure 1A). However, the amount of information required to produce a decent hologram is too large to be managed even by modern day electronics, leading to a limited refreshing rate or field of view (FOV) (Tay et al., 2008, Blanche et al., 2010, Butt et al., 2012, Yue et al., 2017). Multiple modulators or holographic projection are used to enlarge the view angle of video rate holography (Li et al., 2016, Zhang et al., 2017, Hahn et al., 2008, Yaraş et al., 2011, Kozacki et al., 2012, Wakunami et al., 2016) by increasing the complexity and the cost of the display systems. The computer-generated holography using metamaterials or metasurfaces can achieve high diffraction efficiency and large FOV (Ozaki et al., 2011, Larouche et al., 2012, Huang et al., 2013, Huang et al., 2015, Ni et al., 2013, Sun et al., 2013, Chen et al., 2013, Li et al., 2015, Almeida et al., 2016, Hu et al., 2019), but the reconstruction of full-color and video rate display are still difficult to accomplish by current methods.

Figure 1.

Work Principle of Metagratings-Based Holographic 3D Display

(A) Schematic of the wavefront reconstructed by hologram.

(B) Schematic of multiview 3D displays based on periodic structures.

(C) Schematic of holographic sampling displays based on metagratings. When the views are closely arranged, the sampled light field makes a good approximation of the continuous light field.

(D) Schematic of the full-color video rate 3D displays by combining an LCD, color filters, and the metagratings.

(E) Schematic of a pixel with three unit cells by combining three LCD units, R/G/B color filters, and the metagratings with varied spatial frequency and orientation. The emergent beams combine to form a colored pixel.

See also Figure S2, and Table S1.

To address the challenge of fulfilling a full-color video rate 3D display, so-called multiview 3D display produces 2D perspective images at a finite number of “views.” 3D display integrated with parallax barrier, lenticular lens array, or microlens arrays all falls into the category (Fattal et al., 2013, Yoon et al., 2011, Xia et al., 2013, Gao et al., 2016, Liu et al., 2017, Li et al., 2017, Martínez-Corral and Javidi, 2018). A prime example of diffraction-based multiview 3D display is made by HP lab a few years ago (Fattal et al., 2013). Periodic diffractive gratings were used to redirect the (semi-)parallel emitting light beams to multiple view angles. As a result, when two perspective images at different direction are captured by eyes, a virtual 3D scene is reconstructed in an observer's brain by binocular parallax. In other words, the reconstruction of 3D virtual objects in a multiview 3D display depends on whether or not there is an observer and whether we observe it with both eyes or with a single eye.

The employment of periodic micro- or nano-structures as the view modulator leads to two bottlenecks that limit the development of multiview 3D display. Firstly, the emergent light from the abovementioned periodic structures are (semi-)parallel beams. Crosstalk and ghost images happen at the transition region from one view to the other (Figure 1B). Secondly, the failure of reconstructing the phase information at each view leads to vergence-accommodation conflict, the major cause of visual discomfort in multiview 3D display. Vergence-accommodation conflict happens when the depth perception stimulated by binocular parallax is after or behind the display screen, whereas depth recognized by single eye is still at the apparent location of the physical display panel, because the image observed by each eye is still 2D (Inoue and Ohzu, 1997). Such mismatch confuses the brain and more severely, may stimulate nausea and vomit after long-time observation.

In this paper, we propose a holographic sampling 3D display by discretizing a continuous light field function at multiple viewpoints (Figure 1C). A view modulator fully covered with metagratings is utilized to modulate the phase information at each pixel, whereas an LCD (liquid crystal display) panel and a color filter are utilized for the reconstruction of amplitude information at R/B/G wavelength. By reconstructing the wavefront at sampling viewpoints, we greatly suppress the processing data and provide an opportunity for video rate full-color 3D display. When the viewpoints are closely arranged, a good approximation of light field function can be achieved. Compared with the multiview 3D display, the proposed holographic sampling 3D display has the advantages of minimum visual discomfort caused by crosstalk and vergence-accommodation conflict, tailorable viewing window, and simple fabrication process. Our work utilizes the thin and flat view modulator to build a 3D display screen, which is compatible with flat display panel, thus paving the foundation of holographic sampling 3D display for being used in portable electronic devices.

Results

The Mechanism of Holographic Sampling 3D Display by Metagratings

Figure 1D illustrates the schematic of a video rate holographic sampling display. By combining the view modulator with a standard LCD panel, both the phase information and the amplitude information of a viewpoint can be restored. In each pixel as shown in Figure 1E, three unit cells in the LCD panel are overlaid with R/G/B color filters, respectively. The white illumination light beam is divided into three monochromatic light beams with modulated intensity. They pass through three metagratings on the view modulator, and then three beams are combined to form a colored pixel with angular information. To summarize, the pixelated metagrating on the view modulator has three functions: (1) it endows the same phase information of monochromatic beams for R/G/B wavelength in a single pixel; (2) it reveals the phase delay of a single viewpoint over the display screen; and (3) it restores the phase delay between viewpoints.

By tailoring the phase delay between viewpoints, the video rate 3D display prototype features full freedom in the design of sampling viewpoints with arbitrary number of viewpoints and the angular separation between views (Figure S2). The balance between FOV, depth of field, and image quality can be designed according to the applications. Several designs are listed in the Supplemental Information for references (Table S1). In addition, the precise phase modulation by metagratings features tailoring of arbitrary viewing zones, such as circular viewing zones, triple viewing zones, cross viewing zones, for window display, tabletop display, or mobile display with horizontal motion parallax or full motion parallax. More importantly, we have complete freedom to design the angular resolution across the view zone. A gradient angular resolution can be achieved for decent optical quality at the central views and compromised optical quality at the periphery, when the total resolution is limited. Last but not the least, the configuration of the 3D display system is compatible with most mobile devices and can be easily integrated in a thin-form device.

The Design of View Modulator with Metagratings for a Single Viewpoint

Considering the optical disturbance at one sampling viewpoint, the light field is the summation of individual optical disturbance from each pixel at the display panel, so a converged light beam to a viewpoint is desirable. Metalens is a potential candidate to modulate light at the micron-/nano-scale. However, the size of a 3D display panel ranges from 5 inch to 100 inch depending on its applications. The big challenge in fabrication forces us to seek alternative solutions: metagratings. The pixelated metagratings that correspond to a single viewpoint form a segment of off-axis Fresnel zone plate. Viewpoints are formed when a series of segmented Fresnel zone plates are shifted and intervened. As a result, we modulate the phase information pixel by pixel. In comparison, periodic microstructures currently being employed (Fattal et al., 2013) in most multiview 3D display modulate a group of pixels, namely a voxel, simultaneously. It can hardly permit the accuracy of light field reconstruction over a large viewing angle across the whole display panel.

Figure 2A shows the schematic of the view modulator with metagratings for a single viewpoint. The variation of the spatial frequency and the orientation of the metagratings reveal the phase delay ranging from 0 to 2π between adjacent pixels and reconstruct the sectional phase of 3D scene for the sampled views of light field. The metagrating vector of the modulator can be calculated by the holographic recording and readout theory (Harvey and Vernold, 1998). The relationship between the incident beam and the emergent beam can be written as:

$${\mathit{k}}_d={\mathit{k}}_i-\mathit{G}$$ (Equation 1)

where |G| = 2π/Λ is the metagrating vector and Λ is the period of the metagrating. The k_i and k_d is the wave vector of the incident and diffracted light beam, respectively. Figure 2B shows an example of a row of the pixelated metagratings in the view modulator with varied spatial frequency and orientation. The structural parameters (the period Λ and the orientation θ) of the metagrating at each pixel can be calculated according to the position of the collimated incident beam and the position of the sampled viewpoint in the free space (Figure 2C). Moreover, the depth of the metagratings in the view modulator is optimized to improve the light efficiency. The detailed design principle of the metagratings can be found in Supplemental Information (Figure S1).

Figure 2.

Schematic of Metagratings in Holographic Sampling Display

(A) Schematic of the metagratings, capable of reconstructing one viewpoint under monochromatic illumination.

(B) Schematic of a row in view modulator for one viewpoint.

(C) The SEM of the metagratings in the flexible view modulator. Scale bar: 20 μm.

(D) The relationship between the increment of the period for the metagrating and the angular separation of view in the holographic sampling display.

(E) The relationship between the increment of the orientation for the metagrating and the angular separation of view in the holographic sampling display.

See also Figure S1.

Considering the design of a group of viewpoints aligned in a line for horizontal parallax, we calculate the phase delay between viewpoints. When the viewpoints are closely arranged in a continuous line, curve, or space, a view zone with sampled wavefront is formed. The design of the proposed holographic sampling 3D display is in favor of excellent angular resolution. The smaller the angular separation between the adjacent viewpoints is, the better the approximation of the sampled wavefront is achieved to the reconstructed light field. Therefore, small angular separation or better angular resolution is beneficial for continuous motion parallax and perceiving the depth cue of virtual 3D objects. With four tunable structural parameters, pixelated metagratings possess superior light manipulation capabilities. The minimum increment step of period and orientation of the metagrating determines the beam width of each focused view point and the minimum angular separation between views. Under the wavelength of 532 nm as an example, the increment of the period and the orientation of metagrating are 1.75 nm and 3.9° for an angular separation of 3° between views. When the angular separation between views drops to 0.7°, the periodic increment of metagrating needs to be as small as 0.12 nm and the orientation increment as fine as 0.95°, representing an incredible challenge to the fabrication of view modulator (Figures 2D and 2E).

Concept Demonstration

The fabrication of the view modulator with the capability to tune the period and the orientation of metagratings at sub-nanoscale is not trivial. A versatile homemade interference lithography system (Wan et al., 2016) was developed in order to generate finely controlled metagrating parameters at different spatial locations. The periodic variation of patterned metagrating is mechanically controlled by the longitudinal travel distance of the diffractive grating placed in a 4F system (Figure S3). The modulated accuracy for period of metagrating by the interference lithography system will be (see the Supplemental Information for more detailed demonstrations):

$$\Delta \Lambda =\frac{\Delta d}{F\left(f_1-d\right)}$$ (Equation 2)

where Δd is the accuracy of the longitudinal movement for the diffraction grating and F is the spatial frequency of interference fringe. The beauty of such interference lithography system is that the scaling factor is up to 100,000 fold. In other words, a mechanical movement of 10 μm along the optical axis corresponds to a variation of 0.1 nm in pitch for the patterned metagrating at a period of 400 nm (F = 2500 lines/mm), as shown in Figure S4. Because 10 μm displacement can be easily realized mechanically, the period of the patterned metagrating can be controlled at an accuracy better than 0.1 nm, which satisfies the design requirement of periodic increment of 0.12 nm for view modulator. Similarly, the orientation of the metagrating can be tuned by the rotation of the diffractive grating placed in the 4F system.

Based on the principle described, we fabricated several view modulators with increased complexity and screen size, the critical parameters of which are summarized in Table 1. The metagratings in the view modulators are nanoimprinted in flexible thin films (Figure 3A). There are 2560 × 1440 pixels in the view modulator, corresponding to 2560 × 1440 pixels in a shadow mask or an LCD panel. The angular separation between views ranges from 0.7° to 10° with a view number ranging from 4 to 64. Under the illumination of an LED white light, the transmitted light through the view modulator exhibits Gaussian distributions. The angular divergence is measured to be 1.4° (FWHM), 1.2°, and 0.9° respectively, in accordance with the theoretical limit. Furthermore, the irradiance beam illuminates the view modulator at an incident angle of 45° so that the zeroth-order diffraction beam from the metagrating is off-centered from the viewing zone.

Table 1.

Optical Performance of Three Typical Prototypes

3D Imaging Characteristics	Metagratings in Figure 4			Metagratings in Figure 3			Metagratings in Supplementary
Screen size	5 inch			6 inch			32 inch
Integration method	Shadow mask			LCD panel			LCD panel
Period of pixelated gratings	R:	G:	B:	R:	G:	B:	R:	G:	B:
	784 nm	640 nm	558 nm	638 nm	543 nm	462 nm	577 nm	520 nm	438 nm
	–	–	–	–	–	–	–	–	–
	903 nm	721 nm	617 nm	997 nm	847 nm	722 nm	1035 nm	935 nm	787 nm
Angular divergence	0.7°			1.2°			1°
Stereo distance	5.16 m			1.20 m			1.20 m
Depth of field	8 cm			5 cm			100 cm
Angular separation	0.7°			3°			3°
Field of view	11.2°			48°			96°
Number of pixels	2560 × 1440			2560 × 1440			3840 × 2160
Sub-pixel size	40 μm × 13 μm			35 μm × 12 μm			110 μm × 38 μm
View number	16			16			32

Also see Figures S3, S4, S6, and S7.

Figure 3.

Typical Prototypes of Holographic Sampling 3D Display

(A) The fabricated flexible view modulator with a thin format.

(B) A 6-inch prototype of proposed holographic sampling 3D display.

(C) The measured radiation pattern from a 16-viewpoint view modulator with metagratings under the illumination by a red LED light source.

(D) Whales, (E) Lego, and (F) Fish 3D images observed from left to right views showing natural motion parallax and colour mixing. Note that no chromatic aberration is observed between views. Scale bar: 2 cm.

See also Figures S3, S4, S6, and S7 and Video S1.

Figure 3B shows a 6-inch prototype of holographic sampling 3D display with a viewing angle of 50°.The radiation pattern from the adopted view modulator in Figure 3C shows a good brightness uniformity with a variation of 8% between views. Moreover, one can easily increase the FOV to more than 150° by increasing the view number or the angular separation, and we record the scene from various views as shown in Figures 3D–3F. The displayed virtual scene preserves natural motion parallax as in the real world (Video S1). In contrast, the lenticular-based 3D displays provide limited head movement with self-repeating views across view zones. Moreover, minimum between-view leakage and angular aliasing was observed.

The Elimination of Vergence-Accommodation Conflict

In multiview 3D display, the mismatch between the vergence and accommodation always causes considerable visual discomfort. Here we demonstrate that single eye can focus on the virtual object in holographic sampling 3D display when the viewpoints are close enough to approximate a continuous 3D light field wavefront. We fabricated several 5- to 6-inch view modulators with 16 viewpoints aligned in a straight line. The angular separation between views was 0.7°.

First, the view modulators were aligned with shadow masks. Under the illumination of R/G/B and white LED light, five lines of letters were projected at multiple virtual depth. A Rubik's cube placed right behind the view modulator was used as a reference in the real 3D space. Then a camera with tunable optical power was placed in front of the view modulator to record the 3D scene (Figure 4A). When we kept the position of the camera fixed and tuned the focal length of the camera, we could mimic the accommodation process of human eye. When the camera was focused at letters projected in front of the display screen (z = +2 cm), the letters behind, as well as characters in the Rubik's cube, were blurry as shown in Figure 4B. As the optical power of the camera decreases, letters at different virtual depth successively turn from blurry to sharp, then to blurry again (Figures 4C–4E). Obviously, the letters were indeed projected virtually at different depth (Video S2). As a result, the virtual 3D scene was reconstructed independent from the viewer. In addition, the virtual objects with broadband spectrum fuse well with each other and the physical world. Because the depth cue of the virtual objects can be captured by the accommodation process of human eye, the vergence-accommodation conflict, which is the major cause for visual discomfort in most multiview 3D display system, can be considerably eliminated.

Figure 4.

Holographic Sampling 3D Display with a Continuous 3D Light Field

For a Figure360 author presentation of this figure, see https://doi.org/10.1016/j.isci.2019.100773.

(A) Schematic of recording process of a 3D scene by a digital camera with tunable optical power to mimic the accommodation process of crystalline lens.

(B–E) Captured photos at different depth from a static holographic sampling 3D display. The camera changes its optical power to focus in front of the view modulator (B), at the front surface of the view modulator (C), behind the view modulator (D), and at further distance from the view modulator (E). Scale bar: 4 cm.

See also Figure S5 and Video S2.

We further aligned a view modulator with an LCD panel. A virtual scene of three letters at different depth is reconstructed. The depth cue of the letters “S”, “V”, and “G” can be captured by a camera (Figure S5). The physical toy of a duck placed right behind the LCD panel is at almost the same depth as the letter “G.” Again, the vergence-accommodation conflict is eliminated by holographic 3D display with closely arranged viewpoints.

Discussion

A laser light source would give better 3D display performances because of the small angular divergence of the irradiance and the narrow spectrum width. LED light is adopted because it is cheap and can be easily integrated in a portable device for directional backlights that are widely used in all kinds of illumination and display systems.

The metagrating-based holographic 3D display is beneficial especially for small screen (<12 inch) 3D display, because the phase modulation by surface profiling for a pixel with a size at tens of micron level by conventional computerized numerical control machining becomes difficult. Yet, we also demonstrate metagratings-based holographic sampling 3D display can be easily scaled up for large screen 3D display used in TV or exhibition applications, as shown in Figures S6 and S7.

To summarize, the metagrating provides a unique method for subsampling of continuous light field function. We have demonstrated that the metagrating with four tunable parameters (period, orientation, and 2D positions) has superior light manipulation capability by confining each view within a diffraction limited spot. Thanks to the significantly increased angular resolution, our eyes can accommodate to focus at the virtual 3D image. Not only crosstalk between views and ghost images are avoided but also the vergence-accommodation conflict is eliminated.

The proposed prototypes have the feature of thin form factor, compatible with current LCD panel, high spatial resolution, wide FOV, video rate display, and the most important, minimized accommodation-convergence conflict. We anticipate the holographic sampling 3D display to be used in commercial applications such as consumer electronic devices, window display, exhibition display, 3D TV, as well as tabletop display.

Limitations of the Study

The collimated illumination source is an important part in the 3D system for the reconstruction of continuous 3D light field without accommodation-convergence conflict. The angular divergence of the collimated irradiance will affect the view distribution. Future studies in the directional backlight module with thin form factor may pave the way for its integration and commercialization.

Methods

All methods can be found in the accompanying Transparent Methods supplemental file.

Published: January 24, 2020

Data and Code Availability

The data that support the findings of this study are available from the authors on reasonable request. See author contributions for specific datasets.