Real-time tomographic holography for augmented reality

Abstract

The concept and instantiation of Real-Time Tomographic Holography (RTTH) for augmented reality is presented. RTTH enables natural hand-eye coordination to guide invasive medical procedures without requiring tracking or a head-mounted device. It places a real-time virtual image of an object's cross-section into its actual location, without noticeable viewpoint dependence (e.g. parallax error). The virtual image is viewed through a flat narrow-band Holographic Optical Element with optical power that generates an in-situ virtual image (within 1 m of the HOE) from a small SLM display without obscuring a direct view of the physical world. Rigidly fixed upon a medical ultrasound probe, an RTTH device could show the scan in its actual location inside the patient, even as the probe was moved relative to the patient.

Most medical imaging augmented-reality systems depend on tracking of the viewer relative to either the world and/or to some image-scanning device [1]. To avoid the undesirable technical and ergonomic requirements imposed by use of a tracking system [2], some researchers have previously mechanically mounted a 2D display and a half-silvered mirror in the proper positions on an ultrasound probe such that by the law of reflection the mirror produces a virtual image of the 2D display that is in situ, i.e. precisely located (registered) in 3D space with the slice of tissue scanned by the ultrasound probe, regardless of viewpoint or probe position [3, 4]. An entire 3D volume can be interrogated tomographically (i.e. in a series of slices) by moving the ultrasound probe to show multiple ultrasound slices in their correct 3D locations; the correct ultrasound data will be displayed at each point in the entire 3D volume inside the patient. This see-through autostereoscopic augmented-reality display technique has been termed Real-Time Tomographic Reflection (RTTR) [4].

Here we present an extension of the RTTR concept, which we first proposed but had not yet implemented in [5]. By replacing the semi-transparent mirror with a narrow-band holographic optical element (HOE), as shown in Figure 1, we are able to maintain all of RTTR's above benefits without our device geometry being constrained by the law of reflection and without the substantial attenuating effects of a semi-transparent mirror. Our approach, analogously termed real-time tomographic holography (RTTH), can produce a 2D virtual image that is always correctly positioned and larger than the source spatial light modulator (SLM) display, with a less obscured view of the patient and with substantial flexibility in the position and orientation of the HOE and the (monochromatically illuminated) SLM. These benefits derive directly from the flexibility of an HOE. A flat-surface (e.g. volume phase) HOE appears transparent if it has sufficiently narrow bandwidth (Bragg window). An HOE can act as a focusing optic and simultaneously as a diffraction grating. The HOE's optical power allows flexibility in the virtual image's size and distance from the HOE, while the HOE's grating aspect allows non-axial flexibility in the position of the HOE and SLM display. Accordingly, the SLM (e.g., an LCD back-lit by a laser) can be permanently located out of the way as depicted in Figure 1, allowing room for the long surgical tools required by liver biopsy or amniocentesis without making the RTTH system unwieldy. The chief difficulty of designing an RTTH system is maintaining a desired geometry while achieving sufficiently sharp focus for the virtual image's position to be adequately (e.g. to within 1 mm) independent of viewpoint, i.e. autostereoscopic and in-situ like RTTR. In contrast to RTTH, head-up displays (HUDs) typically have virtual-image distances from 3 m to ∞ and loose registration requirements.

Fig. 1.

An HOE can be used to project a nearly viewpoint-independent autostereoscopic 2D virtual image at the actual 3D location and orientation of the data in real time. The size, shape, and position of the virtual image are not fixed by that of the SLM image source.

Designing an RTTH system with sufficient viewpoint independence (focus) to enable natural hand-eye coordination to guide invasive procedures is non-trivial. Producing a virtual image ≤ 1 m away and larger than the SLM dispaly requires the system to have a relatively short effective focal length (EFL), and the requirement for stereoscopic human viewing (essential for depth perception) imposes a hard limit on the minimum size of the HOE viewing aperture. Furthermore, conveniently positioning the SLM image source relative to the virtual image to allow both an unobstructed view and room for operating tools, e.g. as depicted in Figure 1, requires a non-axial optical design. Finally, because the HOE is large and the virtual image is near, the resultant range of viewpoints is relatively large as well. Thus, an RTTH system designer is tasked with designing a well-focused, fast (i.e. low f/#), non-axial optical system with a wide viewing angle. Fortunately, RTTH system design can include customization of the image source to compensate for both planar geometric distortions and field curvature in the rest of the optical system. The former can be corrected by using real-time pre-processing software to introduce in-plane warping of the image, and the latter by incorporating a custom non-planar fiber-optic faceplate as part of the SLM-based image source. Note that the pixel size of the face plate/SLM image source affects the virtual image's resolution, but not it's viewpoint independence. Viewpoint independence, i.e. the amount of perceived spatial drift in virtual image points across the allowable viewpoint range, is the most important resolution metric for a working RTTH system, and it is equivalent to the point spread function of the optical system (face plate to HOE) over the viewing aperture encompassing the entire specified field of view.

We now present the first instantiation of RTTH, a proof-of-concept for which we chose to project a planar virtual image of modest size (104×112 mm) at a distance of 1 m from the HOE. Although somewhat greater than the optimal distance for our ultimate clinical application of ultrasound, using a 1 m distance helped relax our requirement for a short EFL. We relaxed our f/# requirement by specifying a square, 5×5-in. HOE (1 in. = 2.54 cm), which is a relatively small aperture for stereoscopic viewing. This combination of HOE size with virtual image size and distance also relaxed our viewing angle requirement. We used a narrow-band 532 nm volume-phase HOE (VHOE) in order to make the HOE appear as transparent as possible, and our SLM image source was a 19×16 mm LCD coupled with a diffusing fiber-optic face plate and illuminated by an expanded 532 nm laser beam. Unfortunately, the small pixel pitch of such an LCD causes it to also act as a two-dimensional diffraction grating when illuminated by a laser, in our case inducing a one-pixel-radius blur on the surface of the face plate. Following standard practice, our design process utilized optical simulation and optimization software, including Zemax EE. Because an unfocused virtual image lacks a single focal plane, we traced the rays in reverse, from the desired virtual image's co-planar field points to the surface of the face plate/SLM image source.

All of our initial attempts at designing such a system failed to achieve sufficient focus, even with numerically optimized corrective optics added between the SLM and the HOE. We gained the additional design freedoms necessary for success by employing non-spherical wavefronts when recording the VHOE [6]. Doing so required that we optically fabricate our VHOE using one traditional (spherical) recording beam and one beam that was specially shaped by an extra construction optic, which was only used to record the HOE, and not during HOE playback. The construction optic was located between the spherical-wavefront-emitting construction point and the VHOE recording surface. We simultaneously simulated (ray-traced) and optimized the system's playback performance across variations in both the RTTH device (the “playback” system) and the HOE-recording system [6], including every aspect of corrective (playback) optics and multiple potential construction optics on each recording beam. We believe we are the first to use this technique to improve the focus of an off-axis, virtual-image projecting HOE. One of the HOE's recording beams was a spherical wavefront from a construction point 45° off-axis at 994.2 mm. The other construction beam, Figure 2, utilized one tilted, offset construction lens (BK7 PCX, R = 38.76 mm) to modify the HOE to counterbalance the playback system's aberrations. The design's playback optical layout, Figure 3, utilized two correction lenses (BK7 PCX, R = 77.26 mm and R = 257.55 mm) to counterbalance aberrations and offload some of the requisite optical power from the HOE. Our system performed acceptably in simulation (Figure 4).

Fig. 2.

A construction optic in one of the construction beams allowed the HOE to counterbalance aberrations.

Fig. 3.

(Color online) (a) Final playback layout of our RTTH system. (b) and (c) Photos showing fetal ultrasound projected inside the mother. The images appear to have higher resolution in person, due primarily to the larger dynamic range and smaller pupil size of the human visual system. The left photograph shows the ultrasound image floating inside the mother, as viewed through the HOE. The right image is “zoomed in,” showing part of the fetus, oriented facing left with the head down.

Fig. 4.

(Color online) Zemax analysis of the worst (left) and best (right) image points. The point spreads across the surface of the face plate can be multiplied by the optical system's magnification factors (5.39 horizontally and 7.23 vertically) to predict the stability of these points in the virtual image. The long, sparse tails represent viewpoints for which the entire virtual image is not simultaneously in view.

We built a working prototype in order to validate our simulation. Wasatch Photonics, Inc. manufactured our HOE utilizing our construction optic according to our design. They used proprietary techniques and a dichromated gelatin (DCG) recording medium to produce an evenly exposed VHOE with high diffraction efficiency and narrow Bragg window. We found that the prototype playback system qualitatively demonstrated the key advantages of RTTH, projecting a virtual image that was focused, larger than the real-time LCD SLM, and off-axis from the LCD SLM. The image's location, L, relative to the RTTH display appeared viewpoint independent and much closer than that of a typical HUD. A large-aperture camera lens, however, revealed an unexpected first-order astigmatism (as would be caused by a cylindrical lens). Although L was correct for the virtual image's vertical focal plane, the horizontal focal plane was displaced 3 cm toward the HOE. The significance of such astigmatism on depth perception over the normal range of human pupil size has yet to be determined.

The viewpoint-independence of L was measured over the viewing cone that kept the entire virtual image simultaneously in view. At 40 mm from the HOE, the viewing cone's cross-section measures 64 mm horizontally by 59 mm vertically, corresponding to an angular viewpoint range, relative to the virtual image's center, of about 2.7° horizontally by 2.5° vertically. Viewpoint independence was quantitatively determined by aligning virtual-image cross-hairs with a co-planar physical grid (Figure 5), and then taking high resolution digital photographs from multiple viewpoints across the viewing cone. Within the viewing cone, the worst-case points in the virtual image changed position by no more than 0.99 mm vertically and 0.83 mm horizontally. Assuming a system properly calibrated to the center of that range, scanned objects should be perceived as being located within 0.5 mm laterally of their actual scanned location. The chief aberrations causing this imprecision are coma and third-order astigmatism, typical for such an off-axis system. Pixel-width lines on the SLM display produced virtual-image lines at most 0.97 mm wide. The aligned physical target also verified virtual image position and orientation, subject to the above astigmatism, at 100 cm from the HOE at an angle of 33±1°. A demonstration of in-situ projection of fetal ultrasound is shown in parts (b) and (c) of Figure 3. For this demonstration, the “real-time” video feed was actually previously recorded ultrasound.

Fig. 5.

(Color online) The virtual image remained aligned with a physical target (a) to within 0.5 mm across viewpoints, e.g. (b) and (c). The maximum pixel width was 0.97 mm (d).

In conclusion, we have successfully designed, built, and tested the first optical system suitable for RTTH, using a large-aperture HOE to project an off-axis, viewpoint-independent virtual image ≤ 1 m away and larger than the SLM display. This research was partially supported by the National Institute of Health R21 grant EB007721, and by the National Science Foundation grant 0308096.