A holographic waveguide based eye tracking device

Abstract

We demonstrated the feasibility of using holographic waveguide for eye tracking. A custom-built holographic waveguide, a 20 mm × 60 mm × 3 mm flat glass substrate with integrated in- and out-couplers, was used for the prototype development. The in- and out-couplers, photopolymer films with holographic fringes, induced total internal reflection in the glass substrate. Diffractive optical elements were integrated into the in-coupler to serve as an optical collimator. The waveguide captured images of the anterior segment of the eye right in front of it and guided the images to a processing unit distant from the eye. The vector connecting the pupil center (PC) and the corneal reflex (CR) of the eye was used to compute eye position in the socket. A 3D printed model eye, which has a similar corneal curvature of human eye and laser pointer tube holder at the tail for simulation of eye gaze on a screen, was used for prototype validation. The benchtop prototype demonstrated a linear relationship between the angular eye position and the PC/CR vector over a range of 60 horizontal degrees and 40 vertical degrees. This prototype eye tracker has a tracking accuracy of 0.72 degree and tracking precision of 0.50 degree over the whole tracking range. These results confirmed that the holographic waveguide technology could be a feasible platform for developing a wearable eye tracker. Further development can lead to a compact, see-through eye tracker, which allows continuous monitoring of eye movement during real life tasks, and thus benefits diagnosis of oculomotor disorders.

1. Introduction

Binocular vision disorder (BVD) is one of the most common encounter in pediatric eye care clinics, second only to refractive errors, 8.5 times more common than all other ocular diseases combined (1). BVD are known to relate to school performance, reading problems and learning disability in school children (2–4). Although most clinical BVD management relies on the clinician’s subjective observation of abnormal oculomotor responses, direct recording of eye movements and accommodation is preferable, not only because it is objective and thus depends less on the expertise of the clinician and the cooperation of the patient but also because quantitative assessment of the ocular dynamics can lead to better understanding of the underlying oculomotor control dysfunction. Furthermore, there is usually a discrepancy between the testing condition for clinical BVD assessment and the condition for daily activities. For example, most vision tests for reading are performed at a standard viewing distance of 40 cm, but for various reasons, the actual reading/writing distance of many children in school or at home is half of that (5). How the excessive eye accommodation and convergence associated with such a short working distance may impact ocular development is unknown. A wearable, see-through device that resembles a pair of normal spectacles in front of the eye and that can monitor oculomotor responses during daily activities would be very useful in understanding and managing real world vision problems.

The most eye trackers nowadays are video-based (6). They use eye cameras to capture the images of the anterior segment of the eye and use image processing techniques to determine the pupil center (PC) and corneal reflection (CR) (7). For obvious reasons, the eye cameras must not obstruct the subject’s view of the outside world. Consequently, existing wearable eye trackers either place eye cameras above or below the eye or use a half mirror in front of the eye to guide the image of the anterior segment to cameras located above the eye or on the temporal side of the head. In the first approach, the camera looks at the eye from a steep oblique angle. Its view of the pupil and CR can be obscured by the eyelids on young children and subjects of certain races. Arranging the eye cameras around the eye also precludes bright pupil tracking, which is preferable to dark pupil tracking in subjects who have very dark irises. The second approach does offer a frontal view of the eye. However, the mirror in front of the eye is bulky and it weight can make the perch of eye tracker on the nose unstable. Such an eye tracker is usually embedded in a headgear similar to a ski goggle so that it can be fastened on the head with strong straps. Subjects may feel reluctant to wear such a device.

A planar holographic waveguide is a diffractive device, made of a pair of holographic couplers pasted on a piece of plane glass substrate. The holographic in-coupler serves to couple light signal into the glass substrate to start total internal reflection and the matching holographic out-coupler couples the light signal out of the glass substrate. The holographic waveguide thus has the effect of displacing light signals or images laterally over a distance of several centimeters (8). Holographic fringes on the in- and out-couplers are typically recorded into the photopolymer films by the interference of two coherent laser beams (8, 9). The films are then pasted on a piece of glass substrate. Optically, it functions like a pair of mirrors but is very compact and light-weight. From the very beginning, the research and development interest in planar holographic waveguide technology is its potential to support light-weight, compact, wearable displays (10, 11). Recent advances in the technology have resulted in some fashionable, wearable, consumer augmented displays, such as the SONY SmartEyeglass. The purpose of this study is not to develop a new holographic waveguide technology but to explore an important application of existing holographic waveguide technology, namely, eye tracking (12). In this new application, the holographic waveguide is not used as a display device but as an imaging device. It captures the images of the anterior segment of the eye, guide the image to a camera on the temple of a pair of glasses for eye position computation, without the cumbersome mirrors. The experimental results obtained in this study will serve as the basis for further development of holographic waveguide based eye tracking technology. The holographic waveguide is a diffractive device. It may attenuate the light signal and reduce image quality. The size of the waveguide may put a limit to the range of eye tracking. There has been no experimental assessment of holographic waveguide for eye tracking purposes. In this research, we developed a prototype eye tracker using an existing holographic waveguide and tested its ability to quantify the position of a model eye.

2. Materials and Methods

2.1. Prototype Eye Tracker and Model Eye

An existing green light (505 nm), transmissive holographic waveguide developed in a previous study for display (13,14) was used to develop and test a prototype eye tracker. The details of the holographic waveguide design can be found in reference (14). As shown in Fig. 1A. It consists of a 20 mm × 60 mm × 3 mm flat glass substrate with integrated 20 mm × 18 mm in- and out-couplers. The in- and out-couplers are photopolymer films with holographic fringes pasted on the glass substrate so that total internal reflection is induced in the glass substrate. An optical diffractive element with a 40-mm focal length was integrated into the in-coupler to serve as an optical collimator. The holographic waveguide was used in this research as an eye imager, the system design of which is shown in Fig. 1A.

Figure 1.

Optical apparatus of the holographic waveguide based eye tracker. (A) Schematic diagram of the optical system. (B) Photo of the setup.

A model eye was built to test the capability of the holographic waveguide imager to monitor eye position as shown in Fig. 1A. It was made of a 25 mm diameter ball. One segment of the ball was removed to create a 11 mm diameter flat surface. A plano-convex lens made of N-BK7 glass (refractive index 1.51, front surface curvature radius 7.7 mm and thickness 5.1 mm) was glued to this surface using optical adhesive (Norland optical adhesive 65, Thorlabs) to simulate the anterior segment of the human eye. An iris image with a 2.5 mm diameter round black spot in the center was inserted behind the flat side of the lens to simulate the iris and the pupil. On the opposite side of the ball, coaxial with the lens, was a tube holder. A laser pointer was inserted in the holder. Its beam provided a visible backward extension of the optical axis of the model eye. This was necessary because the model eye needed to be set to known positions during prototype eye tracker evaluation. To do so, we erected a screen on the back side of the model eye, made a marker on the screen whose location intersected the backward extension of the optical axis of the model eye when it was at a desired position, and directed the laser beam to that marker (Fig. 1B). The model eye was designed by a mechanical design software and printed by a high resolution 3D printer (Stratasys Objet30 Prime). To minimize the reflection from the flat surface of the lens, we roughened this surface using a piece of sand paper. The model eye was then mounted on a frame with two vertical supporting spherical surfaces holding the eye ball in the center. This allowed the model eye to be rotated around its own center as shown in Fig. 1B

A collimated green light from an LED source illuminated the model eye parallel to the norm of the corneal apex of the model eye when it is in the primary position (Fig. 1A). Our pilot tests showed that the field of view of the holographic waveguide we were using was not large enough to support reliable eye tracking. Consequently, a relay imaging system consisting of two achromatic lens, Lens 1 (100 mm in focal length) and Lens 2 (40 mm in focal length), was employed to image the anterior segment of the model eye onto an intermediate plane with a demagnification of ~2.5. Then the intermediate image served as the object of the waveguide imager, as shown in Fig. 1A. Demagnification is necessary to achieve a large field of view (24 mm × 24 mm) enough for imaging the whole anterior segment of the model eye. The light wave from this intermediate image was collected by the in-coupler of the holographic waveguide, propagated in the glass substrate, was coupled out of the waveguide, and was finally focused by a camera lens (8 mm in focal length) onto the camera senor of a Flea3 camera (FLIR FL3-U3-20E4C-C) as shown in Fig. 1A. The camera has 1600 × 1200 pixels, with a pixel size of 4.5 μm × 4.5 μm. In the experiment, a region of interest (600 × 600 pixels) is acquired at 20 frames per second. Images were taken when the model eye was rotated to different positions and were fed to a computer for further processing. The relative distance and orientation between the waveguide and the camera were carefully adjusted to ensure high diffractive efficiency and image quality. A holographic waveguide imager prototype that implemented the design of Fig. 1A is shown in Fig. 1B.

2.2. Eye Tracker Calibration Theory

Modern video-based eye tracking usually utilizes two optical landmarks of the eye, the pupil center (PC) and the corneal reflection (CR). Because of a fixed axial separation between the PC and CR, the PC-CR vector is directly related to the rotation of the eyeball around its center of rotation and is less affected by the lateral motion between eye camera and the eyeball. A custom Matlab program was used to compute the locations of PC and CR in each image of the anterior segment. The length and direction of the PC-CR vector was used to infer the angular eye position. We denote the offset vector between the centers of PC and CR as (V_x, V_y). The unit of V_x and V_y is in pixels.

The goal of the eye tracking is to associate eye tracker output, in our case, V_x and V_y, with a direction in space (gaze direction). A calibration process is needed to establish this association. In this process, eye tracker outputs at a set of known eye positions are collected to build a mathematical relationship that can be used to infer any eye position within the calibrated region from eye tracker output. To calibrate the prototype eye tracker, we used 25 calibration locations of a 5 × 5 array, [−30, −15, 0, +15, +30] deg horizontal and [−20, −10, 0, +10, +20] deg vertical (Fig. 1B). Minus angles were to the left and up and positive angles are to the right and down. The (0, 0) location represented the primary position of the model eye. All angles were measured from the center of the model eye. The calibration locations were marked on a screen perpendicular to the primary position of the model eye. For each calibration location, for example, S_x = −25 and S_y = +10, a pair of eye tracker output, V_x and V_y were obtained. A simple, linear mathematical model to associate (S_x, S_y) and (V_x, V_y) are given by below (15)

$$S_x=a_1+a_2{V}_x+a_3{V}_y$$ (1)

and

$$S_y=a_4+a_5{V}_x+a_6{V}_y$$ (2)

The 6 unknown coefficients a=[a₁, a₂, a₃, a₄, a₅, a₆] need to be obtained to compute an eye position (S_x, S_y) from a pair of eye tracker output. To obtain this coefficient vector a, we directed the model eye to the 25 calibration locations in sequence, compute the V_x and V_y values for each calibration location and plug the values into Eqs. 1 & 2. This generates a set of 50 linear equations.

$$\left(\begin{array}{l}\text{1 }{V}_{x1}{V}_{y1}0\text{ 0 0 }\\ \text{ : :}\\ \text{1 }{V}_{x25}{V}_{y25}0\text{ 0 0}\\ 0\text{ 0 0 1 }{V}_{x1}{V}_{y25}\\ \text{ : :}\\ 0\text{ 0 0 1 }{V}_{x25}{V}_{y25}\end{array}\right)\left(\begin{array}{l}{a}_1\\ a_2\\ a_3\\ a_4\\ a_5\\ a_6\end{array}\right)=\left(\begin{array}{l}{S}_{x1}\\ S_{x2}\\ :\\ S_{x24}\\ S_{x25}\\ S_{y1}\\ S_{y2}\\ :\\ S_{y24}\\ S_{y25}\end{array}\right)$$ (3)

We can simplify the notation of this equation as

$$Va=S$$ (4)

where V is a matrix of eye tracker outputs, S is a vector of known calibration locations and a is the unknown coefficient vector. Eq. 4 is an overdetermined linear equation set because number of equations is larger than the number of unknowns. The coefficient vector a was solved using a least squared method in Matlab. Once we obtain the coefficient vector a, we can compute any eye gaze position on the screen from the corresponding eye tracker outputs using Eqs. 1 & 2. A higher order relationship between eye tracker outputs and eye gaze positions can also be established (15).

3. Results

3.1. Eye Tracker Calibration

The prototype eye tracker was calibrated over a 60×40 deg area on a 5×5 rectilinear array. The laser beam from the back side of the model eye was visually directed to the markers of the calibration locations printed on the screen (Fig. 1B). The diameter of the laser spot on the screen was ~2 mm (~0.8 deg). A pilot test showed that the illumination to the border of the pupil began to fade when the model eye was rotated beyond 30 deg horizontally on either side. When the model eye was rotated more than 20 deg vertically, the supports above and below the model eye began to interfere with the laser pointer holder on the back side. Figure 2 shows images of the anterior segment of the model eye taken through the holographic waveguide imager when the model eye was directed to the 25 calibration locations. Each image contains 600×600 pixels. The pixel size was 60 μm by 60 μm on the pupil plane. The custom program identified the pixels of the pupil edge and fitted them with an ellipse (blue circle). The center of the best-fitting ellipse was designated as the pupil center (green dot). The program also identified the center of the corneal reflection, high-light spot, and its center (red circle and plus sign). The offset vector (V_x, V_y) at each point was computed and plugged into Eq. (3). The coefficient vector a was [−4.64, −0.90, 0.065, 4.56, 0.078, 0.93].

Figure 2.

Images of the model eye at the 25 calibration locations taken by the prototype holographic waveguide eye tracker. The numbers in the upper left corner of each panel are the eye position in degrees. The blue circles and the green dots are the elliptic fits of the edge of the pupil and their centers. The red dots are the centers of the corneal reflection.

3.2. System Accuracy and Precision

To evaluate the system accuracy and precision of the prototype holographic waveguide eye tracker, we perform 10 repeated measurements at the 25 calibration locations. The eye tracker outputs were converted to eye position in degrees using Eqs. 1 & 2 and the calibration vector a obtained from Section 3.1.

Figure 3 shows the scatter plots of the ten repeated measurements from the 9 eye positions in the upper left corner of the visual field. The stars represent the eye positions determined by the prototype eye tracker and the red circles represent the nominal eye positions. The accuracy of an eye tracker is defined as the distance between the actual target location and the average one of the ten eye tracker determined eye locations, indicating how closely the average eye position is relative to the target location (15). For example, at eye position (−30, 10) deg, the accuracy is 0.31 deg, while the accuracy at eye position (−15, 0) is 1.0 deg. The precision of an eye tracker is defined as the root mean square error of the ten repeated eye positions, indicating the dispersion of the repeated eye positions (15). For example, at eye position (−30, 10) deg, the precision is 0.28 deg, while the precision at eye position (−30, 20) is 0.8 deg.

Figure 3.

Scatter plots, accuracy and precision of the nine test locations in the upper left quadrant of the field. R means accuracy, and P denotes precision.

The accuracies of all the 25 tested locations are shown in Table 1. The average accuracy is 0.72 deg. with a standard deviation of 0.28 deg. The precisions of all the 25 tracking are shown in Table 2. The average precision is 0.50 deg. with a standard deviation of 0.12 deg.

Table 1.

Accuracies at different test eye positions.

Unit: degree	Sx=−30	Sx =−15	Sx=0	Sx=15	Sx=30
Sy=20	0.40	0.82	0.72	0.32	0.31
Sy=10	0.31	0.98	0.40	0.77	1.2
Sy=0	0.71	1.0	0.88	0.51	0.66
Sy=−10	0.72	0.78	1.02	0.98	1.12
Sy=−20	0.48	0.39	0.74	0.48	1.23

Table 2.

Precisions at different test eye positions.

Unit: degree	Sx=−30	Sx =−15	Sx=0	Sx=15	Sx=30
Sy=20	0.80	0.64	0.40	0.50	0.54
Sy=10	0.28	0.49	0.57	0.41	0.49
Sy=0	0.57	0.60	0.60	0.31	0.44
Sy=−10	0.33	0.60	0.57	0.54	0.56
Sy=−20	0.43	0.40	0.70	0.36	0.48

4. Discussions and Conclusions

To assess the feasibility of applying holographic waveguide technology to oculomotor function assessment, we used a ready-made holographic waveguide to build a prototype eye tracker and tested its ability to determine the position of a custom-made model eye. Without elaborate optimization of waveguide fabrication, eye illumination, image acquisition and image processing, we demonstrated that holographic waveguide could deliver images with sufficient quality for monitoring eye position over a wide range with moderate accuracy and precision. The following research and development will be conducted to improve tracking performance. First, the primary application of holographic waveguide is display. Consequently, it ubiquitously operates in visible light range, like the one used in the current research. Efforts will be made to develop near infrared holographic waveguides for imaging purposes. Second, while catching the frontal image of the eye requires a holographic in-coupler in front of the eye, there are non-diffractive ways to couple the image out of the waveguide, for example, a prism or a freeform element (16, 17). This will greatly improve the light efficiency of the imager with the additional advantage of reducing waveguide size. Third, new holographic waveguides will be designed to meet the field of view requirement for eye tracking without the aid of the relay lenses. This can be achieved by adjusting the wavelength, grating periods, index of refraction of the glass substrate of the holographic waveguide according to the set of grating equations that defines the waveguide field of view (14). Fourth, the waveguide can also be employed to guide illumination light to the eye, thus offers the opportunity for implementing more efficient illumination schemes, such as combination of bright and dark pupil tracking. Fourth, multiple in-couplers can be placed in front of the eye to capture images from different angles. This can greatly improve tracking range and accuracy. Finally, the real potential of holographic waveguide in research and clinical management of BVD is its ability to integrate accommodation measurement with eye tracking into one wearable device. The frontal placement of the waveguide in-coupler allows capturing images of the fundus through the pupil, thus making real-time monitoring of accommodation responses possible. We are conducting a research to explore this possibility. In conclusion, holographic waveguide can be a viable platform for developing wearable see-through oculomotor assessment devices. It is worth noting that although preliminary results using a prosthetic model eye has been presented in the SPIE conference (18), a comprehensive assessment of the holographic waveguide for eye tracking is still lacking due to the irregular shape of the prosthetic model eye.