THE HUMAN PUPIL AND THE USE OF VIDEO-BASED EYETRACKERS

Abstract

Video-based devices for measuring gaze direction are widespread. However, there is a built-in imprecision in such devices in the event that pupil diameter changes during the experiments. Data are presented to demonstrate this effect. The possibility of correcting eye-position records for the imprecision is discussed and preliminary examples of such correction are presented.

INTRODUCTION

Most current research involving measurement of gaze direction (“eye position”) employs video-based eyetracking devices, which provide a relatively simple and cost-effective way of obtaining two-dimensional records. The usual method of determining eye position used by these devices is based on the relationship between the pupil center¹ and image of one or more light sources formed by the corneal first surface (the first Purkinje image of the source). For angles of eye position up to ±30 deg relative to straight-ahead, the pupil center shifts approximately linearly with eye position, while the minified corneal reflection shifts considerably less. The horizontal and vertical differences between pupil center and corneal reflection location, once they have been calibrated using fixation targets at known locations, provide fairly accurate estimates of eye position. Some degree of nonlinearity can be dealt with by fitting polynomials to the calibration data. For small head movements, both translational and rotational, if fixation is maintained on the same relatively distant target, the video camera sees the front of the globe shifted a little in the field of the camera but with an unchanged relationship between pupil and corneal reflection. Thus, there is considerable stability of the eye position estimate in the face of head movements. (This is not the case if pupil position or corneal reflection alone is used, although either does provide a raw signal related to gaze direction. In such a situation, small head movements generate large contaminating signals which are interpreted as eye movements. Mounting the video camera firmly on the head can reduce the effect of translational head movements, although rotational head movements remain a problem.)

Although video-based eyetracking devices are simple and useful in many circumstances, there is at least one potential problem with using them to measure small variations in eye position: if pupil diameter changes, the center of the pupil does not remain fixed relative to the eyeball. As discussed in earlier work, shifts in pupil center between light and dark conditions can be as large as several tenths of a millimeter, with the direction of shift differing from subject to subject and between eyes of the same subject (Wyatt, 1995). Since a 1 mm shift of the pupil center, with a fixed pupil diameter, approximately corresponds to a 10 deg eye turn, variations of 0.1 or 0.2 mm in pupil position should approximately correspond to 1 or 2 deg eye movements (Wyatt, 1995). In other words, shifts of the pupil center associated with changes of pupil size can generate spurious signals of gaze direction. The present study addresses this issue.

METHODS

Seven normal subjects were recruited from the community at the SUNY State College of Optometry. All subjects had received a comprehensive ocular examination in the University Optometric Center at SUNY College of Optometry; no indications of ocular disease were found. Best corrected visual acuity was 20/20 or better for all subjects. The research adhered to the tenets of the declaration of Helsinki and was approved by the SUNY State College of Optometry Institutional Review Board. After the nature of the experiment was explained, written informed consent was obtained from each subject prior to testing.

Subjects were studied with a video-based eyetracker (ISCAN EC-101, ISCAN, Inc., Burlington, MA) while fixating stimuli on a CRT computer monitor (Radius PressView 21SR, Miro Displays, Inc., Germany; 38.0 × 27.8 cm active area) driven by a Macintosh G3 computer. A fixation target was located centrally on the monitor screen, and additional calibration targets were placed ±3 deg from the fixation target along the horizontal and vertical meridians. Subjects sat in an examination chair, with the eye-screen distance set at 75 cm. A head-rest was provided behind the subject’s head, and they were asked to lean back against it. Other stabilizing devices (e.g., chin-rests) were not employed.

Data recorded were: horizontal and vertical position of the corneal reflection, horizontal and vertical position of the pupil center, and pupil diameter. Where appropriate, a digital indicator of stimulus behavior was also recorded. Sampling rate was 60/sec. Data are reported here in raw pixels of the eyetracker video system. At the magnification and 75 cm eye-screen distance used, 1 mm in the frontal plane of the eye was equivalent to 21.0 pixels of the video system. This was determined using “pupil” diameter measurements of black printed disks on a white background, placed at the position of the subject’s eyes.

On calibration trials, 1.5 sec of data were recorded while subjects fixated each target in turn (center, left, top, right, and bottom as seen by the subject). Additional 1.5 sec trials of central target fixation were conducted to examine fluctuation during repeated trials. All data were obtained from right eyes.

On trials studying the effect of light-induced changes in pupil diameter, subjects fixated the central target steadily. A large rectangular stimulus on the display monitor was turned on and off (2 sec on, 2 sec off, 2 sec on, etc) for 16 sec, and data were recorded. Stimulus “on” luminance was 54 cd/m²; “off” luminance was approx 0.001 cd/m². Several cycles of stimulation were provided before recording began, to permit the pupil responses to settle into a steady cyclical behavior. During analysis of the records, it proved useful to partition the data into periods of pupil constriction [ d(Pup diam)/dt < 0 ] and periods of redilation [ d(Pup diam)/dt > 0 ]. Since the time derivative of video-determined pupil diameter is quite noisy, additional smoothing was necessary, and the negative exponential smoothing provided in SigmaPlot (SPSS, Inc.) was employed (polynomial degree = 1, data proportion = 0.015)². In order to obtain approximate confidence interval estimates for pupil position (actually pupil center position minus corneal reflex position) vs. pupil diameter data, the data were taken into IGOR Pro (Wavemetrics, Inc.). Calibration trials and light/dark trials were run at least twice on each subject and average data are generally reported here, except where examples of individual light/dark trials are presented.

RESULTS

Data are reported mainly for five of the subjects recruited. Data from the remaining two subjects showed substantial upper lid intrusion contaminating vertical data. Their horizontal data were similar to data for the remaining five subjects; one case is included in examples of horizontal data presented.

In Fig. 1, examples of raw data are shown from three 1-sec periods of fixating targets 3 deg left of center, center, and 3 deg right of center. From top to bottom, for each fixation period, horizontal pupil center position (pixels) is shown as upright triangles, horizontal corneal reflex position (pixels) is shown as inverted triangles, and the difference between the two (pixels) is shown as circles (“P–CR”). The solid line added to the P-CR data is a spline plot of the result of smoothing the data with a simple 5-bin filter (weights: 0.3152, 0.2438, and 0.0986; corner freq approx 9 Hz).

Fig. 1.

Data from a 3-point calibration of a subject fixating targets 3 deg left of straight-ahead, straight ahead, and 3 deg right. Top row of data: horizontal pupil center; middle row: position of corneal reflex (1^st Purkinje image) of the infrared illumination source; bottom row: “P-CR”, the difference between the pupil center and the corneal reflex in pixels. Inset: plot of P-CR vs. target position. The dashed straight line in the inset is was the best fit: P-CR(pixels) = 11.2 – 1.8*Gaze direction(deg).

The inset plot in Fig. 1 shows the average and standard deviation of P-CR in pixels for the three fixation periods plotted against direction of gaze. (The ±1 SD error bars were smaller than the symbols, and are plotted inside the symbols.) The three average values were fitted well by the linear regression (dashed line; slope = −1.81 pixels/deg, r² = 0.997). The SD of the values of P-CR for these three fixation periods was, on average, 0.38 and 0.20 pixels for raw and smoothed data, respectively. Using the linear regression, average variability of the smoothed P-CR data amounted to about (0.20 pixels)/(1.81 pixels/deg) = 0.11 deg of eye position (about 7 min arc) for the three fixation periods.

The pupil diameter (data not shown) for the same three fixation periods was 86.1 ± 0.7, 86.6 ± 0.9, and 85.5 ± 1.0 pixels or approximately 4.10 ± 0.05 mm, on average, for this subject. If the 5-bin smoothing used above was applied to the pupil diameter data, the average SD was reduced to 0.6 pixels or about 0.03 mm.

Data for the same subject during 16 sec of visual stimulation, while fixating the central fixation target, are presented in Fig. 2. The same smoothing used in Fig. 1 has been applied to these data. The upper graph shows pupil diameter during the 2-sec-on /2-sec-off stimulation. (The square wave at the bottom of the graph indicates stimulus timing.) On average, pupil diameter varied from 56.8 to 80.0 pixels, or from about 2.7 to 3.8 mm (amplitude of constriction = 1.1 mm). The center graph in Fig. 2 shows the horizontal distance between the pupil center and the corneal reflex. It is immediately apparent that this signal, which is routinely scaled and used as a signal of horizontal eye position, covaried with pupil diameter. The extent of variation was, on average, 2.1 pixels, amounting to 1.15 deg of apparent horizontal eye movement for this subject. The bottom graph in Fig. 2 shows pupil “velocity” (rate of change of pupil diameter with respect to time); the curve represents the smoothed data as described in Methods.

Fig. 2.

Data for one light/dark trial described in the text. Stimulus (indicated at bottom of top graph): square-wave presentation of a bright stimulus (4 sec period; 2 sec on, 2 sec off, etc). Top: pupil diameter; middle: horizontal P-CR in pixels–the quantity used to determine eye position; bottom: time derivative of pupil diameter.

Data similar to those of Fig. 2 are presented for another subject in Fig. 3. The same general pattern of results is evident; on average, pupil diameter varied from 76.9 to 119.1 pixels, or about 3.7 to 5.7 mm in this eye (amplitude of constriction = 2.0 mm). The record of horizontal pupil center position relative to corneal reflex varied by approximately 1.4 pixels, amounting to 0.78 deg of apparent horizontal eye movement for this subject.

Fig. 3.

As Fig. 2, for another subject.

Pupil position as a function of pupil diameter

Data for the subject of Fig. 2 are presented in a different manner in Fig. 4: left-hand and right-hand graphs show horizontal and vertical P-CR data, respectively, plotted against pupil diameter. As described in Methods, constriction and redilation segments of the data were separated, and are plotted in the top and middle pair of graphs, respectively. One 16-sec trial contains 4 stimulus-on periods and 4 stimulus-off periods. As may be seen in Figs. 2 and 3, each stimulus-on period contained a relatively brief (approx 0.5 sec) constriction response, followed by some redilation, and each stimulus-off period contains further redilation. Thus, a single pair of stimulus-on and stimulus-off periods contains one relatively brief constriction period and one longer redilation period. The data in Fig. 4 were pooled from two 16-sec trials, and include 8 constriction periods and 8 redilation periods. The data points from each constriction or redilation period (shown as dots) are connected by lines to indicate data points obtained from a single constriction or redilation. The bottom pair of graphs show the average relationships during constriction and redilation (solid and dashed heavy lines, respectively), together with the estimates of the 95% confidence intervals determined with IGOR software (solid and dashed thin lines). For this subject, there was little difference between the horizontal relationships during constriction vs. redilation, while there was a modest difference in the vertical relationships (see below).

Fig. 4.

Separation of P-CR values into periods of pupil constriction and periods of pupil redilation. Left column shows data for horizontal; right for vertical. In the top two rows, individual samples are dots; sequential neighbors are connected by lines. The bottom pair of graphs shows 2^nd order polynomial fits to the data for constriction (solid heavy line) and redilation (dashed heavy line), with approximate 95% confidence intervals indicated by thin solid and dashed lines.

Fig. 5 shows data for four more subjects in the same format as the bottom pair of graphs of Fig. 4. In comparing graphs for different subjects, it should be noted that the absolute values of the ordinate scale (P-CR expressed in eyetracker pixels) may differ substantially from one subject to the next, reflecting differences in pupil position relative to corneal reflex in different eyes.

Fig. 5.

Data for 4 more subjects. Each pair of graphs is derived in the same way as the bottom pair of graphs of Fig. 4.

The data of Figs. 4 and 5 show some features in common: the plots of the horizontal P-CR relationships all had negative slopes. Since all eyes were right eyes, this indicates that larger pupils had centers more temporal than smaller pupils. The plots of the vertical P-CR relationships showed more variability in this regard: 3 of the 5 had positive slopes, 1 had essentially zero slope, and 1 had a negative slope. In addition, a parabolic regression was used to fit the data, and many of the relationships showed significant amounts of curvature.

A significant amount of hysteresis was present in the P-CR relationships; i.e., the average behavior during constriction (solid curves) and redilation (dashed curves) differed. This was especially true for horizontal P-CR, where the constriction relationships tended to be convex downward or straight, while the redilation relationships were convex upwards. This means that for the central portions of each relationship plot, the horizontal P-CR value was typically smaller during constriction than during redilation. For vertical P-CR, there was less hysteresis; it may be seen in Figs. 4 and 5 that the averages during constriction and redilation were quite close. To compare the two directly, each subject’s calibration data were used to convert to degrees of eye rotation. For horizontal P-CR, hysteresis amounted to 0.26 ± 0.13 deg (range: 0.11 to 0.43 deg), and the range of pupil changes in the present experiments gave a range of “pseudo eye movements” of 0.81 ± 0.25 deg (range: 0.53 to 1.22 deg). For vertical P-CR, hysteresis amounted to 0.09 ± 0.05 deg (range: 0.04 to 0.18 deg) and the range of “pseudo eye movements” was 0.54 ± 0.29 deg (range: 0.02 to 0.85 deg).

DISCUSSION

It is clear from the data that movements of the pupil center, relating to changes in pupil diameter, can create signals of apparent eye movement if the pupil center is used as part of the gaze direction estimate. This is not surprising, since the effect had been predicted (Wyatt, 1995); however, the effect does not appear to be routinely acknowledged in the literature.

Because each eye has an idiosyncratic light/dark pupil displacement, the direction and amplitude of the apparent eye movement will also be idiosyncratic to a degree, although the horizontal effect will have the same sign because of the negative slopes for horizontal P-CR plotted against pupil diameter (Figs. 4 and 5). The size of the overall effect for the subjects in the present experiments was on average 0.81 deg (horizontal) and 0.54 deg (vertical), with the largest cases being 1.22 deg horizontal and 0.85 deg vertical. As noted in Results, the negative slope of the horizontal plots implies that larger pupils had centers more temporal than smaller pupils, in agreement with earlier work (Wyatt, 1995).

In addition to the overall apparent eye movements created in this manner, there was some hysteresis present; the relation between apparent eye movement and pupil diameter was somewhat different during pupil constriction than during pupil redilation. The maximum extent of the hysteresis for a given eye in these experiments was on average 0.26 deg (horizontal) and 0.09 deg (vertical), with the largest cases being 0.43 deg horizontal and 0.18 deg vertical. Taken in the context of the general finding that more constricted pupils generally have more nasal centers, the finding that horizontal P-CR for the same pupil diameter was smaller during constriction than during redilation implies that the pupil center lags behind the diameter change during constriction compared to during redilation. (During constriction, the center is nearer the position it has for larger pupils than during redilation.) This has implications for anisotropic aspects of iris structure and function.

To sum up, the overall horizontal apparent eye movements were 60% larger than the vertical, and the horizontal hysteresis effect was three times larger than the vertical. The size of pupil diameter changes in the present experiments (the largest minus the smallest diameter for each subject) ranged from 1.15 to 2.12 mm (ave ± SD = 1.65 ± 0.43 mm). Calculating the apparent eye movement per mm of pupil diameter change, the result is 0.53 ± 0.27 deg/mm (horizontal) and 0.31 ± 0.19 deg/mm (vertical). This result was calculated without considering the curvilinear relationships between P-CR and pupil diameter.

It may be objected that the “spurious” eye movement signals in Figs. 2 and 3 might represent real eye movements, since there was no independent measure of gaze during the trials. It would have been interesting, if challenging, to employ simultaneously a second eyetracker that did not depend on pupil properties, such as a search coil or an eyetracker based only on Purkinje images, but such devices were not available. However, for the eye movements to be real, the subjects would have had to rhythmically change gaze direction in time with their pupil responses, by means of slow (not saccadic) eye movements, and contrary to instructions. This is an extremely unlikely set of circumstances; in fact, making voluntary smooth eye movements without a moving target is beyond most subjects. In addition, the subjects were trained psychophysical subjects who also participated in experiments with attention focused on the fixation target by means of off-blinks of the fixation targets, to which subjects responded with button presses for longer off-blink durations. Substantial eye movements would have have caused errors, but the results for those trials (unpublished) showed that the subjects were very reliable at fixation on trials much longer than the light-dark trials employed in the present work.

Practical implications

For some applications, this effect is not a major issue. In some situations, limited accuracy is required. In other situations, only saccadic eye movements are of interest. (Because of the slow dynamics of the pupil system, the effect on the eye-position signal is also slow. Therefore, saccadic eye movements can be distinguished from both real and apparent slow eye movements.) Clearly, the present results do not indicate potential problems in situations where pupil diameter does not vary significantly. If experiments (or clinical tests) were being performed on older subjects, and illumination was near constant, substantial pupil diameter changes would be unlikely. Thus, for example, during visual field testing of older patients, substantial variations in pupil size would be unlikely. However, the same is not the case for younger subjects in similar circumstances; the author has observed sporadic large changes in pupil diameter during threshold sensitivity testing of younger subjects, presumably due to fluctuations of autonomic innervation to the iris, which could be related to variations in mental state. Pupil diameter variations that are systematically related to accommodative effort would also constitute a potential problem if eye position were being monitored.

The present results suggest that apparent eye movements caused by pupil changes are not likely to be a problem as long as required accuracy of eye position is no smaller than about 1 deg, except in some extreme cases.

Compensating for the pupil effect

As noted, there are situations in which apparent eye movements related to pupil changes do constitute a problem. Given the present results, one approach to dealing with the problem would be to correct the measured eye movements for changes related to pupil diameter. Fig. 6 shows three examples of performing such a correction. Each graph in Fig. 6 is a plot of horizontal P-CR during a light/dark trial. It was assumed for these corrections that pupil position was a univariant function of pupil diameter; i.e., the center was always at the same location for a given diameter, independent of history. (Hysteresis was ignored.) Each of these subjects performed two light/dark trials. The function relating P-CR to pupil diameter was determined for one of the two trials and the resulting function was used to correct the other trial. (The eye was assumed to remain stationary during the light/dark trials, except for the third subject of Fig. 6, where several small saccadic eye movements were apparent; those data segments were omitted in determining the function.) The heavy traces show the raw data; the lighter traces show the corrected data. The standard deviation of the data for the three trials of Fig. 6 was reduced by 55%, 54%, and 27% as a result of the correction. (For all subjects, the reduction ranged from 25% to 55%, with an average of 39%.) The resulting SD’s, expressed in degrees of eye rotation, averaged about 0.18 deg. For assessing the possibility of correcting the eye-position record, the axis showing the greatest variation of P-CR with pupil size was selected. The third trial of Fig. 6 shows that the correction process did not affect the amplitude of small saccadic eye movements that were present; the reason for this is that pupil diameter does not change during small saccadic eye movements since the dynamics of the former are much slower than those of the latter.

Fig. 6.

Three examples of initial attempts at correcting eye-position data for the pupil effect described in the present work. Heavy line shows raw P-CR values from one light/dark trial. Thin line shows the effect of correcting the raw values using the relationship between P-CR and pupil diameter.

In principle, it would be possible to take the hysteresis of Figs. 4 and 5 into account in the correction process. However, initial attempts at this provided no more than slight further improvement in the correction. The minimal success was found to be partly due to a rather complex temporal relationship between P-CR and pupil diameter which presumably depends on iris mechanical properties.

The possibility for further improvement in correction does remain. One piece of evidence supporting this is a positive correlation between corrected P-CR and raw P-CR, indicating that the correction process could go further. The correlation was 0.62 ± 0.21 (SD), was always positive, and did not depend on whether the trial was one selected for deriving the P-CR/pupil diameter relationship, or one corrected using the relationship. A more successful correction might include consideration of the dynamics of the relationship between pupil diameter and pupil position.