Examining the Influence of Task Set on Eye Movements and Fixations

Abstract

The purpose of the present study was to examine the influence of task set on the spatial and temporal characteristics of eye movements during scene perception. In previous work, when strong control was exerted over the viewing task via specification of a target object (as in visual search), task set biased spatial, rather than temporal, parameters of eye movements. Here, we find that more participant-directed tasks (in which the task establishes general goals of viewing rather than specific objects to fixate) affect not only spatial (e.g., saccade amplitude) but also temporal parameters (e.g. fixation duration). Further, task set influenced the rate of change in fixation duration over the course of viewing but not saccade amplitude, suggesting independent mechanisms for control of these parameters.

1. Introduction

Successful interaction with complex visual environments requires the ability to transition efficiently from one type of visual behavior (e.g., searching for an object in a scene) to another (e.g., evaluating affective dimensions of a scene) depending on situational demands and the goals of the observer. As such, human vision during natural scene perception is often an active process whereby observers selectively seek out information in the visual environment relevant to cognitive or behavioral goals (Aloimonos, Weiss, & Bandyopadhyay, 1987; Findlay & Gilchrist, 2003). Because high-precision visual information is available for only a small portion of the visual field at the center of gaze, with visual acuity and color sensitivity declining into a low-resolution surround, humans must move their eyes from one object or location to the next in order to center the new visual information where resolution is highest. This process of directing fixation through a scene (gaze control) is critical to the support of ongoing cognitive and behavioral activity (Henderson, 2003).

There are at least three important aspects of gaze control during scene perception: 1) where fixations are directed (fixation location), 2) how long individual fixations last (fixation duration), and 3) how spatial and temporal eye movement parameters change over time. Previous studies have shown that these three aspects of gaze control are influenced by both visual factors (stimulus-driven or bottom-up gaze control) and cognitive factors (knowledge-driven or top-down gaze control) (see Henderson & Hollingworth, 1998; Henderson, 2003; Henderson & Ferreira, 2004 for reviews). The present examination is concerned with a specific cognitive factor that influences gaze control during scene perception: task-related control, which refers to goal-directed influence involving either a general ‘gaze-control policy’ or a specific strategy relevant to the current task, and which influences real-time control decisions based on current cognitive needs.

Top-down control of visual behavior involves several cognitive systems (e.g., short-term memory; long-term visual, spatial, and semantic memory), as well as the goals of the observer (i.e., task-related control), which work to selectively direct the eyes to important information in the environment (Henderson & Ferreira, 2004). Early evidence for top-down gaze control during scene perception was provided by Yarbus (1967), who, by manipulating type of viewing instruction, demonstrated qualitative differences in eye movement and fixation patterns depending on the viewing task in which one engaged. For example, when instructed to estimate the ages of family members in a painting, fixations clustered on faces. In contrast, when the same observer viewed the same painting but was instructed to estimate the material wealth of the family, fixations were more widely distributed and tended to land on objects. As the only difference between conditions was the type of viewing instruction provided, Yarbus concluded that eye movement and fixation patterns are influenced not only by visual factors but also by the viewer's task and goals.

Yarbus' work is a classic example of task-related control, as fixation location clearly depended on the strategy associated with viewing instructions. More recent experimental work also supports the conclusion that task set influences fixation location. For instance, studies examining eye movements during complex behaviors such as making tea or a sandwich have shown that observers tend to fixate task-relevant objects rather than the most visually salient objects (Land, Mennie, & Rusted, 1999; Land & Hayhoe, 2001). Similarly, when instructed to count the number of people in a scene, fixations tended to be directed toward scene regions likely to contain people (Torralba, Oliva, Castelhano, & Henderson, 2006). Finally, the distribution of fixation locations has been shown to depend on whether a viewer is searching for an object or trying to memorize a scene (Castelhano, Mack, & Henderson, 2009; Henderson, Weeks, & Hollingworth, 1999).

The studies above clearly demonstrate that task-related control influences fixation location. It is also important, however, to examine the point in time at which this influence tends to occur. Castelhano et al. (2009) instructed participants to either search for a specific object or memorize a specific object. Several global measures of eye movements were found to vary as a function of task, including the spatial distribution and total number of fixations (memory > search), total scan path length (memory > search), average saccade amplitude for the first five saccades (search > memory), and first saccade latency (memory > search). Given that saccade amplitude is related to fixation location, the difference in initial saccade amplitude and first saccade latency between tasks suggests that task-related control over fixation location occurs immediately following scene onset. This conclusion is supported by a number of other studies which have also observed early effects on fixation location, noting that initial saccades tend to take the eyes in the appropriate direction of a search target in a scene, presumably because information about the global gist and spatial layout of the scene that can be generated from the first fixation provides important information about where a particular object is likely to be found (Castelhano & Henderson, 2003; Chen & Zelinsky, 2006; Eckstein, Drescher, & Shimozaki, 2006; Neider & Zelinsky, 2006; Torralba et al., 2006).

Fixation duration is another critical aspect of gaze control. Studies of change detection and object identification have both shown that objects typically require close or direct fixation in order to be identified and for their visual details to be perceived (Henderson & Hollingworth, 1999; Hollingworth et al., 2001a; Hollingworth et al., 2001b). Further, fixation duration is closely linked to both short-term (Henderson & Hollingworth, 1999; Hollingworth & Henderson, 2002; Irwin & Gordon, 1998) and long-term memory encoding (Hollingworth & Henderson, 2002; Loftus, Nelson, & Kallman, 1983; Nelson & Loftus, 1980). The amount of time spent at fixation, therefore, is crucially linked to information processing and can reflect ongoing visual processing during scene viewing (Henderson & Pierce, 2008). Whereas fixation location is rather sensitive to task effects, control over fixation duration during scene perception appears to be relatively insensitive to different task goals.¹ Castelhano et al. (2009) examined the influence of task on fixation duration and did not find a difference in either average fixation duration or initial fixation duration (i.e., first five fixations) between their search and memory tasks. Instead, they found that task influenced first gaze duration (sum of all fixations made in a specified scene region from first entry to first exit), with longer gaze durations falling on objects in the memory task relative to the search task. Thus, although the effect of task did not directly influence individual fixation durations, it did have an indirect effect by influencing molar measures of fixation time such as first gaze duration.

Castelhano et al. attributed this finding to differences in viewing strategy associated with the different goals of each task. In the memory task, object information required more thorough processing to be encoded into memory, and so a strategy was adopted to increase the number of fixations on objects to facilitate this processing. During search, however, participants presumably used information regarding the gist and spatial layout of the scene that was acquired as early as the first fixation (Castelhano & Henderson, 2003; Henderson et al., 1999; Torralba et al., 2006) in order to locate the target. Earlier work by Loftus (1972), who found that memory for scene regions was not related to average fixation duration, but rather to the number of fixations made in the region, supports this conclusion. Thus, consistent with the notion that fixation duration is a relatively stable component of visual behavior (Jacobs, 1986; Motter & Belky, 1998; Phillips & Edelman, 2008), top-down task effects primarily arise in the control of ‘where’ the eyes move as opposed to ‘when’.

A number of scene perception studies have reported that fixation durations change over time. Although some have reported little to no change in fixations as viewing time increases (De Graef et al., 1990), several studies have noted a tendency for fixations to increase during initial viewing periods and to stabilize during later viewing (e.g., Antes, 1974; Friedman & Liebelt, 1981; Pannasch, Helmert, Herbold, Roth, & Walter, 2008; Unema, Pannasch, Joos, & Velichkovsky, 2005). Castelhano et al. (2009), for example, analyzed the duration of the first five fixations following scene onset for their search and memory tasks and found for both tasks that fixation durations increased as viewing time increased and then remained stable during later viewing (after ∼2 s). Likewise, in a preference rating task where participants had to choose which of ten paintings they preferred, Antes (1974) found that fixations during early viewing were shorter (∼215 ms) than those during later viewing (∼310 ms). Saccade amplitude has also been shown to change during viewing, with amplitudes decreasing as viewing time increases, and to stabilize during later viewing (Antes, 1974; Castelhano et al., 2009; Pannasch et al., 2008; Unema et al., 2005). Thus, there appears to be a tendency for short fixations and large saccades to characterize early viewing and for long fixations and small saccades to characterize later viewing. One explanation for this finding relates this tendency to a change in viewing strategy, which entails a quick, initial scan of the scene during early viewing, followed by further scrutiny of local regions during later viewing (Antes, 1974; Castelhano et al., 2009; Scinto, Pillalamarri, & Karsh, 1986). It is noteworthy that Castelhano et al. (2009) did not observe an effect of task on this pattern of scanning, suggesting this trend may be a relatively systematic or general scan strategy employed by the oculomotor system.

Others have similarly observed a tendency for large saccades and short fixations to characterize early scene viewing (∼ first 2 sec) and for small saccades and long fixations to characterize later viewing. According to the two-visual systems hypothesis, the relationship between fixation durations and saccade amplitudes over time is systematic and indicative of two qualitatively distinct modes of scanning: global and local scanning (Frost & Poppel, 1976; Pannasch et al., 2008; Pannasch & Velichkovsky, 2009; Unema et al., 2005; Velichkovsky et al., 2002, 2003, 2005). Accordingly, global processing can be seen as an orienting phase, during which shorter fixations and larger amplitudes dominate, presumably to capture the gist of a scene and establish regions of interest. Local processing, on the other hand, is associated with longer fixations and shorter amplitudes, reflecting a more in depth and object centered analysis. It has been argued that global scanning is restricted to early viewing (first ∼2 seconds following scene onset), after which local scanning dominates (Pannasch et al., 2008; Unema et al., 2005; Velichkovsky et al., 2005). This pattern of change in fixations and saccades has been shown to be relatively stable over a variety of viewing conditions, including differences in cognitive demand (Pannasch et al., 2008; Unema et al., 2005), as well as differences in the affective state of individuals and scenes (Pannasch et al., 2008). Such stability suggests that the interplay between modes of scanning is a fundamental component of gaze behavior elucidated by common mechanisms along a strict time course (Pannasch et al., 2008; Unema et al., 2005). The effect of task on this pattern of scanning, however, has not been examined.

1.1. Overview of present study

In sum, recent task-based research has contrasted visual search tasks, which involved task instructions that explicitly specify the object to which gaze should be directed and thus place strong constraints over object selection during viewing, with memorization tasks, which involved general instructions to memorize the scene and therefore do not place direct constraints on which objects should be fixated. In other words, previous work has compared experimenter-directed tasks (i.e., tasks in which the experimenter explicitly specifies the object to which gaze should be directed, thereby placing strong constraints over object selection during viewing) with participant-directed tasks (i.e., tasks involving viewing instructions that do not state which objects should be selected, thereby leaving it to the participant to translate the general task goals into a series of fixations on relevant objects). The results of this work indicate that task effects emerge immediately and change over time, and generally influence ‘where’ the eyes move (fixation location) as opposed to ‘when’ (fixation duration). This contrast between experimenter-directed search and participant-directed memorization forms the primary quantitative evidence regarding the influence of task on gaze control during scene viewing. Given evidence for a systematic relationship between saccade amplitude and fixation duration, however, which would seem to run counter to the notion that task would affect one parameter (saccade amplitude) but not the other (fixation duration), it is possible that the effects of task may be overestimated by the fact that the object selection is primarily experimenter-directed in these search tasks, but participant-directed in memorization. For instance, when searching for scene-constrained and scene-unconstrained targets, participants were able to detect scene-constrained targets faster and with fewer eye movements (Neider & Zelinsky, 2006). This suggests that differences in eye movements between experimenter-directed and participant-directed tasks in previous work may be due, at least in part, to differences in the utility of contextual information, not simply task differences. Although the contrast between search and memorization provides clear evidence for task-related effects, it does not address the possibility of task-related effects when object selection in both tasks is participant-directed. For example, in the original Yarbus study (1967), task instructions (estimate ages, determine wealth) rarely specified individual target objects (as in search), leaving it to the participant to translate the general task goals into a series of fixations on relevant objects. Although there were clear qualitative differences in fixation patterns, the Yarbus study does not address quantitative effects of task differences on local aspects of eye movement control, such as fixation duration and saccade amplitude, nor the evolution of such effects over the course of a trial.

In the present study, eye movements were compared across four tasks, all of which were participant-directed. First, as in previous studies, a scene memorization task was included that required participants to memorize a scene. Whereas some previous studies have emphasized object memory by informing participants that they will be tested on their memory for specific objects in the scene (e.g., Castelhano et al., 2009; Pannasch et al., 2008; Unema et al., 2005), the present instructions emphasize scene memory. This takes the focus away from instructing participants to fixate individual objects and gives them free reign to fixate anywhere they choose to. Second, a pleasantness-rating was included in which participants were instructed to rate their overall impression of the scene. Critically, neither task specified a particular target object to which gaze should be directed. Third, a visual search task was employed in which participants were to search through a scene for a small ‘N’ or a small ‘Z’. Importantly, the target letter could appear anywhere in the scene, meaning participants were free to determine which objects to select and fixate, as in the memorization and pleasantness tasks, creating a participant-directed search task. Given the target could appear on any object at any location, and that scene knowledge (i.e., of typical object locations) was not predictive of this location, this methodology allows examination of eye movements during visual search that are due to participant-directed selection of fixation locations (as opposed to search tasks that are experimenter-directed, in which the to-be-fixated target object is specified in the instructions, e.g., Castelhano et al., 2009). Finally, a free-view task in which no specific task instruction was provided; participants were free to direct their eyes in any manner they saw fit.

We sought to quantify participant-directed task effects by examining: a) which eye movement parameters are influenced by task set (e.g., fixation duration, saccade amplitude), b) how these parameters change over time, and c) the time course of this control (early versus late onset).

2. Method²

2.1. Participants

Fifty-three undergraduate students from the University of British Columbia and the University of Nebraska-Lincoln received course credit for participation in the present study. All had normal or corrected-to-normal vision and were naive about the purpose of the experiment. Participants were randomly assigned to one of four task conditions (free-view=12, memory=13, pleasantness-rating=14, and search=14). As the same picture set was used for each viewing condition and because looking at the same picture more than once could strongly influence the way one looks at it across viewings, a between-groups design was used to eliminate the possibility of cross-talk between more than one task for a participant.

2.2. Stimuli

Images were 67 computer-generated natural scenes (Figure 1) adapted from a set used by Hollingworth (2007). Scene stimuli subtended 16.9° × 22.8° of visual angle. Each scene depicts a common real-world environment, such as a living room or a city landscape. A small ‘N’ or ‘Z’ was embedded within every scene across all conditions to serve as targets in the visual search condition. Given that the targets were purposely embedded to make detection extremely difficult (so as to encourage search throughout the duration of the trial), it is unlikely that they were detected in the other conditions. To be sure, all participants were asked if they had noticed the targets following completion of the study. The majority of participants in the search condition rarely reported finding even a single target. None of the participants from the other conditions ever reported seeing an ‘N’ or a ‘Z’.

Figure 1.

Typical examples of the computer-generated natural scenes that were used as stimuli in the present study.

2.3. Apparatus

Experimental programs were written using Visual C++. Stimuli were displayed on a Pentium IV computer with a Dell monitor in a testing room equipped with soft lighting and sound attenuation. Eye movements were recorded using an SR Research Ltd. EyeLink II system (Mississauga, Ontario, Canada), which has high spatial resolution and a sampling rate of 500 Hz. Thresholds for detecting the onset of saccadic movements were accelerations of 8000°/s², velocities of 30°/s, and distances of 0.5° of visual angle. Movement offset was detected when velocity fell below 30°/s and remained at the level for 10 consecutive samples. Though viewing was binocular, only the dominant eye was monitored. Participants were seated approximately 44 cm from the monitor and made responses using both eye movements and keyboard strokes.

2.4. Procedure

Participants in the search condition were told that the letter ‘N’ or ‘Z’ was present in every scene and that it was their task to search for it. At the end of each trial, participants were to make a key-press indicating which letter they believed to have been present in the scene (if the letter was not found, which was the case on the vast majority of trials, participants were to guess which letter was present). Participants in the memory condition were asked to memorize each scene in preparation for a recognition test at the end of the session (the test was not actually given). In the pleasantness-rating condition, participants were instructed to evaluate how much they liked each scene on a scale from 1 (do not like the scene at all) to 7 (like the scene very much). At the end of each trial, ratings were entered via key press. Finally, those in the free-view condition were instructed to simply view each scene freely with no additional instruction.

After a description of their task, a nine-point calibration procedure was performed prior to the beginning of the experiment, followed by a nine-point calibration accuracy test. Calibration was repeated if any point was in error by more than 1° or if the average error for all points was greater than 0.5°. At the beginning of each trial, a fixation point appeared in the middle of the screen. To initiate a trial, participants were instructed to fixate the point and press the space bar to initiate the trial. Each scene was presented for 8 sec, during which participants engaged in the task described to them moments earlier. On some trials, a probe appeared just after 6 sec for an unrelated task. As this may influence eye movement patterns post probe presentation, the present analyses were restricted to the first 6 sec of each trial independent of whether a probe appeared on the trial.

2.5. Data analysis

Prior to all analyses, fixation durations less than 90 ms and greater than 2000 ms were discarded as outliers. Fixations and saccades either starting or landing outside of the screen area were also excluded. Finally, to ensure that eye blinks were not mistaken for saccades, fixations immediately preceding or following an eye blink were discarded as well.³ These procedures resulted in the elimination of 7.5% of fixations and 7.3% of saccades across conditions.

To determine whether participant-directed eye movement behavior is affected by task instruction, spatial and temporal eye movement measures were then examined. All analyses were conducted at the level of the 69,369 total individual eye movements, which were nested within 67 trials and within 53 participants, and in which trials and participants were crossed (given that each person saw each image). Multilevel models with participants and trials as crossed random effects (Hoffman & Rovine, 2007) were used to account for the imbalance in the timing of eye movements in examining the time course of search in seconds across viewing conditions. Models were estimated via SAS PROC MIXED using maximum likelihood estimation and Satterthwaite denominator degrees of freedom; the model syntax is available from the authors upon request.

Fixation duration and saccade amplitude served as the dependent variables (outcomes), as these parameters describe ‘when’ (fixation duration) and ‘where’ (fixation location) the eyes move. Per trial there were a mean number of 17.12 (SD = 5.72) fixations and 18.43 (SD = 3.70) saccades. In general, previous work suggests that the average fixation duration is ∼330 ms and the average saccade amplitude during scene perception is between 4° and 5° (Henderson & Hollingworth, 1998; Rayner, 1998). Across all tasks of the present study, the mean fixation duration was 256.19 ms (SD = 135.67) and the mean saccade amplitude was 7.19° (SD = 5.72).

Inspections of the temporal patterns across the duration of the trials suggested a quadratic pattern of change (see left panel of Figures 2 and 3 for observed trends), which was supported by significant overall quadratic effects of viewing time in both outcomes (see right panel of Figures 2 and 3 for predicted trends). In a quadratic model of change over time, the intercept is the predicted outcome whenever time = 0, the linear effect of time is the instantaneous linear rate of change in the outcome per unit time whenever time = 0 (i.e., slope of the tangent line to the curve at that point in time), and twice the quadratic effect of time is how the linear effect of time changes per unit time (i.e., rate of acceleration or deceleration, which is not conditional on time = 0 when it is the highest-order polynomial term in the model). Interactions of viewing condition with each term were then included to examine how the time course of viewing would be modified by task instruction. Viewing time was centered at 1, 2, or 5 sec in separate models in order to evaluate the differences across viewing conditions predicted at those specific viewing times, as reported below. After controlling for the effects of viewing condition, significant remaining sources of random variation included mean differences across trials (e.g., a random intercept for trials), mean differences across persons (e.g., a random intercept for persons), and differences across persons in the linear effect of viewing time (e.g., a random linear time slope for persons). Models also including a random quadratic effect of viewing time across persons did not converge for fixation duration, and so only the intercept and linear effect of time were random across persons for fixation duration, whereas all effects were random across persons for saccade amplitude.

Figure 2.

The left panel shows observed mean fixation duration (in ms) per viewing condition across viewing time (500 ms time bins). These data are for descriptive purposes only and were not used for analysis of task effects. The right panel shows fixation duration (in ms) per viewing condition across viewing time (in sec) predicted from a quadratic model of viewing time across all trials and persons.

Figure 3.

The left panel shows observed mean saccade amplitude (in visual degrees) per viewing condition across viewing time (500 ms time bins). These data are for descriptive purposes only and were not used for analysis of task effects. The right panel shows saccade amplitude per viewing condition across viewing time in seconds predicted from a quadratic model of viewing time across all trials and persons.

Although measures of absolute fit are not available for unbalanced data such as these, effect size was assessed using pseudo-R² statistics (Singer & Willett, 2003) that reflect the change in each of the relevant variance components before and after including the effects of viewing condition. Accordingly, for fixation duration, the main effect of condition accounted for 32% of the random intercept variance, the interaction of condition by linear viewing time accounted for 3% of the random linear slope variance, and the interaction of condition by quadratic viewing time accounted for ∼0% of the residual variance (given that the quadratic slope was not random across persons for fixation duration). For saccade amplitude, the main effect of condition accounted for 57% of the random intercept variance, the interaction of condition by linear viewing time accounted for 6% of the random linear slope variance, and the interaction of condition by quadratic viewing time accounted for 15% of the random quadratic slope variance (given that the quadratic slope was random across persons for saccade amplitude).

3. Results and Discussion

3.1. Fixation duration

The right panel of Figure 2 shows the predicted fixation duration (in ms) for each viewing condition across the length of the trial (in sec) as predicted from a quadratic model. As shown, fixation durations appeared longer in the free-view and memory conditions than in the pleasantness and search conditions. Further, although fixation duration increased across the length of the trial in each condition, the rate of increase decelerated across the length of the trial in each condition except the search condition. Differences across viewing conditions in each of the time parameters (intercept, linear effect, and quadratic effect) are reported in Table 1 and are described below.

Table 1.

Estimated intercepts, linear effects of time, and quadratic effects of time per viewing condition for fixation duration as evaluated at 1, 2, or 5 seconds of viewing time.

Effect	1 Second of Viewing		2 Seconds of Viewing		5 Seconds of viewing
	Est	SE	Est	SE	Est	SE
Intercept:
Free-View	260.35***	7.57	274.03***	7.91	294.07***	10.79
Memory	251.32***	7.28	264.65***	7.60	287.26***	10.37
Pleasantness	223.04***	7.01	239.18***	7.33	266.60***	10.00
Search	222.45***	6.99	231.98***	7.32	259.57***	9.98
Free-View vs Memory	-9.03	10.44	-9.39	10.90	-6.81	14.91
Free-View vs Pleasantness	-37.31***	10.25	-34.85**	10.72	-27.47	14.66
Free-View vs Search	-37.90***	10.23	-42.05***	10.71	-34.50*	14.65
Memory vs Pleasantness	-28.28**	10.04	-25.46**	10.49	-20.66	14.35
Memory vs Search	-28.87**	10.02	-32.66**	10.48	-27.69	14.34
Pleasantness vs Search	-0.59	9.82	-7.20	10.29	-7.03	14.07
Linear Effect of Time:
Free-View	15.43***	2.61	11.93***	2.05	1.43	2.44
Memory	14.77***	2.51	11.88***	1.97	3.20	2.34
Pleasantness	17.90***	2.40	14.39***	1.90	3.88	2.25
Search	9.61***	2.34	9.45***	1.87	8.95***	2.21
Free-View vs Memory	-0.66	3.62	-0.05	2.85	1.77	3.38
Free-View vs Pleasantness	2.46	3.54	2.46	2.79	2.45	3.31
Free-View vs Search	-5.82	3.50	-2.49	2.77	7.52*	3.28
Memory vs Pleasantness	3.12	3.47	2.51	2.74	0.69	3.24
Memory vs Search	-5.16	3.43	-2.43	2.72	5.75	3.21
Pleasantness vs Search	-8.29**	3.35	-4.95	2.66	5.06	3.14
Quadratic Effect of Time:
Free-View	-1.75***	0.44
Memory	-1.45***	0.43
Pleasantness	-1.75***	0.41
Search	-0.08	0.39
Free-View vs Memory	0.30	0.62
Free-View vs Pleasantness	0.00	0.60
Free-View vs Search	1.67**	0.59
Memory vs Pleasantness	-0.30	0.59
Memory vs Search	1.36**	0.58
Pleasantness vs Search	1.67**	0.56

Note: Est = estimate, SE = standard error.

^*p < .05,

^**p < .01,

^***p < .001.

First, with respect to the intercept (i.e., predicted mean fixation duration), there were significant omnibus differences across viewing conditions as evaluated at 1 sec of viewing time, F(3, 53.63) = 7.29, p < .001. Fixation durations for the free-view and memory conditions were equivalent but were significantly higher (all p's < .01) than those for the pleasantness and search conditions, which were also equivalent. The same pattern of intercept differences across conditions was found as evaluated at 2 sec of viewing time, F(3, 53.68) = 7.10, p < .001. However, by 5 sec of viewing time, omnibus intercept differences across conditions were no longer significant, F(3, 52.84) = 2.54, p = .066. Only fixation durations in the free-view condition were still significantly higher than those in the search condition (p = .022), with equivalent fixation durations across the other conditions. Thus, when viewing instructions were participant-directed, task set influenced the duration of individual fixations as early as 1 sec following scene onset. Furthermore, as differences between tasks were less pronounced by 5 sec, this suggests that the influence of task difference decreased as viewing time increased.

Second, with respect to the instantaneous linear effect of viewing time, although there were no omnibus differences across viewing conditions as evaluated at 1 sec, F(3, 229.54) = 2.16, p = .093, the linear effect of time at 1 sec was significantly more positive (p = .014) for the pleasantness condition than the search condition (and the linear effect of time was significantly positive within each condition). This linear slope difference at 1 sec is consistent with the finding above that task set began exerting effects on individual fixations as early as 1 sec. No significant differences across conditions were found, however, for the instantaneous linear effect of time as evaluated at 2 sec, F(3, 89.98) = 1.15, p =.332, in which the linear effect of time was still significantly positive in each condition. By 5 sec of viewing time, although there were still no omnibus differences across viewing conditions, F(3, 176.55) = 2.02, p = .112, the instantaneous linear effect of time at 5 sec was significantly less positive (p = .023) for the free-view condition than the search condition, and was only significantly positive in the search condition. These results indicate that fixation durations increased during the first two seconds of scene viewing for each task. By 5 sec, however, only the search condition had a significant positive slope, suggesting fixation durations had stabilized between 2 and 5 seconds for each of the non-search tasks.

Third, with respect to the quadratic effect of viewing time (which is the same across time as the highest-order polynomial term), there were significant omnibus differences across viewing conditions, F(3, 6,045.94) = 3.96, p = .008. The linear effect of time decelerated significantly less across time (p's < .02) for the search condition than the other conditions, which were all equivalent and in which the quadratic effect of time was significantly negative. In other words, fixation durations for the search condition continued to increase for the entire length of the trial, whereas durations for the other conditions no longer increased after 5 sec of scene viewing. Importantly, however, the interaction of viewing instruction condition with the quadratic effect of viewing time indicates free-view, memory, and pleasantness each decelerated in their rate of increase substantially more so than search. Taken together, when viewing instructions were participant-directed, task set influenced individual fixation durations, as well as the rate of change across the 6 sec viewing period.

3.1. Saccade amplitude

The right panel of Figure 3 shows the predicted trajectories of saccade amplitude (in visual degrees) from a quadratic model of viewing time (in sec) for each viewing condition. As shown, saccade amplitudes appeared smaller in the free-view condition than in the search, memory, and pleasantness conditions. In contrast to fixation duration, saccade amplitude appeared to increase only slightly across the length of the entire trial in each condition, with similar amounts of deceleration across conditions. Differences across viewing conditions in each of the time parameters (intercept, linear effect, and quadratic effect) are reported in Table 2 and are described below.

Table 2.

Estimated intercepts, linear effects of time, and quadratic effects of time per viewing condition for saccade amplitude as evaluated at 1, 2, or 5 seconds of viewing time.

Effect	1 Second of Viewing		2 Seconds of Viewing		5 Seconds of viewing
	Est	SE	Est	SE	Est	SE
Intercept:
Free-View	5.00***	0.33	5.21***	0.34	5.32***	0.30
Memory	7.51***	0.32	7.53***	0.32	7.26***	0.29
Pleasantness	7.55***	0.31	7.73***	0.31	7.18***	0.28
Search	8.59***	0.31	8.70***	0.31	7.92***	0.28
Free-View vs Memory	2.51***	0.46	2.31***	0.46	1.94***	0.42
Free-View vs Pleasantness	2.55***	0.45	2.51***	0.46	1.86***	0.41
Free-View vs Search	3.59***	0.45	3.48***	0.46	2.60***	0.41
Memory vs Pleasantness	0.04	0.44	0.20	0.45	-0.08	0.40
Memory vs Search	1.08*	0.44	1.17**	0.45	0.66	0.40
Pleasantness vs Search	1.04*	0.43	0.97*	0.44	0.74	0.39
Linear Effect of Time:
Free-View	0.26*	0.12	0.17*	0.08	-0.10	0.11
Memory	0.05	0.12	-0.01	0.08	-0.17	0.11
Pleasantness	0.27*	0.11	0.09	0.07	-0.45***	0.10
Search	0.20	0.11	0.02	0.07	-0.53***	0.10
Free-View vs Memory	-0.21	0.17	-0.18	0.11	-0.08	0.15
Free-View vs Pleasantness	0.01	0.17	-0.08	0.11	-0.35*	0.15
Free-View vs Search	-0.06	0.17	-0.15	0.11	-0.44**	0.15
Memory vs Pleasantness	0.22	0.17	0.09	0.10	-0.28	0.15
Memory vs Search	0.15	0.17	0.02	0.10	-0.36*	0.15
Pleasantness vs Search	-0.07	0.16	-0.07	0.10	-0.08	0.14
Quadratic Effect of Time:
Free-View	-0.04	0.03
Memory	-0.03	0.03
Pleasantness	-0.09***	0.02
Search	-0.09***	0.02
Free-View vs Memory	0.02	0.04
Free-View vs Pleasantness	-0.05	0.04
Free-View vs Search	-0.05	0.04
Memory vs Pleasantness	-0.06	0.04
Memory vs Search	-0.06	0.04
Pleasantness vs Search	0.00	0.03

Note: Est = estimate, SE = standard error.

^*p < .05,

^**p < .01,

^***p < .001.

First, with respect to the intercept (i.e., mean predicted saccade amplitude), there were significant omnibus differences across viewing conditions as evaluated at 1 sec of viewing time, F(3, 53.02) = 22.36, p < .001. Saccade amplitudes for the free-view condition were significantly less than those for the memory, pleasantness, and search conditions (all p's < .001). Furthermore, amplitudes for the memory and pleasantness conditions were each significantly less than those for search (all p's < .02). The same pattern was found as evaluated at 2 sec of viewing time, F(3, 52.98) = 20.48, p < .001. When evaluated at 5 sec, however, there were still significant omnibus differences, F(3, 53.23) = 14.52, p < .001—with amplitudes for the free-view condition significantly less than the other conditions (all p's < .001)—but the intercepts for the memory and pleasantness conditions were no longer significantly different from the search condition. These results are consistent with previous work (Castelhano et al., 2009) in which task set had an early effect on saccade amplitudes (here, as early as 1 sec), but less of an effect as viewing time increased, at least in terms of mean differences.

Second, with respect to the instantaneous linear effect of viewing time, there were no omnibus differences across viewing conditions as evaluated at 1 sec, F(3, 52.36) = .71, p = .549, or as evaluated at 2 sec, F(3, 52.92) = 1.09, p = .36. At 1 sec, the linear effect of time was significantly positive for the free-view and pleasantness conditions. At 2 sec, the linear effect of time was significantly positive for the free-view condition only. In other words, amplitudes increased at 1 sec for the free-view and pleasantness conditions, but did not change for the memory or search conditions. At 2 sec, amplitudes increased for the free-view condition, but did not change for the other conditions. When evaluated at 5 sec, the linear effect of time was significantly negative for the search and pleasantness conditions (meaning amplitudes decreased) whereas there was no change in the free-view and memory conditions. Moreover, significant omnibus differences were found, F(3, 49.71) = 4.04, p = .012. The linear effect of time as evaluated at 5 sec was significantly more negative for the pleasantness and search conditions compared with both the free-view condition, and for the search condition than the memory condition (all p's < .02). This indicates that amplitudes decreased significantly more at 5 sec for the pleasantness and search conditions compared with the free-view condition, as well as for search compared with memory. Thus, in contrast to previous work in which saccade amplitudes have been observed to decrease during early viewing, neither the free-view nor memory conditions showed evidence of decrease. Instead, amplitudes for the free-view condition actually increased at 1 sec and at 2 sec, whereas amplitudes for the memory condition did not change across the entire 6 sec viewing period. Similar to the memory condition, the pleasantness and search conditions also did not change at 1 sec or at 2 sec. Though, at 5 sec, amplitudes for these conditions did decrease.

Finally, with respect to the quadratic effect of viewing time (which is the same across time as the highest-order term), although there were no omnibus differences across viewing conditions, F(3, 51.02) = 1.63, p = .195, the quadratic effect of time was significantly negative in the pleasantness and search conditions, meaning amplitudes for these conditions decelerated whereas amplitudes for free-view and memory changed at an equivalent rate over the course of the 6 sec viewing period. As there are no significant differences between tasks, however, these results indicate the quadratic rate of change in saccade amplitude was not affected by task. Taken together, when viewing instructions were participant-directed, task set influenced the size of individual saccades, but not their rate of change over the trial.

4. General Discussion

In the present study, participants completed one of four tasks (memory, pleasantness, search, or free-view) under general viewing instructions which were participant-directed to examine eye movements and fixations as a function of task. Specifically, we were interested in which parameters are influenced by task-related knowledge and how these parameters change over time. There were two main findings. The first was that task set biased not only saccade amplitude but also fixation duration, and the second was that task set biased the rate of change in fixation duration over the course of viewing but not saccade amplitude, which would seem indicative of independent mechanisms influencing control of fixation duration and saccade amplitude.

4.1. Task effects on saccade amplitude and fixation duration

Recent task-based research has contrasted visual search tasks, which explicitly specify the object to which gaze should be directed and thus place strong constraints over object selection during viewing, with memorization tasks, which involve general instructions to memorize the scene and therefore do not place direct constraints on which objects should be fixated. The results of this work indicate that task effects emerge immediately and generally influence ‘where’ the eyes move as opposed to ‘when’ (Castelhano et al., 2009). Consistent with this work, the present study also found an early effect of task set on saccade amplitude, as early as one second into the viewing period. Whereas Castelhano et al. found an effect for the first but not last five saccades made within the scene (10 sec viewing period), the effect here persisted throughout the entire (6 sec) viewing period, though the effect did weaken as viewing time increased. Given the difference in viewing time between studies, it is likely that the amplitude differences found during later viewing were attributable to this (especially considering the trajectory and rate of change for each task indicate convergence at longer viewing times; see Figure 3). It is also important to note that Castelhano et al. compared only memory and search tasks and the differences observed here during later viewing are attributable to the free-view condition, which seems distinct from the other goal-directed tasks. Indeed, consistent with Castelhano et al., there were no differences between any other conditions during later viewing (i.e., at 5 sec).

More important was the effect of task set on fixation duration. In contrast to the Castelhano et al. study where no effect of task was observed across the viewing period or during early viewing (first five fixations), the present study did find that task set biased fixation duration. Here, fixation durations were generally greater for free-view and memory compared with search and pleasantness. Crucially, here the effect was present primarily during early viewing only (i.e., at 1 sec and at 2 sec), with the only difference during later viewing (i.e., at 5 sec) between the free-view condition and the search condition. Thus, direct effects of task set on individual fixation durations were observed here when selection was participant-directed, whereas indirect effects were observed in the Castelhano et al. study when selection was experimenter-directed (i.e., fixation times on objects were mediated by the number of fixations).

There are at least two reasons why the findings reported here differ from those of previous work. The first relates to the use of different task instructions. In the Castelhano et al. study, viewing instructions may have biased eye movements and fixations toward scene regions likely to be relevant to the viewing goal (i.e., experimenter-directed task). For example, Neider and Zelinky (2006) demonstrated scene context as a top-down source of control. When searching for a target constrained by scene context, observers are able to draw on expectations as to where a target is likely to appear in order to guide search behavior (e.g., airplanes tend to appear in the sky), but not when scene context is unconstrained, such as for objects that do not have preexisting semantic associations with specific scene regions. Thus, task effects in the Castelhano et al. study may reflect the effect of task when fixation selection is constrained by properties of the scene. The present viewing instructions, in contrast, emphasized global aspects of the scene, meaning that the present results reflect the effect of task when fixation selection is not constrained by properties of the scene. Second, Castelhano et al. may not have observed direct effects of task set on fixation duration given that only the first five fixations were examined. Here, reliable effects of task set on fixation duration were observed at 1 sec and at 2 sec, meaning that unless differences between tasks are sufficiently large during early viewing, task effects on fixation duration may go undetected if only average fixation duration is examined or if only the first few fixations are examined.

4.2. Evolution of task effects

In terms of the patterns of change in fixation duration and saccade amplitude over the course of scene viewing, two findings stand out. The first is that, contrary to predictions of the two-visual systems hypothesis (which predicts that fixation durations should increase over the first two seconds of scene viewing and stabilize during later viewing while saccade amplitudes should decrease over the first two seconds and stabilize during later viewing), the present results indicate that saccade amplitudes changed very little over the course of the viewing period. This is most apparent for the memory condition, in which no significant change was detected when evaluated at either 1, 2, or 5 seconds. In fact, when change was detected it tended to be in either the opposite direction of what would be predicted (amplitudes actually increased for the free-view and pleasantness conditions when evaluated at 1 sec; amplitudes continued to increase for the free-view condition when evaluated at 2 sec) or long after the proposed stabilization point of two seconds (amplitudes for the search and pleasantness conditions did not show significant evidence of decrease until five seconds into viewing).

On the basis of data indicating that saccade amplitude and fixation duration both reach an asymptotic level after some amount of time into scene viewing (Antes, 1974; Buswell, 1935; Friedman & Liebelt, 1981; Unema et al., 2005), it has been suggested that these parameters are governed by a common mechanism (Pannasch et al., 2008; Unema et al., 2005; Velichkovsky et al., 2005). Assuming this is the case, then it follows that the time course of change in each parameter should be similar, and furthermore, any factor that influences one should also influence the other. Thus, the second finding, which also stands in contrast to the two-visual systems hypothesis, is that the pattern of change in fixation duration was influenced by task whereas the pattern of change in saccade amplitude was not. In other words, the relationship between these two parameters was different across time. Taken together, these results are inconsistent with the notion that fixation duration and saccade amplitude always function together systematically and imply distinct control mechanisms.

These findings should not be taken to suggest that fixation duration and saccade amplitude are not useful diagnostic criteria for indexing global and local processing modes; nor do they imply that these parameters are not closely related. Rather, the present work indicates that other factors, such as the type of task one performs and the degree to which the task is constrained by scene context/knowledge, are necessary to index different modes of processing. Considering the average saccade amplitude during scene perception is between 4° and 5° (Henderson & Hollingworth, 1998; Rayner, 1998), it is noteworthy that the average saccade amplitude for the memory, search, and pleasantness tasks was >7°, whereas it was ∼5° in the free-view task. Given that global processing relies on low spatial frequency information—this being the reason for the expectation of larger amplitude saccades to characterize this mode of processing—then the saccade data presented here suggest that global processing may have dominated the memory, search, and pleasantness tasks, whereas local processing may have been more prominent in the free-view task. The relative dominance of one processing mode over the other would explain why amplitudes changed very little over the course of viewing for each task, as well as account for the rather large disparity in amplitude between the free-view task and the other tasks. Furthermore, as global processing is characterized by short fixation durations, then the dominance of one mode over the other is also partially consistent with the fixation duration data, as the memory, search, and pleasantness tasks tended to have shorter durations than free-view. That the patterns of change in fixation durations and saccade amplitudes over the course of a trial follow very different time courses depending on task suggests that these parameters are independently determined and that additional factors are necessary to infer global/local processing from patterns of eye movements and fixations.

Why would fixation duration and saccade amplitude function systematically in some situations but not others? One factor that may differentially influence durations and amplitudes relates to the utility of visual information for directing when and where the eyes should move. For instance, when information at fixation was degraded, Castelhano and Henderson (2008, Experiment 3) found that saccade amplitudes increased as the amount of degradation increased, whereas fixation durations remained relatively stable across degradation conditions. This was taken to suggest that individuals adopted a strategy of making larger saccades in order to acquire more visual information. If the participant-directed tasks in the present study are seen as reducing the utility of visual information compared with experimenter-directed tasks, given that contextual information is not as useful for task performance, then this explanation (i.e., saccade amplitude depends on visual information) could account for the rather large saccades in the memory, search, and pleasantness conditions: to complete their task without the assistance of scene context (e.g., to guide search), participants made larger saccades in attempt to acquire as much visual information as possible. Within this interpretation, however, one might wonder why this strategy was not adopted in the free-view task. One possibility is that visual information is important for saccade amplitude control only when task-relevant information is present, meaning that because all visual information is potentially relevant while free-viewing there is no need to adopt a spatial strategy to maximize information acquisition. Instead, a better strategy might be to increase the amount of time at fixation. This would explain why fixation durations in the free-view task were larger than the other tasks, as well as account for the relatively stable, average sized saccades across the trial.

Another factor that may differentially influence fixation durations and saccade amplitudes relates to the level of complexity at which the scene is globally represented (e.g., superordinate versus basic versus subordinate levels of category representation). For example, the same scene could be represented as an indoor, manmade scene, as an office space, or as a personalized office space (e.g., Greene & Oliva, 2009). Furthermore, how the scene is represented may depend on the nature of the task. For instance, when searching for a stapler, there is probably little need to represent a scene of an office as anything more complex than ‘office’, as the location of staplers within different office exemplars tends to be consistent (on the desk). However, when searching an office scene for a pair of socks, representing the scene as a personalized office (as opposed to a general category), will likely aid performance. From this perspective, the gradual increase in fixation duration could represent a progression or an adjustment toward a more complex or complete global representation, with asymptotic values reflecting the optimal level of representational complexity necessary to complete the task. Considering the free-view task, for example, it is unlikely that a complex global representation will aid performance and so local processing commenced much earlier than, for example, the search task. Building a more sophisticated global representation might also explain why the increase in fixation duration for the search task followed a linear trend as opposed to a quadratic, like each of the other tasks. Given that visual information was of little use in locating the target in our participant-directed search task, individuals may have adopted a strategy that builds a complex global representation of the scene in order to guide search. The linear trend, therefore, might reflect the possibility that individuals were still building or adjusting this representation for the length of the trial (which makes sense considering search targets were never found). The quadratic trends for memory and pleasantness tasks, in contrast, would reflect a point at which a useable global representation and the subsequent transition to local processing are acquired.

4.3. Conclusion

When goal-directed eye movements are not constrained by information within the immediate environment, but rather are self-directed, the effect of task has direct consequences on the amount of time allocated to individual fixation durations. Furthermore, task-related knowledge influenced the rate of change in fixation duration over the course of viewing but not saccade amplitude, suggesting independent control mechanisms for ‘when’ and ‘where’ to move the eyes. These results contribute to a growing body of literature on fixation duration control and have important implications for models of eye movements. In particular, current computational models typically focus on accounting for fixation location rather than the timing of fixations (e.g., Itti & Koch, 2000; Torralba et al., 2006), which makes sense if individual fixation durations can be treated as a constant. The present work, however, demonstrates that this is not always the case and that models of scene viewing should be extended to account for ‘when’ the eyes move.