Cerebral hemodynamics during scene viewing: Hemispheric lateralization predicts temporal gaze behavior associated with distinct modes of visual processing

Abstract

Systematic patterns of eye movements during scene perception suggest a functional distinction between two viewing modes: an ambient mode (characterized by short fixations and large saccades) thought to reflect dorsal activity involved with spatial analysis, and a focal mode (characterized by long fixations and small saccades) thought to reflect ventral activity involved with object analysis. Little neuroscientific evidence exists supporting this claim. Here, functional transcranial Doppler ultrasound was used to investigate whether these modes show hemispheric specialization. Participants viewed scenes for 20 seconds under instructions to search or memorize. Overall, early viewing was right lateralized, whereas later viewing was left lateralized. This right-to-left shift interacted with viewing task (more pronounced in the memory task). Importantly, changes in lateralization correlated with changes in eye movements. This is the first demonstration of right hemisphere bias for eye movements servicing spatial analysis and left hemisphere bias for eye movements servicing object analysis.

Statement of Public Significance

When viewing a scene for the first time, people usually start by quickly scanning large regions of the scene and then zoom-in on smaller regions for longer periods of time. This behavior of the eyes is thought to distinguish between two underlying processing modes, with the initial scanning thought to reflect a processing mode concerned with spatial information (location of ground, walls, and surfaces) and the slower scanning thought to reflect a processing mode concerned with object identification. However, little neuroscientific evidence exists supporting this hypothesis. The present study recorded the eye movements and brain activity of healthy adults as they viewed scenes of everyday environments. We found that change over time in how the eyes scanned a scene was associated with distinct brain activity. This finding provides the first neural evidence to support the hypothesis that scene viewing initially prioritizes spatial information and shifts to object recognition later.

It has long been noted that when presented with a novel scene there is an initial tendency for observers to quickly scan a large portion of the scene, followed by more intense inspection of smaller regions as viewing time progresses (Buswell, 1935; Antes, 1974). Recent studies of eye movements during scene perception suggest that these two modes or strategies of viewing enable distinct visual processing functions (Follet, Le Meur, & Baccino, 2011; Foulsham, Alan, & Kingstone, 2011; Mills, Hollingworth, Van der Stigchel, Hoffman, & Dodd, 2011; Over, Hooge, Vlaskamp, & Erkelens, 2007; Tatler & Vincent, 2008; Pannasch, Helmert, Roth, Herbold, & Walter, 2008; Pannasch, Schulz, & Velichkovsky, 2011; Unema, Pannasch, Joos, & Velichkovsky, 2005; Velichkovsky, Rothert, Kopf, Dornhoefer, & Joos, 2002; Velichkovsky, Joos, Helmert, & Pannasch, 2005; Wedel, Pieters, & Liechty, 2008). During early viewing (the first few seconds following presentation of a novel scene), sequences of large amplitude saccades separated by short duration fixations enable the encoding of global scene aspects such as overall spatial layout and gist. With longer viewing time, amplitudes decrease and durations increase, enabling the encoding of local scene aspects such as the identity of specific objects. This temporal gaze behavior has been taken to suggest that the two modes of scene viewing, scanning and inspection, are associated with distinct brain regions specialized for processing spatial layout and object identity, respectively (Pannasch et al., 2008, 2011; Unema et al., 2005; Velichkovsky et al., 2002, 2005).

Several lines of evidence suggest that two separate streams of information project from primary visual cortex to other brain regions: a ventral stream, which projects toward temporal areas of the brain and is involved in object analysis, and a dorsal stream, which projects toward parietal areas and is involved with spatial analysis (Milner & Goodale, 1995). On the basis of the eye movement findings described above, it has been suggested that natural viewing behavior can be categorized into two distinct modes that are associated with processing in the dorsal and the ventral visual pathways (Pannasch et al., 2008, 2011; Unema et al., 2005; Velichkovsky et al., 2002, 2005). Ambient or preattentive scanning behavior—evident from large saccades combined with short fixations—would reflect dorsal pathway processing. In contrast, focal or attentive inspection behavior—evident from small saccades combined with long duration fixations—would reflect ventral pathway processing. In support, Velichkovsky et al. (2002, 2005) used a recognition paradigm to demonstrate that focal mode fixations are needed to recognize local, as well as global, features of a situation, particularly in dynamic settings (correct reactions to hazardous traffic events). Currently, however, there is little neuroscientific evidence indicating that these viewing modes are indeed linked to processing in ventral and dorsal visual pathways.

Recently, Marsman, Renken, Haak, and Cornelissen (2013) have used fixation-based event-related functional magnetic resonance imaging (fMRI) to investigate the neural correlates of ambient and focal processing during natural viewing. Although eye movement-related activity was observed in the ventromedial and ventrolateral visual cortices, the results did not support the two visual systems notion that ambient and focal viewing modes modulate dorsal and ventral processing given that ambient viewing behavior is predicted to be located in parietal regions. Instead, these results were taken to reflect that, during ambient viewing, information is processed at a statistical level, where global features are extracted (Marsman et al., 2013). The temporal resolution of fMRI, however, may not be sensitive enough to detect small, rapid fluctuations in brain activation in response to small, rapid eye movement events, especially considering that both brain and eye movement patterns change over a relatively small time frame. Marsman et al. (2013) advances work aimed at resolving these limitations, but their results should be interpreted cautiously. Specifically, well-known time-sensitive lateralization effects in parietal regions (Fink, Halligan, Marshall, Frith, Frackowiak, & Dolan, 1996; Heinze, Hinrichs, Scholz, Burchert, & Mangun, 1998) suggest that the poor temporal resolution of fMRI may mask brain-region-specific eye-movement-related activity. Here, we take a novel approach to investigate the issue of cerebral specialization of temporal gaze behavior. Specifically, we use functional transcranial Doppler ultrasound (fTCD) to investigate hemispheric specialization of scanning and inspection viewing behavior.

fTCD is a neuroimaging technique that uses ultrasound reflections emitted into the cranium to measure cerebral blood flow velocity (CBFV) in basilar cerebral arteries with high temporal resolution. When paired with psychological experiments, fTCD can be used to investigate the brain’s functional organization. During such experiments, the ultrasound probe is positioned at the transtemporal window, allowing access to CBFVs of the left and right anterior, middle, and posterior cerebral arteries. fTCD is useful in psychological studies because CBFV is known to increase with oxygen demand, meaning that fTCD assessments serve as a proxy measurement for cerebral activation (Aaslid, Newell, Stooss, Sorteberg, & Lindegaard, 1991; Truemper & Bashford, 2015). Moreover, the difference between left and right velocities can indicate which hemisphere is more active during the execution of specific cognitive tasks. For instance, previous work has shown that CBFV increases during task performance relative to a resting baseline, the magnitude and hemispheric lateralization of which varies with task demands (Stroobant & Vingerhoets, 2000). The present study used fTCD to investigate neural correlates of temporal gaze behavior. The fundamental question addressed is whether CBFV lateralization traces can serve as the basis for inferences regarding distinct underlying brain areas that are used for different eye movement patterns.

The two hemispheres of the brain are not equivalent, and relative function differences between the two (functional hemispheric asymmetries) have been observed for several cognitive functions. For instance, most individuals show a right-hemisphere dominance for visuospatial processing (Vogel, Bowers, & Vogel, 2003) and a left-hemispheric dominance for production and processing of language (Bethmann, Tempelmann, De Bleser, Scheich, & Brechmann, 2007). A number of explanations for the existence of functional hemispheric asymmetries have been offered, including enhanced ability to perform multiple tasks concurrently (Rogers, Zucca, & Vallortigara, 2004), increased neural capacity due to avoidance of duplicated neural networks (Vallortigara, 2006), and enhanced uni-hemispheric processing speed due to avoidance of interhemispheric transfer via the corpus callosum (Ringo, Doty, Demeter, & Simard, 1994). Thus, each hemisphere seems to process visual stimuli in a different manner, and such functional organization seems desirable. Importantly, anatomical and physiological studies supporting this dissociation have further demonstrated that asymmetrical projections of the ascending visual pathways described above underlie parts of these lateralized visual behaviors (Ocklenburg & Güntürkün, 2012). Thus, there is strong reason to suspect that in addition to well-known dorsal visual processing of spatial layout and ventral visual processing of object identities, the right and left hemispheres may also selectively contribute to processing of space and identity, respectively. In other words, functional hemispheric asymmetry is likely to correlate with temporal gaze behavior.

Evidence to support this hypothesis comes from a wealth of data suggesting that global and local perception is mediated by separate subsystems in the right and left hemispheres, respectively (Jiang & Han, 2005). For instance, studies using compound stimuli have shown that the right hemisphere has a bias for processing the global features of an object, whereas the left hemisphere is biased toward the processing of local stimulus features. Such asymmetries have been observed in response time studies (Hübner, 1998; Martin, 1979), electrophysiological studies (Heinze & Münte, 1993; Malinowski, Hübner, Keil, & Gruber, 2002; Proverbio, Minniti, & Zani, 1998; Volberg & Hübner, 2004), imaging studies (Fink et al., 1996; Heinze et al., 1998), and lesion studies (Delis, Robertson, & Efron, 1986; Lamb, Robertson, & Knight, 1990; Robertson & Lamb, 1991). Thus, global/local processing theories offer another, perhaps complementary account of temporal gaze behavior, according to which the time course of processing differs among levels of stimulus structure such that information at the global level is available earlier or processed faster and thus predominant early during viewing for attentional guidance, whereas information at more local levels is available later or processed slower and thus predominant later in processing for attentional guidance (cf. Navon, 1977).

The present study measured eye movements and cerebral hemodynamics during scene search and memorization tasks to investigate whether periods of scanning behavior associated with visual analysis of overall layout and fine detail demonstrate right and left hemisphere predominance, respectively. The experimental procedure and trial timeline is shown in Figure 1. Based on the above considerations, we expect changes in fixation duration and saccade amplitude over time to correlate with changes in hemispheric lateralization over time. Specifically, eye movement patterns thought to characterize ambient/global visual processing (short fixations, large saccades) should be associated with a right hemisphere bias, whereas eye movement patterns thought to characterize focal/local visual processing (long fixations, small saccades) should be associated with a left hemisphere bias. Thus, as ambient/global visual processing is thought to dominate during early viewing, with focal/local visual processing becoming more dominant later during viewing, we expect to see corresponding changes in hemispheric dominance over time such that a right hemisphere bias during early viewing will transition to a left hemisphere bias later during viewing. In addition, as viewing task is known to influence eye movement behavior (Castelhano, Mack, & Henderson, 2009; Dodd, Van der Stigchel, & Hollingworth, 2009; Mills, Dalmaijer, Van der Stigchel, & Dodd, 2015; Yarbus, 1967), there may be different patterns of lateralization and/or changes in lateralization over time in a search task relative to a memory task. Mills et al. (2011) found that fixation durations were longer and saccade amplitudes smaller in a memory task relative to a search task, suggesting that memory was characterized more by focal/local processing than search (or that search was characterized more by ambient/global processing than memory). Moreover, the specific task has a strong influence on the direction of hemispheric asymmetries. For instance, Studer and Hübner (2008) demonstrated that the hemispheres are differently involved in categorizing objects at the basic or subordinate level. Based on these and other findings (Bethmann et al., 2007; Stroobant & Vingerhoets, 2000; Vogel et al., 2003), we expect the memory task to be more left lateralized than the search task, with this difference becoming more prominent over viewing time.

Figure 1.

Trial timeline and example sequence. Each trial began with a 25 second baseline period. Next, a central fixation point was presented; participants were required to fixate this point and press the spacebar. Immediately following keypress, a written task cue was presented for 2.5 seconds instructing participants to “search” or “memorize”. Finally, the imperative scene was presented for 20 seconds, during which time participants performed the cued task (scenes were presented in full color during the experiment).

Method

Participants

Thirteen undergraduates from the University of Nebraska-Lincoln participated in exchange for course credit (mean age = 21.4 +/− 1.7 years; 8 female; 10 right-handed). All participants had normal or corrected-to-normal vision, were naïve to the purpose of the study, and were informed of their rights of participation according to the University of Nebraska-Lincoln institutional review board.

Stimuli

Scene stimuli were 40 full-color computer-generated real-world scenes used in previous work under similar experimental conditions (Dodd et al., 2009; Mills et al., 2011). Scenes subtended 16.9° × 22.8° of visual angle and depicted common real-world environments such as a living room or a city landscape (none of which contained people).

Eye Movements

Eye-movements were recorded with a desktop mounted SR Research EyeLink 1000 (SR Research Ltd., Mississauga, Ontario, Canada). Chin and forehead rests were used to maintain the participant’s viewing position and distance. Viewing was binocular but only the right eye was recorded. Thresholds for detecting the onset of saccadic movements were accelerations of 8000°/s², velocities of 30°/s, and a minimum amplitude of 0.5°. Movement offset was detected when velocity fell below 30°/s and remained at that level for 10 consecutive samples. Calibration entailed a nine-point accuracy test followed by a nine-point validity test and was repeated if any point was in error by more than 1° or if the average error for all points was greater than 0.5°.

Transcranial Doppler

Cerebral blood flow velocities (CBFV) were recorded from interrogation of the bilateral middle cerebral arteries (MCAs) using transcranial Doppler ultrasound (TCD) (DWL DopplerBox X, Compumedics Germany Gmbh) with a custom-made fixation headset and 2 MHz transducers (Compumedics Germany Gmbh). The depth setting on the TCD was initially set to expected depths for the MCA based on published values (Alexandrov & Neumyer, 2004), and the strongest signal was found by manual adjustment of the depth and transducer position. Accordingly, the MCAs were insonated at depths between 43 and 55 mm, with Doppler gate size between 8 and 10 mm. The sampling frequency of the recorded data was 100 Hz.

Procedure

The experimental procedure and trial timeline is shown in Figure 1. There were a total of 40 trials, each lasting ~47.5 seconds and consisting of four event periods (baseline, fixation, cue, and task periods). Period I was a 25-second baseline period during which a black screen was displayed and participants were instructed to be still and avoid making any eye movements. This period was used to obtain resting baseline measurements of CBFV. Period II was a fixation period during which a central fixation point was presented. Participants were instructed to press the spacebar while maintaining fixation on this point in order to initiate a trial. Thus, the duration of this period was variable. Period III was a 2.5-second cue period during which a word cue was presented (either “search for N or Z” or “memorize the scene”) indicating which task was to be performed on that trial. Period IV was a 20-second task period during which a scene was presented and participants performed the cued task. As our interest was in eye movement and brain responses during each viewing task as opposed to task performance per se, it was important to ensure that participants performed each viewing task for the full duration of the task period. Accordingly, the letter targets in the search task were very small and well camouflaged so that detection would be unlikely, thereby encouraging continued search throughout the duration of the trial. At the end of each search trial, participants were instructed to whisper which target had been found or to whisper a random letter if a target was not found. In the memory task, participants were instructed to memorize the scene in preparation for a memory that would be given at the end of the experiment (the test was not given). Thus, participants did not know which aspects of a scene their memory would be tested on, meaning scene memorization could not be terminated early as a result of having encoded a memorization target. Tasks were randomly ordered from trial-to-trial, and task cues preceded each trial. Scene stimuli were displayed on a Pentium IV PC with a 19-inch VGA monitor (85 Hz) at a viewing distance of 90 cm. Testing took place in a dimly lit, sound attenuated testing room. Experimental sessions lasted 45–60 minutes.

Data Processing

Eye Movement Data

Only saccades commencing during the 20-second task period were included for analysis. Saccades with amplitudes less than 1° were removed to exclude corrective and microsaccades, and only fixations less than 1,500 ms and greater than 90 ms were included to exclude outliers. Saccades with amplitudes greater than 25° were also excluded, as were blinks. After all exclusions (17.4%), 33,410 eye movement events remained for analysis.

TCD Data

All data from TCD were processed using custom-written MATLAB algorithms (R2014b v. 8.4.0, Mathworks, Natick, MA, USA). Two filters were applied in the pre-processing stage: (1) a median filter with a length of 5 samples to remove spurious noise in the envelope waveform, and (2) a lowpass filter (189^th order equiripple finite impulse response filter, 1 dB attenuation at 0.25 Hz and 40 dB attenuation at 1 Hz, filtered data corrected for time lag) applied using the function filter() in MATLAB for the purpose of removing noise and cyclical variation due only to normal cardiac rhythm. Additionally, data above 2.0 times the mean of the envelope waveform or below 0.3 times the mean of the envelope waveform were replaced with the trimmed mean of the envelope waveform (the mean with the highest 2.5% and lowest 2.5% of values removed) to further remove spurious noise in the envelope waveform (Knecht, Henningsen, Deppe, Huber, Ebner, & Ringelstein, 1996). No subject had more than 5% of samples in the left or right envelope replaced (11 of 13 had < 1% replaced).

After pre-processing the recorded waveform from the TCD, by-subject averages were taken across all 40 baseline periods separately for the left and right sides. The percent change from baseline during the cue and task periods was then calculated for each sample within a trial for the left and right sides, according to Equation 1 (Deppe, Ringelstein, & Knecht, 2004; Knecht et al., 1996; Knecht, Deppe, Ebner, Henningsen, Huber, Jokeit, & Ringelstein, 1998; Knecht, Deppe, Ringelstein, Wirtz, Lohmann, Dräger, Huber, & Henningsen, 1998):

$${\text{dV}}_{\text{Left}(\text{Right})}(\text{stp})(\%)=100\%\ast ({\mathrm{V}}_{\text{Left}(\text{Right})}(\text{stp})-{\mathrm{V}}_{\mathrm{b},\text{Left}(\text{Right})(\mathrm{p})})/{\mathrm{V}}_{\mathrm{b},\text{Left}(\text{Right})(\mathrm{p})}$$ (1)

where dV_Left(Right)(stp) is the percent change from baseline for sample s on trial t within person p, V_{b,Left(Right)(p)} is the baseline for person p for the left(right) side, and V_Left(Right)(stp) is the envelope waveform for sample s on trial t within person p for the left (right) side. Finally, lateralization was calculated by subtracting dV(stp)_Right from dV(stp)_Left, as shown in Equation 2 (Deppe et al., 2004; Knecht et al., 1996; Knecht et al., 1998a, 1998b):

$$\mathrm{\Delta }{\mathrm{V}}_{\text{Search}(\text{Memory})}(\text{stp})(\%)={\text{dV}}_{\text{Left}}(\text{stp})(\%)-{\text{dV}}_{\text{Right}}(\text{stp})(\%)$$ (2)

where ΔV_{Search(Memory)}(stp) is the CBFV lateralization of sample s on trial t for person p. These CBFV lateralization traces were used for analysis.

Results

The extent to which three outcomes (fixation duration, saccade amplitude, and CBFV lateralization) exhibited correlated change processes during natural viewing of real-world scenes was examined at the level of individual eye and brain responses, which were nested within 13 participants and within 40 scenes, and in which participants and scenes were fully crossed (given that each participant viewed each scene). As these outcomes are correlated within participants, multivariate growth curve modeling was used to predict all three outcomes for each participant simultaneously, such that the model included separate residual variances and a residual covariance across outcomes from the same participant. Models were estimated within SAS (9.3) PROC MIXED using restricted maximum likelihood estimation in order to accommodate incomplete (e.g., missing) and unbalanced (e.g., individually-varying intervals between occasions of measurement) data and provide unbiased population estimates under the assumption that the data are missing at random.

We first examined the shape of the growth curve for each outcome. As can be seen in Figure 2, which plots the observed means as a function of time (rounded to 1 second bins) for each outcome collapsing across task, the pattern of change over time in each appears to follow a trend that may be approximated by a simple quadratic function. Accordingly, quadratic models for the effect of time were estimated for each outcome.¹ The significance of model parameters was evaluated using Wald tests for fixed effects (time, task, and their interaction) and likelihood ratio tests for random effects. A model with scenes and participants specified as crossed random effects (Baayen, Davidson, & Bates, 2008; Hoffman, 2014) and with by-subject random slopes specified for within-subject effects of time (Barr, Levy, Scheepers, & Tily, 2013) was attempted but failed to converge on a positive definite solution. As such, by-scene random effects were not modeled. Moreover, models including by-subject random slopes for the quadratic effect of time also failed to reach a positive definite solution (even when the random effect of scene was omitted from the model), and so only a by-subject random slope for the linear effect of time was included.

Figure 2.

Observed means for each outcome as a function of viewing time (1 second time bins), collapsing across task. The left panel shows the means for the eye movement outcomes during the task period. Fixation duration and saccade amplitude both displayed quadratic-like change such that their rates of change were steepest during early viewing and leveled-off as viewing progressed. The right panel shows the means for the CBFV lateralization outcome (left side minus right side) during the cue and task periods (only observations during the task period were analyzed). Positive values reflect left lateralization; negative values right lateralization. The reference line for 0 on the y-axis indicates bilateral activation. Early viewing was right lateralized whereas later viewing was left lateralized. As with the eye movement outcomes, this right-to-left shift displayed quadratic-like change over time such that the rate of change was steepest during early viewing and leveled-off with viewing time. Error bars represent +/−1 standard error.

Next, we report the fixed effects for the two eye movement outcomes, followed by the fixed effects for the CBFV lateralization outcome. Finally, the extent to which individual-level change over time is related across eye movement and CBFV lateralization outcomes is then examined via cross-outcome associations among the by-subject random slopes for the linear effects of time. As standardized estimates of association (i.e., correlations) are unavailable when individually-varying time intervals are modeled, correlations were computed based on the estimated variances and covariances (as described below).² It should be mentioned that the present CBFV data have been reported elsewhere (Hage et al., in press). Note, however, that whereas Hage et al. analyzed the CBFV data using least squares estimation methods, here, maximum likelihood methods were used (as described above). The present eye movement data have not been reported elsewhere.

Eye Movements

Figure 3 shows the observed mean change over time (rounded to 1 second bins) in fixation duration and saccade amplitude during the task period for the search and memory tasks; Figure 4 shows the model predicted means as a function of quadratic time. Parameter estimates, standard errors, and p-values for each outcome are shown on Table 1 (model 1a). Significant quadratic effects of time were observed for each outcome (ps < .001) such that fixation duration increased at a decelerating rate, whereas saccade amplitude decreased at a decelerating rate. Significant effects of task on saccade amplitude were also observed. At the start of the task period, amplitudes were larger in the search task than the memory task, evident by a significant effect of task at time 0 (p = .002). Furthermore, the significant linear time by task interaction (p = .013) indicates that the linear effect of time was less negative in the memory task than the search task. In other words, the rate at which amplitudes decreased during early viewing was shallower in the memory task. Thus, at the start of the task period, amplitudes were larger and changed over time more rapidly in the search task than in the memory task. There were no significant effects of task on fixation duration, though, the main effect of task trended toward longer durations in the search task than the memory task (p = .224). Overall, these results are in line with the pattern of temporal gaze behavior observed in previous studies (Buswell, 1935; Antes, 1974; Follet et al., 2011; Foulsham et al., 2011; Marsman et al., 2013; Mills et al., 2011; Over et al., 2007; Pannasch et al., 2008, 2011; Tatler & Vincent, 2008; Unema et al., 2005; Velichkovsky et al., 2002, 2005; Wedel et al., 2008). Importantly, this behavior differed by task, agreeing with the task-dependent nature of eye movement control reported elsewhere (Castelhano et al., 2009; Dodd et al., 2009; Mills et al., 2011, 2015; Yarbus, 1967). Even more important, the effect of task was observed during early viewing, indicating that eye movements were not impervious to higher level factors during this time. We return to this point in the discussion.

Figure 3.

Observed mean fixation duration and saccade amplitude as a function of viewing time (1 second time bins) during the task period for the search (left panel) and memory (right panel) tasks. Both tasks show pronounced temporal gaze behavior: fixation duration increased steeply during the first few seconds and then leveled-off, whereas saccade amplitude decreased before leveling-off. Error bars represent +/−1 standard error.

Figure 4.

Model predicted mean saccade amplitude (left panel) and fixation duration (right panel) as a function of viewing time during the task period for each task. Significant temporal gaze behavior (decelerating reduction in saccade amplitude, and decelerating increase in fixation duration, with viewing time) was observed in both tasks. Importantly, the rate of change in saccade amplitude during the first several seconds of viewing was significantly more rapid in the search task than in the memory task. The tasks did not differ significantly on the time course of change in fixation duration. Note that both outcomes have been standardized (M = 0, SD = 10). Error bars represent +/−1 standard error.

Table 1.

Fixed effect parameter estimates, standard errors, and p-values for the fixation duration, saccade amplitude, and CBFV lateralization (left side minus right side) outcomes, as estimated by three separate models. In model 1a, quadratic models were estimated for the effects of viewing time (centered at 0) on each outcome. Model 1b was the same as model 1a, however, the first second of viewing was discarded. Finally, model 2 estimated a two-part piecewise slope model for the effects of viewing time in which one slope term captured linear change during the first second of viewing (Time01) and another slope term captured quadratic change during the rest of the viewing period (Time119 and Time119²). In each model, task (search or memory) was effect coded, and each outcome was standardized (M = 0, SD = 10). Significant estimates (p < .05) are highlighted in bold.

Model	Fixed Effect Parameters	Fixation Duration			Saccade Amplitude			Lateralization
		Est	SE	p <	Est	SE	p <	Est	SE	p <
1a: Quadratic Time	Intercept	−2.900	.270	.000	2.162	.293	.000	−1.612	.434	.000
	Time	.556	.062	.000	−.438	.065	.000	.318	.046	.000
	Time2	−.020	.003	.000	.016	.003	.000	−.012	.002	.000
	Task	−.656	.540	.224	−1.827	.587	.002	−.034	.869	.969
	Time*Task	−.019	.123	.878	.324	.130	.013	.293	.092	.002
	Time2*Task	.003	.006	.648	−.011	.006	.101	−.008	.004	.036
1b: Omit First Second	Intercept	−1.705	.293	.000	1.521	.310	.000	−1.334	.452	.003
	Time	.367	.069	.000	−.358	.072	.000	.297	.050	.000
	Time2	−.013	.004	.000	.014	.004	.000	−.012	.002	.000
	Task	−.558	.585	.341	−1.934	.621	.002	.606	.904	.503
	Time*Task	−.040	.137	.770	.394	.144	.006	.198	.099	.046
	Time2*Task	.004	.007	.560	−.015	.008	.049	−.005	.004	.248
2: Piecewise Slopes	Intercept	−3.749	.965	.001	2.341	.656	.001	−1.531	.608	.022
	Time01	2.179	.431	.000	−.977	.452	.031	.311	.308	.312
	Time119	.353	.063	.000	−.350	.067	.000	.299	.045	.000
	Time1192	−.013	.003	.000	.014	.004	.000	−.013	.002	.000
	Task	−.996	.743	.180	−1.052	.769	.171	−.731	.867	.399
	Time01*Task	.288	.862	.738	−.632	.903	.484	1.468	.615	.017
	Time119*Task	−.018	.126	.887	.348	.134	.010	.171	.090	.056
	Time1192*Task	.002	.007	.766	−.013	.007	.086	−.003	.005	.547

It is worth noting that the effect of task on fixation durations is inconsistent with previous work, where longer fixation durations were observed during scene memorization than scene search (Castelhano et al., 2009; Mills et al., 2011). It is possible that the somewhat unusual nature of the search task—namely, that all 20 trials were in some sense target-absent trials given that no participant correctly reported finding even a single target—may help to explain this apparent inconsistency as this may have impacted the strategies participants developed over the course of the experiment. It could be questioned, for instance, whether participants were actually searching. In the absence of any viable search targets, the search task may have become something more like free-viewing. Though speculative, there is some support offered by the finding that fixation durations during memory and free-viewing tasks display similar time course profiles (Mills et al., 2011). Of course, it is also possible that fixations were unusually short in the memory task, or a combination of both (i.e., fixations were slightly longer than usual in the search task and slightly shorter than usual in the memory task, resulting in no difference between tasks). For instance, it is possible that the 20 second duration of a trial and 25 second intertrial interval (both of which are much longer than in standard search and memory tasks) were somewhat disruptive and altered how the task was approached. Whatever the basis for the inconsistency in the effect of task, that eye movements nonetheless conformed to the expected pattern of temporal gaze behavior (i.e., increasing fixation durations and decreasing saccade amplitudes over time) despite such methodological deviations from previous studies speaks impressively to the robustness of the behavior.

Cerebral Blood Flow Velocity (CBFV) Lateralization

CBFV lateralization was calculated by computing the bilateral CBFV envelope percent change from baseline (Equation 1) and then subtracting the percent change from baseline in the right hemisphere from that in the left hemisphere (Equation 2). Figure 5 shows the observed mean CBFV lateralization as a function of time (rounded to 1 second bins) during the task period for the search and memory tasks; Figure 6 shows the model predicted means as a function of quadratic time. Parameter estimates, standard errors, and p-values for each outcome are shown on Table 1 (model 1a). Overall, CBFV was right lateralized at time 0, evident by a significant negatively signed coefficient for the intercept (p < .001). The significant linear effect of time at time 0 indicates that CBFV became less right lateralized over time (p < .001), and the significant quadratic effect of time (p < .001) indicates that this right-to-left linear rate of change in CBFV lateralization diminished across time. This pattern interacted with task. At time 0, each task was significantly right lateralized (evident by significant intercepts for each task; ps < .048), with no significant difference between them (p = .969). Importantly, the linear effect of time was significantly more positive for memory than search (p = .002), indicating that the right-to-left change in CBFV lateralization was more rapid in the memory task. In addition, there was a significantly more negative quadratic effect of time for memory than search (p = .036), indicating that the linear rate of right-to-left change in CBFV lateralization diminished more rapidly over time in the memory task. Thus, both tasks were right lateralized during early viewing (first ~1–3 seconds); however, whereas the memory task quickly become strongly left lateralized (beginning at ~4–6 seconds into the task period and peaking at ~12 seconds), the search task slowly became less right lateralized and never showed significant left lateralization (instead showing bilateral activity). It is important to note that a left lateralized finding for memory, for example, should not be interpreted as suggesting that the memorization task exclusively uses the left hemisphere. Rather, it shows that a proportion of the underlying cognitive processes are left dominant.

Figure 5.

Observed mean CBFV lateralization (left side minus right side) as a function of viewing time (1 second time bins) during cue and task periods for each task. Positive values reflect left lateralization; negative values right lateralization. The reference line for 0 on the y-axis indicates bilateral activation. Error bars represent +/−1 standard error.

Figure 6.

Model predicted mean CBFV lateralization (left side minus right side) as a function of viewing time during the task period for each task. Positive values reflect left lateralization; negative values right lateralization. The reference line for 0 on the y-axis indicates bilateral activation. Memory was right lateralized at the start of the task period and became strongly left lateralized over time. Search was right lateralized during early viewing, with bilateral activation during later viewing time. Error bars represent +/−1 standard error.

Correlation between Eye Movements and CBFV Lateralization

To examine the extent to which individual-level variation in change over time is related across eye movement and CBFV lateralization outcomes we examined their pattern of covariation, which can be made more meaningful by converting the cross-outcome by-subject random linear time slope covariances into standardized correlations. Correlations were computed using the estimated variances and covariances, as shown in Equation 3:

$$r_{\mathrm{A},\mathrm{B}}={\mathit{COV}}_{\mathrm{A},\mathrm{B}}/(\text{SQRT}({\mathit{VAR}}_{\mathrm{A}})\ast \text{SQRT}({\mathit{VAR}}_{\mathrm{B}}))$$ (3)

Where r_A,B is the correlation between outcomes A and B, COV_A,B is the covariance between outcomes A and B, VAR_A is the variance in outcome A, and VAR_B is the variance in outcome B. The significance of these cross-outcome correlations was assessed via Wald tests for the model-based covariances, which should be appropriate given that the covariances are not bounded at 0 (the null hypothesis). Table 2 (model 1a) shows the cross-outcome covariances, standard errors, p-values, and standardized correlations among the by-subject random linear time slopes. A positive correlation between by-subject random linear time slopes (which, in contrast to interpretation of the fixed effects, is not conditional on time 0 given that it is the highest order random effect term) would indicate that participants with a larger slope on outcome A than other participants are predicted to have a larger slope on outcome B than other participants, whereas a negative correlation would indicate that participants with a larger slope on outcome A than other participants are predicted to have a smaller slope on outcome B than other participants.

Table 2.

Cross-outcome covariance estimates, standard errors, p-values, and standardized correlations among the by-subject random linear time slopes. Significant estimates (p < .05) are highlighted in bold.

Cross-Outcome Covariance	Model 1a: Quadratic Time				Model 1b: Omit First Sec
	Est	SE	p<	r	Est	SE	p<	r
Duration with Amplitude
Memory	−.035	.013	.007	−.777	−.061	.016	.000	−.986
Search	−.003	.010	.783	−.127	−.011	.011	.318	−.639
Duration with Lateralization
Memory	.045	.019	.018	.268	.051	.024	.032	.251
Search	−.034	.016	.044	−.324	−.022	.018	.224	−.234
Amplitude with Lateralization
Memory	−.028	.014	.047	−.396	−.032	.020	.072	−.319
Search	.001	.015	.964	.010	.002	.018	.906	.030

First, if eye movements shift over time from an ambient or global mode (short fixations, long saccades) to a focal or local one (long fixations, short saccades), then their time slopes should be related such that an increase over time in fixation duration should correlate with a decrease over time in saccade amplitude. In support, for the memory task, there was a significant negative correlation between the by-subject random linear time slopes of the fixation duration and saccade amplitude outcomes (r = −.777, p = .007). That is, participants with a larger (more positive) slope for the fixation duration outcome than other participants are predicted to have a smaller (more negative) slope for the saccade amplitude outcome than other participants, so that a greater increase over time in fixation duration predicted a greater decrease over time in saccade amplitude. The same was true of the search task, though, the correlation was not significant (r = −.127, p = .783).

Second, if ambient-to-focal or global-to-local changes over time in eye movements is related to a right-to-left shift over time in CBFV lateralization, then their time slopes should be related such that an increase over time in fixation duration, or decrease over time in saccade amplitude, should correlate with an increase over time in CBFV lateralization. In support, for the memory task, there was a significant positive correlation between the by-subject random linear time slopes of the fixation duration and CBFV lateralization outcomes (r = .268, p = .018). That is, participants with a larger (more positive) slope for the fixation duration outcome than other participants are predicted to have a larger (more positive) slope for the CBFV lateralization outcome than other participants, so that a greater increase over time in fixation duration predicted a greater increase over time in left-hemisphere CBFV lateralization. There was also a significant negative correlation between the by-subject random linear time slopes of the saccade amplitude and CBFV lateralization outcomes for the memory task (r = −.396, p = .047). That is, participants with a smaller (more negative) slope for the saccade amplitude outcome than other participants are predicted to have a larger (more positive) slope for the CBFV lateralization outcome than other participants, so that a greater decrease over time in saccade amplitude predicted a greater increase over time in left-hemisphere CBFV lateralization.

In contrast, for the search task, there was a significant negative correlation between the by-subject random linear time slopes of the fixation duration and CBFV lateralization outcomes (r = −.324, p = .044). That is, participants with a larger (more positive) slope for the fixation duration outcome than other participants are predicted to have a smaller (less positive) slope for the CBFV lateralization outcome than other participants, so that a greater increase over time in fixation duration predicted a smaller increase over time in left-hemisphere CBFV lateralization. There was no correlation between the by-subject random linear time slopes of the saccade amplitude and CBFV lateralization outcomes for the search task (r = .010, p = .964), indicating that change over time in saccade amplitude was not predictive of change over time in CBFV lateralization.

Finally, a likelihood ratio test comparing the fit of models with and without by-subject time by task slopes indicated that the difference between tasks in the size of these correlations was significant (p < .001), suggesting less hemispheric dominance overall during search than memory. As mentioned above, it is possible that this difference reflects the fact that there may have seemed to be no targets in the search task, leading some or all participants to abandon search. Alternatively, it is also reasonable that a search task recruits more bilateral processes, especially considering that search was difficult and that task difficulty is associated with greater bilateral activity (e.g., Pollman, Zaidel, & von Cramon, 2003; Schnittger, Johannes, Arnavaz, & Münte, 1997).

Alternative Specifications for the Effect of Viewing Time

Inspection of the first second of viewing in Figure 3 suggests that estimating quadratic models for the effect of viewing time may not adequately fit the observed trends and therefore may be prone to some biasing, especially for saccade amplitude. Whatever the bias though, it should affect the slope estimates for each task in roughly the same way given that the functional form of change over time was similar in each. As such, we would not expect the observed task differences to disappear simply by discarding the first second of viewing time, and indeed this was the case: the observed slope differences between tasks hold even when the first second is discarded (see model 1b on Table 1). Therefore, the observed task difference does not appear to be due to fitting quadratic models. In addition, we also analyzed the full viewing period using an alternative model for the effect of viewing time. This model was a two-part piecewise slope model in which one slope term captured change during the first second of viewing and another slope term captured change during the rest of the viewing period. The results of this analysis replicated the quadratic model (see model 2 on Table 1). As such, we are confident that the observed task differences are not an artifact of fitting quadratic models to the observed growth trends.

Discussion

The present study measured eye movements and cerebral blood flow velocities during scene search and memorization tasks to investigate whether periods of scanning behavior associated with visual analysis of overall layout and fine detail demonstrate right and left hemisphere predominance, respectively, as well as whether any such laterality is task-dependent. Consistent with previous reports (Buswell, 1935; Antes, 1974; Follet et al., 2011; Foulsham et al., 2011; Marsman et al., 2013; Mills et al., 2011; Over et al., 2007; Pannasch et al., 2008, 2011; Tatler & Vincent, 2008; Unema et al., 2005; Velichkovsky et al., 2002, 2005; Wedel et al., 2008), we found that fixation duration increased and saccade amplitude decreased with viewing time (Figure 4). One account of this temporal gaze behavior is the two visual systems hypothesis (Pannasch et al., 2008, 2011; Unema et al., 2005; Velichkovsky et al., 2002, 2005), which suggests that this behavior reflects two distinct viewing modes associated with specialized cortical processing: an ambient mode (characterized by short fixations and large amplitudes) thought to reflect activity in the dorsal visual pathway and thus involved with spatial analysis, and a focal mode (characterized by long fixations and small amplitudes) thought to reflect activity in the ventral pathway and thus involved with object analysis. In line with this view, analysis of cerebral blood flow velocities in left and right medial arteries via fTCD indicated greater involvement of the right hemisphere during early viewing (first 3–4 seconds), with greater involvement of the left hemisphere during later viewing (Figure 2). Moreover, this right-to-left change in hemispheric dominance with viewing time correlated with ambient-to-focal changes in fixation duration and saccade amplitude (though, only for the memory task). Thus, eye movement behavior enabling visual analysis of overall scene layout (short fixations, large amplitudes) was linked with a right hemisphere bias, whereas eye movement behavior enabling visual analysis of detailed scene information (long fixations, small amplitudes) was linked with a left hemisphere bias. To our knowledge, these findings are the first evidence that patterns of eye movements long hypothesized to service distinct cortical functions indeed exhibit different cortical activity.

However, the right-to-left temporal pattern of cortical activity described above interacted with viewing task such that the right-to-left shift was more rapid in the memory task than the search task. As a result, search was right lateralized for a longer period of time during early viewing than memory. In fact, whereas memory was strongly left lateralized for much of the task period, search showed no evidence of left lateralization at any point during this period (Figure 6). Instead, search simply became less right lateralized over time, resulting in bilateral activity during later viewing. Furthermore, the correlation between right-to-left change in hemispheric dominance and ambient-to-focal change in eye movements was stronger in the memory than search task. The influence of viewing task on this temporal gaze activity is not entirely predicted from the two visual systems account. Analysis of overall layout is supposed to dominate early viewing, which is reflected in short duration fixations and large amplitude saccades during the first few seconds of viewing (Pannasch et al., 2008, 2011; Unema et al., 2005; Velichkovsky et al., 2002, 2005). The two visual systems account suggests this pattern of eye movements reflects dorsal activity. Processing in the dorsal stream is thought to be cognitively impenetrable, as evidenced by its resistance to visual illusions (Milner & Goodale, 1995). Accordingly, if early gaze behavior reflects dorsal activity, then we would expect behavioral and cortical activity to be equivalent in search and memory tasks during this time. Instead, we observe differences in eye movements (Figure 4) and corresponding cortical activity (Figure 6).

Others have offered similar functional classifications for scanning and inspection viewing patterns, discriminating for instance between global and local (Mills et al., 2011; Tatler & Vincent, 2008; Wedel et al., 2008) or between coarse and fine (Over et al., 2007) viewing behavior. Our finding of functional hemispheric asymmetry in eye movement behavior is also consistent with these theories, which suggest that global structural information (e.g., location of ground, walls, surfaces, and so forth) is prioritized early during viewing (cf. Navon, 1977). In addition, this finding is consistent with numerous demonstrations of a right hemisphere bias in the processing of global layout and a left hemisphere bias in the processing of local detail (Fink et al., 1996; Lamb et al., 1990; Robertson & Lamb, 1991). The key difference between these theories and the two visual system account described above is that the latter presumes that the scanning mode operates independent of attention.

According to Navon’s (1977) global precedence hypothesis, visual processing of an attended scene or object is temporally structured, proceeding from a global analysis of large forms to increasingly finer-grain analyses of component features. Consistent with this proposal, recent work indicates that the default (i.e., in the absence of task demands) temporal processing sequence is indeed global-to-local (Mills & Dodd, 2014). Along these lines, one account of temporal gaze behavior is that the time course of processing differs among levels of stimulus structure such that global information is available earlier or processed faster than local information and is therefore predominant early in processing for attentional guidance, whereas local information requires more effort (i.e., time) to become available or is processed slower than global information and is therefore predominant later in processing for attentional guidance. Evidence for this hypothesis comes from studies of scene and object classification, which suggest that global and local perception is mediated by separate subsystems in the right and left hemispheres, respectively (Jiang & Han, 2005). For instance, studies using compound stimuli have shown that the right hemisphere has a bias for processing the global features of an object, whereas the left hemisphere is biased toward the processing of local stimulus features (Delis et al., 1986; Fink et al., 1996; Heinze et al., 1998; Heinze & Münte, 1993; Hübner, 1998; Lamb et al., 1990; Malinowski et al., 2002; Martin, 1979; Proverbio et al., 1998; Robertson & Lamb, 1991; Volberg & Hübner, 2004). It should be noted, however, that given the limited spatial resolution of fTCD it remains a possibility that attention for spatial layout is mediated by dorsal stream mechanisms.

The reverse relationship between the time course of change in fixation duration and CBFV lateralization in the search task is surprising and contrary to the hypothesis that longer fixations should be related to greater left hemisphere activity. We suggest two possibilities for the unexpected results in the search task. First, it is possible that the unique nature of the search task (target-absent-like trials; 20 second trials; 25 second intertrial intervals) impacted the strategies participants developed over the course of the experiment (e.g., given the 20 second search window, participants may have adapted their saccadic timing strategy in response to the demands of the task to favor a more elaborate scanning strategy, or may have abandoned search in the later seconds of each trial). Second, the difficulty of the search task may have recruited more bilateral processes (e.g., combined verbal and visuospatial activities such as translating scene gist into a semantic code), which seems especially plausible given that task difficulty is associated with greater bilateral activity (e.g., Pollman et al., 2003; Schnittger et al., 1997). These two accounts are not mutually exclusive and both are consistent with the idea that a proliferation of processes contributes to the control of fixation duration during scene viewing such that both direct control mechanisms (decisions about when to terminate fixation are made on-line during the current fixation) and indirect control mechanisms (termination of the current fixation is determined by factors that do not depend on the presence/absence of visual information) exist (Henderson & Pierce, 2008; Henderson & Smith, 2009; Nuthmann & Henderson, 2012).

In conclusion, the present work is the first demonstration of a right hemisphere bias for oculomotor behavior thought to service spatial analysis and a left hemisphere bias for oculomotor behavior thought to service object analysis. Current models of eye movements emphasize bottom-up and top-down control factors. The present findings, among others (Mills et al., 2015; Tatler & Vincent, 2008), suggest that incorporating oculomotor behavioral biases into models of gaze control is likely to improve our understanding of when and where we choose to move our eyes under natural viewing conditions. It is also worth noting that in the present tasks, viewing was not constrained by the nature of the semantic relationship between the target of the task and the content of the scene (congruent or incongruent). Rather, the identities of the target and the scene were orthogonal (search) or unspecified (memory). Thus, the present task instructions did not specify how to carry out the task, which is more akin to many real-world task conditions. The present results suggest that under such conditions the perceptual system is more prepared to process global versus local information. Moreover, such findings in the context of natural viewing indicate that relating brain activity and eye movements is a fruitful way of studying psychological processes without imposing an artificial task structure. This approach is particularly useful to demonstrate how brain dynamics underlying perceptual and cognitive processes unfolds over time in naturalistic conditions. Finally, hemispheric studies are useful for linking brain activity to cognition generally because the two hemispheres and their functions are more alike than different, potentially making it easier to relate brain activity to cognitive processes. For instance, analysis of hemispheric asymmetries and eye movements may reveal the time course of attention or semantic processing, which may be highly informative for the labeling of fixations. Such findings could contribute to the interpretation of scan paths and fixations during real-world activities.