Visual Sampling Predicts Hippocampal Activity

Abstract

Eye movements serve to accumulate information from the visual world, contributing to the formation of coherent memory representations that support cognition and behavior. The hippocampus and the oculomotor network are well connected anatomically through an extensive set of polysynaptic pathways. However, the extent to which visual sampling behavior is related to functional responses in the hippocampus during encoding has not been studied directly in human neuroimaging. In the current study, participants engaged in a face processing task while brain responses were recorded with fMRI and eye movements were monitored simultaneously. The number of gaze fixations that a participant made on a given trial was correlated significantly with hippocampal activation such that more fixations were associated with stronger hippocampal activation. Similar results were also found in the fusiform face area, a face-selective perceptual processing region. Notably, the number of fixations was associated with stronger hippocampal activation when the presented faces were novel, but not when the faces were repeated. Increases in fixations during viewing of novel faces also led to larger repetition-related suppression in the hippocampus, indicating that this fixation–hippocampal relationship may reflect the ongoing development of lasting representations. Together, these results provide novel empirical support for the idea that visual exploration and hippocampal binding processes are inherently linked.

SIGNIFICANCE STATEMENT The hippocampal and oculomotor networks have each been studied extensively for their roles in the binding of information and gaze function, respectively. Despite the evidence that individuals with amnesia whose damage includes the hippocampus show alterations in their eye movement patterns and recent findings that the two systems are anatomically connected, it has not been demonstrated whether visual exploration is related to hippocampal activity in neurologically intact adults. In this combined fMRI–eye-tracking study, we show how hippocampal responses scale with the number of gaze fixations made during viewing of novel, but not repeated, faces. These findings provide new evidence suggesting that the hippocampus plays an important role in the binding of information, as sampled by gaze fixations, during visual exploration.

Introduction

Our experience of the visual world is supported by the continuous foveal sampling of different parts of the environment interleaved with saccadic eye movements. Information about the visual environment is thought to accumulate across saccades (Irwin, 1991; Rayner and Pollatsek, 1992; Pertzov et al., 2009), allowing for the encoding of high-resolution visual information that can later be used in the service of cognition and behavior (Yarbus, 1967; Melcher, 2001; Henderson, 2003; Melcher and Morrone, 2007). Indeed, the number of gaze fixations made to a scene or a face predicts the subsequent memory of the viewed image independent of the overall viewing time (Loftus, 1972; Chan et al., 2011; Olsen et al., 2016). Moreover, when fixations are restricted during memory encoding or retrieval, memory is impaired compared with free viewing conditions (Henderson et al., 2005; Johansson et al., 2012; Johansson and Johansson, 2014). These findings suggest that the information accumulated across gaze fixations is used not only in the short-term guidance of where to look next (Caspi et al., 2004; Shen and Paré, 2014), but also contributes to long-term memory representations that guide behavior (Castelhano and Henderson, 2005).

The medial temporal lobe and the hippocampus in particular have long been implicated for their role in the formation of such long-term memory representations (Milner et al., 1998). The hippocampus is thought to index or bind parsed information from neocortical regions to form vivid associative, relational, or episodic memories (Teyler and DiScenna, 1986; Davachi, 2006; Squire et al., 2007; Teyler and Rudy, 2007; Olsen et al., 2012; Eichenbaum and Cohen, 2014; Moscovitch et al., 2016) and is particularly sensitive to novel information processing (Kumaran and Maguire, 2007, 2009, Suzuki et al., 2011a, 2011b; Vannini et al., 2013; Kremers et al., 2014). The hippocampus has been implicated for its role in guiding where to look during memory retrieval (Ryan et al., 2000; Hannula et al., 2007, 2012; Ryals et al., 2015), but no human neuroimaging study has yet examined directly the relationship between hippocampal activity and visual sampling behavior during encoding.

In nonhuman primates, hippocampal and entorhinal neurons are modulated by both gaze fixations and saccades (Ringo et al., 1994; Sobotka and Ringo, 1997; Sobotka et al., 1997; Hoffman et al., 2013). The hippocampus is embedded in a densely connected anatomical network with connections that allow for information flow between it and the visual and oculomotor systems (Shen et al., 2016). Recently, we observed altered patterns of gaze fixations during stimulus viewing for the developmental amnesic, H.C., who presents with a compromised hippocampal system (Olsen et al., 2015, see also Voss et al., 2011). Building on research linking visual sampling with memory formation, these neuropsychological, neuroanatomical, and electrophysiological findings suggest that the hippocampus may play a role in the binding of information accumulated during visual sampling. In this simultaneous fMRI–eye-tracking study, we presented participants with novel and repeated stimuli during a nominal encoding task to test the hypothesis that blood oxygen level-dependent (BOLD) responses in the hippocampus are positively related to visual sampling. Given that our previous work with H.C. showed altered viewing patterns when faces were used as the stimuli, the present study used face stimuli as well. We show that hippocampal responses scale with the extent of visual sampling in each trial and that this relationship only occurs during the presentation of novel stimuli. Furthermore, the number of gaze fixations made during the initial presentation of a face stimulus predicts the extent of hippocampal engagement on subsequent presentations, suggesting that gaze fixations may be related to the development of a lasting representation (Melcher and Morrone, 2007). These results thus provide empirical support for the idea that the hippocampal and oculomotor systems are inherently linked and raise the possibility that eye movements may support hippocampal binding functions.

Materials and Methods

Participants.

Twenty healthy young adults (8 females) with normal or corrected-to-normal vision participated in this study (age: mean = 22.95 years, SD = 2.68; education: mean = 16.74 years, SD = 2.47) in exchange for monetary compensation. All participants were recruited from Toronto community and had no neurological or psychological conditions. The study was approved by the Research Ethic Board at Rotman Research Institute at Baycrest. All participants provided written informed consent.

Stimuli.

One-hundred-twenty color images of nonfamous faces (480 × 480 pixels) were used in this study (see Fig. 1). Half of the images were of female faces and half were of male faces. Face images were taken from a larger stimulus database that has been used in our prior work (Ryan et al., 2007; Heisz and Ryan, 2011). A single scrambled image of a face stimulus was used for the control condition (i.e., during null trials; see Fig. 1). Stimuli were presented using a computer with screen refresh rate of 60 Hz (Dell).

Figure 1.

A, Task procedure. Participants were presented with faces that repeat across the experiment. Novel faces are introduced throughout, as are null event trials (scrambled images). B, Illustration of two stimuli with regions of interest (i.e., eyes, nose, face, mouth, and hair) depicted. Fixations from all participants are presented for the two given faces; the size of the circle represents the viewing duration of the fixations (longer durations = larger circle). C, Distribution of the number of fixations (across all images and participants). D, Proportion of fixations directed to each face part during novel face processing.

Procedure.

Each trial began with a 2 s presentation of a fixation cross “+” against a gray background (see Fig. 1). After the fixation cross, a face (viewing angle = 7.63° × 7.63°) was presented for 4 s. Participants were asked to judge whether each face was older or younger than 35 years of age by pressing 1 for >35 years old or 2 for <35 years old on an MRI-compatible response box. If the image was of a scrambled face (null event trial), participants were instructed to press 3. Participants were instructed to respond as quickly and accurately as possible.

Six blocks were presented, each containing 70 trials. Of the 70 trials, 20 trials presented novel faces (i.e., novel condition), 16 trials presented a face that had been viewed previously once (repetition1 condition), 12 trials presented a face that had been viewed previously twice (repetition2), eight trials presented a face that had been viewed previously three times (repetition3), and 14 trials presented scrambled images (scrambled). For each block (i.e., fMRI run), the 20 novel faces were assigned to 10 “sets” of two faces (one female, one male) and this assignment was counterbalanced across participants. Within each block, the 20 novel and 36 repeated faces were also organized into seven “miniblocks” (each containing eight faces) such that novel faces were introduced throughout each block instead of being clustered at the beginning of the blocks (for more details, see Johnson et al., 2008). The first three miniblocks were used to establish multi-item presentations and allowed for the control of the lag between items. The final four miniblocks contained equal numbers of first, second, third, and fourth presentations. The mean lag between the first and second, second and third, and third and fourth presentation was 8.1, 7.9, and 8.0 face stimuli, respectively. Male and female faces were balanced within each face condition. Each scanning block lasted 7 min and 28.8 s. The sequence of the face versus scrambled trials was optimized using Optseq2 (http://surfer.nmr.mgh.harvard.edu/optseq/) to obtain adequate design efficiency (Dale, 1999). Due to apparatus malfunction, three participants completed five of the six scanning blocks.

Structural and fMRI.

A 3 T Siemens MRI scanner with a standard 32-channel head coil was used to acquire MRI images. Head movements were minimized by inserting soft cushions into the head coil. For the structural MRI scan, T1-weighted high-resolution MRI volumes were obtained using a standard 3D MPRAGE (magnetization-prepared rapid-acquisition gradient echo) pulse sequence (160 slices; FOV = 256 × 256 mm; 192 × 256 matrix; 1 mm isotropic resolution, TE/TR = 2.63/2000 ms, flip angle = 9 degrees, and scan time = 386 s). For the fMRI scan, the BOLD signal was assessed using T2-weighted EPI acquisition procedure with 204 time points (TE = 27 ms, TR = 2200 ms, 3.5 mm slices (with 0.5 mm gap and a bottom-up interleaved order) and a flip angle of 62° (FOV = 225 × 225 mm; 64 × 64 matrix, 2.3 × 2.3 mm in-plane resolution). The images were acquired in an oblique axial orientation parallel to the longitudinal axis of the hippocampus. T1 image acquisition used the same slice orientation.

Stimuli were presented with Experiment Builder (SR Research) back projected to a screen (projector resolution: 1024 × 768) and viewed with a mirror mounted on the head coil. Responses were collected with an MRI-compatible response box.

Eye tracking.

An MRI-compatible eyetracker (Eyelink 1000; SR Research) with a sampling rate of 1000 Hz and spatial resolution of 1° was used to monitor participants' eye movements in the MRI scanner. Calibration was done using the built-in Eyelink 9-point calibration procedure at the beginning of the experiment. Drift correction was performed between trials when necessary to ensure tracking accuracy. Fixations and saccades were categorized using Eyelink's default eye movement event parser. Specifically, a velocity threshold of 30°/s and acceleration threshold of 8000°/s were used to classify saccades (saccade onset threshold = 0.15°). Events not defined as saccades or blinks were classified as fixations.

Eye movement measures.

Eye movement data from Eyelink eyetracker (i.e., in EDF file format) was read into MATLAB (The MathWorks) using freely available toolboxes (edfimport toolbox: https://osf.io/fxumn/ and iTrack toolbox: https://github.com/jashubbard/iTrack) and custom MATLAB scripts. For each trial, the number of gaze fixations was counted during the time window in which face images were on the screen (i.e., 4 s).

fMRI data preprocessing.

SPM8 (Statistical Parametric Mapping, Welcome Trust Center for Neuroimaging, University College London, UK; www.fil.ion.ucl.ac.uk/spm/, version 4661) in the MATLAB environment was used to preprocess the T2-weighted functional images. First, for each participant, anatomical images and several raw functional images selected randomly from each run were checked for quality control and no artifacts were found. Then, slice timing was corrected using sinc interpolation with the midpoint slice as the reference. All functional images were then aligned using a six-parameter linear transformation. Next, for each participant, T2 image movement parameters obtained from the previous step, as well as T2 image global signal intensity, were checked manually using the freely available toolbox ART (http://www.nitrc.org/projects/artifact_detect/) to detect volumes with excessive movement and abrupt signal changes. Volumes indicated as outliers (2.45 volumes per participant, i.e., <0.25%) by ART default criteria were examined visually, confirmed, and later excluded from statistical analyses. Next, anatomical images were coregistered to the aligned functional images, and segmented into white matter, gray matter, and cerebrospinal fluid using SPM8 default tissue probability maps. These segmented images were then used to calculate the transformation parameters mapping from the individuals' native space to the MNI template space. Next, the resulting transformation parameters were used to transform all functional images to the MNI template. For each participant, coregistration and normalization quality were checked by inspecting 12 randomly selected functional images. The functional images were finally resampled at 2 × 2 × 2 mm resolution and smoothed using a Gaussian kernel with an FWHM of 6 mm. The first five fMRI volumes from each run were discarded to allow the magnetization to stabilize to a steady state. Volumes for the last four TRs (with only the fixation cross on the screen) were also truncated, resulting in 195 volumes preprocessed in each run.

fMRI analysis.

SPM8 was used to conduct the first (i.e., individual)-level whole-brain voxelwise parametric modulation analyses (described below) using the number of fixations as the regressor to evaluate the relationship between visual sampling and BOLD response for each participant. As we had specific a priori brain regions of interest, the hippocampus, an ROI analysis approach was used to examine the effects of eye movements (i.e., number of fixations) on hippocampal activation. Then, the mean β estimates within each ROI were calculated and the ROI effects tested using one-sample t tests at the group level across participants.

An extensive literature has shown that the fusiform face area (FFA) plays an important role in the perceptual processing of faces (Kanwisher, 2010) and is a high-level visual processing module that projects to the hippocampus (Felleman and Van Essen, 1991). Therefore, we also included the FFA in our ROI analyses to explore whether gaze fixations also predicted activity in regions devoted to the perceptual processing of faces. This allowed us to determine whether visual exploration modulates differentially the hippocampus versus neocortical regions such as the FFA.

Parametric modulation analysis.

To test whether the number of fixations was associated with hippocampal activation, we conducted a parametric modulation analysis in SPM8 using trial-by-trial number of fixations as a modulator. First, at the individual-level general linear model (GLM) analysis, we entered trial onset times and durations for all trials of the four face conditions (i.e., novel, repetition1, repetition2, and repetition3) and one null event condition (i.e., scrambled) as condition main effect regressors. Then, for each face condition in each run, a parametric modulator was added by entering the trial-by-trial number of fixation scores. Both linear and quadratic modulation was considered. Therefore, for each face condition in each run, there were three regressors: a face main effect regressor, a linear parametric modulation regressor, and a quadratic parametric modulation regressor. No parametric modulator was entered for the null event condition. These regressors were convolved with the SPM8 canonical hemodynamic response function to form the final design matrix. We also included six raw motion parameters obtained from image alignment preprocessing and one additional composite movement parameter by aggregating movement trajectory measures at six brain edge plane locations (approximately at the center of each bounding box surface; for detailed calculation, see art.m lines 432–465 in the ART toolbox) to further covary out movement-related artifacts. The default high-pass filter with a cutoff of 128 s was applied. A first-order autoregressive model AR(1) was used to account for the serial correlation in fMRI time series in the restricted-maximum-likelihood estimation of the GLM.

To examine the effect of the number of fixations during viewing of novel faces on brain activation, we constructed two t-contrasts. The first contrast was the mean linear modulation effect and the second contrast was the mean quadratic modulation effect across all runs in the novel condition. t-contrasts for eye movement modulations effects during repeated face conditions were also constructed. We also constructed a t-contrast to compare the number of fixation modulation effects in the novel condition with the mean modulation effects of all repeated face conditions. Finally, a t-contrast comparing the main effects of the novel versus all repeated conditions were constructed, which was used to localize the FFA.

For the ROI analysis, we first calculated mean contrast estimate values within each ROI from the first level voxelwise analyses and tested at the group level using two-tailed one-sample t tests. All voxelwise contrast estimate images obtained from the first level analyses were also carried into SPM8 group-level analysis (one-sample t tests) with participants as a random factor. This step produced whole-brain voxelwise images so that activation outside the current ROIs could be visualized (Table 1).

Table 1.

Brain regions that showed stronger activation in the trials in which participants made more fixations

Cluster	Anatomical areas	Cluster size	t value	p value	MNI coordinates
					x	y	z
1	Calcarine_R	8035	7.57	0.0000	24	−94	2
2	Frontal_Inf_Tri_R	260	5.57	0.0000	40	34	8
3	Cingulum_Mid_R	51	5.39	0.0000	6	−2	26
4	Fusiform_L	63	5.16	0.0000	−34	−40	−22
5	Hippocampus_L	473	5.02	0.0000	−18	−6	−20
6	Frontal_Mid_Orb_R	136	4.61	0.0001	32	42	−10
7	ParaHippocampal_R	19	4.45	0.0001	16	−6	−22
8	Precentral_R	20	4.41	0.0002	36	−14	68
9	Cerebelum_7b_L	40	4.34	0.0002	−26	−72	−48
10	Vermis_7	35	4.30	0.0002	4	−72	−24
11	Cerebelum_9_R	64	4.25	0.0002	16	−46	−42
12	Temporal_Sup_L	16	4.21	0.0002	−58	−18	4
13	Frontal_Inf_Orb_L	97	4.20	0.0002	−36	38	−16
14	Paracentral_Lobule_L	43	4.19	0.0002	−8	−22	56
15	Fusiform_L	12	4.16	0.0003	−38	−16	−22
16	Hippocampus_R	44	4.08	0.0003	20	−34	−2
17	Frontal_Inf_Oper_R	91	4.03	0.0004	46	8	26
18	Parietal_Inf_R	50	3.97	0.0004	30	−52	48
19	Cerebelum_Crus2_L	81	3.95	0.0004	−4	−82	−36
20	Thalamus_L	11	3.93	0.0005	−8	−26	10
21	Frontal_Inf_Tri_L	61	3.90	0.0005	−40	30	12
22	ParaHippocampal_R	37	3.83	0.0006	22	−14	−18
23	Precentral_R	32	3.60	0.0010	58	−10	46
24	Calcarine_L	24	3.56	0.0010	−6	−96	10
25	Cerebelum_9_L	26	3.51	0.0012	−6	−62	−44
26	Cerebelum_Crus2_R	13	3.40	0.0015	10	−78	−34
27	Occipital_Mid_R	19	3.37	0.0016	30	−64	36
28	Frontal_Inf_Tri_L	14	3.25	0.0021	−46	20	30

All clusters survived the threshold of p < 0.005, with 10 voxel extension, no correction, to facilitate future meta-analysis (Lieberman and Cunningham, 2009). The names of the anatomical regions in the table were obtained using the automated anatomical labeling toolbox for SPM (Tzourio-Mazoyer et al., 2002).

R/L, Right/left hemisphere; Tri, triangularis; Mid, middle; Sup, superior; Inf, inferior; Orb, orbital; Oper, opercular.

Next, we explored whether a greater number of fixations during viewing of novel faces predicted larger reductions in brain activity (i.e., repetition suppression) during the subsequent presentation (i.e., repetition1) of the same faces. To this end, we modified the design matrix of our main analyses by replacing the fixation scores during the second presentation of the face (repetition1) with those of the novel condition. This way, the novel and repetition1 conditions had the same number of fixations modulator. The extra four trials in the novel condition that were not subsequently repeated were excluded from this analysis. Then, we constructed a t-contrast to examine the β value differences for the two parametric modulation effects (i.e., novel modulation minus repetition1 modulation) to reflect the effects of the number of fixations during novel face viewing on repetition suppression effects in brain activation. The logic of this analysis can be explained as follows. First, the GLM for the novel face and repetition1 condition can be written as Y₁ = a₁ + β₁ * ParaM₁ + ε₁ (Eq. 1) and Y₂ = a₂ + β₂ * ParaM₁ + ε₂ (Eq. 2), respectively, where Y denotes brain activation data after controlling for effects of each face condition and all regressors of no interest, ParaM₁ denotes the parametric modulator (i.e., the number of fixations to novel faces), and a and ε denote the intercepts and residuals for these GLMs. In this study, we estimated β₁ and β₂ simultaneously using one GLM design matrix as specified earlier and obtained the contrast β₁ − β_2. If we subtract Equation 2 from Equation 1, we obtain Y₁ − Y₂ = a₁ − a₂ + (β₁ − β₂) * ParaM₁ + ε₁ − ε₂ (Eq. 3). Using a new intercept a to replace a₁ − a₂, a new residual term ε to replace ε₁ − ε₂, and a new parameter β to replace β₁ − β₂, we obtain Y₁ − Y₂ = a + β * ParaM₁ + ε (Eq. 4), which indicates that the parameter estimate β, which is equivalent to β₁ − β₂ in our analysis, is exactly the regression coefficient that tests whether fixations to novel faces predicted decreases in brain activation from the first to the second face condition. Finally, at the second-level analysis, we examined whether this effect was larger than zero with a one-sample t test, using the identical procedure as mentioned in other analyses.

Although nonlinear modulation effects were considered in these analyses, significant effects were only observed with the linear regressor. Therefore, only the results from the linear regressor are presented hereafter.

Control analyses.

Control analyses were conducted to exclude alternative interpretations for the results obtained from our main analyses mentioned above. First, a greater number of fixations may simply indicate that participants spent more time actively processing the faces and/or found the age judgment more difficult. To exclude the possibility that eye movement modulation effects were confounded with time spent processing the faces, we controlled for the effect of trial-by-trial reaction time under the assumption that reaction times reflect the length of time dedicated to active face processing. Specifically, using the identical parametric modulation analysis procedure as in our main (i.e., the first) analyses, we added trial-by-trial reaction times as a parametric modulator, in addition to the number of fixations, for the novel condition. Only linear modulation effects were considered for both eye movement and reaction time modulators in this condition. The design matrix construction for all other conditions (of no interest) remained the same. For the first-level analysis, the regressor orthogonalization option in SPM8 (spm_get_ons.m and spm_fMRI_design.m) was turned off to ensure that the two parametric modulators were treated with the same status (Mumford et al., 2015). This way, we obtained unique effects of the number of fixations controlling for the potential confounding effect of reaction time. Except for these modifications, all other analysis procedures were identical to the original parametric modulation analysis. In the second control analysis, we entered the total duration of fixations as a covariate in a similar control analysis. Specifically, we added together durations of all fixations in each trial to make a total duration regressor. Because the first fixation in each trial may not start exactly with (e.g., sometimes earlier than) the stimulus onset and the last fixation may not end exactly with (e.g., sometimes later than) the offset of the stimulus, we used the stimulus onset and offset as the starting and ending time point for calculating the duration of the first and last fixations. This control analysis was included to investigate whether effects of the number of fixations were confounded by total time spent viewing the face itself.

Next, we probed the possibility that the viewing of certain face features (e.g., related to face sex or physical appearance) and/or the size of the face may drive the relationship between the number of fixations and hippocampal activation. To this end, face sex and face size (i.e., the total pixel area within the face and hair boundary) were entered as covariates in two separate parametric modulation analyses with identical procedures as described in the previous control analyses.

ROI definition.

The bilateral anatomical hippocampal ROIs in individuals' native space were first obtained using FreeSurfer recon-all function, version 5.3 (http://surfer.nmr.mgh.harvard.edu.myaccess.library.utoronto.ca) (Fischl, 2012). Then, the same normalization parameters obtained from SPM normalization procedure were used to transform these hippocampus masks to MNI normalized space.

For the functional FFA ROIs, we contrasted novel versus all repeated face conditions. Then at the group level, we localized the local maximum activation in the bilateral fusiform gyrus at the threshold of p = 0.05 (with 10-voxel extension and family wise error multiple-comparisons correction). To make the final functional ROI masks, a spherical volume with 8 mm radius was placed around the maximum activation voxel (left FFA: [−34 −46 −20], right FFA: [36 −38 −20]; see Fig. 2A). We note that the pattern of the results remained the same when using FFA ROIs localized at the individual level using face versus scrambled picture contrast (mean coordinates: [−38.3 −49.6 −19.9] for the left FFA and [40.0 −47.0 −20.4] for the right FFA).

Figure 2.

The number of gaze fixations positively predicts activation in the hippocampus (HPC) and FFA bilaterally. A, Individual anatomical HPC ROIs (in blue) are shown for a representative participant in MNI space. Group-level functional FFA ROIs are illustrated in green and activation clusters are presented in red for viewing of novel versus repeated faces (familywise error = 0.05). B, Results for the anatomical HPC ROIs and functional FFA ROIs (two-tailed t test). For illustration purposes, brain section (C) and surface (D) views are also presented at p = 0.005, 10-voxel extension with no corrections. For the brain surface views, L indicates left hemisphere and R the right hemisphere.

Statistical thresholding.

The threshold for statistical significance was set at p < 0.05 for the ROI analyses that were used to test our main hypotheses (i.e., the one-sample t tests). Results from whole-brain voxelwise analyses were thresholded at p = 0.005 (uncorrected) with 10 voxel extension to facilitate future meta-analysis (Lieberman and Cunningham, 2009). The automated anatomical labeling toolbox (Tzourio-Mazoyer et al., 2002) was used to identify the anatomical labels for all activated regions in each analysis.

Results

Gaze fixations

We calculated the mean and SD for the number of fixations for each condition in each run, which were then averaged across runs and participants. On average, participants made 9.96 fixations per trial during viewing of novel faces (within condition SD = 2.47). The distribution of the number of fixations aggregated from all trials and all participants is presented in Figure 1C. For the repetition1, repetition2, and repetition3 conditions, on average, participants made 9.36 (SD = 2.47), 8.99 (SD = 2.58), and 9.07 (SD = 2.42) fixations, respectively. Mean fixation duration was 368.79 ms (within-condition SD = 106.96 ms), 384.25 ms (SD = 122.73 ms), 385.64 ms (SD = 129.58 ms), and 387.94 ms (SD = 129.18 ms) for the novel, repetition1, repetition2, and repetition3 conditions (durations > 2000 ms excluded). Faces were divided into five regions: eyes, nose, mouth, face (excluding the eyes, nose, mouth regions), and hair. As can be seen from Figure 1, B and D, the eyes attracted the largest number of fixations during viewing of novel faces, consistent with previous reports (Bindemann et al., 2009; Heisz and Ryan, 2011; Riggs et al., 2014; Bortolon et al., 2016). Similar fixation distributions across different face features were found in the repetition1, repetition2, and repetition3 conditions (eyes: 46%, 46%, and 46%; nose: 19%, 18%, and 17%; face: 23%, 24%, and 24%; mouth: 6%, 5%, and 5%; hair: 6%, 6%, and 7% for the 3 repeated face condition, respectively).

Variations in the number of fixations made to different faces were unlikely to be driven purely by the bottom-up properties of the face stimuli because the number of fixations specific to a given image was not correlated across participants (e.g., r < 0.03 for the novel face processing condition). Because <1% of the variance in the number of fixations made across face images can be accounted for by between-subject similarity, this suggests that the trial-by-trial variations in the number of fixations is largely driven by processes idiosyncratic to each participant (Peterson and Eckstein, 2013).

fMRI results

Number of gaze fixations predicted activation in the hippocampus and FFA during viewing of novel faces

Linear parametric modulation effects of the trial-by-trial number of fixations on brain activations during viewing of novel faces were examined to determine whether the number of fixations predicted activity in the hippocampus and FFA. As shown in Figure 2B, the ROI analysis showed that the bilateral hippocampus was more strongly activated for the novel trials in which participants made more fixations (t = 4.759 and 2.620, p = 0.0001 and 0.017 for the left and right hippocampus, respectively), with the effect being stronger in the left than the right hippocampus (t = 2.278, p = 0.035). Bilateral FFA also showed stronger activation for the trials in which participants made more fixations (t = 3.478 and 3.023, p = 0.0025 and 0.007 for the left and right FFA, respectively; Fig. 2B), with no differences between the left and right FFA (p = 0.23). Brain section and surface views, thresholded at p = 0.005 uncorrected, with 10 voxel extent threshold, for illustration purposes, are presented in Figure 2, C and D. Although specific hypotheses were not made for other brain regions, the whole-brain results showed that the number of fixations also positively modulated early visual regions. To facilitate future meta-analysis (Lieberman and Cunningham, 2009), whole-brain clusters that survived the threshold of p = 0.005, 10-voxel extension (without multiple-comparison corrections), can be found in Table 1.

The same pattern of results was obtained after fixations with durations larger than 2.5 SDs of the mean within each run were excluded (t = 4.962, 3.089, 3.360, and 2.90, p = 0.0001, 0.006, 0.003, and 0.009 for the left and right hippocampus and left and right FFA, respectively). Moreover, excluding trials in which <2 fixations were recorded within the face image yielded the same pattern of results (t = 4.742, 2.753, 3.558, and 3.111, p = 0.0001, 0.013, 0.002, and 0.006 for the left and right hippocampus and left and right FFA, respectively), indicating that the current findings were not driven by outlier trials in visual sampling behavior. Examining individuals' results with and without smoothing confirmed that the effects on hippocampus and FFA were not due to smoothing from adjacent brain regions (without smoothing, t = 4.335, 2.166, 3.40, and 2.592, p = 0.0004, 0.043, 0.003, and 0.018 for the left and right hippocampus and left and right FFA, respectively). Moreover, in a separate analysis, we added a linear fixation modulator to the scrambled trials and confirmed that the number of fixations did not predict hippocampal and FFA responses in this condition (t = −1.705, −0.183, 0.131, and −0.208, p = 0.105, 0.857, 0.897, and 0.837 for the left and right hippocampus and the left and right FFA, respectively). The modulation effects of the number of fixations during novel face viewing were significantly larger than those during the scrambled trials for the left hippocampus and bilateral FFA (t = 3.920, 2.061, 2.522, and 2.574, p = 0.0009, 0.053, 0.021, and 0.019 for the left and right hippocampus and the left and right FFA, respectively). Together, these findings demonstrate that increased visual exploration during viewing of novel faces is associated with increased activity in the hippocampus and the FFA.

Number of fixations did not predict hippocampal activation during viewing of repeated faces

The linear modulation effects of the number of fixations on hippocampus and FFA activation was examined during each repeated face condition (repetition1, repetition2, and repetition3) in the same manner as used for the analyses for the viewing of novel faces. As shown in Figure 3, the number of fixations was not significantly associated with hippocampal activation during viewing of faces that had been repeated two, three or four times (p = ∼0.25–0.98). However, for the FFA, the number of fixations still had significant positive modulation effects during viewing of repeated faces (p = ∼0.0252–0.0378; Fig. 3A–C). Comparing the novel versus the repeated face conditions directly showed that the effect of the number of fixations on hippocampal activation was significantly stronger during novel than repeated face viewing (t = 3.404/2.917, p = 0.003/0.0088, for the left and right side, respectively; Fig. 3D). However, the fixation modulation effect on activation of the FFA was not significantly different during viewing of novel versus repeated faces (t = ∼0.39–1.13, p = ∼0.7–0.27; Fig. 3D). Combining the ROIs from the right and left hemisphere, changes in the fixation modulation effect from novel to repeated face viewing were significantly larger in the hippocampus than FFA (t = 2.16, p = 0.044), indicating that visual exploration had a different modulatory effect across repetitions for the hippocampus versus the FFA.

Figure 3.

Modulation effects of the number of gaze fixations on activation in the hippocampus (HPC) and FFA during viewing of the faces that have been repeated. From top to bottom: faces were shown twice (A; repetition 1), three times (B; repetition 2), and four times (C; repetition 3). D, Differences in the modulation effects of the number of fixations on neural activation during viewing of novel versus all repeated face conditions. ***p < 0.005; **p < 0.01; *p < 0.05.

We note that the lack of association between the number of fixations and the hippocampal responses in the repeated face conditions is not due to a lower mean level fixation in the repeated compared with the novel, face-processing condition. This is because condition regressors were entered in the GLM analysis to account for condition mean effects. Therefore, the effects of the fixation modulator were above and beyond the condition main effects. Moreover, the fixation parametric modulation regressors were all mean centered in SPM. Therefore, the parametric modulation effects of numbers of fixations were only determined by how the variability across trials in fixation data was related to the variability of the brain activation data. As reported earlier, the variability of the number of fixations was equivalent in the repeated, compared with the novel, face-viewing conditions. We also estimated single-trial brain activation β values in a separate GLM analysis; i.e., a β series analysis (Rissman et al., 2004), and calculated the across-trial β variability for the four face-viewing conditions. The mean SDs of β estimates for the left/right hippocampus were as follows: 0.961/1.079, 1.014/1.195, 0.996/1.142, and 1.007/1.136 for the novel, repetition1, repetition2, and repetition3 conditions, respectively. Therefore, the variability of the hippocampal responses was not diminished during the repeated face-viewing conditions. Together, these data confirm that the lack of association between the number of fixations and the hippocampal responses during repeated face viewing was not due to reductions in either the mean or variability in the fixation or fMRI data.

Number of fixations predicted neural activity after controlling for behavior and stimulus features

To corroborate the findings from the main analysis, we conducted control analyses to exclude potential confounding effects related to the behavior of individual participants (i.e., reaction time, total face viewing time) and stimulus features (i.e., face sex and size). The number of fixations was not strongly correlated with these potential confounding variables (e.g., r = 0.077, −0.328, 0.096, and 0.10) for the mean correlation of fixation (across runs and participants) with reaction time, total face viewing time, face sex, and face size, respectively. However, because the correlations can be higher than the mean correlation in some runs, control analyses were conducted to test whether the fixation modulation effects were robust. We added each potential confounding variable in separate GLM analyses as another linear parametric modulator to obtain the unique effects of the original modulator (i.e., number of fixations) above and beyond effects of the potential confounding variable. These analyses confirmed that the number of fixations still positively predicted activation in the hippocampus and FFA (p < 0.05; see Fig. 4 for detailed statistics).

Figure 4.

The number of gaze fixations made during viewing of the faces positively predicts activation in the hippocampus (HPC) and FFA, bilaterally, after controlling for reaction time (A), total fixation duration (B), sex of faces (C), and size of face images (D). Results are plotted using anatomical HPC ROIs and functional FFA ROIs (two-tailed t test).

Together with the above findings, these results suggest that there is an intrinsic relationship between the number of fixations indicative of visual exploration neural activity in the hippocampus and FFA that is not merely due to the bottom-up aspects of the stimuli or other aspects of individual behavior.

More fixations during viewing of novel faces predicted larger subsequent repetition suppression effects in the hippocampus

Previous studies have shown that encoding of repeated, compared with novel, stimuli leads to less activation of the hippocampus and that this repetition suppression is associated with successful memory formation (Miller et al., 1991; Brown and Aggleton, 2001; Kumaran and Maguire, 2007, 2009, Suzuki et al., 2011a, 2011b; Vannini et al., 2013; Kremers et al., 2014). We thus explored whether higher numbers of fixations during viewing of novel faces linearly predicted larger reductions in brain activity during subsequent presentations of the same faces (i.e., in the repetition1 condition). As shown in Figure 5, the number of fixations made during viewing of the novel faces positively predicted repetition-related suppression in the hippocampus (t = 2.944/2.126, p = 0.008/0.047), but not in the FFA (t = 0.595/0.389, p = 0.559/0.702). Comparing the effects in the hippocampus and FFA directly after combining the ROIs from the right and left hemisphere yielded significant differences (t = 2.143, p = 0.045).

Figure 5.

Trials with a higher number of gaze fixations during viewing of the first presentation of the face led to larger activation suppression (i.e., β value) in the left and right hippocampus (HPC) during the second viewing of the face. The effect was not significant for the bilateral FFA. After combining the left and right ROIs, the effect was significantly larger for the HPC than FFA. **p < 0.01; *p < 0.05.

In addition, we tested the correlation between the number of fixations during novel and repeated face viewing (i.e., repetition1 condition). Specifically, we first calculated the correlation between the two fixation count variables in each run for each participant. We then averaged the Fisher's z-transformed correlations across runs to obtain an averaged correlation for each participant. The data showed that the mean correlation across participants was small but positive, r = 0.16 (one-sample t test on the Fisher's z-transformed correlation values: t = 4.696, p < 0.001), indicating that, on trials in which a relatively high number of fixations were made to novel faces, there was a tendency for a relatively high number of fixations to be made the second time that face is shown. Parallel to the neuroimaging finding mentioned above, we also found that more fixations in the novel face condition were associated with larger reduction in fixations from the first to the second face exposure, r = 0.65 (one-sample t test across participants: t = 23.89, p < 0.0001). Together, these results support the idea that more fixations during the novel face viewing may facilitate the formation of lasting memory representations (Althoff and Cohen, 1999; Grill-Spector et al., 2006; Ryan et al., 2007; Hannula and Ranganath, 2009; Hannula et al., 2010; Heisz and Ryan, 2011).

Discussion

In this study, increases in the number of gaze fixations made during the viewing of novel faces were significantly associated with stronger activation in the hippocampus, a structure critical for the binding of lasting memory representations (Squire, 1992; Cohen and Eichenbaum, 1993; Eichenbaum and Cohen, 2001), as well as early visual (e.g., occipital) and perceptual processing regions (e.g., FFA). The relationship between gaze fixations and hippocampal responses was robust and remained significant after controlling for potential confounding variables related to individual behavior, such as reaction time or total viewing time, and variables related to stimulus differences, such as face sex or size. The number of fixations predicted hippocampal activation during viewing of novel, but not repeated, faces. This is in contrast to the FFA, which showed a significant positive relationship between the number of fixations and activity regardless of novelty. Increases in gaze fixations made during the viewing of novel faces predicted stronger repetition suppression (i.e., a decreased response from novel to repeated face processing) in the hippocampus. Given that the only difference between viewing of a novel face and viewing of a repeated face is in a participant's prior viewing history, any change in neural activity from novel to repeated viewing must be due to the influence of memory. The modulation of the repetition suppression effect by gaze fixations thus suggests that the gaze fixations enacted during the viewing of novel faces may be related to the development of a lasting representation (Grill-Spector et al., 2006; Kumaran and Maguire, 2007, 2009; Johnson et al., 2008; Suzuki et al., 2011a, 2011b; Vannini et al., 2013; Kremers et al., 2014). To the best of our knowledge, this is the first evidence from human neuroimaging showing that gaze fixations predict hippocampal activation, suggesting that visual sampling behavior and hippocampal responses are inherently linked.

Information regarding the visual environment accumulates across fixations (Irwin, 1991; Rayner and Pollatsek, 1992; Pertzov et al., 2009; Shen and Paré, 2014). This allows for the encoding of disparate high-resolution visual information into coherent memory representations to support ensuing cognition and behavior (Yarbus, 1967; Melcher, 2001; Henderson, 2003; Melcher and Morrone, 2007). Representations maintained in memory can affect, and be affected by, gaze fixation patterns (Loftus, 1972; Ryan and Cohen, 2004; Henderson et al., 2005; Holm and Mäntylä, 2007; Castelhano et al., 2009; Foulsham and Kingstone, 2013; Olsen et al., 2014). Studies with older adults and neuropsychological cases have found altered fixation patterns that may be functionally linked to these participants' memory deficits (Ryan et al., 2000; Hannula et al., 2007; Chan et al., 2011; Voss et al., 2011; Shih et al., 2012; Olsen et al., 2015; Rondina et al., 2016a). Neuroimaging studies also show that hippocampal responses are predictive of (or aligned with) eye movement patterns that express memory retrieval (Hannula and Ranganath, 2009; Ryals et al., 2015). These findings suggest that gaze fixation patterns are indicative of retrieval of representations that were bound by the hippocampus. However, to date, no functional neuroimaging study in humans has examined directly whether visual sampling behavior such as the number of fixations made by the viewer predicts hippocampal responses during encoding. Given that the hippocampus receives input from visual neocortical systems (Felleman and Van Essen, 1991; Bussey and Saksida, 2007; Lee et al., 2012) and facilitates the binding of visual information processed by these neocortical regions (Davachi, 2006; Eichenbaum and Cohen, 2014), we predicted that increases in visual sampling would be associated with stronger activation in visual neocortical regions as well as the hippocampus, presumably to support the formation of lasting representations. The current finding that more fixations predicted stronger activation in these brain regions (even after controlling for potentially confounding factors) is consistent with this hypothesis and suggests that visual sampling may be directly related to the formation of representations bound by the hippocampus.

These findings are consistent with previous neurophysiological studies finding that that visual sampling (either the onset of fixations or the onset/offset of saccades) modulated neuronal activity directly in the hippocampus and other temporal and occipital regions (Ringo et al., 1994; Sobotka and Ringo, 1997; Sobotka et al., 1997; Rajkai et al., 2008; Hoffman et al., 2013; Jutras et al., 2013). For example, in a series of studies, Ringo and colleagues observed that neurons in visual neocortical and medial temporal lobe regions, including the hippocampus, responded within 300 ms after the onset of a saccade (Ringo et al., 1994; Sobotka and Ringo, 1997; Sobotka et al., 1997). Evidence also shows that the phase of theta-band oscillations in the hippocampal local field potentials, which have been related to memory processing (Buzsáki and Moser, 2013), can be reset or realigned by fixation or saccadic onset in humans and in monkeys (Hoffman et al., 2013; Jutras et al., 2013). Moreover, neural activation was modulated by fixations/saccades in visual neocortical and medial temporal lobe regions even in full darkness and during sleep (Ringo et al., 1994; Sobotka and Ringo, 1997; Lee and Malpeli, 1998; Rajkai et al., 2008; Andrillon et al., 2015), indicating that the physiological coupling between eye movements and neural activity may be obligatory and likely serves to prime subsequent neural activity in regions devoted to perceptual processing and memory (Rajkai et al., 2008; Andrillon et al., 2015).

The manner by which the inherent coupling between eye movements and neural activity is related to the formation of long-term memory representations during encoding had not been examined by previous neurophysiological studies. We reasoned that, if visual sampling and hippocampal responses were related to the development of a lasting representation, then visual sampling should no longer be related to hippocampal responses during repeated presentations of the faces if those representations had been established. Consistent with this hypothesis, we found that the number of fixations positively predicted hippocampal activation only during viewing of novel, but not repeated, faces. Increased fixations during viewing of novel faces also predicted stronger repetition suppression in the hippocampus. These results are consistent with the rich literature demonstrating that hippocampal responses are sensitive to novelty and that repetition-related suppression in the hippocampus is related to successful representation formation (Miller et al., 1991; Brown and Aggleton, 2001; Grill-Spector et al., 2006; Kumaran and Maguire, 2007, 2009; Johnson et al., 2008; Suzuki et al., 2011a, 2011b; Vannini et al., 2013; Kremers et al., 2014). Therefore, it is likely that previous behavioral findings of better memory performance associated with higher numbers of fixations (Loftus, 1972; Henderson et al., 2005; Castelhano et al., 2009) were due to stronger neural activity in regions such as the hippocampus that was not measured in previous work.

We acknowledge that, during viewing of repeated faces, it is possible that initial fixations were related to hippocampal responses that reflect the retrieval of the face representation in memory and that, after such retrieval, hippocampal responses declined and were no longer related to gaze fixations. Alternatively, some proportion of the total fixations during repeated face viewing may strengthen the representations that were initially formed during the novel face viewing and thereby continue to contribute to hippocampal responses, but this effect may be masked by other fixations with a function that was not specifically related to the strengthening of those representations. We were unable to distinguish among these alternatives within the current paradigm. Moreover, the current study was not designed to disentangle the role of (covert and/or overt) attention in the relationship between fixations and hippocampal responses. However, given that the number of fixations did not predict hippocampal responses in repeated face conditions and that fixation spatial distributions were comparable in all conditions, the observed fixation–hippocampus relationship is likely not be due to deployment of overt spatial attention per se. Future studies could address these questions.

In contrast to the hippocampus, activity in the FFA showed a positive relationship with the number of fixations during viewing of both novel and repeated faces. This suggests that the FFA may serve to gather and process information within the faces that is ultimately fed forward to the hippocampus, but such activity within the FFA may not support the lasting face representation per se. Consistent with this, we found that more fixations made during viewing of novel faces did not significantly predict stronger repetition suppression in the FFA. Therefore, the relationship between visual exploration and neural activation may not be identical across different regions along the visual processing hierarchy.

In conclusion, the current study provides novel evidence for a relationship between visual exploration and hippocampal activity in humans. The movement of the eyes may reflect the ongoing binding process that occurs along the visual processing hierarchy (Lee et al., 2012), particularly within the hippocampus, which plays a key role in integrating information across space and time to form a lasting representation (Konkel and Cohen, 2009; Staresina and Davachi, 2009; Olsen et al., 2013; Eichenbaum, 2014; Rondina et al., 2016b). Together with recent anatomical evidence found in the macaque (Shen et al., 2016), we suggest that there is an inherent link, functionally and anatomically, between the brain's oculomotor system and its hippocampal system.