The spatial distribution of attention predicts familiarity strength during encoding and retrieval

Abstract

The memories we form are determined by what we attend to, and conversely, what we attend to is influenced by our memory for past experiences. Although we know that shifts of attention via eye movements are related to memory during encoding and retrieval, the role of specific memory processes in this relationship is unclear. There is evidence that attention may be especially important for some forms of memory (i.e., conscious recollection), and less so for others (i.e., familiarity-based recognition and unconscious influences of memory), but results are conflicting with respect to both the memory processes and eye movement patterns involved. To address this, we used a confidence-based method of isolating eye movement indices of spatial attention that are related to different memory processes (i.e., recollection, familiarity strength, and unconscious memory) during encoding and retrieval of real-world scenes. We also developed a new method of measuring the dispersion of eye movements, which proved to be more sensitive to memory processing than previously used measures. Specifically, in two studies, we found that familiarity strength—that is, changes in subjective reports of memory confidence—increased with i) more dispersed patterns of viewing during encoding, ii) less dispersed viewing during retrieval, and iii) greater overlap in regions viewed between encoding and retrieval (i.e., resampling). Recollection was also related to these eye movements in a similar manner, though the associations with recollection were less consistent across experiments. Furthermore, we found no evidence for effects related to unconscious influences of memory. These findings indicate that attentional processes during viewing may not preferentially relate to recollection, and that the spatial distribution of eye movements is directly related to familiarity-based memory during encoding and retrieval.

How we view the world is influenced by our memory: for example, we know approximately where to look when searching for objects based on past experiences (Torralba, Oliva, Castelhano, & Henderson, 2006; Võ & Wolfe, 2015). Conversely, our ability to learn about new visual information depends on how we look at that information: where we direct our attention largely determines what we encode into memory (e.g., Henderson & Hollingworth, 1999, 2003; Hollingworth, 2006). This bidirectional relationship between memory and visual attention has long been supported by a wealth of research in both memory and attention (Hannula, 2010; Henderson, 2003; Meister & Buffalo, 2016). However, recent evidence indicates that understanding the relationship between memory and visual attention may require consideration of the type of memory involved, because different memory processes may be related to attention in different ways. For example, memory for prior experiences can be based on a variety of underlying memory processes, such as conscious recollection for details of an experience, assessments of stimulus familiarity, as well as unconscious forms of memory that occur without awareness (Roediger, 1990; Schacter, Chiu, & Ochsner, 1993; Yonelinas, 2001, 2002). A number of studies suggest that conscious recollection may have a uniquely strong relationship with visual attention (Holm & Mantyla, 2007; Kafkas & Montaldi, 2012; Mantyla & Holm, 2006; Sharot, Davidson, Carson, & Phelps, 2008), but other studies indicate that attentional mechanisms may be related to familiarity as well (Kafkas & Montaldi, 2011), and still others suggest that many experience-related changes in attention may be due to unconscious memory (Hannula, 2010; Hannula & Ranganath, 2009; Ryan, Althoff, Whitlow, & Cohen, 2000). In addition, there is emerging evidence that these memory processes may be differentially related to distinct aspects of spatial attention as indexed by eye movements (Ramey, Yonelinas, & Henderson, 2019), but very little is known about the spatial attentional mechanisms involved in successful memory encoding and retrieval, even irrespective of memory processes. Resolving these issues is important both for accurately characterizing the functional nature of different memory processes, and for understanding the manner in which attentional processes influence—and are influenced by—memory for past experiences.

There are several ways in which visual attention, as indexed by eye movements, can influence or be influenced by memory during encoding and retrieval. For example, during encoding, attention can determine which image components are ultimately stored in memory (Henderson & Hollingworth, 1999, 2003; Hollingworth, 2006). In addition, memory for past encounters with an image may impact how we subsequently view that image (Althoff & Cohen, 1999; Ryan et al., 2000; Smith, Hopkins, & Squire, 2006; Smith & Squire, 2008, 2017), and conversely, preliminary evidence suggests that during memory retrieval, attention may influence memory by determining which areas of a scene are utilized as retrieval cues (Foulsham & Kingstone, 2013; Valuch, Becker, & Ansorge, 2013). However, it is not clear how the distribution of attention across scenes is related to different memory processes.

Examinations of eye movements during memory encoding and retrieval have thus far largely involved measures indexing general oculomotor behavior (e.g., the number of fixations made during viewing, and the average saccade amplitude or distance between fixations) rather than the distribution of visual attention per se. Nonetheless, these studies have provided useful clues as to how visual attention may relate to memory. For example, an increased number of fixations during encoding has been found to predict better overall memory performance on a subsequent memory test (Kafkas & Montaldi, 2011; Loftus, 1972; Sharot et al., 2008), suggesting that sampling more information, independent of encoding duration, may improve memory. However, studies aimed at determining how the number of fixations and average saccade amplitude during encoding predict subsequent recollection or familiarity have not yet yielded a clear conclusion. Specifically, one study found that the number of fixations made while viewing images was related to both recollection and familiarity strength (Kafkas & Montaldi, 2011), but a similar study found that the number of fixations was not specifically related to either memory process (Sharot et al., 2008). Both studies concluded that recollection was related to decreased saccade amplitude during encoding, which suggests that saccade amplitude may uniquely predict subsequent recollection.

At time of retrieval, previously studied images generally elicit fewer fixations than new images (Althoff & Cohen, 1999; Ryan et al., 2000; Smith et al., 2006; Smith & Squire, 2008, 2017), suggesting that memory leads to more efficient processing of repeated images. However, it is not yet clear how this effect relates to different memory processes. One study found that, compared to familiarity, recollection was related to decreased saccade amplitude and fewer fixations (Sharot et al., 2008), whereas another found that recollection was related to increased saccade amplitude and more fixations (Kafkas & Montaldi, 2012). In addition to conflicting results regarding how eye movements during retrieval may be related to recollection and familiarity, there is debate surrounding whether experience-driven changes in attention are instead a result of unconscious memory, rather than any form of conscious memory (Ryan et al., 2000; Smith et al., 2006; Smith & Squire, 2008, 2017). In a study directly examining how recollection, familiarity, and unconscious memory may drive experience-related changes in attention during an implicit visual search task, we found that recollection (i.e., conscious memory) and unconscious memory outside of awareness—but not familiarity—were each related to distinct spatial patterns of eye movements (Ramey et al., 2019). Specifically, we found that recollection was uniquely associated with increased accuracy of the first eye movement in a trial in terms of heading towards the learned location of a search target, and that unconscious memory was uniquely associated with an improvement in search efficiency throughout the trial. These results indicate that both recollection and unconscious memory may relate to distinct patterns of spatial attentional allocation, but it remains to be seen whether analogous findings apply during explicit encoding and retrieval.

Finally, investigations of the extent to which image regions are revisited between successive viewings have provided a rare, direct examination of spatial allocation of attention during encoding and retrieval. Overall memory tends to be better when participants resample scene regions at retrieval that they had initially visited at encoding (Foulsham & Kingstone, 2013; Wynn et al., 2016). Furthermore, this effect appears to be bidirectional, because forcing participants to resample more regions improves memory for a scene in some cases, and cueing memory prior to viewing increases the degree of resampling upon viewing (Foulsham & Kingstone, 2013; Holm & Mantyla, 2007). Importantly, there is evidence that this effect may be recollection-related, such that increased resampling has been found to uniquely relate to recollection, but familiarity strength has not yet been examined (Holm & Mantyla, 2007; Mantyla & Holm, 2006).

The notion that recollection, rather than familiarity, is particularly related to attentional allocation may seem intuitive given that recollection involves memory for specific details of an experience (Yonelinas, 2002), and eye movements allow for the extraction of detailed visual information (Henderson, 2003). Familiarity-based recognition, on the other hand, is posited to involve a sense of global similarity (Yonelinas, 2002), and the overall gist of visual stimuli may be ascertained without the need for eye movements (Henderson, 2003). Despite this, however, previous results using general oculomotor measures are inconsistent with respect to the directions of the associations between different eye movement patterns and recollection, and to what extent familiarity strength is involved (Kafkas & Montaldi, 2011, 2012; Sharot et al., 2008). Thus, the existing literature shows that visual attention is related to memory at both encoding and retrieval, but it is not yet clear how it is related to specific memory processes.

One possible reason for the conflicting findings is that memory strength has not been taken into account in most studies, which could cause familiarity strength effects to appear as recollection effects (Montaldi, Spencer, Roberts, & Mayes, 2006; Yonelinas, Otten, Shaw, & Rugg, 2005). That is, while recollection-based responses most often involve high-confidence recognition, familiarity-based responses vary widely in recognition confidence (Yonelinas et al., 2005). Therefore, when a dichotomous measure is used such that recollection is compared to all levels of familiarity confidence, as has most often been done in eye movement investigations of memory processes, observed relationships between attention and memory could be due to associations with overall memory strength instead of recollection. In fact, in one study that did assess memory strength, the reported relationship between the number of fixations and recollection did not hold when recollected stimuli were compared to stimuli with only high-strength familiarity, as opposed to all familiar stimuli (Kafkas & Montaldi, 2012). This result suggests that some effects interpreted to be recollection-specific may actually have been a result of the high memory strength that recollection entails rather than recollection per se.

In a similar vein, conflicting conclusions regarding whether eye movements reflect conscious or unconscious influences of memory (Hannula, 2010; Ryan et al., 2000; Smith et al., 2006; Smith & Squire, 2008, 2017) may also be related to a failure to consider memory strength. For example, in most of these previous studies, conscious memory was assessed using dichotomous old/new recognition measures in which items falling above the participants’ response criterion are treated as conscious (i.e., the hits), whereas items falling below the criterion are treated as unconscious (i.e., misses). However, because familiarity strength can vary, it is not clear if eye movements associated with misses reflect truly unconscious memory outside of awareness, or simply low levels of familiarity. In order to tease apart these possibilities, familiarity strength needs to be assessed using confidence-based measures to determine whether eye movement changes are related to conscious or unconscious memory.

Although confidence-based methods have been used in the memory literature in the past to examine unconscious memory, examining it alongside recollection and familiarity may be particularly important for understanding attention given the debates that have arisen regarding all three processes, outlined above. Specifically, it is possible that recollection, familiarity strength, and unconscious memory may each be characterized by unique relationships with viewing behavior, which would not be possible to assess without directly isolating them. For example, when only recollection and familiarity are examined, unconscious effects could be mistakenly attributed to familiarity, whereas when only unconscious and conscious memory are examined, familiarity effects could be mistakenly attributed to unconscious memory. However, to our knowledge, no study of attention during encoding and retrieval has examined familiarity and unconscious memory, or recollection and unconscious memory, in conjunction.

As mentioned above, another aspect of the relationship between eye movements and memory that has yet to be examined is the extent to which the spatial allocation of visual attention during encoding and retrieval, rather than general oculomotor measures (i.e., number of fixations and saccade amplitude), is related to different memory processes. This distinction is particularly important in light of findings that recollection and unconscious memory were each related to unique indices of spatial attention during search—but general oculomotor measures were not able to dissociate these memory processes (Ramey, Henderson, & Yonelinas, 2019). Importantly, the studies outlined above that examined saccade amplitude interpreted the relationship between decreased saccade amplitude during encoding and subsequent recollection to indicate that recollection is related to making less dispersed fixations (Kafkas & Montaldi, 2011; Sharot, Davidson, Carson, & Phelps, 2008)—that is, constraining viewing to a small scene region. However, because the spatial locations of fixations were not considered, it is not clear how recollection and familiarity may relate to the dispersion of attention across a stimulus during encoding. Furthermore, it is not yet known how spatial dispersion of attention during encoding and retrieval may relate to memory performance for scenes in general, in addition to the question of underlying memory processes.

Current Research

In the present study, we addressed these questions by examining how spatial eye movement patterns during encoding and retrieval of scenes are related to recollection, familiarity, and unconscious memory for those scenes. To this end, we used a recently developed confidence-based memory assessment method that allowed us to isolate the effects of these different memory processes (Ramey et al., 2019). We also developed two new eye movement measures of spatial attention to quantify resampling and dispersion of attention across a stimulus.

In two experiments, participants viewed a series of real-world scenes while their eye movements were tracked. The second experiment served as a replication of the first, using a different stimulus set and a different group of participants. During an initial study phase, participants viewed a series of scenes in two encoding tasks. In one encoding task, participants were asked to memorize each scene, whereas in the other, they were asked to judge each scene for its aesthetic appeal. The two different encoding blocks were included to test the generalizability of any effects obtained as well as to verify that the effects were not limited to conditions in which participants intentionally encoded the scenes. During a subsequent test phase, participants viewed the same scenes that they had viewed during the study phase (i.e., old scenes) along with randomly intermixed new scenes, and were asked to provide a recognition judgment for each scene. Memory awareness was measured by asking participants to rate memory confidence for each scene on a 6-point scale during the recognition judgment. Participants were told that if they could consciously recollect some qualitative aspect of the initial learning event, such as what they thought about when the scene was encountered earlier, they should respond “Recollect old (6);” otherwise, they rated their memory confidence by responding “I’m sure it’s old (5),” “Maybe it’s old (4),” “I don’t know (3),” “Maybe it’s new (2),” or “I’m sure it’s new (1).”

To isolate eye movement patterns that were related to recollection, we examined the old scenes that were confidently recognized as old and for which participants reported being able to retrieve specific details about the study event (i.e., “recollect old”). To assess eye movement patterns related to familiarity, we examined intermediate levels of memory confidence for old scenes, ranging from high familiarity strength (i.e., “I’m sure it’s old”) to low familiarity strength (i.e., “I’m sure it’s new”). To isolate eye movement patterns related to unambiguously unconscious memory, we examined the old scenes that participants were confident had not been studied (i.e., receiving a response of “I’m sure it’s new”). That is, examining only “sure new” responses ensured that we excluded any scenes for which there was even a weak sense of conscious memory (i.e., the “maybe it’s old,” “I don’t know” and “maybe it’s new” scenes). Thus, any observed eye movement differences between new and old scenes given a response of “I’m sure it’s new” could not be attributable to conscious memory.

In addition to employing a recent method of isolating recollection, familiarity, and unconscious memory, recent computational advances have allowed us to develop a new measure to directly assess the allocation of visual attention across a stimulus using the spatial distribution of eye movements. As outlined above, the measures that have been the primary focus of past studies of eye movements during encoding and retrieval are number of fixations and saccade amplitude (Kafkas & Montaldi, 2011, 2012; Sharot et al., 2008), which provide useful information about general oculomotor behavior but do not capture the deployment of visual attention across a stimulus. For example, an increase in the number of fixations does not necessarily entail more dispersed visual attention, as fixations are often clustered in a small region. Given that visual processing and successful encoding each heavily relies on where attention is deployed (e.g., Henderson & Hollingworth, 1999, 2003; Hollingworth, 2006), directly assessing the spatial distribution of viewing may be the key to understanding the relationship between visual attention and memory. Thus in addition to assessing previously used measures of oculomotor behavior such as saccade amplitude and number of fixations, we introduce two new measures of attention: one that uses cluster analysis to quantify spatial dispersion of eye movements across a stimulus, and one that provides a continuous measure of resampling of regions between viewings.

Given the conflicting findings surrounding the relationship between memory processes and eye movements during encoding and retrieval, a variety of outcomes are of interest. First, it is possible that the spatial allocation of visual attention via eye movements may selectively relate to recollection. This would be consistent with proposals that recollection is particularly related to visual attention, compared to familiarity or unconscious forms of memory (Holm & Mantyla, 2007; Kafkas & Montaldi, 2011, 2012; Mantyla & Holm, 2006). It would also indicate that attentional processes interact with some forms of memory but not others during encoding and retrieval. Alternatively, spatial attention may be associated with familiarity and/or unconscious forms of memory, suggesting that attentional processes interact more broadly with memory during encoding and retrieval. In addition, different memory processes may be related to different patterns of eye movements during encoding and/or retrieval—analogous to our prior findings using a visual search task (Ramey et al., 2019)—indicating that different types of memory are related to attention in different ways.

General Method

In two experiments, participants’ eye movements were tracked while they viewed a series of real-world scene photographs during encoding (i.e., study phase) and retrieval (i.e., test phase). In the test phase, recognition memory was assessed for each scene, allowing us to examine how eye movements during encoding and retrieval related to recollection, familiarity, and unconscious memory for scenes. The second experiment served as a replication of the first, using different scenes and participants.

Experiment 1

Participants.

Twenty-two undergraduates from the University of California, Davis completed the experiment for course credit. The sample size was selected to provide more than 80% power to detect the weakest effect of recollection on eye movements obtained in our prior study (Ramey et al., 2019). All participants had normal or corrected-to-normal vision, and provided informed consent in accordance with the study protocol as approved by the university IRB. The quality of each participant’s eyetracking data was assessed by computing the mean percent signal across all trials, to determine whether there was excessive track loss due to blinks or calibration loss. All participants had greater than the preselected criterion of 75% signal (M = 96.5%) (Henderson & Hayes, 2017), such that they lost less than 25% signal; all participants were thus retained for analysis.

Stimuli.

Stimuli were 204 photographs of real-world indoor scenes. All scenes were presented in color at 1024x768 pixels subtending a visual angle of approximately 25°xl9° at presentation, and were free of people, animals, and text. Of these 204 scenes, 4 were used in practice trials, 150 were presented at study and test, and 50 were presented only at test. Stimulus presentation was counterbalanced, such that each scene appeared in different conditions (i.e., in one of the two study tasks, or as a new lure during test; see procedure) for different participants, to mitigate stimulus effects¹.

Apparatus.

Participants’ eye movements were recorded using an SR Research EyeLink 1000+ tower mount eyetracker, sampling at 1000hz. A forehead and chin rest were used to reduce head movements, and eye movements were recorded from one eye though viewing was binocular. Stimuli were displayed on a monitor 85cm from the eyetracker, and the experiment was controlled with SR Research Experiment Builder software (SR Research, 2010a).

Procedure.

The experiment lasted 1.5 hours, and consisted of a study phase followed by a filled 30min delay, as well as a subsequent test phase (see Figure 1). Eye movements were recorded throughout the study and test phases. In both phases, each trial (i.e., each scene presentation) was preceded by a central fixation cross. Participants were given breaks every 50 trials and between phases, and the eyetracker was recalibrated after each of these breaks.

Fig. 1.

Illustration of the procedure. A) Study phase. Half of the scenes were presented in an aesthetic judgment task (i.e., participants were instructed to judge the image aesthetically and rate it as “dislike,” “neutral,” or “like”), whereas the other half were presented in a memorization task (i.e., participants were instructed to memorize the image and rate it as “not memorable,” “neutral”, or “memorable”). B) Delay between study and test, during which participants completed unrelated questionnaires. C) Test phase in which participants rated their recognition confidence.

Study phase.

During the study phase, participants were presented with 150 unique scenes split into two task blocks: an aesthetic judgment task and a memorization task. These tasks were selected to ensure that any effects obtained during encoding were not a product of a given task, but rather generalized across tasks (as prior work has shown that eye movements vary systematically between tasks; Castelhano, Mack, & Henderson, 2009; Henderson, Shinkareva, Wang, Luke, & Olejarczyk, 2013; Kardan, Berman, Yourganov, Schmidt, & Henderson, 2015; Mills, Hollingworth, Van der Stigchel, Hoffman, & Dodd, 2011). The order of the tasks was counterbalanced such that half of the participants completed the aesthetic judgment task first, whereas the other half completed the memorization task first. In each task, 75 scenes were presented for 3.5s each, allowing for an average of 12 fixations per trial. Each task was preceded by two practice trials to familiarize participants with the procedure.

In the aesthetic judgment task of the study phase, participants were asked to rate each scene based on how aesthetically pleasing they found it to be. Each trial consisted of a 3.5s scene presentation, followed by a grey response screen containing the prompt “What is your opinion of the photo?” as well as the key mappings for each response option. Responses were made on the keyboard, had no time limit, and consisted of “dislike,” “neutral,” and “like;” the response data were not used.

The memorization task of the study phase followed the same general procedure, but participants were instead asked to memorize the scenes. After each scene, they were asked to rate how memorable they found the scene to be. Participants were asked to give this response to ensure that the sequence of events in the memorization task was analogous to the aesthetic judgment task. Responses included “not memorable,” “neutral,” and “memorable;” again, the response data were not used.

Delay.

Between the study and test phases, participants were moved to a computer in a different room to complete a 30min distractor task that included questionnaires (e.g., personality scales; see Appendix) that were not related to the present study.

Test phase.

In the test phase, participants were presented with a series of scenes and asked to rate their memory for each scene. The test phase consisted of 200 trials: 150 old scenes, which had been presented in the study phase, and 50 new scenes, which had not been presented previously. Each scene was presented for 3.5s, as in the study phase, and was subsequently replaced by a recognition judgment screen.

For the recognition judgment, participants indicated whether or not they recognized the scene from the study phase. They were given as much time as they needed to select their response. Response options fell on a 1-5 and recollect scale made up of “sure new,” “maybe new,” “don’t know,” “maybe old,” “sure old,” and “recollect old” (Ramey et al., 2019; Yonelinas, 2002). Participants were instructed and tested on how to use this scale prior to beginning the test phase. A response of “recollect old” indicated that a participant could recall details of their experience of having seen the image in the study phase. Examples given to participants included remembering the study task in which they initially viewed the scene (i.e., memorization or aesthetic judgment), remembering an emotion they felt during prior exposure to the scene, and remembering ambient noise or sensations experienced during previous viewing of the scene. Participants were explicitly instructed that responses of “recollect old” and “sure old” were categorically different, rather than varying in memory strength. The other responses fell on a continuous gradient ranging from no memory to strong memory for an image, with a response of “sure old” indicating memory strength comparable to that of “recollect old” but without the additional episodic details.

Experiment 2

The methods and procedure were the same as those used in Experiment 1, with the following exceptions.

Participants.

Forty-five undergraduates from the University of California, Davis completed the experiment for course credit. The sample size was selected to provide more than 98% power to detect the weakest effect of recollection on eye movements obtained in our prior study (Ramey et al., 2019), and more than 90% power to detect the weakest recollection effect obtained in Experiment 1. Eyetracking signal was greater than 75% in all participants (M = 94.7%), so all subjects were retained for analysis.

Stimuli.

Stimulus characteristics were similar to Experiment 1, with the exception of content: rather than using only indoor scenes, a mix of new indoor and outdoor scenes were used. We also expected that the increased diversity of scenes would lead to higher recognition accuracy and more “recollect” responses, which would give us more power to detect recollection-specific effects.

Data Reduction and Analysis

Measures.

Fixations and saccades were segmented with EyeLink’s standard algorithm using velocity and acceleration thresholds (30°/s and 9500°/s²; SR Research, 2010b). Eye movement data were imported offline into Matlab using the EDFConverter tool. We computed the eye movement measures as follows.

Dispersion.

The extent to which eye fixations are focused on a small number of regions or dispersed across a large number of regions of an image has been suggested to play an important role in how well an image is remembered, and which memory processes are involved (Kafkas & Montaldi, 2011, 2012; Sharot et al., 2008). To quantify attentional dispersion, we calculated the number of regions that were fixated in a scene, as well as the distance between those regions. This allowed us to determine how eye movements were spatially distributed, by taking into account how many fixation clusters were formed and how those clusters were distributed across the scene.

Specifically, we used a clustering algorithm to group fixations based on their spatial relation to each other, and created a composite dispersion score based on both the number of fixation clusters in a trial and the distance between those clusters. To this end, we first submitted the fixation locations for each trial, in terms of the (x,y) coordinates recorded by the eyetracker (with each coordinate reflecting one pixel in the 768x1024 pixel screen), to a k-means clustering algorithm (Maechler, 2015). Then, using a silhouette algorithm that assessed how similar each fixation was to its own cluster versus other clusters (Rousseeuw, 1987), we identified the optimal number of clusters that described the fixation locations (the minimum number of clusters was two). Using the optimal clustering solution, we then computed the average distance between the centroids of those clusters. Lastly, we created the dispersion score by multiplying the number of clusters by the average distance between the clusters, to capture the extent to which eye fixations were distributed across the display (Fig. 2a–f). Therefore, higher values indicate that fixations were more distributed across a scene. This dispersion score was used in subsequent analyses, and we refer to it as dispersion.

Fig. 2.

Visualizations of primary eye movement measures of interest. The rings in A-F represent fixations. Each cluster is denoted by a different color, such that the color of a fixation indicates its cluster membership. A) An example of a low-dispersion trial. B) An example of a medium-dispersion trial. C) An example of a high-dispersion trial. A-C are trials comprised of 12 fixations, which was the mode number of fixations, to illustrate how dispersion can vary for a given number of fixations. Similarly, D-F each contain three clusters of fixations, to illustrate how dispersion can vary for a given number of clusters. D) An example of a low-dispersion trial. E) An example of a medium-dispersion trial. F) An example of a high dispersion trial. G) An example of a smoothed fixation map, presented as a heatmap, of one subject viewing an image at study. H) The fixation map for that same subject and image during the test phase. I) The regions that overlap between the study and test maps, for the purposes of illustration. (The resampling measure captures the correlation between G and H.)

Resampling.

To determine the extent to which the same regions were viewed between the study and test presentations of a scene, we created maps of the regions visited in each trial (Fig. 2g–h). To create these maps, we generated a matrix of fixation locations for each trial, in terms of (x,y) coordinates recorded by the eyetracker. A Gaussian low-pass filter with a cutoff frequency of −6dB (Henderson & Hayes, 2017) was then applied to the matrices to account for the fact that only the single pixel at the center of fixation is recorded by the eyetracker, whereas visual acuity is more diffuse (Bylinskii, Judd, Oliva, Torralba, & Durand, 2018). The resulting map represents the density of fixations at each pixel in a scene (Fig. 2g–h). For a given participant, we computed the Pearson correlation coefficient between the fixation map generated while they viewed a scene at study with the fixation map of that same scene at test (Fig. 2i). The resulting resampling value reflects the extent to which fixation locations were similar between study and test, such that higher values indicate that there was more overlap in the regions visited between study and test of a scene. It should be noted that resampling captures the proportion of fixations at retrieval that were devoted to revisiting previously viewed regions, such that it controls for the number of fixations made.

Additional measures.

To better interface with prior literature, we also examined other, converging measures of eye movement behavior that have been used in past eyetracking studies. First, we assessed the number of regions visited in a trial, which has been taken to represent the dispersion of viewing. However, it should be noted that this measure does not consider the distance between regions, and a high number of regions visited could result from viewing constrained to a relatively small portion of the scene. For this analysis, we divided each scene into 64 evenly spaced rectangles (16x12 pixels), each of which defined a region. We then determined how many unique regions were fixated in a trial.

In addition to the number of regions visited, we also examined two eye movement measures that did not take spatial information into account. First, we calculated the number of fixations made in a trial. Second, we assessed saccade amplitude, which is the average distance between fixations in a trial. Each of these measures has also been considered by some to serve as a proxy for dispersion of viewing (Kafkas & Montaldi, 2011, 2012; Sharot et al., 2008).

Statistical models.

Statistical analyses were conducted using linear mixed effects models with crossed random effects of participant and image, which allowed us to harness trial-by-trial (i.e., within-subjects) data while controlling for individual differences and stimulus effects. The models were estimated using the ImerTest package in R (Kuznetsova, Brockhoff, & Christensen, 2017), and were fit using maximum likelihood. The degrees of freedom and t values used were output by the linear mixed effects model for the variables of interest. The degrees of freedom were computed using the Satterthwaite approximation, and were rounded to the nearest integer in the manuscript. The models for each analysis were specified by regressing the eye movement measure in question on the memory variable, which depended on the type of memory being assessed: conscious recollection, unconscious memory, and familiarity strength (Table 1). Effect sizes were calculated as classical Cohen’s d, as 2t/√df (Rosenthal & Rosnow, 1991), for the recollection and unconscious memory models, and as a standardized regression coefficient (β) for the linear gradient of familiarity strength.

Table 1.

Linear Mixed Effects Model Specifications for Each Analysis

Fixed effect: Memory contrast	Random effects	Included scenes
Familiarity strength: “Sure new”	Image, participant	All old scenes except those
through “Sure old”		given a “recollect” response
Recollection: “Recollect” versus	Image, participant	All old scenes given responses
“Sure old”		of “recollect” or “sure old”
Unconscious memory: “Sure new”	Image, participant	All scenes, both old and new,
old scene versus “Sure new” new scene		given a “sure new” response

Note. In each model, the outcome was the eye movement measure of interest (e.g., dispersion, resampling). The fixed effect was the memory response given to a scene, or, in the case of unconscious memory, the old versus new status of the scene. Random effects were selected a priori to control for potential confounding influences of participant and scene, given the repeated measures design.

Results

Memory Accuracy

In Experiment 1, the percentage of scenes that received a recognition confidence response of “recollect,” “sure old,” “maybe old,” “don’t know,” “maybe new,” “sure new,” respectively, were 34%, 19%, 16%, 15%, 11%, and 5% for old scenes, and 3%, 5%, 15%, 21%, 27%, and 29% for new scenes. In Experiment 2, the percentage of scenes receiving these respective responses were 46%, 25%, 11%, 8%, 6%, and 4% for old scenes, and 2%, 3%, 8%, 13%, 27%, and 47% for new scenes. These results suggest that participants were able to discriminate between old and new scenes, and used the full range of response options. Furthermore, as expected, Experiment 2 yielded more recollection responses to old scenes, and higher recognition accuracy overall.

Task Effects

It is possible that the two tasks that participants performed while studying the scenes (i.e., memorization and aesthetic judgment) yielded fundamentally distinct relationships between memory responses and the eye movement measures examined. To address this possibility, each eye movement measure was regressed on the interaction between memory response and the task performed at study. There were no significant interactions between study task and memory response at study or test, for dispersion or resampling (ps >.12). Therefore, subsequent analyses collapsed across task.

Dispersion at Study Predicting Subsequent Memory

Familiarity strength.

To assess how the spatial distribution of attention during the study phase predicted subsequent memory for the scenes, we compared dispersion of fixations between scenes that were given different memory responses in the test phase (for a review of the subsequent memory method, see Paller & Wagner, 2002). We first examined the effects of dispersion on familiarity strength by assessing whether there was a linear relationship between dispersion at study and subsequent familiarity confidence, using old scenes that were given any response except “recollect old” (Table 1). In both studies, we found that higher dispersion during study predicted higher subsequent familiarity strength, (Exp 1: β = .10, t(2110) = 4.90, p <.0001; Exp 2: β = .07, t(3575) = 4.01, p <.0001; Fig 3a–b). This suggests that more dispersed viewing during encoding of a scene, such that attention is more distributed across the scene, leads to subsequently higher familiarity.

Fig. 3.

Dispersion at study and test by memory response. Estimated marginal means controlling for participant and image are plotted, and the error bars represent the standard error of these estimated means from the model. A) Dispersion during the study phase in Experiment 1, sorted by subsequent memory response. B) Dispersion during the study phase in Experiment 2. C) Dispersion during the test phase in Experiment 1. The “new” bar includes scenes that were new in the test phase, and therefore contained no memory. All other bars besides “new” only include old scenes. D) Dispersion during the test phase in Experiment 2.

Recollection.

We then examined the extent to which dispersion of attention predicted subsequent recollection, by comparing scenes that were subsequently endorsed as “sure old” (i.e., high-confidence familiarity) with scenes subsequently endorsed as “recollect old” (Table 1). In both experiments, dispersion was numerically higher for recollected than for high-confidence familiar scenes, but the difference only reached statistical significance in Experiment 1, t(1734) = 2.37, p = .02, d = 0.11 (Experiment 2: t(4584) = 0.67, p = .50, d = 0.02). This suggests that more dispersed viewing may lead to a slight increase in recollection above high-confidence familiarity under some conditions, but the effect is not as consistent as the effect of dispersed viewing on familiarity strength.

Dispersion at Test Related to Memory

Familiarity strength.

We examined dispersion of viewing during the test phase, in which participants inspected each scene to determine if they recognized it. To assess how dispersion during the test phase related to familiarity strength, we compared dispersion between scenes that had been viewed in the study phase across different memory responses ranging from “sure new” through “sure old,” as in the model used for the study phase data (Table 1). Dispersion decreased significantly as familiarity strength increased in both Experiment 1, β = −.06, t(2109) = −2.94, p = .003,and Experiment 2, β = −.08, t(3534) = −4.92, p <.0001. These results suggest that less dispersed viewing during retrieval is related to increased familiarity strength.

To ensure that these effects were unique to old scenes (i.e., scenes for which participants had memory), we submitted the new scenes to the same analysis. Dispersion was not significantly related to subjective experiences of familiarity in these new scenes in Experiment 1, β = −.05, t(1054)= −1.73, p = .08, nor Experiment 2, β = .02, t(2191)= 0.97, p > .25. This suggests that dispersion was not simply related to memory responses in the absence of true memory. However, we note that the familiarity strength effect in old scenes was driven largely by the higher confidence familiarity responses (i.e., “sure old” and “maybe old” responses), and there were many fewer new scenes associated with this level of confidence.

Recollection.

To assess how recollection was related to dispersion of attention in the test phase, we again compared dispersion between scenes given a response of “recollect” and scenes given a response of “sure old” (Table 1). Similar to the effects of familiarity strength, recollection was related to a reduction in dispersion in both Experiment 1, t(1735) = −4.01, p <.0001, d = −0.19, and Experiment 2, t(4589) = −4.36, p <.0001, d = −0.13. It therefore appears that both recollection and familiarity are related to less dispersion during retrieval. Similar to familiarity strength, there was no significant relation between dispersion and recollection in new scenes in Experiment 1, t(60) = −0.15, p > .25, nor in Experiment 2, t(106) = −0.17, p > .25.

Unconscious memory.

To assess unconscious memory, we compared scenes that were previously viewed with scenes that were newly presented in the test phase². Importantly, we only examined scenes that were given a response of “sure new,” indicating that participants were confident that they had not seen them before (Table 1). This strict criterion for unconscious memory (i.e., only considering scenes that were endorsed as “sure new” rather than all misses) ensured that none of the scenes used in the unconscious memory contrast were contaminated by conscious recollection or familiarity, and that the scenes differed only in terms of whether or not the participant had seen them previously. Dispersion did not relate to unconscious memory in either Experiment 1, t(397) = −0.44, p = .66, d = −0.04, or Experiment 2, t(1236) = 0.04, p = .97, d = 0.00. Furthermore, Bayesian analysis provided substantial evidence for this null effect in Experiment 1, BF₁₀ = 0.13, and strong evidence for the null in Experiment 2, BF₁₀ = 0.07, such that any numerical difference was more than six times more likely to be explained by chance than by unconscious memory³. This suggests that unconscious memory did not influence dispersion of viewing.

Resampling Scene Regions Between Study and Test

Familiarity strength.

To assess how visiting scene regions at test that were initially visited at study related to memory for a scene, we compared the degree of resampling between scenes given different memory responses. Higher resampling scores reflect both increased resampling of previously visited regions, and as a corollary, decreased sampling of new regions. We first examined familiarity strength, by assessing whether resampling was linearly related to familiarity strength (Table 1). Resampling was significantly higher for scenes that were more familiar in both Experiment 1, β = .10, t(2064) = 4.59, p <.0001, and Experiment 2, β = .14, t(3543) = 8.12, p <.0001, which suggests that revisiting the same regions between study and test is associated with increased familiarity for a scene.

Recollection.

Resampling was then compared between scenes endorsed as “recollect old” and scenes endorsed as “sure old” (Table 1), to determine if resampling was related to recollection. Resampling was numerically higher for recollection than for high-confidence familiarity in both experiments, but the effect only reached statistical significance in Experiment 2, t(4098) = 3.05, p = .002, d = 0.10 (Experiment 1: t(1643) = 0.38, p = .7, d = 0.02).

Additional Analyses

In order to facilitate comparisons to previous studies, we examined several commonly used eye movement measures with respect to the current recollection, familiarity, and unconscious memory contrasts, and present the results of these analyses in Table 2. In general, these measures led to results that were similar to those obtained using the dispersion measure, which is also provided in Table 2 for ease of comparison between measures. Additionally, we conducted a model comparison to determine if the presently developed eye movement measures (i.e., dispersion and resampling) predicted memory above and beyond previously used measures, and found that a model with dispersion and resampling was superior in both experiments (see Appendix). Lastly, we re-ran the analyses examining dispersion and unconscious memory by collapsing all misses, to ensure that the lack of effects was not due to our strict definition of misses (see Appendix).

Table 2.

Memory Effects for Eye Movement Measures Related to Dispersion of Attention

Phase	Measure	Experiment	Familiarity β	Familiarity p	Recollection d	Recollection p	Unconscious d	Unconscious p
Study
	Sac. amplitude	1	.07	<.001	−0.04	.43	-	-
	Sac. amplitude	2	.05	<.001	−0.05	.12	-	-
	# fixations	1	.11	<.0001	0.15	.002	-	-
	# fixations	2	.11	<.0001	0.13	<.0001	-	-
	# regions visited	1	.10	<.0001	0.05	.31	-	-
	# regions visited	2	.10	<.0001	0.09	.002	-	-
	Dispersion	1	.10	<.0001	0.11	.02	-	-
	Dispersion	2	.07	<.0001	0.02	.50	-	-
Test
	Sac. amplitude	1	.01	.6	−0.06	.21	−0.07	.46
	Sac. amplitude	2	−.06	<.001	−0.16	<.0001	0.02	.71
	# fixations	1	.04	.03	−0.24	<.0001	−0.02	.88
	# fixations	2	−.02	.16	−0.13	<.0001	0.04	.47
	# regions visited	1	−.02	.44	−0.22	<.0001	−0.07	.49
	# regions visited	2	−.09	<.0001	−0.19	<.0001	0.05	.35
	Dispersion	1	−.06	.003	−0.19	<.0001	−0.04	.66
	Dispersion	2	−.08	<.0001	−0.13	<.0001	0.00	.97

Note. See Table 1 for descriptions of models for each memory process. Sac. amplitude = saccade amplitude. Significant results are bolded.

Saccade amplitude.

To better assess how the present results fit with past findings (Kafkas & Montaldi, 2011, 2012; Sharot et al., 2008), we examined saccade amplitude, which is the distance between fixations. The relation between saccade amplitude and memory was less consistent across the present experiments than dispersion, such that some effects only emerged in Experiment 2. However, its relation with familiarity strength was more consistent than with recollection, and was in the same direction as dispersion. In contrast to the present results, a prior study reported that average saccade amplitude during encoding increased with familiarity strength, but was lower for recollection than high-confidence familiarity (Kafkas & Montaldi, 2011). One possible reason for the discrepant results is the fact that the prior study did not control for participant-level effects, whereas the present study did. In support of this hypothesis, when we removed the random effect of participant that we included to eliminate potentially confounding individual differences, we found that we were able to replicate the pattern of saccade amplitudes obtained by Kafkas and Montaldi (2011) during encoding: increased saccade amplitude predicted subsequent familiarity strength (ps <.0001 in both experiments), but decreased saccade amplitude predicted subsequent recollection (ps <.05 in both experiments). When participant-level effects were controlled for, on the other hand, only the familiarity strength effect remained (Table 2). In contrast, the presently developed measure of dispersion did not show a reversal of the recollection effect when the participant covariate was removed. This suggests that saccade amplitude effects that were previously attributed to trial-by-trial changes in eye movements may instead reflect a relation between individual differences in saccade amplitude (Henderson & Luke, 2014) and individual differences in recollection—whereas dispersion may be less subject to individual differences.

Number of fixations.

The pattern of results with respect to the number of fixations in a trial was largely the same as that observed using the dispersion measure. Specifically, more fixations made at study predicted subsequent recollection and high familiarity strength, whereas fewer fixations at test were related to recollection and high familiarity strength (see Table 2).

Number of regions visited.

To provide a converging measure with the presently developed measure of spatial dispersion, we computed the number of regions visited. The results largely paralleled those obtained using the dispersion measure, such that familiarity strength was related to the number of regions visited, with the exception of Experiment 1 at retrieval. Interestingly, while dispersion at encoding was related to recollection in Experiment 1 but not Experiment 2, the opposite was true for number of regions visited.

See the Appendix for supplemental measures, correlations between eye movement measures, and analyses ensuring that the pattern of results was not altered by the differences in accuracy rates and scene type between experiments, or the criterion for unconscious memory.

Discussion

In the present study, we examined how recollection, familiarity strength, and unconscious memory related to the deployment of spatial visual attention during encoding and retrieval of real-world scenes. Participants’ eyes were tracked as they viewed a series of scenes during a study and test phase, and they provided confidence-based recognition judgments for each scene during the test phase. Recollection, familiarity strength, and unconscious memory were isolated based on the recognition judgments, and three different sets of analyses were used to assess how eye movement patterns related to these memory processes. In two experiments, we found that familiarity strength was robustly associated with viewing patterns during both encoding and retrieval. Specifically, we found that a more dispersed distribution of eye movements during encoding predicted subsequently stronger familiarity, whereas less dispersed eye movements during retrieval were related to stronger familiarity. These effects also emerged in additional measures of general eye movement behavior (i.e., saccade amplitude, number of fixations, and number of regions visited), indicating that there is a consistent relationship between familiarity strength and eye movement behavior. Furthermore, we found that increased resampling of previously viewed regions during retrieval was related to familiarity strength as well. Interestingly, we found that recollection followed the same trends as familiarity strength, with no evidence for effects unique to recollection-based memory. That is, like familiarity strength, recollection was also predicted by more dispersed eye movements at encoding, less dispersed eye movements at retrieval, and resampling of regions between encoding and retrieval; however, these effects were somewhat less reliable across studies. Moreover, we found no evidence for an influence of unconscious memory, such that no differences in eye movement patterns were observed between new scenes and old scenes for which participants did not have conscious memory (i.e., high-confidence misses).

Prior investigations of recollection and familiarity have concluded that the two processes fundamentally differ in their relationship with patterns of sampling behavior during both encoding and retrieval (Kafkas & Montaldi, 2011, 2012). Specifically, it has been suggested that recollection involves decreased saccade amplitude compared to familiarity during both encoding and retrieval (Kafkas & Montaldi, 2011; Sharot et al., 2008, but see Kafkas & Montaldi, 2012), and that overlap in regions visited between encoding and retrieval is uniquely related to recollection (Holm & Mantyla, 2007; Mantyla & Holm, 2006). These effects have been interpreted as showing that recollection and familiarity have qualitatively different relationships with eye movement behavior, such that recollection has a particularly strong relationship with eye movements to allow for the retrieval of specific details. However, it is possible that memory strength differences between recollection and familiarity may have confounded prior results, such that comparing recollection to all levels of familiarity strength—rather than strength-matched familiarity—may have obscured effects arising from differences in memory strength (Montaldi et al., 2006; Yonelinas et al., 2005). Taking memory strength into account, the present results reveal that familiarity strength is very strongly related to attentional dispersion, as well as other measures of eye movement behavior including saccade amplitude, across both encoding and retrieval. Moreover, we found that recollection and familiarity did not involve qualitative differences in dispersion of viewing or resampling of regions: The relationship of these eye movements with recollection paralleled their relationship with familiarity strength. Together, these results suggest that eye movement patterns during naturalistic encoding and retrieval may not clearly dissociate along the lines of recollection and familiarity.

The current results indicate that both familiarity and recollection were related to the dispersion of viewing and resampling of previously studied scene regions. Future work, however, will be needed to clarify the precise roles that these eye movement patterns play in memory, particularly during retrieval. For example, the eye movements observed during the retrieval phase could reflect either processes facilitating memory retrieval (e.g., resampling studied regions in a scene may increase study-test similarity and so may facilitate retrieval; Wynn et al. 2016), or they could reflect decision processes that occur after the memory is retrieved (e.g., increased resampling of studied regions could reflect evaluation of the strength or the quality of the specific details that were retrieved from memory; Holm and Mantyla 2007). Prior work has indicated that limiting eye movements during retrieval can significantly reduce recollection-based memory responses (Mantyla & Holm, 2006; Schwedes, Scherer, & Wentura, 2019; Schwedes & Wentura, 2019), suggesting that the presently observed eye movements may have been involved in facilitating recollection; however, we cannot rule out the possibility that they may have reflected additional post-retrieval processes as well. Similarly, the eye movement effects related to familiarity could also reflect pre-retrieval and/or post-retrieval processes. For example, there is evidence that familiarity is related to perceptual fluency, such that ease of identification of stimuli contributes to increased familiarity for those stimuli (Whittlesea, 1993). One possible explanation for the observed relationship between eye movements and familiarity in the present study, therefore, is that attention focused on previously visited regions during retrieval reflects more fluent visual processing and leads to the subjective experience of familiarity. But another possibility is that the eye movement effects primarily reflect post-retrieval processing driven by familiarity. Teasing apart these possibilities may be particularly useful in furthering our understanding of the interplay of attention and memory.

Given that eye movement changes resulting from experience are frequently considered to be underpinned by memory that occurs outside of awareness (e.g., Hannula, 2010; Hannula & Ranganath, 2009; Ryan, Althoff, Whitlow, & Cohen, 2000), it is perhaps surprising that we found no evidence that eye movements were related to unconscious memory in the present studies. One possible explanation for this discrepancy is that we simply lacked the statistical power to detect unconscious memory effects. However, this explanation seems unlikely for a number of reasons. First, even though power is a concern with a nonsignificant standard hypothesis test, the Bayes factors that we achieved in both experiments met the convention for confidence that the null results were not due to lack of power, but instead were more likely to have occurred under the null hypothesis. Second, we doubled the sample size in Experiment 2 such that we had more than 98% power to detect previously obtained effects of memory on eye movements—but again found no evidence for a difference between new and missed old scenes. Third, we ran an additional analysis collapsing “sure new” and “maybe new” trials into a single “all misses” category, thereby doubling the number of trials included in the analysis, and this further verified the results of the initial analysis (see Appendix). Fourth, inspection of the numerical differences between missed and new scenes in both Experiments 1 and 2 indicated that the results were reversed from what would be expected of an unconscious memory effect, given prior findings. Another potential explanation for the lack of effect is that unconscious memory may influence some eye movement measures, but not the specific eye movement measures we examined in the current study. However, we did examine a variety of measures (i.e., dispersion, number of fixations, saccade amplitude, number of regions visited), including those that have previously been thought to relate to unconscious memory (i.e., number of fixations; Althoff & Cohen, 1999; Ryan et al., 2000), but none of them showed any relation with unconscious memory. The present findings, therefore, may be useful in informing the debate surrounding whether changes in oculomotor behavior and visual attention at retrieval are related to conscious or unconscious memory: When unconscious memory was isolated from confounding influences of weak conscious memory, we found only conscious effects and substantial evidence against unconscious effects.

Despite the evidence for a lack of unconscious effects in this paradigm, it is possible that there are other tasks that are more appropriate for detecting unconscious memory effects. As one example, in a prior study we found that during memory-guided search (i.e., contextual cueing, which is an implicit memory task), the overall efficiency of participants’ scanpaths was influenced by unconscious memory, but not recollection or familiarity (Ramey et al., 2019). Thus, one possibility warranting further investigation is that unconscious memory effects on attention may be suppressed or masked under certain explicit retrieval conditions, such as those used in the present study. Although this is a relatively unexplored area, there is evidence that conscious and unconscious memory can compete for expression in some circumstances (Henson, Shallice, Gorno-Tempini, & Dolan, 2002). For example, in an implicit test of memory for faces (i.e., fame judgments), Henson and colleagues (2002) found that activity in the fusiform gyrus was reduced for faces that had been studied earlier, which is considered a marker of neural priming. In an explicit version of the task (i.e., recognition memory test), however, the neural priming effects were no longer observed but were replaced by medial temporal lobe activity—a marker of conscious, explicit memory. Thus, the current results should not be interpreted as ruling out the possibility that unconscious memory may impact some types of eye movements under some conditions, but they do suggest that under explicit retrieval conditions, eye movements and the allocation of visual attention are more tightly coupled with conscious memory.

The finding of a robust relation between familiarity strength and eye movement behavior has important implications for understanding the relationship between memory and attention more broadly. For example, building on prior results showing that making more fixations during encoding predicts better subsequent memory (Kafkas & Montaldi, 2011; Loftus, 1972), the present results suggest that dispersing those fixations broadly across a stimulus also predicts improved memory—regardless of the task performed during encoding (i.e., memorization or aesthetic judgment). Perhaps surprisingly, our findings at retrieval suggest that instead of widely distributing attention in a similar fashion to encoding, the opposite pattern is optimal: restricting fixations to a more constrained area of the stimulus at retrieval is related to better memory. Moreover, those constrained fixation patterns may function to focus attention on regions that were previously visited—as suggested by the relation between increased resampling of previously viewed regions (and therefore decreased sampling of new regions) and overall memory strength. Together, these results are broadly consistent with an account of attentional deployment wherein attention is widely distributed at encoding to facilitate maximal sampling of information, and focused in on the most relevant regions at retrieval to facilitate comparisons with internal memory representations.

The presently developed measures of dispersion and resampling may prove to be particularly useful new tools for understanding the relationship between memory and attention. First, the present resampling measure provides a continuous metric of resampling behavior that does not require arbitrary cutoffs, whereas past assessments of memory processes and resampling behavior have all utilized categorical measures (i.e., the proportion of the first 3 test fixations that were within 2 degrees of study fixations; Holm & Mantyla 2006; Holm & Mantyla 2007). Second, dispersion demonstrated a more consistent relationship with familiarity than did previously used measures such as number of fixations and saccade amplitude: Whereas dispersion was strongly related to familiarity strength in both experiments at both encoding and retrieval, the number of fixations and saccade amplitude were not reliable across experiments (Table 2), and dispersion outperformed prior measures in a model comparison (Appendix). Furthermore, dispersion was not influenced by subject and image effects, whereas the relationship between saccade amplitude and recollection was reversed depending on if subject effects were controlled for—suggesting that dispersion is more robust across statistical methods. The current measure of dispersion also has clear potential for applications beyond memory research, to questions of visual attention more broadly. For example, examining cluster-based dispersion may be useful for understanding the processes involved in perceptual discriminations such as change detection, given that change-related differences have been found in converging eye movement measures such as saccade amplitude (Henderson & Hollingworth, 2003). Furthermore, prior work indicates that saccade amplitudes and the number of fixations vary between emotional and neutral scenes (Bradley et al., 2011), and dispersion may provide a more robust, sensitive method of examining differences in attentional distribution for emotional stimuli. Therefore, given its stability and potential for applications beyond memory research, dispersion may prove to be a sensitive new index for visual attentional deployment that could be useful for attention researchers in general.

The present results may also have important implications for longstanding theoretical debates in both attention and scene memory, particularly with respect to the importance of stimulus properties versus cognitive factors in the control of attention and memory for scenes (e.g., Henderson, 2007; Van der Stigchel et al., 2009). For example, much of the research into how attention is controlled has been focused on predicting attention by quantifying various scene properties, such as the salience (Itti & Koch, 2000; Itti, Koch, & Niebur, 1998) and semantics (Henderson & Hayes, 2017, 2018) of different scene regions. Some dominant theories have even assumed that attention is controlled primarily by bottom-up visual features, with cognitive factors serving only to modulate the prioritization of visually salient regions (Henderson, 2007; Itti & Koch, 2001; Tatler, Hayhoe, Land, & Ballard, 2011). In contrast, a growing body of research has indicated that cognitive factors unrelated to scene properties, such as task goals, can guide attention independently of salience (Henderson, 2003, 2007; Tatler et al., 2011). The role of episodic memory as a source of cognitive guidance, however, has not yet been well-defined—and the present results suggest that subjective familiarity strength may comprise a unique form of attentional guidance warranting further investigation. In a similar vein, much research has been devoted to examining memory for scenes in terms of scene memorability: the intrinsic aspects of scenes that tend to elicit better memory (e.g., Bainbridge, Hall, & Baker, 2019). For example, some findings have indicated that scenes containing certain features such as faces tend to be more memorable (e.g., Isola, Parikh, Torralba, & Oliva, 2011), and that scenes eliciting greater consistency between different participants’ viewing patterns tend to be more memorable (Mancas & Le Meur, 2013). In fact, the effects obtained in prior studies of memory and visual attention (e.g., Kafkas & Montaldi, 2011; Loftus, 1972; Sharot et al., 2008) could perhaps reflect differences in image properties: For example, certain scenes may be more memorable and also elicit more fixations, both due to inherent scene properties (e.g., having many interesting regions), thus leading to the observed relation between memory and an increased number of fixations. Given that we controlled for image effects, however, the present findings may be uniquely poised to contribute to our understanding of how scenes are remembered independently from scene properties. Specifically, the present findings are the first to our knowledge to point to a robust role of visual attention in scene memory strength that is disentangled from the influence of image properties.

Both the results and newly developed methods of the present study may prove useful in motivating future investigations in both attention and memory research. For example, the present findings combined with prior evidence for eye movement measures that separately index recollection and unconscious memory during search (Ramey et al., 2019) suggest that recollection, familiarity, and unconscious memory can each be indexed using eyetracking, which may be particularly useful for assessing memory processes in nonverbal and patient populations. Additionally, given the sensitivity of the presently developed measure of attentional dispersion to familiarity strength—and the fact that many cognitive processes may indirectly support memory (e.g., Johnson, 1992)—future research aimed at determining whether dispersion of attention might also index phenomena such as cognitive load, mind-wandering, or aspects of executive functioning may prove fruitful. Finally, if causal investigations show that manipulating attentional dispersion can increase familiarity strength, a dispersion-based intervention could perhaps be fruitfully applied to improve learning in a real-world setting.

Context

How we view the world is intimately related to how we remember it, but characterizing the specific viewing behavior and memory processes involved in this relationship has proven to be a complex undertaking, poised at the interface of two fields. The present study emerged out of a new, interdisciplinary collaboration that aims to characterize memory processes and eye movement behavior. In prior work, we found that recollection and unconscious memory outside of awareness were selectively related to distinct patterns of search-related eye movements during memory-guided search. The current findings, on the other hand, outline a clear role for familiarity strength in multiple eye movement patterns during naturalistic viewing. Taken together, these results indicate that recollection, familiarity, and unconscious memory are each related to visual attention and eye movement behavior, but the specific eye movement patterns examined and the manner in which memory is deployed (i.e., search versus recognition) are key determinants of which memory processes are involved.

Fig. 4.

Degree of resampling of scene regions between study and test by memory response. Resampling is given as the correlation between the fixation density maps at study and test. Estimated marginal means controlling for participant and image are plotted, and the error bars represent the standard error of these estimated means from the model. A) Resampling in Experiment 1. B) Resampling in Experiment 2.

Appendix

Post-hoc Analyses

Model Comparison.

To provide a direct test of the sensitivity of the presently developed measures of visual attention (i.e., dispersion and resampling) above and beyond previously used oculomotor measures (i.e., number of fixations and saccade amplitude) in predicting memory, we conducted a model comparison. Specifically, we determined whether adding dispersion and resampling predictors to a model regressing familiarity strength on number of fixations and saccade amplitude provided a better fit to the data. We found that adding dispersion and resampling produced a superior model in both experiments, at both encoding and retrieval, ps<.0001.

Experiment Effects.

To determine whether the differences between experiments (i.e., memory accuracy, and the use of a different set of scenes) led to differences in the relation between memory and the eye movement measures, we compiled both experiments into one data set and re-ran the analyses. Every significant effect obtained in either of the experiments alone replicated to ps<.007 in the combined data set: dispersion predicted familiarity strength and recollection during encoding and retrieval, and resampling predicted familiarity strength and recollection. When a covariate of the experiment to which each data point belonged was included in the model, all of the effects held; this covariate of experiment was not significant in any contrast, ps>.07. There were also no interactions between experiment of origin and memory in predicting the eye movement measures, ps>.05.

Furthermore, one possible concern is the addition of outdoor scenes in Experiment 2, which were included to increase the distinctiveness of the stimuli and therefore the proportion of high-confidence memory responses, whereas only indoor scenes were used in Experiment 1. In analyses of data from Experiment 2, we found that the indoor versus outdoor status of a scene did not interact with memory in predicting any of the eye movement effects. Together, these analyses suggest that the results were robust across different scenes and accuracy rates.

Collapsing all Misses.

It is possible that the lack of effect of unconscious memory on dispersion could be related to the strict criterion we used for the unconscious memory contrast (i.e., “sure new” scenes, that is, high-confidence misses). Therefore, to ensure that this was not responsible for the null effects, we collapsed “sure new” and “maybe new” responses into a single “new” response and re-ran the retrieval dispersion analysis. There was still no effect in Experiment 1, t(921)= 0.54, p=0.59, d=0.04, or Experiment 2, t(2300)= 1.56, p=0.12, d=0.07. Thus, even with approximately double the number of trials included in the analysis and a more lax definition of misses, there was no effect of unconscious memory on dispersion.

Correlations Between Eye Movement Measures.

Both number of fixations and saccade amplitude have previously been interpreted at times to reflect spatial distribution and clustering of eye movements during viewing, but neither measure takes spatial information into account. To determine if these measures may serve as a proxy for spatial distribution, we assessed their correlation with each other and the presently developed measure of dispersion using the combined data from both experiments (Table A1). Interestingly, despite the fact that both measures have been interpreted to represent spatial distribution of eye movements, saccade amplitude and number of fixations were only weakly associated, r = .09, p <.001. However, both measures were moderately correlated with dispersion, rs > .41, ps <.001 (Table A1). Moreover, when number of fixations and saccade amplitude were submitted to a principal component analysis, the first principal component exhibited a stronger correlation with dispersion than either measure did individually, r = .87. This suggests that both number of fixations and saccade amplitude do contain non-overlapping subsets of information about spatial distribution, such that they converge towards describing spatial distribution when combined, but they do not provide a complete picture of how widely viewing is distributed across a stimulus.

Table A1.

Pearson Correlations Between Measures Related to Dispersion, Across Both Experiments

	# fixations	Sac. amplitude	# regions	Dispersion
# fixations	1	.09	.71	.45
Sac. amplitude	.09	1	.33	.41
# regions	.71	.33	1	.50
Dispersion	.45	.41	.50	1

Note. All correlations were significant to p <.0001. Sac. amplitude = saccade amplitude.

Additional Analyses.

To increase the interface of the present findings with the memory and visual attention literature in general, we ran supplemental analyses. First, to confirm the effect of memory on dispersion, we examined whether dispersion significantly differed between study and test as well as between old and new scenes at test. For old scenes, dispersion was lower at test than at study in Experiment 1, t(6366)= −3.42, p<.001, d= −0.09, and Experiment 2, t(13218)= −21.44, p<.0001, d= −0.37. Similarly, at test, dispersion was significantly lower for old than new scenes in Experiment 1, t(4065)= −4.07, p<.0001, d= −0.13, and Experiment 2, t(8654)= −8.6, p<.0001, d= −0.18. Second, average fixation duration was examined. During encoding, fixation duration was related to familiarity in Experiment 1, β = −0.082, t(2148)= −3.92, p<.0001, and Experiment 2, β = −0.088, t(3633)= −5.22, p<.0001. Fixation duration during encoding was also related to recollection, albeit weakly, in Experiment 1, t(1743)= −2.05, p= .041, d= −0.1, and Experiment 2, t(4761)= −2.15, p= .032, d= −0.06. At retrieval, fixation duration was not related to familiarity in Experiment 1 or Experiment 2, ps>.13. Fixation duration during retrieval was not related to recollection in Experiment 1, t(1730)= 1.34, p= .18, d= 0.06, but it was in Experiment 2, t(4771)= 3.54, p<.001, d= 0.1. There were no unconscious effects on fixation duration, ps>.44.

Additional Measures

Additional measures were collected, largely as fillers, that were not analyzed. As part of the delay between the study and test phase (Fig. 1b), participants filled out simple questionnaires as filler measures (i.e., 24-hour food recall, perceived stress scale, a short ADHD symptom questionnaire, and a short form personality scale). Additionally, after the recognition probe, participants gave source judgments for each scene indicating which study task they initially viewed the scene in. The source judgments were included to address a question that does not pertain to the topic of the present paper. They were not analyzed, but we plan to analyze and report them in a separate manuscript in conjunction with the results of a separate experiment.