Looking for the neural basis of memory

Abstract

Memory neuroscientists often measure neural activity during task trials designed to recruit specific memory processes. Behavior is championed as crucial for deciphering brain-memory linkages yet is impoverished in typical experiments that rely on summary judgments. We criticize this approach as being blind to the multiple cognitive, neural, and behavioral processes that occur rapidly within a trial to support memory. Instead, time-resolved behaviors such as eye movements occur at the speed of cognition and neural activity. We highlight successes using eye-movement tracking with in vivo electrophysiology to link rapid hippocampal oscillations to encoding and retrieval processes that interact over hundreds of milliseconds. This approach will improve research on the neural basis of memory as it pinpoints discrete moments of brain-behavior-cognition correspondence.

The challenge of merging brain and behavior

It is common in neuroscience to champion behavior, especially when trying to understand the neural basis of cognition [1-4]. However, not all behaviors are on equal footing. This is because cognition and behavior are not always simultaneous, and cognitive processes of interest are not always linked to behaviors that are suitable for identifying corresponding neural activity. Such disconnects have led some to question the very utility of behavior [5]. Here, we emphasize the importance of behavior in the cognitive neuroscience of episodic memory but argue that useful behaviors are those that can be directly related to cognitive and neural processing. We assert that naturalistic eye movements are ideal because they occur on the same timescale as many neural processing events of interest and often reflect moment-to-moment changes in memory processing. We highlight recent successes using eye-movement tracking in conjunction with in vivo electrophysiology. This approach has resolved key neurocognitive processing events that have eluded previous memory experiments lacking the temporal resolution of eye movements. We hope that articulating this theoretical opinion advances experimental frameworks to test the cognitive relevance of transient neural events whose function has evaded discovery by typical experimental techniques.

Processing too fast to see in typical memory experiments

Cognitive neuroscience has been pivotal for understanding the neural basis of episodic memory (see Glossary). A typical experimental approach has been as follows. During study, encoding occurs when novel stimuli are presented, and memories are formed. During test, retrieval occurs when probes are presented, and memories are reactivated. Neural signals of encoding are revealed by comparing neural activity for stimuli that are subsequently remembered versus forgotten. Similarly, comparisons among trials that vary in memory status at test identify neural correlates of retrieval. Experiments seldom focus on consolidation during the delay period. Although others have highlighted limitations of this trial-based experimental approach [6], here we focus on the specific challenges concerning episodic memory.

This logic of separately inspecting encoding and retrieval trials to identify relevant neural correlates is often criticized because such trials are not “process pure”. For instance, although stimuli may be novel in the context of the experiment during study when encoding occurs, retrieval can occur simultaneously for stimulus information learned before the experiment (e.g., semantic memory for meaningful stimuli such as words or objects and/or episodic memory for experiences with conceptually similar stimuli [7]).

However, here we offer the novel point that “process impurity” can occur even within a trial, as multiple, rapid, and ongoing cognitive and neural processes interact. Relative to cognition and brain activity, typical experimental trials often last many seconds during which many processing operations can occur, even when the trial is comprised of only a single, static stimulus. For instance, retrieval can occur within a given task trial for information learned earlier in the trial (Figure 1A, Key Figure). Because binding previously encoded information to the currently attended stimulus determines whether and how it will be remembered [8], such retrieval from earlier in the trial can, in turn, influence encoding and subsequent behavior.

Key Figure, Figure 1. Eye movements as real-time indicators of memory processing and neurophysiological activity.

(A) During a study trial, an individual makes a sequence of saccades (red arrows) to fixate (red circles) on content within a visual scene. In this conceptual example, when a participant views the white succulent (fixation 3), retrieval of previously viewed content (the latte, fixation 2) causes the gaze to return to the previously viewed location (fixation 4). This eye movement thus provides a highly precise marker of when retrieval occurred during the study trial, and this temporal precision can be used to identify a corresponding rapid neurophysiological event (a sharp-wave ripple (SWR) shown in the lower trace). (B) During a test trial, memory guides visual exploration to recognize the scene. The first two fixations are reinstated in the same sequence as during the study episode, recalled from memory. Viewing of previously unexplored content (fixations 3 and 4) allows integration of new content into memory for the scene, thus reflecting encoding during the test trial. The temporal precision of these eye movements allows for identification of corresponding neurophysiological activity (a bout of theta oscillations, shown in the lower trace).

Likewise, there are many opportunities for encoding during test trials, especially for complex stimuli that were only partially encoded when initially presented during study (Figure 1B). Indeed, the idea that memory formation rapidly co-occurs with retrieval is well recognized. If encoding and retrieval could not co-occur, it would be impossible to later remember episodes that involved memory retrieval. During memory tests, subjects can accurately report information learned during test trials, revealing encoding processes that occur at retrieval [9, 10]. When experimenters ignore the complex timing of these processes by focusing on trials in aggregate, encoding and retrieval blur together, which is perhaps why they often appear to have essentially the same neural correlates [11, 12] despite involving both shared and distinct cognitive and neural processes. Indeed, it is well recognized that updated memories are formed by integrating novel and previously learned information [13]. Without the ability to temporally resolve these processes, it may be difficult if not impossible to unravel their neural basis.

Neuronal processes also occur at timescales far faster than the typical trial timecourse. For example, hippocampal sharp-wave ripples (SWRs) are prominent in rodent models of memory [14] (Box 1). SWRs are believed to support recollection by driving synchronous replay of activity in the cortex. Intracranial EEG in humans can measure ripple oscillations [15], with coupling between ripples in the medial and lateral temporal cortex before spontaneous recollection [16]. Furthermore, during encoding of word pairs (Figure 2A), sequences of single-unit activity recorded from the anterior temporal lobe co-occurred with local ripples and coded for specific memories [17]. These temporal sequences were reinstated before recall. From these data, it is tempting to conclude that ripple oscillations reflect retrieval.

Box 1: Electrophysiological states of the hippocampal circuit.

SWRs are well established in animal models (for a review, see [14]), with later work characterizing potential homologs in humans [105-108]. Recently, cognitive studies in humans have examined hippocampal SWRs and cortical ripples during memory encoding and retrieval [16-18] and implicate a role of ripples in the recall of visual memories. However, ripple oscillations are observed across many task states in humans [17, 18, 109] and nonhuman primates [46]. The notion that SWRs support retrieval/consolidation relies on the assumptions held above, which may or may not be valid. Thus, the functional relevance of SWRs is unclear. Studies in rodents suggest that SWRs are spontaneously generated in CA3, driving synchronous activity throughout the hippocampal circuit, driving replay in cortical networks to support retrieval and consolidation [110, 111].

Neuronal firing across the hippocampal circuit is also shaped by theta oscillations, four to eight hertz rhythms that modulate plasticity and facilitate communication through synchronous neuronal firing [112]. The theta rhythm has been shown to coordinate distinct network states in the hippocampus, with maximal entorhinal inputs to CA1 at the theta trough and CA3 inputs at the theta peak [49]. Thus, phases of the theta rhythm determine when processing in the CA1 is dominated by external inputs (i.e. supports encoding) or by pattern completion processes in CA3 (i.e. supports retrieval). Distinct phases of theta also modulate the amplitude of distinct high and low gamma bands associated with retrieval and encoding states, respectively [113-117]. It is possible to identify these distinct states within individual theta cycles [118], affording the hippocampal circuit the ability to rapidly (i.e., within hundreds of milliseconds) switch between encoding and retrieval modes.

These two states generally exhibit an antagonistic relationship in rodents. Waking SWRs and theta preferentially occur during attentive stillness and volitional exploration, respectively [14]. Acetylcholine differentially affects both processes, as cholinergic stimulation blocks sharp-wave ripples while enhancing theta band power and synchrony within the hippocampus [119].

Figure 2. Do SWR events reflect retrieval during both study and test?

(A) In [17], participants encoded word pairs during 4-sec study trials (top) and recalled the paired associate given one word as a cue during test trials (bottom). Sequentially organized single-unit activity occurred during study and these sequences were replayed just before accurate recall, locked to ripple events observable in the local field potential. However, ripple events occur multiple times during both study and test, as shown, with comparable rates across both task periods. Do these ripples reflect retrieval and, if so, why are they so prevalent during the study trial? (B) Hypothetical eye movements during the trial in A could reveal when memory processing has occurred within the trial. (C) In [47], participants performed a target-detection task in which they learned to fixate on a specific object within each scene. When viewing repeated scenes (left), fixations started in a random location and moved nearer the target over the course of the trial. SWR events (shown in red) became more likely as fixations approached the target location on repeated as compared to novel trials (middle). This indicates that SWRs reflected the guidance of visual exploration by memory retrieval (right). Figures modified with permission from [17] and [47].

However, functional interpretation of SWRs is complicated because they persist for only a few hundred milliseconds and cognition is not typically measured at this timescale. Many SWRs occur during typical study and/or test trials. For example, in a recent experiment [18], hippocampal SWRs were recorded while participants performed a free-recall task. The SWR rate increased before spontaneous recall, consistent with their hypothesized role in retrieval. However, in the same study, SWRs were also prevalent during the encoding of novel stimuli. SWR rates nearly doubled when viewing novel stimuli compared to recall, and SWR rates following encoding predicted subsequent recall.

Do multiple SWRs during an ostensible “encoding” period indicate that the association between SWRs and retrieval is incorrect? Alternatively, do these findings suggest that retrieval happened multiple times during study trials? Indeed, retrieval is an excellent learning mechanism [19] and therefore could support memory formation. In either case, the methods were not sufficient for disambiguating function because there were no sub-trial indicators of cognitive processing concurrent with ripples. Similar ambiguity exists in studies of SWRs in rodents [5]. Although rodent SWRs may predict the outcomes of decision-making processes, these outcomes are temporally disconnected from behavioral and cognitive states during SWRs.

Similar uncertainty clouds the interpretation of hippocampal theta oscillations (Box 1), which typically persist for only a few cycles in humans (approximately 350 to 800 milliseconds) [20]. Increased theta is linked to associative memory encoding [21-23] and retrieval [21]. It has been argued that when trial-level comparisons do not emphasize associative processing, decreased theta indicates encoding [24]. These conclusions overlook the conditions necessary to produce theta oscillations, such as demands for volitional exploration [22, 25]. We argue that eye tracking can reveal the rapid cognitive processes involved in generating these behaviors and thereby identify the functional relevance of hippocampal theta.

Real-time memory tracking

When viewing a static image, saccades reposition the fovea on average every 200 to 300 milliseconds (see [26] for representative data across a variety of tasks) – a timescale similar to electrophysiological events in the hippocampus (Figure 1). Cognitive processes influence both the frequency and pattern of eye movements and hippocampal activity alike [27, 28]. Furthermore, eye movements can indicate memory encoding and retrieval. Viewing novel as compared to repeated scenes increases saccade frequency, with less time processing inputs during fixations [29]. The influence of memory on eye movements often occurs before [30] or in the absence of accurate recognition judgments [31], with effects as early as the first fixation [32]. Because eye movements can reflect memory at the speed of individual fixations, we argue they provide an ideal means to link cognitive and neural processing.

Eye movements are useful due to their speed and because they can identify when specific cognitive processing events are likely to occur. Eye movements can reveal relational memory processing when the gaze moves between previously related (as opposed to unrelated or perceptually similar) stimuli [33]. Relational processing can also be identified in memory tasks where scene content is manipulated during testing. Saccades to manipulated content indicate intact relational memory and depend upon the hippocampus [29, 34]. Spatiotemporal memory is reflected when subjects reinstate at test the sequence of eye movements made during study [35-37]. More generally, eye movements can indicate when retrieval of spatial information has occurred as saccades position the gaze to locations based on memory [38]. By using partial cues to evoke memories, eye tracking can pinpoint pattern completion, even when it causes false memories to perceptually similar lures [39].

Determining whether and when a memory process is engaged relative to an observed eye movement warrants consideration. To make such an inference, one must assume a correspondence between the gaze position and ongoing memory processing. One may assume that encoding occurs during fixations to previously unviewed content within a scene. Similarly, in the case of memory-guided eye movements, we assume retrieval processes occur in the moments leading up to saccade execution. However, the simple act of viewing a previously attended location does not necessarily imply that a retrieval process has occurred. Re-fixating upon previously viewed content occurs under many circumstances [40]. For example, refixations may result from optimizing task performance with high working memory demands [41], or rehearsal in visual working memory [42]. Thus, rather than assuming a specific process has occurred based on the location of a fixation, predictions linking specific types of eye movements, neural activity, and subsequent behavior must be tested.

Eye movements can reveal cognitive states present from moment to moment, on the timescale of hundreds of milliseconds. This unusually specific behavioral marker of cognition is advantageous because behavior provides a timestamp to identify related neural processing. Contrast this with how behavior is typically measured in memory experiments, as a judgment that is not necessarily synchronized with the relevant processing, typically made at the end of a trial of arbitrary duration. Few such judgments can be made per trial, thus providing relatively coarse information on the many cognitive processing operations that occurred throughout the trial. Eye movements are advantageous in providing temporally precise behavioral readouts of ongoing memory-related cognition for use in neural timeseries analysis.

Revealing the neural basis of memory with eye movements

Fully capitalizing on eye movements requires that they be used in conjunction with equally precise measures of neural activity. For instance, eye movements modulate hippocampal activity measured indirectly with fMRI, both in the amount [28] and type [43] of visual exploration during encoding and retrieval [44]. However, the BOLD response temporally filters neural activity, so experiments often aggregate eye movements across the trial to relate to trial-level activity estimates. fMRI can therefore miss neural activity associated with specific eye movements within the trial that reflect discrete memory processing events. In general, trial-level fMRI activity correlates of memory will include some relatively unknown amount of signal related to eye movements and their corresponding cognitive operations [27].

In contrast, recent experiments have combined eye tracking with the high temporal and spatial resolution of invasive electrophysiology. In one experiment, eye movements were monitored during a scene memory task while electrophysiological activity was recorded from the hippocampus of nonhuman primates [45]. During study, the first saccade to novel visual content produced a reset of ongoing theta oscillations in the hippocampus. The consistency of this effect predicted later scene recognition. Thus, encoding of visual information in hippocampal circuits is tightly linked to sampling behaviors, and the coordination of hippocampal activity with dynamic behavioral events affects memory.

Eye movements can identify the neural mechanisms of retrieval by providing the precise timing of retrieval events. A series of studies have identified exploratory SWRs (as opposed to those observed during rest or sleep) during visual search in nonhuman primates [46, 47]. In these experiments, target objects were selected from nontargets embedded within a visual scene. By tracking the gaze throughout extended search trials, shifts of attention to the target location indicated when retrieval was likely to occur (Figure 2B). During search, target fixations concurrent with SWRs predicted successful target detection, whereas SWR rates and fixations to the target alone did not [46], and ripple rates increased with proximity to search targets [47]. Thus, exploratory SWRs are time-locked to memory processes that support retrieval of current task goals (i.e., the target object). A key strength of these studies is that they use discrete eye movements to focus in on cognitively relevant periods and uncover the cognitive relevance of SWRs.

Eye tracking can resolve when encoding and retrieval operations co-occur within the span of a single trial. In one experiment with human subjects, eye tracking was used to determine when encoding and retrieval occurred during a spatial memory task that simultaneously taxed both processes [48]. Subjects encoded the location of objects presented on a visual scene. During a memory test, objects were re-presented in a location that either matched or mismatched the original location, and subjects indicated whether the object was in a novel location. During mismatch trials, subjects not only fixated on the object at the novel location (reflecting stimulus encoding as measured on a subsequent memory test) but also increased fixations to the original location, indicating memory retrieval had occurred. Both fixation types were tightly linked to hippocampal theta oscillations. As in nonhuman primates [45], saccades to novel content caused a reset in ongoing theta, which coherently persisted during subsequent encoding. In contrast, retrieval-guided fixations were phase-locked to theta in the moments leading up to their execution (Figure 3A). Phase-locking associated with these two types of eye movements occurred at distinct phases of theta, as expected if it coordinates when encoding and retrieval operations occur [49]. Furthermore, theta phase-amplitude coupling (PAC) to relatively low vs. high gamma was associated with eye movements that reflected retrieval vs. encoding, consistent with rodent data showing temporal coordination of these processes by theta [50]. Iterative encoding and retrieval processes thus occur within the span of a single trial, with distinct phases of theta associated with unique gamma states and eye movements.

Figure 3. Eye movements indicate dynamically changing encoding and retrieval processes synchronized to hippocampal theta oscillations.

(A) The phase of theta reflects when encoding and retrieval occur when perceptual information conflicts with information in memory. In [48], subjects studied objects at certain scene locations, and the locations were updated/changed at test. Following fixations to the updated location (green), ongoing theta oscillations reset such the trough of theta occurred during fixations to novel content. In contrast, when fixations were directed to the original location of the object (i.e., guided from memory, purple) theta oscillations were consistently aligned to the peak of theta just before the fixation. (B) During free viewing of scenes in [51], participants revisited previously viewed locations (depicted as returning the gaze to the coffee mug after already viewing it during study) based on short-term retrieval. Just before these revisitation fixations, the prevalence of theta oscillations markedly decreased, indicating a neural signature of retrieval. In contrast, theta oscillations were more prevalent during encoding of these locations (i.e., following the onset of the fixation). (C) These findings suggest that when scene content must be compared to memory, encoding and retrieval processes occur on distinct theta phases (left). Without conflicting mnemonic information, the presence or absence of theta bursts reflect distinct processing states. Figures modified with permission from [48] and [51].

Given the sequential nature of eye movements, neural responses to one eye movement can contaminate surrounding behavioral events of interest. When viewing familiar scenes, fixations to novel content often precede retrieval-guided fixations [48]. As a result, encoding and retrieval processes overlap in time with potentially co-occurring neural responses. To circumvent this issue, researchers often focus on eye movements preceded or followed by periods devoid of saccades [45, 48] or with saccades that are matched between experimental conditions of interest [48, 51]. Deconvolution-based techniques make it possible to construct fixation-related potentials even when eye movements lead to overlapping neural responses [52, 53]. Similar fixation-related changes in neural activity can be used with fMRI [54-56]. However, given the sluggishness of the BOLD response combined with the rapid and sequential nature of eye movements, power to detect relevant neural activity is challenging in structures with lower signal such as the hippocampus [57]. Thus, this potential interpretive limitation is best addressed via the rapid temporal resolution of in vivo electrophysiology.

A clearer view of memory reinstatement

Eye movements provide rich spatiotemporal information that can reveal the quality of memory processing. Studies of hippocampal function typically focus on questions of how spatial and temporal information is bound into memory representations [58-60]. Because vision is a primary form of exploration in primates [61], eye movements are well-suited to these research questions because they translate continuous visual inputs into discrete sequences associated with specific spatial locations. Eye-movement behaviors thus provide a means of tracking the spatiotemporal information in memory during both encoding and retrieval.

It is widely held that memory retrieval involves the reinstatement of neural activity present during initial encoding. Indeed, functional neuroimaging studies have shown hippocampal activity at encoding and retrieval predicts both item-specific and episode-specific reinstatement of individual memories [62-65]. However, experiments have not typically considered behavioral reinstatement, such as in patterns of eye movements, that likely accompanies neural reinstatement. Theories of visual memory propose that the sequence of eye movements becomes incorporated into memory during encoding, and memory recall involves reinstatement of the same sequence of eye movements [66, 67]. Recent studies provide behavioral evidence that successful scene retrieval is marked by such reinstatement [35-37], making reinstatement of fixation sequences a behavioral measure of memory reinstatement. This raises a key interpretive challenge for previous studies of neural pattern reinstatement, as neural reinstatement could have been secondary to behavioral reinstatement.

This issue is exemplified in a recent study wherein subjects performed a memory task in which they imagined scenes from either long-term or short-term memory [68]. Simultaneous eye tracking and fMRI revealed reinstatement of neural activity and fixations when imagining visual content from both long-term and short-term memory. Further, neural reinstatement positively correlated with fixation reinstatement, indicating that eye movements may be functionally relevant in generating neural reactivation. However, the relative order of neural and fixation reinstatement cannot be resolved via fMRI because of its temporal sluggishness. Thus, neural activity measures with better temporal resolution are needed to determine whether for a given brain region such as the hippocampus, neural reinstatement is a cause of eye-movement reinstatement, a consequence of it, or whether neural and eye-movement reinstatement are mutually influential as part of an interactive retrieval process.

A comprehensive account of memory reinstatement includes understanding the neurocognitive processing during memory formation that sets the stage for later reinstatement. In the context of eye-movement tracking, several findings indicate that active visual exploration is particularly effective for the creation of memories that are later reinstated. When individuals study complex material, short-term memory guides visual exploration such that viewing occasionally returns to re-sample previously viewed content after brief delays [69-71]. This revisitation eye-movement pattern is associated with successful memory formation and depends upon hippocampal function [71]. Revisitation could therefore serve as a temporally specific behavioral marker of hippocampal-guided eye movements, increasing sampling of information that is necessary for successful memory formation [72-74]. Under this account, one would expect that hippocampal retrieval-related activity would immediately precede (i.e., drive) revisitation eye movements whereas encoding-related activity would follow them, reflecting the beneficial effect of revisitation on memory formation. Network models suggest hippocampal activity can propagate to oculomotor systems within the duration of a single fixation [75], providing a mechanism by which memory can guide visual sampling at the level of individual fixations. Testing this hippocampal role in the interaction of retrieval and encoding as expressed through eye-movement behavior thus requires the ability to resolve neural processing across the hundreds of milliseconds over which revisitation transpires.

A recent study [51] measured human hippocampal electrophysiology associated with revisitation eye movements in a memory task using naturalistic scenes (Figure 3B). Revisitation during the initial viewing of scenes predicted reinstatement of the spatiotemporal sequence of eye movements during subsequent scene recognition. Revisitation fixations thus provide specific temporal markers of when short-term (i.e., within-trial) memory retrieval informed where to direct the gaze and thereby improved spatiotemporal memory encoding.

The role of the hippocampus in driving revisitation eye movements was indicated by a shift in hippocampal neurophysiology immediately before their occurrence. In the moments leading up to individual fixations, hippocampal theta predicted whether the next saccade would return the gaze to a previously viewed location (Figure 3B) or proceed to novel content. Before these revisitation fixations, hippocampal theta shifted towards a feedback regime, predicting upcoming theta activity across the visual network. The relative absence or presence of theta oscillations thus indicates when upcoming visual sampling is primarily driven by sensory inputs or retrieval from short-term memory. Encoding of scene content during revisitation fixations was different from other non-revisitation fixations, as theta bouts were markedly more prevalent during these fixations, potentially reflecting a mechanism by which foveated content can become integrated with a hippocampal representation of the scene. Theta oscillations in humans thus exhibit considerable variability throughout single free-viewing trials, reflective of concurrent demands for encoding and memory-guided exploration. Notably, a typical trial-level analysis approach would not have identified these reinstatement-specific encoding mechanisms, as the brief revisitation eye-movement events and their electrophysiological correlates would have been lost in the aggregate trial recording.

By integrating across these studies of SWR [46, 47] and theta [45, 48, 51] function, an understanding of how hippocampal processing supports distinct memory processes begins to emerge. Periods when retrieval guides visual exploration are marked with increases in SWRs and decreases in theta oscillations. In contrast, theta oscillations are prominent during visual exploration of novel content, especially for scene features that are later reinstated as part of a spatiotemporal sequence. Although hippocampal mechanisms for generating fixation-sequence reinstatement in humans have not yet been identified, SWRs are likely involved. Because of the speed of these neural processes, causally influencing the hippocampus and evaluating effects on eye-movement reinstatement will be essential in determining the functional role of the hippocampus in reinstatement.

Other real-time approaches

Other eye-tracking measures may be more directly related to specific cognitive processes. Pupil dilation tracks both arousal [76, 77] and locus coeruleus activity [78, 79] in real time, and has been associated with both memory load [80], as well as memory formation and retrieval [81-83]. Because multiple cognitive processes are reflected in the same physiological or behavioral outputs, it can be beneficial to monitor multiple component processes at once. For example, eye-movement tracking is necessary to disambiguate the neural basis of arousal from processes that direct attention to arousing or salient content [84-86]. These measures are particularly relevant for understanding the influence of the amygdala on memory formation, as it both modulates hippocampal function [85] and directs the gaze to salient objects in the environment [87].

Carefully controlled fixation-dependent designs offer advantages over free-viewing paradigms at the expense of ecological validity. For example, saccades are known to reset ongoing theta oscillations in the hippocampus [45, 88]. By time-locking stimulus presentation to fixation onsets, it is possible to assess whether visual processing fluctuates according to ongoing phases of theta oscillations [89, 90]. Converging evidence from scalp EEG [90] suggests perceptual cycles emerge from theta oscillations [91]. Similar theta-frequency dynamics emerge in measures of memory performance [92], making it possible to test theories of theta function even in the absence of direct hippocampal recordings. However, given multiple neural sources of theta, understanding the generators of such behavioral fluctuations requires concurrent observations of neural activity.

Memory-guided exploration is not limited to the visual domain. Navigation of complex environments, either virtual [93-95] or real [96], provides opportunities to track memory in real-time. However, because vision is central to primate exploration [61], eye-tracking can reveal critical components of navigation such as the contributions of visual landmarks [96, 97]. Even in rodents, head-scanning behaviors produce theta oscillations and single-trial encoding of place fields [98].

Computational models provide the ability to estimate emergent cognitive states from complex high-dimensional data including brain activity itself [2]. Such modeling can align dynamically changing high-level cognitive states to neural activity. For example, machine learning techniques have been used to differentiate whether memory encoding, retrieval, or integration occurs based on fMRI signals in the hippocampus and medial prefrontal cortex [99]. These same models have successfully been used to estimate cognitive states in an independent study [100], showing how integration during learning protects against interference from related memories. Incorporating eye movements and other real-time behavioral measures could improve the utility of this class of model, potentially yielding additional information about rapid cognitive dynamics that cannot easily be resolved with fMRI.

Concluding Remarks

We have described how complex task design and in-depth behavioral quantification can be leveraged to gain insights into the neurophysiology of episodic memory. Rather than purely observational or rigid, task-based designs (e.g., subsequent memory paradigms), we argue that future studies would benefit from characterizing behavioral dynamics that are specific indicators of cognitive function. Here, we have highlighted how eye movements have been used in this way, indicating when encoding and retrieval operations occur and providing specific behavioral markers of spatiotemporal information in memory. These studies are merely a starting point to understand the complex interplay between hippocampal processes and naturalistic, memory-guided behaviors. Many questions remain (see Outstanding Questions). Further, leveraging multiple real-time behavioral and physiological measures (e.g., pupil dilation [101, 102], facial expression [103], and olfaction [104]) could enhance identification of cognitive and neural processes involved in memory. Complex behaviors such as eye movements are not merely sources of noise that confound true cognitive processing but are rather an essential part of the cognitive repertoire produced by the interplay of neural systems [1, 6]. Progress in cognitive neuroscience requires understanding these behaviors and their neural origins.

Outstanding Questions.

1. How do discrete memory processing events revealed by eye movements relate to aggregate trial-level activity measured via methods such as fMRI?

2. In what circumstances are eye movements the cause versus the consequence of memory encoding and retrieval?

3. What brain regions are more involved in driving memory-related eye movements versus responding to the input provided by them?

4. What hippocampal mechanisms are primarily involved in reinstatement of eye movements and corresponding neural activity patterns?

5. Do different neural mechanisms support reinstatement from short-term and long-term memory?

6. Do theta oscillations play a causal role in the generation of eye-movement behaviors?

7. In what circumstances is retrieval reflected by decreased theta power versus phase coordination of neural activity and visual sampling?

Highlights.

• Eye movements reflect specific memory processing events and occur at the speed of cognition and neurophysiological activity.

• Eye movements therefore are better suited for identifying brain-behavior-cognition linkages in episodic memory than are coarse-grained behaviors.

• Invasive neurophysiological recordings in conjunction with eye-movement tasks have clarified the role of sharp-wave ripples and theta oscillations in episodic memory.

Glossary

Consolidation

The process by which encoded information is stabilized into a durable memory trace. In most experimental settings that focus on encoding or retrieval, short delays are often used to minimize the influence of consolidation

Encoding

The process by which sensory inputs and internal states are rapidly stored in memory

Episodic memory

The ability to recall past experiences, which can be used to inform memory judgments and to engage in activities such as planning or simulating the future. It relies on multiple cognitive processes including encoding, storage or consolidation, and retrieval

Gamma oscillations

High frequency (typically 30 – 120 Hz) synchronous neuronal activity observed in the local field potential

Reactivation

The process by which patterns of neural activity present during initial learning are activated again at a later point in time

Retrieval

The process by which previously encoded information is reactivated by a memory cue, allowing memory to influence behaviors such as eye, recognition judgments, or recall responses

Sharp-wave ripples

Highly synchronous neural complexes observed in the hippocampus consisting of strong deflection in the local field potential (sharp wave) and concurrent high-frequency oscillations (ripples, 80-140 Hz in humans)

Theta oscillations

Low frequency synchronous neuronal activity implicated in memory processing and long-range communication between brain regions. Distinct forms of theta occur along the hippocampal long axis in humans, with relatively slow theta (~3 Hz) in the anterior hippocampus and fast theta (~8 Hz) in the posterior hippocampus

Typical experimental approach

Most tasks contain three distinct phases to study the neural basis of memory: study, delay, and test. Memory tests vary on what cues are used to guide retrieval. On recognition tasks, the same stimuli are used as memory probes, and subjects must judge whether the stimulus is old or new. Cued recall tasks use a partial cue (e.g., one of two simultaneously presented stimuli) to trigger retrieval. Subjects are required to generate the missing associate. Free-recall tasks do not provide external retrieval cues. After encoding lists of information, subjects must use internally generated cues to report as many studied items as possible