Hierarchical process memory: memory as an integral component of information processing

Abstract

Models of working memory commonly focus on how information is encoded into and retrieved from storage at specific moments. However, in the majority of real-life processes, past information is used continuously to process incoming information across multiple timescales. Considering single unit, electrocorticography, and functional imaging data, we argue that (i) virtually all cortical circuits can accumulate information over time, and (ii) the timescales of accumulation vary hierarchically, from early sensory areas with short processing timescales (tens to hundreds of milliseconds) to higher-order areas with long processing timescales (many seconds to minutes). In this hierarchical systems perspective, memory is not restricted to a few localized stores, but is intrinsic to information processing that unfolds throughout the brain on multiple timescales.

“The present contains nothing more than the past, and what is found in the effect was already in the cause.”

Henri L Bergson

Memory as a component of all neural processes

In real life, multiple timescales of prior information continuously influence the processing of information in the present. Consider, for example, how prior information shapes language comprehension: each phoneme achieves its meaning in the context of a word, each word in the context of a sentence, and each sentence in the context of a discourse. Thus, past information gathered over milliseconds-, seconds-, and minutes-long timescales all contribute to comprehension. More generally, memories of recent events continuously support the processing of incoming information.

Working memory as a specialized memory store

When information from the recent past is needed for task performance, it is conventionally described as being stored in working memory (WM) [1, 2]. Theories of working memory traditionally focus on memory stores: how information enters and leaves them, their capacity, and the robustness of stored information to interference and decay. The separation between information storage and information processing is rooted in analogies with digital computer architectures, where the systems that perform information processing (e.g., CPUs) are separated from the memory systems that store information (e.g., RAM, caches, and hard-disks). Thus, in computer-inspired models of memory, new information is temporarily stored in limited-capacity working memory buffers, or old information is made available for present processing when it is loaded into the buffers from long-term memory (LTM) storage (Figure 1A). In such models, the systems of memory storage (WM and LTM) are functionally distinct (and in some cases physically separated, Figure 1B) from the systems that support online information processing tasks, such as visual and auditory object recognition, biological motion perception, decision making, and the organization of movement [3].

Figure 1.

A) In a “systems of memory” framework, the information storage (boxes) is functionally separated from the information processing units (arrows). B) A tentative mapping of the working memory model components onto the brain. Figure adapted from [1], reprinted with permission from Nature Publishing Group.

The innovation provided by the multi-store model (Figure 1) was in specifying how a general purpose working memory resource was instantiated via a control system (“central executive”) operating on a set of functionally specialized buffers (the phonological loop, visuo-spatial sketchpad, and episodic buffers); this clarified how working memory might relate to task performance in wide-ranging task domains. However, as researchers came to consider the number of working memory subsystems that would be required to support memory for different kinds of information over multiple timescales, these subdivisions of working memory began to appear inadequate [4]. Moreover, the neural circuits identified as working memory buffers (e.g., the phonological loop) appear in many cases to be the same as the neural circuits that perform the relevant processing (e.g., of phonological and linguistic information) [5].

Newer perspectives on working memory no longer require a physical separation between memory storage and ongoing information processing, but they still maintain a functional separation between stores and processing [4, 6]. For example, in contemporary theories of visual working memory, the visual memory representations are located in the visual processing stream itself. Nonetheless, the representations in visual WM are functionally separated from new visual input, as top-down fronto-parietal signals are required to shield the contents of WM from interference. Thus, information is still considered to be stored in and retrieved from working memory, which has a distinct functional status from the representations of incoming information.

Although working memory models effectively capture behavioral and neural data related to goal-directed control of prior information (e.g., maintaining a visual array over a delay period), we propose that they can also conceptually obscure our understanding of ubiquitous real-life processes in which memory has to be integrated with ongoing processing. We argue that memory and online processing are entangled in many everyday situations, such as reading a book or conversing with a friend. Thus, we suggest replacing the question “how is information stored and then retrieved for later processing?” with the question “how does prior information continuously shape processing in the present moment?”

Questioning the separation between memory and ongoing processing

The separation of the contents of memory from ongoing information processing has long been questioned. A prominent example is the network memory model [7], which embraced the idea that there are no dedicated memory systems and that memory is an integral part of all neural networks, arguing for a shift in research focus from “systems of memory” to the “memory of systems”. The network memory model also incorporated a hierarchical organization of the systemic memory, as higher order areas combined and abstracted memories accumulated from lower areas in the hierarchy. In a similar spirit, the theory of active memory questioned the separation between short-term memory and LTM [8]. In this framework, memory is considered a single entity, either active or inactive, with active memories envisioned as a subset of especially labile memories that are currently being used by the brain to process incoming information [9]. More recently, the separation between memory units and processing units has been questioned by neuroscientists who propose local interactions between memory and perception within visual areas [10–12] and beyond [13], as well as by some long-term memory researchers who argue for the involvement of the medial temporal lobe memory system in perceptual processes [14–17].

Neurobiologically, the idea of functional separation of memory into processes and stores is not well substantiated. In neural circuits, there does not appear to be a separation between neurons that process information and neurons that store information. For example, in the sea slug Aplysia Californica, there are no systems specialized for memory that are separated from perceptual and motor systems [18]; in contrast, short-term and long-term changes in the synaptic efficacy of sensory and motor neurons support learning and memory. More generally, in mammalian circuits, it is known that patterns of prior information reshape synapses over minutes and years [19–21] and can alter levels of activation, potentiation, and excitability over milliseconds and minutes [22–24]. Thus, at the biological level, prior information continually influences information processing in the present, and memory is intrinsic to virtually all neural processes.

Psychologists have also critiqued the segregation of memory and ongoing processing. An early step in this direction came in linking memory performance to the hierarchical depth of processing of a stimulus [25, 26]. Following the rise of connectionist models [27], even more radical theories arose, proposing that the memory needed for in task performance (e.g., in language comprehension) is an intrinsic property of distributed circuits that continuously process the language input [28]. In such models, individual differences in language comprehension are ascribed not to differences in capacity, but to differences in linguistic expertise (i.e. the organization of information processing within the language processing circuits). Another model that emphasized the importance of expertise for explaining mnemonic performance was the long-term working memory (LTWM) model [29], which aimed to account for variations in memory as a function of domain expertise and stimulus semantics.

Finally, computational modeling work has shown in a variety of ways how memory and information processing can be combined in the same circuits [30]. Abstract connectionist models provided some of the earliest and most influential examples [31], while more recent dynamical systems models of information processing have also posited varying integration timescales, leading each level of a hierarchical system to exhibit differing capacities to integrate information over time [32–34].

Process memory framework

Synthesizing these prior ideas with recent empirical work from neurophysiology and neuroimaging, we now outline a framework for how memory serves online information processing. In this framework, we emphasize that traces of past information should not be segregated from ongoing neural processes. To dissociate our notion of memory from the traditional notion of encapsulated memory stores, we will use the term “process memory” (PM) throughout the paper. We use the term process memory, in a broad sense, to mean active traces of past information that are used by a neural circuit to process incoming information in the present moment. Furthermore, we argue for a hierarchical organization of process memory, in which the timescale of memory-dependent processing gradually increases from early sensory areas to high order areas. The new framework is broadly consistent with the family of distributed memory models described above [7, 8, 10].

In the process memory framework, virtually all cortical circuits have the ability to accumulate information over time (Figure 2, red circles). We operationalize the processing timescale of each brain region by measuring its temporal receptive window (TRW): the window of time in which prior information from an ongoing stimulus can affect the processing of newly arriving information. The TRW is defined as a temporal analogue of the spatial receptive field. Some areas have a short TRW (e.g., tens to hundreds of milliseconds), enabling them to integrate a few phonemes to detect a word. Other areas have a medium TRW (e.g., several seconds), enabling the integration of sequences of words while parsing a sentence. Still other areas have a long TRW (e.g., tens to hundreds of seconds) needed for the integration of sentences over time while comprehending a narrative.

Figure 2.

A) A hierarchy of process memory framework. Memory is integral to the operation of each cortical area and there is no separation between the processing units and information storage units. Furthermore, the processing timescale (operationalized by measuring each region’s temporal receptive window [TRW]) increases in a topographically organized manner, from milliseconds in early sensory areas up to minutes in high-order areas. B) A schematic process memory hierarchy for auditory and visual stimulation (for actual data see Figures 3–4 and ref [39]). C) Primary vs. modulatory process memory. Two additional processes (blue circles) modulate the primary process memories (red circles) that are located along the hierarchy: attentional control processes (e.g., fronto-parietal network interactions with short-TRW linguistic regions could enable maintenance of a target word across a delay period) and episodic memory processes (e.g., MTL/hippocampal interactions, most likely with long-TRW regions such as retrosplenial cortex, could enable reactivation of an autobiographical episode).

The TRW increases in an orderly hierarchical fashion from early sensory areas to higher-order perceptual and cognitive areas (Figure 2, size of red circles). Memories of the recent past are not stored in a few dedicated memory stores, but are organized hierarchically across cortical regions that process incoming information. We will now outline evidence for the process memory model, before returning to discuss the relationship between process memory and existing models of working memory (WM) and long term memory (LTM).

A method for mapping processing timescales

To characterize the TRW for each area of the cerebral cortex, we measured the extent to which traces of prior events (recent memory) influenced moment-to-moment neural activity (online processes) during minutes-long real-life stimuli (such as stories and movies – see Box 1). First, minutes-long stimuli were broken into smaller and smaller temporal units (e.g., paragraphs, sentences, words) and the order of the units was scrambled, thereby varying the temporal structure of the stimulus at multiple timescales while preserving the atomic elements (e.g., using the identical movie frames or elementary sound clips in all conditions). Next, neural activity during the intact and scrambled stimuli was examined for evidence of whether online responses changed as a function of the structure of prior events. Areas with short processing timescales (i.e., short TRWs) were expected to respond in the same way at any given moment regardless of the prior context. Areas with long processing timescales (i.e. long TRWs) were expected to modulate their responses to a given event as a function of prior context over many seconds; for example, responses to a particular word would be affected by information from a previous paragraph. Areas with intermediate processing timescales (i.e., intermediate TRWs) were expected to be sensitive to the structure of past events over a few seconds, e.g., at a sentence-level timescale.

Box 1. Dissociation between memory and process is not feasible in real-life contexts.

In the systems of memory perspective, memory is segregated from the neural systems that process sensory input and is kept in dedicated working memory (WM) and long-term memory (LTM) stores (Figure 1). Based on such conceptualization, many memory studies focus on delay periods in which information has to be actively maintained but not processed (e.g., match-to-sample tasks, Figure IA), and in which the integration of past with present information is undesirable (e.g., remembering a target word in a list of distractors). On the behavioral level, dual-task and filled-delay studies have taught us that working memory capacity for arbitrary items is limited (usually to 4+/−1 items [81, 82]) and that remembered content during a filled-delay period is fragile, labile, and susceptible to interference [3]. On the physiological level, these tasks have revealed sustained and selective responses during the delay period in prefrontal cortex and lateral inferior parietal cortex [83, 84]. These findings are of lasting value for explaining how people maintain and manipulate information according to rules using attentional control [57]; however, they do not necessarily provide evidence for segregation of processing and memory storage systems. Actively holding information in mind during a delay period seems to rely on attentional mechanisms, not on dedicated working memory buffers [85, 86], and the evidence suggests that the same brain areas that perform primary processing are also involved in the maintenance of information across delays [4, 6, 87].

Contemporary views of working memory [4, 6] are compatible with the process memory framework we propose (Figure 2), but there is a key difference in emphasis. In contrast to working memory studies that focus on delay periods, in which selected information must be segregated from new input, we highlight the active and ongoing accumulation and integration of information that occurs during online processing. To that end, we explored the kind of memory that is critically required during continuous natural stimulation, where prior information must be integrated with (rather than segregated from) new input. For example, when listening to a spoken sentence (Figure IB), the brain must concurrently detect acoustic features and integrate them with prior sounds to recognize words, while at the same time integrate each word with the preceding elements in the sentence. By examining neural processing of naturalistic temporally extended stimuli, we underline the importance of an under-studied question in memory research: how memories of past information gathered across multiple timescales are used by the brain to continuously process incoming information.

Figure I.

A) Example of a delayed match-to-sample task in which the to-be-remembered information is separated in time from the subsequent cue that triggers the use of the information. B) Example of an everyday sentence in which there is a need to integrate each incoming word with the preceding words as the sentence unfolds over time. Red arrows connect a pronoun and its referent; black arrows indicate the need to integrate phrases within and across sentences.

Neural response dynamics during these temporally-extended real-life stimuli were assessed using inter-subject correlation [35]. The response reliability of any given brain region was measured by calculating the correlation of that region’s timecourse across multiple subjects during exposure to the same stimulus. High correlations between subjects at any given location in the brain indicate the presence of stimulus-driven reliable responses at that location. For more details about the inter-subject correlation method see [35, 36].

A hierarchical topography of process memory

Mapping temporal receptive windows using fMRI, electrocorticography (ECoG), and single unit recording has revealed a large-scale topographic organization of processing timescales along the auditory and visual processing streams [37–39]. Figure 3A presents the gradual transition along the superior temporal gyrus, from short TRWs in early auditory cortex to long TRWs in the temporoparietal junction and angular gyrus, as measured using fMRI while subjects listened to a story scrambled over multiple timescales. Early auditory areas (A1+) responded reliably (i.e. high inter-subject correlation) at all scrambling levels, from the intact full story (FS), to scrambled paragraphs (P), scrambled sentences (S), scrambled words (W), and backward speech (B). These sensory regions were denoted as having short process memory (short TRWs). Further up the processing hierarchy, more and more of the stimulus history was found to affect processing in the present moment. In areas with especially long process memory (long TRWs), such as the temporoparietal junction (TPJ), angular gyrus (AG), and medial prefrontal cortex (mPFC), the cortical activity at each moment depended on information that had previously arrived over tens of seconds. In these higher order areas the responses were reliable only at the full story (FS) and paragraphs (P) levels [38]. The process memory hierarchy was not confined to the processing of temporally extended linguistic input, as a similar topographic organization was found in the visual system when subjects viewed silent movies [39]. Finally, a similar topographic gradient of TRWs was observed in the visual and auditory processing streams for an audio-visual movie using ECoG (Figure 3B), which replicated the fMRI findings with a direct neurophysiological measurement [37].

Figure 3. Hierarchical topography of temporal receptive windows of auditory information.

A) fMRI map of the gradual transition from short to long TRWs along the temporal-parietal axis mapped using audio narratives. The color of each voxel indicates the shortest timescale of coherence in the stimulus that produced a reliable inter-subject response (red: story played backward; yellow: story with word-order scrambled; green: story with sentence-order scrambled; blue: story with paragraph-order scrambled). (A, inset) BOLD timecourses in early auditory areas (A1+) were reliable across subjects exposed to the same stimulus; this was true at all scrambling levels, from the intact full story (FS), to scrambled paragraphs (P), scrambled sentences (S), scrambled words (W), and backward speech (B). Further up the processing hierarchy, more and more of the stimulus history affected responses in the present moment. At the top of the hierarchy, areas such as the temporal parietal junction (TPJ) responded reliably only at the full story and paragraph levels. Figure adapted from [38].

B: ECoG map of gradual transition from short to long TRWs using an audio-visual movie. The TRW indices computed within five individual subjects are displayed on a standard surface. Shorter TRWs were predominantly found near primary sensory areas, while longer TRWs were found further away from sensory areas. The TRW index was defined as the difference in response reliability between the intact and scrambled stimuli, as a proportion of the sum of their respective reliabilities. (B, inset) Early auditory areas (A1+) responded reliably across all scrambling levels, from the intact full movie (FWD), to the coarse scrambled movie (CRS), and fine scrambled movie (FIN). Further up the processing hierarchy, more and more of the stimulus history affected responses in the present moment. At the top of the hierarchy, areas such as the lateral prefrontal cortex responded with much greater reliably at the intact movie and coarse scrambled movie levels. Figure adapted from [37].

Slow neural dynamics are more pronounced in areas with long process memory

In the primate brain, the TRW of an area (i.e., its process memory timescale) covaries with the timescale of its intrinsic neural dynamics. In other words, intrinsically faster neural dynamics are observed in areas with shorter TRWs, whereas intrinsically slower neural dynamics are observed in areas with longer TRWs. Recently, the spike-count autocorrelation during short resting periods was measured in seven cortical areas in the macaque monkey, revealing a hierarchical ordering in which sensory and prefrontal areas exhibited shorter and longer timescales, respectively [40] (Figure 4). Similarly, using ECoG [37] it was observed that neuronal population activity in higher-order regions exhibited a greater proportion of low frequency fluctuations (and increased temporal autocorrelation), while in early sensory areas there was a greater proportion of high frequency fluctuations (and decreased temporal autocorrelation). In both studies, the gradient of timescales of neural dynamics was observed in the absence of any stimulus, suggesting that it may be an intrinsic property of neural circuits. Similar dynamical organizations have been reported using fMRI [41].

Figure 4. Spike-count autocorrelation reveals a hierarchical ordering of intrinsic timescales.

A) Across multiple experiments, single unit data were recorded from seven cortical areas in the macaque monkey: MT, LIP, LPFC, OFC, ACC, S1 and S2. B) Areas arranged according to their anatomical hierarchy as defined by their long-range projection patterns. C) Intrinsic timescales increase gradually along the visual-prefrontal hierarchy. Error bar indicates standard error of fit parameters. Figure adapted from [40], reprinted with permission from Nature Publishing Group.

Abbreviations: ACC, anterior cingulate cortex; LPFC, lateral prefrontal cortex; LIP, lateral intraparietal cortex; MT, Middle temporal cortex; OFC, orbitofrontal cortex; S1, S2, Primary and secondary somatosensory cortex (anterior and lateral parietal cortex).

Together, these results suggest that areas with faster neural dynamics accumulate information over shorter timescales, whereas areas with slower neural dynamics accumulate information over longer timescales. Thus, the abundant slow fluctuations of neural dynamics [42], which are commonly ignored or treated as entirely artifactual, can actually be connected to the processing of real-life information, which is structured on timescales of milliseconds, seconds and minutes [37, 41, 43]. Recent modeling has connected the hierarchical organization of dynamical timescales to changes in excitatory-inhibitory balance (via changes in spine density) [44]; other models have proposed a role for large-scale anatomical organizations [45] and the topography of neuromodulators also appears likely to also play a role.

Scaling of temporal receptive window size as a function of information rate

The results presented so far suggest that TRWs increase gradually from sensory areas with short TRWs up to higher-order areas with long TRWs. The next study asked whether the size of the TRW should be defined in fixed temporal units (e.g., milliseconds, seconds, and minutes) or informational units (e.g., phonemes, words, and sentences). Fortunately, temporal units and informational units are easily dissociated in real-life speech. The fastest speakers of American English will articulate a sentence about twice as fast as the slowest speakers [46]. If temporal integration windows are defined based on informational units, then neural response timecourses should be rescaled in time when the incoming information is rescaled in time. In accordance with such a prediction, a temporal scaling of neural responses was observed throughout auditory, linguistic and extra-linguistic brain areas in response to a linear scaling (speeding or slowing) of the incoming speech rate [47]. These data suggest that the process memory integration window should be measured on relative rather than absolute timescales, with the brain scaling its neural dynamics in response to compression or dilation of the input.

The temporal rescaling of neural responses in accordance with rescaled stimuli began to break down when stimuli were presented at double speed (50% duration stimulus); this is also when intelligibility began to be impaired [47]. Furthermore, it is important to emphasize that, even when the timescales of processing were re-scaled, the TRW hierarchy was preserved. Thus, just as spatial receptive fields can spatially rescale as a function of task, context, and attention demands [48–51], process memory integration windows can temporally rescale according to the rate at which information is arriving [52, 53].

How process memory can be linked with other types of memory

We have argued here for the parsimonious idea that process memory is an integral feature of many cortical processing systems. Moreover, we proposed that process memory increases in a hierarchical manner across the cerebral cortex. How does the process memory framework relate to the classic distinctions [54] between sub-types of memories, both declarative (semantic and episodic) and non-declarative (procedural, priming and conditioning)?

The frameworks are compatible, but answer different questions. Traditional memory categories are defined based on the kinds of stimuli remembered (e.g., visual or auditory), the type of learning (e.g., one-shot or repeated exposures), and the kinds of behavior that the memory supports (e.g., recollection or recognition). The process memory framework is concerned with active memory that is intrinsic to the information processing taking place within a circuit. Thus, there may be similar amounts of process memory, supported by similar circuit mechanisms, even across circuits with very different functions. For example, premotor circuits may contain some of the procedurally learned memories for steering a bicycle, whereas visual cortex may contain the memories that later support recognition of an emotional facial expression. Nevertheless, both organizing a steering movement and recognizing a dynamic facial expression may require integrating information over a second of time. Thus, the type of processes performed in the circuit will define the functional properties of the stored information, but even very different processes may have a common functional signature and some common mechanisms for integrating information. Our model is also compatible with alternative processing-based parcellation of memory, such as [55].

In the process-memory framework, the same cortical neurons that process information (e.g. in sensory perception) also store the information. Nonetheless, additional processes are needed to manipulate, control, and consolidate these process-memories in specific contexts. In particular, there are two major modulatory processes that act on the primary process-memories: attentional control processes supported by fronto-parietal circuits (related to traditional working memory) and binding and consolidation processes supported by medial temporal lobe circuits (related to episodic memory) (Figure 2C).

Process Memory and the Attentional Perspective of Working Memory

In many contemporary perspectives, the term “working memory” has become almost synonymous with attention and cognitive control [56]. The deep connection between working memory and attention arises because, in classic working memory paradigms such as digit manipulation and delayed-match-to-sample, subjects are asked to actively preserve some aspects of the incoming information and task goals in mind (see Box I). When the to-be-remembered content is fragile and labile, attention must be used to shield prior information from interference with new information [3, 57]. Shielding against processing of new input while actively maintaining information in memory is a special case of the more common situations discussed above, in which memory and processing of incoming information are intertwined (see Box I).

Although attentional control is a fundamental aspect of cognition, and we incorporate it as a key modulator in our model (Figure 2C), its role in ongoing perception and comprehension may be obscured by the label “working memory”. The work of memory is performed in virtually every neural circuit, and attentional systems modulate this ongoing processing in accordance with rule- or goal-related constraints. Thus, both the designated memory buffers framework and the attentional-control framework of working memory fail to account for the tight reliance of online neural processes on memory across multiple levels of the processing hierarchy. One perspective on top-down modulation that is more consistent with our distributed model is provided by hierarchical predictive coding models (see Box II Q4).

Box 2. Outstanding questions.

1. Which biophysical circuit models can integrate memory with online processing? Network recurrence is a powerful mechanism for incorporating memory into online processing, but it remains challenging to model how information accumulated over many seconds of time can modify online processing in a local neural circuit [88–90].

2. What is the role of the MTL system in the accumulation and manipulation of information over minutes of time? Areas with long TRWs, at the apex of the hierarchy, seem to be able to accumulate information over minutes. These areas are functionally connected to the MTL system. Will MTL lesions diminish the ability of cortical areas with long TRWs to accumulate information over many minutes?

3. Is information transmitted upward along the process memory hierarchy in a continuous or pulsatile manner? For example, does an area with a “sentence” timescale transmit information continuously to an area with a “paragraph” timescale, or is the communication primarily at the end of the sentence?

4. Does process memory require top-down feedback within the hierarchy? Computing predictions further into the future is made easier with more information about the past; thus process memory and predictive coding may be deeply intertwined. Are predictive signals transmitted in a top-down manner along the hierarchy, from areas with longer timescales to areas with shorter timescales?

5. How is process memory affected by the allocation of attention? In contrast to two-state, multi-state, and continuous state models of working memory [91–93], the process memory framework does not assign a central role to attention. Because attention is capacity-limited, it is unclear how it can support temporal integration simultaneously across diverse circuits and hierarchical levels, as required in many daily life situations (see Box 1). Nonetheless, attention certainly modulates process memory (see Figure 2C), and the nature of this modulation requires further investigation.

6. How can we quantify the influence of prior information? Information theoretic frameworks may enable us to precisely measure the information that the history of a circuit contains about its future, across a range of timescales [94]. Synergy metrics [95–97] can quantify the additional information about the present state that is provided by joint knowledge of past and present input.

7. What is the relationship between process memory and information integration during decision making? Neurons in lateral parietal and frontal areas can accumulate time-varying evidence for or against choice alternatives [98–100], demonstrating history dependent processing in a local circuit. This history dependence is usually simple, as the circuit response depends only on the present input and the present circuit state; future studies should investigate information integration in tasks that require more complex history dependent processing.

Process Memory and the Medial Temporal Lobe

Our framework suggests that processing timescales increase gradually along the cortical hierarchy, from early sensory areas with short (milliseconds-long) integration windows up to higher order areas with minutes-long integration windows (see Figure 2A–B). The responses in areas with the longest processing timescales, at the top of the hierarchy, seem to be influenced by information accumulated over many minutes. These areas overlap broadly with the default mode network (DMN), and include the angular gyrus, retrosplenial cortex, precuneus, posterior cingulate cortex, and mPFC. Many studies implicate these areas in the encoding and retrieval of episodic memories (memory for situated experiences tied to a particular time and place [58, 59]), as well as in a variety of high-level cognitive processes such as decision making [60–62], self-representation [63–65], prospective planning [63], and social reasoning [65].

The DMN is functionally and anatomically connected to the medial temporal lobe (MTL) and hippocampus [66, 67], but it remains unclear to what extent the hippocampus is needed in order for the DMN to retain information over many minutes. Hippocampal damage strikingly impacts the ability to retain episodic memories [68], but exactly how long new information can be maintained in cortical areas without hippocampal involvement is not clear. Behaviorally, under natural conditions, it appears that information can be sustained for a few minutes without reliance on the hippocampus. For example, hippocampal amnesics can retain stimulus information for long enough to engage in a conversation [68], summarize a short passage of prose [69, 70], and play a complex communicative game [71]. Such observations suggest that a meaningful continuous context (e.g., a conversation or listening to a story) may enable information to persist in cortical areas for few minutes without relying on the hippocampus (though hippocampal circuits do appear to contribute to ongoing processing when they are intact [72–74]). Furthermore, the important role of the hippocampus in encoding and retrieval of episodic memories over long timescales [75] does not exclude its involvement in binding of relational information over short time scales [72, 76–78]. Measurements of neural responses to scrambled narratives in MTL lesion patients will provide important constraints on how these regions modulate process memory.

Links between process memory and active long-term memory

Stored knowledge is just as crucial for online processing of incoming information as is recently acquired information. After all, understanding the meaning of a word, sentence, or idea relies heavily on information gathered throughout the lifetime of the individual. Because online processing, at each level of the hierarchy, relies both on recent memories (e.g., information accumulated during an ongoing conversation) and distant memories (e.g., long term knowledge), both types of memory have to be integrated within each neural circuit. Thus, the argument against dedicated working memory stores can naturally be extended to question the notion of dedicated LTM stores that are separate from the circuits that process information [79].

Inspired by prior researchers [8], we suggest that memory should be conceived as a single entity that can be either in an active (process memory) state or an inactive (long-term memory) state. Process memory refers to prior information that is currently used by an area to process incoming information; to influence ongoing processing, the prior information must be in an active state. The active information is composed of the stimulus information accumulated in a given circuit throughout the event, as well as a subset of long term memories activated during the processing of the incoming information. Note that the word “active” is not necessarily synonymous with sustained elevation in firing rates [80], given that information can be sustained in a neuronal circuit by short-term calcium-mediated synaptic facilitation in the absence of recurrent activity [22]. Inactive long-term memories, by contrast, are simply the long-lasting structural features of a circuit (e.g., synaptic patterning) that are not currently affecting the processing of incoming information in that circuit. Inactive long-term memories may be brought into an active state via the influence of other active process-memories in the circuit, or via modulation from medial temporal lobe input or fronto-parietal control areas (see Box II Q2 and Q5).

Concluding Remarks

In this article, we have argued that the traditional dissociation between memory and ongoing information processing is artificial. Diverse cortical functions, ranging from the smooth pursuit of a swiftly moving object, to resolving an anaphoric reference, up to the integration of information across multiple paragraphs of text, all require active integration of past information with new information. We have reviewed data from multiple sources indicating that timescales of processing vary in a hierarchical fashion across the cerebral cortex, with shorter (milliseconds to seconds) timescales in sensory regions and a gradient of lengthening timescales (seconds to minutes) in higher order cortices. Instead of compartmentalizing memory into increasingly specialized storage systems, we highlight the fact that memory is an integral component of the processing conducted in each neural circuit. Process memory is an especially important factor in real-world cognition and perception, which requires continuous information integration, not only maintenance over delays. We propose the process memory framework as a biologically motivated model of the memory that is intrinsic to ongoing, integrative information processing in naturalistic settings.

Highlights.

• Real life neural processing is shaped by stimulus history across multiple timescales

• This requires continuous information integration, not only maintenance over delays

• A “process memory” framework is proposed, integrating memory and online processing

• Process memory is hierarchically organized and intrinsic to circuit and cell dynamics