The Integration of Distributed Memory Traces

Memories are stored in distributed, neocortical networks. These networks can hold episodic memories of specific events and their contextual details, or semantic memories that comprise factual knowledge about the world, concepts, or rules (Tulving, 1972). The defining features of these different memory types are thought to be encoded in modality-specific, posterior representational regions (e.g., auditory or visual sensory association cortices) (Marr, 1970; Frankland and Bontempi, 2005). Yet, as we remember, we reconstruct memories as a whole. The so-called “binding problem” describes the need to converge and integrate information from different storage sites (Damasio, 1989). Presumably, this “binding” underlies memory retrieval (Shimamura, 2011), but also the holistic experience of present percepts (Treisman, 1998), and it might be achieved in two steps. First, neuronal synchronization between distant brain regions was proposed as a functional mechanism for the integration of information (Engel et al., 1997). Second, higher-level association regions are assumed to support feature convergence. Here, I will focus on the latter aspect. One region that might perform such a role is the angular gyrus (AnG), which has been implicated in numerous cognitive operations, among them attention and mnemonic processing (Wagner et al., 2005; Kuhl and Chun, 2014), as well as the recombination of information (Shimamura, 2011; Bonner et al., 2013; Seghier, 2013; Price et al., 2015; Wagner et al., 2015). However, evidence for “multimodality” has been lacking so far, and it was unclear whether integration would differ when retrieving different memory types.

In their recent publication, Bonnici, Richter et al. (2016) tackled this issue of multimodal feature integration during episodic and semantic retrieval using fMRI. In a first step, the authors carefully selected cue words (nouns) that were preferentially associated with auditory, visual, or audiovisual features. Thus, unimodality or multimodality was manipulated by using cues that elicited specific associations. Second, during the actual fMRI study, subjects were presented with a cue word on the screen and either vividly recalled previously seen video clips (episodic condition) or generated as many words as possible that were associated with the specific cue (semantic condition). During word generation, subjects were asked to continuously refer back to the cue, thereby focusing their attention on the specific modality imposed (e.g., visual associations, such as “sky”, “light”, or “yellow” for the cue word “sun”). The authors found increased AnG activation for multimodal compared with unimodal processing during episodic retrieval, as well as event-specific, multimodal memory representations that were successfully detected by a pattern classifier. For semantic processing, however, no such distinction was apparent, and unimodal as well as multimodal retrieval engaged the AnG to a similar degree. These findings highlight a special affinity of the AnG for semantic information, regardless of modal complexity, which might be explained by its general function in higher-level, semantic cognition (Binder and Desai, 2011). In addition, the results emphasize a functional segregation within the AnG. Episodic relative to semantic retrieval involved more dorsal AnG subregions, possibly related to the coding of egocentric spatial information and scene processing while recalling the video clips (Howard et al., 2013; van Assche et al., 2016). Last, the authors found that auditory and visual information was processed by the middle temporal and fusiform gyrus, respectively. These findings support the hypothesis of a distributed, neocortical memory system that stores modality-specific features in their respective representational regions and recombines them during retrieval.

The traditional model of systems consolidation hypothesizes that the memory trace is initially held by the hippocampus (Marr, 1970; Frankland and Bontempi, 2005). Over time, hippocampal-neocortical interactions prevent memory decay and strengthen associative connections, such that the memory trace eventually becomes stored within a neocortical network that includes the mPFC and posterior representational regions (Takashima et al., 2006, 2009). In this model, the hippocampus and mPFC integrate information at different times. Thus, multimodal feature integration is not a function uniquely subserved by the AnG, and it is currently unclear how these multiple integrational “hubs” differentially contribute to long-term memory storage and retrieval. One possibility is that their contribution shifts over time, perhaps organized in the form of a “hierarchical indexing scheme” (Teyler and Rudy, 2007). Hippocampal memory representations are thought to provide an associative index that points toward modality-specific representational regions, and this indexing function is taken over by the mPFC with time. Indeed, Bonnici, Richter et al. (2016) reported a trend toward increased hippocampal activation during multimodal compared with unimodal recent episodic retrieval (p = 0.059). An extended model of systems consolidation might further include the AnG. Here, indices held by the hippocampus or mPFC may point toward the AnG, which in turn recombines the distributed features and holistically “reinstates” the memory representation (Kuhl and Chun, 2014; Wagner et al., 2015; Bonnici, Richter et al., 2016).

The organization of such a hierarchical indexing scheme, and whether memory representations are bound and integrated by the AnG, might depend on (1) the current stage of consolidation (recent vs remote retrieval) and (2) the type of material to be consolidated (episodic vs semantic). Previously, Wagner et al. (2015) investigated the retrieval of schema memories and showed that their different components, or features, converged within the AnG after 24 h consolidation, but not before. The multivoxel patterns only partly generalized across days, suggesting a change in memory representations over the passage of time. Thus, whereas vivid episodic memories appear to be represented within the AnG during recent retrieval (Kuhl and Chun, 2014; Bonnici, Richter et al., 2016), representations of context-independent semantic, or schematic, memories might be “reinstated” in the AnG only after consolidation (Wagner et al., 2015). This suggests multiple, temporally distinct routes into long-term storage for these diverse memory types. Additionally, in the case of semantic information, its level of abstraction might play a relevant role. General knowledge, rules, or broad schemas share a high level of abstraction and are thus fit to guide behavior in multiple situations. In contrast, more specific schemas might be based on fewer encounters with similar material, are thus less abstracted, and contain more episodic details. This also highlights the difficulty of classifying memories as episodic or semantic because semantic retrieval can be contaminated by episodic aspects and vice versa. In the study by Bonnici, Richter et al. (2016), subjects were asked to generate as many words as possible that were related to a specific cue word during semantic trials. For example, the cue word “sun” and its associations “sky”, “light”, and “yellow” might also bring back vivid episodic memories of a day at the beach. Thus, an episodic or semantic categorization is imposed onto the data by the researcher, and a more continuous or a data-driven approach might be useful for future studies.

Moreover, a detailed comparison of the core regions within the so-called “retrieval network”, including the hippocampus, mPFC, and the AnG (Rugg and Vilberg, 2012; Wagner et al., 2015), might elucidate their differential contributions to memory integration during episodic and semantic retrieval. Previous work approached this issue by using univariate analyses methods, multivariate pattern techniques, and functional connectivity analysis (Bonner et al., 2013; Price et al., 2015; Wagner et al., 2015; Bonnici, Richter et al., 2016). Nevertheless, the neural substrates underlying multimodal feature integration remains unclear, and it is not known which regions “drive” this computation. A potential next step to demonstrate integration and the direction of regional interactions would be to use effective connectivity measures (e.g., dynamic causal modeling) or novel developments from the machine learning domain. For example, one possibility would be to model the neural processing hierarchy along the ventral visual pathway up to higher-level association cortices with deep neural networks (LeCun et al., 2015). This could help to unravel the level of representational complexity at each processing step within the hierarchical model, or, by using recurrent model connections, could be used to map information flow during multimodal feature integration.

In conclusion, the findings of Bonnici, Richter et al. (2016) yield an important step in a new direction in that they demonstrate multimodal feature integration within the AnG, whereas unimodal (episodic) retrieval mainly engaged the respective representational regions. This strengthens the assumption of a distributed, neocortical memory network, and the distinction between episodic and semantic AnG processing might indicate different routes into long-term memory. In future studies, direct comparison between “hubs”, such as the AnG, the hippocampus, and the mPFC, has the potential to pinpoint their differential contributions to the binding and integration of distributed memory traces during retrieval. Finally, the flexible nature of mnemonic networks warrants an investigation of memory integration over time.