Data augmentation as a framework for modeling hippocampal contributions to generalization

Abstract

The hippocampus plays a critical role in generalization, enabling us to flexibly repurpose prior experiences to perform novel tasks. Here we suggest that data augmentation—a machine learning strategy to improve generalization by refactoring prior experience—offers a useful framework to conceptualize and model hippocampal function. We begin by outlining how data augmentation operates across two timescales: the traditional “offline” setting, where refactoring training data yields more general representations, and an “online” setting, where retrieved experiences can be flexibly refactored at test time to support zero-shot inference. We suggest that these ‘offline’ and ‘online’ computational strategies map onto functions supported by the hippocampus. Critically, we argue that these computational tools can be leveraged to develop formal ‘linking functions’ between experimental evidence and theoretical claims, such that a unified modeling approach can be used to predict the diverse behaviors that depend on the hippocampus—from navigating in high-dimensional sensory environments to more abstract inferences. We hope this perspective, and the modeling strategies it makes available, will support new efforts to formalize and evaluate theories of hippocampal function.

Intelligent systems must generalize from limited experience. Humans and other animals have remarkable strategies for achieving this feat. After walking around a new environment, for example, we can perform many navigation-related behaviors—from retracing our steps to identifying shortcuts, or planning novel routes between two locations. That is, a single experience can be used to support many downstream behaviors. This process depends on the interplay between two related abilities: the ability to access prior experiences, and the ability to transform those experience in a way that meets subsequent task demands. A wealth of evidence implicates the hippocampus in these operations, from lesion studies showing impairments in spatial navigation [1] or linguistic generalizations [2], to neural signatures including replay [3], preplay [4], place cells [5], and time cells [6]. What computational principles allow a limited set of experiences to support such diverse behaviors?

Many conceptual frameworks have been proposed to account for these empirical findings (Box 1), including cognitive maps [7, 8], pattern separation [9, 10], Complementary Learning Systems [CLS 11], predictive representations [12–14], inferential reasoning [15], and compositionality [16–18]. Each framework captures important aspects of hippocampal function, but they offer distinct accounts of how a single experience can be repurposed to support such diverse behaviors. Understanding the relationship between each of these theories and experimental evidence requires some kind of ‘linking function’ (i.e., a means to infer the theoretical implications of empirical observations), yet this remains an informal process, even for prevailing computational frameworks (Box 2). ‘Stimulus-computable’ models from deep learning provide one route to concretize the relationship between theory and data; theories of hippocampal function can be used to design models that operate over the same stimuli presented to experimental participants (e.g., sensory environments, distances to travel), and can be evaluated via their quantitative fit to experimental measurements (e.g., trial-by-trial behavioral dynamics, neural activity). As such, the models themselves can provide the linking function that relates theoretical claims to empirical observations, and distinct theoretical frameworks can be evaluated using the some empirical framework. However, this requires computational techniques that are well-suited to instantiate theories of the hippocampus.

Box 1 — Relating to prior theories of hippocampal function.

Many computational perspectives have been offered on hippocampal contributions to learning and generalization. Complementary Learning Systems [CLS; 11] offers the view that the hippocampus primarily complements cortical learning by allowing rapid learning without interference. However, it initially had relatively little focus on contributions to generalization. More recent models have considered how hippocampal(-cortical) recurrence could support generalization [20] by connecting related memory episodes. A similar perspective is taken more recently by models describing a broad range of replay findings as context-driven memory reactivation [21]. Other accounts have focused on the hippocampus forming predictive maps [12, 13] or more generally predictive representations [14]. Perhaps the most closely related perspective is from Zeithamova et al. [15], who argue that recall of past events can allow making connections with present ones, either when acting or during encoding. While our perspective is generally compatible with these prior models, it offers the unifying normative interpretation of hippocampal-cortical interactions as data augmentation—recombining memories with each other, and with prior knowledge, to improve current or future generalization. We believe that this perspective offers an overarching computational description of the process by which, for example, the hippocampus might help to “organize knowledge for flexible behavior” [17]. This perspective is consistent with the idea that it is useful for the hippocampus to be less compressive than cortex, and “[preserve] experiences in a relatively raw format for later interpretation” [22]. However, the data augmentation perspective allows drawing on predictions about the detail of how individual experiences are processed to support different kinds of generalization.

Box 2— Linking functions: tightly coupling theory and evidence.

The relationship between experimental evidence and scientific theory is underdetermined [37]; the same data can be plausibly interpreted in many ways. As such, the implications of empirical data must be inferred. This mapping between measurements and their theoretical implications is commonly described as a ‘linking function,’ and there is a rich tradition that aims to formalize this process [38]. While computational models are thought to address this ambiguity [e.g. 39], prevailing models of hippocampal function remain imprecise because the models operate over idealized, hand-tailored inputs (e.g., one-hot vectors) which are designed to reflect known stimulus properties. Consequently, the relationship between model operations and experimental variables must still be inferred, and the assumptions embedded in the input representation (e.g., which stimulus properties matter, how they relate to one another) do considerable theoretical work. In contrast, ‘stimulus-computable’ models greatly reduce this ambiguity by operating over the same sensory data presented to experimental subjects [40]. In this formulation, the model performs the same task that animals are confronted with, from the same sensory data, such that the linking function is the model itself, and model predictions embody theoretical claims. This approach has already been used to understand neuroanatomical structures within the medial temporal lobe. It was long debated whether medial temporal structures like perirhinal cortex (PRC) support performance on perceptual tasks; there were competing interpretations of the available data. Using a stimulus-computable modeling framework to formalize the relationship between experimental data and theoretical claims reduced this ambiguity, resolving this debate [41]. While the hippocampus invites the same approach, this demands a new class of computational models that can perform memory-related tasks, given the same sensory data presented to experimental subjects. In such a model, different strategies for data augmentation could serve as computational specifications for the process of reprojecting data to enhance generalization; the models’ internal representations and behavior under those different strategies would then make distinct, testable predictions about the neural activity and behavior of the animal on the task being modeled.

Here, we suggest that technical frameworks from machine learning offer novel routes to formalize and evaluate theories of hippocampal function. We develop this idea by reviewing machine learning literature on ‘data augmentation’ [19], from vision to language. We identify how data augmentation can operate at two complementary timescales: offline during training to build robust representations, and online via retrieval to support flexible inference on the present task. We relate these modeling strategies to the diverse behaviors that depend on the hippocampus. Our goal is not to propose a novel computational theory of the hippocampus, per se, but to suggest an opportunity for a modeling framework that could allow bridge between computational models and experimental or real-world settings, thus allowing better evaluation of theories of hippocampal function.

1. Data Augmentation in Machine Learning

Machine learning systems face a fundamental challenge: performance improves with extensive training data, yet these data are expensive to obtain. More fundamentally, models are typically deployed in domains not well represented in the training data. Data augmentation is a strategy to address these limitations by synthesizing new training examples from existing data. The core idea is simple: one data point, when appropriately transformed, generates multiple learning signals. This approach has proven effective across many domains, from computer vision to language modeling. Below we outline how these ideas, operating at different timescales, serve complementary functional roles.

1.1. Offline augmentation: Enhancing data to learn robust representations

We begin with vision, where data augmentation first became prevalent. How can a visual system learn reliable representations from a limited training dataset? For example, any image of a cat is just one of infinite possible images of that cat; the viewpoint, background, or camera could all change, without changing the cat’s identity. Data augmentation attempts to discover the structure underlying an image, by learning a representation that is consistent across different re-projections of that image. For example, rotating or recoloring an image will change its pixels, but not the identity or the spatial relationships of the entities within the image. By learning to ignore these nuisance variables, vision models are able to better learn about the entities represented in the images [19]—thus producing vision encoders that can extract useful features for many downstream visual tasks—from classification to object segmentation [23].

Data augmentation can also be applied across higher levels of visual abstraction. Similar strategies can be used for ‘multi-view’ models that infer the underlying spatial structure of an environment [24]. These data augmentation strategies are not limited purely to the domain of vision; vision language models, like CLIP, commonly used data augmentation strategies over linguistic information; the same image can be paired with multiple captions (e.g., “a cat sitting on a bed,” “a house cat in a bedroom”) [25]. This cross-modal augmentation enables using language as a flexible interface for retrieving visual knowledge.

The goal of these data augmentations is to learn abstractions that can be used to support many downstream behaviors. Single-image transformations yield appearance-level invariances; multi-view integration across modalities yields geometric structure; vision-language transformations learn stable image-semantic mappings. While this approach is not motivated by biological plausibility, nonetheless, many systems trained with these data augmentation strategies exhibit emergent alignment¹ with human visual processing; models trained via image-text alignment predict human perception of semantics [27], while multi-view transformers predict human 3D shape perception [28].

Sequential tasks:

Sequential tasks, such as language processing, only multiply the problems of limited data. Any path through a city, or sentence on a topic, is just one of infinite ways that the underlying structure could be experienced. Thus, in sequential tasks from Reinforcement Learning (RL) to language modeling, machine learning researchers have likewise relied on data augmentation.

As a simple example, consider the statement “A is the parent of B.” This relationship could be encoded in many other surface forms, e.g., the reversal “B is the child of A.” Surprisingly, language models that have been trained on the relation in one direction do not generalize to the reverse [29]. Thus, researchers have used data augmentation to help the system learn the more abstract information; explicitly training models on reversed sequences [30], or augmenting sequences with masked copies that encourage learning both dependencies [31].

However, sequential data affords much richer augmentations, beyond simple, fixed transformations. Indeed, language models themselves are powerful and flexible tools for transforming information. Thus many works [32–36] use the model’s own capabilities to elaborate training documents by reasoning about possible extrapolations and connections between them—by extracting entities and reasoning about their relations [34], or chaining statements from multiple documents to make new inferences [32, 33]. These strategies bootstrap from the model’s existing language abilities to enrich the experiences at hand. By generating alternative “views” of the information underlying the learning experiences, the system is able to learn more effectively.

1.2. Online augmentation: Flexible inference through context-dependent retrieval

How can a system flexibly re-purpose stored knowledge for tasks it was not originally trained to solve? In the offline approaches above, abstractions that may be useful in the future are encoded into the system by augmenting the data accordingly during training. However, at test time the system might face entirely novel challenges that were not anticipated during training. Language models provide a useful setting to understand how a system can adaptively re-purpose its knowledge to solve unexpected tasks via online data augmentation.

Online data augmentation addresses this limitation by flexibly modifying how knowledge is queried. In short, the system retrieves² relevant information and provides it as context, augmenting the present information with past experiences, and relying on the model’s ability to appropriately combine these multiple sources. For the parent-child example, this means retrieving the learned fact (“B is the parent of A”) and presenting it in context, rather than expecting the model to retrieve and transform it internally. When information is provided in context, language models successfully perform reversals, make transitive inferences, and better answer novel questions [32, 35, 42]. (Alternatively, models can learn to do approximate retrieval via generation [43].) This strategy extends beyond language to visual reasoning, where vision-language models can generate intermediate visual representations at test time to support spatial inferences [44, 45].

Unlike offline data augmentation, where transformations are fixed during training, online augmentation is flexible and task-dependent. The key insight is that systems can bootstrap from their existing capabilities to transform retrieved content to serve novel goals—such as reasoning about retrieved documents. What makes this “augmentation” is that the same knowledge is re-factored differently (reversed, re-composed, or chained), depending on current goals. This ability to recombine learned elements in novel ways emerges naturally in systems that can retrieve knowledge in a form that preserves its relational structure, rather than compressing everything into fixed associations.

1.3. Data augmentation at different timescales

Offline and online data augmentation provide complementary strategies for sample-efficiency. Offline augmentations apply predetermined transformations to learn features useful for downstream behaviors; online augmentations apply context-dependent transformations to enable flexible inference at test time. These computational strategies are unified in recognizing that any given experience contains a single projection of the underlying data; a photograph of a scene, a sentence about an idea. Re-projecting these data from different perspectives can extracts more information than any single projection could provide. The difference between online and offline augmentation is simply when that re-projection occurs.

2. The analogy between data augmentation and hippocampal function

There are useful parallels between hippocampal function and the data augmentation strategies outlined above. In essence, data augmentation requires two components: an experience, and operations to transform it. We suggest the hippocampus preserves information related to single events in a way that supports subsequent transformation [cf. 22]. Offline, these experiences can be reinstated and interleaved with ongoing learning, allowing cortical structures to build more general representations from sets of related events [11]. Online, prior experiences can be re-instated, and then re-factored, in order to meet the demands of the current task [46]. In both cases the hippocampus provides information that can be operated on by—and through the interaction with—other neural structures. We elaborate on the offline and online settings below (and in Box 3).

Box 3 — Interpreting other prior findings as data augmentation.

Many other prior findings on hippocampal activity can be reinterpreted from the perspective of data augmentation. We outline a few more of these connections here.

Abstractions and schemas:

Hippocampal replay recomposing learning experiences according to previously-learned abstract structures [47] is similar to how offline augmentation in language models can reproject learning experiences through prior knowledge to yield new data. Other structural knowledge like schemas [48] may similarly support this type of augmentation; in turn, replay that integrates related experiences may help to extract more abstract schemas from those experiences [49].

Compositional reasoning:

Online hippocampal replay has also been implicated in reasoning about the composition of a structure by generating possible compositions of previously learned components [50, 51]; this can be interpreted as augmentation through recomposing prior experiences to make new predictions.

Semantic search:

Findings that the hippocampus supports online enumeration of semantic properties of a concept [52]—and in particular, the argument that it does so by offering shortcuts between dissimilar semantic features through bound representations [18]—can also be interpreted from the (online) data-augmentation perspective as augmenting each concept with direct connections to its experienced features.

Offline augmentation and systems consolidation.

A long tradition holds that the hippocampus supports the gradual transfer of knowledge to neocortical structures via systems consolidation—for example, complementary learning systems theory [CLS 11, 53]. A candidate mechanism facilitating this division of labor is replay: during rest, the hippocampus ‘re-instates’ patterns of neural activity that occurred during the initial encoding of those experiences [54].Remarkably, replay does not always recapitulate exact experiences; it also produces never-experienced trajectories [3, 4], reverse unrolls from a reward 47, 55, and sometimes exhibits a bias away from recently experienced trajectories 3. While the nature of these hippocampal-neo-cortical interactions is an ongoing research question [21], it is clear that manipulating which memories are reactivated during sleep changes subsequent performance [56, 57]. In effect, replay enables a limited set of experiences to be re-factored, such that the neocortex can learn structure that was only latent in the individual learning events. This is data augmentation in the precise sense that we began with: the hippocampus supplies an experience, via replay, which can be operated on by other neural structures. This is not a novel claim about hippocampal function; rather, data augmentation provides a coherent framework to model these diverse properties of the hippocampus, and the downstream behaviors they enable. With these tools, theoretical claims can be made into explicit design choices: a claim that replay prioritizes weakly-learned experiences, for instance, is instantiated as a sampling policy over stored experiences, whose consequences for consolidation and subsequent behavior can be directly evaluated (Box 4).

Box 4 — Modeling hippocampal function; a worked example.

Our proposal’s value is ultimately in the modeling approaches it enables. To make this strategy concrete, we engage with a question that has been the topic of considerable attention: which experiences are replayed by the hippocampus? Normative accounts have proposed that replay is a function of gain and need [66]; how much reactivating a memory improves subsequent decisions, and how likely that memory is to be relevant. Framed in relation to our data augmentation proposal, this is a policy for curating the augmentation set, determining which experiences to re-factor. However, determining which experiences are replayed does not specify how information must be replayed to support subsequent behavior.

Consider navigation. First, an animal must process their sensory trajectories through an environment, and encode this information within the hippocampus in a manner that preserves it for future replay. Offline, these experiences are thought to be replayed proportional to gain and need, but there are many possibilities for how these experiences might be re-factored to support adaptive behaviors: possibilities include re-projecting sensory information between coordinate frames to plan a route, reversing a sequence to propagate rewards backward [55], or composing segments of trajectories to construct a shortcut [3]. Prior modeling work instantiates these as relatively abstract operations over idealized inputs that have no concrete relationship to the sensory experiences that animals are confronted with. As such, there is not a formal ‘linking function’ between the model operations and experimental variables, and so the relation to theoretical claims is speculative.

In contrast, the computational frameworks we have introduced can operate over the same sensory data as experimental subjects. As such, model design choices—and by proxy, the theoretical claims they instantiate—can be evaluated empirically, via the model’s quantitative fit to animal behavior on the same experimental task. This approach forces a commitment that verbal theories can leave implicit: which operations are supplied by the hippocampus, and which are accomplished through interaction with other structures. Whether a shortcut reflects composition within the hippocampus or emerges from hippocampal–cortical exchange; whether the hippocampus re-instates experiences veridically or transforms them (e.g., by compressing them into more generalizable structure [12]) are questions a stimulus-computable model must answer explicitly. Framing these theoretical claims as modeling choices allows them to be resolved empirically.

Online data augmentation in spatial navigation.

An animal traversing a novel environment processes sensory information associated with that environment’s unique configuration of landmarks, distances, objects, and actions. This single trajectory can only provide a limited representation of the environment. But that structure is latent within the sequence, and it can be discovered. Just as re-projecting a single image aims to recover the scene it captured, re-projecting a trajectory can reveal the environment’s spatial structure. While many dimensions could be used to reproject these data, egocentric and allocentric coordinate frames are particularly useful for navigation. An egocentric encoding (‘turn left, walk ten steps’) captures action sequences but not environmental structure; an allocentric encoding (‘food north of the tree’) captures world coordinates but not the affordances necessary to traverse a scene. Re-projecting between these coordinate frames, combined with geometric operations like reversal and composition, unlocks what a single sequence cannot: novel shortcuts, backward routes, and answers to “where was that object?” The hippocampus supports exactly these operations. During an episode, replay and preplay transform stored experiences using these geometric operations—identifying shortcuts [3], connecting separate sequences once a link has been established [58], reversing experiences [55, 59], and even preplaying routes to potential future goals [4]. Online, these transformations can both enablie the kinds of online data-augmentation strategies outlined above to support current behavioral goals [cf. 46]; offline, they can support consolidating the spatial primitives that online planning and behavior require.

Online data augmentation in relational generalizations.

The principles outlined above extend beyond spatial domains to abstract relational reasoning. When an animal learns a single fact—for example, “A is larger than B”—this provides just one projection of the underlying relational structure. But that latent structure can be re-projected (reversed to “B is smaller than A,” or chained with “B is larger than C” to infer “A is larger than C,”). Language models that encode such facts solely into their parameters fail these transformations—they cannot reverse learned relations or make multi-step transitive inferences [29, 60]. The same knowledge maintained in retrievable form, however, supports systematic transformation [32, 33]. Hippocampal lesions produce analogous deficits: failures in transitive inference requiring integration across experiences [61] and inflexibility in rephrasing relations [2]. Intact hippocampal function, by contrast, enables flexible relational generalization through replay-driven recombination of prior experiences [62]—integrating experiences to support transitive inferences [20, 63, 64], or generalization after contingency changes [65]. Just as spatial navigation requires re-projection across coordinate frames, relational reasoning requires re-projection across relational structures—operations the hippocampus supports through episodic retrieval.

3. Conclusions

We have outlined how data augmentation strategies from machine learning could be used to understand and model hippocampal function. Our goal is not to introduce a novel theory of hippocampal function, but to outline a perspective that could help to integrate many existing theories by interpreting replay, consolidation, and flexibly task-dependent retrieval as instances of a common set of operations. Because data augmentation is a computational strategy coupled to stimulus-computable methods, it would allow us instantiate theories of hippocampal function as models that operate over the sensory data animals actually receive , rather than the hand-designed inputs prevailing models require. Realizing this promise is now potentially a feasible undertaking. A natural first step is to build such a model for a single, well-characterized behavior (e.g., the identifying a novel shortcut) where competing theories make concrete predictions. These disagreements can then be adjudicated by their fit to the behavior, and neural activity, of animals performing the same task. This will demand experimental paradigms rich enough to allow nontrivial generalization from naturalistic input, which remain scarce [67]. But we see much of the value of this framing in the demands it imposes: it forces distinctions that verbal theories can leave implicit [cf. 39]—which operations the hippocampus supplies, and which emerge through its interaction with other structures—into explicit modeling choices that can be weighed against data. We hope that this will prove a productive step toward a more detailed understanding of the computational role of the hippocampus.

Figure 1:

An overview of our perspective, relating machine learning findings and methods to phenomena of hippocampal contributions to generalization. (top) The deficit: machine learning systems without data augmentation, and animals with hippocampal-lesions, show impaired generalization. (middle) Data-augmentation has been shown to ameliorate these deficits in machine learning, and we argue that the hippocampus may contribute to a similar process. (middle, second row) Data augmentation can take various forms, including refactoring an individual learning experience based on prior knowledge, or recombining multiple experiences. (bottom) Ultimately, we aim for an end-to-end, stimulus computable model that predicts how neural activity and behavior depend on individual learning experiences.