Coordinated representations for naturalistic memory encoding and retrieval in hippocampal neural subspaces

Abstract

Our naturalistic experiences are organized into memories through multiple processes, including novelty encoding, memory formation, and retrieval. However, the neural mechanisms coordinating these processes remain elusive. Using fMRI data acquired during movie viewing and subsequent narrative recall, we examine hippocampal neural subspaces associated with distinct memory processes and characterized their relationships. We quantify novelty in character co-occurrences and the valence of relationships and estimate event memorability. Within the hippocampus, the novelty subspaces encoding each type exhibit partial overlap, and these overlapping novelty subspaces align with the subspace involved in memorability. Notably, following event boundaries, hippocampal states within these subspaces align inversely along a shared coding axis, predicting subsequent recall performance. This novelty-memorability alignment is selectively observed during encoding but not during retrieval. Finally, the identified functional subspaces reflect the intrinsic functional organization of the hippocampus. Our findings offer insights into how the hippocampus dynamically coordinates representations underlying memory encoding and retrieval at the population level to transform ongoing experiences into enduring memories.

In this study, the authors show that novelty encoding aligns with memory formation in the hippocampus but not with memory retrieval, and that hippocampal components involved in each process reflect its intrinsic functional organisation.

Introduction

In our daily environment, we often encounter objects and agents with multiple types of simultaneously varying information. Consider a scenario where a previously harmonious couple experiences conflict when a new person is introduced into their relationship. An outside observer may initially perceive novel information, such as the presence of a new individual or changes in the partner’s mood, form memories of this information, and later retrieve them to guide adaptive behavior^1,2. These memory processes encompass three fundamental cognitive functions: i) encoding novel information across multiple attributes, ii) forming memories, and iii) retrieving them when needed. While previous studies have linked distinct subregions of the hippocampus and medial temporal lobe to individual memory processes, such as novelty encoding^3–6, memory formation^7–9, and retrieval^10–12, how these processes are coordinated remains poorly understood.

Previous behavioral findings have suggested that the process of novelty encoding influences the formation of memories^13–16. Within the hippocampus, two possible relationships between these processes emerge: novelty encoding and memory formation may occur orthogonally, or they may operate in a coordinated manner. How does the hippocampus compute and coordinate these processes for memory encoding and retrieval? To address this question, we examine the parallel and orchestrated computation underlying these memory processes by analyzing neural subspaces, where the temporal sequence of neural activity patterns related to each process unfolds¹⁷. The alignment of neural subspaces offers mechanistic insights into how multiple cognitive processes could be coordinated, ranging from fully orthogonalized to overlapped or falling somewhere in between^18–20. Conceptually, the subspace alignment indicates a shared computation between two cognitive processes. For instance, the presence of overlapping subspaces between the processing of past and current information in task-switching behavior suggests interference from past information in the current context, potentially leading to a high switching cost²¹.

To explore the relationship between these putative computational mechanisms, a task paradigm that effectively elicits multiple cognitive processes as well as a suitable quantitative analysis method are needed. A movie stimulus proves particularly fitting for engaging various processes concurrently, given its dynamic nature that introduces novel information²². Notably, movie stimuli often portray social information, which is a salient feature of daily life²³, making them pertinent for investigating cognitive processes related to naturalistic memory. Moreover, the narrative structure of movies offers an opportunity to investigate memory encoding and retrieval²⁴. Given the advantages of these stimuli, numerous studies have demonstrated their effectiveness in engaging a range of cognitive processes, including memory in hippocampal regions^25–29.

In this study, we leveraged the concept of neural subspaces and a narrative movie stimulus to investigate the coordination of memory processes. To identify neural subspaces associated with memory processes, we applied targeted dimensionality reduction^30–32 (TDR) to fMRI blood oxygenation level-dependent (BOLD) data acquired as participants viewed a movie and subsequently recalled its narrative. TDR has been widely used in electrophysiology for extracting low-dimensional subspaces that describe cognitive processes of interest. TDR extends PCA by specifically targeting how neural populations encode task variables associated with cognitive processes^30,31. Unlike PCA, which does not aim to link principal components to specific cognitive functions, TDR is designed to identify PCs that are directly related to cognitive processes by applying PCA to the neural population responses associated with task variables derived from linear regression modeling. We focus on neural dynamics occurring around event boundaries, where discontinuities in sensory and semantic experiences prompt the processing of preceding events and subsequent memory formation^{26,29,33–35}, and around recall initiations, where the memory retrieval process operates³⁶. After extracting neural subspaces in the hippocampus associated with novelty encoding, memory formation, and retrieval, we propose three hypotheses. The initial hypothesis is that the neural subspaces involved in encoding novel information across distinct attributes are either aligned or orthogonal (Fig. 1A). The second hypothesis is that the axes of novelty subspaces, whether aligned or orthogonalized across multiple attribute dimensions, are linked to the memorability subspace, which reflects a memory formation process (Fig. 1B). Considering the significance of retrieving stored knowledge to assimilate new experiences into existing memories, the third hypothesis proposes that the memory retrieval subspace remains aligned with the memorability subspace (Fig. 1C).

Fig. 1. Schematic representation of hypotheses.

A Hypothetical relationships between neural subspaces encoding two types of novel information: co-occurrence and valence novelty. (left) Aligned subspaces suggest that novelty levels are encoded in both their counterpart and respective subspaces. (right) Orthogonal subspaces indicate exclusive encoding of each novelty type in its respective subspace. B Hypothesized relationship between novelty encoding and memory formation processes. (left) If novelty processing is linked to memory formation, novelty levels are encoded in the memorability subspace, and vice versa. (right) If the processes are independent, the neural dynamics within the memorability subspace do not account for novelty levels. C Hypothesized relationship between memory formation and retrieval processes. (left) Aligned subspaces suggest shared computations between memory formation and retrieval. (right) Orthogonal subspaces indicate the independence of retrieval from formation processes. The color bars represent the range of novelty or memorability levels from low to high.

To test our hypotheses, we defined two distinct forms of novel social information within our movie stimuli: character co-occurrences (i.e., changes in the set of characters present) and the valence of character relationships (i.e., transitions between positive, negative, or neutral interactions among characters). These social categories correspond to associative and contextual novelty as previously defined in the literature^14,37. Initially, we found that two neural subspaces within the hippocampus, each associated with a different type of novel information, partially overlap, with major axes exhibiting nonorthogonality (i.e., novelty coding axis). This novelty coding axis was further linked to the memorability subspace within the hippocampus (i.e., shared coding axis), suggesting that the aligned subspace underlies both novelty encoding and memory formation. Notably, we observed an inverse correlation between neural states corresponding to novelty and memorability levels along the shared coding axis, indicating greater recall performance for less novel events. However, the alignment between the novelty and memorability subspaces was specific to memory formation during movie viewing and did not extend to memory retrieval during narrative recall, indicating a distinction between the processes of novelty encoding and memory retrieval. Conversely, the neural dynamics within the memory retrieval subspace reflected participants’ recall performance, exhibiting an alignment of neural states along the memory coding axis with the memorability subspace observed immediately before commencing the recall of each event. Additionally, we found that the hippocampal subspaces are organized along the hippocampal longitudinal axis, providing insight into the functional organization of the hippocampus. Overall, we propose that the alignment between subspaces and the rearrangement of neural states underlie the coordinated representation of multiple memory processes within the hippocampus.

Results

Novelty in events influences subsequent recall

During fMRI scanning, participants viewed a three-episode movie from a Korean web drama (total length 43 minutes), depicting evolving social relationships among characters (Fig. 2A). Independent annotators segmented the movie into distinct events by marking the onset of new scenes whenever changes in topics or character relationships were observed. Matrices representing two types of critical social information for comprehending the movie narrative—the co-occurrence among six characters and the valence of their relationships—were constructed for each segmented event through detailed annotation. The co-occurrence matrix captured the frequency of characters appearing concurrently within each event, while the valence matrix quantified the sentiment of interactions between characters based on sentiment scores of their interaction words (see Methods). We assessed the degree of novelty in co-occurrence and valence information for each event by computing the Pearson correlation between the matrices of the current event and the cumulative sum of the matrices up to that event, reflecting the disparity between the information acquired from the current event and prior knowledge (Fig. 2B, C and see Methods). For example, if the valence scores of relationships in the current event are inconsistent with those in previous events (e.g., a loving couple suddenly starts fighting), the valence novelty of the current event would be high. There was no significant correlation between eventwise co-occurrence and valence novelty (Fig. 2D, r = 0.17, p = .263, 95% CI = [−0.13, 0.45], a two-sided test), suggesting that these two types of social information were distinct. To rule out the effect of time progression in the movie on novelty measurements (e.g., later events having lower novelty), we further calculated the correlation between each type of novelty and its timing within the movie. We did not find significant correlations (Supplementary Fig. 1A; co-occurrence: r = 0.049, p = 0.755, 95% CI = [−0.26, 0.34]; valence: r = 0.197, p = 0.206, 95% CI = [−0.11, 0.47], from a two-sided test).

Fig. 2. Task structure and measurements.

A During fMRI scanning, participants viewed a movie followed by free recall of its narrative. Participants’ recall transcripts were segmented into corresponding movie events. The analysis focused on neural dynamics within ±10 seconds around event boundaries and recall onset for each event. B The co-occurrence novelty level of each event was assessed by calculating the correlation between the sum of co-occurrence matrices of prior events and the current event’s co-occurrence matrix and then subtracting this value from one. C Valence novelty for each event was quantified by computing the correlation between the sum of valence matrices of prior events and the current event’s valence matrix. D Co-occurrence novelty did not significantly correlate with valence novelty across events (p = 0.263, from a two-sided test). Each point represents an event. Events were categorized into a 3×3 condition matrix based on novelty degrees for further analysis. The numbers in parentheses denote regressors identifying neural subspaces for each novelty condition (left: co-occurrence novelty, right: valence novelty). E Memorability was quantified as the number of words in participants’ recall transcripts that correctly matched the movie annotation per event divided by the total annotation word count for that event. F Memorability scores across novelty levels (Co-occurrence novelty: p = 0.040, Valence novelty: p = 0.033, from a two-sided paired t-test). The hand-drawn faces (designed by pikisuperstar on Freepik at https://www.freepik.com) and the head with a brain (designed by Vectorslab on Freepik at https://www.freepik.com) were used for visualization purposes.

Following each episode, participants verbally recounted their memories of the movie in chronological order with maximum detail, without any external sensory cues or guidance from experimenters (Fig. 2A). The memorability of each event was assessed by computing the ratio of words in recall transcripts that matched the words in the movie annotation for the corresponding event (Fig. 2E and see Methods). On average, participants recalled 76.1% of the events in the movie (mean memorability score = 0.23, s.d. = 0.08; Supplementary Fig. 1B), and the pattern of recall success across events was consistent among participants (mean Jaccard similarity = 0.72; Supplementary Fig. 1C). To evaluate the influence of novelty on memorability, we categorized events into three levels—high, middle, and low thirds—for both co-occurrence and valence novelty and compared their corresponding memorability scores. Regardless of the novelty type, events with low novelty received significantly greater memorability scores than events with high novelty (Fig. 2F, co-occurrence: t(23) = 2.18, p = 0.040, 95% CI of the difference = [0.001, 0.042]; valence: t(23) = 2.26, p = .033, 95% CI of the difference = [0.002, 0.050], from a two-sided paired-sample t-test), indicating that participants more accurately recalled events with minimal changes in co-occurrence and valence than events with significant changes.

Neural subspaces in the hippocampus span the novelty and memory-coding axes

We introduced quantitative measures to characterize the novelty and subsequent memorability of each movie event. Our primary aim was to identify the neural subspaces encoding different types of novel information and forming memories during ongoing naturalistic experiences. We focused on the hippocampus (Fig. 3A), a widely known central hub for encoding novelty, forming and retrieving memories^3–12,38. Based on previous studies reporting increased hippocampal activities after event boundaries^25–28, we hypothesized that novelty and memorability are encoded in hippocampal neural subspaces around event boundaries during movie viewing. To test this hypothesis, time courses were extracted from the hippocampus within a time window spanning 6 seconds before to 14 seconds after each event boundary (Supplementary Fig. 2A, B), accounting for the four-second hemodynamic response delay (−10 to +10 seconds). Using TDR on the neural data, we constructed neural subspaces from all voxels in the hippocampus (Fig. 3B; Supplementary Fig. 2C). The data from each movie event were categorized into nine novelty (a combination of three novelty levels in co-occurrence and valence; Fig. 2Dright) and nine memorability conditions (ranging from low to high). We then projected the averaged hippocampal responses for each condition onto these subspaces, generating neural trajectories around event boundaries (see Methods).

Fig. 3. Analysis of neural subspaces.

A Hippocampal ROIs are visualized on the medial view of an inflated cortical surface. B Subspace analysis. 1) The finite impulse response (FIR) was estimated for each voxel at each time point. 2) The beta coefficient matrix was derived from a linear regression model incorporating combinations of different novelty levels. 3) Principal component analysis (PCA) was applied to the beta matrix. (lower) Averaged responses for each novelty/memorability condition were projected onto the principal component space. The encoding performance was evaluated to verify whether each subspace accurately reflects the corresponding memory processes. For each time point, a line connecting the two most distant neural states among the conditions was drawn. All neural states were projected onto this line, and R-squared values were calculated as encoding performance, accounting for novelty/memorability levels. C (left) Encoding performances of observed hippocampal subspaces at each time point around event boundaries are shown. The red lines represent the mean R-squared difference between the observed and null distribution. The gray area indicates ± 2 s.d. of differences. Black horizontal lines denote the statistical significance of encoding performances at each time point (p < 0.05, from a one-sided permutation test). (right) Encoding performances averaged across 10 time points preceding and following event boundaries. The gray dots represent the observed encoding performance compared to each of the 1000 encoding performances obtained from the null distribution (all ps < 0.001, from a one-sided permutation test).

For each of the three subspaces (co-occurrence novelty, valence novelty, and memorability), 12 eigenvectors from the dimensionality reduction process explained 80% of the variance (Supplementary Fig. 3A). All subsequent analyses were conducted within these 12-dimensional subspaces. To confirm that the neural trajectories within each subspace accurately represented the respective memory processes, we computed R-squared values as the extent to which the neural states within these subspaces were arranged along a coding axis according to novelty and memorability levels (i.e., encoding performance). The R-squared values of neural states associated with novelty and memorability conditions were initially calculated at each time point (Fig. 3Cleft) and subsequently averaged over 10-second intervals preceding and following event boundaries (Fig. 3Cright). To determine the statistical significance of the encoding performances, either at individual time points or as averages, a null distribution comprising 1000 random subspaces with permuted condition labels was generated, against which the observed R-squared values were compared. The resulting neural trajectories within each subspace significantly encoded corresponding memory processes above chance (Fig. 3Cright): co-occurrence novelty (before event boundaries: mean R² difference = 0.38, p < 0.001; after event boundaries: mean R² difference = 0.53, p < .001, from a one-sided permutation test); valence novelty (before: mean R² difference = 0.31, p < 0.001; after: mean R² difference = 0.56, p < 0.001); and memorability (before: mean R² difference = 0.34, p < 0.001; after: mean R² difference = 0.30, p < .001). These findings are hippocampus specific and not merely the results of our analysis method. Unlike those in the hippocampal subspaces, neural states in the dorsomedial prefrontal cortex (dmPFC) subspace were arranged along a coding axis based on relationship valence, exclusively before event boundaries, suggesting valence coding of the dmPFC³⁹ (Supplementary Fig. 3B). Our analysis yielded robust average-response trajectories across participants within each subspace (Fig. 4A), indicating that the reconstructed subspaces captured the dynamics of novelty encoding and memory formation during movie viewing. Notably, while we discretized both novelty encoding and memory formation into discrete levels for our TDR analysis, our results do not indicate whether the hippocampus processes novelty/memorability through discrete or continuous coding mechanisms, which requires further investigation.

Fig. 4. Alignment of hippocampal subspaces encoding two novelty types.

A Neural trajectories representing average population responses for each condition. (Top) Conditions sorted by co-occurrence novelty levels within the co-occurrence novelty subspace. (Bottom) Conditions sorted by valence novelty levels within the valence novelty subspace. The color coding of the lines corresponds to each co-occurrence and valence novelty level. The black dots indicate 10 seconds before the event boundaries. Neural trajectories were constructed using temporally smoothed neural data for clear visualization. (Inset) Examples at a selected time point. B Alignment scores between novelty subspaces were assessed by calculating the cosine similarity between PCs, revealing the nonorthogonality of the first two PCs (PC 1: p = 0.012, PC 2: p = 0.005, from a one-sided permutation test). The gray area indicates ± one s.d. from the mean alignment score of the null distribution. C Dynamics of co-occurrence novelty encoding within the valence novelty subspace. The red lines represent the mean R-squared difference between the observed and null distribution. The gray area indicates ± 2 s.d. of differences. D To quantify the encoding performance for each novelty, average R-squared values were computed across 10 time points preceding and following event boundaries within both novelty subspaces. Significant encoding of both novelties was found on the first PC within each subspace, exclusively following event boundaries (from a one-sided permutation test). The gray dots represent the observed encoding performance compared to the null encoding performance.

The hippocampal subspaces for two types of novelty encoding partially overlap

We hypothesized that the coding axes for different novelty types would align across hippocampal subspaces, suggesting integrated processing of multiple forms of novelty (hypothesis 1, Fig. 1A). Such alignment would indicate that nonorthogonal subspaces contribute to integrative information processing within the hippocampus⁴⁰, facilitating coordinated neural representation. Specifically, we predicted that the two types of novelty would be encoded along an aligned coding axis between the neural subspaces, reflecting shared computational components. To examine this alignment, we measured i) the alignment score between the two novelty subspaces and ii) evaluated the encoding performance for each novelty type within the other type’s subspace. Furthermore, we determined the number of dimensions implicated in the shared coding axis by evaluating encoding performance across subspace dimensions.

We computed the cosine similarity between the eigenvectors of the principal components (PCs) of the two novelty subspaces as the alignment scores. The first (cos θ = 0.62, p = 0.012, from a one-sided permutation test) and second PCs (cos θ = 0.36, p = 0.005) were found to be nonorthogonal, indicating coordinated representations between the two types of novelty encoding (Fig. 4B). To further explore the dynamics of coordinated representations, we evaluated the encoding performance across time. We found that co-occurrence novelty was encoded in the first PC of the valence novelty subspace, particularly following event boundaries (Fig. 4C, D; R² difference = 0.15, p = 0.033, from a one-sided permutation test). Similarly, valence novelty was encoded in the first PC of the co-occurrence novelty subspace after event boundaries (Fig. 4D; R² difference = 0.08, p = .049, from a one-sided permutation test). This result indicates that the neural population combinations required to compute each novelty level are overlapping. Notably, co-occurrence novelty was exclusively encoded along the first PC of the valence novelty subspace, with no significant encoding observed in other PCs (Supplementary Fig. 4A, B). Valence novelty was also primarily encoded along the first PC of the co-occurrence novelty subspace and the other PCs showed no significant encoding (Supplementary Fig. 4C, D), indicating alignment between the first PC of the co-occurrence and valence novelty subspaces (novelty coding axis). Both co-occurrence novelty (before: R² differences > 0.14, ps < 0.002; after: R² differences > 0.10, ps < 0.007) and valence novelty (before: R² differences > 0.14, ps < 0.001; after: R² differences > 0.28, ps < 0.001) were encoded in their respective subspaces, regardless of the number of subspace dimensions (Supplementary Fig. 4A, C). Furthermore, we did not find evidence of coordinated representation between the two types of novelty, when analyzing the neural data with a time lock at event onset (i.e., the start of the event), indicating that the coordinated representation of different novelty types may be specifically associated with event boundaries rather than occurring throughout the event (Supplementary Fig. 5). It should be noted that these results do not imply that meaningful information is predominantly captured by the first two PCs only within the subspaces, with the remaining PCs lacking significance. On the contrary, including the remaining PCs clearly improves the encoding of each novelty type within its respective subspace (Supplementary Fig. 4A, C). More specifically, while PC 1 in each novelty subspace encodes both the respective novelty and the counterpart novelty, the subsequent PCs significantly contribute to encoding the respective novelty. In sum, these findings suggest that each form of novelty is processed around event boundaries and that the coordination between the two processes occurs specifically after event boundaries along the aligned coding axis of the two subspaces.

The inverse relationship between computations in novelty and memorability subspaces during memory encoding

Our behavioral results demonstrated an inverse relationship between the degree of novelty of movie events and participants’ memory performance for those events (Fig. 2F). We hypothesized that this behavioral relationship would be reflected in the alignment of the neural subspaces associated with novelty encoding and memory formation (hypothesis 2; Fig. 1B). Computing the alignment score between the first PCs of each novelty subspace (novelty coding axis) and the memorability subspace revealed no significant alignment (co-occurrence novelty – memorability: cos θ = 0.41, p = 0.340; valence novelty – memorability: cos θ = 0.11, p = 0.878, from a one-sided permutation test). However, the two cognitive processes could still rely on shared computational components, albeit with varying levels of importance across them. Thus, we computed the alignment score between the novelty coding axis and the second PC of the memorability subspace and found significant alignment between them (co-occurrence novelty – memorability: cos θ = 0.45, p = 0.006; valence novelty – memorability: cos θ = 0.39, p = 0.013), indicating coordinated representations between novelty encoding and memory formation along this shared coding axis (Fig. 5A).

Fig. 5. The linkage between the novelty and memorability subspaces.

A The shared coding axis within the hippocampal subspaces associated with two types of novelty encoding and memory formation processes extends over the first PC of the novelty subspaces and the second PC of the memorability subspace. The ratio of the explained variance of the corresponding PC is shown beside each spatial gradient map. B Correlations between subspace alignment and recall performance (the ratio of recalled events). For both co-occurrence and valence novelty subspaces, greater alignment with the memorability subspace is associated with better recall performance, whereas lesser alignment is associated with poor recall. Each dot represents an individual participant (from a two-sided test). C Dynamics of co-occurrence novelty encoding within the memorability subspace. Co-occurrence novelty encoded in the second PC of the memorability subspace, particularly after event boundaries. The red lines represent the mean R-squared difference between the observed and null distribution. The gray area indicates ± 2 s.d. of differences. D Directionality of computations for novelty encoding and memory formation along the shared coding axis. Before event boundaries, neural states corresponding to each memory process were disarranged, but following event boundaries, these states inversely aligned along the shared coding axis, reflecting an inverse relationship between novelty and memorability in behavior. The dashed lines represent + one s.d. determined from the null distribution (t = +3: p = 0.014, t = +4: p = 0.008, from a one-sided permutation test).

What are the behavioral implications of the alignment between the novelty and memorability subspaces? The hippocampus encodes information by evaluating whether current inputs match (i.e., familiarity) or mismatch (i.e., novelty) previous experiences⁴¹. We hypothesized that the degree to which this hippocampal subspace for novelty encoding aligns with memory formation mechanisms would be positively correlated with the strength of the resulting memory, thereby enhancing recall performance. To test this hypothesis, we computed the alignment score between novelty and memorability subspaces for each participant and assessed the correlation between these alignment scores and their recall performances. Our results revealed a significant positive correlation, demonstrating that participants exhibiting greater alignment between these subspaces performed better in narrative recall tasks (Fig. 5B; co-occurrence novelty – memorability: p = 0.43, p = 0.035; valence novelty – memorability: ⍴ = 0.46, p = 0.023, from a two-sided test). Further analyses, which explored the relationships between permuted subspace alignments and recall performances also supported the robustness of these relationships (Supplementary Fig. 6A, B). This relationship provides evidence for the functional relevance of the observed hippocampal subspace alignment in supporting effective memory formation.

Next, we investigated the timing of when novelty encoding and memory formation processes align. Our findings suggest that both types of novelty were represented in the memorability subspace following event boundaries, with strong evidence for co-occurrence novelty (R² difference = 0.23, p = 0.009, from a one-sided permutation test) and a marginal trend for valence novelty (R² difference = 0.10, p = 0.077), but not preceding them (co-occurrence novelty: R² difference = 0.03, p = 0.212; valence novelty: R² difference = −0.04, p = 0.892) (Fig. 5C, Supplementary Fig. 7A, B). Additionally, memorability levels were encoded in the shared coding axis of the two novelty subspaces after event boundaries (Supplementary Fig. 7C, D). However, further analyses did not show this coordinated representation around the event onset or event midpoint (Supplementary Fig. 8). These findings indicate that the coding axis of the hippocampal subspace dedicated to novelty encoding aligns with that of the subspace responsible for memory formation and that their coordination occurs specifically after event boundaries.

We further examined the directional alignment of neural states along the shared axis between their neural subspaces. To assess this within the neural dynamics, we projected average neural responses for each novelty and memorability condition onto the shared coding axis within their respective hippocampal subspaces using the first PC of novelty subspaces and the second PC of the memorability subspace. We then computed the Spearman rank correlation between the projected values of the neural trajectories along these axes across memorability and each novelty level at each time point to measure the directionality between novelty and memorability computations. After event boundaries, the neural states corresponding to novelty and memorability levels were aligned in opposite directions (t = +4 seconds; the average of directionality: p = −.62, p = .008, from a one-sided permutation test) (Fig. 5D; Supplementary Fig. 9A for each type of novelty). Each process was also reflected in its respective neural trajectories following event boundaries (Supplementary Fig. 9B–D). However, while the directionality remained positive 4 seconds before event boundaries, indicating alignment in the same direction, neural states corresponding to each novelty and memorability level were not orderly arranged along the shared coding axis (Fig. 5D, Supplementary Fig. 9). This finding suggested that initially misaligned neural states rearrange in opposite directions along the shared coding axis of the three subspaces (i.e., two types of novelty and memorability) after event boundaries, reflecting memory performances.

The relationship between hippocampal subspaces during memory encoding and memory retrieval

We thus far have demonstrated the coordinated representations of novelty encoding and memory formation during encoding periods (i.e., movie viewing). However, previous studies have also suggested that shared neural representations in the hippocampus occur during both memory encoding and subsequent retrieval of naturalistic events^25,42–46. To examine the computational characteristics of these common neural representations associated with memory encoding and retrieval, we compared hippocampal subspaces during movie viewing and free recall. We hypothesized that the memorability and memory retrieval subspaces would be aligned, indicating shared neural computations between memory encoding and retrieval processes (hypothesis 3, Fig. 1C). Due to the limited number of events recalled compared to those in the movie, the recalled events were categorized into six memorability levels. Our analysis focused on the period from −6 seconds to +14 seconds around the recall onset of events, as memory retrieval primarily occurs at the beginning of recall rather than at its end³⁶ (Fig. 2A). We identified the memory retrieval subspace by employing TDR on the neural data acquired from narrative recall tasks and calculated the alignment scores between the two memory subspaces. The results showed partial overlap between the memorability subspace during movie viewing and the memory retrieval subspace during recall (memory coding axis) (PC 1: cos θ = 0.56, p = 0.036; other PCs: cos θ < 0.21, from a one-sided permutation test; Fig. 6A, Supplementary Fig. 10A), indicating mutual computations between the memory encoding and retrieval processes.

Fig. 6. Neural subspace associated with memory retrieval.

A The first PC of the memory retrieval subspace is aligned with the first PC of the memorability subspace but not with the first PCs of the novelty subspaces (from a one-sided permutation test). B Dynamics of the memory retrieval process. Hippocampal states within the memory retrieval subspace were arranged in order along the memory coding axis immediately before the onset of recall. The red lines represent the mean R-squared difference between the observed and null distribution. The gray area indicates ± 2 s.d. of differences (t = −3 seconds; R² difference = 0.42, p < 0.01, * p < 0.05, from a one-sided permutation test). C In contrast to the memorability subspace (Co-occurrence novelty: p = 0.006, Valence novelty: p = .013, from a one-sided permutation test), the second PC of the memory retrieval subspace did not align with the first PCs of the novelty subspaces (Co-occurrence novelty: p = 0.594, Valence novelty: p = 0.295, from a one-sided permutation test). The gray dots represent each of the 1000 cosine similarities obtained from the null distribution. The diamonds indicate the cosine similarity obtained from the observed subspace. The dashed lines represent the threshold of chance level (p = 0.05) determined from the null distribution.

We further hypothesized that neural states in the retrieval subspace do not remain constantly linked to the memorability subspace but rather momentarily align when internal cues prompt the retrieval of previously encoded information⁴⁷. To assess the dynamics, we projected the average neural response from each of the nine conditions onto the memory retrieval subspace across time, ensuring methodological consistency. The encoding performance of neural trajectories within the retrieval subspace was evaluated by projecting neural states onto the memory coding axis associated with the memorability subspace. Notably, three seconds before initiating the recall of each movie event, the neural states within the memory retrieval subspace aligned along the memory coding axis (Fig. 6B; t = −3 seconds; R² difference = 0.42, p < .01, from a one-sided permutation test), indicating a distinctive onset of memory retrieval processes within the retrieval subspace. However, following the termination of recall, the neural states associated with retrieval processes became misaligned along the memory coding axis (Supplementary Fig. 10B). Thus, these results revealed that the memory retrieval subspace successfully captured the temporal dynamics of the retrieval process, aligning it with the memory encoding process.

While we observed alignment between the novelty and memorability subspaces during movie viewing, prior findings suggested that novelty detection is confined to the memory encoding phase^38,48, raising questions about the generalizability of this relationship to the retrieval process. The alignment between the first PC of the retrieval subspace and the novelty coding axis was not statistically significant (Fig. 6A, co-occurrence novelty: cos θ = 0.27, p = 0.774; valence novelty: cos θ = 0.15, p = 0.906, from a one-sided permutation test). Further analysis of the second PC of the retrieval subspace also yielded no significant alignment with the novelty coding axis (co-occurrence novelty: cos θ = 0.06, p = 0.594; valence novelty: cos θ = 0.12, p = 0.295, from a one-sided permutation test), in contrast to the memorability subspace (Fig. 6C). These results indicate that while processing novelties across multiple attributes influences the formation of memories of narrative contents during encoding, the underlying mechanism for novelty processing does not significantly contribute to the memory retrieval process.

The novelty and memorability subspaces align with the canonical axes of the hippocampus

Our findings showed that functionally defined subspaces within the hippocampus successfully captured the dynamic alignment of memory-related processes. Then, how are these functional subspaces related to the intrinsic dimension of the hippocampus? Previous studies have identified the canonical space along the longitudinal axis of the hippocampus using resting-state functional connectivity or gene expression methods^49,50. We hypothesized that the novelty and memorability subspaces identified in our study bore a systematic relationship to the canonical space within the hippocampus, a principal organizational framework previously described^49–51. To assess this relationship, we identified the hippocampal canonical space by applying unsupervised dimensionality reduction (i.e., PCA) to the entire time series of neural data collected during movie viewing and narrative recall. Consistent with prior work⁴⁹, the principal axes of the hippocampal canonical space exhibited a spatial distribution extending along the longitudinal axis of the hippocampus (Fig. 7Acenter).

Fig. 7. Overview of hippocampal subspaces for novelty encoding and memory processes.

A Canonical space and functional subspaces of the hippocampus. The first three canonical components of the hippocampal distributed along the longitudinal axis (center). (i) The memorability subspace was found to correlate with the first PC of the canonical space, following the longitudinal axis. While the memory retrieval subspace was aligned with the memorability subspace, it did not exhibit a similar alignment with the first PC of the canonical space. (ii) Partial overlap was observed between the novelty subspaces along the longitudinal axis within the hippocampus, correlating with the second PC of the hippocampal canonical space. (iii) Both novelty subspaces are linked to the memorability subspace along the shared coding axis (Fig. 6C). (iv) No significant links were found between the memory retrieval subspace and either novelty subspace (Fig. 6C). The black lines indicate the statistically significant alignments of two spaces, whereas the gray lines denote the absence of statistically significant alignment. B The relationship between canonical PCs and each memory subspace. PC 1 in the memorability subspace was significantly correlated only with canonical PC 1 (p = 0.001, from a one-sided permutation test), while PC 1 in each novelty subspace was significantly correlated only with canonical PC 3 (Co-occurrence novelty: p = 0.022, Valence novelty: p = 0.036, from a one-sided permutation test).

We found that the first PC of the canonical space significantly aligned with the first PC of the memorability subspace (cos θ = 0.85, p = 0.001, from a one-sided permutation test), but there was no significant alignment with the first PC of the memory retrieval subspace (cos θ = 0.65, p = 0.69) (Fig. 7A). Additionally, the third PC of the canonical space significantly correlated with the first PC of the novelty subspaces, showing significant relationships with both co-occurrence (cos θ = 0.34, p = 0.022) and valence novelty (cos θ = 0.35, p = 0.036) (Fig. 7A). Furthermore, the spatial gradients of the eigenvectors displayed similarities between each of the novelty and memorability subspaces and the canonical space (Fig. 7A). In contrast, no significant relationships were observed in the other scenarios (Fig. 7B): the co-occurrence novelty (ps > 0.11) or valence novelty (ps > 0.273) with other PCs, and memorability (ps > 0.581) with other PCs. The positive bias observed in the null distribution of the canonical PC 1 may indicate the correlation between the baseline dynamics in the canonical space and the functional subspaces. Similar positive biases in alignment indices have been previously reported^19,52 when using randomly shuffled conditions. These results suggest that the hippocampal subspaces identified in our study are systematically organized along the longitudinal axis, revealing an intricate functional organization within the hippocampus.

Discussion

To explore the computational mechanisms underlying naturalistic memory encoding and retrieval in the hippocampus, we identified neural subspaces associated with three distinct memory processes. Using a variant of the TDR method, we analyzed fMRI BOLD responses from human participants while they watched a movie and later recounted its narrative. First, we found partial overlap between hippocampal subspaces involved in encoding novelty from two crucial attributes for comprehending social narratives in movie–character co-occurrences and relationship valence. Moreover, a shared coding axis between these two novelty subspaces aligned with the memorability subspace, suggesting a common computational mechanism underlying both novelty encoding and memory formation. Notably, the neural states corresponding to novelty and memorability levels aligned inversely with this shared coding axis following event boundaries, mirroring the observed recall performance across different novelty levels. Additionally, the memory retrieval subspace, which captures the temporal dynamics of memory retrieval processes, showed alignment with the memorability subspace. However, the alignment between novelty and memory subspaces was specific to memory encoding, not retrieval. Finally, the functionally relevant subspaces identified in our study corresponded to the canonical space of the hippocampus, emphasizing the intrinsic spatial organization of hippocampal activities.

Memory encoding involves various cognitive processes, including encoding novelty from incoming information and transforming it into memories, which are facilitated by the hippocampus^3–12,50,53. While these individual processes during naturalistic experiences have been previously studied, the coordination of these processes within the hippocampus has remained elusive. Our approach, which quantifies novelty and memorability in naturalistic stimuli, enabled us to identify subspaces associated with these memory processes and their partial alignment. While interpreting the temporal dynamics of encoding performances related to memorability remains challenging (Fig. 3C), the subspace alignment in predicting subsequent recall behavior (Fig. 5B), and the high correlation between the subspace and the hippocampal canonical component (Fig. 7), suggest the robustness and significance of the observed dynamics. In contrast to prior research, which has focused primarily on neural subspaces associated with sequential processes (e.g., from motor preparation to execution^19,20 or from evaluation to comparison⁵²), our study examines the coordination of simultaneous cognitive processes. To comprehend and remember the narratives of the movie, participants must simultaneously process and integrate various types of social information, such as the co-occurrence and valence of social interactions, potentially leading to the observed alignment of subspaces for parallel cognitive processes. This shared computation manifests an inverse relationship between novelty encoding and memory formation, likely influenced by schema-driven unexpectedness⁵⁴, which might negatively affect memory encoding⁵⁵.

Amidst the continuous flow of information within movies, we found a distinct transition of neural states in the hippocampal subspaces around event boundaries. These boundaries denote moments of discontinuity in experiences, potentially triggering the organization of current information into memories^25,33,34. Numerous prior studies have reported neural activities coinciding with event boundaries, with recent research suggesting that the reactivation of previously encountered events contributes to consolidating knowledge at these boundaries^28,56. Our findings offer insights into how event boundaries trigger the alignment of hippocampal states involved in both novelty encoding and memory formation processes along a shared coding axis. The observed arrangement of neural states along the shared coding axis within both novelty and memorability subspaces following event boundaries suggests that the hippocampus reconfigures its population dynamics to represent novelty and memorability levels simultaneously. This coordination of multiple types of information processing (e.g., character association and interaction context, which we measured) aligns with hierarchical models of information integration linked to memory formation^25,57. In particular, the hippocampus is well known for its role in maintaining an event model that combines multiple elements, including agents and objects, as well as time, space, and tasks, and organizing them into episodic memories⁵⁸. Our study suggests that this integration process occurs through the alignment of hippocampal functional subspaces, specifically after the termination of current events. This alignment may involve the offline reactivation of past events at event boundaries, providing a potential mechanistic explanation for how the brain structures continuous experience into discrete memory episodes⁵⁶.

The relationship between memory encoding and retrieval processes in the hippocampus has long been a focus in the field of memory research^11,38,59. Previous studies have often attempted to identify subregions involved in both processes⁶⁰ or assign them exclusively to one process, implying functional specialization within the hippocampus^61,62. While previous research has focused on the specific functional roles of hippocampal subregions, such as anterior versus posterior divisions^63,64 or CA1/CA3/DG subfields^65,66, our study offers a complementary perspective. By analyzing population dynamics across the entire hippocampus, we reveal low-dimensional neural subspaces where memory encoding and retrieval share computational components that influence memorability. The temporally selective alignment of hippocampal states within the retrieval subspace along the memory coding axis at the onset of verbal recall suggests that this dynamic alignment between the two processes serves as a potential mechanism for retrieving previous experiences^11,36,67. Because the axes of the functional subspace represent computational components for memorability levels (e.g., from low to high) rather than event-specific representations (e.g., neural patterns for individual events), our findings suggest that this alignment operates in an event-general way, highlighting the shared computational processes that determine event memorability during both memory formation and retrieval. Notably, novelty processing and memorability computations were coupled during encoding, but this coordination did not extend to retrieval. This dissociation indicates that integrating novel information is a selective process confined to memory formation that does not influence the retrieval of previously formed memories^38,47.

Dimensionality reduction methods, employed to identify neural subspaces in electrophysiological data, have been highly successful in revealing computational mechanisms across diverse cognitive domains^{18–21,30,31,68}. Only recently have such methods been applied to fMRI data utilizing a geometrical approach owing to the limitations imposed by the slow dynamics inherent in the fMRI signal⁶⁹. To address this challenge, we focused on cognitive processes (e.g., memory encoding and retrieval), which unfold at a slower pace than sensory or motor processing, and considered the specific temporal window during which memory processes occur in naturalistic experiences (e.g., event boundaries). Our extended application of neural subspace analysis tailored for fMRI data reveals the dynamic interplay and computational principles of coordinated memory processes.

Multivariate pattern analysis in fMRI studies, including pattern classification and pattern similarity analysis^70–72, has proven to be effective in delineating neural correlates of individual cognitive processes, such as forming^73,74 or retrieving memories⁷⁵. However, in naturalistic experiences characterized by a continuous stream of complex stimuli, a myriad of cognitive processes unfold concurrently. Subspace analysis has emerged as a particularly useful tool for elucidating the interplay among these multiple cognitive processes. Furthermore, it holds promise in providing insights into higher cognition in humans, such as reasoning and judgment, which entail intricate interactions among a broad range of individual processes⁷⁶.

While our study provides insights into the coordination of memory processes within the hippocampus, several limitations should be addressed in future research. First, our study focused on novelty in social information, but other types of novelty (e.g., spatial, temporal, or goal-oriented information) also significantly influence memory formation⁷⁷. Future studies should investigate how these diverse forms of novelty relate to and impact memory formation processes. Second, our findings are based on a free narrative recall paradigm^28,29, and their generalizability to recognition memory paradigms⁷⁸ remains to be determined. Third, our results pertain to immediate memory retrieval, leaving open the question of whether these findings extend to remote memory retrieval, such as recall after sleep. Nevertheless, this study opens avenues for future investigations into the brain networks associated with memory processes. Applying advanced analysis methods, such as communication subspace analysis⁷⁹, could elucidate how neural populations in the hippocampus coordinate with those in other medial temporal lobe (MTL) regions known to compute familiarity and novelty signals^80–83 to support naturalistic memory encoding and retrieval. Furthermore, our findings on narrative memorability raise questions about the relationship between different forms of memorability. For example, how does narrative memorability relate to visual image memorability, which has been proposed to be computed in the visual cortex-MTL-prefrontal cortex network^84,85? These questions, along with some limitations of our current study, present intriguing avenues for future research, which could provide a more comprehensive understanding of the mechanisms underlying naturalistic memory encoding and retrieval across different modalities and cognitive domains.

Methods

This research complies with all relevant ethical guidelines and was approved by the Sungkyunkwan University Institutional Review Board (2018-05-005).

Participants

Twenty-four participants aged between 19 and 25 (M = 22, SD = 2.18; 6 females) from the Sungkyunkwan University community were included in the analyses. Initially, 32 fluent Korean speakers with normal or corrected-to-normal vision and hearing were recruited and provided informed consent in accordance with the Institutional Review Board guidelines of Sungkyunkwan University. They received monetary compensation for participating in the fMRI scans. Data from eight participants were excluded due to excessive head motion, with more than 5% of the data exceeding the threshold of FD > 0.5 in at least one fMRI run. The majority of the excluded participants were naive to verbally recounting inside an MRI scanner, which likely contributed to their excessive head movement during the recall task. We did not conduct sex and gender analyses, as we did not have specific hypotheses regarding the influence of these variables on memory processes.

Task structure

All the tasks were performed in an MRI scanner (Fig. 2A) and were presented and controlled using MATLAB with the Psychophysics Toolbox⁸⁶. Participants completed six fMRI runs, including separate movie watching and narrative recall runs for three episodes.

Movie-viewing task

Participants watched the first season of the Korean YouTube web drama ‘Love Playlist’ (available at https://www.youtube.com/playlist?app=desktop&list=PLS–ClexQbQ1lg6TttQTcE3a60T_noSnh), focusing on the depicted social interactions. No participants had prior exposure to this series, and no behavioral responses were required during the fMRI scan. The series portrays romantic and friend relationship dynamics among six university students, including evolving relationships, romantic entanglements, breakups, love triangles, and unrequited love. The entire series was divided into three episodes of approximately 14 minutes each, with a 10-second blank screen preceding and a 16-second blank screen following each episode. To provide continuity, each episode began by replaying the final 30 seconds from the previous one, and data from these repeated segments were excluded from analysis. The audio was delivered via MR-compatible headphones (OptoACTIVE II, Optoacoustic Ltd.).

Narrative recall task

Before watching the movie, participants viewed an unrelated 90-second movie clip for a recall exercise and received feedback such as “Recount character names clearly.” After each episode of the series, they were instructed to verbally recount the narrative chronologically with maximum detail, with an emphasis on social relationships. The recall task began with a 10-second blank screen, followed by a fixation cue indicating the onset of the recall task. Participants were required to recall each episode in detail for a minimum of five minutes, with the option to revisit earlier events if desired. We emphasized that while maintaining chronological order was important, providing detailed recall was even more crucial. The recall task was manually terminated 10 seconds after the participants stated, “I’m done.” On average, participants recalled each episode for 422 seconds (s.d. across episodes and participants = 144 sec, range = 180 ~ 866 sec). Recalls were recorded via an MR-compatible microphone (FOMRI III, Optoacoustics Ltd.).

Movie annotation

Event segmentation

Four independent annotators, naive to the movie, segmented it into distinct events by marking the start and end times of each event as they viewed it. They identified new events when observing changes in topic, location, time, characters, or relationship dynamics, assigning a title to each segmented event and providing their rationale for the observed changes²⁵. Among the 61 events initially identified, only those lasting more than 20 seconds with event boundaries receiving consensus from three or more annotators within a 5-second range were retained for subsequent analyses. This selection process resulted in 44 events, but the first event was excluded from analyses due to its unavailability for novelty measurements, leaving 43 events (mean duration = 48.45 seconds). To further validate our event segmentation protocol, we conducted an additional experiment with an independent group of fifteen participants using the spontaneous protocol, where participants were asked to press a key whenever they recognized the start and end of an event^28,29,87. All event boundaries identified by the annotators were recognized as boundaries by over 60% of the participants in the spontaneous protocol (Supplementary Fig. 2A). Furthermore, consistent with previous findings, we observe increased hippocampal activation at these event boundaries^28,87 (Supplementary Fig. 2B).

Detailed annotation

A trained annotator who did not participate in the fMRI experiment or event segmentation provided a detailed second-by-second annotation of the movie. This annotation included 1) identifying socially interacting characters as senders or receivers of information, 2) describing characters’ actions, 3) describing their emotions, and 4) providing scene descriptions. For example, in a pub scene where character A talks with characters B and C, with character B laughing and character C showing no interest, the annotation included the following details: character A was designated as the sender and engaged in talking; characters B and C were identified as receivers; character B’s action was laughing, and their emotion was pleasant; character C’s emotion was described as bored; and the scene was described as ‘character A, B, and C are talking in the pub’. For consistency, synonymous actions and emotions (e.g., cry and sob) were standardized (e.g., cry).

Recall transcript

Audio recordings from the recall task were transcribed into text files with millisecond-level word timings using NAVER CLOVA speech recognition⁸⁸. These transcripts were then organized into words per second, aligning with the TR of the fMRI data. The first author segmented the recall transcripts into events corresponding to the movie event structure and recorded the start and stop times for recounting each movie event.

Novelty and memorability of movie events

Measure of novelty

The main narrative of the movie employed in our study revolves around dynamic social interactions among the characters. These social interactions are categorized into two components: the co-occurrences of characters, which reflect their association, and the valence of their interactions, denoting the context. For instance, the frequent appearance of character A alongside character B would indicate a strengthened association between them. Similarly, if character C consistently interacts with character D in a positive manner, the context of their interaction is deemed positive. Leveraging these components, we quantified the degree of novelty for each movie event by assessing co-occurrence and valence novelty.

$$\text{Co}-{\text{occurrence}}_{\text{AB},\text{t}}=\left\{\begin{array}{c}\hfill 1ifappearedtogether\hfill \\ \hfill 0ifnot\hfill \end{array}\right),$$ 1

where A and B represent individual characters, and t denotes each time point.

Initially, we computed directional co-occurrence and valence scores between every pair of characters (co-occurrence and valence matrices) for each event. We counted the frequency of characters appearing together during each event to determine the co-occurrence score. Specifically, when characters A (sender) and B (receiver) were both present in the annotation, we assigned a directional co-occurrence score of 1 from A to B at time t (Co-occurrence_AB,t = 1). Conversely, the directional co-occurrence scores (Co-occurrence_BC,t and Co-occurrence_CB,t) were set to 0 for characters not appearing together.

$${\text{C}}_{\text{AB},\text{ev}}=\frac{{\sum }_{t=start}^{end}{\text{Co}-\text{occurrence}}_{\text{AB},\text{t}}}{Durationoftheevent}$$ 2

$${\text{Co}-\text{occurrence}}_{\text{ev}}=\left[\begin{array}{cccc}\hfill 0\hfill & \hfill C_{AB,ev}\hfill & \hfill \cdots \hfill & \hfill C_{AF,ev}\hfill \\ \hfill C_{BA,ev}\hfill & \hfill 0\hfill & \hfill \cdots \hfill & \hfill \cdots \hfill \\ \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill 0\hfill & \hfill C_{EF,ev}\hfill \\ \hfill C_{FA,ev}\hfill & \hfill \cdots \hfill & \hfill C_{FE,ev}\hfill & \hfill 0\hfill \end{array}\right],$$ 3

where Co-occurrence_ev is a 6 x 6 matrix (for six characters in the movie) and ev denotes an individual event.

The eventwise co-occurrence score (co-occurrence matrix) was then calculated by dividing the sum of directional co-occurrences during an event by the event duration. For example, if characters A (sender) and B (receiver) appeared together for 16 seconds during a 42-second long event, the co-occurrence score_AB,ev was computed as 16/42 = 0.38.

$${Novelty}_{Co-occurrence,ev}=1-corr\left({\sum }_{e=1}^{ev-1}{\text{Co}-\text{occurrence}}_{\text{e}},{\text{Co}-\text{occurrence}}_{\text{ev}}\right)$$ 4

Subsequently, co-occurrence novelty for each event was quantified by calculating 1 − the Pearson correlation between the cumulative co-occurrence matrices from prior events across all the episodes watched up to the current event and the co-occurrence matrix of the current event. This reflects the deviation of the current event’s character co-occurrence patterns from the cumulative patterns in previous events.

We assessed moment-to-moment valence information between characters by calculating sentiment scores for annotated action and emotion words using the Korean Sentiment Word Dictionary⁸⁹ (KSWD), which ranges from −2 (strongly negative) to +2 (strongly positive).

$${Valence}_{AB,t}=\sum \text{Sentiment scores of action and emotion words}$$ 5

We aggregated these sentiment scores for each TR for all action and emotion words. For example, if character A annoyed character B and character B was irritated at time t, we assigned a sentiment score of −1 for the relationship between characters A and B (Valence score_AB,t = −1) and between characters B and A (Valence score_BA,t = −1) based on the sentiment scores of the words ‘annoy’ and ‘be irritated’ from the KSWD.

$${\text{V}}_{\text{AB},\text{ev}}=\frac{{\sum }_{t=start}^{end}{\text{Valence}}_{\text{AB},\text{t}}}{Durationoftheevent},$$ 6

$${\text{Valence}}_{\text{ev}}=\left[\begin{array}{cccc}\hfill 0\hfill & \hfill V_{AB,ev}\hfill & \hfill \cdots \hfill & \hfill V_{AF,ev}\hfill \\ \hfill V_{BA,ev}\hfill & \hfill 0\hfill & \hfill \cdots \hfill & \hfill \cdots \hfill \\ \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill 0\hfill & \hfill V_{EF,ev}\hfill \\ \hfill V_{FA,ev}\hfill & \hfill \cdots \hfill & \hfill V_{FE,ev}\hfill & \hfill 0\hfill \end{array}\right],$$ 7

where the Valence_ev is a 6 x 6 matrix calculated by dividing the sum of sentiment scores over an event by the event’s duration.

$${Novelty}_{Valence,ev}=1-corr\left({\sum }_{e=1}^{ev-1}{\text{Valence}}_{\text{e}},{\text{Valence}}_{\text{ev}}\right)$$ 8

Valence novelty was assessed using the same method as co-occurrence novelty applied to the valence matrices. We assumed that the processing of valence information for a specific relationship in the current event might be independent of the preexisting valence information from other relationships. Hence, we confined the measurement of valence novelty to the relationships present in the current event. Given that novelty computation requires two successive events (i.e., past and current), the first event of each episode was excluded from analysis.

Measure of memorability

We evaluated the memorability of each movie event by counting the number of words in the participants’ recall transcripts that correctly matched the words in the detailed scene descriptions from the annotation. Initially, the words in the recall transcript from all participants were mapped to the corresponding words in the scene descriptions. To ensure consistent evaluation of recall performance and account for semantic variations in participants’ responses, we implemented a standardization procedure for word choices used by participants during recall. This process involved manually substituting varied terms (e.g., Facebook, Instagram, SNS, etc.) with corresponding common terms from the scene description (e.g., social media). Our analysis revealed that, on average, 19.73% of the words recounted by participants were standardized (s.d. across participants = 2.67%; Supplementary Fig. 11A). To validate this process, we used the fastText algorithm⁹⁰ to extract word embeddings and measured the word embedding similarity between the original words recounted by participants and the standardized terms (e.g., Facebook to social media). The results indicated that the standardized terms accurately captured the intended meaning of the participants’ original words (Supplementary Fig. 11B; mean embedding similarity = 0.33, p < 0.001, one-sided permutation test).

$${Memorability}_{ev}=\frac{N_{\text{correct words corresponding to the event}}}{N_{\text{total words in the event}}}$$ 9

Then, for each event and participant, we calculated the number of correctly recalled words and normalized it by the total number of words in the corresponding event description from the annotation, accounting for variations in the amount of content across events. This procedure provided a quantitative measure of the degree of memorability for each event.

Imaging procedure

Acquisition

Neuroimaging data were acquired at the Center for Neuroscience Imaging Research of the Institute for Basic Science using a 3 Tesla Siemens MAGNETOM Prisma equipped with a 64-channel head coil. Functional images were obtained through a T2*-weighted echo-planar imaging (EPI) sequence (TR = 1000 ms, TE = 30 ms, FOV = 240 mm, multiband factor = 3, in-plane acceleration factor (iPAT) = 2, and 3 mm isotropic voxel size with 48 slices covering the whole brain). Additionally, high-resolution anatomical images were collected using a T1-weighted magnetization-prepared rapid gradient echo (MPRAGE) sequence (TR = 2200 ms, TE = 2.44 ms, FOV = 256 mm, and 1 mm isotropic voxel size).

Preprocessing

The neuroimaging data were preprocessed using fMRIprep⁹¹ (version 21.0.2) with default settings, which included corrections for head motion and registration of each participant’s brain to the MNI152NLin2009cAsym template for spatial normalization. Subsequently, we regressed out nuisance variables⁹², encompassing six motion parameters and their derivatives, global signals, framewise displacement values, six components from anatomical component correction (aCompCor) obtained from fMRIprep, and polynomial regressors up to the second order. Finally, the BOLD time series of the movie-viewing and narrative-recall runs were scaled, spatially smoothed (FWHM = 5 mm), and z scored within each individual run.

Regions of interest

The hippocampal voxels were identified using the Brainnetome atlas⁹³. We conducted the same analysis across other cortical regions, such as the parahippocampal cortex, entorhinal cortex, perirhinal cortex, superior temporal gyrus, superior temporal sulcus, temporoparietal junction, posterior medial cortex, dorsomedial prefrontal cortex, ventromedial prefrontal cortex, visual cortex, and auditory cortex. However, consistent results were observed only in the hippocampus (Supplementary Fig. 12). Therefore, our reporting focused exclusively on results derived from the hippocampus.

Identification of a canonical space and subspaces for memory processes

We delineated a canonical space and three subspaces for memory processes, each associated with 1) novelty encoding, 2) memory formation, and 3) memory retrieval processes.

Canonical space

To establish the canonical space, we first aggregated the neural data collected during movie watching and narrative recall, involving 24 participants, 664 voxels, and t TRs (t represents length of movie watching and narrative recall data in total), concatenated across three movie-viewing runs and across participants. This aggregation produced a two-dimensional data matrix (M_664,92119). Subsequently, principal component analysis (PCA) was applied to this entire time series of neural data, represented as M_n,92119, where n indicates the number of PCs. Our analysis focused on the first three PCs, which together accounted for 12% of the total variance within the hippocampus.

Subspace for novelty encoding

To identify the neural subspace associated with novelty encoding, we analyzed neural data around event boundaries spanning from −6 to +14 TRs (21 TRs) for 43 events (M_24,43,664,21). We categorized movie events into nine distinct conditions based on the combination of three levels of co-occurrence and valence novelty using one-third percentiles. Subsequently, a finite impulse response (FIR) model was applied to each of the nine conditions, yielding an FIR time series (M_24,9,664,21, where 9 represents nine novelty conditions) for each participant. The FIR estimation step aimed to average neural responses across events within each condition without assuming a specific hemodynamic response function.

Next, we applied multivariate linear regression to the FIR time series to disentangle the neural representations of the two types of novelty. The regression model was defined as follows:

$$R_{v,t}\left(k\right)={\beta }_{v,t}\left(1\right)\cdot \text{co}-\text{occurrence}\left(k\right)+{\beta }_{v,t}\left(2\right)\cdot valence\left(k\right)+intercept$$ 10

Here, R_v,t(k) denotes the FIR-estimated response of voxel v at time t under condition k, where k represents the co-occurrence and valence novelty levels of each condition arranged as [−1, − 1, − 1,0,0,0, + 1, + 1, + 1] and [−1,0, + 1, − 1,0, + 1, − 1,0, + 1], respectively. The regression coefficient β_v,t(n) indicates the extent to which the FIR-estimated response of voxel v at time t is modulated by each novelty level n (i.e., co-occurrence: n = 1, and valence: n = 2). To estimate β_v,t(n), we define a regressor matrix, F_v, for voxel v, consisting of three rows for co-occurrence novelty, valence novelty, and an intercept.

$${\beta }_{v,t}={\left(F_v{F}_v^T\right)}^{-1}{F}_v{R}_{v,t}$$ 11

The resulting beta coefficient (β) matrix, denoted as M_24,3,664,21, incorporates two types of novelty and an intercept. We averaged the β matrix across participants for each novelty type, yielding M_2,664,21. We then applied PCA to the matrix corresponding to each novelty type (M_2,n,21, where n represents the number of PCs). Finally, we identified the two PC-based novelty subspaces using the eigenvectors of each PC. To assess the statistical significance of the identified subspaces, we generated a null distribution of 1,000 random subspaces by permuting the condition labels and altering the co-occurrence (k) and valence (k) elements (e.g., co-occurrence (k): [0,0, − 1, + 1, − 1,0, + 1, + 1, − 1]; and valence (k): [0, − 1, − 1, + 1,0, + 1, − 1,0, + 1]).

Subspace for memory formation

To identify the memorability subspace, we sorted the 43 movie events into nine memorability conditions based on the average memorability scores of movie events across participants and applied an FIR model to the neural data across memorability conditions (M_24,9,664,21, where 9 indicates the nine memorability conditions). The memorability (k) is the memorability level of each condition, normalized by subtracting the mean.

$$R_{v,t}\left(k\right)={\beta }_{v,t}\left(3\right)\cdot \text{memorability}\left(k\right)+intercept,$$ 12

where R_v,t(k) denotes the FIR-estimated response of voxel v at time t under the memorability condition k. We then employed a linear regression model and PCA on the FIR time series utilizing the same approach as described in the identification of novelty subspaces.

Subspace for memory retrieval

Unlike movie events, in which all participants watched identical content, the events recalled varied among them (Supplementary Fig. 1A). Consequently, to accommodate this variability, we recategorized these events into six memorability conditions, each defined using one-sixth percentiles of the memorability scores from each participant’s data. To account for the smaller number of events recalled by participants compared to the total number of events in the movie, we reduced the nine conditions originally applied to movie events to six conditions for the recalled events.

We first analyzed the neural data around the onset of the recall, ranging from −6 to +14 TRs (21 TRs) (M24,i,664,21, where i indicates the number of recalled events by each participant). We then applied an FIR model to the neural data for memorability conditions (M24,6,664,21, where 6 denotes the number of memorability conditions). We applied a linear regression model to the FIR time series (M_24,664,21).

$$R_{v,t}\left(k\right)={\beta }_{v,t}\left(4\right)\cdot {memorability}_{recall}\left(k\right)+intercept,$$ 13

where R_v,t(k) denotes the FIR-estimated response of voxel v at time t under the memorability condition k. The remaining procedure for identifying retrieval subspaces was identical to that used for the novelty subspace, except for the variable of interest, and the number of random subspaces varied. While we generated a null distribution of 1000 random subspaces for the novelty and memorability subspaces, considering the 9! = 362,880 potential combinations from nine conditions, only 100 random subspaces were generated for the memory retrieval subspace due to the reduced number of possible combinations, 6! = 720. For comparison, we repeated the same analysis for the time period around the end of recall, spanning from −6 to +11 TRs. We quantified the explained variance in the three hippocampal subspaces and found that 12 PCs accounted for 80% of the total variance in all subspaces (Supplementary Fig. 3A).

Encoding performance in subspaces

We first projected an averaged FIR time series corresponding to each memory-related process (M_c,664,21, where c represents the number of conditions) onto distinct hippocampal subspaces, generating condition-level hippocampal trajectories (M_c,n,21) (Fig. 4A). When calculating the β matrix to formulate the memory retrieval subspaces, we initially considered six conditions. However, for consistency in assessing encoding performances across all subspaces, we expanded the conditions to nine when projecting neural trajectories onto the memory retrieval subspace.

Using projected neural trajectories (M_c,n,21), we established a one-dimensional axis by connecting the two most distant neural states, agnostic to the conditions, and projected other neural states onto this axis for each TR. We assumed that if an estimated subspace accurately represents a particular process, the neural states projected onto the coding axis would be systematically arranged, reflecting the extent of that process, ranging from low to high. The values of the two most distant states were assigned values of 0 and 1, respectively, and we computed the projected values for the other states between them. Then, we calculated the R-squared values using an ordinary least square regression model.

$$\widehat{S}=\beta \cdot codingregressor+intercept,$$ 14

$$R^2=\frac{\sum {\left({\widehat{S}}_i-\overline{S}\right)}^2}{\sum {\left(S_i-\overline{S}\right)}^2},$$ 15

where S represents the actual projected values of neural states onto the axis and Ŝ represents the estimated values of these neural states. The R-squared values elucidate the extent to which the projected values of neural states could be explained by coding regressors for each novelty (e.g., co-occurrence novelty: [−1, − 1, − 1,0,0, + 1, + 1, + 1]; valence novelty: [−1,0, + 1, − 1,0, + 1, − 1,0, + 1]) and memorability ([−4, − 3, − 2, − 1,0, + 1, + 2, + 3, + 4]), including a constant term.

To assess the statistical significance of the encoding performance in the observed subspace, we compared the observed encoding performance with that of random subspaces and counted the number of null encoding performances that exceeded the observed encoding performance. The encoding performances were computed at each TR (Fig. 3Cleft, 4 C, 5 C, 6B) and averaged over 10 TRs before or after event boundaries for comparison (Fig. 3Cright, 4D).

Alignment scores between hippocampal subspaces

We defined the alignment score as the cosine similarity computed between the eigenvectors of each principal dimension obtained through PCA.

$$cosinesimilarity=\cos \left(\theta \right)=\frac{E_a\cdot E_b}{\mathrm{\mid \mid }{E}_a\mathrm{\mid \mid }{E}_b\mathrm{\mid \mid }},$$ 16

where E represents an eigenvector of a principal dimension. To assess the statistical significance of subspace alignment, we compared the alignment scores between the observed subspaces (e.g., the co-occurrence and valence novelty subspaces) and those from generated random subspaces. We generated a null distribution of 1000 random subspaces for both novelty encoding and memory formation and 100 random subspaces for the memory retrieval process. The statistical significance of the alignment between the memory retrieval subspace and other subspaces was assessed by comparing their alignment scores with those between the observed memory retrieval subspace and random subspaces generated for novelty and memorability.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Author contributions

D.K. and W.M.S. conceptualized the research and designed the experiment. D.K. performed the experiments, conducted all the analyses, and visualized the results. J.K. and S.B.M.Y. assisted with the analysis. D.K. and S.B.M.Y. prepared the original draft. All the authors discussed the results and commented on the manuscript. S.B.M.Y. and W.M.S. supervised the research.

Peer review

Peer review information

Nature Communications thanks the anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Data availability

Behavioral data is available at GitHub, https://github.com/somvid/Hippocampal-subspaces. FMRI data is available at https://openneuro.org/datasets/ds005468. Source data are provided with this paper.

Code availability

Analysis codes are available at GitHub, https://github.com/somvid/Hippocampal-subspaces.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-025-55833-x.