Enhanced Intersubject Correlations during Movie Viewing Correlate with Successful Episodic Encoding

SUMMARY

While much has been learned regarding the neural substrates supporting episodic encoding using highly controlled experimental protocols, relatively little is known regarding the neural bases of episodic encoding of real-world events. In an effort to examine this issue, we measured fMRI activity while observers viewed a novel TV sitcom. Three weeks later, subsequent memory (SM) for the narrative content of movie events was assessed. We analyzed the encoding data for intersubject correlations (ISC) based on subjects’ subsequent memory (ISC-SM) performance to identify brain regions whose BOLD response is significantly more correlated across subjects during portions of the movie that are successfully as compared to unsuccessfully encoded. These regions include the parahippocampal gyrus, superior temporal gyrus, anterior temporal poles, and the temporal-parietal junction. Further analyses reveal (1) that these correlated regions can display distinct activation profiles and (2) that the results seen with the ISC-SM analysis are complementary to more traditional linear models and allow analysis of complex time course data. Thus, the ISC-SM analysis extends traditional subsequent memory findings to a rich, dynamic and more ecologically valid situation.

INTRODUCTION

Understanding how the experience of an episode in our daily lives is transformed into an enduring memory is a critical question for memory research. Although many would agree that understanding memory formation for real-world events is of utmost priority, relatively little cognitive neuroscience research addresses this issue. Instead, the vast majority of experimental protocols used to explore the neural bases of episodic memory formation (Paller and Wagner, 2002; Sanquist et al., 1980) commonly use paradigms in which memoranda are presented as individual items, often single words, devoid of continuous context outside of the laboratory setting (Buckner et al., 2000; Winocur and Weiskranitz, 1976). Although this approach is well suited to isolating transient responses to discrete events, it bears little resemblance to the real-world environment in which memories are formed.

Real-life episodic encoding is the result of continuous, multimodal, overlapping perceptual and cognitive operations within a highly contextualized framework (Bartlett, 1932; Suddendorf and Busby, 2005; Tulving, 1983). Furthermore, the probability of encoding a particular event into memory, even if temporally distinct from neighboring events (as in event-related designs), may be influenced by ongoing cognitive operations that do not conform to event boundaries as defined by the experimental design. Indeed, recent data have revealed that brain activation prior to stimulus presentation is correlated with successful memory formation for the presented item (Adcock et al., 2006; Otten et al., 2002) suggesting that, even within highly controlled stimulus presentation paradigms, recent exposure to list items and other factors are likely to impact upon memory formation. Though the importance of real-life experimental protocols has long been recognized (Cohen, 1996; Hanson and Hanson, 1996; Hanson and Hirst, 1989; Neisser, 1978), it has proven difficult to harness the attributes of naturalistic environments in well-controlled, reproducible laboratory settings (Dudai, 2002).

In an effort to examine memory formation under more real-world conditions, we conducted a subsequent memory experiment that employed complex, naturalistic stimuli. During functional scanning, subjects viewed a novel audio-visual movie consisting of an episode of a TV sitcom in its entirety. A movie was chosen because movies are capable of simulating aspects of real-life experiences by fusing multimodal perception with emotional and cognitive overtones (Eisenstein and Leyda, 1947; Morin, 2005) within a carefully produced context. The use of cinematic material to probe memory can be traced to the early days of cinema (Boring, 1916) but has only infrequently been used since (Carroll and Bever, 1976; Hanson and Hirst, 1989).

Three weeks after the encoding phase, cued recall and recognition memory for the narrative content of the movie was assessed. Each subject's subsequent memory performance was used to reveal memory formation-related activation patterns in the functional imaging data collected during encoding. The analysis approach adopted in this study builds on recent findings suggesting that the time course of activation in many brain regions is strongly correlated across individuals watching the same movie (Hasson et al., 2004) or listening to the same narrative (Wilson et al., 2008). Here, we performed an intersubject correlation (ISC) analysis based on subject's subsequent memory (SM) to reveal brain regions for which the ISC is greater during successful as compared to unsuccessful memory formation. The use of the ISC method combined with a SM paradigm (ISC-SM hereafter) makes no explicit assumptions about the magnitude or shape of the evoked spatiotemporal response pattern of any given brain region. This is in contrast, and complementary to, more traditional SM paradigms that focus on revealing regions showing a difference in the response magnitude during successful as compared to unsuccessful encoding.

Using this approach, we identified brain regions showing enhanced ISC during successful memory formation. Each of these regions showed one of two activation patterns. One activation pattern was seen in the parahippocampal gyrus (PHG), a region that has previously been associated with successful episodic encoding. A different activation pattern was seen in the temporal pole, superior temporal gyrus (STG), medial prefrontal cortex (mPFC), and temporal parietal junction (TPJ). These after brain regions have not typically emerged from more traditional word list-learning paradigms. Interestingly, however, these regions have been implicated in studies examining various processes broadly related to the perception and analysis of social information (for review, see Gallagher and Frith, 2003) that was highly prevalent in the movie stimulus. We also compared the ISC-SM analysis with a standard general linear model (GLM—see Experimental Procedures) and found some, but not total, overlap in the results, suggesting that brain regions showing enhanced ISC during successful encoding, may not always show greater magnitude of activation during those same successful trials. By contrast, other regions, like the inferior frontal gyrus, while showing enhanced magnitude of activation during successful encoding, did not exhibit stronger ISC-SM during those same events. Taken together, these data suggest that the two methods may reveal distinct correlates of memory formation.

RESULTS

Behavioral Measures of Memory

On average, collapsing across all confidence levels, subjects successfully remembered 66.5% ± 2.3% of all test questions 3 weeks after viewing the movie (Figure 1B). To verify that performance on our memory test cannot be deduced simply from the cues themselves, in a separate behavioral study (Furman et al., 2007), we found that subjects (n = 8) who did not view the movie attained a chance hit rate of 53% ± 2% correct (i.e., 41 ± 2 correct answers out of 77; see Figure 1B, dotted line). Separating the analysis across confidence levels revealed that, on average, 71.4% ± 5.8% of cued recall answers, 85.9% ± 3.6% of high-confidence recognition (HCR) answers, 65.7% ± 3.9% of low confidence recognition (LCR) answers, and 49.3% ± 3.1% of “guess” answers were correct (Figure 1B). For the fMRI subsequent memory analyses, “remembered” responses were limited to correct HCR and cued recall responses (Davachi et al., 2001; Wagner et al., 1998) while “missed” responses included all incorrect recognition answers across confidence levels. Thus, subjects “remembered” on average 34.1% ± 4.8% of all questions and “missed” on average 33.5% ± 2.4% of all questions, leading to similar statistical power in estimating the response for each condition.

Figure 1. Memory Test and Behavioral Performance.

(A) Memory was assessed using an interactive computerized questionnaire. On each trial, a visual snapshot from the queried movie segment was presented with a memory question. Subjects were instructed to recall the answer to the question if they were confident they could. If they were not, a two-alternative force choice recognition phase ensued (left panel) during which they would choose their answer and rate their confidence (“highly confident,” “fairly confident,” or “guessing”; right panel).

(B) Behavioral measures of performance. Subjects’ performance is presented as overall percent correct and percent correct for high- and low-confidence level. Dotted line represents the chance level of performance obtained by a group of subjects that completed the questionnaire without seeing the movie (Furman et al., 2007). Error bars are standard error of the mean (SEM). (Abbreviations: HCR = high-confidence recognition; LCR = low-confidence recognition).

Enhanced Intersubject Correlation during Successful Memory Formation

To detect brain regions exhibiting intersubject correlated temporal patterns of BOLD response during subsequently remembered events, we modified the previously described intersubject correlation analysis to include each subject's behavioral performance. The ISC-SM analysis consisted of the following computational steps (see also flowchart in Figure S1, available online): first, based on each subject's performance on the long-term memory test, each of the 77 events distributed over the length of the movie was labeled as later “remembered” (correct answer, using recall or high-confidence recognition) or “missed” (incorrect recognition answers across all confidence levels). Thus, for any pair of subjects, the labeled segments fell into three pairwise conditions: (1) events both subjects subsequently remembered (blue stripes in Figure 2), (2) events both subjects subsequently missed (gray stripes in Figure 2) and (3) events for which subjects’ responses were different (i.e., one remembered and one missed; mixed stripes in Figure 2). Then, we computed the ISC for each of the conditions: “both subjects remembered,” “both subjects missed” and “subjects responded differently” on a voxel-by-voxel basis for all subject pairs and this resulted in three sets of correlation coefficients (remembered, missed,or different) per voxel. Finally, we compared the resulting correlation coefficients across the different categories to identify voxels showing significantly higher ISC for both remembered compared to both missed events (i.e., voxels in which the BOLD activation was significantly more correlated across subjects during the encoding of movie segments that were later remembered versus those segments that were later missed). The statistical significance of the increased ISC for remembered events was assessed using both parametric t tests (Figure 3; p < 0.05) and a nonparametric permutations test (see below). A 3D randomization technique to estimate a corrected cluster-level confidence for the entire volume was used to correct for multiple comparisons (Forman et al., 1995; Goebel et al., 2006). For more methodological details see Experimental Procedures and Figure S1.

Figure 2. ISC Based on SM.

Example BOLD time course responses extracted from the same voxel of two representative subjects are shown. Probed events within each time course are color-coded according to subsequent memory performance with remembered in blue and missed in gray. For a given pair of subjects, memory performance and corresponding BOLD time courses are binned into three categories: both subjects remembered, both subjects missed, and subjects responded differently. Next, the ISC is calculated for each category separately, as detailed under Experimental Procedures, resulting in three separate correlation coefficients, or the ISC-SM. Finally, a statistical contrast is performed searching for voxels with significantly greater ISC during encoding of events later remembered compared to those missed.

Figure 3. Regions Showing Enhanced ISC during Events Later Successfully Remembered.

Statistical maps displayed on sagittal (top) and coronal (bottom) sections of group mean anatomical brain depict brain areas with significantly enhanced ISC during movie segments that were subsequently remembered compared to segments that were subsequently missed. Voxels are colored according to the ISC R value for the both remembered condition. Moving from anterior to posterior, these regions include the right temporal pole (TP; [48, 11, –21]) (Talairach coordinates), bilateral anterior and posterior superior temporal gyrus (anterior STG: right [56, –12, –1]; left [–57, –12, 1]; posterior STG: right [52, –35, 4]; left [–57, –35, 4]), bilateral anterior parahippocampal cortex (aPHG; right ([25, –29, –18]; left [–29, –35, –17]), bilateral posterior parahippocampal gyrus (pPHG; right [23, –39, –14]; left [–27, –44, –14]), and bilateral temporal parietal junction (TPJ; right [51, –53, 9]; left [–55, –55, 12]). RH, LH denote the right and left hemisphere, respectively.

Plotted in Figure 3 are the results for the ISC-SM analysis after correction for multiple comparisons. The pseudocolor (yellow to orange) represents the average R value of the both remembered correlation coefficients, and they are graphed only in voxels that show a significant ISC-SM effect (both remembered > both missed contrast). Moving from anterior to posterior, these regions include the right temporal pole (BA 38), superior temporal gyrus (STG, BAs 21 and 22), anterior and posterior parahippocampal cortex (PHG, BAs 35 and 36), the temporal parietal junction (TPJ, BA 39), and occipital cortical regions (BAs 18 and 19). A similar map was obtained when comparing both remembered to subjects responded differently (data not shown). The medial prefrontal cortex (mPFC) and the transverse occipital sulcus (TOS) also showed significant ISC-SM effects; however, they did not survive the strict correction for multiple comparisons (see uncorrected map in Figure S2). For the postulated role of these regions in memory formation under the conditions of movie viewing, see Discussion below. Finally, we did not find any brain regions that exhibited the opposite effect of greater ISC for the missed over the remembered events.

In order to compare brain regions that show a significant ISC-SM effect to those that show a robust ISC irrespective of memory, we computed the voxel by voxel ISC across the entire movie ignoring subsequent memory performance. Importantly, regions showing a significant ISC-SM effect (Figure 4, yellow) only partially overlapped with those regions showing a significant ISC across the entire movie, when memory is not taken into account (Figure 4, red). Regions of overlap are highlighted in orange (Figure 4). Specifically, while activation in most of the posterior visual cortex was significantly correlated across subjects (Figure 4, red), successful memory encoding did not further modulate the correlation. This is in comparison to the effects seen in the posterior PHG, TPJ, and superior temporal gyrus, where significant ISC were seen both during movie viewing and during successful memory formation (orange). Finally, other regions such as the right temporal pole and more anterior regions of PHG showed significant ISC only during events that were subsequently remembered compared to those that were missed (Figure 4, yellow), but did not show significant ISC when the analysis included all events.

Figure 4. Comparison of ISC Computed with and without Subsequent Memory Constraints.

Regions showing significant ISC during movie viewing are shown in red (same threshold as in Figure 3) and regions showing significant greater ISC during events later remembered versus missed are shown in yellow (replicated from Figure 3). The regions where these two maps overlap are shown in orange. The layout is identical to Figure 3. Visual cortical areas (y = –64 to –80) exhibited high ISC during movie watching (red), but this correlation was not enhanced for remembered compared to missed events. Other regions, such as pPHG(y= –37), STG (x = 49), and TPJ (x = 49) exhibited significant ISC for the entire time course, with enhanced correlation for the later remembered events (orange); while the ISC in the temporal pole (y = 7) and aPHG (y = –27) was significantly correlated only for events that were later remembered (yellow).

Nonparametric Assessment of the ISC-SM

In order to independently assess the reliability of the resulting regions that emerged using the ISC-SM analysis, we performed nonparametric permutation tests both at the whole-brain level (Figure S3) and independently on the mean time course data from the ROIs that emerged from the ISC-SM analysis (Figures 5B and S4B). Specifically, for each ROI, we first assessed the chance of getting a significant ISC-SM difference that is not based on subjects’ veridical memory performance. To this end, we randomly shuffled remembered and missed memory labels recursively in order to produce a distribution of ISC for “mock” labels. Next, the ISC-SM differences for the random distributions (“mock both remembered” – “mock both missed”) were calculated and compared to the veridical ISC-SM value (i.e., when using the actual behavioral performances). The veridical ISC-SM difference was deemed statistically significant in each ROI (Figure 5B) if it was at the positive tail of the mock ISC-SM distribution, over the 95% confidence interval. Finally, for the whole-brain analysis, we repeated voxel-by-voxel ISC-SM analysis (as in Figure 3) for each random permutation of the behavioral labels, creating “mock” maps. We used these maps in order to compute a corrected statistical threshold of the veridical ISC-SM map. This threshold was the cluster size above which there were no significant voxels across the entire volume in 90% of the randomly created “mock” maps (p < 0.02, cluster size of 27 voxels). Compare the corrected map, Figure 3, with the uncorrected ISC-SM map, Figure S2, yellow.

Figure 5. Trigger Averaged Responses in ISC-SM Regions of Interest.

(A) Regions of interest (ROIs). The top five ROIs (red) were defined from the ISC-SM map (Figures 3 and S2) and are displayed on coronal slices from the group average anatomical brain: right medial prefrontal cortex (mPFC), right temporal pole, right temporal parietal junction (TPJ), right posterior parahippocampal gyrus (pPHG), and right anterior parahippocampal gyrus (aPHG). An additional control region was sampled from the vicinity of the calcarine sulcus (V1/V2).

(B) Determination of significance using a nonparametric permutation method. Behavioral measures of each subject were permutated and then randomly reassigned. Plots present the distribution of the following statistic: the difference both remembered – both missed correlation coefficients, computed from the permutated “mock” behavioral data (for details see under Experimental Procedures). Red vertical line in each panel represents the value of the statistic when calculated using veridical behavioral measures. The red asterisk denotes significance of the nonparametric test. Note: the statistic calculated using veridical behavioral data is found at the positive tail of the distribution (>95% confidence level) in all regions except the control region (V1/V2).

(C) Response amplitude averaged by the fact-availability. “Fact availability” was defined as the time point in the movie at which the probed content first becomes available. The BOLD time course from all subjects was Z normalized and averaged according to each subject's behavioral performance: remembered epochs (blue lines, include correct answers using recall and high-confidence recognition) or missed epochs (gray lines, include incorrect recognition answers). An X denotes a significant difference between the two conditions (two tailed t test). Black X represents significantly higher remembered trace, red X represents significantly higher missed trace. Yellow bars highlight time points where remembered and missed events show significant magnitude differences in the mPFC, right temporal pole, and TPJ, but not in the PHG.

(D) Response amplitude averaged by episode boundary. All plots same as above, except in this case responses were trigger-averaged starting at the boundary between probed events. Green bars serve to indicate both the initial event-locked response seen in PHG but not in other regions as well as the later divergence of remembered and missed responses in PHG but not in other regions. (See Figure S4 for the same data using a deconvolution approach).

The resulting whole-brain ISC-SM nonparametric corrected map overlapped (Figure S3) nicely with the regions resulting from the parametric t test map (Figure 3). Similar results were also obtained with ROI analysis (Figure 5B). Shown in Figure 5B is the veridical ISC-SM difference graphed on the distribution of difference values using the mock data in each ROI. The mean ISC-SM difference between the mock both remembered and mock both missed centered on 0 ± 0.2, suggesting no difference in ISC between both remembered and both missed events for the shuffled data set (Figure 5B). In contrast, in these regions, the veridical ISC-SM coefficient difference (red vertical line in each panel) was at the positive tail of this distribution over the 95% confidence interval (Figure 5B), suggesting significant differences between the ISC for both remembered compared to both missed events only when using the veridical behavioral responses. The only brain region that proves significant in the nonparametric ROI analysis (see Figure 5B) but does not pass the whole-brain nonparametrically derived corrected threshold is the mPFC.

Finally, in posterior visual areas, the veridical ISC-SM statistic is found in the middle of this mock distribution (i.e., no significant difference between both remembered and both missed correlation), further verifying our initial finding that activity in these regions does not show enhanced ISC during encoding of events later remembered (see control ROI in calcarine sulcus (V1/2) in Figure 5B, and lack of correlation in posterior regions in Figures 3 and S3).

Response Magnitude

The ISC method measures the intersubject similarity in the temporal pattern of BOLD response without making any a priori assumptions about the spatiotemporal response profile. Hence, a significant ISC-SM effect can stem from different types of activation profiles. For example, enhancements or reductions in response amplitude or even a complex mixture of the two, can lead to high ISC as long as the activation patterns are similar across subjects. Thus, in order to characterize the activation profiles in the memory-formation-related regions emerging from our ISC-SM analysis, allowing us to bridge our results with previous SM studies using rapid event-related designs, we extracted the BOLD time course for remembered and missed events within each ROI showing enhanced ISC-SM effects (Figure 3)using both trigger-averaging and deconvolution approaches (see Experimental Procedures). (Note both approaches yielded similar results. Figure 5 presents results using trigger-averaged data, and Figure S4 presents results from the deconvolution analysis, as well as time course data from low confidence remembered events.)

One challenge of using a continuous movie stimulus is in determining at what point an event occurred. In the present study, the movie was segmented into 77 probed events spaced approximately 20 s apart. Based on this structure, we examined the BOLD response starting at two different onset points: (1) the moment when the content related to each question was first available (“fact availability”) and (2) the transition between successive episodes within the movie sequence (“episode boundaries”). Within each ROI, the MR signal from all subjects was Z normalized and trigger averaged (or deconvolved) according to each subject's subsequent memory performance.

Examination of trigger-averaged BOLD time courses revealed two distinct activation patterns. The first was seen in the mPFC, temporal pole, STG, and TPJ, where BOLD activation for the later remembered events was significantly greater than that seen for the missed events (Figure 5). Importantly, this effect is evident shortly after the probed fact is first available (point 0 on x axis; see Figure 5C, yellow shading). The second pattern, evident in PHG, was not characterized by a significant magnitude difference in the response amplitude between remembered and missed events directly after the first availability of the probed fact (Figure 5C, yellow bars). Instead, a biphasic BOLD response was observed. Specifically, the initial response was equivalent for both later remembered and missed events (Figure 5D, left side green bars) and this was then followed by a relatively late amplitude difference between remembered and missed events (Figure 5D, right side green bars). Thus, although activation in the PHG demonstrates a reliable ISC-SM effect, the corresponding magnitude difference is relatively weak and is delayed. Finally, as a control, the same analyses were conducted in V1/2 where no significant magnitude differences were seen between memory conditions (Figure 5, lower panel).

Using Resulting Response Profile as Regressor in a Linear Regression Analysis

In principle, one could use the activation patterns emerging from the time course analyses to create, de novo, a linear regressor in order to reveal additional brain regions exhibiting similar activation patterns. We performed this exercise using the first profile described above. In brief, we modeled an elevation in amplitude during encoding of later remembered events starting approximately 4 TRs post-fact-availability (see Figure 5C upper panel, Fact availability) with no significant above-baseline response (in this case, mean response across the ROI) for missed events. The results of group multisubject GLM random effects analysis for this model are shown in Figure 6A. For comparison, Figure 6B displays results from the ISC-SM analysis (taken from Figure 3).

Figure 6. GLM Results Using a Simple Linear Regressor.

(A) Regions showing a significant correlation with the derived linear regressor (GLM, with a random effects group map, n = 12). Data shown on sagittal views of the group-average anatomical brain.

(B) For comparison, a contrast ISC-SM map is presented, as shown in Figure 3. Results of the derived linear model partially overlap with results of the ISC-SM analysis in the right temporal pole, STS, and TPJ (white circles) but do not overlap in the PHG (ISC-SM only, red circle) and IFG (linear fit only, yellow circle). The mPFC was significant in the uncorrected ISC-SM map (Figure S3) but did not survive the strict correction but again appears here in the linear fit (yellow circle).

Interestingly, we observed only partial overlap between the results of this GLM analysis (Figure 6A) and those of the ISC-SM analysis (Figure 6B), suggesting important distinctions between subsequent memory effects based on BOLD magnitude differences and those based on ISC-SM. Brain regions showing memory-related response enhancement in both analyses were the bilateral temporal poles, TPJ and STG (white circles in Figure 6). However, the GLM analysis revealed additional regions of interest in the mPFC and bilateral inferior frontal gyrus (IFG) that were not evident in the corrected ISC-SM analysis (yellow circles in Figure 6). While the mPFC did emerge from the uncorrected ISC-SM analysis (Figures 5B and S2), the IFG was not previously seen in any ISC-SM analyses. This was not due to a thresholding issue as follow up analyses revealed very low ISC in these IFG regions during subsequently remembered events (right IFG, r = 0.0001; left IFG, r = 0.00006; mean r = 0.0008). Thus, the linear model revealed a significant elevation in response amplitude for the remembered events compared to missed events in bilateral IFG, but the ISC-SM analysis suggests that the spatiotemporal characteristics of this response are not strongly correlated across subjects. Finally, the PHG region did not emerge in GLM analysis (Figure 6A, red circle), further corroborating the existence of differential activation patterns in the network of brain regions supporting successful encoding of long-term memory under conditions resembling real-life (see also Figures 5C and 5D).

DISCUSSION

Making sense of the complex, temporally overlapping, multisensory information encountered in everyday life is a fundamental task that the brain readily accomplishes. The results presented herein reveal patterns of brain activation correlated with successful memory formation during the viewing of a complex, temporally extended audiovisual stimulus.

The data demonstrate, first, that the ISC-SM approach is viable for examining memory formation for real-life-like episodes whose complex structure may not lend themselves to more traditional analysis approaches. Moreover, using the ISC-SM analysis, we were able to isolate two distinct activation patterns associated with successful episodic encoding. Our results further suggest that ISC-based measures of memory formation are distinct from more traditional measures based solely on magnitude differences. Finally, the regions that correlate with successful memory formation for the contents of the movie bear a striking resemblance to a brain network associated with various aspects of social cognition (for review see Gallagher and Frith, 2003). While speculative, this overlap suggests that engagement in social cognitive operations during the experience of everyday events is correlated with memory formation, at least for the narrative contents of those experiences.

Assessment of ISC based on subsequent memory measures revealed that ISCs are enhanced during successful episodic memory formation. Critically, however, not all regions that show a strong ISC during movie viewing (i.e., posterior visual cortex; Figure 4, red) also show an enhanced ISC during successful memory formation. Only the STG, TPJ, and posterior PHG showed both a significant ISC during movie viewing irrespective of memory formation and enhanced ISC during successful memory formation (Figure 4, overlap regions in orange). Moreover, other regions (i.e., temporal poles and anterior PHG) showed a significant ISC only during successful memory formation (Figure 4, yellow). We postulate that the level of specificity of the ISC-SM can dissociate between stimulus-driven versus internally generated operations important for memory formation. Specifically, if a given brain region supports primarily stimulus-driven operations, a strong ISC may be seen irrespective of memory formation. By contrast, regions supporting more internally generated cognitive operations contributing to encoding may be more likely to show ISC only when successful encoding is considered. Regions that show both strong ISC and ISC-SM effects, like the STG, TPJ, and posterior PHG, may then be both stimulus-driven as well as support higher-level operations important in memory formation.

In fact, broadly consistent with this notion is the overall pattern that the most posterior regions of cortex show strong ISC irrespective of memory, the most anterior regions only show ISC-SM effects, while midlevel regions appear to show both. These results may be in accord with previous work examining the neural bases of event segmentation (Zacks et al., 2001), where posterior brain regions are reported to show a response to both fine and coarse grained segmentation boundaries, while frontal cortex only shows a response to the coarse grained events. It is hence tempting to postulate that activation in frontal regions correlates with memory for the overall structure of the event and in posterior regions with memory for the fine details of the action. If this is correct, then a memory test focused on detailed perceptual features rather than on narrative structure of the movie may reveal the opposite pattern of results, with now visual cortex showing both significant ISC as well as an ISC-SM effect.

Social Cognitive Operations Predict Narrative Memory Formation

Unlike traditional experiments that have consistently revealed subsequent memory effects for still images or words in the medial temporal lobes (MTL) and inferior frontal gyrus (IFG), our results additionally implicate the STG, TPJ, mPFC and temporal poles in memory formation. While these regions have not typically emerged in episodic memory experiments using simple words or still images as memoranda, they were identified here using a complex movie stimulus depicting everyday kinds of social interactions. Furthermore, they were not identified solely as being active during stimulus viewing, but instead were identified based on the consistency with which participants engaged these regions during successful memory formation.

This finding is intriguing because recent research has implicated all four of these regions in different aspects of social cognition and perception (Amodio and Frith, 2006; Gallagher and Frith, 2003; Ochsner, 2004). Briefly, whereas mPFC has been associated with drawing higher-level inferences about the beliefs and feelings of social targets, posterior cortical regions have been associated with decoding the social meaning of nonverbal cues. For example, although the functions associated with different mPFC subregions is still a matter of debate, the dorsal region observed here is similar to those consistently activated in studies of judgments about one's own or another person's emotions, personality traits, or intentions (for review, see Amodio and Frith, 2006; Mitchell et al., 2004; Ochsner, 2004; Ochsner et al., 2005). By contrast, regions along the STG have been implicated in auditory-visual integration in monkeys (Ghazanfar et al., 2005) and, in humans, with visuomotor integration, language processing and narrative comprehension (Ferstl et al., 2005; Schmithorst et al., 2006; Xu et al., 2005). The nearby STS is similarly activated by perception of nonverbal cues, including moving lips, eyes, and facial expressions, as well as point-light stimuli depicting biological motion (Blake and Shiffrar, 2006; Puce et al., 2007; Puce and Perrett, 2003; Thompson et al., 2005). The temporal poles have been long associated with memory retrieval (Nakamura and Kubota, 1996). In fact, lesions to this area have been reported to disrupt memory for a global narrative while leaving memory for single sentences intact (Fletcher et al., 1995; Frisk and Milner, 1990; Milner, 1958). One may speculate that the engagement of the temporal poles in our paradigm may reflect the integration of the unfolding narrative into a coherent structure. Finally, although still under intense debate, activation in the TPJ has been previously reported in tasks that require the representation of false beliefs (Gallagher and Frith, 2003; Saxe et al., 2004; Saxe and Kanwisher, 2003); however, this region also has been emerged in tasks requiring shifts of attention (Corbetta et al., 2000; Mitchell, 2007).

While the precise role of each of these regions is not yet clear, they do emerge consistently as a network important in social cognition. While admittedly preliminary, our finding of enhanced ISC-SM for the remembered events in these same regions, strongly suggests that modulation of social cognitive processes can impact episodic memory formation for real-time human interactions. In support of this notion, it has been shown that similar regions are active when subjects are asked to make memory decisions based on knowledge of their own social connections (Kumaran and Maguire, 2005). Moreover, previous work has shown that activation in the mPFC during social encoding tasks correlates with later recognition memory for items (e.g., words or faces) presented during the encoding episode (Mitchell et al., 2004). The present data extend these results by (1) demonstrating that activation in the entire network, not just the mPFC, correlates with successful memory formation and (2) that this activation correlates with memory for social interactions, per se, as measured by questions concerning the narrative rather than recognition of single items alone. Given that much of what we encounter and value on a daily basis is fundamentally social, these findings highlight the important role that social cognitive processes play in episodic memory formation, which to date has been relatively underappreciated in the cognitive neuroscience of memory.

Comparison of ISC-SM to a Linear Regressor Approach

The analysis of the fMRI data in this study was conducted using two different approaches: (1) the ISC-SM approach and (2) a simple linear regressor model in which the shape of the regressor was derived directly from patterns of BOLD activation gleaned from ROIs identified from the ISC analysis. We found that these two approaches yielded complementary, yet not fully overlapping, results. On the one hand, overlapping results were seen in the TPJ, temporal poles and regions along the STG. These regions exhibited statistically stronger ISC during successful versus unsuccessful memory formation and, at the same time, the magnitude of activation in these regions was also significantly greater during successful compared to unsuccessful memory formation.

On the other hand, other regions emerged only in one or the other analysis. Specifically, the PHG exhibited enhanced ISC during successful memory formation but did not emerge in the regression analysis. Conversely, while previous studies have revealed that activation in the left IFG is modulated by controlled semantic retrieval and selection operations (Badre et al., 2005; Gold and Buckner, 2002; Thompson-Schill et al., 1997) and is enhanced during successful associative memory formation (Davachi et al., 2003; Ranganath et al., 2004; Uncapher et al., 2006), this region did not show enhanced ISC during successful memory formation. However, when using a linear regression, we found that the BOLD response in IFG was significantly greater during encoding of later remembered compared to missed events (Figure 6), consistent with these previous reports. The lack of an ISC-SM effect in IFG could emerge from two distinct scenarios: (1) large individual variability in the engagement of IFG mechanisms during successful memory formation or (2) a strong ISC during both remembered and missed events. Analysis of the correlation values in both left and right IFG during successful encoding revealed low R values, supporting the first scenario and suggesting that mechanisms supported by IFG may, indeed, be highly variable across individuals.

These nonoverlapping results importantly suggest that the ISC-SM approach reveals a correlate of memory formation that is distinct from more traditional linear modeling approaches based solely on magnitude differences. The ISC-SM detects reliable temporal response patterns that correlate with successful memory formation across subjects. These correlations can be induced by increases in the response amplitudes (as observed for example in the STS, TPJ) but can also be induced by decreases in the response amplitudes and/or by a complex mixture of the two. Thus, a significant ISC-SM effect may reflect the extent to which different subjects are, among other things, attending to similar aspects of the movie stimulus, retrieving semantic and episodic details relevant to comprehending the stimulus and, perhaps, are making predictions about what might happen as a result of presently viewed scenarios. In other words, there may be an “optimal” manner of processing the complex movie stimulus that enhances the probability of long-term memory formation that the ISC-SM analysis is allowing us to measure that is, importantly, not dependent on overall response magnitude in any given region.

Conclusion

We identify patterns of brain activation correlating with successful long-term memory formation during the viewing of a complex, dynamic, and temporally extensive movie stimulus. Along with emerging multivariate connectivity measures (Haxby et al., 2001; Haynes and Rees, 2005; Kamitani and Tong, 2005; Polyn et al., 2005), the ISC-SM may prove fruitful in examination of higher-level cognitive operations in cases where it may be challenging to invoke an accurate model (e.g., when there may be too many free parameters, like in the case of a complex viewing situation) or in cases where different models are suitable for different brain regions. Future work can now focus on revealing how the results may be modulated by critical factors such as the movie content and subjective event segmentation (Avrahami and Kareev, 1994; Hanson and Hanson, 1996; Neisser 1988; Zacks and Tversky, 2001). When narrative content is stressed, as in the present study, we find that a brain network previously implicated in various aspects of social cognition, shows significant intersubject correlations that are enhanced during successful memory formation.

EXPERIMENTAL PROCEDURES

Participants

Twelve right-handed subjects (age 23 ± 4 years, 7 females) participated in the experiment. Data from four additional subjects was not included in the study due to excessive movement in the magnet (two participants) and failure to stay awake during scanning, as evident from eye-tracking monitoring throughout the session (two participants; see below). All participants had normal or corrected-to-normal vision, provided written informed consent, and were remunerated for their participation. All participants were native English speakers and were prescreened to insure that they had never seen any of the episodes of the selected TV sitcom prior to the experiment and had no familiarity with either the characters, the topics of the series, or the specific plot of this episode. Procedures were in compliance with the safety guidelines for MR research and were approved by the University Committee of Activities Involving Human Subjects at New York University.

Study Material and Memory Test

The stimulus used was a 27 min long episode from an English-speaking television sitcom (Curb Your Enthusiasm, by Larry David, Season 1, Chapter 7: “AAMCO”), which portrays a character's real-life-like actions in an urban environment. The episode depicted ordinary types of events (e.g., a dinner party, a minor car accident, and arguments with friends). Memory performance was assessed using a computerized interactive questionnaire (in-house software). A total of 77 questions were composed to test memory for components of the narrative content of the episode, at an approximate sampling rate of one question per 20 s. A special emphasis was placed on composing questions that target the main occurrence within each 20 s movie segment, that could only be answered with information presented during the relevant segment, and could not be deduced from other segments of the episode. Thus, a question could not be answered using information obtained before or after the segment of the movie it covered. Questions were presented in the same order that respective events occurred in the movie. In a separate behavioral study we ruled out the possibility that the ordering of the test questions may have primed subjects’ memory performance (Furman et al., 2007).

Each test question was accompanied by presentation of a still frame that served as a visual cue. Subjects were given two options for answering each question: they were instructed to attempt to recall the answer and enter a free-text response if they were highly confident that they could answer correctly (cued recall mode; Figure 1A, left panel). If not confident, they were presented with a two-alternative-forced-choice recognition test and asked to rate their confidence for the recognition portion (highly confident, fairly confident, or guessing; Figure 1A, right panel). Subjects answered each question only once, using either recall or recognition. Both response modes displayed the same visual cue and question.

Experimental Procedure

Participants took part in two experimental sessions, study and test, which were spaced 3 weeks apart. The interval between study and test was determined on the basis of a separate set of behavioral experiments targeting memory performance over time (Furman et al., 2007). The study session took place inside an MRI scanner (see scanning procedures below), while the test session took place in a quiet windowless room containing a desktop computer. During the study session, participants underwent fMRI scanning while they watched the movie. Video and sound were supplied by a PowerMac G4 system. Audio was fed into the standard Siemens bed system, and video was sent to an LCD projector and projected on a screen behind the subjects. The visual stimuli were viewed via a mirror mounted on the head coil, and sound was delivered via Siemens system pneumatic earphones. External padding around the head was added in order to minimize head movement and scanner noise, and in order to optimize quality and comprehension of the spoken dialogue. Participants were not informed of a pending memory questionnaire and were not asked to explicitly memorize movie content. Eye movement data were monitored and collected during scanning using an infrared eye tracker (ASL model 504) to ensure that subjects were watching the film.

MRI Acquisition

We used functional magnetic resonance imaging at 3T (Allegra, Siemens, Erlangen) to measure blood-oxygen level-dependent (BOLD) changes in cortical activity. During each fMRI scan, a time series of volumes was acquired using a T2*-weighted EPI pulse sequence (TR 2000 ms, TE 30 ms, flip angle 80° , 35 coronal slices, 3 × 3 × 3 mm voxels, FOV 192 mm). All images were acquired using a head coil (transmitter/receiver; Nova Medical, model NM011). T1-weighted high-resolution (1 × 1 × 1mm) anatomical images were acquired with an MP-RAGE pulse sequence, for each observer, to allow accurate cortical segmentation, 3D reconstruction and volume-based statistical analysis.

fMRI Data Analysis

fMRI data was preprocessed and analyzed using the BrainVoyager QX v1.8.6 software package (Brain Innovation) and in-house supplementary software. Preprocessing of functional scans included slice-time correction, head movement detection (two scans with head movement larger than 2 mm were rejected) and correction, high-frequency temporal filtering, and removal of linear trends. Subjects’ data were transformed into Talairach space (Talairach and Tournoux, 1988) and spatially smoothed by a Gaussian filter of 6 mm full width at half maximum (FWHM). To remove transient preprocessing artifacts emerging at the beginning and the end of the scanning sessions, the first 10 and last 19 time points, which include also the running credits, were excluded from the analysis. The duration of the credit roll at the end of the episode was 61 s. After removing the last 38 s, 23 s of the credit still remained; this was sufficient to capture the delayed BOLD response for the last event in the movie (corresponding to test question #77).

To correct for multiple comparison in the whole-brain maps (Figures 3, 4, and 6), we implemented a randomization technique to estimate a corrected cluster-level confidence for the entire volume based on the method described by Forman et al. (1995). The method uses a nonparametric Monte Carlo simulation which calculates the likelihood to obtain a cluster of randomly generated voxels across the entire volume based upon the individual voxel probability threshold (p > 0.05). To reduce the number of multiple comparisons, the correction was restricted to include only gray matter regions. It is important to note that even before the correction all of our results were confined only to gray matter regions (see uncorrected map in Figure S2). The correction took into account the resolution of the sampling (3 mm × 3 mm) and the spatial smoothness applied to the data set (FWHM 6mm), and closely followed the procedure introduced by Forman et al. (Forman et al., 1995; Goebel et al., 2006).

ISC Constrained by Subsequent Memory

The ISC-SM was comprised of the following computational steps (see analysis flowchart in Figure S1).

I. Behavioral Parcellation

Each subject's subsequent memory performance (answers on computerized questionnaire) were labeled as either remembered (correct answer, using recall or high-confidence recognition) or missed (incorrect recognition answers). Memory performance was then compared between each pair of subjects, and answers were sorted into three groups: both subjects remembered, both subjects missed, and subjects responded differently (see also Figure 2). Questions answered by incorrect recall, correct low-confidence, or correct guesses were excluded from further analysis.

II. Talairach Alignment

All brains were aligned to Talairach coordinate space.

III. Pairwise ISC-SM Analysis

For each voxel in the Talairach space, we computed the intersubject correlation (ISC) values for time course data segments according to behavioral parcellation (note that parcellations can be different for each subject pair). For each subject pair, time course segments of each category were concatenated, and a single correlation was computed, resulting in three sets of correlation coefficients per voxel (both remembered, both missed, different response). fMRI time series segments were aligned according to the moment within each event when probed information is first provided (“fact availability”), adding two TRs to account for hemodynamic delay.

IV. Statistical Parametric Mapping

In order to find voxels in which the ISC is higher for the remembered events, we used both parametric and nonparametric tests.

Parametric t Test Analysis

A voxel-by-voxel t test was computed on the difference between the both remembered and both missed correlation coefficients (Figure 3), after applying a Fisher transformation to individual coefficients. The maps were corrected using a cluster-size threshold method (see above).

Nonparametric Permutation Analysis

In addition, we applied a nonparametric permutation test at two independent levels: a whole-brain voxel-by-voxel level (Figure S3) and on the ROIs emerging from the parametric map (Figure 5B). Specifically, the remembered and missed memory labels were randomly permutated within each subject and were then randomly reassigned to subjects. Correlation coefficients for mock both remembered and mock both missed time course segments were calculated for each ROI and for each voxel in the whole-brain analysis, according to the randomly assigned behavioral parameters. This procedure was repeated 1000 times for each ROI and 100 times for each voxel in the whole-brain analysis, resulting in a distribution of the statistic for the mock both remember and mock both missed conditions. Finally, within each ROI/voxel the ISC-SM difference (both remember – both missed) was calculated, and the resulted mock distribution was compared to the veridical ISC-SM value (i.e., when using the actual behavioral performances). The difference between the ISC for the remembered and missed events was deemed significant in each ROI/voxel if the veridical ISC-SM statistic was at the positive tail of the mock distribution, over the 95% confidence interval. In the whole-brain analysis, a t test was performed between the mock both remember and mock both missed conditions for each permutation. We then chose a corrected threshold cutoff for the veridical ISC-SM map (p < 0.02, cluster size of 27 voxels) in which there was no significant voxel across the entire volume in 90% of the random maps. In both whole-brain voxel-by-voxel statistical maps (parametric t test, Figures 3 and S3; and nonparametric permutation test, Figure S3), the value plotted in significant voxels is the mean R value of both remembered correlation coefficients across all subject pairs.

BOLD Signal Magnitude Analysis

In order to compute signal magnitude changes, time course data from each ROI identified in the ISC-SM analysis were Z normalized and averaged by event type according to each subject's behavioral performance. Signal was averaged using two possible trigger onset points: one according to the onset of each event (“episode boundary”) and another according to the moment within each event when the answer to the question is first provided (“fact availability”). The trigger averaging analysis included averaging the normalized data for remembered and missed events separately across subjects, and performing two-tailed t tests at each time point in order to determine statistical significance of differences between event types. In addition to the simple trigger averaging, a finite impulse response (FIR) deconvolution analysis (Glover, 1999) was performed separately for each event type, and the resulting traces were averaged across subjects and submitted to the same statistical tests as above (see Figure S4). Both analyses (trigger averaging and FIR deconvolution) were performed for three types of events—remembered (correct recall or high confidence recognition), missed (incorrect recognition answers), and low-confidence (correct low-confidence recognition answers). Figure 5 presents trigger averaging results of remembered and missed events, while Figure S4 presents the deconvolution analysis results of all three event types. Statistical tests were always performed only between remembered and missed event types.

Derived Linear Model Captures Memory-Induced Activation

We created a single linear regressor modeled directly on the responses for later remembered and missed events in one of the activation profiles resulting from the triggered-average ROI analysis. Remembered events were modeled as a six TR boxcar convolved with a hemodynamic response function. Missed events were not given a response since no magnitude change from baseline was found in the triggered average time course results (see Figure 5B, top panel). A unique regressor was built for each subject according to their subsequent memory performance, modeling only correct answers using recall and high confidence recognition. A GLM random effects group analysis was performed using these regressors (n = 12).