An Unexpected Sequence of Events: Mismatch Detection in the Human Hippocampus

Abstract

The ability to identify and react to novelty within the environment is fundamental to survival. Computational models emphasize the potential role of the hippocampus in novelty detection, its unique anatomical circuitry making it ideally suited to act as a comparator between past and present experience. The hippocampus, therefore, is viewed to detect associative mismatches between what is expected based on retrieval of past experience and current sensory input. However, direct evidence that the human hippocampus performs such operations is lacking. We explored brain responses to novel sequences of objects using functional magnetic resonance imaging (fMRI), while subjects performed an incidental target detection task. Our results demonstrate that hippocampal activation was maximal when prior predictions concerning which object would appear next in a sequence were violated by sensory reality. In so doing, we establish the biological reality of associative match-mismatch computations within the human hippocampus, a process widely held to play a cardinal role in novelty detection. Our results also suggest that the hippocampus may generate predictions about how future events will unfold, and critically detect when these expectancies are violated, even when task demands do not require it. The present study also offers broader insights into the nature of essential computations carried out by the hippocampus, which may also underpin its unique contribution to episodic memory.

This functional imaging study suggests that the human hippocampus may generate predictions about how future events will unfold and may critically detect when these expectancies are violated.

Introduction

We have all had the experience of walking into a familiar place, for example our living room, and immediately noticing that a new object such as a painting is present (i.e., stimulus novelty), or equally that the furniture has been rearranged (i.e., associative novelty). Whereas the neural mechanisms underlying the coding of stimulus novelty have been widely studied [1,2], surprisingly little is understood about the way in which associative novelty is represented within the human brain.

How might the brain enable us to rapidly and apparently effortlessly detect such unexpected occurrences, when familiar objects appear in new configurations (i.e., associative novelty)? One attractive possibility suggested by theoretical models is that novelty detection relies upon a comparison between current sensory input and stored representations [3–9]. This computation has been proposed to occur in the hippocampus, given the unique nature of its anatomical circuitry [3–5,10,11]. The hippocampus, therefore, is viewed to detect “associative mismatches” in the environment, i.e., when current sensory input conflicts with expectations deriving from associative retrieval of past experience [3–5,10–12]. By this account, we only notice that our favourite armchair has moved because we have expectations, derived from prior experience, about where it should be located.

Empirical evidence in rodents supports the hypothesis that the hippocampus acts as an associative mismatch detector [13–15]. In a recent study [13], neural activity in the hippocampus was recorded in response to moving an escape platform to a new location in a water maze. Hippocampal novelty signals were only observed when rats had been trained to expect the platform to be in a specific location. In contrast, novelty signals were not observed in response to the mere presence of associative novelty, where prior expectations were absent, i.e., when the platform was moved to a new location on each trial. This study, therefore, provides evidence that hippocampal novelty responses specifically reflect the detection of associative mismatches contingent on the retrieval of prior expectations (i.e., “match”), and are not generated by the presence of associative novelty per se when prior expectations are absent.

In humans, however, few neuroimaging experiments have investigated hippocampal responses to associative novelty, in which familiar items appear in novel configurations [16,17], with the majority focussing on stimulus novelty [18–22]. Critically, these studies [16,17], although observing hippocampal responses to associative novelty, were not designed with the aim of distinguishing an associative mismatch process from a more general response to associative novelty per se. Existing work, therefore, has not provided direct evidence that the human hippocampus operates as an associative mismatch detector, a process that has been proposed to play a cardinal role not only in novelty detection, but also more generally in the functioning of a network subserving memory [3,23,24]. Indeed it has been suggested by computational models that such comparison-based computations allow mismatches between current sensory input and stored internal representations to induce hippocampal dynamics that favour the learning of new information (i.e., encoding), rather than the retrieval of old information (i.e., retrieval) [3,23].

In this functional magnetic resonance imaging (fMRI) study, we use a repetition paradigm to characterize brain responses to sequence novelty, in which familiar items (i.e., seen once previously) are viewed in a new temporal order. We focussed on this type of (temporal) associative novelty because theoretical models of hippocampal function have emphasized its role in bridging discontinuities over time [25,26], and specifically in representing episodic memories as sequences of events [10,27–30]. Moreover, the role of the hippocampus in novelty detection has been closely linked with its capacity to store and recall sequences [5,10,11,29]. Indeed, the capacity to predict future events on the basis of prior experience, and react when they do not turn out as expected, is of clear evolutionary importance [9,10].

We therefore constructed event sequences consisting of the consecutive presentation of four objects (i.e., a quartet), superimposed on one of 30 background scenes to which subjects had been pre-exposed (see Materials and Methods) (Figure 1A and 1B). Following a short delay, the same object quartet was re-presented in one of three possible sequential orders (factor: sequence type, levels: S_rep, S_half, and S_new), on either the same or a different background scene (factor: object-context, levels: C_same and C_different). In the S_rep condition, the objects were re-presented in exactly the same order as previously (i.e., A-B-C-D; see Materials and Methods and Figure 1), whereas in the S_new condition, the sequential order of objects during the second presentation was entirely different from that during the first presentation (e.g., C-A-D-B). In the S_half condition, the first half of the object sequence was the same as during the first presentation, but the second half was different, with the last two items in the sequence presented in reverse order (i.e., A-B-D-C). In common with numerous previous studies exploring how the brain automatically processes novelty regardless of its behavioral relevance, e.g., [20,31], subjects performed an incidental task unrelated to our experimental manipulations (1-back target detection task; see Materials and Methods).

Figure 1. Experimental Design.

(A) Color pictures of objects were presented in groups of four constituting an object quartet. Objects were presented consecutively for 1 s each, with a 200-ms gap between objects, over a background scene that was displayed continuously throughout the presentation of each object quartet (first presentation: FIRST). Subjects were pre-exposed to the set of background scenes used, prior to the scanning experiment. There followed a 2-s period during which a fixation cross was displayed.

(B) Following this 2-s period, the same object quartet was re-presented in one of three sequence types (S_rep, S_half, or S_new) on either the same or different background scene (C_same [as illustrated in (B)] or C_diff) (second presentation; a 3 × 2 factorial design). The sequential order of objects during the second presentation was A-B-C-D (S_rep), A-B-D-C (S_half), and either C-A-D-B (as illustrated) or B-D-A-C (S_new) (with the first presentation of the object quartet denoted by A-B-C-D). Importantly, each object was trial-unique and only presented twice during the experiment (i.e., in first and second presentations). Subjects performed an incidental 1-back target detection task throughout the experiment (see Materials and Methods). Note that the stimuli shown here are visually similar (but not identical, for copyright reasons) to those used in the experiment.

The design of our study, therefore, in contrast to prior neuroimaging studies in this area [16,17], allowed us to distinguish between an associative mismatch process and a more general response to sequence novelty per se. As defined above and in previous work [10,14], an associative mismatch occurs when there is a discrepancy between the stimulus that is predicted to appear next and the stimulus that actually appears. Associative mismatches, therefore, can be seen to be contingent on a prior “match” process reflecting associative retrieval of specific predictions about what will occur. Critically, therefore, an associative mismatch situation only arises at the third object in the S_half condition when predictions derived from prior experience (i.e., the first presentation) and cued by the first two objects in the sequence (i.e., match) are subsequently violated. In contrast, a significant associative mismatch does not occur in the S_new condition, the condition with the greatest level of sequence novelty (i.e., change from first presentation). This is the case since reordering of all aspects of the sequence, including the salient “start” section, which may act as a reference point or anchor for other elements [32], likely disrupts match processes reflecting the retrieval of specific predictions concerning which object will appear next [33,34]. Our experimental design, therefore, was explicitly designed to dissociate the condition with the greatest associative mismatch (S_half) from that containing the greatest level of sequence novelty per se (S_new). We reasoned, therefore, that a finding of greatest hippocampal activation in S_half would support the hypothesis that the hippocampus operates as an associative mismatch detector [3,4,8–10]. Conversely, if we observed maximal hippocampal responses to the S_new condition, this would be consistent with a general response to sequence novelty per se.

We, therefore, set out to characterize brain responses to sequence novelty, a form of associative novelty not previously explored (experiment 1). We aimed to test the hypothesis that the human hippocampus detects associative mismatches between what is predicted to happen based on prior experience, and what actually occurs [3,4,8–10]. Further, in a subsequent behavioral experiment (experiment 2), we asked whether the violation of predictions in the S_half condition is associated with detectable change in subjects' behavior using a reaction time (RT) measure. More generally, we hoped that determining whether the human hippocampus functions as an associative mismatch detector during novelty detection would provide insights into the nature of essential computations carried out by the hippocampus, processes that may also underlie its contribution to episodic memory.

Results

Experiment 1: Neuroimaging

Behavioral data.

In the incidental 1-back target detection task performed during scanning (see Materials and Methods), subjects detected consecutive repetitions of objects with 90% (±6%) accuracy. Average RTs for the detection of object repeats for each sequence type, collapsed across the object-context factor were: 522 ± 22 ms (S_rep), 539 ± 25 ms (S_half), and 526 ± 26 ms (S_new). There was no significant effect of sequence type in terms of either RT (F_(2,32) = 0.82, p = 0.45) or accuracy (F_(2,32) = 0.62, p = 0.54). These results provide evidence that subjects were focussed on the incidental target detection task during the experiment, and suggest that they attended equally to each of the three sequence types. Subjects were thoroughly debriefed following scanning in order to assess how they performed the task. As expected, none of the subjects reported naming the objects to a significant extent or explicitly trying to memorize or retrieve the sequences of stimuli during the experiment, confirming the naturalistic processing of object sequences (i.e., without explicit requirements to learn). Instead, subjects remained focussed on the target detection task, in line with our instructions.

Neuroimaging data.

We first identified the overall pattern of neural activity elicited by the initial viewing of sequences of novel objects (i.e., first presentation of the object quartet). This was done by comparing the first presentation of the object quartet with the baseline condition during which subjects viewed familiar objects always presented in the same sequence and in the same background scene (FIRST > BASE; see Materials and Methods). There was significantly greater activation in this contrast in a distributed network previously implicated in the processing stimulus novelty [2], including regions such as the prefrontal cortex and notably the left hippocampus and right entorhinal/perirhinal cortex (Figure S1; Table S1).

Hippocampal novelty responses are maximal in the S_half condition.

Our main interest was in determining whether hippocampal activation in relation to novel sequential information reflects either a general response to sequence novelty, or alternatively an associative match-mismatch process whereby initial predictions derived from prior experience (i.e., the first presentation) and cued by the first two items in the sequence (match), are subsequently violated (associative mismatch). Accordingly, we directed our analyses to evaluating differences in neural activity between the three possible sequential orders (S_rep, S_half, and S_new), in which objects could appear during the second presentation. Given our experimental aims, the forthcoming analyses refer to main effects of sequence type (i.e., collapsed across object-context). Of note, there was no significant interaction between our experimental factors (sequence type and object-context), even at liberal statistical thresholds (i.e., p < 0.01 uncorrected for multiple comparisons). Moreover, no significant activation (i.e., at p < 0.001 uncorrected for multiple comparisons) within predicted regions of interest (i.e., the medial temporal lobe [MTL]) was observed when we considered the main effects of object-context (i.e., collapsed across sequence type). Furthermore, we did not observe any significant activation in other brain areas in this comparison (p < 0.05 corrected).

We first compared neural activity during the S_half and S_new conditions (S_half > S_new). This contrast revealed significant activation in the body of the left hippocampus (Figure 2). No significant hippocampal activation was observed in the reverse comparison (S_new > S_half), even at liberal statistical thresholds. Having determined that hippocampal activation is greater in the S_half condition as compared to the S_new condition, we next compared the S_half and S_rep conditions with each other. Activation in a very similar region of left hippocampus was identified in this contrast (S_half > S_rep; Figure 3), as well as a region of right entorhinal/perirhinal cortex. No areas in the MTL were found to be significantly more active in the reverse comparison (S_rep > S_half). Hence, these results demonstrate that a region in the left hippocampal body responds maximally in the S_half condition. In contrast, no significant hippocampal activation was observed when the S_rep and S_new conditions were directly compared with each other (i.e., S_new > S_rep or S_rep > S_new), even at liberal statistical thresholds.

Figure 2. Main Effect of Sequence Type in Left Hippocampus: S_half > S_new.

Results of comparison of S_half and S_new conditions (S_half > S_new, collapsed across object-context factor). “Glass brain” figures are displayed above. Activations shown on the averaged structural MRI scan of the 17 participants (displayed below). The color bar indicates the t-statistic associated with each voxel, and the Z-score equivalent. Activation in left hippocampus (HC) (x, y, z = −27, −18, −18; z = 3.59) is circled in sagittal, coronal, and axial planes. Activation in left medial prefrontal cortex (x, y, z = −9, 36, −15; z = 3.58) evident on axial section and significant at p < 0.001 uncorrected. Threshold is set at p < 0.005 uncorrected for display purposes.

Figure 3. Main Effect of Sequence Type in Left Hippocampus and Right Entorhinal/Perirhinal Cortex: S_half > S_rep.

Results of comparison of S_half and S_rep conditions (S_half > S_rep, collapsed across object-context factor). “Glass brain” figures are displayed above. Activations shown on the averaged structural MRI scan of the 17 participants (displayed below). The color bar indicates the t-statistic associated with each voxel and the Z-score equivalent. Activation in left hippocampus (HC) (x, y, z = −27, −15, −18; z = 3.51) evident in sagittal and coronal sections. Activation in right entorhinal/perirhinal (ERC) (x, y, z = 21, −9, −30; z = 4.07) evident on coronal and axial sections. Threshold is set at p < 0.005 uncorrected for display purposes.

Given the S_rep and S_half conditions are indistinguishable early on (i.e., first two items in sequence), any significant differences in hippocampal activation between these two conditions must have arisen later on in the sequence, in particular in relation to the third item. In order to empirically demonstrate this, we performed an additional analysis (see Materials and Methods) in which the trial onset was set either at the first, or third, item in the sequence, and neural activity was modelled with a delta/stick function. When the trial onset was set at the third object, the pattern of neural activations observed was highly similar to that obtained in the “mini-block” analysis detailed above. As such, significantly greater hippocampal activation was observed in the S_half condition as compared to the S_rep condition, when the trial onset was set at the third object (i.e., S_half > S_rep: x, y, z = −27, −15, −18; z = 3.96; Figure S2). In marked contrast, no significant hippocampal activation was observed in this comparison when the trial onset was set at the first object, even at liberal statistical thresholds. Hence, these results strongly suggest that the hippocampal activations in our paradigm are generated specifically in response to later items in the S_half condition, most likely at the third object under conditions of associative mismatch when prior expectations suffer violation by sensory reality.

Our results, therefore, in demonstrating that hippocampal activation is maximal in the S_half condition, go clearly against the notion that the hippocampus responds to sequence novelty per se. Instead, our findings suggest that in the S_half condition, expectations initially set in train (match) are subsequently violated (associative mismatch) at the third object in the sequence, resulting in the generation of a mismatch (or novelty) signal within the hippocampus. Thus, the pattern of hippocampal activation during viewing of novel sequential information would seem to be best characterized by an associative match-mismatch process, providing compelling support for computational models in which this putative neural mechanism plays a key role in novelty detection [3,4,8–10].

Right entorhinal/perirhinal cortex exhibits novelty responses in both conditions in which sequence novelty is present (S_half and S_new).

The pattern of activation observed in the right entorhinal/perirhinal cortex (hereafter referred to as right entorhinal cortex) in relation to the three sequence conditions was qualitatively different from that observed in the hippocampus. Although significantly greater activation was observed in this area in the S_half condition as compared to the S_rep condition (S_half > S_rep; Figure 3), no significant activation was found when the S_half condition was compared to the S_new condition (i.e., S_half > S_new or S_new > S_half). That the right entorhinal cortex responds more generally to sequence novelty per se is supported by the significantly greater activation in this region observed when we considered the main effect of sequence change (i.e., S_half + S_new > S_rep; Figure 4). Notably, no hippocampal activation was observed in this comparison. As such, the entorhinal cortex, but not the hippocampus, exhibited differential responses during the second presentation to both conditions in which sequence novelty was present (S_half and S_new). Therefore, our results suggest that whereas the pattern of hippocampal activation in relation to the three sequence types is consistent with the operation of associative match-mismatch processes, the pattern observed in the entorhinal cortex is instead better characterized by a more general response to sequence novelty per se.

Figure 4. Main Effect of Sequence Type in Right Entorhinal/Perirhinal Cortex: S_half + S_new > S_rep.

Significantly greater activation observed in right entorhinal/perirhinal cortex (ERC) in comparison of S_half and S_new conditions against S_rep condition (S_half + S_new > S_rep, collapsed across object-context factor). “Glass brain” figures are displayed above. Activations shown on the averaged structural MRI scan of the 17 participants (displayed below). The color bar indicates the t-statistic associated with each voxel, and the Z-score equivalent. Activation in right entorhinal/perirhinal cortex (x, y, z = 21, −6, −30; z = 3.56) circled in sagittal, coronal, and axial planes. Threshold is set at p < 0.005 uncorrected for display purposes.

Experiment 2: Behavioral

Our neuroimaging results demonstrate that greatest hippocampal activation occurs under conditions of associative mismatch (i.e., the S_half condition), when expectations initially set in train (match) are subsequently violated (associative mismatch). We next asked whether predictions generated on the basis of previous experience (i.e., first presentation), and subsequently violated at the third object in the S_half condition, are associated with a detectable change in subjects' behavior (i.e., using an RT measure as an index of behavior during the second presentation). During the scanning experiment, subjects performed an incidental task (i.e., 1-back target detection; see Materials and Methods) so as to avoid contaminating the brain activation data with the influence of frequent motor responses. In the behavioral experiment, however, a group of naive subjects performed a speeded semantic categorisation task on each object presented (see Materials and Methods), chosen in order to avoid any explicit requirements to learn, and thus preserve the naturalistic processing of event sequences.

As in the neuroimaging study, our analyses were directed to evaluating differences in RT between the three sequence types comprising the second presentation (S_rep, S_half, and S_new). Subjects made few errors throughout the experiment (average performance: 96.8 % ± 1.1 % correct responses) with no significant differences in number of errors between sequence types: S_rep (5.6 ± 3.0), S_half (6.8 ± 2.3), and S_new (5.6 ± 3.0): F_(2,14) = 1.0, p = 0.4). As we expected, there was a significant main effect of sequence type on RT (two way analysis of variance [ANOVA]; factors: sequence type and item position: F_(2,14) = 7.7, p < 0.01) with subjects fastest in the S_rep condition (mean 573 msec) and slowest in the S_new condition (mean 598 msec), providing behavioral evidence of the utility of prior predictions in the S_rep sequence (Figure 5). There was also a significant interaction between sequence type and item position (F_(6,42) = 2.8, p < 0.05). Crucially, this effect was revealed by a subsequent ANOVA to be due to a significant RT cost when initial predictions in the S_half condition were violated at the third position in the sequence, as evidenced by a significant interaction between item position (second and third) and sequence type (S_half and S_new: F_(1,7) = 17.0, p < 0.01; Figure 5). A two-tailed t-test also confirmed that subjects were significantly slower at the third position in the S_half condition as compared to the S_new condition (t₍₇₎ = 3.5, p < 0.01). These results, therefore, provide compelling evidence that sequential predictions generated on the basis of a single prior experience significantly influence subjects' subsequent behavior, most strikingly manifesting as a marked RT cost in the S_half condition when expectations conflict with current experience.

Figure 5. Results of Experiment 2 (Behavioral): RT Cost of the Violation of Prior Expectations in the S_half Condition.

During this experiment, a separate group of naive subjects performed a speeded semantic categorization task (see Materials and Methods). There was a significant effect of sequence type (i.e., collapsed across item position), with subjects fastest in the S_rep condition (black diamonds), reflecting the utility of sequential predictions based on a single prior experience (i.e., first presentation: see Results for details). Critically, a significant RT cost was observed in the S_half condition (red squares) at thethird item position, when prior predictions are violated by sensory reality. Note, no error bars are displayed because reported statistical comparisons reflect within-subject effects (as opposed to the condition-specific between-subject variance measure represented by bars). Green triangles indicate S_new condition.

Discussion

We used functional neuroimaging to test the hypothesis that the human hippocampus acts as an associative mismatch detector during the processing of novel sequential information, a form of associative novelty not previously studied. Hippocampal activation was observed to be maximal in the S_half condition, where predictions concerning which object will appear next in the sequence are violated by sensory reality. In contrast, the pattern of activation exhibited by the entorhinal cortex suggested a general response to sequence novelty per se. Further, the results of a separate behavioral experiment demonstrated that the presence of associative mismatch specifically at the third object in the S_half condition results in a substantial RT cost, providing evidence of the significant influence of prior expectations, and their violation, on current behavior. The present study, therefore, provides empirical support for theoretical models in which associative match-mismatch processing constitutes a cardinal mechanism underpinning the hippocampal contribution to novelty detection. More generally, our results also offer insights into the nature of fundamental computations carried out by the hippocampus, which may also underlie its unique contribution to episodic memory.

We observed that hippocampal activation was maximal in the S_half condition, in which the first half of the sequence (i.e., first two items) was repeated and the second half novel. This finding argues strongly against a hippocampal response to sequence novelty per se, which would predict maximal responses in the S_new condition. Instead, our findings are consistent with theoretical frameworks in which the hippocampus is viewed to operate as an associative mismatch detector during novelty detection [3,4,8,9], and in particular during the processing of sequence novelty [5,10,11]. According to this recently proposed computational model [5,10,11], a sensory cue (i.e., the first object in the sequence) triggers autoassociative recall of the entire stored sequence within the CA3 region of the hippocampus. The readout of sequential information is said to be time compressed because it is played out at a rate more rapid that the actual occurrence of events [10]. This process results in the output of sequential predictions to the CA1 region allowing the organism to anticipate what will happen next based on previous experience. The CA1 region is proposed to act as an associative match-mismatch processor, whereby predictions arriving from CA3 are compared with the sensory reality arriving directly from the entorhinal cortex [10]. Hippocampal novelty signals, therefore, are viewed to occur specifically in response to an associative mismatch situation, i.e., when there is a discrepancy between the stimulus that is predicted to appear next in the sequence, and the stimulus that actually appears. This situation only occurs in the S_half condition, and not in the other two sequence conditions in which predictions about which object would appear next were either confirmed at each stage (S_rep), or likely never set in motion due to reordering of the entire sequence (S_new) [33]. Results of the behavioral experiment provided evidence that predictions concerning which object would appear next, derived from a single prior experience (i.e., first presentation) significantly influenced subjects' behavior in our paradigm (i.e., during the second presentation). Critically, subjects' RT increased significantly between the second and third objects in the S_half condition, in the presence of an associative mismatch between prior expectations and impinging reality. As such, our results accord well with previous studies demonstrating the substantial RT cost when expectations are violated [35]. In contrast, no such acute slowing of subjects' RT (i.e., increase in RT between successive items in the sequence) was observed at any stage in the S_new condition, providing further confirmation that a significant associative mismatch situation does not arise in this condition. The present study, therefore, provides empirical support that the human hippocampus, like its rodent counterpart [13,14], does indeed detect associative mismatches in the environment, a process that significantly influences subsequent behavior and is widely held to play a critical role in novelty detection.

The entorhinal/perirhinal cortex, in contrast, was observed to respond to both conditions in which sequence novelty was present (S_half and S_new), as compared to the condition in which sequential information was repeated (S_rep). This pattern of activation, which is consistent with a general response to sequence novelty per se, may reflect the phenomenon of repetition suppression [1,2,36]. Repetition suppression refers to the reduced neural activity elicited by familiar, as compared to novel stimuli or stimulus configurations. Indeed, activation in a region of the right MTL close to that identified in our study (in response to sequence novelty) has been previously reported in relation to stimulus novelty [37]. Repetition suppression may be mediated by the tuning of neural representations, such that familiar stimuli tend to activate fewer neurons and therefore evoke less neural activity [1,36]. Although, the phenomenon of repetition suppression has been predominantly studied in relation to the coding of stimulus novelty [1,36], similar mechanisms have also been suggested to underlie the processing of associative novelty [1,38–40]. In this fashion, therefore, the entorhinal cortex may exhibit a more general response to sequence novelty, in contrast to the computationally refined associative match-mismatch processes operating within the hippocampus.

Our findings, therefore, in demonstrating that sequence novelty responses occur in both the hippocampus and the entorhinal cortex, accord with the view that the processing of associative information occurs in different areas of the MTL, and not solely the hippocampus [41,42]. However, our results suggest that functional specialisation within the MTL may exist along other lines, given the qualitatively different pattern of activation in relation to the three sequence types observed in the hippocampus, as compared to the entorhinal cortex. The hippocampus alone was found to operate as an associative mismatch detector, a process that is thought to rely upon both comparison-based and autoassociative computations [3,10]. As such, our results support the hypothesis that a key aspect of the hippocampal contribution to memory may perhaps be a unique ability to retrieve, through the process of pattern completion, the entire previously stored pattern from a partial input cue (e.g., first item in a sequence) [10,43–45]. In this way, the hippocampus may make a specific contribution, as distinct from other areas in the MTL, to the process of recollecting additional contextual details relating to an episode, and more generally to autobiographical memory [1,29].

We considered the possibility that encoding-related processes, as opposed to those involved in novelty detection, may contribute to the observed activations within the MTL. Indeed, it is widely accepted that events judged to be novel are afforded preferential status in terms of subsequent encoding into memory [2,18]. However, it would seem highly unlikely that the observed activations within the hippocampus reflect primarily encoding-related processes, rather than associative mismatch computations. First, the hippocampus was the only region in the brain found to exhibit a pattern of neural activity consistent with that of an associative mismatch detector. Hence, it is difficult to argue that a different brain region is responsible for initially detecting associative mismatches in the S_half condition, with the hippocampus merely engaged at a later stage (i.e., during subsequent encoding). Second, we observed a qualitatively different pattern of activity in relation to the three sequence types in the hippocampus and entorhinal cortex. This finding strongly suggests that these two MTL regions perform different computations during novelty detection, rather than merely participating in the encoding of novel information. Hence, although the engagement of encoding-related processes may naturally follow the detection of associative mismatch by the hippocampus, our results provide support for the theoretical validity [11] of distinguishing between these two processes and their underlying mechanisms.

Interestingly, our paradigm shares some similarities with a task that has been widely used in both animals and humans [29,46,47]. In this delayed non-matching to sample (DNMS) task, subjects must remember either a sample stimulus (e.g., single object: “item DNMS task”) or stimulus conjunction (e.g., object-location: “associative DNMS task”) across a variable delay period in order to select the correct stimulus (or stimulus-conjunction) at test. It is important to note, however, that in our paradigm, subjects performed an incidental target detection task so as to emphasize the naturalistic processing of object sequences and avoid any explicit requirements to make old or new judgements. As such, subjects were not required to, and reported that they did not, actively maintain sequential information across the delay period. There is considerable evidence that MTL lesions produce deficits in the DNMS task when sample stimuli are trial-unique (i.e., where no prior representations exist within the brain) even at delays of no more than a few seconds (e.g., between 6–8 s in references [48–50]). Of note, the effective delay in our paradigm was in the region of 10 s (i.e., time interval between the start of the first presentation and the critical third object in the second presentation), with several intervening objects presented during this time window. That selective hippocampal lesions only inconsistently produce impairments on the item-based DNMS task, but appear to produce deficits on associative DNMS tasks [50], has been suggested to reflect the primary role of the hippocampus in recollective or associative, as opposed to single-item recognition-related processes [29]. Our results, therefore, are consistent with convergent evidence that the MTL is critical for rapidly forming neural representations of novel trial-unique stimuli (i.e., as used in our experiment), which are accessible at test, and therefore support successful performance on the DNMS task [46,49,51].

The present results, therefore, suggest that the hippocampus acts as an associative mismatch detector, identifying when specific predictions based on “one-shot” sequential experiences are violated by impinging sensory reality. In contrast, other brain regions (e.g., striatum and prefrontal cortex) are widely held to detect expectancy violations under different circumstances, where learning occurs over repeated instances and is thought to be instantiated by prediction error–type signals [52–54]. The hippocampus, therefore, drawing on its putative capacity for the formation of one-shot associative representations [29,43,44,55], may play a specific role in predicting future events on the basis of a single prior episode (i.e., sequence of events). There is also evidence that the hippocampus plays a wide role in the detection of novel events that are unexpected given the general context of the experiment [6,31,56], rather than because they violate specific associatively retrieved predictions (i.e., as in our experiment). For example, the hippocampus appears to respond to infrequently presented oddball stimuli occurring against a background of repetitively presented stimuli, although the magnitude of this response appears to decrease over the course of the experiment [6,31,57]. It has therefore been suggested that the hippocampus may represent the statistical likelihood of expected outcomes in a given setting [35,58,59]. It should, however, be noted that hippocampal lesions appear not to affect the mismatch negativity (MMN) component of the event-related potential (ERP), a finding that has been attributed to the simple nature of the stimuli employed in such auditory paradigms [60]. The present study provides evidence that the hippocampus plays a specific role in detecting violations of predictions formed on the basis of a single sequential experience. Our results, therefore, accord with a wider role for the hippocampus in the detection of unpredictable events, through its ability to act as a comparator between past and present experience, although future studies are required to establish the range of circumstances under which this process occurs.

In conclusion, the present study establishes the biological reality of associative match-mismatch computations within the human hippocampus, a process that has been widely held to play a cardinal role, not only during novelty detection, but also in determining the appropriate operating mode of a network subserving memory (i.e., encoding vs. retrieval). Our results also suggest that the hippocampus may play an important role in predicting on the basis of a single prior experience how future events will unfold, and critically in detecting when violations of these expectancies occur. In the future, it will be of considerable importance to establish the range of situations in which the human hippocampus performs associative match-mismatch computations, and in particular whether these computations also operate during the processing of other types (e.g., spatial) of associative novelty.

Materials and Methods

Experiment 1: Neuroimaging.

Subjects: Seventeen healthy, right-handed native English speakers, who were currently undertaking or had recently completed a university degree, participated in this experiment (age range 18–32 y, average age 26 y; nine female). All subjects were free from neurological and psychiatric disease and gave informed written consent to participate in accordance with the local research ethics committee.

Stimuli: Approximately one thousand color pictures of objects were used in this study. Pictures of objects were obtained from the Hemera Photo-Objects image collection (http://www.hemera.com/) and included pictures of household objects, animals, and cars, but no faces. Objects were placed on a white background. There was an approximately equal balance of animate and inanimate objects in the main experimental conditions. In addition, the allocation of stimuli was randomized between subjects and experimental conditions. Examples of objects visually similar to those used in the experiment are shown in Figure 1.

Thirty-one pictures of background scenes were used in this study. They consisted of outdoor scenes (e.g., beach or mountains) and were obtained from the Hemera Photo-Objects image collection and a variety of other sources on the Internet. In order to minimize possible effects of scene novelty, subjects were pre-exposed to all background scenes prior to the scanning experiment, viewing each three times while performing an incidental task (i.e., rating the aesthetic appeal of the pictures). The allocation of stimuli was also randomized between subjects with each scene appearing approximately equally frequently in each of the experimental conditions. An example of a background scene visually similar to one used in the experiment is shown in Figure 1.

Experimental task and procedures: The four objects in each quartet were presented consecutively for 1 s each (first presentation; FIRST) with a 200-ms inter-stimulus gap. Each object within a quartet was presented centrally over a background scene, which was displayed continuously throughout the presentation of each object quartet. Following the display of a central fixation cross (2-s duration), the same object quartet was re-presented (second presentation), on either the same or different background scene (C_same or C_diff) in one of three sequence types (S_rep, S_half, or S_new). Thus, the design of our experiment was 3 × 2 factorial (with 25 trials of each condition) with factors: sequence type and object-context.

Importantly, each object was trial-unique, only appearing twice in total (i.e., in first and second presentations). The sequential order of objects during the second presentation was A-B-C-D (S_rep), A-B-D-C (S_half), and C-A-D-B or B-D-A-C (S_new) (with the first presentation of the object quartet denoted by A-B-C-D). The two possible orders for the S_new condition were selected so as to ensure all objects occupied a different ordinal position in the second presentation, with no forward (or backward) sequential associations preserved between first and second presentations. There was also a baseline condition (BASE; 25 trials) in which one object quartet was repeatedly presented with objects always in the same sequence and on the same background context. Subjects were pre-exposed to this quartet in a prescan training session, rendering objects, sequence, and background context entirely familiar in this condition. The order in which each experimental condition was presented was randomized with the constraint that a given condition did not occur more than two times consecutively.

Prior to scanning, subjects were informed that they would be participating in a speed target-detection task. Our choice of task reflected the desire to emphasize the naturalistic processing of object sequences (i.e., without explicit requirements to learn), while ensuring that subjects attended to the objects themselves. Subjects were instructed to respond with a key press if the same object repeated twice in a row (1-back task). There was no task related to the background scene, in line with the notion that in the real world, context forms the backdrop in which events occur. In order to encourage the processing of object quartets as individual “episodes,” subjects were told that they should only look out for two-in-a-row object repeats within (and not between) object quartets. Thus, although subjects were aware prior to scanning that objects would be presented in groups of four, they were not informed of our experimental manipulations. Subjects were warned that we would be monitoring their reaction times and thus were encouraged to stay alert throughout in order to respond to a repeat (i.e., target) as fast as possible, without compromising accuracy. Over the course of the entire experiment, there were 51 trials in which targets (i.e., two-in-a-row object repeats) were present, requiring subjects to respond with a key press. These trials were excluded from the main analyses. Target trials were equally distributed between all experimental conditions (including the baseline condition), such that the probability of a target occurring was constant across first and second presentations of object quartets.

After scanning, subjects participated in a debriefing session in which they were asked in general about how they performed the task, and any strategies they might have used.

fMRI design: The temporal pattern of stimulus presentation was designed to maximise statistical efficiency while preserving psychological validity, in line with established procedures [61–63]. The trial onset asynchrony (TOA) is not a simple integer multiple of the repetition time, TR (time for acquisition of one scanning volume = 4.05 s) [61]. Importantly, the haemodynamic response to events that occur a few seconds apart is explicitly modelled (via a haemodynamic response function [HRF]), and therefore can be estimated separately for each event type by implementing the general linear model as is standard when using statistical parametric mapping software (SPM2) (http://www.fil.ion.ucl.ac.uk/spm) (also see below) [62].

Imaging parameters and acquisition: T2*-weighted echo planar (EPI) images with BOLD (blood oxygen level–dependent) contrast were acquired (268 volumes/session; three scanning sessions) on a 1.5 tesla Siemens Sonata MRI scanner (Siemens plc, Erlangen, Germany) using a specialized sequence to minimize signal dropout in the MTL [64]. We used the following scanning parameters to achieve whole brain coverage: 45 oblique axial slices angled at 30° in the anterior-posterior axis, TR 4.05 s, 2-mm thickness (1-mm gap), echo time (TE) 30 ms, in-plane resolution 3 × 3 mm, field-of-view 192 mm, 64 × 64 matrix. A preparation pulse (duration 1 ms, amplitude + 1 mT/m*ms) was used in the slice selection direction to compensate for through-plane susceptibility gradients predominant in the hippocampus [65]. High-resolution (1 × 1 × 1 mm) T1-weighted structural MRI scan were acquired for each subject after functional scanning. These were co-registered to the functional EPIs, and averaged across subjects to aid localization.

fMRI data preprocessing (SPM2): Images were analyzed in a standard manner using the statistical parametric mapping software SPM2 (http://www.fil.ion.ucl.ac.uk/spm). After the first six “dummy volumes” were discarded to permit T1 relaxation, images were spatially realigned to the first volume of the first session, followed by spatial normalization to a standard EPI template, resulting in a functional voxel size of 3 × 3 × 3 mm. Normalized images were smoothed using a Gaussian kernel with full width at half maximum of 8 mm.

fMRI data analysis (SPM2): Following preprocessing, the event-related fMRI data were analyzed in SPM2 using the general linear model (GLM) using established procedures [61,62]. As a first step, vectors corresponding to the onset times for each of the condition types were created. Our main interest was in the 4.8-s period during which the object quartet was presented. Therefore these vectors were then coded as boxcar functions (of 4.8-s duration) constituting regressors of interest in the GLM (i.e., the first-level design matrix). These regressors were then convolved with the canonical HRF. One regressor of no interest was also included coding for target trials. In addition, subject-specific movement parameters (derived from realignment) were included as regressors of no interest. A high-pass filter with a cutoff of 128 s was employed to remove low-frequency drifts. Temporal autocorrelation was modelled using an AR(1) process.

Model estimation proceeded in two stages. In the first stage, condition-specific experimental effects (parameter estimates, or regression coefficients, pertaining to the height of the canonical HRF) were obtained via the GLM in a voxel-wise manner for each subject. In the second (random-effects) stage, subject-specific linear contrasts of these parameter estimates, collapsed across the three sessions, were entered into a series of one-sample t-tests (as is standard when using statistical parametric mapping [SPM] and a factorial design [61]), each constituting a group-level statistical parametric map. Condition-specific percentage signal changes were calculated using the MarsBar SPM toolbox (http://marsbar.sourceforge.net).

Given our clear a priori anatomical hypotheses, we report activations within the MTL at p < 0.001 uncorrected for multiple comparisons, with an extent threshold of more than five contiguous voxels. Hippocampal activations survived small volume correction (SVC), performed using an anatomical mask drawn around the left hippocampus on the average structural MR image for all participants. The SVC procedure, as implemented in SPM2 using the family-wise error (FWE) correction (p < 0.05), allows results to be corrected for multiple non-independent comparisons with a defined region of interest, in this case the left hippocampus. Activations in other brain regions are reported for completeness at a threshold of p < 0.001 uncorrected for multiple comparisons, but were only considered significant if they survive whole brain correction for multiple comparisons at p < 0.05 (in line with established procedures [61]). All activations are displayed on sections of the average structural image of all the participants. Reported voxels conform to MNI (Montreal Neurological Institute) coordinate space. Right side of the brain is displayed on the right side.

We also performed two additional event-related analyses: in one analysis, the trial onset for each condition was set at the critical third object in each quartet, and in the other, the trial onset was set at the first object in the quartet. As in the primary analysis detailed above, vectors corresponding to the onset times for each of the condition types were created. These vectors were then coded as delta (i.e., stick) functions constituting regressors of interest in the GLM (i.e., the first-level design matrix). Next, these regressors were convolved with the canonical HRF [62]. One regressor of no interest was also included coding for target trials. In addition, subject-specific movement parameters (derived from realignment) were included as regressors of no interest. A high-pass filter with a cutoff of 128 s was employed to remove low-frequency drifts. Temporal autocorrelation was modelled using an AR(1) process. Further analysis proceeded in the same way as the primary analysis.

Experiment 2: Behavioral subjects:

This experiment was conducted in a separate group of naive subjects (n = 8; average age 27 y; three female) to ask whether predictions generated on the basis of previous experience (i.e., first presentation), and subsequently violated in the S_half condition, are associated with a detectable change in subjects' subsequent behavior (i.e., using an RT measure as an index of behavior during the second presentation). All subjects were free from neurological and psychiatric disease and gave informed written consent to participate in accordance with the local research ethics committee.

Stimuli, experimental design, and procedures: A different set of 540 color pictures of objects also obtained from the Hemera Photo-Objects image collection (http://www.hemera.com/) were used in this experiment, divided into an equal number of four semantic categories (animals, food, manmade objects, and vehicles). Objects were placed on a white background. Given the aim of this behavioral study and our neuroimaging results, no background scenes were used in this experiment. Objects were presented in 135 quartets in total, in the three experimental conditions (S_rep, S_half, and S_new: 45 trials of each). The allocation of stimuli was randomized between subjects and experimental conditions.

In line with the results of a pilot behavioral study, the timings of object presentation were slightly changed from that of the neuroimaging study, in order to afford subjects adequate time to perform the semantic categorization task. As such, the four objects in each quartet were presented centrally for 1.5 s each (i.e., first presentation). As in the neuroimaging experiment, following the display of a central fixation cross (2-s duration), the same object quartet was re-presented (second presentation) in one of three sequence types (S_rep, S_half, or S_new). Of note, the presentation of stimuli was constrained such that two objects in the same semantic category would not appear consecutively. This was done in order to avoid possible effects on the RT of consecutive responses using the same key. However, it was permitted that two objects in a given semantic category could appear in each quartet.

During the scanning experiment, subjects performed an incidental task (i.e., 1-back target detection) so as to avoid contaminating the brain activation data with the influence of frequent motor responses. In this experiment, however, subjects performed a speeded semantic categorisation task on each object presented, entering their response using four keys on a standard keyboard. This task was chosen in order to avoid any explicit requirements to learn, and thus preserve the naturalistic processing of event sequences. Subjects were warned that we would be monitoring their reaction times and thus encouraged to respond as fast as possible, without compromising accuracy. Prior to the actual experiment, subjects were instructed as to which key mapped to a given semantic category and received practice to familiarize themselves with the task (i.e., 25 trials, approximately 7 min). The mapping of keys to semantic categories was the same in all subjects. The experiment was divided into five runs lasting approximately 8 min each, with a 1-min break in between each run. Following the completion of the experiment, subjects were debriefed in exactly the same way as described previously. As expected, none of the subjects reported actively maintaining sequential information across the delay period, instead remaining focused on the semantic categorization task.

Analysis: As in the neuroimaging study, our analyses were directed to evaluating differences in RT between the three sequence types comprising the second presentation (S_rep, S_half, and S_new). Additionally, RTs were separated according to position in the sequence (e.g., third item in object quartet) in order to allow us to examine the RT cost of violation of prior predictions in the S_half condition at the third item in the sequence. Analysis was conducted using SPSS software (http://www.spss.com/index.htm). Data were entered into a 3 (sequence type) × 4 (item position) ANOVA. Further, ANOVAs were conducted in order to examine effects specific to particular sequence types and item positions (see Results for details). Effects are reported as significant at p < 0.05.

Supporting Information

Figure S1. Brain Areas Significantly More Active during First Presentation as Compared to Baseline Condition: FIRST > BASE.

Significantly greater activation observed in this comparison (FIRST > BASE) in a distributed network of brain regions, including left hippocampus (x, y, z = 24, −24, −15; z = 3.79), right entorhinal/perirhinal cortex (x, y, z = 21, -−12, −27; z = 4.96), left medial prefrontal cortex (x, y, z = −3, 36, −6; z = 4.07), and visual cortex (see Table S1). During the baseline condition, one object quartet was repeatedly presented with objects always in the same sequence and on the same background context (see Materials and Methods). Subjects were pre-exposed to this quartet in a prescan training session, rendering objects, sequence, and background context entirely familiar in this condition. “Glass brain” figures are displayed above. Activations shown on the averaged structural MRI scan of the 17 participants (displayed below). The color bar indicates the t-statistic associated with each voxel and the Z-score equivalent. Threshold is set at p < 0.001 uncorrected for multiple comparisons.

(3.5 MB TIF)

Figure S2. Event-Related Analysis with Trial Onset Set at Critical Third Object: Main Effect of Sequence Type (S_half > S_rep).

Results of comparison of S_half and S_rep conditions (S_half > S_rep, collapsed across object-context factor) in analysis in which trial onset was set at the critical third object in the sequence in which an associative mismatch occurs in the S_half condition (see Materials and Methods). “Glass brain” figures are displayed above. Activations shown on the averaged structural MRI scan of the 17 participants (displayed below). R, right side of the brain. The color bar indicates the t-statistic associated with each voxel and the Z-score equivalent. Activation in left hippocampus (x, y, z = −27, −15, −18; z = 3.96) evident in sagittal and coronal sections. Activation in right entorhinal/perirhinal (x, y, z = 21, 9, −30; z = 3.94) also evident on coronal and axial sections. Threshold is set at p < 0.005 uncorrected for display purposes.

(2.8 MB TIF)

Table S1. Brain Areas Significantly More Active during First Presentation as Compared to Baseline Condition: FIRST > BASE.

All values, p < 0.001 uncorrected.

(25 KB DOC)

Abbreviations

ANOVA

analysis of variance

DNMS

delayed non-matching to sample

fMRI

functional magnetic resonance imaging

GLM

general linear model

HRF

haemodynamic response function

MTL

medial temporal lobe

RT

reaction time

Funding Statement

This work was funded by a Wellcome Trust senior research fellowship in basic biomedical science to EAM.