Item memory, context memory and the hippocampus: fMRI evidence

Abstract

Dual-process models of recognition memory distinguish between the retrieval of qualitative information about a prior event (recollection), and judgments of prior occurrence based on an acontextual sense of familiarity. fMRI studies investigating the neural correlates of memory encoding and retrieval conducted within the dual-process framework have frequently reported findings consistent with the view that the hippocampus selectively supports recollection, and has little or no role in familiarity-based recognition. An alternative interpretation of these findings has been proposed, however, in which it is argued that the hippocampus supports the encoding and retrieval of ‘strong’ memories, regardless of whether the memories are recollection- or familiarity-based. Here, we describe the findings of eight fMRI studies from our laboratory: one study of source memory encoding, four studies of the retrieval of contextual information, and three studies of continuous recognition. Together, the findings support the proposal that hippocampal activity co-varies with the amount of contextual information about a study episode that is encoded or retrieved, and not with the strength of an undifferentiated memory signal.

Introduction

There is a broad consensus that recognition memory is supported by two different processes, usually referred to as recollection and familiarity (Mandler, 1980; Yonelinas, 2002; Wixted and Mickes, 2010). Recollection occurs when a recognition test item elicits retrieval of qualitative information about the study episode. This information includes not only the identity of the studied item, but also details about the context in which it was studied. By contrast, familiarity supports a sense of past occurrence that is devoid of contextual information. Whereas both recollection and familiarity can support simple ‘old/new’ recognition judgments, judgments based on the content of an episode – source memory or ‘Remember’ judgments for example (see below) – depend largely on recollection.

The functional characteristics of recollection and familiarity, and their neural substrates, are currently matters of debate. Contentious issues include the questions of whether the memory signal associated with recollection is better modeled as a threshold or a continuous process (Yonelinas et al., 2010; Wixted and Mickes, 2010), and whether recollection and familiarity are differentially dependent upon the hippocampus (Eichenbaum, Yonelinas, and Ranganath, 2007; Squire, Wixted, and Clark, 2007). Here, we focus on the second of these issues as it is informed by functional neuroimaging studies, although we touch upon the first issue also.

The proposal that recollection, but not familiarity, is dependent upon the hippocampus has been advanced by numerous authors. Several lines of evidence have been interpreted in favor of the proposal, although none has gone unchallenged. For example, whereas some studies of patients with lesions restricted to the hippocampus have reported disproportionate deficits in estimates of recollection (e.g., Mayes et al., 2004; Holdstock et al., 2002; Aggleton et al., 2005), studies of other, seemingly similar patients, have reported that recollection and familiarity are impaired to an equivalent extent (e.g., Wais et al., 2006; Manns et al., 2003; Cipolotti et al., 2006; Jeneson et al., 2010). And whereas there is compelling evidence from studies of experimental animals that hippocampal lesions can have little or no impact on recognition memory of single items as assessed by tasks such as delayed non-match to sample (for review, see Brown and Aggleton, 2001) or spontaneous exploration (e.g., Good et al., 2007), it is unclear how directly these findings relate to the constructs of recollection and familiarity, which are difficult to operationalize in experimental animals (although see Fortin et al., 2004).

An important line of evidence in support of the proposal that recollection is disproportionately dependent on the hippocampus comes from functional neuroimaging studies in healthy humans that employed event-related fMRI. Such studies have investigated both the neural correlates of the encoding processes associated with successful recollection on a later memory test, and the correlates of successful recollection at the time of retrieval. In these studies, recollection was almost invariably operationalized either in terms of successful versus unsuccessful memory for a contextual feature of the study episode (source memory), or as the difference in neural activity when recognition of a test item was accompanied by a phenomenal sense of recollection compared with when it was accompanied only by a sense of familiarity (‘Remember vs. Know’, Tulving, 1985). In studies of both encoding (e.g., Ranganath et al., 2004; Uncapher & Rugg, 2005; Otten, 2007; Davachi, Mitchell, and Wagner, 2003; Duarte, Henson, and Graham, 2011; Kensinger and Schacter, 2006) and retrieval (e.g., Diana, Yonelinas, and Ranganath, 2010; Eldridge et al., 2000; Yonelinas et al., 2005; Montaldi et al., 2006; Cohn et al., 2009) it has been reported that, relative to familiarity-driven recognition, recollection is associated with enhanced hippocampal activity. These findings have been taken as evidence that that the hippocampus plays a selective role in supporting recollection, possibly because of its unique ability to bind the different components of a study episode into a cohesive memory representation (Diana, Yonelinas, and Ranganath, 2007; Mayes, Montaldi, and Migo, 2007).

Recently, an alternative account of these fMRI findings has been advanced (Squire, Wixted, and Clark, 2007; Wixted, Mickes, and Squire, 2010). By this account, encoding- and retrieval-related hippocampal activity are neural correlates not of recollection, but of ‘strong’ memory - as operationalized by the accuracy and confidence of recognition judgments -regardless of whether the strength of the memory is due to recollection, high familiarity, or a combination of the two memory signals. In support of this account, Wixted, Mickes, and Squire (2010) argued that contrasts intended to identify the neural correlates of recollection, for example, contrasts between items attracting correct versus incorrect source judgments, or items endorsed ‘Remember’ versus ‘Know’, are invariably confounded with differences in item recognition accuracy and hence memory strength. Wixted, Mickes, and Squire (2010) noted that even in studies in which confidence judgments were employed to segregate ‘unrecollected’ test items according to their level of familiarity (Ranganath et al., 2004; Montaldi et al., 2006; Yonelinas et al., 2005; Cohn et al., 2009), recognition accuracy was invariably higher for items endorsed as recollected than it was for highly familiar items (but see Montaldi and Mayes, 2010). Wixted, Mickes, and Squire (2010) argued that it was this difference in item accuracy, and not the distinction between recollection and familiarity, that was responsible for the differential hippocampal activity reported in those studies.

In the following sections, we describe the findings from a series of studies from our laboratory that bear directly on the question of whether the construct of memory strength is sufficient to account for modulation of encoding- and retrieval-related hippocampal activity, as this activity is assessed with fMRI. On the basis of these findings, we argue that hippocampal activity is sensitive neither to variation in the strength of an undifferentiated memory signal, nor to whether a test item elicits a subjective sense of recollection. Instead, hippocampal activity reflects the amount of contextual information that is encoded during a study episode, or that is retrieved in response to a test item.

Measurement of memory strength and recollection

Before turning to our empirical findings we briefly review the behavioral measures that we and others have employed to estimate memory strength and recollection in studies of recognition memory. As the construct is employed in the literature cited above (e.g. Squire, Wixted, and Clark, 2007; Wixted et al., 2010), strength is operationalized in terms of the accuracy (pHit/(pHit+pFalse Alarm)) and confidence with which recognition memory test items are correctly endorsed as studied, and we follow that practice below. We defer discussion of the value of the construct of memory strength in understanding the neural correlates of memory performance until later.

We have employed two different procedures to operationalize recollection. The first is the ‘Remember/Know’ procedure, introduced by Tulving (1985; see Migo, Mayes, and Montaldi, in press, for a recent review of the procedure from a methodological perspective). In its simplest form, the procedure requires subjects to signal whether recognition of a test item is accompanied (Remember judgment) or is not accompanied (Know judgment) by the retrieval of a contextual detail or details about the item’s study presentation. Remember and Know judgments are assumed to map onto the constructs of recollection and familiarity, respectively. The procedure has been criticized on the grounds that it merely distinguishes between relatively strong and relatively weak memories (e.g., Donaldson, 1996). Recent evidence suggests however that this is not necessarily the case. Rather, independently of differences in memory strength, Remember and Know judgments can indeed segregate test trials according to whether or not contextual (source-specifying) information was retrieved, (Wixted and Mickes, 2010; Ingram, Mickes, and Wixted, 2012). The second procedure adopted to operationalize recollection requires subjects to make an explicit judgment about a specific contextual feature of the study episode, for example, whether a test word was presented at study in a red or a green font. It is typically assumed that accurate retrieval of such ‘source’ information is indicative of the recollection of the relevant contextual feature, and hence of the retrieval of qualitative information about the study episode. It is important to note, however, that failure to retrieve source information does not necessarily mean that recollection failed; the possibility that recollection occurred, but that it did not include retrieval of contextual features relevant to the source judgment (‘non-criterial recollection’, see Yonelinas and Jacoby, 1996), cannot easily be discounted.

In several of the experiments described below, we employed variants of the Remember/Know and source memory procedures to identify and characterize recollection-related hippocampal activity. Additionally, we estimated the memory strengths of ‘recollected’ and ‘unrecollected’ test items when this was possible.

Hippocampal activity and the encoding of source and item information

Below, we describe data drawn from a larger study of the effects of age on the neural correlates of source memory encoding (Mattson et al., unpublished data), focusing on findings from the 17 young participants (ages 18–27, 6 male, all right-handed with no reported neurological or psychiatric histories). During the scanned study phase, subjects viewed a series of 180 color pictures of objects, each of which was preceded by a cue that signaled whether the object should be judged as to whether it would be more likely to be found indoors or outdoors, or was smaller or larger than a shoebox. fMRI data were acquired and analyzed according to our standard methods (e.g., Gottlieb and Rugg, 2011). In an unscanned test phase that began approximately 25 min after exiting the scanner, subjects viewed the 180 critical study pictures intermixed with 90 new items. The requirement was to first make an ‘old/new’ judgment on each picture and, for each item endorsed ‘old’, to then judge whether the item had been subjected to an indoors/outdoors or a size decision at study. Both the item and source judgments were made using confidence ratings: ‘confident old’, ‘unconfident old’, ‘don’t know’, ‘unconfident new’ and ‘confident new’ for item memory, and ‘confident location’, ‘unconfident location’, ‘don’t know’, ‘unconfident size’ and ‘confident size’ for source memory.

Behavioral performance on the test task is summarized in Tables 1 and 2. Recognition accuracy (and thus memory strength – see above) for items receiving a ‘confident old’ item memory judgment was at ceiling, whereas the accuracy of items accorded an ‘unconfident old’ judgment was significantly lower (accuracies of .99 and .88 respectively, p < .01). Additionally, as has been reported previously (Kirwan, Wixted, and Squire, 2008; Wais, Squire, and Wixted, 2010), source accuracy was substantially higher for items accorded confident than unconfident item memory judgments (.86 and .68 respectively, p < .001). Therefore, to avoid any confound between memory strength and source accuracy, only study trials that went on to receive a confident item judgment were employed to identify the neural correlates of source encoding.

Table 1.

Item memory performance for old and new trials by response type (Mattson et al., unpub.)

Trial type	CO	UO	DK	UN	CN
Old	.76 (.02)	.08 (.01)	.03 (.01)	.07 (.01)	.06 (.01)
New	.01 (.00)	.01 (.00)	.04 (.02)	.15 (.04)	.78 (.06)

Note: Response abbreviations correspond to the following: CO, confident old; UO, unconfident old; DK, don’t know; UN, unconfident new; CN, confident new. Standard errors are shown in parentheses.

Table 2.

Source memory performance for confidently recognized old items according to confidence of source judgment (Mattson et al. unpub.)

Source confidence	Source correct	Source incorrect
Confident	.67 (.03)	.07 (.01)
Unconfident	.17 (.03)	.06 (.01)

Note: Performance values do not sum to 1.00 due to exclusion of don’t know source responses. Standard errors are shown in parentheses.

The fMRI analyses focused on contrasts between three classes of study items: ‘source hits’, ‘source misses’, and ‘item misses’. Source hit items were those that went on to receive both a confident old item judgment and a confident, accurate source judgment on the subsequent memory test. Source misses refer to study items that also received a confident old item judgment on the subsequent test, but for which the source judgment was either inaccurate or not given (don’t know response). Item misses refer to items that were either recognized with low confidence or were misclassified as new on the subsequent test. Thus, the contrast between source hits and source misses permitted identification of the neural correlates of the encoding of ‘strong’ source memories when item memory strength was high and equivalent for items associated with successful versus unsuccessful encoding of source information. The contrast between source misses and item misses was employed to identify the neural correlates of the encoding of strong item memories.

The outcomes of these contrasts are illustrated in Figure 1. For the contrast between source hits and source misses (height thresholded at p < .001 with a cluster extent threshold corrected (p < .05) for a mask encompassing the MTL), a single cluster localized to the right hippocampus was identified (MNI coordinates (27, −19, −17); Z = 3.84; see Figure 1A). The associated parameter estimates are illustrated in Figure 1A, where it can be seen that item misses elicited a level of activity similar to that elicited by source misses. When the same statistical thresholds were applied to the source miss > item miss contrast, no voxels were identified within the MTL at the corrected extent threshold (7 voxels). Reducing the height threshold to p < .005 led to the emergence of a 19 voxel cluster located in left perirhinal cortex (MNI coordinates (−36, −10, −29); Z = 3.37; see Figure 1B). As is evident from the parameter estimates illustrated in Figure 1B, this latter effect was also evident for source hits.

Figure 1.

Findings from Mattson et al. (unpublished data). A: The top panel displays the outcome of the source hit > source miss contrast (thresholded at p < .001). The bottom panel displays the average across-subjects parameter estimates for each response category in the right hippocampal peak voxel (coordinate of 27 -19 -17). B: The top panel displays the outcome of the source miss > item miss contrast (thresholded at p < .005). The bottom panel displays the average across-subjects parameter estimates for each response category in the left perirhinal peak voxel (coordinate of -36 -10 -29).

To summarize, we found encoding-related activity in the hippocampus to be predictive of successful source memory, whereas activity in perirhinal cortex predicted successful item memory. These findings are consistent with those reported in several prior studies in which item memory strength was not equated between study items that attracted correct and incorrect source judgments (e.g., Davachi et al., 2003; Ranganath et al., 2004; Kensinger and Schacter, 2006; Duarte, Henson, and Graham, 2011). Thus, the present findings suggest that prior findings were not a consequence of the confounding effects of memory strength. One caveat to this conclusion arises from the fact that the accuracy of item memory in the present experiment was at ceiling. We cannot rule out the possibility that although the memory strengths associated with accurate and inaccurate source judgments were both very high, a difference in strength nonetheless existed between the two response categories.

The foregoing experiment is not the first to address the question whether encoding-related activity in the hippocampus is predictive of source memory performance on a later memory test when item memory strength is equated. Kirwan, Wixted, and Squire (2008) described an experiment similar in several respects to the one described here. When these authors contrasted the activity elicited on study trials for which the subsequent source judgment was accurate or inaccurate, they found that study words that were both later recognized and for which the source (one of two encoding tasks) was correctly retrieved elicited greater right hippocampal activity than did recognized words associated with inaccurate source memory (a result replicated in the study we report here). No hippocampal effect was evident, however, when the analysis of subsequent source memory effects was restricted to items that had been recognized with high confidence, and was conducted by performing a linear trend analysis to identify study activity that co-varied with the accuracy and confidence of the subsequent source memory judgments. Kirwan, Wixted, and Squire (2008) proposed that their initial finding of a hippocampal subsequent source memory effect was due to the confounding influence of memory strength (like us, they found that items attracting correct source judgments on the later memory test were recognized with higher confidence than items attracting inaccurate judgments). At odds with this proposal, however, subsequent memory effects associated with differing levels of memory strength were found not in the right hippocampal region that demonstrated the subsequent source memory effect, but rather, on the perimeter of the hippocampus bilaterally (see Song, Jeneson, and Squire, 2011 for similar findings). An alternative possibility is that the dependence of the right hippocampal subsequent source memory effect on whether memory strength was or was not equated reflects differences in the efficiency of the statistical methods used to identify the effect in each case (linear trend analysis as opposed to a binary contrast).

Hippocampal activity and amount of recollected information

In this section we describe previously unpublished analyses of data from three studies, focusing on the relationship between retrieval-related hippocampal activity and amount of information recollected in response to a recognition memory test item. The first study (Vilberg and Rugg, 2007) comprised two experiments (Ns of 14 in each case) that employed almost identical procedures. As in the original paper, the data are presented here collapsed across the two experiments. In each experiment, subjects initially studied a series of picture pairs under the requirement to imagine the two denoted objects interacting. In the scanned test phase, the items comprised a series of single pictures, two thirds of which were studied and one third of which were new. One of four responses was required for each test item: if both the test picture and its studied pairmate could be recollected a ‘Remember 2’ (R2) response was to be given, if the test picture could be recollected along with a detail or details about the study episode other than its pairmate, a ‘Remember 1’ (R1) judgment was required, items that were judged as studied but for which recollection was absent were assigned a ‘Know’ (K) response, and items for which the study status was uncertain, or that were judged as unstudied, were given a ‘New’ response. We assumed that R2 judgments would, on average, be associated with the recollection of more information than would R1 judgments. This assumption was validated with a preliminary behavioral study and a follow-up memory test conducted after subjects had completed the scanned test phase.

The second study (Vilberg and Rugg, 2009a) employed a design that was essentially identical to that just described (Vilberg and Rugg, 2007). However, word pairs rather than picture pairs were presented at study (the task being to generate a sentence incorporating the two words), and single words were employed as test items. As in the earlier study, the test requirement was to make one of two different Remember responses (R2 or R1) depending on whether or not recollection included memory for the test item’s studied pairmate, and to respond Know to items that were judged old solely on the basis of a sense of familiarity.

In the third experiment (Vilberg and Rugg, 2009b) subjects initially studied a series of picture pairs superimposed on a complex scene. As in the first experiment, the requirement was to imaging the two objects interacting. To vary the amount of information that would be encoded, and thus the amount that would be recollected on the later memory test, the study items were presented for either 6s or 1s. Test items comprised studied and unstudied single pictures, with the requirement to make an R, K, or New judgment to each item. We confirmed that the manipulation of study duration modulated the amount of information recollected with a preliminary behavioral study and a post-scan test on which, for each item endorsed ‘R’, subjects were required to verbally report the recollected content. Compared to the 1s study trials, the 6s trials were associated with retrieval of significantly more contextual details.

Trials in each of the three experiments associated with the recollection of a relatively large amount of information are hereafter referred to as R-large trials (R2 hits in experiments 1 and 2; R hits for items studied for 6s in experiment 3), whereas trials associated with recollection of a relatively small amount are referred to as R-small (R1 hits in experiments 1 and 2; R hits for items studied for 1s in experiment 3). Table 3 summarizes item recognition accuracy for the R-large, R-small and K response categories in the first and second studies (accuracy cannot be calculated as a function of amount recollected in experiment 3 because the different amounts were not associated with distinct response categories). As can be seen from the table, accuracy increased as a function of amount of information recollected (R-large vs. R-small: F(1, 44) = 7.11, p < .025; R-small vs. K: F(1, 44) = 19.89, p < .001), reaching ceiling for the R-large condition. Importantly, accuracy exceeded 90% for K responses, indicating that item memory was strong even in the absence of phenomenal recollection. In the third study, accuracy (collapsed across the study duration manipulation) for R and K judgments was also high, although significantly higher in the case of R judgments (.99 and .89 respectively; F(1, 17) = 9.96, p < .01).

Table 3.

Item recognition accuracy for old items classified as R-large, R-small, and K in the experiments reported by Vilberg and Rugg (2007; 2009a)

Experiment	R-large	R-small	K
Vilberg & Rugg (2007)	.99 (.00)	.98 (.01)	.91 (.02)
Vilberg & Rugg (2009a)	.99 (.00)	.97 (.01)	.91 (.02)

Note: Accuracy is defined as hits/(hits + false alarms). Standard errors are shown in parentheses.

At our standard pre-experimental statistical threshold (p < .001), separate voxel-wise analyses of each of these data sets revealed little evidence for differences in hippocampal activity across the different conditions. Consistent effects were evident, however, when a region-of-interest approach was adopted. In our first study (Vilberg and Rugg, 2007) we had reported that exploratory analyses at a lowered threshold (p < .01) identified a left hippocampal region where activity elicited by R-large items exceeded the activity elicited by items endorsed K (peak coordinate -21, -21, -18; Z= 2.44). The parameter estimates associated with R-large, R-small, and K hit trials for a 3mm sphere centered on that voxel are shown for each experiment in Figure 2A–C. Parameter estimates for studied items misclassified as new (Misses) are also included in the figure for the first and third experiments (there were too few Miss trials in the second experiment to permit estimation). As is evident from Figure 2, in all three studies retrieval-related activity elicited by R-large trials was greater than that for K trials. Moreover, in studies 2 and 3, R-large activity also significantly exceeded the activity elicited on R-small trials. In no study did the activity elicited on R-small trials exceed that for K trials. According to Fisher’s procedure for combining significance levels of independent tests of the same null hypothesis (Fisher, 1950), the conjoint one-tailed significance levels across the three studies for the R-large > R-small and R-large > K contrasts were p < .001 and p < .002 respectively. The conjoint probability of the R-small > K contrast was far from significant (p > .1).

Figure 2.

Parameter estimates associated with R-large, R-small, and K hit trials for a 3mm radius region of interest centered at -21, -21, -18 are displayed for the three experiments conducted by Vilberg and Rugg (A: 2007; B: 2009a, C: 2009b). The region of interest is superimposed on an individual subject’s T1-weighted anatomical image in panel D. Misses are displayed for illustrative purposes only. ** = p < .01, * = p < .05.

The outcome of this retrospective analysis of these three studies is clear: at least in the hippocampal region identified here, retrieval-related activity was greater when subjects recollected relatively large amounts of information about a study event (R-large trials) than when they recollected relatively small amounts of information (R-small trials) or when phenomenal recollection was absent (K trials). Moreover, R-small items elicited a level of hippocampal activity that was statistically indistinguishable from that elicited by items endorsed as K, despite the marked difference in memory strength between the two classes of item.

Hippocampal activity, phenomenal recollection and source retrieval

The aim of the experiment described in this section (Yu, Johnson, and Rugg, in press) was to investigate retrieval-related activity in the hippocampus as a function of both phenomenal recollection and the amount of source-specifying information retrieved. Subjects studied a series of pictures that were presented to the left or right of central fixation. While undergoing scanning, they viewed a mixture of studied and unstudied items with the requirement to make an initial R/K/New judgment to each item and, for any item endorsed R or K, to then signal the item’s study location, using a 6-point confidence scale (‘high confidence left’, ’moderate confidence left’, ‘low confidence left’, ‘high confidence right’, ‘moderate confidence right’, ‘low confidence right’). We assume that the greater the confidence with which an accurate source decision was made, the greater the amount of source-specifying information that was retrieved.

Accuracy of item memory (memory strength) for R and K judgments is summarized in Table 4, broken down for items given an R response according to the confidence/accuracy of the associated source memory judgment. Highly and moderately confident accurate source judgments are designated as R-high and R-mod response categories. Low confidence and inaccurate judgments make up the R-weak category. K judgments were collapsed over source accuracy and confidence to form a K-all category. Item accuracy was markedly greater for items endorsed R than K, and increased as a function of source accuracy. Importantly, accuracy for items endorsed R but for which the source judgment was unconfident or inaccurate (‘R-weak’ items; 92%), and accuracy for ‘K-all’ items (77%) spanned the range argued by Wixted, Mickes, and Squire (2010) to be sufficient to give rise to differential hippocampal activity (see Introduction).

Table 4.

Item memory accuracy as a function of subsequent source memory judgments in Yu, Johnson, and Rugg (in press)

Experiment	R-High	R-Mod	R-Weak	K-All
Yu et al. (in press)	.99 (.02)	.97 (.03)	.92 (.09)	.77 (.18)

Note: Accuracy is defined as hits/(hits + false alarms). Standard errors are shown in parentheses.

The key fMRI findings are summarized in Figure 3. In brief, the hippocampal activity elicited by K-all items and items receiving an R-weak response did not differ (Figure 3B). Instead, activity elicited by items endorsed R co-varied with the confidence/accuracy of the associated source judgment. Therefore it appears that hippocampal activity co-varied not with memory strength, but with the amount of contextual (source-specifying) information that was recollected. A subsidiary analysis investigated whether the failure to find a difference in hippocampal activity for R-weak versus K-all trials was due to the inclusion in the K-all response category of items attracting highly or moderately confident source judgments. We addressed this question by forming a ‘K-weak’ response category analogous to the R-Weak category described above. Like the R-weak > K-all contrast, the R-weak > K-weak contrast also failed to identify hippocampal effects.

Figure 3.

A: Outcome of an omnibus ANOVA employed to identify response category effects in the study of Yu, Johnson, and Rugg (in press) (p < .001, 2-tailed). B: Across subject parameter estimates from the peak voxel at -30, -16, -20. *** = p < .001, * = p < .05.

Not shown in Figure 3 are the findings for studied items incorrectly endorsed as new (Misses). These items elicited hippocampal responses that were reliably greater than those elicited on K trials, a finding that echoes the results of the continuous recognition studies described in a later section. The finding should be interpreted cautiously, however, since items judged old were associated with two successive responses (R/K followed by a source judgment), whereas items judged new received only a single response, leading to a confound between judged study status and response demands. The same confound is present in the studies of Wais, Squire and Wixted (2010) and Smith, Wixted, and Squire (2011) (see below).

Recollection, familiarity, and retrieval-related hippocampal activity

The findings described in the preceding two sections are strongly convergent. They suggest that retrieval-related hippocampal activity is sensitive neither to differences between test items in their memory strengths, nor to whether or not an item elicits a phenomenal sense of recollection (i.e., attracts an R endorsement). Rather, hippocampal activity co-varies with the amount of information recollected about a study episode. Below, we discuss the implications of these findings for understanding the role of the hippocampus in memory retrieval.

It might be argued that our conclusion that retrieval-related hippocampal activity is insensitive to memory strength is at odds with our own data. In each experiment where the analysis was possible, we found that recognition accuracy increased in tandem with the amount of information recollected (Tables 3 and 4 and Figures 2 and 3). Is it possible, then, that the hippocampus was responding to the strength of item memory rather than to the amount of information recollected? We think this possibility is unlikely for two reasons. First, item memory accuracy differed reliably between the R response category associated with the least amount of recollected information and the K category, but in no case was there a concomitant difference in hippocampal activity. This was so even in the study of Yu, Johnson and Rugg (in press), where the recognition accuracies for R-weak and K-all trials (.92 and .77, respectively) spanned the range argued by Wixted, Mickes, and Squire (2010) to be sufficient to account for prior reports of recollection-related enhancement of hippocampal activity. The second reason for rejecting the possibility that our findings can be attributed to differences in memory strength comes from a subsidiary analysis reported in Yu, Johnson, and Rugg (in press). Two sub-groups of subjects were formed for whom recognition accuracy was equated between the R-high and R-mod response categories, and between the R-mod and R-weak categories. In both sub-groups, hippocampal activity was significantly greater when it was elicited by items from the category associated with the more confident source judgment.

The findings reported above are at variance with those reported in two recent studies: Wais, Squire, and Wixted, (2010), and Smith, Wixted, and Squire (2011). In Wais, Squire, and Wixted (2010), equivalent levels of hippocampal activity were elicited by test items attracting correct and incorrect source memory judgments when the items were equated for memory strength. Comparison of the levels of source accuracy in that study and the study of Yu, Johnson, and Rugg (in press) described above suggest a possible explanation for these divergent findings. In Wais, Squire, and Wixted (2010) mean source accuracy for test items recognized with high confidence was .67, and accuracy for items recognized at the adjacent confidence level was .63. These two item classes were combined for the fMRI analyses, meaning that source accuracy would have fallen somewhere between these two values. Source accuracy levels in the study of Yu, Johnson, and Rugg (in press) were considerably higher (.88 for R-high items). It is possible therefore that the failure of Wais, Squire, and Wixted (2010) to detect effects of source accuracy in the hippocampus reflected the relatively low levels of source recollection in that study. Smith, Wixted, and Squire (2011) combined recognition confidence ratings on a 20-point scale with the R/K procedure. They reported that the hippocampal activity elicited by items endorsed R exceeded the activity elicited by items given a K response when memory strength (operationalized by recognition confidence) was not equated, but that this difference was abolished when the R vs. K contrast was limited to items attracting the highest confidence ratings. Both classes of items did however elicit hippocampal responses that exceeded the responses elicited by missed items. Smith, Wixted, and Squire (2011) interpreted their findings as evidence for the involvement of the hippocampus in the retrieval of strong memories, regardless of whether the memory is supported by recollection or familiarity. An intriguing alternative possibility is that item memory judgments made with very high accuracy and confidence (and subjectively experienced as strong familiarity) are supported by ‘item-specific’ recollection rather than the familiarity signal that supports item memory more generally (J. Wixted, pers. comm. 12/08/2011; see also Wixted and Mickes, 2011). By this account, such judgments might be expected to depend on some of the same neural substrates - perhaps including the hippocampus - as the process of contextual recollection that supports successful source memory. To the extent this account is valid, our proposal that encoding- and retrieval-related hippocampal activity index the amount of contextual information encoded in association with a study item will require qualification.

As already noted, a consistent finding across the four experiments described above was that hippocampal activity did not differ between items endorsed as R or K when the R judgment was associated with the retrieval of relatively little information about the study episode (the R-small and R-weak trials in Figures 2 and 3). Thus, the endorsement of a test item as ‘recollected’ is not necessarily associated with enhanced activity relative to the activity elicited by items judged as familiar only (at least at the sensitivity afforded by conventional fMRI). It is unclear how this null finding should be interpreted. One possibility is that it is a reflection of the ‘process impurity’ of the Remember/Know procedure. By this account, R judgments are made when memory strength (whether based upon recollection, familiarity, or a combination of the two signals) exceeds a criterion level (Donaldson, 1996; Rotello et al., 2005). Thus, on average, there may be little or no difference between items endorsed as R or K in respect to the amount of qualitative information retrieved. The finding that, when equated for memory strength, items endorsed as R are markedly more likely to be associated with accurate source memory judgments than items endorsed as K (Wixted and Mickes, 2010; see also Ingram, Mickes and Wixted, 2012) indicates that this account does not hold for R judgments in general. It remains possible, however, that ‘near criterion’ R judgments reflect a combination of a weak recollection signal little different from that associated with many items endorsed as K (though see below) and strong familiarity. Such judgments would be associated with relatively high memory strength and the retrieval of little qualitative information, the characteristics associated with R-small and R-weak judgments in the four experiments described above.

Alternatively, the failure of the hippocampus to discriminate between R-small/R-weak and K items may merely reflect the inability of the fMRI BOLD signal to detect differences in hippocampal activity associated with recollection of only a small amount of information. By this argument, subjects are capable of distinguishing accurately between trials on which recollection is successful or fails, responding R and K accordingly. However, whereas the retrieval of even a small amount of such information is sufficient to boost recognition accuracy above that for items for which recollection fails, it is insufficient to elicit a detectable hippocampal response (cf. Squire, Wixted, and Clark, 2007). The currently available data do not allow adjudication between this and the foregoing possibility.

As was noted in the Introduction, the question whether recollection should be characterized as a threshold or continuous process is a debated issue. According to one influential model (Yonelinas, 2001), recollection is thresholded: unstudied test items almost never elicit a recollection signal, and studied items either elicit a signal or fail to do so. An alternative model (Mickes, Wais, and Wixted, 2009; Wixted and Mickes, 2010; Ingram, Mickes, and Wixted, 2012) argues that, like the signal supporting familiarity, the recollection signal is continuous: all test items elicit recollection to some degree, and whether an item is endorsed as recollected (R) depends on the setting of a response criterion. At first sight, the behavioral and fMRI data from the four experiments described above appear to support the second of these two positions. The behavioral data clearly indicate that recollection is graded. As demonstrated in the experiments of Vilberg and Rugg (2007; 2009a, b), subjects can reliably report recollecting differing amounts of information across a series of test trials. And in the study of Yu, Johnson, and Rugg (in press), the accuracy and confidence of source memory judgments co-varied, as would be expected if the judgments were supported by a continuously varying memory signal (see also Mickes, Wais, and Wixted, 2009). Moreover, as was described above, retrieval-related hippocampal activity co-varies with the amount of information recollected, a finding strongly indicative of a graded memory signal. These findings are incompatible with an ’all or none’ model of recollection in which it is assumed that recollection either fails or is complete. As has been noted previously, however (e.g., Yonelinas et al., 2010), evidence that recollection is graded does not rule out threshold models in general, since these models are not predicated on the all-or-none assumption. Rather, the key assumption of threshold models is that unstudied test items, along with some proportion of studied items, fail to elicit a discriminating signal. An example of a model of memory retrieval for which this assumption is a good approximation is provided by Norman and O’Reilly (2003; see also Norman, 2010).

Hippocampal activity during continuous recognition

The great majority of studies of retrieval-related hippocampal activity employed separate study and test phases. A few studies however have employed the continuous recognition procedure, when new and old items are interleaved within a single list (Brozinsky, Yonelinas, Kroll, and Ranganath, 2005; Huijbers, Pennartz, and Daselaar, 2010; Johnson, Muftuler, and Rugg, 2008; Suzuki, Johnson and Rugg, 2011a; Suzuki, Johnson, and Rugg, 2011b). In the simplest case items are presented twice, with the requirement to discriminate between first and second presentations (see, for example, Brozinsky, Yonelinas, Kroll, and Ranganath, 2005).

Below, we describe findings from three experiments from our laboratory that employed different variants of the continuous recognition procedure. The experiments allowed investigation of the neural correlates of differences in the memory strength of test items and of individual differences in the ability to retrieve contextual information associated with a repeated test item. The findings converge with those reported above from our experiments employing separate study and test phases: retrieval-related enhancement of hippocampal activity is a neural correlate not of increased memory strength, but of successful contextual retrieval.

In the first two experiments (Johnson, Muftuler, and Rugg, 2008; Suzuki, Johnson and Rugg, 2011a), items were presented a total of four times. The designs of the two studies were very similar, differing principally with respect to their response requirements and stimulus materials. Specifically, whereas in Johnson, Muftuler, and Rugg (2008) the task was simply to discriminate between old and new items, in Suzuki, Johnson and Rugg (2011a), the requirement was to discriminate between test items on the basis of the number of presentations (responding to first and third presentations on one key, and second and fourth presentations on another). The findings from the two studies were very similar. Despite the substantial ‘strengthening’ of memory across the successive presentations (hit rates in Johnson, Muftuler, and Rugg (2008) were .80, .94 and .98 for first, second and third repetitions respectively, against a false alarm rate of .06), no hippocampal voxels could be identified in either experiment where activity increased as function of strength (in contrast to what was observed in cortical regions such as the precuneus). Rather, as is illustrated in Figure 4, hippocampal activity decremented as a function of repetition.

Figure 4.

The effects of item repetition on hippocampal activity in the study of Suzuki, Johnson and Rugg (2011a). Upper: Repetition-sensitive hippocampal regions. Lower: across-subject mean parameter estimates for correctly endorsed items as a function of number of presentations.

We conjectured that the negative relationship between hippocampal activity and memory strength observed in these studies was a consequence of the failure of repeated items to engage recollection. By this argument, the subjects in these studies relied primarily on differences in the familiarity of the test items to select a response. In the absence of recollection, hippocampal activity indexed the relative novelty of the eliciting items, reflecting the extent to which the items engaged processes supporting episodic encoding.

We assessed this proposal by employing a continuous recognition procedure in which accurate performance depended on recollection (Suzuki, Johnson, and Rugg, 2011b). Items comprised pictures surrounded by a colored frame (orange, blue or gray). The task requirement was to respond ‘new’ to first presentations, and ‘old’ to repeated items that were surrounded either by a gray frame, or a frame of the same color as when the item was first presented (‘targets’; i.e. orange-orange or blue-blue). Crucially, repeated items that were surrounded by a differently colored frame (‘non-targets’; orange-blue or blue-orange) required a ‘new’ rather than an ‘old’ response. Therefore, to avoid making a false alarm to non-target items it was necessary to use recollection to ‘oppose’ their familiarity (Jacoby, 1991). Hence, the contrast between the activity elicited by non-targets attracting correct versus incorrect responses should identify neural correlates of recollection. Subjects’ ability to perform the task varied widely, with recollection performance (operationalized by the difference between the probabilities of correct responses to targets and incorrect responses to non-targets; Jacoby 1991) varying between .13 and .81. As is illustrated in Figure 5, we identified a right hippocampal cluster where recollection-related activity co-varied across subjects with the behavioral estimate of recollection. Thus, consistent with the findings reported from studies employing separate study and test phases, successful recollection in continuous recognition is associated with enhanced hippocampal activity. Nonetheless, the different classes of studied item all elicited lower levels of hippocampal activity than did unstudied items, in keeping with the findings of the two preceding studies.

Figure 5.

Hippocampal recollection effect in the study of Suzuki, Johnson, and Rugg (2011b). Upper: Right hippocampal region where recollection-related activity co-varied across-subjects with recollection performance. Lower: scatter plot illustrating the relationship between the neural and behavioral indices of recollection. Each dot represents a separate subject.

Clearly, the findings described in this section are incompatible with the idea that retrieval-related hippocampal activity is always a positive function of memory strength. At least during continuous recognition, increased memory strength can be associated with decreased hippocampal activity. Similar results were reported in one other continuous recognition study (Brozinsky, Yonelinas, Kroll, and Ranganath, 2005), and an analogous finding has been reported in several studies that employed separate study and test phases. In these studies, items attracting ‘new’ judgments (correct rejections or misses) elicited larger hippocampal responses than did correctly detected studied items (e.g., Rugg, Henson, and Robb, 2003; Daselaar, Fleck, and Cabeza, 2006; Vilberg and Rugg, 2009c; Yu, Johnson and Rugg, in press; see also Figure 2C). These findings have usually been interpreted as evidence for the novelty-induced engagement of hippocampally-mediated encoding processes (e.g., Duzel et al., 2003; Nyberg, 2005), consistent with the long-standing proposal that there is a bias favoring the encoding of relatively novel over relatively familiar items (Tulving and Kroll, 1995). Direct evidence in support of this interpretation of hippocampal ‘novelty effects’ was provided by Stark and Okado (2003), who reported that unstudied recognition memory test items that went on to be retrieved on a later memory test elicited higher hippocampal activity than forgotten items.

The findings from our continuous recognition studies indicate that strengthening of the familiarity signal is associated with a qualitatively different hippocampal response than the response that co-varies with the ‘strength’ of contextual recollection. The dissociation in the direction of the two responses (a negative correlation with familiarity, but a positive one with recollection) suggests that the construct of memory strength - as operationalized by accuracy of item memory - is of little utility in understanding the functional significance of retrieval-related hippocampal BOLD activity.

As was just noted, findings of greater hippocampal activity for unstudied than for studied items are usually interpreted as evidence for the engagement of encoding processes by relatively novel (unfamiliar) items. It is important to note that such an account does not rule out the possibility that the differential hippocampal responses elicited by familiar and unfamiliar items could, in principle, provide a signal capable of supporting recognition memory judgments (though see Johnson, Mufutler, and Rugg. 2008 for reasons why this is unlikely). Were this to be the case, it would call into question the proposal (e.g., Brown and Aggleton, 2001; Eichenbaum, Yonelinas, and Ranganath, 2007) that the hippocampus does not support familiarity-based recognition. It would not, however, license the conclusion that the hippocampus is necessary for this form of recognition.

A final noteworthy point is that, at least at the spatial scale afforded by fMRI, familiarity-driven decrements and recollection-related enhancements in hippocampal activity can co-exist in the same region. For example, the hippocampal cluster illustrated in Figure 5, where activity covaried with a behavioral index of recollection, also demonstrated robustly greater activity for new items than for studied items. Analogously, Vilberg and Rugg (2009c) reported overlapping recollection and novelty effects in the anterior hippocampus in a study that employed separate study and test phases (see also Figure 2C). These findings raise the question of whether the overlap between hippocampal recollection and novelty (or familiarity) effects exists at the level of individual neurons or neuronal circuits, and hence whether the same neuronal populations support both the encoding and retrieval of episodic information.

Concluding comments

We have described data from a variety of studies relevant to the question of whether the construct of memory strength is sufficient to account for the enhanced encoding- and retrieval-related hippocampal activity that has typically been attributed to the engagement of processes selectively supporting recollection. We have argued that the construct of strength is not sufficient; rather, hippocampal activity co-varies with the amount of contextual or, more generally, episodic information that is encoded or retrieved, and not with the strength of an undifferentiated memory signal. This does not mean however that the construct of recollection provides the best way to understand hippocampal function. Indeed, our own findings consistently indicate that the phenomenal sense of recollection, as indexed by a Remember judgment, is not necessarily accompanied by a hippocampal response greater than the response elicited by items judged as familiar only (Know judgment). The role of the hippocampus in episodic memory will likely be more completely understood in the context of a computationally explicit model that specifies when and how this structure contributes to recollection (similar arguments have been made previously, e.g., Davachi, 2006; Diana, Yonelinas, and Ranganath, 2007; Norman, 2010; Wixted and Squire, 2011). Among the findings that any such model should accommodate is the sensitivity of hippocampal fMRI BOLD activity to the amount of information that is encoded or retrieved about a study episode.

Highlights.

1. Item memory strength is not a major determinant of hippocampal encoding- or retrieval-related activity

2. Hippocampal activity co-varies positively with the amount of contextual information encoded or retrieved

3. Hippocampal activity co-varies negatively with familiarity strength