Repetition effects elicited by objects and their contexts: An fMRI study

Abstract

Event‐related fMRI responses were recorded during a recognition memory test for previously studied visual objects. Some studied objects were superimposed on the same context (landscape scenes) as at study, some were superimposed on a different studied context, and some were paired with new contexts. Unstudied objects were paired with either a studied or a new context. Relative to all other stimulus classes, test stimuli where both components were unstudied elicited enhanced responses in lateral and ventral extrastriate visual cortex. This effect, which is analogous to a previously described electrophysiological result obtained with the same experimental procedure, had the same magnitude regardless of whether a test item was composed of one or two studied components, or whether a single studied component was task relevant or task irrelevant. The findings point to the existence of repetition‐sensitive neural mechanisms that operate in a non‐linear manner. Hum. Brain Mapping 19:145–154, 2003. © 2003 Wiley‐Liss, Inc.

INTRODUCTION

Numerous studies have investigated the neural correlates of the repetition of stimuli such as words and pictures in both direct (explicit) and indirect (implicit) memory tests. Most of these studies have employed event‐related brain potentials (ERPs) to detect these correlates [for reviews see Friedman and Johnson, 2000; Rugg and Allan, 2000]. More recently, electrophysiological studies have been joined by others employing event‐related functional magnetic resonance imaging (fMRI) [for review, see Rugg and Henson, 2002]. Together, the ERP and fMRI studies have focused on the nature and the functional significance of the differential neural activity elicited by experimentally familiar stimuli (for example, items first presented during a pre‐test study phase) as opposed to stimuli that are experimentally novel.

With few exceptions, these studies have investigated the neural correlates of repetition while holding constant the context in which the experimental items were experienced on successive presentations. In a departure from this pattern, Tsivilis et al. [2001] employed ERPs to investigate the brain activity elicited during a recognition memory task in which background context was systematically manipulated. Subjects first studied a series of pictures of objects, each of which was presented against one of a set of landscape scenes that served as the contexts. At test, the requirement was merely to discriminate between studied (old) and unstudied (new) pictures. Some of the old pictures were paired with the same background as at study (Same pairs), others were paired with a studied background different from that at study (Rearranged), and some old pictures were paired with contexts that had not been presented at study (Old–New). New pictures were paired either with studied (New–Old) or unstudied (New–New) contexts.

The findings from this study were analyzed primarily in relation to contextual influences on putative ERP correlates of recognition memory. There was, however, one unanticipated finding, and it is this that provides the focus of the present study. Relative to the ERPs elicited by the New–New items, ERPs to each of the other classes of stimuli elicited an early (onset ∼ 100 msec) positive deflection, the amplitude of which was maximal at frontopolar scalp sites. This finding was striking for two reasons: first, ERP repetition effects are rarely found with such an early onset. Typically, such effects do not emerge until some 200 msec or more post‐stimulus [Rugg, 1995]. Second, the effect was ungraded; it was as large when elicited by stimuli containing a single, task‐irrelevant, old component (New–Old items), as it was for stimuli where both components were old (Same and Rearranged items). Thus, the effect appears to represent an early, relatively automatic discrimination between stimuli that are experimentally novel and stimuli containing at least one familiar component. Whereas it is not possible to localize the intracerebral sources of a scalp electromagnetic field solely from knowledge of its scalp distribution, a distribution with a focal frontopolar maximum would most plausibly reflect generators localized to anterior regions of the brain. On the basis of its early onset latency and anterior scalp distribution, Tsivilis et al. [2001] conjectured that the effect was a reflection of differential neural activity in repetition‐sensitive neurons of the kind identified in the anterior temporal cortex of the monkey. From earlier than 100 msec post‐stimulus, and in a range of task settings, these neurons demonstrate reduced firing rates to repeated relative to experimentally novel stimuli [for review, see Brown and Xiang, 1998]. Tsivilis et al. [2001] suggested that the frontopolar ERP repetition effect might either be a direct reflection of the activity of this neuronal population, or that it might reflect activity in one or more of the prefrontal areas with which the anterior temporal cortex is connected [Rempel‐Clower and Barbas, 2000].

Our primary goal was to determine whether there is an fMRI analogue of the frontopolar ERP repetition effect. We took advantage of the fact that, unique among the effects described by Tsivilis et al. [2001], the frontopolar effect was characterized by a difference in the neural activity elicited by New–New items relative to all other item types. Employing an experimental procedure that was essentially identical to that of Tsivilis et al. [2001], we adopted an analytic approach that identified where in the brain there were common differences in the activity elicited by New–New items versus each of the other classes of stimuli. Thus, we were able to identify the regions that demonstrated this characteristic pattern of effects, assess their nature (e.g., whether they result from a relative enhancement or decrement in New–New activity), and determine whether the identified regions include those suggested by Tsivilis et al. [2001] as possible contributors to the analogous ERP effect. Findings from contrasts directed toward the effects of context on the fMRI correlates of successful recognition memory [Rugg and Henson, 2002] will be reported separately.

SUBJECTS AND METHODS

Subjects

Twelve volunteers (nine women; mean age 21 years, range 19–25) participated in the experiment in return for remuneration. All reported themselves to be right‐handed, native English speaking, and with no history of neurological or psychiatric problems. Informed consent was obtained from each volunteer prior to the experiment, the procedures of which were approved by the National Hospital for Neurology and Neurosurgery and Institute of Neurology joint Medical Ethics Committee.

Tasks

The experiment consisted of a study phase followed by a recognition memory test, both performed in the scanner. At study, volunteers made judgments about objects superimposed on landscape scenes (see Fig. 1). They were asked to mentally place each object in a specific location in the landscape, creating an internal narrative to justify the placement. The study phase consisted of a sequence of 120 critical stimuli, with a short rest halfway through.

Figure 1.

Stimuli and conditions used in the experiment. Each stimulus consisted of a digitized picture, composed of an object superimposed on a landscape. During a study phase, subjects had to mentally place the object in the landscape. During a subsequent test phase, subjects had to decide whether or not they had seen the object before during the experiment. Some of the old objects were paired with the same background as at study (Same pairs), others were paired with a studied background different from that at study (Rearranged pairs), and some old objects were paired with contexts that had not been presented at study (Old–New pairs). New objects were paired either with studied (New–Old pairs) or unstudied (New–New pairs) contexts.

About 3 min after completion of the study phase, volunteers performed a yes/no recognition memory test. All 120 studied objects were re‐presented, along with 80 new objects. Each object was superimposed on a landscape. Old objects were superimposed on the same landscape as that with which they appeared at study (Same pairs), a landscape that had appeared at study but not with that object (Rearranged pairs), or a new landscape (Old–New pairs). New objects were superimposed on either a previously seen landscape (New–Old pairs) or a new landscape (New–New pairs). For each test stimulus, volunteers had to decide whether or not they had seen the object before during the experiment. They were told that while some of the landscapes might also appear familiar, this was irrelevant to the task. One of two buttons had to be pressed according to the old/new object decision. The responding hand was counterbalanced across volunteers. Both speed and accuracy were stressed, and volunteers were asked to avoid guessing “old.”

Before entering the scanner, volunteers received short practice study and test lists so as to familiarize them with the task. To verify their understanding of the study task, volunteers gave an overt reason for their choice of placement of the object in the landscape during the practice. Before the experiment proper, volunteers viewed each of the landscapes that would subsequently be encountered in the study phase for 2 sec. This was done to familiarize volunteers with the landscapes prior to their first appearance in the study phase, and thereby to minimize differences between first and successive presentations of each landscape.

All stimuli were displayed on a black background. Landscapes and objects subtended approximate horizontal and vertical visual angles of 11.8 and 11.8, and 5.9 and 5.9 degrees, respectively. At study, stimulus duration was 10 sec, and stimuli were presented consecutively. Objects were displayed in a randomly chosen quadrant of their background landscape. At test, stimulus duration averaged 550 msec, with a random variation (introduced by a programming error) between 500 and 600 msec. The interval between successive stimuli was 4.1, 8.2, or 12.3 sec (average interstimulus interval 4.9 sec). In the test stimuli, objects always appeared in the center of the display, so as to minimize eye movements. A white frame (corresponding to the perimeters of the landscapes) was continuously present on the screen and an “x” at the center of the frame served as a fixation point.

Stimulus lists

Two sets of corresponding study and test lists were created from a pool of digitised color photographs of 226 objects and 118 landscapes. Each stimulus was composed of one of the objects, superimposed on one of the landscapes. Assignment of objects to landscapes was random with the restriction that no obvious semantic or associative relationship existed between them (e.g., a boat superimposed on a lake). Objects were outlined in yellow to facilitate figure–ground separation (see Fig. 1). The objects were randomly selected from several semantic categories, including tools, furniture, clothing, household appliances, toys, and vehicles. Only objects that were easily identifiable (as determined from pilot studies) were included. The landscapes consisted of views of mountains, lakes, fields, waterfalls, beaches, forests, and so on; none depicted buildings, people, or animals. Landscapes were easily discriminable from each other.

The two stimulus sets differed in the allocation of objects to landscapes, and object‐landscape combinations to experimental conditions, and were counterbalanced across subjects. Each set was composed of 200 critical stimuli. Because of an error in the compilation of the lists, 174 of the objects and 82 of the landscapes were used in both sets, while the remaining 52 objects and 36 landscapes were unique to one or other set. Additional objects and landscapes were used to create short practice study and test lists.

Each study list consisted of a random sequence of 120 of the critical stimuli, composed of 120 unique objects superimposed on 20 different landscapes. Thus, each landscape was presented six times during study. A filler item was added to the beginning and middle of the list. Each test list consisted of a random sequence of 200 critical stimuli, interspersed with 40 fixation‐only trials to allow the estimation of the fMRI response in each experimental condition. One hundred and twenty of the critical stimuli contained studied objects: 40 were paired with the same studied landscape (Same pair), 40 with a different studied landscape (Rearranged pair), and 40 with a new landscape (Old–New pair). Eighty test stimuli contained new objects: 40 paired with a studied landscape (New–Old pairs), and 40 with a new landscape (New–New pairs). Each studied landscape appeared six times, twice each in Same, Rearranged, and New–Old pairs. Two filler items were added to the beginning of the test list. A different presentation order was employed for each subject.

MRI scanning

Scanning took place during the test phase only. A 2T Siemens VISION system (Erlangen, Germany) was used to acquire both T1‐weighted anatomical volume images (1 × 1 × 1.5 mm voxels, MPRAGE sequence) and T2*‐weighted echoplanar (EPI) images (64 × 64, 3 × 3 mm pixels, TE = 40 msec) with blood oxygenation level dependent (BOLD) contrast. Each EPI volume comprised 36 2‐mm thick axial slices separated by 1.5 mm, positioned to cover all of the brain except the cerebellum. In a single session, 368 volumes were acquired continuously with an effective repetition time (TR) of 2.74 sec/volume. The first five volumes were discarded to allow for T1 equilibration effects.

MRI analysis

The data were analyzed using Statistical Parametric Mapping [Friston et al., 1995], version SPM99 (Wellcome Department of Imaging Neuroscience, London, UK). For each subject, volumes were realigned to the first volume and resliced using a sinc interpolation in space to correct for movement artifacts. To correct for differences in slice acquisition time within a volume, the signal measured in each slice was shifted relative to the acquisition of the middle slice using a sinc interpolation in time. Each volume was normalized to a standard EPI template volume (based on the MNI reference brain) [Cocosco et al., 1997] of 3 × 3 × 3 mm voxels in the space of Talairach and Tournoux (1988) using nonlinear basis functions. Finally, the EPI volumes were smoothed with an 8‐mm full‐width half‐maximum isotropic Gaussian kernel to accommodate residual anatomical differences across volunteers.

The haemodynamic response to the onset of each event type of interest was modeled with two basis functions: a canonical hemodynamic response function (HRF) [Friston et al., 1998], and a delayed HRF, shifted to onset 2.74 sec (i.e., one TR) later than the canonical HRF. The use of both an “early” and a “late” response function was based on suggestions from previous reports [Henson et al., 2000; Otten et al., 2001] that the time of maximal activation is later for some brain regions than the sensory regions on which the canonical HRF is based. The early and late response functions, when convolved with a sequence of delta functions representing the onset of each event, comprised the covariates in a general linear model, together with a constant term. The covariates for the late HRF were orthogonalized with respect to those for the early HRF so as to give priority to the early covariate [Andrade et al., 1999]. Thus, loadings on the orthogonalized late covariate account for residual variance in the data not explained by the early covariate. The data were high‐pass filtered to a maximum of 1/120 Hz, and both model and data were smoothed temporally with a 4‐sec full‐width half‐maximum Gaussian kernel. Parameter estimates for each covariate were calculated from the least squares fit of the model to the data.

Planned contrasts (see Results ) were employed to test parameter estimates for both early and late covariate. The results from the late covariate did not add meaningfully to the aims of the present study, and are not reported (results can be obtained from the authors on request). The linear combination of parameter estimates for each contrast was stored as a separate image for each volunteer. These contrast images were entered into one‐sample t‐tests to permit inferences about condition effects across volunteers (i.e., a random effects analysis). The images were transformed into the unit normal Z‐distribution to create statistical parametric maps (SPMs). The maxima of suprathreshold regions were localized by rendering them onto both the mean normalized structural images and the MNI reference brain [Cocosco et al., 1997]. They were labeled using the stereotactic system and nomenclature of Talairach and Tournoux [1988]. Stereotactic coordinates correspond to the standard MNI brain [Cocosco et al., 1997]. Only activations involving contiguous clusters of at least 5 voxels are reported.

RESULTS

Behavioral Performance

Accuracy and response time (RT) data are shown in Table I. ANOVAs revealed that neither index differed for either the three classes of hit, or the two classes of correct rejection.

Table I.

Recognition memory performance

	Same	Rearranged	Old–New	New–Old	New–New
Accuracy (%)	81.9 (7.6)	80.4 (6.6)	80.8 (9.5)	95.6 (5.2)	94.8 (6.0)
RT (msec)	1,086 (120)	1,114 (114)	1,093 (82)	1,141 (68)	1,157 (105)

Values are across‐subject means (SD).

fMRI Data

The fMRI analyses were restricted to trials associated with correct recognition judgments. Voxels exhibiting a common difference for New–New versus all other conditions were identified by an inclusive masking procedure. First, pairwise directional contrasts were performed between the New–New condition and each of the other conditions. These contrasts, which were thresholded at P < 0.0005 (giving a two‐tailed threshold of P < 0.001), identified voxels where activity in the New–New condition was either greater or lower than that in each of the others. Second, separately for each test direction, the findings for each contrast were masked to identify overlapping voxels, that is, voxels exhibiting a significant difference in activity in all four contrasts. No voxels were identified where activity in the New–New condition was consistently lower than that in the remaining conditions. The reverse contrast, however, revealed a large population of voxels in lateral occipital and inferior occipito‐temporal regions where activity was consistently greater in the New–New condition (see Fig. 2).

Figure 2.

A–D: Maximum intensity projections illustrating regions that showed signal increases for the New–New condition relative to the Same (A), Rearranged (B), Old–New (C), and New–Old (D) conditions. E: Voxels exhibiting a common difference for the New–New versus all other conditions, identified by an inclusive masking procedure (see text). Activations are rendered onto the cortical surface of the Montreal Neurological Institute normalized canonical brain (Cocosco et al., 1997). All figures thresholded at P < 0.001.

To investigate the relative magnitude of the differences between the New–New and the other conditions, we conducted two additional analyses. In the first analysis, estimates were obtained of these differences from representative voxels in left and right lateral occipital and fusiform regions (see Fig. 3). The voxels were chosen on the criterion that they exhibited local maxima for the contrast between the New–New condition and the weighted average of the remaining four conditions, obviating the possibility of bias in favor of any one condition. As can be seen from Figure 3, the responses elicited in the conditions other than New–New are of similar magnitude, with no sign of a graded response in respect of the conditions in which both stimulus components were old (Same and Rearranged), as opposed to those with only one old component (Old–New and New–Old). This impression was confirmed by ANOVA of the parameter estimates of the responses elicited in each condition. In none of the voxels was there evidence that these estimates varied according to condition (min P > 0.1).

Figure 3.

Parameter estimates on the early covariate from representative voxels in left and right lateral occipital (left) and left and right fusiform (right) regions for each of the five experimental conditions. Activations are rendered onto the normalized T1 anatomical images averaged across subjects. Coordinates are given below each bar graph. Error bars represent SEM.

The second analysis addressed whether any of the voxels identified by the inclusive masking procedure described above differed when activity was elicited by stimuli containing two, as opposed to one, old component. The analysis took the form of a bidirectional contrast between the average of the Same and Rearranged conditions, and the average of the Old–New and New–Old conditions, masked by the outcome of the inclusive masking procedure (see Fig. 2). Even at the liberal threshold of P < 0.005, no voxels were identified where activity differed according to the number of old stimulus components. Thus, the results of this analysis agree with those performed on selected voxels (Fig. 3), suggesting that the effect illustrated in Figure 2 reflects an ungraded, “all or none” response to familiarity.

DISCUSSION

Robust differences were observed between the event‐related BOLD responses elicited by experimental stimuli containing two novel components (New–New items) and the responses to each of the other stimulus classes. These differences did not vary according to whether one or both stimulus components were familiar; relative to New–New items the presence of a single familiar component, whether task‐relevant or task‐irrelevant, led to a response reduction as large as that seen when both components were familiar (see Fig. 3). Thus, we were able to identify an fMRI analogue of the frontopolar ERP repetition effect [Tsivilis et al., 2001].

Whereas the present findings conform qualitatively to the pattern observed for the previously reported ERP effect, the localization of the fMRI effects is much different from what was expected on the basis of the ERP data. As noted in the Introduction, the ERP effects took the form of early‐onsetting positivities (relative to the New–New condition) with maxima over the frontopolar scalp. This led to the conjecture that the frontopolar repetition effect reflects differential neural activity either in anterior temporal cortex or in the prefrontal cortical areas that receive projections from this region. In striking contrast to this conjecture, the present findings localized the fMRI analogue of the ERP effect to lateral and ventral extrastriate visual cortex, with no sign of differential activity in either the anterior temporal or prefrontal cortex (this latter conclusion remained valid even when the statistical threshold for the analysis was lowered by an order of magnitude, i.e., to P < 0.005). The reasons for the apparent discordance between the ERP findings and the present data are uncertain. On the one hand, there may not be a discordance. Under this scenario, the ERP effect is the scalp reflection of early‐onsetting, differential extrastriate activity revealed by fMRI, and the seemingly counter‐intuitive scalp distribution of the effect is merely a consequence of the net orientation of the active tissue. On the other hand, the two effects may reflect the activity of distinct neural systems. For example, the extrastriate activity described here may be a “re‐entrant” effect, dependent upon more anteriorly located repetition‐sensitive mechanisms [see Dale et al., 2000]. This scenario is by no means implausible since there are a number of reasons why electrophysiological and haemodynamic measures of brain activity might dissociate [Rugg, 1999], any of which could apply here. It is not possible to decide between these alternatives on the basis of current evidence. This may, however, be possible through the application of source modeling methods applied to ERP data sampled with sufficiently high spatial resolution.

What could be the functional significance of the effects observed in the present study? For two reasons, it seems very unlikely that the effects merely reflect the differential consequences of recognition memory judgments made to the different classes of stimuli. First, they were observed regardless of whether items were associated with positive (Same, Rearranged, Old–New) or negative (New–Old) recognition decisions. Second, we have observed similar effects in a follow‐up study [Tsivilis et al., unpublished observations] in which semantic classification rather than recognition memory judgments were made for each object. Thus, the effects appear to reflect processing differences contingent on whether or not the stimulus contains a familiar component, regardless of task or response requirements.

The differences common to the New–New items and the remaining conditions were due exclusively to larger New–New responses. Greater responses for experimentally novel relative to experimentally familiar items have been reported in numerous previous studies and, when found in the context of indirect memory tasks, have often been interpreted as a neural correlate of priming, the facilitation of stimulus processing engendered by repetition, often in the absence of conscious memory [Schacter and Buckner, 1998]. According to this view, the reduced activity elicited by repeated stimuli reflects the more limited processing required to identify such stimuli relative to stimuli that have not been encountered recently. In keeping with this account, the areas exhibiting response reductions in the present experiment are all ones expected to play a role in the initial identification of experimental stimuli. Indeed, the areas are in good correspondence with the extrastriate regions identified in previous event‐related fMRI studies of object priming [Buckner et al., 1998; Koutstaal et al., 2001].

If the present findings do reflect priming‐related response reductions, they have two interesting implications for the conditions under which such reductions can be found. First, they suggest a striking non‐linearity in the underlying mechanism, in that the presentation of a single familiar stimulus component was sufficient to “saturate” the effects. An interesting question for the future is whether this occurred because the two components engaged the same extrastriate regions, and thus whether a different pattern of effects would emerge if components arguably dependent upon different regions, faces and words, for example, were employed. Second, the present findings indicate that the mechanism responsible for response reduction is relatively insensitive to attentional factors, in as much as the task‐irrelevant background contexts were no less effective in eliciting response reductions than the objects superimposed upon them. It should, however, be noted that this second conclusion is subject to a caveat arising out of the fact that, unlike the objects, backgrounds were presented on multiple occasions at both study and test. It remains to be determined, therefore, whether the conclusion extends to the situation where, like the relevant stimulus component, the irrelevant component is repeated once only.

An alternative account of the present findings is that the enhanced activity elicited by the New–New items represents a “novelty response.” By this argument, these stimuli were rendered particularly salient by the combination of their relative infrequency of occurrence and possession of two novel components. Thus, they acted somewhat like “oddball” stimuli, engaging attentional and processing resources beyond those allocated to the remaining stimulus classes. In favor of this account are findings demonstrating enhanced activity in posterior fusiform cortex in response to visually‐presented oddball words [Strange et al., 2000]. Also supportive is the finding that selective attention to visual objects is associated with enhanced activity in the same extrastriate regions that were identified in the current study (Rees et al., 1999). Against this account, however, is the failure to find enhanced New–New responses in regions, notably the posterior medial temporal lobe and prefrontal cortex, identified as novelty‐sensitive in previous studies [for review, see Habib, 2001]. As with the priming account outlined above, the novelty account implies an “all or none” mechanism; a single novel stimulus component, even when the focus of attention (i.e., New–Old items), was insufficient to elicit any sign of a “novelty response.”

The present data do not allow a clear adjudication between these competing accounts of the enhanced activation elicited by the New–New items [see Habib, 2001]. Findings somewhat analogous to those described here were reported by Rombouts et al. [2001]. Both in an encoding task and during a test of recognition memory, these authors contrasted responses to visual scenes that had been repeatedly presented during the experiment with responses elicited by scenes being viewed for the first time. In both cases, novel scenes elicited greater responses in extrastriate visual regions overlapping with those reported here, as well as in a wide variety of other regions including, in the recognition task, the medial temporal lobe. It is possible that the greater extrastriate activity observed by Rombouts et al. [2001] for novel scenes is attributable, at least partially, to the mechanisms also responsible for the present findings. Unfortunately, their results can easily be accommodated by both of the competing accounts described above, and thus offer no guide as to which is the more valid.

A final consideration concerns the relevance of the present findings to the understanding of the neural correlates of object recognition. The bilateral occipital and fusiform regions showing greater responses in the New–New condition encompass what has become known as the Lateral Occipital Complex (LOC) [Malach et al., 1995]. It has been suggested that the LOC is selectively involved in the ability to recognize objects like those employed in the present experiment [for review see Grill‐Spector et al., 2001]. Reminiscent of reports of “adaptation effects” in the LOC [Grill‐Spector and Malach, 2001], activity in this region was reduced, relative to the New–New condition, for repeated objects. However, the presentation of novel objects in a familiar context also resulted in reduced LOC activity relative to the New–New condition. Moreover, there was no detectable difference in LOC activity for the contrast between repeated and novel objects when their contexts were familiar. These findings suggest that the object recognition processes supported by the LOC are context‐sensitive, even when the contexts are stimuli (scenes) that do not themselves elicit strong activity in this region [Epstein and Kanwisher, 1998].

In summary, as predicted on the basis of previous electrophysiological findings [Tsivilis et al., 2001], we have demonstrated a pattern of brain activity that distinguishes between visual stimuli comprising two novel stimulus components, only one of which is task‐relevant, and stimuli where either one or both components are familiar. Belying the electrophysiological findings, this activity was localized to extrastriate visual cortex, including regions previously implicated in visual object recognition. Whether these findings reflect differences in the neural resources necessary to identify novel as opposed to familiar stimulus components, or whether instead they reflect a response to stimulus novelty, remains to be established. In either case, their ungraded nature points to the existence of repetition‐sensitive neural mechanisms that operate in a non‐additive manner.