Retrieving Object Color: The Influence of Color Congruity and Test Format

Abstract

Incongruous relative to congruous episodes engender better memory, but it is unclear whether recollection or familiarity is responsible. Hence, objects were encoded in either natural (yellow banana) or unnatural (blue banana) outline colors. ERPs were recorded while memory was assessed by item (IT) and source (ST) tasks. During IT, unnatural- relative to natural-color objects produced better memory and more positive parietal activity (500-600 ms) indicative of recollection. Surprisingly, the converse occurred in ST. Because the encoding task required a natural/unnatural decision, an unnatural-color object would have required activation of its natural counterpart to make an informed decision. Thus, source confusion during ST relative to IT would have led to a recollection disadvantage for unnatural-color objects.

INTRODUCTION

Incongruous (“bizarre”) relative to congruous episodes appear to result in a memory advantage [1,2,3], as one might conclude from everyday experience. In a laboratory analog of this phenomenon, Michelon and colleagues [4] asked participants to encode congruous (e.g., teapot) and incongruous (e.g., front of a bus fused with the head of a deer) pictures of common objects while they were scanned in the fMRI environment (Experiment 1). The hemodynamic data indicated a broad network that was activated to a greater extent by incongruous relative to congruous pictures. The areas activated included regions of the inferior frontal gyrus that have been implicated previously in semantic elaborative encoding activities that enhance subsequent memory performance [5,6,7]. In Experiment 2, the objects were encoded and then subsequently tested via recognition memory. In line with prediction, memory sensitivity was greater for incongruous compared to congruous pictures. Semantic elaboration typically engenders contextually-rich memory traces which leads to better recognition memory compared to items that are not so elaborated [8]. On the basis of their fMRI findings, Michelon et al. [4] concluded that incongruous objects were remembered better because they were elaborated to a greater extent than congruous objects.

However, it is unclear from the data provided by Michelon et al. [4] which mnemonic processes were responsible for the memory advantage of incongruous over congruous objects. Dual-process theories of recognition memory differentiate between familiarity- and recollection-based processes (e.g., [9]). Familiarity involves simply knowing that an event has been experienced before unaccompanied by item-specific information, whereas recollection involves the retrieval of contextual detail that accompanied the initial experience (for example, the object plus the color in which it was painted). Because semantic elaboration at encoding should produce strong, contextually-rich, memory traces, an increase in recollection- relative to familiarity-based processes would be expected to underlie the recognition advantage for incongruous objects.

In an attempt to differentiate behaviorally between familiarity- and recollection-based processes, memory-performance for congruous (natural color, e.g., yellow banana) and incongruous (unnatural color, e.g., blue banana) study items (see also [10]) was assessed via item (IT; old/new recognition) and source-memory (ST) tasks using the processing-dissociation paradigm [11]. Performance on the IT can be supported by both familiarity and recollection because participants are instructed to respond ‘old’ to all studied items. By contrast, performance on the ST requires the recollection of item-specific features from the study-phase as participants are asked to make a selective ‘old’ response to a specific class of items, e.g., those that were studied with natural colors. Those studied with, e.g., unnatural colors and new items receive the same response. Performance on the ST is typically greater than chance only if the two classes of old objects can be distinguished via recollection. Hence, we predicted better recognition performance for unnatural- compared to natural-color objects in both IT and ST, which would reflect the greater use of recollection-based processes for unnatural relative to natural items.

For additional evidence to support the hypothesized recollection advantage of unnatural- over natural-color objects, we recorded event-related potentials (ERPs). A long research tradition indicates that when the ERPs elicited by correctly recognized old items are compared to correctly rejected new items, several temporally and topographically distinct, positive-going, old/new or episodic memory (EM) effects result [12]. Two of these, the medial prefrontal (∼300-500 ms) and parietal (∼500-700 ms) EM effects have been associated with, respectively, familiarity- (but see [13]) and recollection-based memory decisions [14]. Thus, it was expected that the predicted performance difference favoring unnatural- over natural-color old objects would be accompanied by a larger parietal EM effect to unnatural- compared to natural-color objects which would, therefore, be indicative of increased recollection-based processes for unnatural items.

METHODS

Subjects

Ten women and 6 men (M = 24 ± 2.8; normal color vision) were recruited. All were native English speakers, in good physical and mental health, and gave informed consent (NY Psychiatric Institute's IRB).

Stimuli

There were 312 unambiguous common-object line drawings divided into 6 lists of 52 items, with lists equated on all relevant variables [15,16,17]. Each object was created in black outline and separate sets were created with identical-thickness, natural- or unnatural-outline colors. These assignments had been validated previously in a pilot study of 7 participants, who viewed each object and judged whether it was outlined in its natural or unnatural color. Only objects that were judged by all raters as either natural or unnatural were retained.

Task Design and Procedure

There were 6 sessions, each with one study and two test blocks (item, source). In each, one of the six lists was used, with the order of list presentation randomized across phases separately for each participant. Of the 52 pictures in a list, 32 were randomly assigned to the study block, while the remaining 20 were assigned as foils to the test block. In the study block, 36 pictures were viewed, half in natural and half in unnatural colors, including 4 fillers, 2 of which were presented at the beginning and end of the block to avoid primacy and recency effects (these were not tested). At encoding, each object was judged as to whether it had been presented in its natural or unnatural color via speeded and accurate, choice RT responses; subjects were told that they would be tested on their memory for the objects and their outline colors. In the item test block, 26 pictures outlined in black were viewed (14 new and 12 old: 6 natural and 6 unnatural); subjects pressed one button if the item was old and the other if it was new. In the source test block, 26 pictures outlined in black were viewed (6 new, 20 old: 10 natural and 10 unnatural). In half of the test blocks, the natural outline color was the target and, in the other half, the unnatural color was the target. Subjects pressed one button if they thought the picture had been studied in the target color, and the other button if they thought the picture had been studied in the nontarget color, or if it was new (for complete details see [18]).

During study, item and source blocks, each object was presented centrally for 500 ms, followed by a 1500-ms fixation period, for a total ISI of 2000 ms. To ensure that subjects knew which button to press, cues were presented below and to the left and right of the objects -- a yellow school bus indicated the “natural” choice; a green school bus indicated the “unnatural” choice. During item-test blocks, “OLD” and “NEW” cues appeared below and to the right and left of the objects. During source blocks, the cues were “OTHER” (for nontarget old and new) and either a yellow school bus when a target had been studied in its natural color, or a green school bus when it had been studied in an unnatural color. The hands assigned during study to “natural” and “unnatural,” during item blocks to “old” and “new,” and source blocks to “target” and “other” were counterbalanced across subjects. To avoid order effects, in half the blocks, the IT preceded the ST and, in the other half, ST preceded IT. Subjects were not informed prior to the study block which test block would be administered first, or whether the target would be natural or unnatural.

EEG Recording

EEG was recorded (sintered Ag/AgCl electrodes) using an elastic cap from 62 scalp sites including left and right mastoids, referred to nosetip [19]. Vertical EOG (supraorbital and infraorbital ridges of the left eye), and horizontal EOG (outer canthi of both eyes) were recorded bipolarly. EEG and EOG (DC; 100 Hz high-frequency cutoff; 500 Hz digitization) were recorded continuously with Synamp amplifiers. Trials were re-referred to averaged mastoids, epoched off-line with 100 ms pre- and 1900 ms post- stimulus periods, and corrected for eye movement artifact [20].

RESULTS

Behavioral Data

Study Phase

During the encoding phase, subjects' decisions were consistent with the ratings that had been collected from the independent raters, 86 percent (±1.3) and 87 percent (±1.7) were judged, respectively, as natural and unnatural object-color pairings (P >0.10). RTs to natural (M = 903 ms ±41) and unnatural (M = 910 ±41) object-color pairings did not differ reliably (P > 0.10), suggesting that they were processed similarly.

Test Phase

To assess whether retrieval performance for natural and unnatural object-color pairings at study was differentially affected by test format, a Test Type (item, source) × Studied Color (natural, unnatural) ANOVA was performed on a measure of memory sensitivity, Pr (Pr = Hits – FAs; [21]) and RT. Only targets from the ST were used. Although there was a trend for Pr to be larger in the IT (M = .84) compared to the ST (M = .79), this difference was not reliable (F(1,15) = 2.72, P >0.10, η² = .15). However, Test Type and Studied Color interacted (F(1,15) = 8.29, P <0.01, η² = .36). Consistent with expectation, Pr was larger for unnatural (M = .86) relative to natural (M = .81) items during the IT. Surprisingly, however, Pr was larger for natural (M = .82) compared to unnatural (M = .76) items during the ST. RTs were shorter in the IT (M = 785 ms) than the ST (M = 908 ms; F(1,15) = 42.62, P <0.0001, η² = .74). However, Test Type and Studied Color interacted (F(1,15) = 12.17, P <0.003; η² = .44). During the IT, relative to natural object-color pairings (M = 817 ms), RTs were faster when unnatural object-color pairings (M = 752 ms) had been studied. By contrast, during the ST, relative to unnatural items (M = 937 ms), RTs were faster when natural object-color pairings (M = 879 ms) had been studied. Mean correct-rejection RTs were 827 ms in the IT and 872 ms in the ST.

ERP Data

Averages were computed for correctly judged natural- and unnatural-color objects during the study phase. During the test phase, only test trials for which the studied objects had been correctly judged as natural or unnatural were used. Averages were computed for correctly identified old and new items in the IT and for correctly identified target and new items in the ST according to the outline color in which the old objects had been encoded at study (natural, unnatural).

During study, there were no reliable differences (300 to 700 ms) between the ERPs elicited by natural compared to unnatural object-color pairings (Fs(1,15) <2.60, Ps >0.1) indicating, like the behavioral data, that they were processed similarly.

To determine whether reliable episodic memory (EM) effects were present (i.e., natural vs. new and unnatural vs. new), averaged voltages of 100-ms duration were calculated at midline fronto-central sites (Fz, FCz, Cz) to capture midline EM effects and at left, midline and right centro-parietal (CP1, CPz, CP2) and parietal (P1, Pz, P2) sites to capture parietal EM effects (Figure 1; [12]). The averaged voltages were subjected to Item Type (new, natural, unnatural) × Electrode Location ANOVAs separately for the IT and ST (using the target ERPs). The midline analyses revealed that, between 300 and 500 ms, there were reliable medial-frontal EM effects for natural and unnatural objects in item (Fs(2,30) > 28.00, Ps <0.0001) and source (Fs(2,30) > 14.00, Ps <0.0001) tests. Analyses at centro-parietal sites indicated significant parietal-EM effects between 500 and 600 ms for item (F(2,30) > 19.90, P <0.0001) and source (F(2,30) > 15.50, P <0.0001) tests. Hence, in accord with previous studies, reliable frontal- and parietal-EM effects were found for natural- and unnatural-color old objects in both tasks, indicating that familiarity- as well as recollection-based processes had been utilized.

Figure 1.

Grand mean ERPs (N=16) in the IT and ST. Arrows mark stimulus onset with timelines every 500 ms. Surface potential scalp topographies (third-order spherical spline [23]) based on the difference means (IT: unnatural-natural; ST: natural-unnatural) are underneath the waveforms. Dots indicate the electrode locations.

However, consistent with prediction, differences between natural- and unnatural-color old items were found for the time interval indicative of the parietal EM effect, i.e., between 500 and 600 ms (Figure 1), as indicated below. Because the ERPs to new objects in the item and source tests did not differ reliably between 300 and 700 ms (Fs <1), we contrasted, between the item and source tests, objects that had been studied with natural and unnatural colors using the averaged voltage between 500 and 600 ms. A Test Type (item, source) × Item Type (natural, unnatural) × Saggital (left, midline, right) × Anterior/Posterior (anterior-frontal [AF3, AFz, AF4], frontal [F3, Fz, F4], central [C3, Cz, C4], centro-parietal [CP3, CPz, CP4], parietal [P3, Pz, P4], parieto-occipital [PO3, POz, PO4]) ANOVA was performed. This more extensive set of electrodes was used to capture hemispheric and/or anterior-posterior influences between test- and/or item-types. Test Type interacted with Item Type (F(1,15) = 11.34, P <0.004, η² = .43). Post-hoc testing indicated that, whereas unnatural relative to natural items showed larger positivity during the IT, the reverse was the case during the ST (Figure 1). Test Type and Item Type also interacted with the Anterior/Posterior dimension (F(5,75) = 5.86, P <0.01, ε = 0.31, η² = .28). As seen in the maps in Figure 1, this was due to the fact that, while the difference between natural and unnatural items was posterior during the IT, during the ST the distribution did not extend posteriorly to the same degree. However, normalization of the data [22] failed to reveal a reliable difference in scalp distribution (F=1.16).

DISCUSSION

We set out to validate the prediction that recollection-based processes would be employed to a greater extent for unnatural- (i.e., incongruous) than natural-color objects (i.e., congruous), thereby driving the memory advantage previously observed for incongruous pictures [2,4]. Pictures of common objects were outlined during the study phases in either natural (congruous) or unnatural (incongruous) colors. Memory for the objects and their outline colors was then assessed by both IT and ST. During item testing, relative to natural-outline colors, objects encoded with unnatural-outline colors showed greater memory sensitivity and faster RTs, supporting the memory advantage reported previously. In addition, the mnemonic advantage for incongruous objects was associated with an increase in parietally-focused positivity. Based on two decades of prior research, the scalp distribution associated with unnatural-color objects is indicative of a recollection-based decision [14]. Hence, in accord with prediction, the data from the IT support the idea that the performance advantage for incongruous relative to congruous objects is based on increased recollection-related processing.

Contrarily, during source testing, which presumably relies on recollection to a greater extent, objects studied with natural-outline colors showed greater memory sensitivity as well as faster RTs than unnatural-color objects, indicating increased recollection for congruous than incongruous object-color pairings. This performance enhancement was also associated with a relatively parietal topography. Thus, the performance advantage favoring natural object-color pairings may also have been driven by recollection-based processes.

Therefore, the results appear to indicate that enhancement of recollection-based processes resulted in performance and positive ERP differences for unnatural objects in the IT and natural objects in the ST. How might this have occurred? During the natural/unnatural decision at encoding, a “natural”-color object could, potentially, be easily integrated in memory because the combined object-color entity matched the representation stored in semantic memory. By contrast, when an “unnatural-“color object was encountered, participants presumably had to also activate the natural color in order to make an informed decision. Although the additional activation of the natural object color is in line with more elaborative encoding and stronger memory traces for unnatural-color objects during the IT, having activated two colors at encoding could have led to source ambiguity during the source-testing phase. On this view, unnatural object-color pairings would have incurred a disadvantage during source testing because they did not match a pre-existing representation and, therefore, could not readily facilitate recollection of the context (i.e., color) with which the object was paired at encoding.

CONCLUSION

The presence of a larger centro-parietal EM effect (500 - 600 ms) for unnatural- relative to natural-color objects coupled with the better memory performance for the former in the IT support the prediction that recollection-based processes were responsible. However, source confusion for unnatural objects most likely underlies the observation that superior recollection was found for natural objects in the ST.