The worth of pictures: Using high density event-related potentials to understand the memorial power of pictures and the dynamics of recognition memory

Abstract

To understand the neural correlates of the memorial power of pictures, pictures and words were systematically varied at study and test within subjects, and high-density event-related potentials (ERPs) were recorded at retrieval. Using both conventional and novel methods, the results were presented as ERP waveforms, 50 ms scalp topographies, and video clips, and analyzed using t-statistic topographic maps and nonparametric p-value maps. The authors found that a parietally-based ERP component was enhanced when pictures were presented at study or test, compared to when words were presented. An early frontally-based component was enhanced when words were presented at study compared to pictures. From these data the authors speculate that the memorial power of pictures is related to the ability of pictures to enhance recollection. Familiarity, by contrast, was enhanced when words were presented at study compared to pictures. From these results and the dynamic view of memory afforded by viewing the data as video clips, the authors propose an ERP model of recognition memory.

Introduction

It is commonly said that “a picture is worth a thousand words.” Although once thought to be a Chinese proverb, this phrase is now attributed to Fred R. Barnard, who used it to promote the use of images in advertisements (Wikipedia, 2006). Nonetheless, for most of us this proverb rings true and we find that pictures provide a richness of information that is rarely matched by verbal description. Pictures have also been proven to enhance memory relative to words. For over 30 years the picture superiority effect has demonstrated that subjects are more likely to remember items if they are presented as pictures versus words (Nelson et al., 1976). More recently, comparing memory for pictures versus words has been used as a tool in memory research. Such studies have helped to better understand aspects of memory including false recognition (Israel & Schacter, 1997), the distinctiveness heuristic (Budson et al., 2005b), and retrieval orientation (Robb & Rugg, 2002). Examining memory for pictures and words has also helped us to better understand memory impaired populations (Budson et al., 2002a; Budson et al., 2005a). In the present study we were interested in better understanding the neural basis of the memorial power of pictures, and in particular how pictures affect three components of recognition memory: familiarity, recollection, and post-retrieval processes. We also thought it was important to be mindful of the above proverb when presenting our findings, and thus we worked to represent our results using images that both display the richness of our data and are easily and intuitively understood.

How familiarity, recollection, and post-retrieval processes work together has been the subject of some debate. Although some researchers believe that a single underlying brain process can explain all of these components, many researchers believe that familiarity and recollection rely on separable brain processes (Yonelinas, 2002). According to Woodruff, Hayama, and Rugg (2006) the process of familiarity reflects an acontexual sense that a test item has been seen before, while recollection involves the retrieval of contextually specific information about the test item. Single-process theorists argue that familiarity and recollection reflect quantitative rather than qualitative differences in retrieved information, and that a single-process model gives a better account of certain behavioral dissociations than a dual-process model (Donaldson, 1996; Dunn, 2004, Slotnick and Dodson, 2005). Dual-processes theorists argue that familiarity and recollection depend on parallel underlying processes and provide independent bases for recognition. It is assumed under the dual process model that recognition judgments can be based on remembering specific details of a previously experienced event (recollection), or on the strength of stimulus familiarity (Woodruff et al., 2006; Yonelinas, 2002).

Event-related potentials (ERPs) have been particularly useful in understanding these three components of recognition memory. Classic ERP studies of recognition memory have shown that studied or “old” items elicit a more positive ERP response than do nonstudied or “new” items (for review, see Friedman and Johnson, 2000). Traditional ERP researchers have broken down this old/new effect into three distinct components, possibly reflecting three distinct cognitive processes.

The first component typically occurs at bilateral frontal electrode sites between 300 and 500 ms, and appears to precede controlled attempts by the individual to recollect information. Rugg, Curran, and others have dubbed this component the FN400 and posit that attenuation of a frontal negativity peaking around 400 ms is associated with familiarity (Curran, 2000; Friedman & Johnson, 2000; Rugg et al., 1998). For example, Curran (2000) created a high familiarity group of test items by using words that were similar to studied items, and found that familiar words elicited a more positive response at frontal electrode sites than did unstudied words. Further, a recent ERP study using confidence ratings showed that the positivity of the FN400 varied directly with familiarity strength (Woodruff et al., 2006). For the purposes of the current investigation, we will refer to this effect as the early frontal effect.

The second and most widely studied component of the old/new effect typically occurs maximally at parietal electrode sites, particularly left, between 500 and 800 ms. Research has shown this parietal effect to be less sensitive to familiarity, and enhanced by items correctly identified as previously studied (Woodruff et al., 2006). Therefore, this positive deflection has been associated with recollection. According to Friedman and Johnson (2000), this parietal effect increases with study-test repetitions (Johnson et al., 1998a), when items are rated as consciously recollected (Smith, 1993; Smith and Guster, 1993), and when a word is retrieved along with study context (Trott et al., 1999; Wilding et al., 1995; Wilding and Rugg, 1996).

Recently, researchers have been working to better understand the role of the parietal lobes in recognition memory. Current hypotheses generated by fMRI studies suggest that the parietal lobes play an important but relatively unknown role in retrieval success (Shannon & Buckner, 2004), which can be generalized across different modalities, response conditions, and task formats (Wagner et al., 2005). The fact that the parietal cortex is active in both imaging and electrophysiological investigations of recognition memory has invited research to integrate both techniques to better understand the role of the parietal lobes in episodic memory (Iidaka et al., 2006).

The last component of the old/new effect typically occurs later in the waveform over frontal electrode sites, with right hemisphere predominance. This late ERP effect is associated with post-retrieval verification and monitoring processes, particularly when the contents of memory are evaluated for details, and features such as contextual information (Allan et al., 1998; Wilding and Rugg, 1996). In other words, this late frontal activity is associated with the ongoing evaluation and monitoring of the product of the retrieval attempt. Imaging research has suggested that this late frontal effect may reflect executive control functions of the prefrontal cortex while retrieving a memory (Buckner et al., 1996; Fletcher et al., 1998).

A number of ERP studies have compared memory for pictures versus words to better understand aspects of recognition memory such as retrieval orientation and the distinctiveness heuristic. Retrieval orientation refers to the specific form of processing applied to a retrieval cue, or a cognitive state that aligns cue processing with the demand of a particular retrieval goal (Rugg & Wilding, 2000). In a study investigating retrieval orientation, Hornberger, Morcom, and Rugg (2004) showed that ERPs elicited by unstudied pictures and words were more negative for both modalities when the study and test modality were not matched. The authors concluded that retrieval orientation effects depend on differences in similarity between study and test modality, and not on the form of the material being retrieved. Hornberger, Rugg, and Henson (2006) supported this conclusion in an fMRI study that showed that the same material-specific cortical regions are engaged in both encoding and retrieval, and this overlap increases the likelihood of successful recognition.

In an investigation of the distinctiveness heuristic—a response mode or orientation in which participants expect to remember vivid details of an experience and make recognition decisions based on this metacognitive expectation—Budson et al. (2005b) showed that when pictures were the study modality, ERP differences were present in the parietal component, whereas when words were the study modality, differences were present in the late frontal component. The authors concluded that the distinctiveness heuristic is a retrieval orientation that facilitates reliance upon recollection to differentiate between item types. When pictures were the study modality, subjects can rely upon recollection to differentiate between items. However when words were the study modality, subjects needed to engage in additional post-retrieval processes to distinguish between items.

Perhaps most relevant to the present study, Schloerscheidt and Rugg (2004) used pictures and words to investigate the impact of study-test modality change on the old/new effect of recognition memory. Participants studied mixed lists of pictures and words. At test, one group of participants made recognition memory judgments with pictures as the test modality, and another group of participants made judgments with words as the test modality. Schloerscheidt and Rugg found that when study-test stimuli were matched, there was increased bilateral positivity during the 300 to 500 ms time interval. The authors suggested that this effect is sensitive to changes in perceptual characteristics of an item between the first and subsequent encounter, and is likely a correlate of perceptually-mediated familiarity. It was also reported that the matched modality condition showed a larger amplitude than the nonmatched condition over left parietal sites during the 500 to 800 ms time interval. They concluded that their results provided additional evidence for the functional dissociation between an early frontal component sensitive to perceptually-driven familiarity, and a later temporo-parietal component sensitive to recollection.

We set out to investigate the effect of pictures versus words at study and test on the old/new memory effect, and whether these effects could dissociate familiarity, recollection, and post-retrieval processes. To examine these effects, we systematically varied study and test modality using a within-subjects design. All subjects saw four study-test phases (word-picture, picture-word, word-word, picture-picture). Because of the findings from Hornberger et al. (2004), which demonstrated the importance of study and test modality being matched versus nonmatched, we felt that it would be difficult to interpret direct comparisons between the four study-test phases. Instead, we planned our primary analyses to either examine the effect of match versus nonmatch, or to take match/nonmatch into account. Therefore, our primary analyses of interest were to examine three main comparisons: (1) congruence of study-test modality: matched (average of word-word & picture-picture) versus nonmatched (average of word-picture & picture-word), (2) study modality: word (average of word-word & word-picture) versus picture (average of picture-picture & picture-word), and (3) test modality: word (average of word-word & picture-word) versus picture (average of picture-picture & word-picture).

Below we review our a priori hypotheses for each of the major ERP components of recognition memory, the early frontal effect (300–500 ms), the parietal effect (500–800 ms), and the late frontal effect (1000–1800 ms) for each of these three main comparisons of interest (also see Table 1). In addition to these analyses however, we also present our results using methodologies that allow the analysis of the entire data set of all 128 electrodes over the full 1800 ms of recording.

Table 1.

ERP hypotheses for the three comparisons at each time interval of interest

	300–500	500–800	1000–1800
Match vs. Nonmatch	Increased bilateral frontal positivity for match	Increased parietal positivity for match	Increased in frontal activity for nonmatch
Study and Test Modality	Increased bilateral frontal positivity for words	Increased parietal positivity for pictures	Increased frontal positivity for words

For the matched versus nonmatched comparison, we hypothesized that there would be increased bilateral frontal positivity during the 300 to 500 ms time interval for the matched compared to the nonmatch condition, because items studied in the same modality would elicit greater perceptually-mediated familiarity (Schloerscheidt & Rugg, 2004). We also posed that there would be greater parietal positivity for the matched versus nonmatched condition during the 500 to 800 ms time interval, because items in the matched study and test modality would be better recollected than items in the nonmatched study and test modality (Schloerscheidt & Rugg, 2004). Finally, we hypothesized that there would be an increase in frontal activity for the nonmatched versus matched condition during the 1000 to 1800 ms time interval reflecting additional post-retrieval verification processes when study and test stimuli are nonmatched compared to when they are matched.

In order to form hypotheses for the comparison of word versus picture study modality, we first must theorize how recognition of words and pictures is likely to occur. Pictures, being relatively distinctive and unique, are likely to be either recollected at test as having been seen before or not. This view of all or none pictorial recognition is supported by Budson et al., 2005b, Schacter et al., 1999, and other studies of the distinctiveness heuristic. Thus, we predicted that there would be greater parietal positivity during the 500 to 800 ms time interval when pictures were the study modality compared to words. Common words, on the other hand, whether they were presented on the study list or not, will have been seen by the participant at some previous time. Thus, all of the words will be recollected at test by the participant’s semantic memory system. Although some words may generate additional episodic recollection if it engendered a specific association when studied, we suspect that participants will distinguish most studied versus nonstudied words at test because the studied words, having been seen more recently, will be more fluent and thus more familiar than the nonstudied words (Wolk et al., 2004). We therefore predicted that familiarity at test would be more predominant when words were studied compared to when pictures were studied, leading to increased frontal positivity during the 300 to 500 ms time interval in the word study compared to the picture study condition. For the late frontal effect, we hypothesized that there would be an increase in frontal activity during the 1000 to 1800 ms time interval for words as the study modality compared to pictures, reflecting additional post-retrieval verification processes needed for the word condition (Budson et al., 2005b). Lastly, our predictions for comparisons of word versus picture test modality were the same as our predictions for study modality, for similar reasons. (Schloerscheidt and Rugg, 2004, provides some support for the late frontal prediction).

Methods

Design Overview

The experimental design systematically varied words and pictures at study and test to generate four separate study-test phases: word-word; picture-picture; word-picture; picture-word. All phases presented 50 stimuli at study and 100 stimuli at test. Subjects completed all four study-test phases in a single two-hour session with breaks between each phase. High-density ERPs were recorded at test.

Subjects

Seventeen subjects participated in the experiment. All subjects were right-handed, English was their native language, 7 were female, ages ranged from 18 to 25 (mean 21.2), and education averaged 14.25 years. The study was approved by the human studies committee of the Edith Nourse Rogers Memorial Veterans Hospital. All subjects gave IRB-approved informed consent before participating in the study, and were compensated at the rate of $25 per hour.

Experimental Material and Methods

The color pictures were obtained from a stimuli set previously used by Gallo, Kensinger, and Schacter (2006). The pool of experimental stimuli consisted of 480 color pictures of nameable objects (nouns) and 480 words corresponding to the names of the objects. From the total pool, 400 pictures were randomly selected. These 400 items were counterbalanced across each of the four study-test phases. The studied and unstudied items were also counterbalanced across subjects. Color pictures were presented in central vision on a white background, with an average height of 13 cm and an average width of 15 cm. Words were presented in central vision in black uppercase letters 4 cm in height, also on a white background. All stimuli were presented on a 21-inch flat screen computer monitor positioned 48 inches from the subject.

During the study portion of all phases, subjects were asked to make like/dislike judgments of the stimuli, and asked to remember the stimuli for a subsequent memory test. Each trial began with a 1000 ms fixation character (“+”) prior to the presentation of study stimuli. Study stimuli were the presented for 2000 ms followed by the question, “Do you like this item?” Subjects were then prompted to button press to signify their like/dislike judgment, followed by the next trial.

Each test trial began with a 1000 ms fixation character (“+”) prior to the presentation of the stimuli. Test stimuli were presented for 1500 ms, followed by this question, “Is this item old or new?” Subjects were then prompted to button press to signify their old/new judgment, followed by the next trial. Subjects were asked to hold their old/new response until the question appeared immediately after stimuli presentation to minimize response-related ERP artifact.

ERP Procedure

Subjects were seated in a hardback chair and fitted with an Active Two electrode cap (Behavioral Brain Sciences Center, Birmingham, UK). A full array of 128 Ag-AgCl BioSemi (Amsterdam, Netherlands) “active” electrodes were connected to the cap in a pre-configured montage according to the Active Two electrode cap, which places each electrode in equidistant concentric circles from 10-20 position Cz (Figure 1). Active electrodes are amplified through the electrode at the source and do not require abrading of the skin or measuring skin-electrode impedance levels. In addition to the 128 scalp electrodes, two mini-biopotential electrodes were placed on each mastoid process. Finally, vertical and horizontal electrooculogram (EOG) activity was recorded from bipolar electrodes placed above and below the left eye and on the outer canthus of the left and right eye. EEG and EOG activity was amplified with a bandwidth of 0.03–35 Hz (3 dB points) and digitized at a sampling rate of 256 Hz. Recordings were referenced to a vertex reference point, but were later re-referenced to a common average reference to minimize the effects of reference-site activity and accurately estimate the scalp topography of the measured electrical fields (Curran et al., 2006; Dien, 1998).

1.

Positions of the 128 electrodes on the Bio-Semi ActiveTwo headcap with the eight regions of interest shown.

The sampling epoch for each test trial lasted for a total of 2000 ms, which included a 200 ms pre-stimulus baseline period. This pre-stimulus period was used to baseline correct averaged ERP epochs lasting 1800 ms. ERPs were averaged and corrected using the EMSE Software Suite (Source Signal Imaging, San Diego, CA). Trials were corrected for excessive EOG activity using the empirical EMSE Ocular Artifact Correction Tool, in which artifact data are manually distinguished from clean data by the investigator. The Ocular Artifact Correction Tool then produces a logarithmic ratio of artifact data versus clean data and subtracts artifact data where it is detected. Trials were discarded from the analyses if they contained baseline drift or movement greater than 90 μV. Individual bad channels (which occurred very infrequently) were corrected with the EMSE spatial interpolation filter.

Statistical Analysis

Analysis of the behavioral data used a repeated measures ANOVA to examine the effects of match condition (match, nonmatch) and test modality (pictures, words) in a procedure similar to Hornberger, Morcom and Rugg (2004).

The statistical approach used in the current study for the ERP data was influenced by a number of sources (Budson et al., 2005b; Curran, 2000; Curran et al., 2006; Dien, 1998; Galan, et al., 1997; Greenblatt and Pflieger, 2004; Handy, 2005; Karniski, et al., 1994; Schloerscheidt and Rugg, 2004; Srinivasan, 2005). Traditional ERP analyses of selected time intervals was guided by previous work showing that the N400 old/new effect typically occurs between the 300 and 500 ms time interval over frontal electrode sites, the LPC old/new effect typically occurs between the 500 and 800 ms time interval over parietal electrode sites, and the late frontal effect typically occurs between the 1000 and 1800 ms time interval over frontal electrode sites (Budson et al., 2005b; Curran et al., 2006; Curran, 2000 Curran, 2004).

To examine the effects of match / nonmatch, study modality, and test modality on recognition memory performance, old/new ERP waves (hits minus correct rejections) were used in the statistical analysis. Mean amplitudes were calculated for the three time intervals (300–500 ms, 500–800 ms, 1000–1800 ms), which were then averaged across groups of electrodes to form eight separate regions of interest [Left Anterior Inferior (LAI), Right Anterior Inferior (RAI), Left Anterior Superior (LAS), Right Anterior Superior (RAS), Left Posterior Superior (LPS), Right Posterior Superior (RPS), Left Posterior Inferior (LPI), and Right Posterior Inferior (RPI)]. Each region of interest (ROI) consisted of a seven-electrode cluster. See Figure 1 for scalp topography of the eight ROIs.

First presented are t-tests at each region for each of these time intervals to examine (1) the effect of match / nonmatch of study-test modality, (2) the effect of pictures at study, and (3) the effect of pictures at test. In addition to these three analyses, non-parametric permutation tests were used to examine for topographic differences in 50 ms intervals throughout the recording epoch. Typically used in imaging studies to compare voxels between two different conditions, non-parametric permutation tests can be useful in analyzing high density ERP data (Galan, et al., 1997; Greenblatt & Pflieger, 2004; Karniski et al., 1994).

Results

To examine the effects of match / nonmatch, study modality, and test modality on recognition memory performance, old/new ERP waves (hits minus correct rejections) were used in the analyses below. In addition to presenting the data as waveforms, we have also presented the data as scalp topographies averaged over every 50 ms from 0 to 1800 ms, and as topographic videos that may be accessed by clicking the hyperlinks in the text, or via our web site, http://www.thebrainlab.org/videos.htm. As discussed below, we have found that different insights into the data can be gained when looking at brain activity in different ways.

The waveforms, topographies, and videos were formed from a mean (range) of 41 (32–50) hits and 42 (36–50) correct rejections. Another possible analysis would be to compare hits and misses as to not confound item type. However, in the current investigation the number of miss trials (mean = 3, range = 0–12) was inadequate to perform such an analysis.

Behavioral Performance Analysis

Two separate ANOVAs were used to examine the effects of match condition and study and test modality on recognition memory performance for these four study-test phases. Recognition accuracy was calculated using the straightforward discrimination index Pr (%Hits - %False Alarms; Snodgrass and Corwin, 1988). Table 2 shows the percent accuracy for each of our experimental conditions. To examine the effect of pictures as the study modality, an ANOVA used the factors of Match Condition (match, nonmatch) and Study Modality (pictures, words). Results of the ANOVA revealed no main effect of Match Condition [F(1, 16) < 1], or significant interaction of Match Condition and Study Modality [F(1, 16) = 1.03, p = .325]. However, there was a main effect of Study Modality [F(1, 16) = 5.56, p = .031] such that discrimination was better when pictures were the study modality than when words were the study modality. To examine the effect of pictures as the test modality, an ANOVA used the factors of Match Condition and Test Modality. The ANOVA revealed no significant main effect of Match Condition [F(1, 16) < 1] or Test Modality [F(1, 16) = 1.03, p = .325]. However, the interaction of Match Condition and Test Modality was significant [F(1, 16) = 5.56, p = .031]. Follow up t-tests revealed that the these effects were mainly attributable to the discrimination for the P-W phase being significantly better than for the W-P phase [t(16) = 2.42, p = .028].

Table 2.

Behavioral performance using index Pr (%Hits - %False Alarms)

Phase or Condition	Accuracy
Word-Picture	.80
Picture-Word	.87
Word-Word	.86
Picture-Picture	.88
Matched	.87
Nonmatched	.84
Pictures at Study	.88
Words at Study	.83
Pictures at Test	.84
Words at Test	.86

Match versus Nonmatch ERP Analysis

To better understand the effect of matched versus nonmatched study and test modality, planned comparison t-tests were performed at all regions of interest for our three time intervals. See Figure 2 for hit and correct rejection waveforms for the match versus nonmatch analysis. Figure 5 shows t-test comparison [t(16)] significance levels at all eight ROIs during each of our three time intervals. During the 300 to 500 ms time interval, region of interest t-test comparison revealed that the LAS region was significantly more positive for the match than the nonmatch condition. During the 500 to 800 ms time interval, the RAI region more positive for the match than for the nonmatch condition, whereas the RPS region was significantly more positive for the nonmatch than the match condition. During the 1000 to 1800 ms time interval, the LAS region more positive for the match than the nonmatch condition, whereas the RPS region was more positive for the nonmatch than the match condition.

2.

Grand average hit and correct rejection ERP waveforms for the match and nonmatch conditions for the eight regions of interest.

5.

Topographic maps displaying the significance levels of t-test comparisons [t(16)] between old/new difference waves for each condition, for each region of interest during the three time intervals of interest.

Next we describe the differences that can be observed in the old/new scalp topographies (Figure 6) and the topographic videos between the match and nonmatch conditions. (Note that the individual topographies in Figure 6 are an average of 50 ms going forward from the labeled time, such that 0 ms is the average from 0 to 49 ms, etc.) The adjectives “strong” and “weak” in the descriptions below refer to the intensity of the signal on the scalp topographies that in turn indicate high and low amplitudes, respectively. In the match condition [insert video hyperlink here] (average of word-word [insert video hyperlink here] & picture-picture [insert video hyperlink here]), (1) a brief period of weak early occipital negativity occurs from about 150 to 250 ms, followed by (2) a strong early frontal positivity from 400 to 500 ms, then by (3) a strong parietal positivity, greater on the left, from 450 to 700 ms, and by (4) sustained late right frontal activity starting at about 600 and continuing until 1650 ms. In the nonmatch condition [insert video hyperlink here] (average of word-picture [insert video hyperlink here] & picture-word [insert video hyperlink here]), (1) the early occipital negativity is not seen, (2) the early frontal positivity occurs somewhat later, from about 450 to 550, (3) the parietal positivity, relatively bilateral, is observed starting at about 500 ms and continuing with sometimes greater and sometimes lesser intensity until 1450 ms, and (4) the late right frontal activity can be seen staring later, at about 800 ms, and continuing until 1700 ms.

6.

Old/New scalp topography maps for each of the individual phases and experimental conditions. Topographies are presented in 50 ms averages going forward.

To support these observed differences between scalp topographies, we used a non-parametric permutation test to examine differences between the match and nonmatch conditions at each electrode position throughout the 1800 ms waveform. Figure 7 shows the topographic p-value map created by the permutation test for the two conditions at 50 ms intervals. (Again, note that the individual topographies in Figure 6 are an average of 50 ms going forward from the labeled time, such that 0 ms is the average from 0 to 49 ms, etc.) Differences were present (1) in the occipital regions bilaterally from 50 to 250 ms, (2) in the right inferior frontal region from 450 ms to 550 ms, (3) in bilateral parietal regions starting at 600 ms, present consistently until 900 ms, and intermittently until about 1450 ms.

7.

Topographic p-value maps created by the nonparametric permutation test. Topographies are presented in 50 ms averages going forward.

Study Modality ERP Analysis

Analogous planned comparison t-tests were performed for pictures versus words at study. See Figure 3 for hit and correct rejection waveforms for the study modality analysis. Figure 5 shows t-test comparisons [t(16)] significance levels at all eight ROIs during each of our three time intervals. During the 300 to 500 ms time interval, the LAS region was more positive for words as the study modality compared to pictures. During the 500 to 800 ms time interval, both superior frontal regions (LAS, RAS) were more positive when words were the study modality compared to pictures, and there was a near significant trend for the RPS region to be more positive for words than pictures. Finally, during the 1000 to 1800 ms time interval, the RAS region was more positive when words were the study modality compared to pictures, whereas the LPI region was more positive when pictures were the study modality compared to words.

3.

Grand average hit and correct rejection ERP waveforms for the study modality conditions for the eight regions of interest.

We now turn to the differences that can be observed in the old/new scalp topographies (Figure 6) and the topographic videos between words and pictures as the study modality. When words were the study modality [insert video hyperlink here], (1) early frontal positivity occurs from about 350 to 500 ms, followed by (2) strong parietal positivity starts relatively bilaterally at 500 ms and continues with sometimes greater and sometimes lesser intensity until 1100 ms, when weaker right parietal activity persists until about 1400 ms, and (3) sustained late right frontal activity is seen starting at about 750 ms, becoming very strong and broad, and continuing until 1700 ms. When pictures were the study modality [insert video hyperlink here], (1) early frontal positivity is seen only weakly from 400 to 500 ms, (2) parietal positivity starts at about 500 ms and then becomes weaker around 750 ms, until it ends at about 1100 ms, and (3) a small area of weak right frontal positivity is seen, less than with words, but starting earlier at about 550 to 1700 ms.

Again to support these observed differences between scalp topographies, we used the non-parametric permutation test to examine differences between the word and picture study modality at each electrode position throughout the 1800 ms waveform (Figure 7). This analysis revealed a few small trends toward differences in mid frontal regions from 250 to 400 ms, and significant differences in parietal, central, and frontal regions starting at 500 ms and continuing more or less until about 1600 ms.

Test Modality ERP Analysis

Analogous planned comparison t-tests were performed for pictures versus words at test. See Figure 4 for hit and correct rejection waveforms for the test modality analysis. Figure 5 shows t-test comparison [t(16)] significance levels at all eight ROIs during each of our three time intervals. During the 300 to 500 ms time interval, the RAI region was highly significant, due to the increased positivity when pictures were the test modality. In addition, the RPI (significantly) and LPI (near significant) regions were more positive when words were the test modality compared to pictures. During the 500 to 800 ms time interval, the RAI region showed a near significant trend towards being more positive when pictures were the test modality compared to words. During the 1000 to 1800 ms time interval, the RAI region was more positive when pictures were the test modality compared to words.

4.

Grand average hit and correct rejection ERP waveforms for the test modality conditions for the eight regions of interest.

Again we now turn to the differences that can be observed in the old/new scalp topographies (Figure 6) and the topographic videos between words and pictures as the test modality. When words were the test modality [insert video hyperlink here], (1) brief early frontal positivity is seen from 450 to 500 ms, followed by (2) parietal positivity started relatively bilaterally at about 500 ms and continued with sometimes greater and sometimes lesser intensity until 1550 ms, with the last 500 ms being predominantly right-sided, and by (3) sustained late right frontal positivity from around 850 to 1700 ms. When pictures were the test modality [insert video hyperlink here], (1) the early frontal positivity started earlier and was stronger from about 350 to 500 ms, (2) the parietal positivity started very similar at 500 ms but ended sooner, by about 1050 ms, and (3) the late right frontal activity started at 600 ms, and was sustained until 1700 ms.

Once again to support these observed differences between scalp topographies, we used the non-parametric permutation test to examine differences between the word and picture test modality at each electrode position throughout the 1800 ms waveform (Figure 7). This analysis revealed bilateral posterior differences from 350 to 600 ms, and then again intermittently from 1150 to 1650 ms.

Discussion

In this study we sought to understand the neural basis of how pictures affect memory compared to words. We systematically varied pictures and words at study and test and examined the neural correlates of recognition memory using event-related potentials. Our analysis took advantage of the fact that previous research has associated three ERP components with memorial processes: the early frontal effect with familiarity, the parietal effect with recollection, and the late frontal effect with post-retrieval processes. Our behavioral results were notable for showing that discrimination was better when pictures were the study modality compared to words, which was mainly attributable to the word-picture phase showing worse discrimination than the other three phases (Table 2). Our ERP results revealed many interesting findings based upon the differences observed in the waveforms (Figures 2, 3, & 4), the t-statistic maps based upon traditional analyses (Figure 5), the scalp topographies (Figure 6), the non-parametric permutation tests (Figure 7), and the videos. In the comparison between the match and nonmatch conditions we found (1) an early occipital negativity from about 150 to 250 ms that was only present in the match condition, (2) the early frontal positivity was shifted 50 ms earlier and was more positive in the match compared to the nonmatch condition, (3) the parietal positivity started 50 ms earlier, was more left-sided, and was much briefer for the match (450 to 700 ms) compared to the later, bilateral, and longer positivity of the nonmatch condition (500 to 1450 ms), and (4) the late right frontal positivity started earlier in the match (600 ms) compared to the nonmatch (800 ms) condition. Comparisons of study modality found that (1) the early frontal positivity started 50 ms earlier and was more positive when words were the study modality compared to pictures, (2) the parietal positivity was present in a smaller region and was briefer when pictures were the study modality (strong 500 to 750 ms, weak until 1100 ms) compared to words (strong 500 to 1100 ms, weak until 1400 ms), and (3) the late right frontal positivity started about 200 ms earlier when pictures were the study modality compared to words, but was stronger and encompassed a larger region when words were the study modality compared to pictures (consistent with Budson et al., 2005b). Lastly, comparisons of test modality found that (1) the early frontal positivity started 100 ms earlier and was more positive when pictures were the test modality compared to words, (2) the parietal positivity ended about 500 ms earlier when pictures were the test modality compared to words, and (3) the late right frontal positivity started about 250 ms earlier when pictures were the test modality compared to words. It is worth noting that in the test modality comparison, each condition (pictures at test, words at test) had either words or pictures as retrieval cues. Azizian et al. (2006) found that the electrophysiological response to process pictures was approximately 18 to 20 ms faster than words. Consideration of this finding must be taken when interpreting the current test modality results, as each of the components of interest began earlier when pictures were the retrieval cue. (Note that this confound was not present for our match/nonmatch and study modality comparisons. Each of these comparisons included the average of one word and one picture phase at test.)

Having reviewed our main ERP findings, we now turn to interpreting what we have learned about how pictures affect memory and in particular, the memorial components of familiarity, recollection, and post-retrieval processes. Prior to discussing these components, however, it is worthwhile commenting on the early occipital negativity observed from about 150 to 250 ms that was only present in the match condition. Previous ERP studies of recognition memory have identified an early component at around 200 ms that is both repetition and modality sensitive and may be related to implicit perceptual processing (Henson et al., 2004; Rugg & Allen, 2000), that is, perceptual priming (Slotnick & Schacter, 2006). Thus we think it possible that the occipital negativity observed in the match but not nonmatch condition may be related to perceptual priming occurring in the former but not the latter condition.

Previous research has suggested that the early frontal effect that we and others have observed is likely related to the neural correlate of familiarity (Curran, 2000; Friedman & Johnson, 2000; Rugg et al., 1998; Woodruff et al., 2006). We found that the early frontal effect was both more positive and shifted earlier for (1) the match compared to the nonmatch condition, (2) the word study modality compared to the picture study modality condition, and (3) the picture test modality compared to the word modality condition. Thus, our results suggest that there is greater familiarity when study and test modalities are matched compared to when they are nonmatched, when words are the study modality compared to pictures, and when pictures are the test modality compared to words. This first suggestion related to the differences between match and nonmatch conditions seems highly intuitive; when the item present at test is an exact semantic and perceptual match with the studied item, its familiarity is greater than when the test item is a semantic but not a perceptual match with the study item, consistent with Schloerscheidt and Rugg (2004). The second and third suggestions are less intuitive and require additional explanation. To begin, if we examine the individual phases, it appears that differences in the two nonmatched phases were primarily responsible for the differences in the study and the test modality analyses; the W-W and P-P phases did not differ for the early frontal effect. Thus, the increased positivity and earlier start for the early frontal effect in the W-P compared to the P-W phase is driving both the study and the test modality differences (Figure 6). Therefore, we need to explain why there should be increased familiarity for the W-P compared to the P-W phase.

One possibility is that this finding may reflect the inherent semantic range of words versus pictures. When words are the study modality, the semantic range to predict the picture at test is focused, compared to the broad semantic range possible when pictures are the study modality to predict the word at test. For example, if shown a “picture of an ant” at study, many words could be generated in response to the picture (“ant,” “bug,” “insect,” “arthropod,” etc.). However, when the word “ant” is shown at study, the range of possible pictures shown at test is much narrower. Thus, seeing a picture of an ant at test after seeing the word “ant” at study may be more likely to seem familiar to subjects compared to seeing the word ant at test after seeing a picture of an ant at study. Another related possibility is that participants may automatically form a mental image of the word seen at study, leading to increased familiarity for the picture of the item seen at test.

The parietal effect has been associated with recollection (Friedman & Johnson, 2000; Johnson et al, 1998; Smith, 1993; Smith and Guster, 1993; Woodruff et al., 2006). Examining the parietal effect for the different conditions, we found: (1) in the match condition the parietal effect stared earlier, was more left sided, and was much briefer compared to the nonmatch condition in which it started later, was bilateral, and lasted longer; (2) in the picture study condition the parietal effect was smaller and briefer compared to the word study condition in which it was larger and more prolonged; and (3) in the picture test condition the parietal effect ended much earlier than in the word test condition. At first glance, these findings are not consistent with our predictions, and require additional explanation. We had hypothesized that the parietal effect would be more robust in the match, picture study, and picture test conditions compared to the nonmatch, word study, and word test conditions, respectively. Instead, we found that the parietal effect was briefer in each of these comparisons. If we look at the left and right posterior superior waves in Figures 2 and 4, we see that the hits for the match and picture test conditions do show the greatest positivity (and in Figure 3 that the hits for the picture study condition are equal to hits for the word study condition). Thus, if we were comparing hits, our hypotheses would have been correct. In looking at these same figures for correct rejections, however, we see that the correct rejections are also more positive in the match, picture study, and picture test conditions compared to the nonmatch, word study, and word test conditions, respectively. Thus, when the old minus new subtraction is performed, the main finding is that the old/new effect is briefer and often more localized on the scalp in the conditions that we hypothesized would be more robust.

If our hypotheses were correct that recollection would be better for the match, picture study, and picture test conditions compared to the nonmatch, word study, and word test conditions, we then need an explanation of why the parietal effect should be briefer and more localized when recollection is better. One possible explanation is that how long the parietal effect lasts and how localized it occurs may depend upon the amount of time and resources needed to achieve recollection. When recollection is easier, less time and resources are required compared to when recollection is more difficult. Thus, when recollection is easier the parietal effect may be briefer and more localized compared to when recollection is more difficult.

Late frontal activity has been previously associated with post-retrieval verification and monitoring processes, when the contents of memory are evaluated for particular features and other details (Allan et al., 1998; Wilding and Rugg, 1996). We found that (1) in the match condition the late frontal effect started earlier compared to the nonmatch condition, (2) in the picture study condition the late frontal effect started earlier, but was much weaker and encompassed a smaller region compared to the word study condition, and (3) in the picture test condition the late frontal effect started earlier compared to the word test condition. The late frontal effect was also predominantly right-sided in all conditions. It should also be noted that, although not frontal, there was more late activity between 800 and 1800 ms in the nonmatch and word test conditions than the match and picture test conditions, respectively. The first aspect of these results that deserves comment is why late frontal activity continued throughout the recording epoch for all conditions. This finding likely resulted because we asked participants to hold their responses until prompted at 1500 ms, and thus the continued frontal activity is likely due to participants keeping their response “in mind,” or alternatively, inhibiting their natural inclination to respond before the prompt. That this late frontal activity started earlier for the match, picture study, and picture test conditions compared to the nonmatch, word study, and word test conditions, respectively, may be attributable to the prior parietal effect being briefer in the match, picture study, and picture test conditions compared to the nonmatch, word study, and word test conditions, respectively. We speculate that the late frontal activity can start earlier if the parietal effect finishes earlier. Additional processing is needed, however, in the nonmatch, word study, and word test conditions, which is reflected in additional late activity either frontally, parietally, or both, compared to the match, picture study, and picture test conditions.

If our understanding of how these ERP components reflect familiarity, recollection, and post-retrieval processing is correct, it is worth pausing to point out an interesting dissociation. Familiarity and recollection were both enhanced in the match compared to the nonmatch condition, and also in the picture test compared to the word test condition. However, familiarity was enhanced in the word study compared to the picture study condition, whereas recollection was enhanced in the picture study compared to the word study condition. Lastly, additional post-retrieval processing was engaged in the conditions in which recollection was not enhanced. Such findings may contribute to the discussion as to whether familiarity and recollection are two dissociable brain processes or not.

How do pictures enhance memory? We found that when pictures were presented at study or test compared to words, recollection was enhanced, and fewer post-retrieval processes were needed. Familiarity, however, was not enhanced when pictures were presented at study, although it was enhanced when pictures were presented at test. Thus, our data suggest that the memorial power of pictures is related to the ability of pictures to enhance one component of memory: recollection. Compared to words, whether pictures are presented at study, test, or both, recollection is enhanced, and thus fewer post-retrieval processes are needed. By contrast, familiarity is enhanced when words are presented at study compared to pictures.

The results of our study may also be informative regarding previous theories of the picture superiority effect. One theory is based on the dual-coding hypothesis, which states that distinct but interconnected mechanisms for picture and word processes are responsible for encoding (Paivio, 1971). Pictures are represented by an image code and words are represented by a verbal code, but these systems are interconnected such that either modality can evoke either code (see Mintzer and Snodgrass, 1999, for review). Paivio (1986 Paivio (1991) proposed that pictures are better remembered because they are more likely to be represented by both image and verbal codes (dual-coding), increasing the probability that they will be recollected compared to words. On the other hand, the sensory-semantic hypothesis posits that pictures possess highly distinctive visual information and features that allow unique encoding in memory (Nelson, 1979). Researchers refer to the inherently unique nature of the sensory or semantic features of pictures as the distinctiveness account. A comprehensive behavioral investigation and review of the literature completed by Mintzer and Snodgrass (1999) supports this distinctiveness account over the dual-coding theory. There have, however, been several convincing studies supporting each of these theories (Bright et al., 2004; Caramazza et al., 1990; Paivio, 1991; Warrington & Shallice, 1984).

Our results support the distinctiveness account. As stated above, the parietal effect was shorter in duration and more localized when pictures were the study modality, likely reflecting the shorter amount of time needed to achieve recollection. These data are most consistent with the distinctiveness account of the picture superiority effect. The unique sensory information provided by a picture may allow faster recollection of a studied item relative to a word. If, on the other hand, the dual-coding theory was correct, although that memorial accuracy would be better for pictures than for words, there is no reason to believe that recollection would be faster.

The preceding discussion naturally raises the following question: if the picture superiority effect is related to the distinctiveness of pictures facilitating recollection, would other types of distinctive stimuli provide a similar facilitation of recollection? Although this is clearly an empirical question to be answered by future experiments, we speculate that pictures are a special class of stimuli by virtue of their teleological and evolutionary importance. Distinguishing different images, whether faces, scenes, animals, plants, etc., was likely important for the survival of an individual. Therefore, although novel abstract images, for example, may be equally distinctive, we think it unlikely that they would facilitate recollection in a manner similar to pictures. Some support for the idea that pictures may be a privileged category of stimuli for the brain, is that studies in animals and humans have demonstrated areas of the brain devoted to very specific picture stimuli, including faces and places (e.g., Spiridon, Fischl, & Kanwisher, 2006; Tsao, Freiwald, Tootell, & Livingstone, 2006). Future studies that compare distinctive but non-pictorial stimuli with pictures will be able to answer this interesting question.

In the present study we have analyzed and presented our data in several different ways to achieve two main goals. We wanted to facilitate the most accurate and full view of our results as possible, and we wanted to make our data more accessible to those who do not routinely look at ERP data. After starting with more traditional ERP analyses, we attempted to look systematically at our entire dataset, rather than restricting our focus to particular brain regions and time windows. Our traditional analyses examined the averages of 8 groups of 7 electrodes each (total of 56 electrodes) across time windows of 200, 300, and 800 ms, whereas our newer analyses examined all 128 electrodes individually across time windows of 50 ms. Note that because of these disparities in the methodologies, differences in statistical significance between the traditional and nonparametric analyses were observed. Additionally, we endeavored to present data topographically, so that it could be more easily interpreted in light of functional neuroimaging data from fMRI and PET studies. Although one cannot draw reliable neuroanatomical inferences regarding the neural generators of ERP data, it is still easier for most researchers to visualize old/new differences that occur within a certain topographic distribution as scalp topographies than as waveforms, particularly when data are obtained from 128 electrodes. Lastly, for dynamic processes that occur over time such as recognition memory, we believe that there is valuable intuitive information that may be difficult or impossible to convey other than by watching that process over time as a video, which is why we have presented all our data in videos as well as static images.

Examining our data using these different methodologies, and in particular as videos, has led us to propose an ERP model of recognition memory based upon our data and the literature. The general schema of this model can be found in Figure 8. Research from many investigators has influenced this model, including that from Wagner, Buckner, and their colleagues (Wagner et al., 2005; Shannon & Buckner, 2004; Buckner et al. 2005) Yonelinas and the research he presents in his extensive review (Yonelinas, 2002), Schacter and colleagues (Schacter, et al., 1998; Schacter et al., 1999), Rugg and colleagues (Rugg & Wilding, 2000; Woodruff et al., 2006), Johnson and colleagues (Johnson et al., 1993), Brainerd and Reyna and their colleagues (Brainerd et al., 1995; Reyna & Brainerd, 1995), McClelland, McNaughton, and O’Reilly (1995), as well as others. And of course, we do not believe this simple model to be either exhaustive or the only way to view ERP memorial processes; it is nothing more or less than a tool that we are using to conceptualize what is occurring during recognition memory and to test in future experiments.

8.

Proposed ERP model of recognition memory.

At test during a visual recognition memory paradigm, an item is presented to a subject. First, a series of obligate or automatic processes occur. After very early sensory processing, occipital activity may occur between about 150 and 250 ms if the item is an identical perceptual match, reflecting perceptual priming (Henson et al., 2004; Rugg & Allen, 2000; Slotnick & Schacter, 2006).

Next from about 300 to 500 ms activity, often frontal, is modulated (typically a decrease in negativity) if the item is familiar (Curran, 2000; Friedman & Johnson, 2000; Rugg et al., 1998; Woodruff et al., 2006). Here perceptual matching is no longer critical, as shown by others as well as our data that demonstrated familiarity for pictures when words were shown at study and vice-versa. However, items that are more familiar for a variety of reasons show earlier and/or stronger activity than less familiar items, including in our study perceptual matching, consistent with Schloerscheidt and Rugg (2004). In addition to prior study, greater familiarity may be related to greater confidence (Woodruff et al., 2006), greater gist memory (Curran et al., 2001; Duzel et al., 1997; Goldmann et al., 2003), and greater fluency (Wolk et al., 2004, 2005). (It should also be noted that some researchers view this 300 to 500 ms modulation of activity as related to conceptual priming, rather than familiarity (Voss & Paller, 2006). However, because conceptual priming enhances conceptual fluency which in turn may cause a subjective experience of familiarity (e.g., Wolk et al., 2004), viewing this activity from 300 to 500 ms as related to conceptual priming would not change the interpretation of our model.)

Associated with recollection, parietal activity is often observed to start between about 450 and 550 ms, and often lasts about 400 to 500 ms (Friedman & Johnson, 2000; Johnson et al, 1998; Smith, 1993; Smith and Guster, 1993; Woodruff et al., 2006). Here our data and analyses may be particularly informative for our model. As mentioned above, we found that the parietal activity was briefer and more tightly localized for conditions in which we expected better recollection, compared to those in which we expected worse recollection. Thus, parietal activity starts when recollective processes begin, and ends when recollection is achieved. Parietal activity will therefore be brief if the time required to achieve recollection is brief, and it will be long if the time required to achieve recollection is long. Although exactly what processes the parietal activity represents unknown, we speculate (along with others) that it may reflect the actual matching of representations stored in memory with perceptual representations (Addis et al., 2006; Schnyer et al., 2005; Wagner et al., 2005).

If additional activity is needed to achieve recollection, or to achieve the specific recollection required, conscious and effortful processing is begun. Here two topographic areas of activity may be engaged, that with a frontal distribution and that with a parietal distribution. Drawing an analogy to models of working memory, we speculate that the parietal activity reflects processing of a sketchpad—again involved in the actual matching of representations stored in memory with perceptual representations—while the frontal activity acts as the executive and helps direct memory retrieval search attempts (Baddeley, 1995).

We therefore suggest that certain aspects of recollection may be conceptualized as being somewhat analogous to working memory. Many studies have shown more difficult tasks involving working memory require bilateral activation, regardless of the nature of the material being manipulated, and that there is also an increase in the number of activated brain regions as the complexity of the task increases (Newman, Carpenter, Varma, & Just, 2003; Jaeggi et al., 2003; Honey, Bullmore, & Sharma, 2000). In fact, parietal activation is positively correlated with reaction time during working memory tasks (Honey et al., 2000). Our data suggests that recollection may be viewed in a similar manner. When recollection is easy, obligate/automatic processing is all that is needed, reflected by brief parietal activity and little or no frontal activity. When recollection is more difficult, additional processing is needed, reflected by more extended and often bilateral parietal activity in addition to an increase in frontal activity.

A number of fMRI and lesion studies provide some anatomical support for our ERP model of recognition memory, and in particular for the role of the parietal and frontal brain regions. One frequent finding is that activity increases in the left lateral parietal region near Brodmann areas 39 and 40 when items are correctly recollected (Donaldson et al., 2001; Habib & Lapage, 1999; Henson et al., 1999a Henson et al., 2000; Konishi et al., 2000; Wheeler & Buckner, 2003). Dobbins and Wagner (2005) also found that the posterior parietal cortex showed greater activation during source recollections than with item novelty decisions. An imaging study examining the neural correlates of associative and item-specific memory found greater bilateral superior parietal activation for the associative recognition trials (Achim & Lepage, 2005). Evidence also suggests that parietal involvement occurs regardless of whether or not the correct response selected: Kahn et al. (2004) reported strong left inferior parietal, left precuneus, and posterior cingulate activation during perceived recognition (false alarms > correct rejections). Similarly, Wheeler and Buckner (2003) reported strong left lateral parietal activation during perceived recognition. Using a feeling-of-knowing paradigm, Schnyer and colleagues (Schnyer et al., 2005) found activation for “know” judgments in a number of posterior regions including the inferior parietal lobe. These authors speculated that activity in these regions reflects the reactivation of the stored memory representation.

The frontal lobes have been linked to several important roles when making recognition memory decisions. The frontal lobes are responsible for response inhibition (Shimamura, 1995), which is important to inhibit responding on the basis of familiarity alone. Even once this inhibition is achieved, numerous investigations have found that the frontal lobes are also necessary to distinguish between identical versus highly similar and familiar items, and thus to avoid false recognition (Budson et al., 2002b, Delbecq-Derouesne et al., 1990; Henson et al., 1999a; Henson et al., 1999b; Melo et al., 1999; Parkin et al., 1996; Parkin et al., 1999; Rapcsak et al., 1999; Rapcsak et al., 2001; Schacter et al., 1996). Lastly, frontal cortex plays an important role in selecting an appropriate response when there are competing options (Bunge et al., 2002; Desmond et al., 1998; Hazeltine et al., 2000).

A number of studies support our model of parietal and frontal interaction. Simons et al. (2005) found that parietal and prefrontal cortex were active when contextual information was retrieved. Bunge et al. (2002) found that the parietal and frontal cortices engaged in complimentary but dissociable roles in response selection: the left parietal cortex, particularly the left intraparietal sulcus and the left inferior parietal lobule, was activated when there was an increased need to maintain representations of possible responses, and bilateral prefrontal regions were activated when the manipulation required selection between competing responses. Also strongly supporting our model is a recent study using a word-based paradigm from Addis & McAndrews (2006) that used connectivity analyses to show a network of memorial multi-synaptic relays in which right inferior frontal gyrus interacts with left inferior frontal gyrus, which in turn interacts with both the left hippocampus and the left inferior parietal lobule, which also interact directly with each other.

Future studies will be able to critically examine aspects of this model to determine its veracity. For example, the difficulty of recollection can be systematically altered in different ways to see if, in each case, difficulty correlates with the time needed to recollect as measured by the parietal effect and by the need for late frontal and parietal activity. One such study we have considered is to present subjects with canonical views of objects at study, and then at test to present them with canonical views, rotated canonical views, and noncannonical views. We predict that in such an experiment the time to recollect, as measured by the parietal effect, will be slightly prolonged in the rotated compared to the canonical view, while the noncanonical view will require additional post-retrieval processes as measured by late frontal and parietal activity. If the model is proven accurate, then different populations—such as older adults and patients with memory disorders—can be evaluated to determine which brain systems allow them to remember.