An electrophysiological signature of summed similarity in visual working memory

Abstract

Summed-similarity models of short-term item recognition posit that participants base their judgments of an item's prior occurrence on that item's summed similarity to the ensemble of items on the remembered list. We examined the neural predictions of these models in three short-term recognition memory experiments using electrocorticographic/depth electrode recordings and scalp electroencephalography. On each experimental trial participants judged whether a test face had been among a small set of recently studied faces. Consistent with summed-similarity theory, participants’ tendency to endorse a test item increased as a function of its summed similarity to the items on the just-studied list. To characterize this behavioral effect of summed similarity, we successfully fit a summed similarity model to individual subject data from each experiment. Using the parameters determined from fitting the summed-similarity model to the behavioral data we examined the relation between summed similarity and brain activity. We found that 4-9 Hz theta activity in the medial temporal lobe and 2-4 Hz delta activity recorded from frontal and parietal cortices increased with summed similarity. These findings demonstrate direct neural correlates of the similarity computations that form the foundation of several major cognitive theories of human recognition memory.

Behavioral and model-theoretic analyses of recognition memory agree that similarity is a major determinant of both recognition accuracy and speed (e.g., Hintzman, 1988; Nosofsky, Little, Donkin, & Fific, 2011). Specifically, subjects’ recognition responses are influenced by a probe's similarity to item on the just-studied list. When a lure is similar to one or more studied items, participants will tend to commit false alarms, and when they correctly reject those similar items, their response times (RTs) are significantly increased. Moreover, a target is more likely to be endorsed when it is similar to the non-target items on the studied list (see Kahana, 2012, for a review).

Assessing the psychological similarity of a probe (be it a target or lure) to the stored representation of the items on a just-presented list requires a quantitative model of the mapping between physical and psychological coordinates, and of the way in which similarities combine across multiple study items. Summed-similarity models which describe both of these processes have been highly successful in accounting for behavioral data on item recognition (e.g., Hintzman, 1988; Kahana & Sekuler, 2002; Lamberts, Brockdor , & Heit, 2003; Nosofsky, 1991; Nosofsky & Kantner, 2006; Nosofsky et al., 2011; Shiffrin & Steyvers, 1997). These models posit that item recognition is a signal-detection process, in which a participant computes the probe's summed similarity to all study items, and bases her/his recognition decision on this measure. Whenever the computed summed similarity exceeds a decision threshold, the participant will respond yes. Thus, when summed similarity is high for probes that are targets, there is little interference and the participant is likely to respond yes. For probes that are lures, high summed similarity constitutes high interference, which also increases participant's tendency to incorrectly respond yes.

Summed-similarity models are generally viewed as models of familiarity. There is evidence that people can base recognition judgments on both recall of specific features of the encoding trial (termed “recollection”) as well as item familiarity. However, for lists of perceptually-similar items pure summed-similarity (familiarity) models provide an excellent fit to the data without positing a recollection component. Given summed-similarity models’ succes in describing trial-by-trial variation in participants’ behavior (e.g., the Noisy Exemplar Model, NEMo; Kahana & Sekuler, 2002; Nosofsky & Kantner, 2006), we sought to determine whether there is an electrophysiological correlate of the model-defined summed-similarity signal. Such a demonstration would provide neural support for the existence of the cognitive construct of summed similarity.

In three studies we examined whether oscillatory activity in both electrocorticographic (ECoG) recordings in neurosurgical patients and scalp EEG recordings in healthy controls correlate with summed similarity computations, with for other relevant variables being controlled for. We focused our attention on oscillatory responses because an extensive body of work has linked neural oscillations to cognitive processes in both humans and animals (see Jacobs & Kahana, 2010, for a review). Of particular relevance to the current studies is the finding that during memory retrieval slow oscillations, in the theta (4-9 Hz) and delta (2–4 Hz) frequency bands, are of higher amplitude for targets than for lures (Düzel, Neufang, & Heinze, 2005; Jacobs, Hwang, Curran, & Kahana, 2006) and also of higher amplitude for high-confidence correct decisions than for low confidence correct decisions (Guderian & Düzel, 2005). In contrast, oscillations at higher frequencies, specifically in the gamma band (28–128 Hz) have primarily been associated with manipulations of memory load (Howard et al., 2003; van Vugt, Schulze-Bonhage, Litt, Brandt, & Kahana, 2010). The ECoG study reported here builds on a prior study by van Vugt et al. (2009), which involved a subset of the same group of neurosurgical patients. By combining novel analyses of those ECoG data with data from two new scalp EEG studies we assayed neural data recorded at multiple spatial scales for direct evidence of summed-similarity computations. To parametrically investigate the neural correlates of similarity in item recognition we used specially-designed synthetic faces whose similarity is well-defined both in physical and psychological representational spaces (see Methods below).

Experiments 1-3

Methods

Because Experiments 1-3 used similar experimental and data-analytic methods, we present the methods for all three experiments together noting differences between studies as appropriate.

Synthetic Faces

We chose human faces as stimuli both for their ecological validity and because their similarities can be both quantified and parametrically varied. Rather than using photographs of faces as stimuli, we adopted a set of synthetic faces derived from real photographs but filtered to remove components like hair and skin texture while preserving the shape, size, and position of key facial features (Wilson, Loffer, & Wilkinson, 2002). Wilson's technique for generating synthetic faces begins by taking 37 physical measurements on a set of (normalized) photographs of Caucasian males, and then reconstructing the faces from the principal components that could be extracted from the matrix of these measurements. A stimulus set of 16 faces was constructed from all permutations of steps of one standard deviation away from the mean face (of the 37-dimensional face space) in the directions of each of the first four principal components (one standard deviation is approximately the threshold for 75% correct discrimination between two faces that are flashed for 110 ms; see Appendix). The resulting faces were low-pass filtered at ten cycles per face width, which is optimal for face processing and removes components like hair and skin texture (Wilson et al., 2002). The advantages of using this face set are that, although the faces are well-controlled, they still can be identified with high accuracy, and it is possible to measure their inter-item similarity precisely. Moreover, the faces are realistic enough to generate strong responses in the fusiform face area (Loffer, Yourganov, Wilkinson, & Wilson, 2005). This is important because previous studies have shown that the fusiform face area is sensitive to similarity distances between faces (e.g., Harris & Aguirre, 2010). As such it should be possible to find neural effects of similarity for these faces.

Multidimensional Scaling Study

We carried out a preliminary multidimensional scaling (MDS) study to characterize the psychological similarities among the items (see Appendix for details). In this MDS study, 23 participants served for two sessions, in which they saw all combinations of three faces (triads) twice, and had to determine the “odd-one-out” (i.e., the face that is least similar to the other two). From these ratings, a similarity matrix was constructed by increasing the similarity value for each of the two non-chosen faces (see Kahana & Bennett, 1994, for details). This similarity matrix was transformed into similarity coordinates for every face using individual-differences multidimensional scaling (INDSCAL/ALSCAL; Takane, Young, & de Leeuw, 1977). We used the four-dimensional stimulus coordinates derived from this procedure (see Appendix) in our summed-similarity analyses. The normalized raw stress measure for this solution was 0.022, which had a dispersion accounted for (DAF) of 0.98 (DAF is analogous to variance accounted for in a regression analysis, Heiser & Bennani, 1997). The attentional weights were very similar across participants as demonstrated by the small standard deviation for each of the four dimensions [mean attentional weight (standard deviation): 0.34 (0.026); 0.34 (0.042); 0.34 (0.024); 0.33 (0.015)].

Short-term item recognition paradigm

Figure 1 illustrates the sequence of events during each trial of the experiment. Following the appearance of a fixation stimulus (an asterisk that appeared for 1000-1075 ms, jittered) participants viewed a short series of faces. Each stimulus appeared for 700–775 ms, followed by a 275–350 ms interstimulus interval. After a 3000–3300 ms retention interval a probe item appeared and participants indicated with a key press whether the probe was a member of the just presented list (a target) or an item not shown on the current list (a lure). After each trial, participants were given accuracy feedback. The participant advanced to the next trial with a keypress. Temporal jitter was used to avoid spurious correlations between ongoing brain activity and task events. Participants were encouraged to be both fast and accurate, and participants in Experiments 2 and 3 were paid a bonus based on both their accuracy and response time. The experiment was programmed in the Python Experimental Programming Library PyEPL (http://pyepl.sourceforge.net; described in Geller, Schleifer, Sederberg, Jacobs, & Kahana, 2007).

Figure 1.

Trial structure of the Sternberg task. This figure illustrates the sequence and timing of events in a trial with three faces.

The three experiments differed in the number of faces presented in each series. Lists used in Experiments 1 and 2 comprised a variable number of faces whereas lists in Experiment 3 always comprised three faces. In Experiment 1 lists could have one, two, or three faces; in Experiment 2 lists could have two, three or four faces. Experiments 1 and 2 also contained lists of non-face stimuli which are not analyzed in the present report.¹

Each experiment also employed a “recent negatives” manipulation to study the effects of proactive interference on brain activity (Monsell, 1978). Whenever a lure probe was shown, it always had been a study item either one, two or three lists back (we refer to this lure recency as “list lag”). Lures that appeared as study items on recent lists are expected to give rise to significant proactive interference. Although we report on the behavioral effects of lure recency, these effects are not a major focus of the present study. With the exception of those probe items that were part of the recent negatives manipulation, lists were constructed so that items could not be repeated on successive lists. Lists were also constructed so that the frequencies of targets and lures and each of the list lengths and the three levels of the recent negatives manipulation were all matched. In addition, targets were equally likely to match a study item from each serial position. Blocks were then constructed by drawing lists at random from the pool, subject to the above constraints.

Every session was preceded by two 16-trial training blocks plus for Experiment 1, 40 additional one-item lists to familiarize the participant with the face stimuli. Participants were given feedback on their average accuracy and response time (RT) at the end of each 30-trial block. Incorrect trials and trials with RTs shorter than 200 ms or longer than 2500 ms (for Experiment 1) or 2000 ms (for Experiments 2 and 3) were removed from the analysis.

Participants

Participants in Experiment 1 were sixteen neurosurgical patients being treated for medically refractory epilepsy (ages 15–58; six female). Patients were monitored with arrays of subdural and/or depth electrodes to localize seizure onset and for mapping cognitive functions prior to resective surgery. Patients were recruited from Brigham and Women's Hospital in Boston, the Hospital of the University of Pennsylvania in Philadelphia, and Universitäts Klinikum Freiburg in Germany, and the research protocol was approved by the appropriate institutional review boards at each hospital. Informed consent was obtained from all participants.

In Experiment 2, thirty-seven young adults (ages 20–32) recruited from the University of Pennsylvania student community participated in two experimental sessions for payment. Each session involved ten blocks of 30 trials, as described above. In Experiment 3, twenty young adults (ages 19–29) participated in four experimental sessions for payment. Over the course of their four sessions, participants saw the same set of 66 study lists 16 times. The lists were divided into three blocks, the order of which was randomized across the four repetitions in each testing session.

ECoG and depth electrode recordings in Experiment 1

The local field potential was amplified and digitally recorded at sampling rates between 250 and 1024 Hz and bandpass-filtered between 0.1 and 100 Hz. Data were subsequently notch-filtered with a Butterworth filter with zero phase distortion to eliminate line noise. The intervals of interest were scanned for epileptic spikes and sharp waves by means of a kurtosis threshold; events were discarded if their kurtosis exceeded a threshold of 5 (see van Vugt et al., 2010, for details).

To synchronize electrophysiological recordings with behavioral events, the computer generating the task sent pulses via an optical isolator into an unused recording channel or digital input on the amplifier to time stamp the digital intracranial EEG (iEEG) recording. The time stamps associated with these pulses aligned the experimental computer's clock with the iEEG clock to a precision well under the sampling interval of the iEEG recording (<4 ms). For all participants, the locations of implanted electrodes were determined by means of co-registered postoperative computed tomographies and preoperative magnetic resonance imaging (MRI) or from postoperative MRIs by an indirect stereotactic technique and converted into MNI (Montreal Neurological Institute) coordinates. Localization of depth electrode contacts in the hippocampus was done manually through clinician's inspection of the postoperative MRIs.

Scalp EEG recordings in Experiments 2 and 3

We recorded scalp EEG signals using a 129-channel EGI Inc. system, with an AC-coupled, high-input-impedance amplifier (200 MΩ, Net Amps, Electrical Geodesics, Inc., Eugene, OR). The sampling rate was 500 Hz, and data were recorded with a 0.1–250 Hz bandpass filter. Individual channels were adjusted until impedances were below 50 kΩ. Bad channels were detected manually off-line, and the remaining channels were used for average re-referencing. Eye blinks were detected using a running-average threshold and a voltage threshold of 100 μV. Data from seven participants for whom more than 20% of events were rejected due to eye blinks were excluded, leaving us with 30 participants. One additional participant was excluded because of average accuracy below 60%. For Experiment 3, one participant was excluded because of average accuracy below 60%, and another for failure to exhibit serial position effects (and accuracy below 70%). All data were notch-filtered in the 58–62 Hz frequency band to remove line noise.

Regression-based analysis of EEG and ECoG oscillations

We used a multiple-regression approach (e.g., Jacobs et al., 2006) to assess the neural correlates of summed-similarity while controlling for the effects of proactive interference and within-list recency (i.e., serial position effects that cause the first and last items of a list to be remembered better). We computed the ordinary least squares regression coefficient of summed-similarity S, response time, and recency on z-transformed oscillatory power during the interval between probe onset and the moment the participant responded. Oscillatory power was determined using six-cycle Morlet wavelets at logarithmically spaced frequencies (2^(x/8) Hz for x ∈ 8, . . . , 56). Recency was defined separately for targets (lag_T ; in units of item position) and lures (lag_L; the number of lists ago the probe was last presented). Note that lure recency is a measure of proactive interference, with more recent lures corresponding to conditions of higher proactive interference. S was computed by fitting a summed similarity model (Kahana & Sekuler, 2002, described below) to the behavioral data from each individual participant. The parameters of this model were optimized with a pattern search algorithm that searched for the minimum of a goodness-of-fit measure that consisted of sum of the root-mean-square deviation between modelled and empirical summed similarity and the difference between modelled and empirical d-prime. In Experiments 1 and 2 S was binned into five levels of summed similarity; in Experiment 3 we had multiple repetitions of each list so we could fit the model to participants’ accuracy data on each individual list.

To assess whether the neural correlates of S changed over the course of the trial and/or differed between low and high frequency bands, we included terms representing the interaction between S and frequency band F (2–4 Hz delta, 4–9 Hz theta and 9–14 Hz alpha were coded as low frequencies; 14–28 Hz beta and 28–90 Hz gamma were coded as high frequencies), and between S and time interval T (first half or second half of the recognition probe-to-response interval). We also included the triple interaction term between S, F , and T . The length of the probe interval varied from trial to trial as a function of the participant's RT (i.e., oscillatory power was Vincentized; it was resampled into time bins whose size scaled with the length of the trial such that each trial had two time bins but their size increased with the participant's response time; Workman & Adams, 1950). Because recency was defined separately for targets and lures, we fit separate models to the electrophysiological data recorded during these two types of trials. For target trials we fit the model:

$$Z{\left(P\right)}_e={\beta }_0+{\beta }_{1}S+{\beta }_{2}la{g}_T+{\beta }_{3}S\times F+{\beta }_{4}S\times T+{\beta }_{5}S\times F\times T+{\beta }_{6}RT,$$

and for lure trials we fit the model:

$$Z{\left(P\right)}_e={\beta }_0+{\beta }_{1}S+{\beta }_{2}la{g}_L+{\beta }_{3}S\times F+{\beta }_{4}S\times T+{\beta }_{5}S\times F\times T+{\beta }_{6}RT,$$

where Z(P)_e indicates the z-transformed power at electrode e.

The resulting regression coefficients were then submitted to an across-subject t-test to establish population effects. For Experiments 2 and 3, we used a cluster-based method on the t-statistics to correct for multiple comparisons (Maris & Oostenveld, 2007). Clusters reflect the intuition that results are more likely to be true if they are present in groups of adjacent EEG channels. The procedure starts by comparing the p-value from each channel's t-test (on the regression coefficients) to a threshold of 0.01. We then assessed to what extent this binarized significance map consisted of clusters of spatially adjacent channels. For each cluster, we summed the t-statistics from all the channels comprising that cluster, which we compared to a permutation distribution to determine whether the cluster was significant. The permutation distribution was created by randomly swapping the sign of individual-subject regression coefficients and recomputing the clusters. Significant clusters are those whose summed t-statistic is in the upper tail of the distribution of summed-t-statistics based on randomized data. For Experiment 1, no such clustering was necessary, because we restricted our analyses to a small set of pre-defined regions of interest (ROIs).

The ROIs (pre-defined for Experiment 1; based on the S × F interaction for Experiments 2 and 3) were then used to examine the slope of oscillatory power on S. We used this S × F interaction to define our ROIs because we were primarily interested in oscillatory correlates of S that were specific for a frequency band. To ensure that any differences we would observe between Experiments 2 and 3 were not an artifact of having ROIs comprising of different sets of electrodes, we defined a parietal and frontal ROI based on the intersection of the ROIs for the two experiments in the lures condition. For each of these ROIs, we then assessed whether the slope of S on oscillatory power was consistent between target and lure trials, and between the two experiments.

Results

Behavioral Analyses

Serial position effects

We observed standard serial position effects in all three experiments, with the general pattern being a large recency effect and a small primacy effect. Serial position had a significant effect on accuracy but not RT in Experiment 1, as revealed by a repeated measures ANOVA (accuracy: F(2, 44) = 5.67, p < 0.01; RT: F(2, 44) = 1.29, n.s.). In Experiment 2 we observed significant serial position effects for both accuracy and RT (accuracy: F(3, 84) = 23.5, p < 0.001; RT: F(3, 84) = 8.0, p < 0.001). In Experiment 3 we also observed reliable serial position effects for both accuracy and RT (accuracy: F(2, 53) = 16.9, p < 0.001; RT: F(2, 53) = 35.6, p < 0.001). For these ANOVAs we considered the longest list length in each of the experiments, which was list length 4 in Experiment 2, and list length 3 in Experiments 1 and 3. Similar results were observed for list length 3 in Experiment 2.

Proactive interference effects

We next examined whether recently studied lures (i.e., lures that were members of recent prior lists) were more likely to be endorsed as targets. Although we did not observe a significant effect of our proactive interference manipulation in Experiment 1, we did observe clear proactive interference effects in both Experiments 2 and 3. In Experiment 2 both RT and accuracy showed a significant proactive interference effect. We quantified this proactive interference effect separately for each participant by regressing their accuracy and RT on list lag (where a list lag of 1 corresponds to high proactive interference and a list lag of 3 to low proactive interference). The mean slope of this regression (0.028) was significantly larger than zero (t(28) = 3.31, p < 0.001). The mean slope of RT (-27.7 ms) exhibited an analogous effect (t(28) = –3.72, p < 0.001). In Experiment 3, we found a significant proactive interference effect for accuracy (the mean slope of accuracy on the list lag of lures was 0.063, t(17)=8.5, p < 0.001) but there was no effect of lure lag on RT (mean slope = 0.59 ms, t(17)= 0.14, n.s.).

Application of summed-similarity model

We hypothesized that the basic computation underlying short-term item recognition is the summed pairwise similarity between an item's memorial representation and the representation of the test probe (Kahana & Sekuler, 2002; Nosofsky et al., 2011). If stimuli are randomly selected from a multidimensional space, the summed similarity of a target to the contents of memory will typically exceed the summed similarity for a lure. Kahana and Sekuler (2002) extended the basic summed similarity framework for item recognition (Hintzman, 1988; Nosofsky, 1991) by showing that the degree of summed similarity required to respond yes varies with list homogeneity (i.e., the mean similarity among list items). With greater list homogeneity, participants will respond yes at lower levels of summed similarity (Kahana, Zhou, Geller, & Sekuler, 2007; Nosofsky & Kantner, 2006; Visscher, Kaplan, Kahana, & Sekuler, 2007). Specifically, according to Kahana & Sekuler's noisy exemplar model (NEMo), participants respond yes when the similarity signal (S) defined as:

$$S=\sum _{i=1}^{LL}{\alpha }_i{e}^{-\tau \mid {\mathbf{s}}_{\mathbf{i}}-\mathbf{p}\mid }+\frac{2\beta }{LL(LL-1)}\sum _{i=1}^{LL-1}\sum _{j=i+1}^{LL}{e}^{-\tau \mid {\mathbf{s}}_{\mathbf{i}}-{\mathbf{s}}_{\mathbf{j}}\mid }$$ (1)

exceeds a decision threshold c_LL (the subscript LL indicates there is a separate optimal decision criterion for every list length). In Equation 1, LL is list length, α_i is a forgetting parameter, τ determines the form of the exponential generalization gradient (Shepard, 1987), s_i is the vector representing the coordinates of the memorial representation of stimulus i as determined by a prior MDS study, p is the representation of the probe item, and β determines the influence of list homogeneity (bold typeface indicates vectors). A Gaussian random variable with mean zero and standard deviation σ is added to each item's memorial representation to simulate the variability in participant's responses across different occurrences of the same item.

Our implementation is a reduced form version of previous versions of NEMo insofar as we assume that σ does not vary across stimulus dimensions and that each dimension is weighted equally in the similarity computation. We also assume that the decision threshold is chosen to yield the maximum likelihood of the simulated data. Although NEMo does not offer a process account of primacy or recency effects, the set of parameters α_i (i = 1 indicating the most recently studied item) allows the model to differentially weight items from different serial positions thereby matching any forgetting or primacy effects that may be present in the data (see also Nosofsky et al., 2011).

Although most previous applications of summed similarity theory have involved fitting group data, we chose to fit the model separately to each individual's behavioral data. In this way our aim of examining the neural correlates of summed similarity would not be subject to model misspecification at the level of individual participants. This approach enabled us to examine the relation between brain activity and summed similarity at the individual subject level, and then to aggregate those tests across participants.

Estimates of the best-fitting parameters we obtained (Table 2) are similar across the three experiments and also similar to those reported in previous applications of NEMo (Kahana & Sekuler, 2002; Kahana et al., 2007; Nosofsky & Kantner, 2006; Visscher et al., 2007; Yotsumoto, Kahana, Wilson, & Sekuler, 2007). Not surprisingly, the alpha parameters mirror the serial position effects in the data with a large recency effect and a small primacy effect across studies. The list homogeneity parameter, β, was negative in all studies and significantly negative in Experiments 2 and 3, which had the most statistical power (t(28) = –2.295, p < 0.05 and t(17) = –5.58, p < 0.01 for Experiments 2 and 3 respectively). This indicates that in general, list homogeneity decreases the decision threshold for responding yes.

Table 2.

Best fitting parameter values for NEMO's fit to behavioral data in Experiments 1-3.

Parameter	Experiment 1		Experiment 2		Experiment 3
	M	SEM	M	SEM	M	SEM
β	-0.31	0.52	-1.06	0.46	-1.68	0.30
τ	1.32	0.17	1.65	0.12	1.89	0.16
σ	1.21	0.084	0.99	0.061	0.82	0.048
α 2	0.58	0.041	0.76	0.038	0.71	0.032
α 3	0.67	0.051	0.65	0.042	0.81	0.033
α 4			0.72	0.038
RMSD	0.047	0.0026	0.021	0.0022	0.0811	0.0040

Note. RMSD: root-mean-square-deviation.

The central tenet of summed similarity theory is that the probability of responding yes to lures and to targets should increase with summed similarity. Figure 2 shows this effect in each of the three experiments. In all of the experiments the false alarm rate reliably increased with summed similarity (Experiment 1: M = 4.75, t(15) = 4.06, p < 0.01; Experiment 2: M = 6.15, t(28) = 7.94, p < 0.001; Experiment 3: M = 6.56, t(17) = 6.23, p < 0.001). For targets the effect was generally smaller, but nonetheless the hit rate was statistically significant in both Experiments 1 (M = 2.32, t(15) = 3.22, p < 0.01) and 3 (M = 1.66, t(17) = 4.72, p < 0.001).

Figure 2.

Summed similarity effect. The probability of saying yes is shown as a function of binned summed similarity (defined in equation 1) for both targets (open circles) and lures (filled circles). Bins are defined as the lowest and highest 50% of the summed similarity distribution. Error bars reflect Loftus-Masson corrected 95% confidence intervals. The fits for Experiment 1 are based on summed similarity for list length 3 only to make it comparable to the fits for Experiment 2, in which there were only list length 3 trials.

Electrophysiological Results

To avoid contaminating any observed interference-based effects with effects caused by differences in accuracy, we restricted our analyses to correct trials. For lure trials, where proactive interference was explicitly manipulated, we regressed oscillatory power at each electrode on summed similarity, lure recency (a measure of proactive interference), and RT. For target trials, where proactive interference was not manipulated, we regressed oscillatory power on summed similarity, target item recency, and RT. We fit the regression models (see Methods) to each individual's data, using the NEMo fits to determine the value of S on each trial. We then asked whether the distribution of regression coefficients across participants deviated significantly from zero, and whether this difference was modulated by time interval (first half vs. last half of retrieval period), frequency range (low vs. high), or both. Based on previous studies of the target-lure effect (e.g., Düzel et al., 2005; Jacobs et al., 2006), we expected to find a neural correlate of summed similarity in the lower frequency bands (particulaly delta and theta).

For our ECoG and depth electrode recordings in Experiment 1, we computed an average regression coefficient for every participant across all the electrodes in a set of 5 predefined regions (see Table 1). Statistical significance was determined using a p-value threshold set by a False Discovery Rate (FDR) procedure (Benjamini & Hochberg, 1995). A FDR of 0.05, as used here, means that one expects to find 5% of the effects deemed significant to be false positives. In our study, a FDR of 0.05 corresponded to a conventional p-value threshold of 0.011.

Table 1.

Numbers of patients and electrodes in each of five anatomical regions of interest.

Region of Interest	Brodmann areas	Num. Patients	Num. Electrodes
Dorsolateral prefrontal cortex	9, 10, 46	9	95
Temporal Cortex	20, 21, 22	16	413
Medial Temporal Lobe	28, 35, 36	14	126
Hippocampus	n/a	6	32
Parietal Cortex	1, 2, 3, 5, 7, 39, 40	13	126

We would expect an oscillatory correlate of summed similarity to be present for both targets and lures. Whereas a main effect of summed similarity, S, would suggest a broadband power modulation, an interaction between S and F (low-versus-high frequency bands) would lend support to a frequency-specific oscillatory correlate. Given prior findings of delta and theta band activity correlating with target-lure status, and of gamma-band activity correlating with memory load, we hypothesized that an oscillatory correlate of summed similarity would appear as an interaction between S and F , or as a three-way interaction between S, F and T (time epoch: early vs. late portion of the retrieval period).

For lures, we found reliable (p < 0.011) neural correlates of S in all five ROIs; however, only the temporal cortex and MTL exhibited reliable effects for both targets and lures, and in both of those ROIs we found significant interactions between S and F. In temporal cortex we observed a significant S × F interaction for both targets and lures; in MTL we observed a significant S × F × T interaction for both targets and lures (complete table in the Appendix). To further explore these interactions, we examined whether regression betas showed a monotonic increase or decrease with S across both target and lure trials. The frequency bands that show such monotonic increases or decreases are shown in Table 3.

Table 3.

Sign of slope of oscillatory power on similarity bins, grouped by ROI and time bin.

		Experiment 1
		Delta 2–4 Hz	Theta 4–9 Hz	Alpha 9–14 Hz	Beta 14–28 Hz	Low-gamma 28–48 Hz	Mid-gamma 48–90 Hz	High-gamma 90–128 Hz
Early	TC	0	0	-	-	-	0	+
	MTL	0	+	0	0	0	+	+
Late	TC	0	-	-	0	+	+	+
	MTL	0	0	0	0	0	+	+

		Experiment 2 & 3
		Delta 2–4 Hz	Theta 4–9 Hz	Alpha 9–14 Hz	Beta 14–28 Hz	Low-gamma 28–48 Hz	Mid-gamma 48–90 Hz	High-gamma 90–128 Hz
Early	Frontal	0 0	+ 0	+ 0	0 0	+ 0	+ 0	0 0
	Parietal	+ +	0 0	+ 0	+ 0	+ 0	0 0	+ 0
Late	Frontal	0 0	- +	- 0	0 0	0 0	0 0	0 0
	Parietal	+ 0	0 +	+ 0	+ 0	- 0	0 0	0 0

Note. ROIs for Experiment 1 are based on Table 1 and for Experiments 2 based on Figure 4/5(b).

“+”: a positive slope for both targets and lures, “-”: s a negative slope for both targets and lures, “0”: no consistent slopes between targets and lures.

TC= temporal cortex; MTL = medial temporal lobe.

Lower half of the table: the first symbol refers to Experiment 2, the second to Experiment 3. Bold face: frequency bands that have consistent slopes between targets and lures in both experiments.

In temporal cortex, lower-frequency oscillatory power (4–14 Hz) decreases with S, and higher-frequency oscillatory power (24–90 Hz) increases with S for both targets and lures. In the MTL, 4–9 Hz theta and 28–128 Hz gamma power increase with S in the early epoch, but the low-frequency effect switches sign in the late time bin (second half of the retrieval interval) resulting in a three-way interaction between S, F , and T. Figure 3 shows that in temporal cortex, theta power decreases with summed similarity while gamma power increases in the late epoch (this change in direction of the relation produces the significant interaction term). In MTL, both theta and gamma power increase with summed similarity, but now in the early epoch. The 2–4 Hz delta band does not show a consistent increase or decrease with summed similarity across target and lure trials.

Figure 3.

Experiment 1. Average normalized oscillatory power as a function of summed similarity bin in the early time bin in the medial temporal lobe (a) and in the late time bin in temporal cortex (b). In the medial temporal lobe, both 4–9 Hz theta and 48–90 Hz gamma power increase with summed similarity. In temporal cortex, theta decreases whereas gamma increases with summed similarity. Error bars reflect standard error of the mean.

Results from our scalp EEG studies (Experiments 2 and 3) are shown in Figures 4 and 5. These figures summarize the regression results for the targets model and the lures model respectively. The first row of each figure shows clusters of electrodes (see Methods) that exhibit significant oscillatory correlates of S that is not modulated by frequency band, retrieval epoch (early vs. late), or item recency. The subsequent rows of topographical plots shows clusters of electrodes that exhibit significant interactions between S and frequency range F (low vs. high), between S and time epoch (T), and between S, frequency range and time window. Red-shaded clusters indicate significantly positive regression coefficients, blue shaded clusters indicate significantly negative regression coefficients.

Figure 4.

Experiments 2 & 3. Topographical plots of significant (p < 0.05) electrode clusters for regression coefficients of the targets model. Electrodes shown in red exhibit a significantly positive regression coefficient across participants for (a) summed similarity (S), (b) interaction between S and frequency, (c) interaction between S and time, (d) triple interaction between S, frequency and time. Similarly, electrodes shown in blue exhibit a significantly negative regression coefficient across subjects. We do not show topographical plots for lag or RT, because those regression coefficients produce no statistically significant clusters.

Figure 5.

Experiments 2 & 3. Topographical plots of significant (p < 0.05) electrode clusters for regression coefficients of the lures model. Electrodes shown in red exhibit a significantly positive regression coefficient across participants for (a) summed similarity (S), (b) interaction between S and frequency, (c) interaction between S and time, (d) triple interaction between S, frequency and time. Similarly, electrodes shown in blue exhibit a significantly negative regression coefficient across subjects. We do not show topographical plots for lag or RT, because those regression coefficients produce no statistically significant clusters.

The most consistent effect across the two scalp EEG studies and the targets and lures models was the finding of clusters of primarily frontal but also left parietal electrodes that exhibited a significant interaction between summed similarity and frequency range (see Figures 4, 5). In electrodes for which this interaction was significant, the effect of oscillatory summed similarity effect differed between lower frequencies (delta, theta and alpha) and higher frequencies (beta and gamma). Blue electrodes indicate that the summed similarity effect was more positive for the lower frequency bands; red electrodes indicate the opposite. Whereas in both experiments we observed similar clusters that exhibited a stronger summed similarity effect for lower frequency bands, in Experiment 2 we also observed a cluster of central electrodes that exhibited a summed similarity effect that was more positive for high frequencies. In Experiment 2 we further observed clusters of significant electrodes that exhibited significant modulation of broadband power with summed similarity.

In Experiment 2, a set of frontal electrodes also showed an interaction between S and time epoch (Figure 5(c)), indicating that the summed similarity effect varied between the early and late epoch (i.e., first half vs. second half of the retrieval period). In this condition, frontal electrodes exhibited a more positive summed similarity effect for the early than for the late epoch. There was a small group of right parietal electrodes for lures that exhibited a triple interaction between S, frequency and time (Figure 4, 5(d)). The interaction between S and frequency band in these electrodes indicates that the frequency-specific summed similarity effect changes over the course of the trial. There was no significant effect of lag or RT on oscillatory activity during retrieval.

To further examine the frequency specificity of the summed similarity effect we computed normalized oscillatory power in the ROI defined by the summed similarity-by-frequency interaction in Figure 5(b). The ROIs were kept the same between Experiments 2 and 3 by taking the intersection of electrodes that were significant for both experiments. We then examined which of the frequency bands (2–4 Hz delta, 4–9 Hz theta, 9–14 Hz alpha, 14–28 Hz beta, 28–48 Hz low gamma, 48–90 Hz mid gamma, and 90–128 Hz high gamma) displayed the pattern of regression slopes expected of a neural correlate of summed similarity. A neural correlate of summed similarity should show a correlation with summed similarity in the same direction for targets and lures. Because we fit our regression model separately for targets and lures, a significant summed similarity-by-frequency-band interaction in the regression model does not guarantee that this condition is satisfied. Table 3 shows that there are patterns of oscillations in all frequency bands that satisfy this criterion.

In Experiment 3 we also observed a significant interaction between summed similarity and oscillatory frequency in frontal and left parietal electrodes (Figure 4,5(c)). We did not observe the negative interaction between summed similarity and frequency in central electrodes that was present in Experiment 2. There was a significant interaction between S and time epoch in a few frontal electrodes. There was no significant triple interaction between S, time and frequency band. The main effect of summed similarity occurred in a small set of right parietal and bilateral frontal electrodes, which increased in oscillatory power with S. As in Experiment 2, there was no statistically significant effect of lag or RT on oscillatory activity.

Considering the data from the two scalp EEG studies together, the most consistent positive correlate of summed similarity across both targets and lures and across both experiments is the delta band activity observed in parietal and frontal cortices. The theta band shows a pattern similar to that in the delta band, but the direction of the theta effect is not consistent across the same time epochs in the two experiments. In Experiment 2 theta increases with S in the first time bin whereas in Experiment 2 it increases with S in the second time bin. Although the increase in delta activity with S replicated nicely across both scalp studies, the effect was somewhat less consistent in the theta band, appearing at different time epochs in Experiments 2 and 3. This difference is not too surprising given the two major differences in procedure across the experiments: in Experiment 3, participants studied a fixed set of lists many times, whereas in Experiment 2, they saw each list once. Another possibility is that the difference is driven by the fact that in Experiment 2, different list lengths are intermixed, which is not the case in Experiment 3. Consequently, participants have to continuously update their decision threshold in Experiment 2, which they do not have to do in Experiment 3.

General Discussion

A number of the most successful cognitive theories of recognition memory posit that people judge a test item as old when its summed similarity to the list items exceeds a threshold value (e.g., Hintzman, 1988; Kahana & Sekuler, 2002; Lamberts et al., 2003; Nosofsky, 1991; Nosofsky & Kantner, 2006; Shiffrin & Steyvers, 1997). This summed similarity signal is regarded as a likely mechanism for the familiarity component of dual process theories of recognition (Rugg & Yonelinas, 2003). The same computational account has also been offered as the basis for categorization decisions when categories are defined by complex regions in multidimensional feature space (Fific, Little, & Nosofsky, 2010). In view of summed similarity's broad significance of as a latent construct in cognitive theories of memory, we asked whether summed similarity exhibited a distinct electrophysiological signature beyond the electrophysiological correlates of recognition accuracy, study-test lag, and other experimental factors.

Across three experiments we found that brain oscillations measured using depth electrodes and cortical surface electrodes in temporal cortex and the medial temporal lobe (Experiment 1) and measured at the scalp (Experiments 2 and 3) are reliably modulated by summed similarity for both targets and for lures. In particular, we found that lower frequency oscillations, in the delta and theta bands, increased with summed similarity for both targets and lures in the medial temporal lobe (Experiment 1), and in clusters of parietal and frontal electrodes (Experiments 2 and 3). Although the general trend is present in all three studies, there were also differences between the experiments: summed similarity correlated with theta band oscillations in the medial temporal lobe, whereas in the scalp recordings we found the effect in the delta band. Between the two scalp experiments, which varied in the list lengths used and the degree of practice that participants had, we observed the summed similarity correlate in the theta band earlier in the retrieval period in Experiment 2, and later in Experiment 3.

Although the prior literature suggests that theta oscillations are a stronger correlate of target-lure status than delta oscillations, the strongest effects in the literature have often been observed at around 4 Hz which is right at the boundary between the traditional theta and delta frequency bands. In addition, several studies have observed that delta and theta oscillations are similarly modulated during cognitive tasks (e.g., Jacobs et al., 2006; Lega, Jacobs, & Kahana, 2011; Mormann et al., 2005).

Related to the present work, several recent studies have begun to look at the neural correlates of similarity itself. For example, Loffler et al. (2005) showed that in the fusiform face area, the BOLD signal responds to changes in structural distance between a mean face and a face further away in the face space. Similarly, Aguirre showed that distance in similarity space was associated with the magnitude of fMRI adaptation in the fusiform face area (Aguirre, 2007). As such, they observed a neural similarity matrix that was correlated with the similarity matrix derived from the stimulus structure. Similarly, Kriegeskorte et al. (2008) demonstrated the existence of neural similarity matrices of a wide range of objects and faces in area IT (inferior temporal cortex), in both BOLD signal and single-unit activity. For verbal stimuli, Manning, Sperling, Sharan, Rosenberg, and Kahana (2012) showed that predominantly low-frequency oscillations across the cortex varied with their semantic similarity distances. Moving from perception to memory, Yago and Ishai (2006) compared the BOLD response in an item recognition task to probes that were the same, similar, ambiguous, or dissimilar to a studied prototype image (painting). They found an increase in BOLD activity with the level of similarity between probe and study item in the bilateral inferior parietal sulcus and superior parietal lobe. In addition, they found a decrease of BOLD activation with probe-item similarity in the superior frontal gyrus and precuneus.

A key assumption in summed similarity theories of item recognition is that participants use all items in the contextually-defined memory set to help distinguish between targets and lures. This assumption leads to the prediction that participants are more likely to endorse a target or a lure when there are other similar items stored in memory. However, summed similarity theory also assumes a non-linear (exponential) function relating similarity to distance in psychological space. The rate at which similarity decays with distance is a parameter of the model, and for the decay-rate parameters estimated in most studies, the actual contribution of most items to the summed similarity computation will be negligible. This means that in practice a summed similarity model will give most weight to items that are very similar to the target or lure and that less similar list items contribute minimally to the summed similarity calculation. Because of this non-linearity, it would require considerable statistical power to distinguish a summed-similarity model from a model that only uses the similarity of the most similar lure or target. Prior studies that have carefully explored this issue have shown that the summed similarity models do in fact provide a better quantitative fit to behavioral data on item recognition than models considering only the most similar list item (e.g., Huang, Kahana, & Sekuler, Under review; Kahana et al., 2007).

Our study provides neural evidence for the summed similarity computation hypothesized to underlie recognition memory and category learning (Brown, Neath, & Chater, 2007; Hintzman, 1988; Kahana & Sekuler, 2002; Lamberts et al., 2003; Nosofsky, 1991). Similarity-based interference is reflected in frontal and parietal delta and theta oscillations measured in scalp EEG, and in theta oscillations recorded from depth electrodes in the human medial temporal lobe. Given that previous studies have found similar neural correlates for old-new item recognition and categorization (Nosofsky, Little, & James, 2012), we would expect to find similar oscillatory correlates of summed similarity in categorization tasks.

Figure 6.

Neural summed similarity effect in the 4–9 Hz theta band for Experiments 2 & 3. Normalized oscillatory power as a function of binned summed similarity (defined in equation 1) for both targets (dashed lines) and lures (solid lines). Bins are defined as the lowest and highest 50% of the summed similarity distribution. Power is averaged over all frequencies in the 4–9 Hz theta band and in the channels within the parietal ROI (see Figure 4/5(b)), separately for the first half of the trials (“Early”) and the second half of the trials (“Late”). Error bars reflect standard error of the mean.

Figure 7.

Neural summed similarity effect for Experiments 2 & 3. Normalized oscillatory power shown as a function of binned summed similarity (defined in equation 1) for both targets (dashed lines) and lures (solid lines). Bins are defined as the lowest and highest 50% of the summed similarity distribution. Power is averaged over all frequencies in the 2–4 Hz delta band and in the channels within the frontal ROI (see Figure 4/5(b)) in the first half of the trial. Error bars reflect standard error of the mean.

Appendix A

Multidimensional representation of synthetic faces

Faces were constructed from four orthogonal dimensions as shown in Figure A1. Inspection of the figure, and of the measurement vector that makes up each of the four theoretical dimensions demonstrates that the first orthogonal dimension corresponds mostly to head width and mouth height. The second orthogonal dimension corresponds mostly to a hairline parameter and lip thickness. The third dimension is related to the curvature of the bottom of the face, and the fourth dimension reflects curvature of the hairline.

Figure A1.

Two-dimensional projection of the four-dimensional theoretical face space. Each face was constructed by moving a fixed distance (“distance-unit”), equal to the 75% correct discrimination threshold (just noticeable difference), in positive or negative directions away from the mean face. All possible permutations of positive and negative shifts for the four principal components of face space yielded a set of 16 faces. Each quadrant shows four faces that have the same coordinates for dimension 1 and 2, but differ in their coordinates for dimensions 3 and 4. For example, the faces in the right bottom quadrant are constructed by a positive distance-unit in the direction of dimensions 1 and a negative distance-unit in the direction of dimension 2. For each of the four faces, the coordinates in the 3rd and 4th dimension are different. The numbers below each face correspond to those in table A1.

The empirically derived MDS coordinates for this face set are shown in Table A1, and map onto slightly different sets of measurements. One of the reasons for this difference with the theoretically derived coordinates is that the empirically derived MDS coordinates were not Procrustes-transformed. We chose to not Procrustes-transform the empirically-derived MDS coordinates, such that we would not alter the data. The first dimension correlates highest with nose width (r(15) = 0.89, p < 0.001) and one of the components of the head shape (r(15) = 0.88, p < 0.01). The second dimension from the MDS study correlates highest with a hairline parameter (r(15) = 0.89, p < 0.001), as well as eye separation (r(15) = 0.85, p < 0.001). The third dimension correlates highest with thickness of the top lip (r(15) = 0.82, p < 0.001), as well a component of head shape (r(15) = –0.73, p < 0.01). Finally, MDS dimension four also correlates most strongly with thickness of the top lip (r(15) = –0.72, p < 0.01) and with head shape (r(15) = 0.70, p < 0.01).

This demonstrates that perception of these stimuli emphasizes specific aspects of the synthetic face stimuli (hairline, head shape and lips), which led us to use the empirically-derived MDS coordinates in our summed similarity analyses. This was further justified by the fact that we found very similar attentional weights on each of these four dimensions (see Methods).

Table A1.

Similarity coordinates of the face stimuli derived from our MDS study (see text).

face	dim 1	dim 2	dim 3	dim 4
1	0.901	0.706	1.060	-1.079
2	1.886	-0.683	-0.227	0.498
3	1.128	0.942	0.022	-1.137
4	1.284	-1.257	-0.231	0.891
5	-0.075	1.511	-1.029	-0.483
6	0.484	-0.390	-1.059	1.690
7	0.384	1.504	-0.951	0.552
8	0.268	-0.468	-0.106	2.199
9	-0.723	0.215	1.953	0.039
10	-0.248	-1.602	0.534	-0.860
11	-0.342	0.087	1.801	-0.789
12	0.402	-1.269	-0.481	-1.391
13	-1.298	1.292	0.925	0.165
14	-1.249	-0.599	-1.242	-0.532
15	-1.725	0.840	0.301	-0.444
16	-1.079	-0.828	-1.271	0.682

Appendix B

Complete results from the regression for Experiment 1

Table B1 gives the complete overview of normalized regression betas for Experiment 1.

Table B1.

Experiment 1. Normalized beta values for the regression coefficients.

	DLPFC		TC		MTL		Hippocampus		Parietal
model	T	L	T	L	T	L	T	L	T	L
S	0.021	0.044	0.008	0.041*	0.011	0.044*	0.032	0.060	0.022	0.032
RT	-0.003	0.007	-0.009	-0.005	-0.015	0.002	-0.013*	0.014	-0.012	0.002
S × F	-0.018	0.044*	-0.023*	-0.049*	-0.003	-0.023	-0.032	-0.040	-0.010	-0.025
S × T	-0.017	-0.060*	-0.008	-0.044*	-0.025*	-0.068*	-0.005	-0.038*	-0.015	-0.060*
S × F × T	0.023	0.069*	0.007	0.041*	0.017*	0.055*	0.003	0.041	0.016	0.066*

Note. Rows: regression coefficients (S= summed similarity; F = frequency band; RT = response time; × = interaction term).

Columns: brain regions: DLPFC = dorsolateral prefrontal cortex; TC = temporal cortex; MTL = medial temporal lobe.

Models are T (targets model) and L (lures model).

^*significant with a 0.05 False Discovery Rate (corresponding to a p-value threshold 0.011).

Bold face: significant for both targets and lures.