Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Apr 5.
Published in final edited form as: Neuroimage. 2019 Oct 21;206:116296. doi: 10.1016/j.neuroimage.2019.116296

Neural correlates underlying the effect of reward value on recognition memory

Blake L Elliott 1,*, Chris Blais 1, Samuel M McClure 1, Gene A Brewer 1
PMCID: PMC8979913  NIHMSID: NIHMS1784117  PMID: 31648002

Abstract

The prioritized encoding and retrieval of valuable information is an essential aspect of human memory. We used electroencephalography (EEG) to determine which of two hypothesized processes underlies the influence of reward value on episodic memory. One hypothesis is that value engages prefrontal executive control processes, so that valuable stimuli engage an elaborative rehearsal strategy that benefits memory. A second hypothesis is that value acts through the reward-related midbrain dopamine system to modulate synaptic plasticity in hippocampal and cortical efferents, thereby benefiting memory encoding. We used a value-directed recognition memory (VDR) paradigm in which participants encoded words assigned different point values and aimed to maximize the point value of subsequently recognized words. Subjective states of recollection (i.e., “remember”) and familiarity (i.e., “know”) were assessed at retrieval. Words assigned higher values at study were recognized more effectively than words assigned lower values, due to increased “remember” responses but no difference in “know” responses. Greater value was also associated with larger amplitudes of an EEG component at retrieval that indexes recollection (parietal old/new component), but had no relationship with a component that indexes familiarity (FN400 component). During encoding, we assessed a late frontal positivity (frontal slow wave, FSW) that has been related to elaborative rehearsal strategies and an early parietal component (P3) thought to index dopamine driven attention allocation. Our findings indicate that the effect of value on recognition memory is primarily driven by the dopamine-driven reward valuation system (P3) with no discernible effect on rehearsal processes (FSW).

It has long been shown that rewarding or important information is remembered better than non-rewarding or unimportant information (Heyer and O’kelly, 2010; Kahneman and Peavler, 1969; Loftus and Wickens, 1970; Weiner and Walker, 1966). The central nervous system has a finite capacity, and humans constantly encounter information far exceeding this capacity. For human memory to operate efficiently and effectively, important information must be selected, prioritized, and remembered over unimportant information (Broadbent, 1958; Cowan, 2000). It is hypothesized that the prioritized encoding of important information into long-term memory is essential for adaptive behavior. Previously encountered information that is important or rewarding bolsters future decision making (Shohamy and Adcock, 2010). The present study uses electroencephalography (EEG) to investigate neural mechanisms during the encoding and retrieval of rewarding information during a value-based recognition memory task.

A common paradigm used to evaluate the selection and prioritization of content into long-term memory is the value-directed remembering (VDR) paradigm. In a typical VDR experiment, participants study a list of words paired with different values for a future memory test. Value can be operationalized by assigning a number next to the word (i.e., “DEVIL 4”; Watkins and Bloom, 1999). Before the experiment begins, participants are instructed that they will earn the number of points corresponding to the value presented with the word if they are later able to remember the word, and that their goal is to maximize their total score. Unsurprisingly, previous findings from experiments using the VDR paradigm show that participants recall more words associated with higher point values than words with lower point values (Castel et al., 2002; Castel, 2008). Additionally, participants can recall the values associated with recalled words, indicating enriched memory encoding (Stefanidi et al., 2018).

A second variant of VDR experiments tests for recognition instead of the ability to recall and permits finer investigation of memory processes related to recollection and familiarity (Mandler, 1980; Wixted, 2007; Yonelinas, 2002). Dual-process models postulate that recollection and familiarity are distinct processes. Recollection affords conscious retrieval of associative information from the study episode, whereas familiarity-based judgments are more automatic assessments of stimulus familiarity (for review, see Yonelinas, 2002). One method for measuring processes related to recollection and familiarity is the “remember/know” paradigm (Tulving, 1985). This paradigm requires participants to make a subjective evaluation for each item that they discriminate in a recognition test.

Evidence from previous studies suggests that recollection and familiarity processes may be dissociated using event-related potentials (ERPs) during retrieval. The underlying processes are indexed by distinct temporal ERP components and scalp topographies. Familiarity is associated with a negative going component around 400 ms post-stimulus (FN400) that is maximal at mid-frontal electrodes. The FN400 has higher amplitude for correctly recognized familiar (or “know” in a remember/know paradigm) stimuli, correctly recognized stimuli studied under shallow encoding conditions, and correctly recognized stimuli in a speeded response task (Rugg and Curran, 2007). These effects increase with higher confidence ratings (Woodruff et al., 2006). Recollection, by contrast, is associated with a later-occurring positive ERP component (500–800 ms) that is maximal over parietal regions of the scalp. This ERP component is termed the “parietal old/new effect” and is greater for stimuli designated “remember” in a remember/know paradigm, for accurate source judgments, and for stimuli studied under deeper encoding conditions (Addante et al., 2012; Duzel et al., 1997; Rugg and Curran, 2007).

Prior literature using VDR recognition memory paradigms typically find that the value-driven gain in memory performance is specific to recollection. A previous study by Adcock et al. (2006) found that the value-driven gain in recognition memory performance was specific to pooled remember and high confidence responses, with no effect of value on low confidence responses. Additionally, event-related fMRI data during the encoding period found increased activity in reward processing regions (i.e., the ventral tegmental area and nucleus accumbens) as well as regions important for memory encoding (i.e., the hippocampus). These findings has been extended to show that value preferentially affects recollection and recruits the reward system in incidental encoding paradigms (Wittmann et al., 2005), intentional encoding paradigms (Wolosin et al., 2012), and also for encoded stimuli where correct recognition avoids punishment (Shigemune et al., 2014). These studies suggest that the dopaminergic reward system is one mechanism which may underlie the value-driven gain to recollective memories.

An alternative hypothesis is that value affects memory via top down executive control processes. This hypothesis posits that after recognizing a stimulus is valuable, a participant may selectively engage an elaborative strategy that produces deeper semantic processing. To investigate the neural underpinnings of value-driven encoding, Cohen and colleagues (Cohen et al., 2014) conducted a study using fMRI during a VDR free recall paradigm. They found that greater differences in activation of brain regions associated with semantic processing (i.e., left inferior frontal gyrus and left posterior lateral temporal cortex) correlated with individual differences in how strongly value affected memory (i.e. the selectivity index, SI). The authors concluded that frontotemporal semantic processing regions that support elaborative rehearsal may be an important mechanism for value-modulated memory encoding. Additionally, recent research using divided attention at encoding during a VDR recognition task indicated that executive resources at encoding are necessary for VDR effects on recollection (Elliott and Brewer, 2019). Finally, individuals with higher working memory capacity are more likely to implement selective encoding strategies during a VDR free recall paradigm (Robison and Unsworth, 2017). These studies suggest that, perhaps in addition to the dopaminergic reward system, executive resources that support elaborative encoding strategies may be another mechanism important for VDR effects on memory.

Because of its temporal resolution, EEG is an invaluable tool to dissociate latent cognitive processes. Two ERP components during recognition memory encoding have been associated with better recognition memory performance, but different encoding processes. The first of these components is an early mid-parietal P3 component, for which greater amplitude has been associated with increased attention allocation, stimulus valuation and reward processing (Polich, 2007; Pfabigan et al., 2014; Sato et al., 2005; Walsh and Anderson, 2012). A second later component, the frontal slow wave (FSW), has been associated with elaborative rehearsal strategies and executive resources (Fabiani et al., 1990; Mangels et al., 2001). These two components are thought to have distinct neural generators. The P3 component has been hypothesized to be generated from the dopamine system (Polich, 2007). The FSW has been hypothesized to be generated by the prefrontal cortex (Mangels et al., 2001). Given prior literature separately implicating these neural systems and manifested cognitive processes in VDR effects, the P3 and FSW components are likely candidates to be modulated by value during memory encoding.

We used a VDR remember-know recognition memory paradigm and EEG to investigate how encoding and retrieval processes are affected by value. Given previous literature, we hypothesized the effect of value on recognition memory (measured with hit rates) to be localized to remember responses, with no difference in know responses. Accordingly, ERPs at retrieval should reflect this dichotomy. Value should selectively enhance the parietal old/new component that indexes recollection, with no effect on the FN400 that indexes familiarity. To the extent that value affects encoding processes, we investigated two distinct components, the P3 component and the late frontal slow wave. If the effect of value is ascribed to increased attention allocation and reward processing, one would expect value to modulate the earlier P3 component. If the effect of value is ascribed to increased elaborative rehearsal processes, one would expect value to modulate the late frontal slow wave component. Further, if one or both of these components index selective memory encoding, the extent to which value modulates these components should be correlated to behavioral measures of participants’ sensitivity to value (as measured by the selectivity index, SI). That is, participants who showed a greater effect of value on ERP correlates at encoding should also show enhanced sensitivity to value during the behavioral test. Altogether, this experiment was designed to understand the effect of value on recognition memory and the underlying electrophysiological signatures.

1. Methods

1.1. Participants

33 participants were recruited from the Arizona State University research participation pool and took part in what they were told was an EEG study on memory. Three participants were excluded for faulty equipment, and two were excluded for not following task directions, resulting in a final sample size of 28 participants. No formal power analysis was conducted. Sample size was chosen based upon the number of participants used in a previous study which utilized the same paradigm (see Elliott and Brewer, 2019; Experiment 1). These methods were similar and the effect sizes reported in that study were large (Cohen’s ds from 0.90 to 1.03). A post-hoc sensitivity analysis revealed that with our sample size (n = 28), adequate power (0.80), traditional alpha value (0.05), correlation among repeated measures (0.6), and nonsphericity correction (1) we had sufficient power in the current study to detect a minimal effect size of f = 0.202 which is a medium effect size.

1.2. Materials and design

The procedures here are the same as Elliott and Brewer (2019), with the exception of EEG recording during encoding and retrieval. For clarity and consistency across papers we largely reproduce their description here. Stimuli were 400 nouns selected from the Toronto noun pool (Friendly et al., 1982). Each participant completed five study-test blocks as follows. The study phases consisted of 40 words each randomly assigned a point value (1, 3, 7, or 9; ten of each specific value). The test phases consisted of 80 words (40 from the most recent list and 40 new nouns, randomly intermixed) presented one at a time without point values. Participants were asked to classify these old and new items at test and to make a judgment on their subjective state of recollection (i.e., “remember”) and familiarity (i.e., “know”).

1.3. Procedure

All experiments followed a study protocol that was approved by Arizona State University’s Institutional Review Board. Written informed consent was obtained from each participant before beginning the study. Participants were instructed that they would be completing multiple study-test blocks in a recognition memory task. The participants were told that their objective was to remember as many words as possible, with the goal of maximizing their score on each recognition memory test. They studied lists of 40 randomly selected nouns presented sequentially for 2 s, with a randomly jittered inter-stimulus interval from 300 to 500 ms in 17 ms increments to decorrelate evoked activity across stimulus presentations. The nouns were randomly paired with an integer indicating that item’s value (1, 3, 7, or 9) with a total of ten of each value in each study block. Participants were told they would earn the point value previously paired with the word if correctly recognized – regardless of confidence rating – and that they would lose 1 point for incorrectly identifying a new word as old.

Each word appeared in the center of the computer screen below a central fixation cross, with the paired value appearing simultaneously above the central fixation cross in the center of the screen. After studying each list of 40 words, a test phase followed with 80 words (with no point values presented) that included all 40 of the words from the previous list and 40 new words randomly intermixed. Test words were presented one at a time and participants judged whether they thought the word was old or new. Participants responded using a standard computer keyboard with the following response option assignments: Z “Definitely New”—left pinky, X “Maybe New”—left ring finger, C “Maybe Know”—left middle finger, V “Definitely Know”—left index finger, and M “Remember”—right index finger. The participants were told to respond with their first instinct, but the test was untimed.

Before the experiment began, participants were briefed on the difference between remembering and knowing with the following instructions (adapted from Herzmann and Curran, 2011):

Make a remember judgment if you not only remember the word, but also consciously remember the experience of studying the word. For example, perhaps you remember the specific value of the word, something else that happened in the room while you were studying it (like a cough or sneeze), an association that came to mind, or what came just before or after the word in the study phase. To give you a real world example, imagine you are walking across campus and recognize someone, but cannot recall their name or where you have met them. You are certain you have seen this person before, but do not remember anything specifically about them or where you met them. This is would be “knowing.” If you recognize this person and remember that it is John whom you met in Biology class, this would be “remembering.”

If participants could not recollect the word but were certain they saw it on the study list, they were asked to use the “definitely know” response, and to use the “maybe know” response if they thought they had studied the word but were not very sure. They were instructed to use the “maybe new” response if they think the word was new but were not certain, and to use the “definitely new” response if they were certain the word was new.

1.4. Behavioral data analysis

Raw hits, “remember” responses, and “know” responses were all calculated as proportions of total responses in that category divided by total number of studied items. “Know” responses included both “definitely know” and “maybe know” responses. All three measures were also corrected for false alarms (FA) by subtracting the false alarm rates from the hit rates within each category. An individual difference measure of a participant’s sensitivity to value was calculated, the selectivity index (SI, Castel et al., 2002). The selectivity index ranges from ‒1 (minimum possible score for number of items recognized) to 1 (maximum possible score for number of items recognized). The SI is calculated as follows:

Selectivity Index=subject's score−chance scoreideal score−chance score

Since VDR effects are typically localized to “remember” responses, the SI was calculated for total hits, as well as for “remember” and “know” responses separately. Brain-behavior correlation analyses were conducted between these measures and the value modulated ERP components at encoding and retrieval. We hypothesize that if value modulates any ERP encoding or retrieval components, the extent to which a participant showed a value-modulated effect should be correlated with the behavioral measure of sensitivity to value at retrieval (SI). Further, if the benefit of value on behavioral responses is localized to “remember” responses, this correlation should be strongest for the SI “remember” responses and no correlation should be observed for the SI “know” responses.

1.5. EEG recording and analyses

The EEG was recorded using a 32-channel cap with the average of the left and right mastoid electrodes as a reference on a NeuroScan SynampsRT system at 1000 Hz band-pass filtered from DC to 400 Hz. Offline data analyses were performed using EEGLAB (Delorme and Makeig, 2004). The data were down-sampled offline to 250 Hz and band-pass filtered from 1 to 30 Hz, using an infinite impulse response (IIR) Butterworth filter, and submitted to a GPU-optimized version of the infomax independent component analysis (ICA; Raimondo et al., 2012) procedure in EEGLAB. The ocular components in the ICA were identified using visual inspection (independently performed and then compared between two researchers) and then removed from the unfiltered raw data. The ICA procedure subsumed 1.1% of the data. The raw 1000 Hz data without ocular artifacts were then filtered to 0.1–30 Hz using an IIR Butterworth filter. Data were time-locked to the onset of the word and epoched from ‒200 to 2000 ms at encoding, ‒200 to 1000 ms at retrieval, and baseline corrected to the first 200 ms pre-stimulus. A minimum of 20 trials per condition was ensured for each subject (mean = 48, max = 50, for all encoding conditions; mean = 35, max = 49, for all retrieval conditions). A moving window 60 ms wide, moving in increments of 20 ms across the epoch, to detect peak-to-peak voltage differences exceeding 80 μV across any channel was used to identify excess electrical noise. If four or fewer electrodes exceeded this threshold, those electrodes were removed and approximated using spherical interpolation. Otherwise, the trial was removed. This resulted in the exclusion of 1.0% (range: 0–10.2%) of correct trials across participants.

From the grand mean waveform two retrieval components were analyzed using mean amplitude. The first component was an early frontal negativity measured between 300 and 500 ms after stimulus onset (the FN400 component). The second component is a late posterior positivity measured between 500 and 800 ms after stimulus onset (the parietal old/new component). Amplitudes were analyzed at the electrodes where activity has been shown to be maximal in previous literature (FZ for the FN400, CP3 for the parietal old/new). Both of these components were analyzed with separate repeated measures ANOVA on the correctly identified words (hits) as a function of value (1, 3, 7, 9). Linear trend analysis was used to determine if the amplitude of these components increased as a function of value.

At encoding, two components were analyzed from the grand mean waveform using mean amplitude. The first component was an early posterior positivity measured between 450 and 650 ms after stimulus onset (the P3 component). The second component is a late sustained frontal positivity measured between 1000 and 2000 ms after stimulus onset (the frontal slow wave component, FSW). Amplitudes were analyzed at the posterior and frontal midline electrodes for both components (FZ for the FSW component, PZ for the P3 component). Both of these components were analyzed with separate one-way repeated measures ANOVA on all studied trials across all levels of value (1, 3, 7, 9). Linear trend analysis was used to determine if the amplitude of these components increased as a function of value. To account for the relative spacing of the values chosen in our experiment, we used the contrast coefficients [−4 −2 2 4].

An individual difference measure of the effect of value on the ERP components at encoding and retrieval was calculated by computing a linear contrast of the ERP waveforms for each value (i.e., (‒4) × “1” (‒2) × “3” + (2) × “7” + (4) × “9”). This approach gave us an electrophysiological measure of the extent to which each individual’s ERP component scaled with encoding value. This measure was then correlated with each participant’s behavioral measure of sensitivity to value (i.e., the selectivity index, SI).

2. Results

2.1. Behavioral results

Memory performance as a function of value and response type (Remember, Know) is summarized in Fig. 1. The behavioral results demonstrate greater recognition accuracy for higher valued words, regardless of the type of recognition judgment. A one-way repeated measures ANOVA with corrections on hit rates across all levels of value (1, 3, 7, 9) revealed a main effect of value (F(1,27) = 16.284, p < 0.001, ηp2=0.38) and a significant linear trend (F(1,27) = 21.977, p < 0.001, ηp2=0.45). When hit rates were conditionalized on the subjective state of awareness supporting those decisions, the value-driven gain in memory performance was due to an increase in remember responses, with no effect on know responses. A 4 (Value: 1, 3, 7, 9) × 2 (Response Type: remember, know) repeated measures ANOVA with appropriate Greenhouse-Geisser corrections revealed a main effect of value (F(1,27) = 16.284, p < 0.001, ηp2=0.38), no main effect of response type (F(1,27) = 0.099, p = 0.755, ηp2=0.004), and a significant interaction between value and response type (F(1,27) = 16.830, p < 0.001, ηp2=0.38). A linear trend analysis revealed a significant interaction of value and response type (F(1,27) = 23.254, p < 0.001, ηp2=0.46), indicating that only accuracy of remember responses increased as a function of value (Fig. 1).

Fig. 1.

Fig. 1.

Open in a new tab

Proportion of correctly recognized items given remember and know judgments as a function of value. Error bars represent 95 percent confidence interval, dashed line represents the total false alarm rate.

2.2. Electrophysiological results

2.2.1. Retrieval ERPs

Fig. 2 illustrates the grand average ERP waveform elicited by correctly recognized old items from electrode FZ, as well as the associated scalp topographies from the measurement window of interest (300–500 ms post-stimulus) as a function of value. Mean amplitude from electrode FZ between 300 and 500 ms post-stimulus (the FN400 component) was entered into a one-way repeated measures ANOVA on value (1, 3, 7, and 9). The results revealed no main effect of value (F(1,27) = 0.229, p = 0.876, ηp2=0.008) and no linear trend (F(1,27) = 0.025, p = 0.874, ηp2=0.001). The data suggests that value had no effect on the FN400 component, confirming our hypothesis. However, the frequentist statistical approach limits our ability to render evidence for the null hypothesis. Therefore, we specified two Bayesian models and examined relative evidence for the null to the alternative hypotheses that FN400 effect was modulated by value (using default prior r scale on fixed effects = 0.5). The Bayes factor provided strong evidence for a null effect of value on the FN400 (BF01 = 10.30).

Fig. 2.

Fig. 2.

Open in a new tab

Electrophysiological results for the FN400 component during retrieval. A) Grand average ERPs from electrode FZ as a function of value. B) Mean amplitude scalp topographies from 300 to 500 ms as a function of value. C) Linear contrast ERP from electrode FZ, error bars represent 95 percent confidence interval. D) Mean amplitude from 300 to 500 ms from electrode FZ as a function of value.

Fig. 3 illustrates the grand average ERP waveform elicited by correctly recognized old items from electrode CP3, as well as the associated scalp topographies from the measurement window of interest (500–800 ms post-stimulus) as a function of value. Mean amplitude from electrode CP3 between 500 and 800 ms post-stimulus (the parietal old/new component) was entered into a one-way repeated measures ANOVA on value (1, 3, 7, and 9). The results revealed no main effect of value (F(1,27) = 1.639, p = 0.187, ηp2=0.06). Importantly, the data reveal a significant linear trend (F(1,27) = 4.518, p < 0.05, ηp2=0.14). The data suggests that the parietal old/new component increased in amplitude as a function of value, confirming our hypothesis.

Fig. 3.

Fig. 3.

Open in a new tab

Electrophysiological results for the parietal old/new component during retrieval. A) Grand average ERPs from electrode CP3 as a function of value. B) Mean amplitude scalp topographies from 300 to 500 ms as a function of value. C) Linear contrast ERP from electrode CP3, error bars represent 95 percent confidence interval. D) Mean amplitude from 300 to 500 ms from electrode FZ as a function of value.

To test whether the value-modulated parietal old/new component at retrieval predicted behavioral measures of a participant’s sensitivity to value in the recognition test, we examined the correlation between the selectivity index (calculated for total hit rates, “remember” and “know” responses) and amplitude of a linear contrast derived from the ERP waveform (Fig. 4). The correlation indicated that the participants who showed the greatest effect of value on the parietal old/new component also showed the most sensitivity to value in their hit rates (r (27) = 0.39, p < 0.05). This correlation was strongest in the SI for the “remember” responses (r (27) = 0.57, p = 0.002) with no correlation observed in the SI for the “know” (r (27) = ‒0.27, p = 0.159).

Fig. 4.

Fig. 4.

Open in a new tab

Correlations between the effect of value on the parietal old/new component at retrieval (measured by the linear contrast amplitude) and the selectivity index. A) Scalp topography of the linear contrast from 500 to 800 ms. B) Correlation between the selectivity index for total hit rates and the linear contrast amplitude from electrode CP3. C) Correlation between the selectivity index for “remember” responses and the linear contrast amplitude from electrode CP3. D) Correlation between the selectivity index for “know” responses and the linear contrast amplitude from electrode CP3. Note: * denotes p < 0.05, ** denotes p < 0.01, two tailed.

2.2.2. Encoding ERPs

Fig. 5 illustrates the grand average ERP waveform elicited by all encoding trials from electrode PZ, as well as the associated scalp topographies from the measurement window of interest (450–650 ms post-stimulus) as a function of value. Mean amplitude from electrode PZ between 450 and 650 ms post-stimulus (the P3 component) was entered into a one-way repeated measures ANOVA on value (1, 3, 7, and 9). The results revealed a main effect of value (F(1,27) = 3.232, p < 0.05, ηp2=0.11). Importantly, the data reveal a significant linear trend (F(1,27) = 8.525, p < 0.01, ηp2=0.24). Further, repeated measures, two-tailed permutation test based on the tmax statistic (Blair and Karniski, 1993) was conducted to verify this finding (see the Supplement). The data suggests that the P3 component increased in amplitude as a function of value.

Fig. 5.

Fig. 5.

Open in a new tab

Electrophysiological results for the P3 component during encoding. A) Grand average ERPs from electrode PZ as a function of value. B) Mean amplitude scalp topographies from 450 to 650 ms as a function of value. C) Linear contrast ERP from electrode PZ, error bars represent 95 percent confidence interval. D) Mean amplitude from 450 to 650 ms from electrode PZ as a function of value.

Fig. 6 illustrates the grand average ERP waveform elicited by all encoding trials from electrode FZ, as well as the associated scalp topographies from the measurement window of interest (1000–2000 ms post-stimulus) as a function of value. Mean amplitude from electrode FZ between 1000 and 2000 ms post-stimulus (the FSW component) was entered into a one-way repeated measures ANOVA on value (1, 3, 7, and 9). The results revealed no main effect of value (F(1,27) = 0.633, p = 0.596, ηp2=0.02) and no linear trend (F = 1.381, p = 0.250, ηp2=0.05). The data suggests that value had no effect on the FSW component. As noted, the frequentist statistical approach does not provide evidence for the null hypothesis. Therefore, we applied the same Bayesian approach we did for the null effect observed in the ERP at retrieval with the FSW component (using default prior r scale on fixed effects = 0.5). The Bayes factor provided very strong evidence for a null effect of value on the FSW (BF01 = 16.16).

Fig. 6.

Fig. 6.

Open in a new tab

Electrophysiological results for the FSW component during encoding. A) Grand average ERPs from electrode FZ as a function of value. B) Mean amplitude scalp topographies from 1000 to 2000 ms as a function of value. C) Linear contrast ERP from electrode PZ, error bars represent 95 percent confidence interval. D) Mean amplitude from 1000 to 2000 ms from electrode FZ as a function of value.

To test whether the value-modulated P3 component at encoding predicted subsequent behavioral measures of a participant’s sensitivity to value in the recognition test, we examined the correlation between the selectivity index (calculated for total hit rates, “remember” and “know” responses) and amplitude of a linear contrast derived from the ERP waveform (Fig. 7). The correlation indicated that the participants who showed the greatest effect of value on the P3 component also showed the most sensitivity to value in their hit rates (r (27) = 0.40, p < 0.05). This correlation was strongest in the SI for the “remember” responses (r (27) = 0.54, p = 0.003) with no correlation observed in the SI for the “know” (r (27) = ‒0.24, p = 0.222).

Fig. 7.

Fig. 7.

Open in a new tab

Correlations between the effect of value on the P3 component at encoding (measured by the linear contrast amplitude) and the selectivity index. A) Scalp topography of the linear contrast from 450 to 650 ms. B) Correlation between the selectivity index for total hit rates and the linear contrast amplitude from electrode PZ. C) Correlation between the selectivity index for “remember” responses and the linear contrast amplitude from electrode PZ. D) Correlation between the selectivity index for “know” responses and the linear contrast amplitude from electrode PZ. Note: * denotes p < 0.05, ** denotes p < 0.01, two tailed.

3. Discussion

Four primary findings emerged in the current study. First, the effect of value-based encoding increased recognition memory performance for higher valued information. Second, this improvement in recognition memory for higher valued information was localized to enhancement of subjective states of recollection (i.e., “remembering”) with no significant changes in subjective states of familiarity (i.e., “knowing”). Third, ERP correlates at retrieval of recollection and familiarity dovetail with the behavioral dissociation in subjective states of awareness. The parietal old/new component (thought to index recollection) showed greater amplitude for higher-valued information, with no effect of value on the FN400 component (thought to index familiarity). Fourth and finally, ERP correlates at encoding indicated that the P3 component, thought to index dopamine driven reward processing or attention allocation, scaled linearly with participants’ sensitivity to value expressed in the subsequent recognition test (i.e., their selectivity index). Conversely, the FSW component, thought to index prefrontal executive encoding processes was not sensitive to value.

These findings are consistent with previous research suggesting that value at encoding selectively enhances strong, recollective memories. Replicating prior studies (Gruber and Otten, 2010; Gruber et al., 2016; Hennessee et al., 2017; Cohen et al., 2017; Elliott and Brewer, 2019), value-directed encoding led to more recollective experiences being reported at retrieval with no influence on familiarity. ERPs have provided critical evidence for dual process models of recognition memory. Two temporally and topographically distinct ERP correlates have been identified that dissociate recollection and familiarity: the parietal old/new component and the mid-frontal FN400 component. These components have been dissociated by manipulations that selectively modulate recollection and familiarity. This study highlights that value-directed encoding is an additional independent variable that can be used to dissociate these processes and their respective ERP components.

The central finding of this study is that value modulated an early P3 component during memory encoding, with no effect on a later frontal sustained positivity (FSW). The FSW component has been hypothesized to index central executive processes and elaborative encoding (Mangels et al., 2001). Memory paradigms where the to-be-remembered stimuli are explicitly studied under elaborative, associative strategies demonstrate greater FSW amplitudes during encoding (Fabiani et al., 1990; Weyerts et al., 1997). Previous research has suggested that this may be one mechanism by which individuals prioritize and encode valuable information (Cohen et al., 2014; Cohen et al., 2016). We failed to find evidence that value affected this component during encoding. One interpretation of the null result observed here is that elaborative rehearsal and executive resources may be utilized less during VDR recognition memory tasks than during VDR free recall tasks. Further, individual differences may exist in the recruitment of these strategic and automatic resources (Robison and Unsworth, 2017). Future studies should address these hypotheses and directly compare the basis of VDR related improvements to episodic memory in recognition and recall tasks.

The P3 component has been classically associated with the odd-ball paradigm (Donchin et al., 1978; Pritchard, 1981). From odd-ball studies, the P3 has been argued to index attention-driven comparison processes and context-updating in working memory (Donchin, 1981; Donchin and Coles, 1988). The P3 amplitude is sensitive to amount of resources engaged during dual-task performance, and therefore may reflect attentional resource allocation (Isreal et al., 1980; Kramer et al., 1985; Wickens et al., 1983). The P3 amplitude has also been shown to increase with reward magnitude, perhaps reflecting downstream effects of dopaminergic reward processing (Sato et al., 2005; Walsh and Anderson, 2012). This latter claim is supported by observations that the P3 component may be generated from the dopamine system, and that individual differences in VTA and NAc BOLD signals during a reward task correlate with P3 amplitude (Pfabigan et al., 2014; Polich, 2007).

The P3 component has been well studied in classic memory paradigms such as free recall and recognition. In these studies, larger P3 amplitude is associated with better memory performance. However, this is only seen for words studied using rote rehearsal strategies. Words studied using elaborative strategies elicited larger frontal slow wave components (Fabiani et al., 1986; Fabiani et al., 1990; Karis et al., 1984). Although these studies have associated greater P3 amplitudes with rote rehearsal strategies, it is unlikely that increased rote rehearsal for higher-valued items explains the effects in the current study. Rote rehearsal strategies typically lead to enhancements on familiarity and not recollection (Yonelinas, 2002). Our results indicated an effect on recollection alone, and thus do not comport with this account.

A recent study conducted by Elliott and Brewer (2019) used a similar paradigm as the current study, with the exception of implementing various divided attention tasks during encoding. We reported that the typical VDR effect was specific to remember responses. Divided attention conditions that blocked articulatory processes did little to change the VDR effect, but divided attention manipulations that blocked executive processes eliminated the VDR effect. Although the results suggested that executive resources were necessary for VDR effects on recollection, it was impossible to conclude whether disrupting executive resources with a secondary task disrupted selective elaborative rehearsal or earlier attentional processes. This was the motivation for the current ERP study. Interpreting our EEG results through this lens, we suspect that executive processes are necessary to trigger reward responses that enhance encoding but are not necessary for ongoing elaborative processing of words to enhance memory strength.

The fact that the amplitude of the P3 component scaled with value fits with the recent theorizing that this component indexes a dopamine-driven computation implemented when encoding higher-valued information. Individual differences in the size of the effect of value on this component was predictive of behavioral measure of how sensitive participants were to value, the selectivity index. This finding bolsters the argument that the P3 component indexes cognitive processes utilized when encoding higher valued information. While the poor spatial resolution of EEG limits our ability to identify underlying subcortical neural generators, the aforementioned studies have provided circumstantial evidence that the P3 component may be at least partially driven by the dopamine system (Pfabigan et al., 2014; Polich, 2007). Additionally, similar VDR recognition memory tasks using fMRI typically find activation in dopaminergic reward processing regions (i.e. the ventral tegmental area and the nucleus accumbens) and memory regions (i.e. the hippocampus) when encoding higher-valued stimuli (Adcock et al., 2006; Shigemune et al., 2014). These findings provide evidence that the mesolimbic reward system may drive value-directed encoding (via dopaminergic midbrain-hippocampal projections, Lisman and Grace, 2005; Shohamy and Adcock, 2010). However, future studies will be needed to conclude that dopaminergic midbrain-hippocampal projections underlie the P3 effect observed here.

Another ERP component which has been associated with value-coding is the feedback related negativity (FRN, also called the reward positivity; Sambrook and Goslin, 2014; San Martín, 2012; Walsh and Anderson, 2012). The FRN is most prominent over electrode FCZ. In our data, a prominent negative component was evident between 350 and 450 ms post-stimulus during encoding. We analyzed data from this location and time period using one-way repeated measures ANOVA on value, as in our main analyses. We found no effect of value on this negative component (see supplementary material). This null result may indicate that value is differentially encoded depending on the task at hand and may have different underlying neural generators. Most tasks that study the FRN involve some sort of decision-making, and not memory formation and retrieval as we investigated. This possibility of multiple neural sources for value encoding is supported by the anatomy of the dopamine system. The dopamine system has heterogenous functional circuits that support different aspects of behavior, including memory formation (associated with midbrain – hippocampal circuitry; Lisman and Grace, 2005), reward learning, decision making, and cognitive control (associated with cortico-striatal circuitry; Montague et al., 2006; Schultz et al., 1997; McClure et al., 2003; Braver and Cohen, 2000; see Haber and Knutson, 2009 for review).

Mental processes are notorious for their surprisingly limited capacity. Capacity limits necessitate selection of to-be remembered information in the environment. Important information must be selected and prioritized and less important information must be attenuated. The current study expands our understanding of these mechanisms. The behavioral and retrieval ERP data demonstrate that value at encoding enhances memory for higher-valued items, and that effect was specific to recollection. Consistent with the effect on behavior, our data showed selective enhancement of a P3 component at encoding, with no effect on a later frontal slow wave component. Based on prior literature, this result may reflect preferential attention allocation arising from midbrain dopaminergic signaling. Value certainly enhances memory, but it does so in a specific manner and not by augmenting all cognitive processes that improve subsequent recall.

Supplementary Material

Supplement

Acknowledgements

BE was supported National Science Foundation Graduate Research Fellowship Program (GRFP) during the completion of this work. GB was supported by the Army Research Institute during the completion of this work (grant number W911NF-17–1-0175). SM was supported by the National Science Foundation during the completion of this work (grant number 1634179).

Footnotes

Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.neuroimage.2019.116296.

References

  1. Adcock RA, Thangavel A, Whitfield-Gabrieli S, Knutson B, Gabrieli JDE, 2006. Reward-motivated learning: mesolimbic activation precedes memory formation. Neuron 50 (3), 507–517. 10.1016/j.neuron.2006.03.036. [DOI] [PubMed] [Google Scholar]
  2. Addante RJ, Ranganath C, Yonelinas AP, 2012. Examining ERP correlates of recognition memory: evidence of accurate source recognition without recollection. Neuroimage 62 (1), 439–450. 10.1016/j.neuroimage.2012.04.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Blair RC, Karniski W, 1993. An alternative method for significance testing of waveform difference potentials. Psychophysiology 30 (5), 518–524. [DOI] [PubMed] [Google Scholar]
  4. Braver T, Cohen J, 2000. On the control of control: the role of dopamine in regulating prefrontal function and working memory. Contr. Cogn. Process.: Atten. Perform XVIII, 713–736. 10.1016/S0165-0173(03)00143-7. [DOI] [Google Scholar]
  5. Broadbent DE, 1958. Perception and Communication Pergamon Press, London. 10.1037/10037-000. [DOI] [Google Scholar]
  6. Castel AD, 2008. The adaptive and strategic use of memory by older adults: evaluative processing and value-directed remembering. In: Benjamin AS, Ross BH (Eds.), The Psychology of Learning and Motivation, vol 48. Academic Press, San Diego, CA, pp. 225–270. [Google Scholar]
  7. Castel AD, Benjamin A, Craik F, Watkins M, 2002. Aging and selectivity in memory. Mem. Cogn 30 (7), 1078–1085. [DOI] [PubMed] [Google Scholar]
  8. Cohen MS, Rissman J, Suthana NA, Castel AD, Knowlton BJ, 2014. Value-based modulation of memory encoding involves strategic engagement of fronto-temporal semantic processing regions. Cognit. Affect Behav. Neurosci 14, 578–592. 10.3758/s13415-014-0275-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Cohen MS, Rissman J, Suthana NA, Castel AD, Knowlton BJ, 2016. Effects of aging on value-directed modulation of semantic network activity during verbal learning. Neuroimage 125, 1046–1062. 10.1016/j.neuroimage.2015.07.079. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Cohen MS, Rissman J, Hovhannisyan M, Castel AD, Knowlton BJ, 2017. Free recall test experience potentiates strategy-driven effects of value on memory. J. Exp. Psychol. Learn. Mem. Cogn 43 (6), 1–21. 10.1037/xlm0000395. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Cowan N, 2000. Processing limits of selective attention and working memory: potential implications for interpreting. Interpreting 5 (2), 117–146. 10.1075/intp.5.2.05cow. [DOI] [Google Scholar]
  12. Delorme A, Makeig S, 2004. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134 (1), 9–21. 10.1016/j.jneumeth.2003.10.009. [DOI] [PubMed] [Google Scholar]
  13. Donchin E, 1981. Surprise!. Surprise? Psychophysiology 18, 493–513. [DOI] [PubMed] [Google Scholar]
  14. Donchin E, Coles MGH, 1988. Is the P300 component a manifestation of context updating? Behav. Brain Sci 11 (3), 357–374. 10.1017/S0140525X00058027. [DOI] [Google Scholar]
  15. Donchin E, Ritter W, Mccallum WC, 1978. The Endogenous Components of the ERP. Event-Related Brain Potentials in Man
  16. Duzel E, Yonelinas AP, Mangun GR, Heinze HJ, Tulving E, 1997. Event-related brain potential correlates of two states of conscious awareness in memory. Proc. Natl. Acad. Sci. U.S.A 94, 5973–5978. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Elliott BL, Brewer GA, 2019. Divided attention selectively impairs value-directed encoding. Collabra: Psychology 5 (1), 4. 10.1525/collabra.156. [DOI] [Google Scholar]
  18. Fabiani M, Karis D, Donchin E, 1986. P300 and recall in an incidental memory paradigm. Psychophysiology 23, 298–308. [DOI] [PubMed] [Google Scholar]
  19. Fabiani M, Karis D, Donchin E, 1990. Effects of mnemonic strategy manipulation in a Von Restorff paradigm. Electroencephalogr. Clin. Neurophysiol 75 (2), 22–35. 10.1016/0013-4694(90)90149-E. [DOI] [PubMed] [Google Scholar]
  20. Friendly M, Franklin PE, Hoffman D, Rubin DC, 1982. The Toronto Word Pool: norms for imagery, concreteness, orthographic variables, and grammatical usage for 1,080 words. Behav. Res. Methods Instrum 14, 375–399. [Google Scholar]
  21. Gruber MJ, Otten LJ, 2010. Voluntary control over prestimulus activity related to encoding. J. Neurosci 30 (29), 9793–9800. 10.1523/JNEUROSCI.0915-10.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Gruber MJ, Ritchey M, Wang SF, Doss MK, Ranganath C, 2016. Post-learning hippocampal dynamics promote preferential retention of rewarding events. Neuron 89 (5), 1110–1120. 10.1016/j.neuron.2016.01.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Haber SN, Knutson B, 2009. The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 35 (10), 4–26. 10.1038/npp.2009.129. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Hennessee JP, Castel AD, Knowlton BJ, 2017. Recognizing what matters: value improves recognition by selectively enhancing recollection. J. Mem. Lang 94, 195–205. 10.1016/j.jml.2016.12.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Herzmann G, Curran T, 2011. Experts’ memory: an ERP study of perceptual expertise effects on encoding and recognition. Mem. Cogn 39 (3), 412–432. 10.3758/s13421-010-0036-1. [DOI] [PubMed] [Google Scholar]
  26. Heyer AW, O’kelly LI, 2010. Studies in motivation and retention: II. Retention of nonsense syllables learned under different degrees of motivation. J. Psychol 27 (1), 143–152. 10.1080/00223980.1949.9915983. [DOI] [PubMed] [Google Scholar]
  27. Isreal JB, Chesney GL, Wickens CD, Donchin E, 1980. P300 and tracking difficulty: evidence for multiple resources in dual-task performance. Psychophysiology 10.1111/j.1469-8986.1980.tb00146.x. [DOI] [PubMed]
  28. Kahneman D, Peavler WS, 1969. Incentive effects and pupillary changes in association learning. J. Exp. Psychol 79 (2 PART 1), 312–318. 10.1037/h0026912. [DOI] [PubMed] [Google Scholar]
  29. Karis D, Fabiani M, Donchin E, 1984. P300 and memory—individual differences in the von Restorff effect. Cogn. Psychol 16, 177–216. [Google Scholar]
  30. Kramer AF, Wickens CD, Donchin E, 1985. Processing of stimulus properties: evidence for dual-task integrality. J. Exp. Psychol. Hum. Percept. Perform 11 (4), 393–408. 10.1037/0096-1523.11.4.393. [DOI] [PubMed] [Google Scholar]
  31. Lisman JE, Grace AA, 2005. The hippocampal-VTA loop: controlling the entry of information into long-term memory. Neuron 46 (5), 703–713. 10.1016/j.neuron.2005.05.002. [DOI] [PubMed] [Google Scholar]
  32. Loftus GR, Wickens TD, 1970. Effect of incentive on storage and retrieval processes. J. Exp. Psychol. Gen 85, 141–147. [Google Scholar]
  33. Mandler G, 1980. Recognizing: the judgment of previous occurrence. J. Psychol. Rev. Psychol. Rev 87 (3), 252–271. 10.1037/0033-295X.87.3.252. [DOI] [Google Scholar]
  34. Mangels J.a., Picton TW, Craik FIM, 2001. Attention and successful episodic encoding: an event-related potential study. Cogn. Brain Res 11 (1), 77–95. 10.1016/S0926-6410(00)00066-5. [DOI] [PubMed] [Google Scholar]
  35. McClure SM, Daw ND, Read Montague P, 2003. A computational substrate for incentive salience. Trends Neurosci 26 (8), 423–428. 10.1016/S0166-2236(03)00177-2. [DOI] [PubMed] [Google Scholar]
  36. Montague PR, King-Casas B, Cohen JD, 2006. Imaging valuation models in human choice. Annu. Rev. Neurosci 29 (1), 417–448. 10.1146/annurev.neuro.29.051605.112903. [DOI] [PubMed] [Google Scholar]
  37. Pfabigan DM, Seidel EM, Sladky R, Hahn A, Paul K, Grahl A, et al. , 2014. P300 amplitude variation is related to ventral striatum BOLD response during gain and loss anticipation: an EEG and fMRI experiment. Neuroimage 96, 12–21. 10.1016/j.neuroimage.2014.03.077. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Polich J, 2007. Updating P300: an integrative theory of P3a and P3b. Clin. Neurophysiol 118 (10), 2128–2148. 10.1016/j.clinph.2007.04.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Pritchard WS, 1981. Psychophysiology of P300. Psychol. Bull 89 (3), 506–540. 10.1037/0033-2909.89.3.506. [DOI] [PubMed] [Google Scholar]
  40. Raimondo F, Kamienkowski JE, Sigman M, Fernandez Slezak D, 2012. CUDAICA: GPU optimization of infomax-ICA EEG analysis. Comput. Intell. Neurosci 1–8. 10.1155/2012/206972, 2012. [DOI] [PMC free article] [PubMed]
  41. Robison MK, Unsworth N, 2017. Working memory capacity, strategic allocation of study time, and value-directed remembering. J. Mem. Lang 93, 231–244. 10.1016/j.jml.2016.10.007. [DOI] [Google Scholar]
  42. Rugg MD, Curran T, 2007. Event-related potentials and recognition memory. Trends Cogn. Sci 11 (6), 251–257. 10.1016/j.tics.2007.04.004. [DOI] [PubMed] [Google Scholar]
  43. Sambrook TD, Goslin J, 2014. Mediofrontal event-related potentials in response to positive, negative and unsigned prediction errors. Neuropsychologia 61 (1), 1–10. 10.1016/j.neuropsychologia.2014.06.004. [DOI] [PubMed] [Google Scholar]
  44. San Martín R, 2012. Event-related potential studies of outcome processing and feedback-guided learning. Front. Hum. Neurosci 6 (November), 1–17. 10.3389/fnhum.2012.00304. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Sato A, Yasuda A, Ohira H, Miyawaki K, Nishikawa M, Kumano H, Kuboki T, 2005. Effects of value and reward magnitude on feedback negativity and P300. Neuroreport 16 (4), 407–411. 10.1097/00001756-200503150-00020. [DOI] [PubMed] [Google Scholar]
  46. Schultz W, Dayan P, Montague PR, 1997. A neural substrate of prediction and reward. Science 275 (5306), 1593–1599. 10.1126/science.275.5306.1593. [DOI] [PubMed] [Google Scholar]
  47. Shigemune Y, Tsukiura T, Kambara T, Kawashima R, 2014. Remembering with gains and losses: effects of monetary reward and punishment on successful encoding activation of source memories. Cerebr. Cortex 24 (5), 1319–1331. 10.1093/cercor/bhs415. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Shohamy D, Adcock RA, 2010. Dopamine and adaptive memory. Trends Cogn. Sci 14 (10), 464–472. 10.1016/j.tics.2010.08.002. [DOI] [PubMed] [Google Scholar]
  49. Stefanidi A, Ellis DM, Brewer GA, 2018. Free recall dynamics in value-directed remembering. J. Mem. Lang 100, 18–31. 10.1016/j.jml.2017.11.004. [DOI] [Google Scholar]
  50. Tulving E, 1985. Memory and consciousness. Can. Psychol./Psychologie Canadienne 26 (1), 1–12. 10.1037/h0080017. [DOI] [Google Scholar]
  51. Walsh MM, Anderson JR, 2012. Learning from experience: event-related potential correlates of reward processing, neural adaptation, and behavioral choice. Neurosci. Biobehav. Rev 36 (8), 1870–1884. 10.1016/j.neubiorev.2012.05.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Watkins MJ, Bloom LC, 1999. Selectivity in Memory: an Exploration of Willful Control over the Remembering Process (Unpublished manuscript).
  53. Weiner B, Walker EL, 1966. Motivational factors in short-term retention. J. Exp. Psychol 71 (2), 190–193. 10.1037/h0022848. [DOI] [PubMed] [Google Scholar]
  54. Weyerts H, Tendolkar I, Smid HGOM, Heinze HJ, 1997. ERPs to encoding and recognition in two different inter-item association tasks. Neuroreport 8 (7), 1583–1588. 10.1097/00001756-199705060-00007. [DOI] [PubMed] [Google Scholar]
  55. Wickens C, Kramer A, Vanasse L, Donchin E, 1983. Performance of concurrent tasks: a psychophysiological analysis of the reciprocity of information-processing resources. Science 221, 1080–1082. [DOI] [PubMed] [Google Scholar]
  56. Wittmann BC, Schott BH, Guderian S, Frey JU, Heinze HJ, Duzel E, 2005. Reward-related fMRI activation of dopaminergic midbrain is associated with enhanced hippocampus-dependent long-term memory formation. Neuron 45 (3), 459–467. 10.1016/j.neuron.2005.01.010. [DOI] [PubMed] [Google Scholar]
  57. Wixted JT, 2007. Dual-process theory and signal- detection theory of recognition memory. Psychol. Rev 114 (1), 152–176. 10.1037/0033-295X.114.1.152v. [DOI] [PubMed] [Google Scholar]
  58. Wolosin SM, Zeithamova D, Preston AR, 2012. Reward modulation of hippocampal subfield activation during successful associative encoding and retrieval. J. Cogn. Neurosci 24 (7), 1532–1547. 10.1162/jocn_a_00237. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Woodruff CC, Hayama HR, Rugg MD, 2006. Electrophysiological dissociation of the neural correlates of recollection and familiarity. Brain Res 1100 (1), 125–135. 10.1016/j.brainres.2006.05.019. [DOI] [PubMed] [Google Scholar]
  60. Yonelinas AP, 2002. The nature of recollection and familiarity: a review of 30 Years of research. J. Mem. Lang 46 (3), 441–517. 10.1006/jmla.2002.2864. [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement
NIHMS1784117-supplement-Supplement.pdf (366.9KB, pdf)

RESOURCES