CNTRICS Imaging Biomarkers Final Task Selection: Long-Term Memory and Reinforcement Learning

Abstract

Functional imaging paradigms hold great promise as biomarkers for schizophrenia research as they can detect altered neural activity associated with the cognitive and emotional processing deficits that are so disabling to this patient population. In an attempt to identify the most promising functional imaging biomarkers for research on long-term memory (LTM), the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) initiative selected “item encoding and retrieval,” “relational encoding and retrieval,” and “reinforcement learning” as key LTM constructs to guide the nomination process. This manuscript reports on the outcome of the third CNTRICS biomarkers meeting in which nominated paradigms in each of these domains were discussed by a review panel to arrive at a consensus on which of the nominated paradigms could be recommended for immediate translational development. After briefly describing this decision process, information is presented from the nominating authors describing the 4 functional imaging paradigms that were selected for immediate development. In addition to describing the tasks, information is provided on cognitive and neural construct validity, sensitivity to behavioral or pharmacological manipulations, availability of animal models, psychometric characteristics, effects of schizophrenia, and avenues for future development.

Introduction

Biomarkers can be objectively measured and provide an index of a pathogenic process or a response to treatment (eg, hypertension and heart disease). In the second phase of the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) initiative (http://cntrics.ucdavis.edu),¹ functional imaging measures were considered as potential biomarkers for schizophrenia because they can detect altered neural activity associated with the cognitive and emotional processing deficits that are so disabling to this patient population. To identify promising imaging biomarkers, CNTRICS held a series of meetings and large-scale surveys, the results of which are summarized in a recent issue of Biological Psychiatry (Vol 70, No. 1). This manuscript focuses on long-term memory (LTM) biomarkers and summarizes results from the third CNTRICS meeting in which a review panel (see Acknowledgments) reached consensus on which nominated LTM biomarkers held the greatest promise for immediate translational development. The manuscript begins with a brief description of this decision-making process followed by descriptions of selected imaging biomarkers within each of 3 LTM domains; item encoding and retrieval, relational encoding and retrieval, and reinforcement learning. These descriptions are provided by coauthors who originally nominated selected paradigms. Coauthors were contacted after the review process was complete and, in most cases, did not participate in the review.

LTM is a multidimensional construct, and, as in earlier phases of the CNTRICS initiative,² it was agreed that 3 specific constructs should be used to guide the nomination process. The first construct is item encoding and retrieval—defined as “the processes involved in memory for individual stimuli or elements irrespective of contemporaneously presented context or elements.” For the item-encoding construct, 5 functional magnetic resonance imaging (fMRI) and 1 electroencephalogram (EEG) paradigms were nominated. Three of these fMRI paradigms—Acquired Equivalence,³ Cambridge Neuropsychological Test Automated Battery (CANTAB): Paired Associates Learning,⁴ and Hannula Face-Scene memory,⁵ were excluded because they measured relational memory without separate estimates of item memory. One fMRI (Item + Feature + Source task⁶) and one EEG paradigm (Item and Source Memory⁷) were recommended for further task development. Both paradigms were viewed favorably as they include separable measures of memory for individual items, memory for associated item features, and measures of the contexts in which the items were studied. However, neither paradigm provided a way to determine if participants were performing the encoding task correctly, which reviewers believed was important information to aid interpretation of any retrieval deficits. The fMRI paradigm recommended for immediate development was the Relational-and Item-Specific Encoding task (RISE⁸), described below.

Relational encoding and retrieval—defined as “the processes involved in memory for stimuli/elements and how they were associated with coincident context, stimuli, or events,” is the second LTM memory construct, for which 3 fMRI, 1 EEG, and 1 behavioral paradigm were nominated. Two of these fMRI paradigms, described below, were selected for immediate development—the RISE⁸ and the Hannula Face-Scene task.⁵ Of the remaining nominated tasks, the CANTAB: Paired Associates Learning task⁴ was not recommended because a functional imaging version of the task had not been developed. The Item + Feature + Source fMRI task⁶ and the Item and Source Memory EEG paradigm,⁷ were again recommended for further development, with recommendations to develop verifiable measures of encoding performance to ensure that participants are adequately engaged in the encoding task. The Acquired Equivalence fMRI task³ was also recommended for further task development. Group members were enthusiastic about the associative inference component of the task, which provides a close link with animal studies, but were concerned about the amount of over-learning required for subjects to acquire the item associations during the initial learning phase.

The final LTM construct is reinforcement learning—defined as “acquired behavior as a function of both positive and negative reinforcers including the ability to (a) associate previously neutral stimuli with value, as in Pavlovian conditioning; (b) rapidly modify behavior as a function of changing reinforcement contingencies and; (c) slowly integrate over multiple reinforcement experiences to determine probabilistically optimal behaviors in the long run.” Of the 4 nominated biomarker paradigms, the fMRI version of the probabilistic reversal learning task⁹ and the fMRI and EEG versions of the probabilistic selection (PS) task¹⁰ were selected for immediate translational development and are described below. The Corlett Associative Learning fMRI paradigm¹¹ was recommended for further task development because the reviewers were concerned that the complexity of the task scenario may be introducing additional cognitive demands that could affect construct validity.

Below are nominating authors’ discussion of selected tasks within each of the LTM domains. When available, information is also provided on cognitive and neural construct validity, sensitivity to behavioral or pharmacological manipulations, availability of animal models, psychometric characteristics, effects of schizophrenia, and avenues for future development.

Item Encoding and Retrieval and Relational Encoding and Retrieval RISE

Description.

The RISE was translated from a basic cognitive neuroscience fMRI paradigm¹² and designed to assess contributions of different encoding and retrieval processes to episodic memory in schizophrenia (figure 1). On item-specific encoding blocks, participants view pairs of common objects and are asked to rate whether either object is living or nonliving. On relational encoding blocks, participants are presented with pairs of objects and asked to judge whether one can fit inside the other. These encoding tasks orient the participants to use a specific type of processing—rather than leaving the approach up to the individual, and provide verifiable measures of encoding performance.

Fig. 1.

Illustration of item-specific and relational test procedures and task stimuli. (A) Memory encoding, (B) memory retrieval.

The RISE also assesses different aspects of retrieval by incorporating tests of item and associative recognition. During item recognition, participants complete an “old”/“new” recognition test consisting of all previously studied objects combined with an equal number of previously unseen foil objects. Next, participants receive an associative recognition test on objects previously studied during relational encoding. Test stimuli include either ‘intact’ pairs consisting of objects originally paired together on the same encoding trial and “rearranged” pairs consisting of objects originally studied on different trials. Subjects are asked to indicate if the pairs are intact or rearranged.

Construct Validity.

The encoding conditions examined on the RISE are based upon the cognitive psychology literature, which distinguishes between 2 different encoding strategy constructs. Common item-specific encoding strategies involve making a semantic decision about an item (eg, “pleasant,” “unpleasant,” “abstract”/“concrete”), whereas relational encoding strategies include imagining 2 or more items interacting or linking 2 or more items in the context of a sentence or story. It is thought that relational encoding promotes memory for associations among items, whereas item-specific encoding enhances the distinctiveness of specific items.¹³ Although both processes are effective, they tend to have different impacts on memory performance, with item-specific encoding optimal when source memory for item details is tested, whereas relational encoding is optimal when memory for associations between items is tested.¹³

Available data suggest that these 2 encoding processes can be dissociated at the neural level. For example, when encoding-related activity was examined on the original version of the task,¹² investigators found that activity in the dorsolateral prefrontal cortex (DLPFC) was higher during relational than during item-specific encoding and specifically predicted successful memory for associations among items. Although activity in the ventrolateral PFC (VLPFC) was also greater for relational encoding, this activity was nonspecific and predicted successful memory for item and associative recognition. This dissociation in DLPFC vs VLPFC encoding-related activity has been repeatedly demonstrated,¹³ leading investigators to conclude that the DLPFC may contribute to LTM through its role in active processing of relationships during encoding, whereas the VLPFC may have a more general role in promoting successful LTM formation.

The decision to include item and associative recognition tasks on the RISE was motivated by evidence that recognition performance can be supported by 2 dissociable processes; assessing familiarity of studied items or recollecting contextual details associated with study events.¹⁴ Recollection is a threshold process in which recognition is supported by retrieval of distinct aspects of the encoding event. In contrast, familiarity is a strength-based process in which recognition is informed by a signal-detection process in which item familiarity is used to discriminate studied from unstudied items. Although no retrieval task can purely dissociate these 2 processes, successful performance on the associative recognition task depends more strongly upon recollection because all items on the task are familiar, and recollection of some aspect of the encoding event is required to discriminate studied and rearranged pairs. In contrast, the item recognition task includes both studied and unstudied items, allowing participants to successfully discriminate new from old items based upon the familiarity strength of those items.

Imaging studies and human and animal lesion data clearly demonstrate that these 2 retrieval processes have dissociable neural representations within the medial temporal lobe.¹⁵ These studies demonstrate that hippocampal dysfunction is specifically associated with impaired recollection, whereas familiarity deficits are unrelated to hippocampal function. Conversely, activity in the perirhinal cortex is specifically associated with familiarity-based retrieval processes. Therefore, by interrogating these dissociable encoding and retrieval processes, the RISE paradigm provides the opportunity to identify the unique contribution of specific subregions in the prefrontal and medial temporal cortex to episodic memory.

Pharmacological and Behavioral Manipulation.

Not yet available

Animal Models.

Not yet available

Performance in Schizophrenia.

Prior research indicates that, against a background of generalized memory impairment, VLPFC control of item-specific encoding and retrieval appears to be less impaired than DLPFC control of relational encoding and retrieval in schizophrenia. Initial evidence of relative sparing of item-specific memory processes arose from several levels of processing (LOP) studies showing that when patients are required to make a semantic judgment about an item during encoding, they show normal LOP effects on item recognition accompanied by intact VLPFC recruitment, whereas DLPFC function remains impaired.¹⁶ In a recent behavioral study of the RISE paradigm,⁸ we found that the nature of the memory impairment in schizophrenia depended upon an interaction between encoding and retrieval processes. Under item-specific encoding conditions, patients were unimpaired in their ability to use familiarity to drive recognition performance. However, in that same item-encoding condition, their recollection was severely impaired and both familiarity and recollection-based retrieval was impaired when patients were required to perform the relational encoding task. These results are inconsistent with a generalized cognitive deficit explanation, in which patient impairments would be expected across encoding and retrieval conditions. Results are consistent with previous LOP studies in demonstrating the value of having patients engage in item-specific encoding and, moreover, suggest that patients benefit through increased facilitation of familiarity-based retrieval processes.

Psychometric Data.

In our initial behavioral study,⁸ internal reliability data on the RISE have been obtained on a sample of 93 patients with schizophrenia and 73 healthy controls. This sample did not include 1 control and 9 patients who were excluded for below-chance performance. Analyses reveal excellent internal consistency. For the item recognition task, intraclass correlation coefficients (ICC) are above .72 in both groups for item-specific and relational encoding. Associative recognition also shows acceptable levels of internal consistency for controls (ICC = .68) and patients with schizophrenia (ICC = .57). Because practice effects are a potential concern in memory study, we developed 3 parallel forms of the RISE, which show high levels of alternate forms reliability for item recognition (control r value = .71, patient r value = .72) and somewhat lower associative recognition reliability (control r value = .67, patient r value = .57). Examination of recognition accuracy (hit rate minus false alarm rate) revealed that both groups performed well above chance on item and associative recognition tasks negating concerns about floor effects. However, item recognition in controls was quite high but below ceiling.

Future Directions.

Immediate goals are to complete analysis of retest reliability of the behavioral version of the RISE and complete fMRI data collection with the imaging version of the task. This will complete initial psychometric analysis and task development phases and permit us to begin to study the effects of behavioral and pharmacological interventions on item-specific and relational memory in schizophrenia.

Relational Encoding and Retrieval

Face-Scene Memory Task

Description.

This paradigm entails a study phase, involving presentation of a set of face-scene pairings to be memorized, followed by a test phase in which relational memory for the pairings is assessed with displays consisting of 3 studied faces superimposed on a studied scene (figure 2).

Fig. 2.

Illustration of study and test trials used in the face-scene experiments.

In the original series of investigations,¹⁷ each study trial began with presentation of a scene (eg, a cityscape) for 3 second. A face was then superimposed on the scene, and the pair remained in view for 5 second. Participants were asked to commit 36 face-scene pairs to memory, each presented in 5 successive study blocks (ie, 5 study exposures). All faces and scenes were previously unfamiliar, and pairings were created arbitrarily. In the subsequent test phase, trials began with presentation of a studied scene (eg, the cityscape) for 3 second, providing a retrieval cue intended to reactivate memory of the relevant face-scene pair, followed by presentation of a 3-face test display superimposed on the scene for 10 second. Three different test displays were used: (1) match displays containing 3 studied faces, 1 of which had been paired with the scene during study; (2) re-pair displays containing 3 studied faces, none of which had been paired with the scene; and (3) novel displays containing 3 new (not studied) faces. For all conditions, faces were superimposed on a studied scene, and faces in any given display all had the same viewing histories (all 3 were seen in each study block or all 3 were never studied).

Relational memory for face-scene pairing is evaluated 2 ways. A direct or explicit measure of memory is taken from performance on a 3-alternative forced-choice recognition test, in which participants indicate which of the 3 faces in each test display had been studied with each of the scenes. An indirect or implicit measure of memory is derived from the tracking eye movements while participants view test displays. The proportion of overall viewing (in terms of number of fixations and viewing time) directed to the particular face studied with a given scene is assessed along with the time course of viewing directed to that face. Proportion of viewing time directed to the associated face is examined in successive time bins starting with 3-face display onset (eg, 0–250 ms, 250–500 ms). These data can determine when disproportionate viewing of the associated face first emerges.

This basic paradigm has been optimized for patient testing^17,18 and fMRI scanning⁵ by reducing the number of study exposures (from 5 to 3 or 1, respectively) and stimulus duration, which increases the number of incorrect trials to facilitate fMRI analysis.⁵ There were relatively few test trials per condition in the original version of the study; therefore, it is advised that the number of corresponding study-test blocks be increased as required by the demands of the study. Regardless of experimental manipulations in task instructions and number of study-test blocks,¹⁷ the task elicits a strong relational memory effect (ie, disproportionate viewing of matching faces) that can be observed in participants as young as 9-month-old.¹⁹

Construct Validity.

This paradigm is based upon the relational memory theory,¹⁵ which proposes distinct neural substrates for memory for individual items vs relations among items. According to this theory, the hippocampus is critical for relational memory.

A number of variants of the face-scene paradigm have been used to study relational memory, demonstrating effects of hippocampal damage¹⁷ and schizophrenia¹⁸ on relational memory, as well as evoked response potential (ERP) signatures²⁰ and hippocampal activity associated with relational memory performance.⁵ Amnesic patients with hippocampal damage were significantly impaired on behavioral and eye movement measures, showing no disproportionate viewing of the matching face.¹⁷ An ERP signature specific to relational vs item memory was observed as early as 270–350 msec after test face onset,²⁰ and hippocampal activity elicited by the scene cue predicted disproportionate viewing of matching faces irrespective of performance accuracy.⁵ These results are consistent with other findings of relational (but not item) memory impairment in hippocampal amnesia and findings that hippocampal activity is specific to relational vs item memory.²¹

Less is known about the role of the PFC, or other brain regions, in performance of this task, though fMRI results⁵ showed that activity differences in the DLPFC during presentation of the scene cue were correlated with correct explicit identification of the matching face. In addition, functional connectivity between DLPFC and the hippocampus was greater for correct than for incorrect trials when the 3-face test displays were presented. These results suggest the possibility that the hippocampus supports automatic, or obligatory retrieval of relational memory representations, and that explicit awareness of the retrieved content depends upon the recruitment of a broader cortical network involving the DLPFC. Additional experiments are required to examine this possibility directly.

Pharmacological and Behavioral Manipulation.

Not yet available

Animal Models.

Although we are not aware of any published reports of an animal model of the face-scene paradigm, we are optimistic that the paradigm can be successfully translated for use with nonhuman animals. Evidence for relational memory can be assessed in this paradigm indirectly via eye movement behavior without requiring any explicit instructions or overt verbal responses,²² permitting study in preverbal infants.¹⁹

Performance in Schizophrenia.

When the face-scene relational memory paradigm was applied to 35 individuals with schizophrenia and 35 healthy controls, explicit remembering was impaired, and disproportionate viewing of the matching face was significantly delayed and reduced in magnitude.¹⁸ This contrasted with performance on nonrelational nonmatch trials in which patients and controls viewed the 3 faces equally, suggesting that patients had a specific deficit in relational memory rather than a generalized cognitive deficit across all memory conditions. These results did not include data on 3 controls and 3 patients with schizophrenia who were unable to follow task instructions.

Psychometric Data.

Not yet available

Future Development.

Efforts are currently underway to examine whether and how differences in memory confidence relate to the expression of eye movement-based memory effects in this paradigm. Once basic behavioral investigations have been completed and the designs optimized, these paradigms will be translated for fMRI scanning and patient testing.

Reinforcement Learning

PS Task

Description.

The PS task¹⁰ measures participants’ ability to learn from positive and negative feedback, by integrating reinforcement probabilities over many trials. Three different stimulus pairs (AB, CD, and EF) are presented in random order, and participants have to learn to choose 1 of the 2 stimuli (figure 3). Feedback follows the choice to indicate whether it was correct or incorrect, but this feedback is probabilistic: In AB trials, a choice of stimulus A leads to positive feedback in 80% of trials, whereas a B choice leads to negative feedback in these trials. CD and EF pairs are less reliable: stimulus C is correct in 70% of trials, while E is correct in 60% of trials. Over the course of training, participants learn to choose stimuli A, C, and E more often than B, D, or F. Note that learning to choose A over B could be accomplished either by learning that choosing A leads to positive feedback or that choosing B leads to negative feedback (or both). To evaluate whether participants make choices guided primarily by their positive or negative history of reinforcement, performance is subsequently probed in a test/transfer phase in which all novel combinations of stimuli are presented, and no feedback is provided. “Go learning” is indicated by reliable choice of the most positive stimulus A in this test phase, when presented with other stimuli (AC, AD, AE, and AF). “NoGo” learning is indicated by reliable avoidance of the most negative stimulus B when presented with the same stimuli (BC, BD, BE, and BF). The extent to which a participant performs better in choose A or avoid B pairs is strongly related to dopaminergic state as indicated by disease, dopaminergic drug manipulation, and dopamine-related genetics.^10,23,24

Fig. 3.

The probabilistic selection task assesses the degree to which participants make choices based on learned positive or negative outcomes. During training, in each trial, participants are presented with one of the pairs shown on top (AB, CD, and EF) and select 1 of the 2 symbols. Feedback then indicates if the choice was correct or incorrect. The probabilities of each stimulus leading to positive feedback are indicated in the figure. Participants may learn to select the more positive stimuli A, C, and E by learning which symbol in each pair is associated with positive feedback (Go learning), which is associated with negative feedback (NoGo learning) or both. In the test phase, participants select between novel combinations of symbols without feedback. Reliable choice of the most positive symbol A over the other symbols (which on average have 50% value) is indicative of “Go learning.” Conversely, reliable avoidance of the most negative symbol B is indicative of “NoGo learning.” The test phase can also be used to assess how participants adjust their behavior (choice and reaction time) as a function of conflict in reinforcement values.

In addition to the probabilistic reinforcement learning effects, the task can also probe other aspects of reinforcement-based decision making. For example, the tendency to rapidly learn from a single instance of reinforcement in the initial trials of the task is thought to rely on distinct process from that involved in integrating feedback probabilities over trials.²⁴ Similarly, when faced with novel test pairs, participants adaptively modulate their response times to prevent premature (impulsive) responding in proportion to the degree of reinforcement conflict. High-conflict choices involving stimuli with similar reinforcement probabilities are associated with longer response times than those associated with divergent reinforcement probabilities, a process thought to depend on interactions between dorsomedial frontal cortex and the subthalamic nucleus.²⁵ Recent data indicate that conflict-related frontal EEG signals are predictive of these within-trial RT adjustments and that this brain-behavior relationship is reversed by subthalamic deep brain stimulation.²⁶

Construct Validity.

Performance in this task is defined by the ability to choose the probabilistically most optimal stimulus. Of course, many factors can contribute to better or worse performance aside from reinforcement learning, including attention, motivation, fatigue, working memory, etc. However, the main measure of interest in the task is within subject (ie, the ability to choose the most positive stimulus is contrasted with that of avoiding the most negative stimulus), thereby controlling for overall performance levels and specifically assessing the contribution of reinforcement. This relative positive to negative feedback learning measure is reliably altered by dopaminergic manipulation across a range of populations. Moreover, a pharmacological imaging experiment with this task revealed that dopaminergic stimulation magnifies striatal reward prediction error signals during learning, which are predictive of subsequent performance benefits in choosing the most rewarding stimulus “A.”²⁷ Similarly, electrophysiological responses to negative outcomes during learning are predictive of subsequent avoidance of the most negative stimulus “B.”²⁸ Together, these results indicate that individual differences in test phase performance are related to modulations of prior learning by dopamine; however, it is possible that dopaminergic manipulations can additionally induce a performance effect by inducing a reward-seeking bias during choice selection. Nevertheless, similar dopaminergic modulations of positive vs negative learning have been observed in other tasks meant to measure similar constructs but using different stimuli, motor responses, and task rules.^23,29

Positive and negative feedback learning in this task is thought to rely on striatal D1 and D2 receptors, respectively. As described above, probabilistic positive and negative feedback learning are sensitive to dopaminergic manipulation. Increases in dopaminergic stimulation, likely in the striatum, lead to better positive learning but cause impairments in negative feedback learning.²³ Conversely, dopamine depletion is associated with relatively better negative feedback learning but worse positive feedback learning. At the individual difference level, genes that control D1 and D2 dopamine function in the striatum are predictive of probabilistic positive and negative learning, whereas genes that control dopamine function in PFC are predictive of rapid trial-to-trial learning.²⁴ Similarly, frontal EEG measures associated with processing of negative feedback are predictive of trial-to-trial adjustments.²⁸ Finally, in response to mediofrontal signals of response conflict, the subthalamic nucleus (a component within the basal ganglia network) is thought to be required for response delay during high-conflict decisions. Supporting this claim, deep brain stimulation of the subthalamic nucleus causes premature responding for these high-conflict choices, reversing the relationship between mediofrontal measures of conflict and response time.^26,30

Pharmacological and Behavioral Manipulation.

As reviewed above, this task is sensitive to pharmacological manipulation. Dopamine agonists, including levodopa and D2 agonists, impair negative feedback learning in Parkinson’s patients, while sometimes improving positive feedback learning.^10,25 In ADHD, stimulant medications (methylphenidate and amphetamine), which elevate striatal dopamine, improved positive but not negative feedback learning.³⁰ In healthy participants, low doses of D2 agonists and antagonists, which may act presynaptically to modulate dopamine release, predictably alter positive and negative feedback learning.^23,27

Animal Models.

There is currently no available animal model of this specific task. However, Costa and colleagues³¹ have developed a forced-choice task requiring mice to learn to choose and avoid behaviors associated with positive and negative tastes. Mice with elevated striatal dopamine levels showed enhanced bias to approach rewarding tastes together with a reduced bias to avoid aversive tastes, similar to the data reported in humans. In monkeys, striatal D1 receptor blockade abolishes the normal response speeding observed when a large reward is available (a measure of Go learning), whereas D2 receptor blockade leads to greater response slowing when smaller than average rewards are available (a measure of NoGo learning).³²

Performance in Schizophrenia.

In a preliminary study, patients with schizophrenia showed large deficits in learning the standard version of the task, which uses Japanese Hiragana characters as stimuli.³³ However, fewer than 50% of patients were able to reach criteria on initial reinforcement learning, so a simplified version was developed. In this new version of the task using verbalizable stimuli (pictures of every day objects such as bicycles), 80% of patients and 86% of controls reached criteria during initial reinforcement learning. Nevertheless, patients showed deficits in early acquisition (thought to rely on prefrontal structures), which correlated with their negative symptoms.³³ In the test phase, patients showed intact “NoGo” learning but selectively impaired “Go” learning. The intact NoGo learning was inconsistent with a generalized cognitive deficit explanation. Furthermore, all the genetic polymorphisms predictive of learning in this task are candidate genes for schizophrenia.

Psychometric Data.

Practice effects have been assessed in Frank and O’Reilly.²³ Different stimuli are used across sessions. On average, participants are faster to learn the task after multiple sessions, but this practice does not systematically affect relative positive vs negative feedback learning.

Future Directions.

A more direct animal model of this task would be beneficial for understanding the precise mechanisms by which dopamine supports different aspects of reinforcement learning, as would further studies using functional neuroimaging and positron emission tomography (PET), of which several are in progress.

Probabilistic Reversal Learning

Description.

The task was developed by Trevor Robbins and Robert Rogers and first published in Lawrence et al³⁴ (An adapted version of this task was developed for use with fMRI by Cools et al.⁹ The primary difference between this and the original version of the task is that the fMRI version requires serial reversals.). On each trial, subjects are presented with 2 visual patterns (rectangles of colored stripes; figure 1), which appear in 2 randomly chosen boxes. Stage 1 is a simple probabilistic visual discrimination consisting of a 2-alternative forced choice between 2 colors. The “correct” stimulus (always the first stimulus touched) receives an 80:20 ratio of positive:negative feedback, with the opposite ratio of reinforcement given for the “incorrect” stimulus. After completing 40 simple discrimination trials in stage one, the reversal stage is administered (also consisting of 40 trials) in which contingencies are reversed without warning, so that the previously “incorrect” color is now correct and the previously correct color is now incorrect (figure 4).

Fig. 4.

Screen display of the probabilistic reversal learning task.

Although subjects receive all 80 trials, a learning criterion of 8 consecutive correct trials is required for data analysis. Primary performance measures are failure or success at each stage, mean errors to criterion, and mean latencies. Failure/success rates are analyzed using the likelihood-ratio method for contingency tables. Perseverative errors are scored both immediately following reversal of reinforcement contingencies and when performance during reversal blocks of 8 trials falls below chance (≤1 correct response). Maintenance errors are scored after the participant has reached criterion (ie, the number of responses to incorrect stimulus/total trials remaining), if there at least 10 trials remaining in the task.

Construct Validity.

Probabilistic learning tasks assess reinforcement learning and neural activity associated with prediction errors. For example, O’Doherty et al³⁵ used probabilistic learning to establish that activity in the (ventral) striatum and orbitofrontal cortex was positively correlated with the prediction error signal. The paradigm is optimized for detecting the electrical signature (feedback-related negativity) of reward prediction errors, regardless of behavioral adjustments, thereby allowing for the separate investigation of prediction errors and behavioral adjustment.

Human lesion and neuroimaging studies have strengthened the construct validity of the task by showing that it implicates neural systems associated with reinforcement learning (ie, the ventral striatum and ventral PFC). Reversal learning is disrupted by frontal lobe lesions,³⁶ specifically in the ventromedial vs DLPFC, and deficits can be specifically attributed to reversal problems rather than initial acquisition problems. Recently, Hornak et al³⁷ documented probabilistic reversal learning deficits in a small number of patients with DLPFC lesions, but posttest debriefing revealed that patients had failed to pay attention to crucial feedback. A subsequent study showed that patients with frontotemporal dementia had impaired probabilistic reversal learning task but intact performance on executive tasks associated with DLPFC function.³⁸

FMRI studies⁹ reveal a reliable pattern of activity in the VLPFC (bordering on the anterior insula), lateral orbitofrontal cortex, anterior cingulate cortex, posterior parietal cortex, and ventral striatum during final reversal errors relative to baseline correct responses. Activity in the VLPFC and ventral striatum is also larger during final reversal errors than during other (eg, probabilistic) errors that did not lead to switching problems.⁹

Sensitivity to dopaminergic manipulations further strengthens the task’s construct validity. Reversal-related activity in ventral striatum is abolished by dopaminergic medication in patients with Parkinson’s disease (PD)³⁹ and by the dopamine-enhancer methylphenidate.⁴⁰ Critically, this effect depended upon individual differences in the degree to which methylphedate potentiated dopamine release in the striatum, as measured with raclopride PET.⁴¹ Greater release was associated with greater impairment.

Pharmacological and Behavioral Manipulation.

Probabilistic reversal learning is sensitive to dopaminergic and serotoninergic manipulations but not to noradrenergic manipulations in humans. Withdrawal of dopaminergic medication, such as levodopa and dopamine receptor agonists in patients with mild PD improves task performance, presumably due to removal of a dopamine overdoses.⁴² These data are consistent with a study by Mehta et al,⁴³ showing that administration of the dopamine receptor agonist bromocriptine impaired performance in young healthy volunteers while improving spatial memory. Additional support for the overdose hypothesis came from a recent pharmacological fMRI study, which revealed that dopaminergic medication in mild PD patients abolished reversal-related activity in the ventral striatum (particularly in the nucleus accumbens³⁹).

Serotoninergic manipulation affects task performance in a qualitatively different way than dopaminergic manipulation. A large genetic study revealed dissociable effects of dopamine and the serotonin transporter polymorphisms (unpublished data), with the dopamine polymorphism affecting perseverative errors and the serotonin polymorphism increasing sensitivity to misleading punishment. This concurs with the pharmacological dopamine data described above, as well as a study by Chamberlain et al⁴⁴ who observed that acute administration of citalopram, a selective serotonin reuptake inhibitor, but not atomoxetine, a selective noradrenaline reuptake inhibitor, increased the number of switches after probabilistic errors (ie, misleading punishment). Thus, in contrast to dopamine, serotonin affects the processing of punishment irrespective of switching.

Animal Models.

There is a long history of work with experimental animals on reversal learning. When assessing convergence between animal and human studies, it is important to consider subtle differences in task design. First, most studies with experimental animals have used deterministic rather than probabilistic contingencies, so that animals never obtain “misleading” punishment or reward. A probabilistic version for rodents was successfully developed only recently.⁴⁵ Second, the nature of reward and punishment is qualitatively different, with reward constituting prolonged periods of access to juice or food in animals but (often) bonus points of positive feedback in humans. On the other hand, punishment may consist of periods of darkness in animals or simply reward omission in animals but bonus point loss or negative feedback in humans.

Nevertheless, there is remarkable convergence. Consistent with neuroimaging studies in humans, reversal learning in rodents has highlighted the importance of dopamine, serotonin, and the orbitofrontal cortex.^45,46 Furthermore, lesions of the orbitofrontal cortex in rodents and nonhuman primates induce a perseverative response tendency to the previously rewarded stimulus, reflecting persistent interference from a prepotent response.⁴⁶ Based on these and other data, investigators have concluded that the orbitofrontal cortex may indirectly facilitate flexibility in downstream regions (such as the amygdala) by signaling the expected value of outcomes rather than by directly inhibiting previously relevant responses.

In addition to the orbitofrontal cortex and the amygdala, experimental animal work has also implicated the striatum in reversal learning. Specifically, lesions of the caudate nucleus induced a perseverative response tendency during (object) reversal learning in monkeys.⁴⁷ Furthermore, work with both rodents and nonhuman primates indicate that lesions of nucleus accumbens also disrupt performance on (object and/or spatial) reversal learning tasks.⁴⁸ However, the impairment following nucleus accumbens lesions is generally not restricted to the reversal stages of the task but extends to initial acquisition stages, suggesting a more general role in the learning of stimulus-reinforcement contingencies rather than in reversal specifically.⁴⁸

Finally, psychopharmacological work with marmosets supports the evidence from human studies, reviewed above, that reversal learning is sensitive to serotoninergic manipulations. For example, Clarke et al⁴⁷ revealed that depletion of serotonin in the orbitofrontal cortex with the neurotoxin 5,7-DHT impairs reversal learning by increasing perseverative responding to the previously rewarded stimulus. It might be noted that the perseverative nature of the deterministic reversal deficit after 5,7-DHT lesions in marmosets is qualitatively different from the inappropriate switching seen after tryptophan depletion in humans. This discrepancy may reflect differences in tasks used in marmosets vs humans (probabilistic vs deterministic; emphasis on punishment) or, more likely, differences in the effect of the manipulation on the degree of serotonin depletion in the brain.

Performance in Schizophrenia.

Waltz and Gold⁴⁹ recently employed a modified version of the above-described probabilistic reversal learning task in 34 patients with schizophrenia and 26 controls. During initial learning about 80% of controls and 68% of patients successfully acquired all 3 reinforcement contingencies, with no group differences in initial acquisition. Although schizophrenia patients and controls performed similarly during initial learning of probabilistic contingencies, patients showed substantial learning impairments when reinforcement contingencies were reversed, achieving significantly fewer reversals. The ability of patients to acquire initial reinforcement contingencies suggests that the impaired reversal learning was not secondary to a generalized inability to perform the task. Similar results were obtained in a study by Weiler et al.⁵⁰

Psychometric Data.

Not yet available

Future Directions.

Psychometric data on practice effects, test retest reliability, and internal consistency should be obtained and reported. Decomposition of the paradigm into its constituent elements will allow analysis of the neurocognitive mechanisms underlying the effects of neurochemical manipulations and schizophrenia on reversal learning.

Funding

National Institute of Health (NIH) (R01MH084826, R01MH084895, R01 MH080066-01); National Institute of Mental Health Fellowship (F32MH075513). NIH, National Alliance of Research on Schizophrenia and Depression, and the Robert Wood Johnson Foundation (Dr J.D.R.); NIH (Dr N.J.C.); NIH, the Human Frontiers Science Program, the Dutch Brain Foundation, and the The Netherlands Organization for Scientific Research (NWO) (Dr R.C.); NIH and Michael J Fox Foundation (Dr M.J.F.); NIH (Dr D.E.H.); Dr C.R. has received research grants from the NIH and serves on the advisory board for Helicon Pharmaceuticals.