The neural basis of reversal learning: An updated perspective

Abstract

Reversal learning paradigms are among the most widely used tests of cognitive flexibility and have been used as assays, across species, for altered cognitive processes in a host of neuropsychiatric conditions. Based on recent studies in humans, non-human primates, and rodents, the notion that reversal learning tasks primarily measure response inhibition, has been revised. In this review, we describe how cognitive flexibility is measured by reversal learning and discuss new definitions of the construct validity of the task that are serving as an heuristic to guide future research in this field. We also provide an update on the available evidence implicating certain cortical and subcortical brain regions in the mediation of reversal learning, and an overview of the principle neurotransmitter systems involved.

Introduction

Cognitive flexibility, the ability to rapidly change behavior in the face of changing circumstances, is disrupted in many psychiatric and neurological disorders. Determining the neural basis of cognitive flexibility is therefore important for understanding the pathophysiology of these disorders and potentially developing treatments. To study the neural substrates of cognitive flexibility in rodents, nonhuman primates, and humans, researchers have often used a set of paradigms collectively referred to as reversal learning. Across species, these paradigms are subtly different, but importantly they all assess cognitive flexibility by evaluating adaptive responding in the face of changing stimulus-outcome (S-O) or response-outcome (R-O) contingencies.

Over the years, reversal learning has become a pre-eminent test of cognitive flexibility and has been used to characterize altered cognitive processes in a host of neuropsychiatric disorders, including substance abuse, obsessive compulsive disorder, psychopathy, Parkinson’s disease, schizophrenia, and to assess cognition at certain developmental time periods such as adolescence (Swainson et al., 2000, Remijnse et al., 2006, Finger et al., 2008, Brigman et al., 2009, Leeson et al., 2009, van der Schaaf et al., 2011, Izquierdo and Jentsch, 2012). Despite its long history of use, reversal learning continues to be an essential experimental paradigm for assessing cognitive function. Indeed, recent years have seen a precipitous rise in the number of published studies using reversal learning, with almost equal focus on rodent, monkey and human subjects (Figure 1).

Figure 1. Publications of reports on reversal learning in rodent, monkey, and human subjects.

Pubmed search terms “reversal learning” from 1950–2014. The early-to-mid 2000’s witnessed the steepest rise in the number of publications on reversal learning. Reversal learning continues to be a widely-used paradigm for assessing cognitive function, with an almost equal focus on rodent, monkey and human subjects.

While the literature on the neural basis of reversal and the interpretation of findings using this task have been reviewed elsewhere (Clark et al., 2004, Izquierdo and Jentsch, 2012, Costa et al., 2015, Hamilton and Brigman, 2015, Wassum and Izquierdo, 2015), our aim here is to: 1) consider how reversal can be measured and compare different versions of the paradigm across species; 2) provide an updated perspective on the construct validity of reversal learning paradigms; 3) discuss current thinking on the major neural circuits mediating the ability to flexibly change behavior; and 4) review the neurochemical modulation of the cognitive processes engaged during reversal learning.

Reversal learning paradigms across species

In the classic reversal learning paradigm used in humans (Fellows and Farah, 2003a), monkeys (Butter, 1969) and rodents (Schoenbaum et al., 2000), subjects are trained to discriminate between two visual stimuli or spatial locations, one of which is rewarded every time it is chosen and the other which is not. After successful discrimination learning has been demonstrated by reaching a criterion level of performance, the outcomes associated with the two stimuli are reversed and subjects are again trained until they meet a performance criterion. Note, that while in this review we focus on instrumental, appetitive forms of reversal learning, Pavlovian associations can also be reversed and outcomes can also be aversive (Morris and Dolan, 2004, Burke et al., 2009).

An advantage of reversal learning paradigms is that they can be employed in multiple species and, as such, can have significant translational value for understanding the neural bases of cognitive flexibility. Reflecting this, many of the key findings concerning the neural mechanisms of reversal learning have been replicated across rodent, non-human primate and human subjects, as discussed later in this review. We briefly review some of the common procedures employed to test reversal learning in different species noting the differences in paradigms, but also their similarities.

For reversal learning tasks in rodents, behavioral apparatus are often outfitted with either two levers, nosepoke portals or a touch-sensitive screen. Mazes are also commonly used to test spatial discriminations and reversals (Jentsch and Taylor, 2001, Bannerman et al., 2003, Palencia and Ragozzino, 2004). With mazes, levers and portals, reversal may be performed solely using information about spatial location or incorporate the use of visual or auditory cues (Neill et al., 2001, Widholm et al., 2003, Boulougouris et al., 2007, Castañé et al., 2010). When a touchscreen is used, a wider variety of visual stimuli become available and spatial and egocentric strategies better controlled for (Izquierdo et al., 2006, Mar et al., 2013, Graybeal et al., 2014). In nonhuman primates, modified versions of the Wisconsin General Testing Apparatus (WGTA) have been used to test reversal learning (Jones and Mishkin, 1972, Stern and Passingham, 1995, Izquierdo et al., 2004a). In one paradigm, an opaque screen is lowered while one of two food wells is baited with a reward. When the screen is raised the monkey is tasked with displacing one of the objects to reveal the reward. Alternatively, animals can be presented with visual stimuli on cards or a touchscreen (Crofts et al., 1999, Clarke et al., 2005, Walker et al., 2009). With either method, selection of the correct stimulus results in delivery or access to a reward. Thus, the paradigms used in both rodents and monkeys are very similar. However, where reversal learning tasks differ between species is in the number of reversals typically completed by the animals. In the standard WGTA version of the task for macaque monkeys, subjects often complete more than seven serial reversals (Izquierdo et al., 2004b), but more recent versions of the paradigm deliver multiple reversals in a single session. In marmoset monkeys, subjects usually complete approximately four reversals across multiple sessions, whereas in rodents, only a single reversal is often completed (c.f. Schoenbaum et al., 2002).

There are also a range of approaches to assaying reversal learning in human volunteers and patient populations, including via the presentation of visual stimuli on a screen that can be responded to with a screen touch or keyboard stroke (Lawrence et al., 1999, Swainson et al., 2000, Cools et al., 2002, Hvoslef-Eide et al., 2015). Reward in these experiments involves a simple notification that a response was correct or the accumulation of simulated or actual monetary compensation (Cools et al., 2002, Fellows and Farah, 2003b). Thus, a difference with most reversal tasks in non-human animals is that rewards are almost always secondary, conditioned reinforcers rather than primary reinforcers such as food or drugs, and can often entail delayed access and a mental representation of reward accrual.

Across species and platforms, the relationship between stimuli and outcomes can in principle be either fully (deterministic) or partially predictive (probabilistic) (Bari et al., 2010a, Ineichen et al., 2012, Dalton et al., 2014, Rygula et al., 2014). Indeed, S-O outcomes can be reversed at the mid-point of sessions with differing probabilistic reinforcement schedules (e.g., 80% rewarded or 60% rewarded) as a way to modulate the subject’s (in this case, a monkey’s) anticipation of upcoming reversals (Walton et al., 2010, Costa et al., 2015). Reversal paradigms in humans are almost always probabilistic, and probabilistic associations can occur numerous times during a single session, in some cases in fewer than five trials (Rolls et al., 1994a, Hampton et al., 2007). Probabilistic tasks slow the rate of learning (which can be fast in human subjects) and helps to reduce the use of simple strategies (as reviewed later) and allows for contrasting of ‘true errors’ and ‘probabilistic errors’ (Cools et al., 2002). Other methodological factors challenge the notion that there are preserved features of reversal learning across species. One such factor is sensory modality of learning. Learning in some modalities (e.g., involving odor or digging-medium discriminanda) is easily acquired in rodents, whereas visual discrimination paradigms are learned more readily in primates. The differences in length of training that result from these species- specific propensities could potentially engage different neural circuitry, thereby complicating translation. Nonethless, despite some differences in the paradigms, there is more convergence and comparability of methods for testing reversal learning across species than disagreement (Figure 2).

Figure 2. Reversal learning assessments using touchscreens in rodent, monkey, and human subjects.

Despite some differences in the paradigms, there is more convergence and comparability of methods for testing reversal learning across species than disagreement. Shown is an example of the touchscreen platform across species. In rats and monkeys the relationship between stimuli and outcomes are either fully (deterministic) or partially predictive (probabilistic), whereas reversal paradigms in humans are almost always probabilistic.

What does reversal learning measure?

Reversal learning requires a subject to flexibly adjust their behavior when the reward-related contingencies that they have previously learned are reversed. For some time, a widely-voiced idea was that reversal leaning paradigms primarily measured inhibitory control of responding (Jones and Mishkin, 1972). Based on experiments in humans, monkeys, and rodents (Fellows and Farah, 2005, Chudasama et al., 2007, Schoenbaum et al., 2009a), this view has been revisited and other tasks have been designed to directly test more specific aspects of cognition related to attention and motor inhibition (for example, stop signal task, 5-choice serial reaction time task). It appears that the brain regions underlying performance on these various tasks are overlapping with those implicated in reversal learning, though there are also functional dissociations (Jentsch et al., 2014).

In the paragraphs below, we review current thinking that reversal learning paradigms require dynamic reward representations including an abstraction of accrued reward and an expectation (belief) of the possibility for change. We discuss how, to perform optimally, the subject’s most adaptive strategy is rule use (similar to a “learning set,” described below) combined with reward learning, and not simply inhibitory (motor response) control. These conclusions have arisen in large measure from important refinements and variants of reversal learning tasks that we will also detail. For instance, in addition to general measures such as the number of correct responses and errors the subject makes after each switch in reward contingencies, experimenters can use consecutive (‘correction’) errors to assess a failure to disengage from ongoing behavior (perseverative responses), monitor performance throughout different stages of learning (e.g., below vs. above chance performance, early, middle and/or late stages) (Jones and Mishkin, 1972, Chudasama and Robbins, 2003, Izquierdo et al., 2006, Brigman et al., 2008, Graybeal et al., 2011, DePoy et al., 2013), and study the microstructure of learning derived from trial-by-trial responses to positive and negative feedback (Rudebeck and Murray, 2008, Brigman et al., 2013, Izquierdo et al., 2013, Stolyarova et al., 2014, Klanker et al., 2015). These measures assess how subjects are learning from the feedback that they received on each trial, and therefore assume that reversal learning is solved by a single reward learning process that integrates reward and non-reward experiences across trials to drive performance. As an example, increased omissions during reversal learning can index a failure to overcome non-reward experiences (Tait and Brown, 2007).

Variations in task-design which allow for different ways of assessing the reward learning process including administering serial reversals, in which the subject learns to perform multiple reversals, for example via repeated stimulus-outcome (S-O) switching (Boulougouris et al., 2007, Castañé et al., 2010, Kosaki and Watanabe, 2012), concurrent reversals - where subjects learn multiple pairs of stimuli - some of which are then reversed and some not (Wilson and Gaffan, 2008), and introducing an additional option for choice. In the latter variant, when a three- or four-choice reversal procedure is used, it becomes easier to distinguish between regressive errors, perseverative errors, and new learning errors (Ragozzino et al., 2003, Kim and Ragozzino, 2005, Lee et al., 2007, Ragozzino and Rozman, 2007, Rudebeck et al., 2008, Seu et al., 2009, Kosaki and Watanabe, 2012, Riceberg and Shapiro, 2012). Increasing the option space beyond two possibilities also has the potential for decoding exploitative versus explorative strategies, consistent with optimal foraging, in the service of maximizing reward procurement.

Evidence from both human and monkey studies shows that strategies or rules are also adopted during reversal learning and in some cases can come to dominate responding (Murray and Gaffan, 2006). Indeed this is the main reason why probabilistic learning tasks are predominantly used with human subjects (Rolls et al., 1994a, Hampton et al., 2007) and increasingly in monkeys (Walton et al., 2010, Costa et al., 2015, Rygula et al., 2015) where, as noted above, learning can proceed very (too) quickly. Probabilistic learning paradigms, especially those with more than two options, reduce the ability of subjects to apply a simple strategy, such as win-stay/lose-shift, as they have to use an integrated history of choices and outcomes to determine the best stimulus to choose.

Probabilistic (or bandit) tasks require the subject to choose from a set of options with unknown reward rates that do not depend on choice history. Probabilistic reversal paradigms also have another advantage: they are more amenable to the application of reinforcement learning (RL) models that can estimate parameters for reward decisions, providing another window into cognitive flexibility (Sutton and Barto, 1998). This is especially true for probabilistic tasks because RL models estimate choice behavior based on an integrated history of previous reward and non-reward experience - a critical feature when “correct” responses are not always rewarded. Within the basic RL model equation, different free parameters such as the learning rate, how quickly values are updated after each trial, and the inverse temperature parameter, also known as the exploration parameter (Katahira, 2015) influence how choice outcomes are integrated into option values (i.e., learning rate) and then used to drive reward-guided behavior. These parameters can be fit to the data and compared across groups, helping to further dissociate the processes involved in reversal learning (Costa et al., 2015). Ultimately, the application of these RL models allows an even more fine-grained understanding of what drives subjects’ choices during reversal learning.

Within the past decade, empirical work on reinforcement learning has been greatly influenced by computational theories (Daw and Doya, 2006), and this is likely to continue. One key feature of the theoretical work related to reversal learning is model-free vs. model-based learning (Doya, 1999, Daw et al., 2005, Dayan and Berridge, 2014). Model-free learning does not require a representation, or map, of how to respond or even an understanding of the identity of the end state, and is most associated with acquiring information about the world on a trial-by-trial basis, experiencing stimuli and their outcomes (as described above). There is certainly one aspect of reversal learning that incorporates this, and as we review in the Neural Substrates section, subcortical regions such as striatum and amygdala may help to generate a reward history from these experiences. By contrast, model-based learning incorporates choices guided by a map or abstract cognitive-type structure that the subject can refer to in guiding their choices. This type of learning may also be engaged in reversal learning, and is associated with cortical function. Both model-free and model-based systems may be online during behavior but one could dominate depending on the task variant. For example, simple deterministic tasks involve model-free learning since the task can be solved by implementing a win-stay/lose shift strategy. However, in a task variant with both options rewarded but with different outcomes (Keiflin et al., 2013), this problem can only be solved by model-based learning. Therefore, reversal learning tasks can incorporate both model-free and model-based aspects, depending on the method employed.

Recently, Costa et al. (2015) proposed a novel Bayesian framework for studying both reward learning and strategy use, in reversal learning (Costa et al., 2015). Within this framework, one model is closely associated with a reward learning process that tracks whether the subject did or did not receive a reward for a particular choice. This is similar to an accumulator model in which a decision reflects evidence accrued from information based on discrete time versus continuous state space, that is not unlike initial pairwise discrimination learning (Vickers, 1970). A second model, which likely interacts with the first, represents the subject’s belief that reversals or changes in reward contingency can occur. It does so by accumulating a different kind of evidence over multiple reversals to generate a likelihood estimate (known as a prior probability) that a reversal can occur. Using this framework, Costa and colleagues were able to model the subject’s “belief” that a reversal could occur, even before the first reversal, in both standard deterministic (Jang et al., 2015) and probabilistic reversal learning tasks (Costa et al., 2015, Jang et al., 2015). The utilization of such beliefs is consistent with model-based learning, described above.

The notion that reward learning and belief that reversals occur likely both contribute to efficient performance of reversal learning and a prospective coding of the anticipated trials is consistent with concepts such as learning set (Murray and Gaffan, 2006), and potentially also fits with formulations of ‘task space’ (Wilson et al., 2014). The idea of task space suggests that subjects generate an abstraction of the current state of performance on-task that is not bound to the stimulus or cue, but instead to a more ‘cognitive’ representation of some hidden variable that is relevant to the task, such as the possibility of changing S-O or task context (Wilson et al., 2014, Saez et al., 2015). Like the formulation of a belief that reversals can occur, the map or task space model is particularly informative when explaining serial reversal learning or switching S-O probe trials - where it can account for improvements in performance resulting from experience. How notions of task space differ from the belief that a reversal can occur is, however, yet to be determined.

Taken together, these studies suggest that we should consider a revised definition of the construct validity of reversal learning. Irrespective of their exact design, reversal learning paradigms all likely test the ability to: 1) learn from the rewards received (as well as absence of reward, or reward omission) upon choosing different stimuli (Stalnaker et al., 2015); 2) estimate the likelihood or prior probability that reversals can occur (Costa et al., 2015, Jang et al., 2015); and/or 3) generate an understanding of task or option space (Wilson et al., 2014, Saez et al., 2015) (Figure 3). Defining these different sub-processes in future work will be important, as it also allows more straightforward comparisons between species and paradigms, and the parsing of specific neural systems that contribute to these different aspects of cognitive flexibility. Another challenge will be to determine the concordance of reversal learning sub-processes with other tasks of cognitive flexibility, such as discounting tasks, set shifting tasks, and responses during reinforcer devaluation or in contingency degradation paradigms.

Figure 3. Sub-processes in reversal learning that contribute to a revised definition for the construct validity of the task.

The widely-accepted idea that reversal leaning paradigms primarily measure inhibitory control (1) of responding, has fallen out of favor. Instead, reversal learning paradigms all likely test the ability to learn from rewards and non-rewards upon choosing different stimuli (2), estimate the likelihood or prior probability that reversals can occur (3), and/or generate an understanding of task or option space (4).

Neural substrates of reversal

Cortical regions

Neuroimaging studies report increased activity in orbitofrontal cortex (OFC) and medial prefrontal cortex (mPFC) in human subjects performing reversals (Nagahama et al., 2001, Cools et al., 2002, Kringelbach and Rolls, 2003, Remijnse et al., 2005, Ghahremani et al., 2010), and patients with lesions of these regions exhibit reversal learning deficits (Rolls et al., 1994b, Fellows and Farah, 2003b, Hornak et al., 2004). Furthermore, a consistent finding from over fifty years of work in nonhuman primates, has been that frontal lobectomies and ablations (Battig et al., 1962, Butter et al., 1963), or more localized aspiration lesions of the OFC, impair various measures of flexible responding for reward (Jones and Mishkin, 1972, Izquierdo et al., 2004a, Machado and Bachevalier, 2007) (for recent reviews, see (Hamilton and Brigman, 2014, Rudebeck and Murray, 2014). In addition, excitotoxic lesions of the OFC in marmosets, that were presumed to spare fibers of passage, were found to selectively impair reversal learning, while leaving other forms of behavioral flexibility intact (Dias et al., 1996b, a, 1997). Similarly, recordings of single neuron activity in the OFC during reversal learning also show that neurons track changes in reward contingency (Thorpe et al., 1983, Schoenbaum et al., 1999, Morrison and Salzman, 2009).

Extending these studies in nonhuman primates, research in rodents has shown that excitotoxic lesions and glutamatergic manipulation of the OFC impairs reversals based on visual stimuli (Bohn et al., 2003, Graybeal et al., 2011, Brigman et al., 2013, Izquierdo et al., 2013), as well as reversal of auditory cues (Burke et al., 2009), spatial responses (Ghods-Sharifi et al., 2008, Young and Shapiro, 2009) and spatially-cued operant responses (Boulougouris et al., 2007, Mar et al., 2011). OFC inactivations or lesions also impair reversal of olfactory and tactile cues in rats and mice (Ferry et al., 2000, Schoenbaum et al., 2002, McAlonan and Brown, 2003, Schoenbaum et al., 2003, Ragozzino, 2007, Bissonette et al., 2008) (Ragozzino, 2007, Churchwell et al., 2009). The importance of the OFC to reversal learning may depend upon the amount of discrimination training the subject receives before reversal. Setting a stringent performance criterion for discrimination produces improvements in reversal learning performance when the OFC is inactive, whereas loss of OFC function causes reversal impairment when the discrimination criterion was easier to attain (Riceberg and Shapiro, 2012). This is possibly due to the role of the OFC varying as a function of the relative ‘stability’ in reward experience.

The role of the mPFC in reversal is distinct from that of the OFC. A majority of studies in rodents report that reversal learning is insensitive to mPFC damage (Birrell and Brown, 2000, Bissonette et al., 2008, Floresco et al., 2008, Churchwell et al., 2009, Cordova et al., 2014) and can even be enhanced by lesions or stress-associated dysfunction in this region (Salazar et al., 2004, Graybeal et al., 2011, Bryce and Howland, 2015). Interestingly, several studies have suggested the mPFC may be recruited in reversal when attentional processes are taxed with difficult discriminanda (Bussey et al., 1997, Brigman and Rothblat, 2008), when multiple contingency changes are tracked continuously (Kosaki and Watanabe, 2012) or when discrete cues are coupled with a high visual or visuospatial component (Meunier et al., 1991, Li and Shao, 1998, Ragozzino et al., 1999, Chudasama and Robbins, 2003, Schwabe et al., 2004, Pickens et al., 2005, Young and Shapiro, 2009, Shaw et al., 2013). Similarly, lesions of the mPFC do not produce gross effects on reversal learning in monkeys (Meunier et al., 1997, Rudebeck et al., 2006). Instead, macaques with mPFC lesions exhibit a slight deficit in the ability to maintain the correct response following a reversal (Chudasama et al., 2013). This also fits with a recent neurophysiology study showing the neurons in the anterior cingulate cortex track rewarded and unrewarded choices over multiple trials during reversal learning (Kawai et al., 2015). Thus, the mPFC may have a more circumscribed role in reversal than the OFC, one that primarily manifests in tasks with a high demand on attention and performance monitoring; consistent with ideas about the involvement of the mPFC in stimulus detection, timing, and error detection (Laubach et al., 2015). However, the precise contribution of the mPFC in reversal learning remains enigmatic and a question for future work.

Understanding the place of the OFC in reversal learning is also evolving. Some authors frame the OFC’s role in reversal as one representing expected outcomes (Schoenbaum et al., 2009b, Rhodes and Murray, 2013, Stalnaker et al., 2015) and task space (Wilson et al., 2014). This outcome representation could utilize value information stored in the OFC (Padoa-Schioppa, 2007, Cai and Padoa-Schioppa, 2014), and/or derive outcome information from subcortical networks tracking critical parameters of the reward environment (e.g., the history and uncertainty of reward availability, its cost and incentive value) (Wassum and Izquierdo, 2015). These conceptualizations are consistent with the finding that OFC lesions do not decrease responding to reward-related cues learned and presented after devaluation (Pickens et al., 2003, Pickens et al., 2005), the observation that OFC is required to store response-outcome associations during reversal learning (Keiflin et al., 2013), and the observation from in vivo electrophysiological recordings that OFC neurons respond to the expected outcome and track reward value across reversal learning (Schoenbaum et al., 2000, Bissonette et al., 2008, Moorman and Aston-Jones, 2014). Moreover, problems updating the value of reward-related cues after OFC lesions could also account for the finding that reversal deficits are typically seen on initial, but not subsequent, reversals (Schoenbaum et al., 2002, Boulougouris et al., 2007, Klanker et al., 2013), given the subject will have accrued more updated information by a second reversal that could serve to mitigate the impact of a suboptimal outcome-valuation system.

Contrasting with earlier data from aspiration lesions in macaque monkeys and excitotoxic lesions in marmosets, fiber-sparing excitotoxic targeting of OFC subregions limited to Walker’s areas 11, 13 and 14 in macaque monkeys have recently been shown to leave reversal intact (Rudebeck and Murray, 2011, Rudebeck et al., 2013). Moreover, aspirating a narrow ‘strip’ of the posterior OFC reinstated reversal deficits. This raises the possibility that damage to white matter tracts near the OFC could at least partially account for impairments attributed to OFC damage (Rudebeck et al., 2013, Rudebeck and Murray, 2014). It is important to note that these findings do not rule out the possibility that areas near to the OFC in macaque monkeys may be important for performance on reversal learning. Indeed given that even small amounts of damage to white matter near OFC cause deficits in reversal learning it seems likely that areas laterally adjacent to the OFC in macaques are engaged during the task, a possibility that has recently received some support (Chau et al., 2015). The importance of the OFC was further demonstrated by the finding that serotonergic depletion in this region impaired reversal learning in marmosets (Clarke et al., 2004, Clarke et al., 2005, Clarke et al., 2007, Walker et al., 2009). As we review in more detail later, levels of serotonin in the OFC, together with DA in putamen, have been estimated to account for a significant proportion of the variance in reversal learning performance in vervet monkeys (Groman et al., 2013). Determining the precise brain regions and systems involved in reversal learning in macaques and how these relate to areas in rodents, other species of monkey, and humans are important avenues for future research.

Striatal and amygdalar regions

In humans, neuroimaging studies demonstrate recruitment of the dorsal (Rogers et al., 2000) and ventral regions of the striatum (Cools et al., 2002) during reversal, while lesions of the basal ganglia are associated with impairments in reversal learning (Rogers et al. 2009). In the rodent, anterograde tracing shows that the OFC projects to the nucleus accumbens (NAc) and dorsomedial striatum (DMS) (Haber et al., 1995, Schilman et al., 2008) and receives reciprocal input from the striatum through the mediodorsal nucleus of the thalamus (Middleton and Strick, 1996). The DMS is implicated in reversal learning by studies demonstrating that neurotoxic lesions of this region impair various forms of reversal learning in marmosets (Clarke et al., 2008, Robbins et al., 2008) and rats (Ragozzino, 2007, Castañé et al., 2010) (Braun and Hauber, 2011). Likewise, neurotoxic lesion of the NAc disrupts spatial, but not visual, reversal learning in monkeys (Stern and Passingham, 1995), as well as probabilistic reversal learning in rats (Dalton et al., 2014), although there is as much, if not more, evidence of unaffected reversal learning following lesions to NAc (Burk and Mair, 2001, Schoenbaum and Setlow, 2003, Castañé et al., 2010).

In addition to receiving strong cortical inputs, the amygdala projects to the NAc and DMS, forming a functional circuit that may support reversal learning. Recent models propose projections from the BLA to the NAc and DMS influence motivated behavior and action selection involving both rewards and punishments, while BLA-OFC interactions generate a high-resolution representation of expected outcome value (Wassum and Izquierdo, 2015). However, various lines of evidence have provided only equivocal support for a contribution of the BLA to reversal learning (Schoenbaum et al., 2003, Stalnaker et al., 2007, Churchwell et al., 2009, Izquierdo et al., 2013) – likely due to variation across studies in the specific methods employed. For example, the BLA may be less engaged in tasks that can be solved without the need to form representations of different outcomes, as in two-choice and/or deterministic reversal tasks where outcome-specific representations do not aid performance. The outcomes encoded by BLA are restricted to the cues that predict them; distinct from the ‘expected outcomes’ that are linked to OFC function. The specific outcomes represented in BLA may be used by OFC to generate expectations, or conversely, OFC may set the learning rate parameter in the RL model by which specific representations are formed in BLA. In two-choice tasks, lesions of rat BLA or whole amygdala in monkey facilitate reversal learning, presumably by enabling a simpler stimulus-response strategy that is not based on outcome representations (Rudebeck and Murray, 2008, Izquierdo et al., 2013). It also may matter when the amygdala goes ‘offline.’ Inactivation or lesions made before specific outcome representations are formed may alter the way the association is learned and, therefore, the way the animal adapts to later changes in contingency, when assessed in the reversal phase. In tasks that do necessitate outcome-specific representations, for example where animals are required to compare outcomes (sucrose versus quinine), or when BLA inactivations occur after initial training but prior to the reversal phase, BLA manipulations disrupt performance (Schoenbaum et al., 2003, Churchwell et al., 2009).

Given these considerations, the BLA can be conceptualized as supporting reversal learning by tracking prior outcomes and comparing them to current outcomes. This idea is supported by computational modeling (Jang et al., 2015) and in vivo recordings showing that monkey amygdala neurons respond according to the deviation between cached and expected rewards (Belova et al., 2007, Bermudez and Schultz, 2010). This is also consistent with the notion that amygdala encodes a dynamic representation of outcome that incorporates both the stimuli that predict those outcomes, and a history of reward (Paton et al., 2006, Morrison and Salzman, 2010).

Neurochemical modulation of reversal

The molecular and neurochemical factors influencing reversal learning and the associated cognitive domains affected are yet to be fully understood (Izquierdo et al., 2012). Here we consider what have been the most intensively studied neurotransmitter systems in reversal learning - serotonin, dopamine and glutamate.

Serotonin

Systemic depletions of serotonin produce generalized effects on reward learning that are not specific to reversal learning (Lapiz-Bluhm et al., 2009). For example, rats systemically treated with para-chloroamphetamine (PCA) or parachlorophenylalanine (PCPA), both of which produce significant depletion of brain serotonin, showed impaired stimulus-reward, discrimination and reversal learning (Masaki et al., 2006, Izquierdo et al., 2012). However, serotonin and serotonin transporter levels in the rodent OFC predict individual variation in reversal learning performance, suggesting a more circumscribed role for the transmitter (Stolyarova et al., 2014, Barlow et al., 2015). A study in vervet monkeys by Groman and colleagues also reported that serotonin levels in ventrolateral PFC/Area 47 correlated with reversal learning performance (Groman et al., 2013). In relation to the previous discussion on the role of OFC in reversal learning, note that Area 47 is outside and laterally adjacent to the areas considered to constitute the OFC in rhesus monkeys. Nonetheless, this result does fit with recent studies suggesting that areas outside of the OFC may be necessary for reversal learning in macaques (Rudebeck et al., 2013).

Specific serotonergic manipulations have ranged from 1) region-selective neuron destruction and depletions (e.g., in the OFC or amygdala) (Park et al., 1994, Rogers et al., 1999, Clark et al., 2004, Clarke et al., 2005, Clarke et al., 2007, Finger et al., 2007, West et al., 2013, Ochoa et al., 2015, Rygula et al., 2015), 2) serotonin receptor blockade (Boulougouris and Robbins, 2010), 3) treatment with serotonin reuptake inhibitors (Brigman et al., 2010b, Brown et al., 2012, Furr et al., 2012a, Wallace et al., 2014), to 4) phenotyping gene deletions and polymorphisms (Homberg et al., 2007, Izquierdo et al., 2007, Vallender et al., 2009, Brigman et al., 2010b, Jedema et al., 2010). A consistent finding to emerge from this work has been that reduced serotonin signaling (in the cortex, rather than striatum) increases perseveration and impairs reversal learning, whereas increasing serotonin generally facilitates reversal learning (Clarke et al., 2004).

There are, however, problems with this blanket conclusion, most notably because specific serotonin receptor subtypes have differential effects. For instance, in rats antagonism of 5-HT2A (via MDL 100–907) impairs reversal performance, while 5-HT2C antagonism (with SB 242084) facilitates performance (Boulougouris et al., 2008, Boulougouris and Robbins, 2010, Furr et al., 2012b, Nilsson et al., 2012), though there is also a report of an overall impairment after this same treatment (Alsio et al., 2015). Performance effects of serotonergic manipulations are likely also influenced by sensory modality, whether ligands are administered centrally or peripherally, and the amount of pre-reversal training allowed. As an example of the latter, post-training probabilistic spatial reversal learning is impaired after 5,7-DHT lesions in rats (Bari et al., 2010a) yet rats failed to learn to touch stimuli to bring about reward after PCPA administration, before any training with contingencies (Izquierdo et al., 2012).

There is then the question of the cognitive processes by which serotonin modulates reversal learning. Some authors have proposed that serotonin signaling affects punishment or absence-of-reward processing by subcortical structures, but also regulates processes ascribed to cortical regions, such as outcome-tracking (Dayan and Huys, 2009, van der Schaaf et al., 2011, den Ouden et al., 2013). The relative balance between these effects may vary with the duration of serotonin alterations, with acute manipulations increasing the gain on negative feedback, whereas chronic treatments affect reward sensitivity (Bari et al., 2010b, Rygula et al., 2014). In rats, BLA serotonin depletions failed to affect deterministic reversal performance, possibly due to the timing of the manipulation relative to training and testing (Ochoa et al., 2015). However, when Rygula and colleagues compared the effects of localized serotonin depletions on a probabilistic visual reversal task in marmosets they found impairments after amygdala depletions were related to increased sensitivity to misleading reward and punishment, while OFC depletion impairments were associated with poor response suppression (Rygula et al., 2014). In summary, serotonergic tone modulates reversal learning across species and appears to do so, at least in part, through affecting processing in OFC.

Dopamine

Dopamine mediates synaptic plasticity in brain regions subserving optimal reversal learning performance, notably the cortex and striatum (Reynolds and Wickens, 2002, Cagniard et al., 2006, Calabresi et al., 2007). In addition, dopamine neurons encode so-called reward prediction errors that coincide with violations in expected outcomes (Schultz, 2013), and can be manipulated to generate and disrupt various forms of learning (Tsai et al., 2009, Adamantidis et al., 2011, Steinberg and Janak, 2013). Indeed, optogenetic excitation of dopamine neurons in the ventral tegmental area (VTA) or substantia nigra pars compacta (SNc) improves spatial reversal learning performance in rats (Adamantidis et al., 2011, Rossi et al., 2013). It is perhaps surprising then that, in contrast to serotonin depletions, depleting dopamine content in the OFC did not disrupt touchscreen reversal learning in marmosets (Clarke et al., 2007). Conversely, depleting dopamine, but not serotonin, in the marmoset striatum led to non-perseverative reversal impairments (Clarke et al., 2011).

This dissociation suggests a prominent role of striatal dopamine in cognitive flexibility – a contention supported by recent empirical data. In human subjects, methylphenidate has been shown to improve reversal learning specifically when it results in an increase in striatal dopamine levels (Clatworthy et al., 2009). Using fast-scan cyclic voltammetry to measure phasic dopamine transients in the rat nucleus accumbens, Klanker and colleagues found phasic dopamine increases in response to unexpected rewards (positive prediction error) during a cued lever-based reversal learning paradigm (Klanker et al., 2015). This fits with the observation that striatal dopamine transporter binding correlates with individual differences in the use of positive feedback during reversal learning (Stolyarova et al., 2014). Together, these findings suggest dopamine might serve as a neurochemical substrate for the use of Bayesian prior beliefs in reversal learning whereby, as discussed earlier, performance is guided by evidence of reversal occurring from prior experience (Jang et al., 2015).

An important corollary question is that of the relative roles of specific dopamine receptor subtypes in reversal learning. A study in mice demonstrated early touchscreen visual-cue reversal deficits following systemic treatment with a D1 receptor agonist (SKF81297) (Izquierdo et al., 2006). The locus of this systemic drug effect remains to be determined, but the nucleus accumbens and BLA are prime candidates. One salient finding here is that BLA-NAc projections invigorate reward-related behaviors in a D1 receptor-dependent manner (Stuber et al., 2011). Whether a similar subcortical D1 receptor mechanism contributes to the early stages of reversal learning performance when new cue-reward associations are being formed is yet to be determined. There is also the outstanding question of cortical D1 receptors given their singular contribution to other forms of cognition (Floresco, 2013, Wass et al., 2013).

Regarding other dopamine receptor subtypes, low dopamine D2 receptor availability correlates with poor reversal learning in mice (Laughlin et al., 2011), monkeys (Groman et al., 2011) and humans (Jocham et al., 2009). Furthermore, D2 receptor gene deletion impairs reversal learning performance in mice (Kruzich and Grandy, 2004, Kruzich et al., 2006, De Steno and Schmauss, 2009), while either D2/D3 receptor agonists (quinpirole, 7-OH-DPAT) or antagonists (raclopride), impair spatial and olfactory reversals in rats (Boulougouris et al., 2009), monkeys (Smith et al., 1999, Lee et al., 2007) or healthy human volunteers (Mehta et al., 2001). These findings suggest that reversal learning depends upon an optimal balance of dopamine D2 receptor function (Izquierdo et al., 2012).

Experience with reward events may calibrate the impact of subsequent events on D2 receptor signaling; events that may be negative or suboptimal by comparison, as in the case of acute drug-withdrawal (Thompson et al., 2015a). More generally, this role is consistent with the idea that dopamine tone scales the performance of previously learned behavior and that poor reward conditions, as experienced in early reversal learning, are characterized by weaker dopamine signaling, co-occurring with D2 receptor activation (Cagniard et al., 2006, Calabresi et al., 2007). In support of this, in monkeys with extensive experience on a two-arm bandit reversal task, neither systemic administration of L-Dopa or haloperidol affected learning, but haloperidol led to increased reliance on prior beliefs about when a reversal would occur and monkeys displayed anticipatory switching (Costa et al., 2015).

Glutamate: N-methyl-D-aspartate receptors (NMDAR)

Glutamate is key substrate for synaptic plasticity and many forms of learning. Consequently, a number of investigators have examined the role of the NMDAR, as well as other glutamate receptors and transporters (Karlsson et al., 2009, Barkus et al., 2012), in reversal learning and other forms of cognitive flexibility (for review, see (Jentsch and Roth, 1999). This work has led to reports that acute, systemic NMDAR blockade (with MK-801) can cause impairments in spatial reversal learning (Lobellova et al., 2013), but also fail to affect reversal performance and instead disrupt set-shifting (Svoboda et al., 2015). The literature on the effects of subchronic NMDAR antagonism with phencyclidine (PCP) (considered a potential model of schizophrenia (Brigman et al., 2010a) is also equivocal. While early studies found reversal learning impairments with acute or subchronic PCP treatment in rodents (Abdul-Monim et al., 2007), more recent reports find no change (Brigman et al., 2009, Janhunen et al., 2015) or an improvement in late-stage reversal learning (Dix et al., 2010, Fellini et al., 2014, McAllister et al., 2015). As with other apparently discrepant data in the reversal literature, the potential for methodological variation to explain some of these differences should not be discounted.

Using gene manipulations and pharmacology, some of these studies have attempted to parse the role of the specific subunits that constitute the heteromeric NMDAR complex and influence its function. Brain-wide deletion of the GluN2A subunit was found to impair touchscreen-based reversal learning in mice; a phenotype that likely reflected a general learning deficit given it was accompanied by impaired discrimination and occurred late in reversal when learning, rather than flexibility, is presumed to be taxed (Brigman et al., 2008, Marquardt et al., 2014). As with any gene knockout approach, the extent to which constitutive GluN2A deletion can be attributed solely to loss of the subunit, and not to more generalized alterations in NMDAR function, is unclear. However, a pharmacological approach to targeting NMDAR subunits, particularly GluN2A, also has issues relating to specificity (Neyton and Paoletti, 2006). Notwithstanding, a very recent study failed to find a deficit in reversal in a touchscreen location-based task following acute systemic treatment with a GluN2A preferring antagonist (NVP-AAM077), whereas treatment with either a subunit non-specific blocker (MK-801) or a GluN2B antagonist (CP 101–606) impaired the same behavior (Kumar et al., 2015).

The ability of GluN2B antagonism or gene deletion to impair reversal learning performance has now been reported across a number of behavioral paradigms. Dalton and colleagues demonstrated that systemic administration of a GluN2B blocker (Ro 25–6891) increased perseveration in a spatial location reversal task (Dalton et al., 2011), without affecting initial discrimination learning. Brigman and colleagues also found selective, perseveration-driven, (touchscreen visual-cue) reversal deficits following cortex-wide GluN2B deletion (Radke et al., 2015) or local blockade of GluN2B (via Ro 25–6981) in the lateral OFC (Brigman et al., 2013, Thompson et al., 2015b). With more widespread forebrain GluN2B deletion or localized antagonism in the dorsolateral striatum, the same authors found that deficits extended to discrimination learning – suggesting GluN2B expressed in different brain regions has dissociable roles in discrimination and reversal phases of a stimulus reversal learning task.

These studies of GluN2B raise the question of whether NMDARs, more generally, exert effects on reversal in a region-specific manner. Antagonizing NMDARs (with MK-801 or AP-5) in the DMS of adult or weanling rats impairs spatial reversal learning (Palencia and Ragozzino, 2004, Watson and Stanton, 2009). Ragozzino and colleagues found that these deficits were accompanied by decreases in acetylcholine efflux (Palencia and Ragozzino, 2006) and could be mimicked by blocking muscarinic M1 (via pirenzepine) or stimulating M2 (via oxotremorine sesquifumurate), but not by antagonizing nicotinic (via mecamylamine), acetylcholine receptors in the DMS (Tzavos et al., 2004, McCool et al., 2008, Ragozzino et al., 2009). Somewhat surprisingly, given this evidence of a key role for DMS NMDARs in reversal, a recent study found no effect of DMS NMDAR antagonism (via AP5) on a visual-cue reversal learning task that was impaired by NMDAR blockade in the NAc (Ding et al., 2014). It may be that differences in the way reversal was tested across these studies accounts for this discrepancy, but further investigation would be needed to draw any firm conclusions.

Concluding remarks

In this review we summarized evidence that reversal learning paradigms generally test the ability to learn specific stimulus-outcome associations, to estimate the likelihood that reversals can occur given accumulated evidence, and/or to generate a representation of option or task space. More investigation is needed to design behavioral tasks to extract and further describe the neural mechanisms of these individual processes, which should then allow even better comparisons between species and across paradigms. With the availability of higher resolution neuroscience techniques, it should be possible to assign cell type- specific roles to each process and therefore, to distinct aspects of cognitive flexibility.

Highlights.

• -Reversal learning is a widely used test of cognitive flexibility across species
• -The idea that this learning primarily measures response inhibition has been revised
• -We describe how it is measured and present new definitions for its construct validity
• -We also present an update of the brain regions and neurotransmitters that support it