Two tales of how expectation of reward modulates behavior

Abstract

Expectation of reward modulates many types of behaviors. Here we highlight two lines of research on reward-modulated perceptual decision making in primates and social context-modulated singing in songbirds, respectively. These two seemingly distinct behaviors are both known to involve cortico-basal ganglia-thalamic circuits. The underlying computations may be conceptualized using a simple, common framework. We summarize and compare our current knowledge of the two fields to motivate new experiments for each field, with the goal of finding general principles for how the brain implements reward-modulated behavior.

Introduction

In goal-directed behaviors, expectations of reinforcement are an integral part of the definition of goals. For example, for animals foraging for food, it is often beneficial to have some knowledge of how much food reward may be expected of currently available food sources and then act accordingly. For animals trying to attract a mate, it is likely beneficial to appreciate what the potential mates prefer, which may be thought of as an expectation of successful mate attraction given certain attributes of the mate-attracting behavior. In this review, we highlight two lines of research that deal with seemingly distinct types of goal-directed behaviors: reward-modulated perceptual decision making in primates and social context-dependent singing behavior in songbirds. We suggest that, despite the apparent differences in species, behavioral modality and the general premise of goals, these two lines of research share many similarities and could inform each other to advance our understanding of the neural basis of how expectation of reinforcement modulates behavior in general.

In the following sections, we first describe the behaviors of interest. We then introduce a common framework to describe the general computations that are needed for these two types of behaviors. Next we describe the cortico-basal ganglia-thalamo-cortical circuits that are (likely) involved in mediating these behaviors, drawing evidence from fMRI studies in humans and neurophysiological studies in monkeys and songbirds. Finally, we put forth some open questions for future studies, guided by a comparison of existing knowledge between the two lines of research.

The behaviors and general framework

Reward-modulated perceptual decision making

To illustrate some key properties of reward-modulated perceptual decision behavior, we focus on examples of adding explicit reward expectation manipulations to a well-known perceptual decision task, the visual motion direction discrimination task [1,2]. On a basic motion discrimination task, a subject sees a field of moving dots on a screen and needs to report the global motion of these dots. On each trial, there are two possible directions (i.e., two possible choices) and a proportion of the dots move in one direction while the other dots move in random directions. This proportion specifies the motion stimulus strength and influences the difficulty of the perceptual judgment: if the motion strength is high, i.e., most dots are moving coherently in one direction with only a few dots moving randomly, it is easy to tell that the global motion is in that direction; conversely, if the motion strength is low, i.e., only a few dots are moving coherently in one direction in a field of randomly moving dots, it is hard to determine the true global motion. Another factor influencing the task difficulty is the viewing time: when the viewing time is controlled by the experimenter, the longer the viewing time, the more likely the subject makes a correct decision; when the viewing time is controlled by the subject, trials with lower motion strength tend to have longer viewing time (i.e., longer response time).

With the basic task design, reward expectation is not explicitly manipulated, except that all correct choices earn a fixed reward and error choices are not rewarded and/or incur a fixed penalty (e.g., time out). One way to explicitly manipulate reward expectation is thus to make the reward size associated with the two choices unequal. In humans and monkeys, this kind of manipulation has robust effects on behavioral performance [3-8]. These effects include: 1) subjects are more likely to pick the choice associated with the larger reward; such a choice bias is more prominent on trials with low motion strength, while minimal on trials with very high motion strength; 2) when subjects control the viewing time, for the same motion strength, response time is shorter for the choice associated with the larger reward; such response time bias also depends on motion strength, with the largest bias occurring on trials with low motion strength.

To understand performance on this task, consider a simple framework illustrated in Fig 1A. At the onset of the decision process, the brain combines sensory information related to the moving dots and contextual information related to the reward contingency, and produces a categorical “left” or “right” choice. Using concepts from the perception literature, a perceptual categorization process can be generally thought of as having two components: sensory representation, reflecting the conversion of sensory inputs into task-relevant evidence, and readout, reflecting the conversion of the task-relevant evidence into a categorical decision [9]. A priori, the contextual information may influence both components. As an example of reward modulation of sensory representation, when the “left” choice is paired with the larger reward, there may appear to be more leftward-moving dots than what are actually present, biasing the evidence toward the “left” choice. As an example of reward modulation of readout, in the same left-choice-with-larger-reward scenario, the subject may require observing a total of 100 rightward-moving dots before making a “right” choice, but only 20 leftward-moving dots to make a “left” choice, thereby reading out the same evidence with a leftward bias. Based on model fitting using “accumulate-to-bound” models, reward-biased readout better recapitulates observed behavioral effects on reward-biased motion discrimination tasks [3-6] and this seems to be a common finding for many unequal-payoff perceptual decision tasks [7,10-13]. During decision formation, the brain may also generate predictions of how much reward is expected. A comparison between these predictions and actual reward received can then instruct changes of the categorical decision-formation process to improve future decisions.

Figure 1.

A common conceptual framework for reward-modulated perceptual decision making (A) and social context-modulated vocal behavior (B). Blue text indicates the conceptual functions (identical for the two panels) and black text provides behavior-specific information.

Social context-dependent singing

Another example in which behavior is altered depending on the context, perhaps including the expectation of reward, is in avian song. Songbirds use song for courtship, territory defense and individual recognition. In zebra finches, a young male learns a tutor song through a trial-and-error process [14], in which the young bird’s own songs are compared against a memorized template [15]. This learning occurs in an unsupervised manner; no external rewards are needed to guide progress or indicate success. Over a period of ~3 months, songs show gradual improvement until a good match is achieved [14]. During this period, song variability also decreases such that the adult song becomes acoustically stereotyped, save for a small amount of residual variability.

This residual variability is modulated by social context, potentially in anticipation of social reward. When a male sings as part of female-directed courtship (“directed song”), trial-to-trial song variability drops to about half of the value measured when the bird sings alone (“undirected song”) [16-18]. Females care about this distinction, spending more time near a speaker playing the mate’s directed song than one playing undirected song [19]. One practical advantage of this behavior is that it is natural and, unlike the primate decision task, does not require extensive training.

We can examine song variability in the same conceptual framework as described above for primate perceptual decisions. Consider a simple framework illustrated in Fig 1B. We make two assumptions: the presence of a female indicates the possibility of a social reward; and production of more stereotyped songs, compared to more variable songs, incurs some cost or reduced benefit to the male bird. This cost may include effort spent in maintaining tighter control of motor production and loss of opportunity for practicing to improve his songs in future. At a given time, the male bird makes a decision, based on sensory information (whether a female bird is present) and expectation of reinforcement (expected cost of performing low-variability songs and probability of attracting that particular female bird). This combination of information is fed into the song system controlling song variability. At the end of a performance, the male bird may either gain a mating opportunity or fail to do so. A failure may help generate a prediction error to guide either the decision process and/or motivate more practicing to improve song stereotypy. To date, songbird researchers have made a binary distinction between directed and undirected song. There has been no exploration into whether song variability is graded or rather switched in a binary fashion from high to low and back. Similarly, unlike studies in primate decision making, there has been no exploration into how song quality varies with manipulations of the probabilities or amounts of reward.

Cortico-basal ganglia-thalamo-cortical loops

Reward-modulated perceptual decision making

The neuronal mechanisms underlying reward-modulated perceptual decisions are only beginning to be explored. In the oculomotor basal ganglia loop (Fig 2A), every major station tested on the basic motion discrimination task with saccade output, (i.e., the frontal eye field (FEF), lateral intraparietal area (LIP), the caudate nucleus and the superior colliculus (SC)), contains neurons with decision-related activity [20-25]. The LIP and caudate nucleus, in particular, play a causal role in perceptual decision-making on the motion discrimination task [26**,27]. On value-based decision tasks that do not manipulate perceptual demand, these same regions display various kinds of reward modulation of visual cue-evoked activity (for a severely truncated list of examples, see [28-31] for LIP, FEF, SC and the caudate nucleus). Thus the oculomotor basal ganglia loop seems a solid starting point to search for neural mechanisms that mediate reward-modulated perceptual decisions. Indeed, theoretical studies have demonstrated that the cortico-basal ganglia circuit can make optimal perceptual decisions under different reinforcement contexts [32,33]; in human subjects performing unequal-reward perceptual tasks, the fronto-parietal network, which includes FEF and LIP, shows activation patterns suggestive of incorporation of reward and sensory information [4,13]; in monkeys performing unequal-reward motion discrimination tasks, LIP neurons show reward-modulated activity that is consistent with changes in readout for decision formation [5**], FEF neurons encode an error-predicting signal that may contribute to decision evaluation [34], and dopamine neurons in the SNc carry reward-predictive activity that may be used as an instructive signal for improving future decisions [35**]. Functional roles of other cortical and subcortical regions remain to be determined.

Figure 2.

Simplified diagram of candidate brain regions for reward-modulated perceptual decision-making (A) and social context-modulated vocal behavior (B). Abbreviations: DA: dopamine; DLM: medial portion of the dorsolateral nucleus of the thalamus; FEF: frontal eye field; GPe: external segment of the globus pallidus; GPi: internal segment of the globus pallidus; HVC: used as a proper name; LIP: lateral intraparietal area; LMAN: lateral magnocellular nucleus of the anterior nidopallium; MD: mediodorsal nucleus of thalamus; RA: robust nucleus of the arcopallium; SC: superior colliculus; SNc: substantia nigra pars compact; SNr: substantia nigra pars reticulata; STN: subthalamic nucleus; VTA: ventral tegmental area.

Social context-dependent singing

The neuronal mechanisms underlying social context-dependent singing behavior have been much more extensively studied. In the schematic diagram of the song system (Fig. 2B), the pathway from HVC, through Area X, DLM and LMAN to RA is considered homologous to the mammalian cortex-basal ganglia-thalamic pathway [36-40]. Several lines of evidence suggest that song variability arises in this pathway. First, whether songs are variable or not, HVC activity is precisely timed with low variability, while RA and LMAN activity show considerable modulation of variability consistent with song variability [41,42**]. Second, silencing or lesioning LMAN dramatically reduces song variability [16,17,43], suggesting that the basal ganglia pathway is causally involved in song variability modulation. Third, in anesthetized animals, single-shock stimuli of HVC give rise to variable responses in DLM and LMAN [44*], suggesting that the basal ganglia pathway is capable of generating neural variability observed in LMAN.

Within the basal ganglia pathway, the social-context modulation of song variability likely depends on dopaminergic modulation of Area × circuitry. Activity of midbrain neurons changes with social context [45,46] and dopamine levels in the extracellular space of Area × are higher during directed song to a female bird, when a social reward might be expected, than during undirected song, when a social reward is not expected immediately [47]. Dopamine influences multiple properties of Area × circuitry, including Area × spiny neuron electrophysiology [48,49], the spontaneous firing rate of the pallidal neurons in Area × and evoked responses in pallidal and DLM neurons by HVC electrical stimulation [44*]. Although Area × may not be the only source of song variability (zebra finches sing with normal song variability after recovering from Area × lesions [50**]), two recent studies provided direct evidence for a causal role of dopaminergic modulation of Area × circuitry in social-context modulation of song variability: infusion of a D1 receptor antagonist into Area × prevents the reduction in variability observed during directed song [44*,51] and activity of Area × GPi-like neurons remains more variable during undirected songs than during directed songs, even after LMAN lesions [42].

Open questions

As may already be apparent from the brief review of our current knowledge of decision making in primates and singing in birds, the former line of research could benefit from more mechanistic investigations, while the latter may be at a stage where an intuitive and predictive computational framework would be valuable to guide future research. For example, in the oculomotor system diagram in Fig 2A, only three regions have been examined so far to determine how single neurons behave during reward-biased perceptual decision tasks, in clear contrast to the wealth of knowledge in the song system, in which we know how social context modulates neural activity at most major stations. In particular, specific computational roles have been proposed for the core basal ganglia structures [32,33**,52-54]. Determining which of these roles are actually in play will be important for our understanding of the neural mechanisms implementing the computations needed for the observed behavior. In addition, past studies have focused on average firing rate of neurons. Is the variability of neural activity also actively modulated according to the reward context in perceptual decision making?

In songbirds, we have made the assumption that song stereotypy has a cost, given that male birds do not seem to sing highly stereotyped songs without a social need. Is there a way to quantify such a cost relative to the social reward expectation? In zebra finches, in which most of the cited work was conducted, the song repertoire does not normally change in adult birds. Could it be that one consequence of practicing is to reduce the cost of song stereotypy, in addition to improving the acoustic quality of songs? Another untested issue is whether song variability is graded or under binary control. Careful analysis of the transition from the initial minutes when the male sings directed song to a newly-introduced female to later stages when he apparently loses interest and sings undirected song, could help clarify this issue. At a more general level, how do the dopamine neurons acquire social context modulation and how do they respond to a successful or courtship performance? As it seems that the same neural structures are involved in both social context-dependent modulation of song variability and vocal learning in young birds, how much do the detailed neural mechanisms overlap between these behaviors? For example, dopamine neurons receive indirect disinhibitory input from Area × via the ventral pallidum, which confers on them selective auditory responses for the bird’s own song [55]. Such auditory selectivity suggests that dopamine neurons may be able to evaluate a bird’s own song relative to the memorized tutor song and provide an internal “reward” signal during song practice. If so, how is this kind of signal multiplexed with social context information?

We have attempted to unify thinking about these two behaviors by placing them into a similar conceptual framework. It is possible that their parallels will strengthen, and this juxtaposition will benefit work on both systems. It is also possible that, as additional data become available, fundamental differences will emerge between them. In either case, we feel that attempting to unify conceptual models of brain function is a valuable goal, provided that these attempts lead to experimentally testable predictions.

Highlights.

• A common framework for reward-modulated perception and vocal behavior

• Both behaviors involve the cortico-basal ganglia pathway

• Reward modulates the readout process in perceptual decision making

• Social context modulates neural variability via dopamine-dependent mechanisms