Motivational Deficits in Schizophrenia and the Representation of Expected Value

Abstract

Motivational deficits (avolition and anhedonia) have historically been considered important negative symptoms of schizophrenia. Numerous studies have attempted to identify the neural substrates of avolition and anhedonia in schizophrenia, but these studies have not produced much agreement. Deficits in various aspects of reinforcement processing have been observed in individuals with schizophrenia, but it is not exactly clear which of these deficits actually engender motivational impairments in SZ. The purpose of this chapter is to examine how various reinforcement-related behavioral and neural signals could contribute to motivational impairments in both schizophrenia, and psychiatric illness, in general. In particular, we describe different aspects of the concept of expected value (EV), such as the distinction between the EV of stimuli and the expected value of actions, the acquisition of value vs. the estimation of value, and the discounting of value as a consequence of time or effort required. We conclude that avolition and anhedonia in SZ are most commonly tied to aberrant signals for expected value, in the context of learning. We discuss implications for further research on the neural substrates of motivational impairments in psychiatric illness.

1 Introduction

The reduced tendency to initiate goal-oriented behavior (avolition; Goldberg and Weinberger, 1988; Saykin et al., 1994) is an aspect of negative symptomatology thought to typify many patients with schizophrenia (SZ). While there are many factors which could lead to avolition/motivational deficits in SZ, the preponderance of studies indicate that SZ patients do not differ from controls in their self-reported experience of pleasure (“consummatory hedonics”; Cohen and Minor, 2008; Gard et al., 2007). Partially based on this evidence, we (Gold et al., 2008) hypothesized that avolition results from a failure to look forward to pleasurable outcomes (“anticipatory hedonics”), by virtue of the assignment of incentive salience to cues. As defined by Berridge and Robinson (1998), a stimulus becomes imbued with incentive salience when it is transformed from a neutral object into an object of attraction that animals will work to acquire. This is the essential outcome of reinforcement learning (RL), and it is thought to be a primary functional role of dopamine in the nervous system (Berridge and Robinson, 1998). The updating of the incentive value of a stimulus is thought to occur via the signaling of reward prediction errors (RPEs), which are mismatches between expected and obtained outcomes. Thus, a failure to update the incentive value of a stimulus could happen for at least three reasons: 1) the signal of the expected outcome is degraded or inaccurate; 2) the signal of the obtained outcome is degraded or inaccurate; or 3) the mechanism for computing the RPE is dysfunctional. Given the evidence that signals related to reward receipt in schizophrenia are intact (Cohen and Minor, 2010), considerable attention has been focused on the other two possibilities: that the signal of the expected outcome is degraded or inaccurate, and that the mechanism for computing the RPE is dysfunctional.

In fact, there is considerable evidence that acutely-ill patients (particularly those that are unmedicated) have genuinely-disrupted RPE signaling (Murray et al., 2007; Schlagenhauf et al., 2014; Schlagenhauf et al., 2009), with important implications for RL and belief-formation. Furthermore, there have been numerous findings of correlations between measures of both positive symptoms in schizophrenia and supposed RPE signals in the brain (Gradin et al., 2011). It is, however, much less certain that RPE signaling is abnormal in chronic, medicated patients (Walter et al., 2009; Waltz et al., 2010), despite clear evidence of reinforcement learning deficits in these patients (Farkas et al., 2008; Waltz et al., 2007). Furthermore, measures of RL performance have been shown to correlate with the severity of motivational deficits in chronic SZ patients. Were RL deficits to persist in stably-medicated SZ patients, despite evidence of intact RPE signaling, it would suggest that aberrant RPE-driven learning observed in medicated SZs may be more a problem of faulty input to the PE computation than dysfunction in the mechanism itself. In this chapter, our purpose is to evaluate the data arguing for and against the idea that the signaling of expected value (EV) in chronic SZ patients relates to motivational deficits, which are thought to persist throughout the illness and be largely unaffected by antipsychotic medications. This area has been the focus of numerous basic and clinical studies. Prior to discussing clinical findings, we will first review the basic concepts and methods that have served to guide the field.

2 Identifying a relationship between EV and avolition: Considerations

2.1 How do we quantify the severity of motivational deficits in schizophrenia?

The first step in linking an aspect of behavioral performance or a purported neural signal to the severity of motivational deficits in a psychiatric population is to establish how one quantifies the severity of motivational deficits. In the field of schizophrenia research, motivational deficits are commonly thought of as a component of “negative symptoms”, or areas of subnormal function (Peralta and Cuesta, 1995; Sayers et al., 1996). Thus, clinical interviews for assessing the severity of negative symptoms in schizophrenia – such as the Scale for the Assessment of Negative Symptoms (SANS; Andreasen, 1984) – involve questions about motivation. In previous studies (Gold et al., 2012; Strauss et al., 2011; Waltz et al., 2009), we have used both the individual Avolition/Role-functioning and Anhedonia/Asociality sub-scores, as well as a combination of the two, to quantify motivational deficits in SZ. Another scale used to quantify the severity of motivational deficits in SZ is the Schedule for the Deficit Syndrome (SDS; Kirkpatrick et al., 1989). The Deficit Syndrome has been described as a separate disease within the syndrome of schizophrenia (Kirkpatrick et al., 2001), whereby patients are characterized primarily by negative symptoms (such as avolition and anhedonia) not attributable to psychotic symptoms or antipsychotic medications. The SDS is an instrument designed to identify this subset of SZ patients, the prevalence of which ranges from 25% to 30% of chronic SZ patients (Kirkpatrick et al., 2001).

Two additional scales have recently been developed, in an attempt to separately ascertain capacities for consummatory and anticipatory hedonics: the Brief Negative Symptom Scale (Kirkpatrick et al., 2011) and the Clinical Assessment Interview for Negative Symptoms (Horan et al., 2011). Validation studies (Kring et al., 2013; Strauss et al., 2012) have demonstrated that these measures show good convergent validity in their relationships with other symptom rating scales, but are not redundant with them (for more detail on these newer assessment tools, please read the chapter by Reddy, Horan, and Green in this volume). Aside from clinical rating scales, There are self-report measures, meant to quantify the severity of anhedonia, specifically, such as the Chapman Scales for Physical and Social Anhedonia (Chapman et al., 1976). The Chapman Scales for Physical and Social Anhedonia can be, and have been, administered to both psychiatric patients and controls, and are thought to serve as “trait” measures of anhedonia, rather than in-the-moment “state” measures.

One of our major goals has been to provide a mechanistic account of motivational deficits in schizophrenia. The first step in that enterprise has been to identify component processes in reward processing and reinforcement learning and to describe their relationships with clinical assessments of motivational deficits. It is possible, however, that experimental measures might do a better job of capturing the phenomenon/disability than a clinical rating scale. Thus, an additional goal of studying motivational deficits in SZ from an experimental standpoint is to develop better assessment tools.

2.2 What constitutes a behavioral EV signal?

In order to argue the claim that degraded value representations contribute to avolition in schizophrenia, we need to be able to isolate the use of value representations experimentally. From a behavioral standpoint, there are several ways to specifically assess the intactness or aberrance of value representations in human subjects. One way is to probe for preferences between conditioned stimuli, following a feedback-driven acquisition period. This approach is used in many operant learning paradigms, such as probabilistic response (or Go/NoGo) learning (Frank and O’Reilly, 2006; Holroyd et al., 2004; Pessiglione et al., 2006), probabilistic stimulus selection (PSS; Frank et al., 2004; Shanks et al., 2002) and probabilistic reversal learning (Cools et al., 2002). Learning, in the context of such paradigms, is driven by mismatches between expected and obtained outcomes, called reward prediction errors (RPEs; Glimcher, 2011; Montague et al., 2004). Choices that lead to better-than-expected outcomes, or positive RPEs, facilitate those choices in those contexts – a process called “Go-learning” (Frank et al., 2004). Choices that lead to worse-than-expected outcomes, or negative RPEs, decrease the likelihood of making those choices in those contexts – a process called “NoGo-learning” (Frank et al., 2004). Prediction-error-driven learning happens on multiple time-scales, with rapid, often single-trial, learning known to rely on intact functioning of the prefrontal cortex, or PFC and gradual, more procedural processes, reliant of signaling in the basal ganglia, or BG (Collins and Frank, 2012; Frank and Claus, 2006; see Figure 1A). Additionally, one can isolate the contribution of expected value to performance deficits by manipulating the expected value of a choice, while holding RPE valence and magnitude constant.

Figure 1.

(A) Taxonomy of reinforcement learning mechanisms and their hypothesized neural substrates (see Frank and Claus, 2006, for details). Abbreviations: PE, prediction error; PFC, prefrontal cortex; VM, ventromedial; VL, ventrolateral; DM, dorsomedial. (B) Sample acquisition pairs from the Gain vs. Loss-avoidance probabilistic stimulus selection task. Individuals learn to choose the best stimulus from each of four pairs, based on the reinforcement probabilities of each stimulus. In two pairs, choices resulted in either a gain or a neutral outcome. In two other pairs, choices resulted in either a loss or a neutral outcome. Arrow thickness reflects probability of specific outcome. Green arrows denote gain, red arrows denote loss, blue arrows denote neutral outcome. (C) Sample transfer pairs from the Gain vs. Loss-avoidance probabilistic stimulus selection task. By prompting subjects to choose between stimuli associated with the same frequency of positive prediction errors, but difference expected values (e.g., between “Frequent Winner” stimuli vs. “Frequent Loss-avoider” stimuli), one can isolate the influence of expected value on choice. Abbreviations: PE, prediction error; EV, Expected Value.

Gold and colleagues (2012) have provided one example of how this might be done, using a paradigm where individuals learn to choose the best stimulus from each of four pairs, based on the reinforcement probabilities of each stimulus. This task involved the presentation, in an acquisition phase (Figure 1B), of two kinds of probabilistic discriminations in a pseudo-random order: 1) “Gain” pairs, involving learning to choose a frequently-rewarded stimulus over one frequently leading to a neutral outcome, and 2) “Loss-avoidance” pairs, involving learning to choose a stimulus frequently leading to a neutral outcome rather than a loss. By prompting subjects to choose, in a transfer phase (Figure 1C), between stimuli associated with the same frequency of positive prediction errors, but difference expected values (e.g., between “Frequent Winner” stimuli vs. “Frequent Loss-avoider” stimuli), one can isolate the influence of expected value on choice.

Alternatively, one can isolate the contribution of expected value to performance deficits by using a computational model of reinforcement learning to model RPE valence and magnitude, as well as expected value, on a trial-by-trial basis. These algorithms are typically derived from the work of Rescorla and Wagner (1972) and Sutton and Barto (1998). In computational models of reward learning and decision making, the contribution of the basal ganglia system is often formalized using an “actor-critic” framework or a “Q learning” framework (Sutton and Barto, 1998). In an actor-critic model, a “critic” evaluates the reward values of particular states, and the “actor” selects responses as a function of learned stimulus-response weights. In a Q learning framework, by contrast, the model learns expected quality (“Q value”) of each action separately, as opposed to representing the value of the “state”. Actions are then selected by comparing the various Q values of each candidate action and probabilistically choosing the largest one. Both algorithms depend on the signaling of prediction errors (mismatches between expected and obtained outcomes) to update value representations. However, whereas the actor in the actor-critic scheme does not consider the outcome values of competing actions, the Q-learning scheme makes these fundamental, updating value representations for alternative choices in context (Gold et al., 2012). There is compelling evidence that the orbitofrontal cortex (OFC) has a critical role in representing these kinds of value representations (Plassmann et al., 2010; Roesch and Olson, 2007; see also the chapter by Bissonette and Roesch in this volume). Certain algorithms allow researchers to estimate prediction error valence and magnitude, as well as expected value, on a trial-by-trial basis. As a consequence, it is possible to distinguish the contribution of an aberrant RPE-driven learning mechanism from the contribution of faulty input to that mechanism, in observed non-normative behavior.

2.3 What constitutes a neural EV signal?

The representation of value in the brain – like most representations and processes in the brain – is likely to be distributed across multiple nodes (Knutson et al., 2005). A growing scientific literature has, in fact, supported the idea that value representations reside in fronto-striatal circuits, centered on ventral striatum (VS) and the ventral and medial aspects of prefrontal cortex (PFC), in particular (Kahnt et al., 2010; Kahnt et al., 2014; Smith et al., 2014; Takahashi et al., 2009). Whereas studies in nonhuman subjects have primarily used electrophysiology to identify value representations in the brain, studies with human subjects have largely used functional Magnetic Resonance Imaging (fMRI), with Monetary Incentive Delay (MID) paradigms being the most commonly used (Knutson et al., 2001; Knutson et al., 2003). Monetary Incentive Delay (MID) paradigms are thought to isolate representations of expected value in the brain, because they are designed with temporal intervals between cues predictive of outcomes and deliveries of outcomes that are long enough and variable enough to model each separately in a functional MRI study (Figure 2). The success of MID paradigms in linking EV representations to model blood-oxygen-level-dependent (BOLD) signals in the VS (Knutson et al., 2001; Knutson et al., 2003), led to the application of similar paradigms to the study of reward anticipation in schizophrenia (Juckel et al., 2006b; Walter et al., 2009; Waltz et al., 2010). It should be noted, however, that both studies in nonhuman animals (Roesch and Olson, 2004; Schoenbaum and Roesch, 2005; Schoenbaum et al., 2009) and more recent studies with human subjects (Kahnt et al., 2010; Kahnt et al., 2014) have emphasized the role of OFC in the representation of value. While OFC has historically been a difficult brain region to capture with fMRI, newer behavioral paradigms and imaging techniques have made this task easier. As a consequence, current and future studies of the neural representations of value in schizophrenia should account for within- and between-group variability in task-related OFC signals.

Figure 2.

Schematic of a single trial from a Monetary Incentive Delay (MID) task, with example time intervals. The subject is first presented with a cue, depicting the valence and magnitude of the potential outcome (see inset). Following a variable inter-stimulus interval (ISI; filled in this case by a black screen with a fixation star), a target stimulus (e.g., a white cross on a black background) appears, prompting the participant to respond within a target time window. Following another ISI, the actual outcome is displayed. If a participant responds within the acceptable response window on a gain trial, the total amount of money increments by the associated amount; if not, the total increments by a nominal (small) amount, or not at all. If a participant responds within the acceptable response window on a loss trial, the total amount of money decrements by a nominal (small) amount, or not at all; if the subject is too slow, the subject loses the full predicted amount. Because the interval between the cue and the outcome is variable and long, the cue and outcome regressors are not collinear, and, thus, brain responses to the cue (thought to be associated with outcome anticipation) and outcome receipt can be modeled separately in functional imaging experiments.

A major consideration in isolating EV signals in the human brain is how the task-related MRI data are modeled. Typically, EV signals in fMRI tasks have been identified through experimental manipulations and in contrasts between beta values for conditions in either whole-brain or regions-of-interest (ROI) analyses. Such studies, for example, might employ a paradigm with conditions where the participant might anticipate a monetary gain or loss, or a monetary gain or neutral outcome, or a large gain and a small gain. An effect of such a manipulation on associated beta values is typically taken as evidence for a regions role in coding for value, and a group-difference in the effect of such a manipulation on associated beta values is typically taken as evidence for an abnormality in neural representations of value in a given group.

More recently, however, neuroimaging researchers have begun to take advantage of the kinds of computational modeling techniques mentioned above, in order to estimate EV, as well as RPE valence and magnitude, on a trial-by-trial basis. This allows one to model BOLD signals with parametric regressors (O’Doherty et al., 2007). This has also been done in the case of studies of reinforcement learning in schizophrenia (Gradin et al., 2011), perhaps allowing researchers to pinpoint neural signals associated with EV with more precision. This methodology also enables one to distinguish cue-evoked, or outcome-anticipation, signals, from feedback-evoked, or outcome-integration signals. While outcome processing is associated with neural activity in multiple systems, RPE-signaling, in particular (at least in the case of the delivery and omission of appetitive stimuli), has consistently been linked to neural activity in the dopaminergic midbrain and its targets in the striatum (McClure et al., 2003; Schultz, 1998; Takahashi et al., 2009). Importantly, while the axiomatic approach described by Caplin and Dean (2008) clearly implicates the striatum in the signaling of appetitive RPEs (Rutledge et al., 2010), it casts doubt on a role for the striatum in the signaling of aversive RPEs, a function more likely to be subserved by the periaqueductal gray (Roy et al., 2014). That is to say: one cannot say with 100% certainty whether punishment-evoked activity in the VS can be interpreted as an RPE signal.

2.4 Distinguishing EV signals from general salience signals

It is one matter to identify neural responses to reward-predicting cues in the brain; it is another matter to link these neural responses specifically to reward prediction, and to distinguish them from other sorts of neural signals, such as salience signals, which might be generated simultaneously (For more information on this topic see Salamone et al in this volume). Because of the difficulty in isolating specific cognitive representations and processes, and because of the overlap in functions attributed to specific brain regions, there is obvious ambiguity as to what can be called an “expected value” signal, and what not. As Kahnt et al. (2014) have noted, for example: in tasks examining only appetitive stimuli, or only aversive stimuli, value signals are perfectly correlated with salience signals.

Multiple research groups have sought to disambiguate value from salience signals in the human brain. In fact, the findings of several groups (Jensen et al., 2007; Zink et al., 2006; Zink et al., 2003) point to a role for the ventral striatum in general salience signaling, rather than exclusively representing the valence of prediction errors. This finding does not preclude a role for VS in signaling RPEs, as prediction errors could be understood as a particular kind of salient event. Ventral striatum may also participate in expected value representation by virtue of inputs from OFC; however, OFC appears to play a more specific role in value representation and does not appear to play a large role in signaling mismatches between expectations and outcomes (Kahnt et al., 2014; Takahashi et al., 2009).

Additional work suggests a role for inferior parietal cortex in the signaling of stimulus-driven salience, as well (Geng and Mangun, 2009; Kahnt et al., 2014). Finally, recent work indicates that anterior insula serves as a hub in a global salience network (with the amygdalae and other structures as nodes; Harsay et al., 2012; Seeley et al., 2007), and is not thought to signal either expected value or signed prediction errors. Thus, while neural responses in anterior insula, the amygdalae, and parietal cortices are often evoked by salient stimuli with incentive value, these responses are more likely to reflect their salience than their incentive value.

2.5 Acquisition of incentive salience vs. on-the-fly computation of EV

It is possible to talk about value-based decision making (DM) in multiple contexts. One sense, which was described above, involved decision making in the context of reinforcement learning, where stimuli acquire incentive value by virtue of their frequent temporal contiguity with valenced outcomes. Another kind of value-based decision making, with which many people are familiar, is the sort of hypothetical decision making based on the instantaneous integration of reward probabilities and magnitudes, first formalized by von Neumann and Morgenstern (1947). This sort of decision-making is the subject of Kahneman and Tversky’s (1979) “Prospect Theory”, the main point of which is to illustrate ways in which human choice is guided by irrational biases. It has also been the subject of several recent neuroimaging studies (De Martino et al., 2006; Tom et al., 2007), which have pinpointed the neural processes associated with online EV computation and the irrational biases that lead to deviations from purely EV-based decision making. In particular, these studies have shown that individual differences in susceptibility to irrational biases (such as disproportionate loss-aversion), correspond to individual differences in decision-related activity in areas such as medial PFC and amygdala. Although we know of no imaging studies of SZ patients, involving hypothetical decision-making situations, such as those described above, several recent behavioral studies (Brown et al., 2013; Heerey et al., 2008; Tremeau et al., 2008) have used such paradigms to investigate the ability to estimate expected value on-the-fly, as well as the influence of irrational biases on this ability, in schizophrenia. The fact that it is possible to make a distinction between the acquisition of incentive salience and the on-the-fly computation of EV begs the question of whether variables related to one type of decision are likely to be more closely tied to clinically-ratable motivational deficits than to the other. We discuss relationships between clinical measures of motivational deficits and variables related to both types of decisions below.

2.6 EV of stimuli vs. EV of actions

Another important distinction to consider in identifying relationships between expected value signaling and clinical ratings of motivational deficits in SZ is whether one means the expected value of a stimulus that has acquired incentive value, or the expected value of a choice. Several recent reviews (Noonan et al., 2012; Rushworth, 2008) have addressed the issue, providing evidence that the expected value of stimuli and choices are subserved by somewhat different neural networks. One version of this dissociation is that ventrolateral areas of PFC are responsible for representing the value of stimuli, whereas medial areas of PFC are responsible for representing the value of actions (Noonan et al., 2010; but see Glascher et al., 2009). A slightly different hypothesis regarding the lateral/medial dissociation in OFC is that the lateral OFC is concerned with the linking of specific stimulus representations to representations of specific types of reward outcome, whereas VMPFC/medial OFC is concerned with evaluation, value-guided decision-making and maintenance of a choice over successive decisions (Noonan et al., 2012). A role for VLPFC in “linking of specific stimulus representations to representations of specific types of reward outcome” may account for the established role of VLPFC in the successful performance of reversal learning tasks (Cools et al., 2002). Noonan et al. (2012) demonstrated that lesions in neither orbitofrontal subdivision caused perseveration; rather lesions in the lOFC made animals switch more. By contrast, lesions in the mOFC caused animals to lose their normal predisposition to repeat previously successful choices, suggesting that the mOFC does not just mediate value comparison in choice but also facilitates maintenance of the same choice if it has been successful. Furthermore, recent findings from Guitar-Masip and colleagues (2011) argue against the claim that cue-evoked activity in the VS reflects cue-evoked reward anticipation (i.e., the value assigned to a stimulus); rather, work from this group indicates that cue-evoked striatal activation reflects action anticipation.

From an experimental standpoint, the type of EV signal being probed may be a function of whether reward delivery in a task is action-dependent, or not. As an example: in the context of a Pavlovian paradigm, a stimulus can acquire value, without the need to respond during training. In the context of an operant learning paradigm, a response is required, and the response, in context, is assigned value. For that reason, it is likely that, when we are talking about EV, with regard to probabilistic response learning, or probabilistic reversal learning, paradigms, we are talking about the EV of actions in context. Furthermore: if VS and medial PFC are more critically involved in action-valuation than stimulus-valuation, then aberrant cue-evoked responses in VS and medial PFC are more likely to point to an abnormal ability to assign value to actions than assign “incentive salience” to stimuli. Furthermore, this suggests that an abnormal ability to assign value to actions would likely be a better model for clinical avolition then an abnormal ability to assign “incentive salience” to stimuli.

3 Integrating EV with cost considerations to guide decision making

3.1 Distinguishing wanting from willingness to work

Decision-making requires one to not only estimate the benefits of actions, but also the costs of actions. Thus, even stimuli that have acquired incentive salience – that have become the objects of wanting – require a willingness to overcome the estimated effort cost of an action. The willingness to expend effort has been the subject of recent investigation, with regard to regard to motivational deficits, in general (Treadway et al., 2009), as well as avolition/anhedonia within psychiatric disorders, such as major depressive disorder (Treadway et al., 2012), chapter by Treadway in this Volume) and schizophrenia (Gold et al., 2013). From a brain mapping standpoint, however, it is very difficult to distinguish the neural substrates of “reward anticipation”, or “wanting”, from the neural substrates of “willingness to work”. Dorsal anterior cingulate cortex (dACC) has been implicated in the representation of both the costs and benefits of actions (Kennerley et al., 2006; Walton et al., 2006), as well as conflict monitoring and resolution, with regard to action selection (Ridderinkhof et al., 2004; Rushworth et al., 2004; van Veen and Carter, 2002). The implication of these studies is that the task of isolating a “selective deficit in representing the expected value of actions”, or the specific contribution of action value representations to avolition, will likely be very difficult. By contrast, isolating the neural substrates of “a deficit in the ability to integrate the prospective costs and benefits of actions” may prove more tractable.

3.2 Distinguishing wanting from willingness to wait

Related to the concept of the “effort cost” of an action, and the “effort discounting” of the value of an action, is the concept of the “delay discounting” of the value of an appetitive stimulus. This refers to the rate at which the value of a stimulus is discounted as a function of time, and is usually estimated by way of queries such as, “Would you rather have $10 now, or $100 in two weeks?” Like effort-cost paradigms and devaluation paradigms, these measures assess not only in-the-moment valuation of stimuli, but also valuation as a dynamic, time-varying process. Steep devaluation of stimuli with time is often taken as a measure of impulsivity (Holt et al., 2003; Kirby and Santiesteban, 2003), associated with OFC lesions, in particular (Rudebeck et al., 2006). Because schizophrenia has also been hypothesized to involve maladaptive decision making, possibly brought on by the abnormal valuation of stimuli and actions, delay discounting paradigms have also been administered to SZ patients (Avsar et al., 2013; Heerey et al., 2007). Such studies are described in the following section.

4 Evidence for faulty EV signaling in SZ

Thus, as Figure 1 illustrates, one can be speaking of a number of different cognitive and physiological processes, when one refers to expected value signaling:

• the ability to estimate EV on the fly (by integrating the probabilities and magnitudes of various outcomes);

• the signaling of the acquired incentive value of a stimulus in the environment;

• the signaling of the learned value of an action/choice; and

• the signaling of the learned value of an action/choice, relative to its estimated effort cost.

In the following sections, we will discuss evidence for and against the disruption of each of these aspects of valuation in schizophrenia patients, in general, as well as in subgroups of SZ patients with prevalent primary negative symptoms.

4.1 Behavioral and modeling evidence for faulty EV signaling in SZ

4.1.1 Probabilistic discrimination learning in schizophrenia

A number of behavioral studies (Waltz et al., 2007; Waltz et al., 2011) have used probabilistic RL paradigms like the Probabilistic Stimulus Selection task (described in Section 2.2) to investigate possible relationships between aspects of RL performance and clinically-rated avolition/anhedonia in schizophrenia. These studies have generally found that SZ patients, as a group, have shown impaired (positive-RPE-driven) Go-learning, of the sort thought to rely on the basal ganglia. However, these impairments in Go-learning have not always been found to be most severe in SZ patients with the highest clinical ratings for motivational deficits – as would be theoretically-attractive (Waltz et al., 2007; Waltz et al., 2011). Systematic relationships with clinical ratings for motivational deficits have, however, been observed with measures of (supposedly PFC-driven) trial-to-trial adjustments, like win-stay and lose-shift rates (Waltz et al., 2007; Waltz et al., 2011). An association between an aspect of RL performance and negative symptoms (which include motivational deficits) was also observed by Farkas et al. (2008), who administered the Rutgers Acquired Equivalence Test (AET) to SZ patients and controls. Specifically, these authors found that feedback-driven acquisition performance correlated with negative symptom scores and was impaired only in deficit syndrome SZ patients. Additional evidence for a specific impairment in feedback-driven acquisition performance in deficit SZ patients comes from a related study by the same group (Polgar et al., 2008), using the “chaining” association task originally developed by Shohamy et al. (2005) and Nagy et al. (2007).

The Iowa Gambling Task (IGT; Bechara et al., 1994; Bechara et al., 1997) is another canonical probabilistic RL task, where subjects receive variable rewards and punishments based on their choices. In this task, subjects can choose gambles from four different decks. Two of the decks offer $100 rewards, and two offer $50 rewards. However, the decks offering the higher rewards also involve large potential losses and choosing these higher paying decks is ultimately disadvantageous. Thus, the choice of the decks with smaller rewards turns out to be more advantageous. Learning these contingencies typically requires an extended period of sampling across decks, as the frequency and magnitude of the punishments vary across decks. Rather remarkably, the initial sample of patients with extensive frontal lesions showed a robust preference for the higher-paying but ultimately disadvantageous decks, and appeared to be almost totally indifferent to punishment (Bechara et al., 1994). Thus, it appeared that their behavior was driven by reward seeking alone, as if the punishments simply failed to occur.

The IGT has been administered in numerous studies of decision making and reinforcement learning in schizophrenia (Kester et al., 2006; Kim et al., 2009; Kim et al., 2012; Lee et al., 2007; Ritter et al., 2004; Sevy et al., 2007; Shurman et al., 2005; Wilder et al., 1998), with mixed results. While SZ patients do not show the dramatic insensitivity to punishments observed in OFC-lesion patients in the early studies of Bechara et al. (1994), SZ patients often show a reduced ability to learn to choose good decks more frequently over time (Kester et al., 2006; Shurman et al., 2005). However, genuine insensitivity to punishments – which may be characteristic of some patient groups, such as those with bipolar affective disorder (Brambilla et al., 2013; Burdick et al., 2014) or orbitofrontal lesions (Bechara et al., 1994; Bechara et al., 1999) – does not appear to be the best explanation of poor IGT performance in SZ patients. Rather, recent evidence (Brambilla et al., 2013) indicates that poor performance on the IGT most likely stems from impairment in the ability to use feedback to adaptively update estimations of choice value. Specifically, SZ patients appear to show intact sensitivity to the frequency of losses, but are less able than control subjects to integrate information about the frequency and magnitude of gains and losses in order to form adaptive representations of expected value (Brown et al., 2015).

Finally, the Balloon Analog Risk Task (BART; Lejuez et al., 2003) is another experimental paradigm designed to examine decision making under risk. In this task, subjects “inflate” a balloon using the space bar on the computer. As the balloon gets bigger, the potential reward gets bigger. The balloon pops randomly somewhere between the first and the 128th potential press, such that the optimal strategy to maximize gains would be to press 64 times each time, thereby ensuring the fewest pops coupled with the maximal retained gains. Subjects are presented with three blocks of 30 trials, during which they can learn to optimize their behavior. Thus, unlike the IGT, eventual loss is certain in the BART, and the question is how much risk subjects are willing to take to increase the magnitude of their reward. Prior studies with the BART have found that several clinical populations with impulse control deficits (such as individuals with substance dependence and individuals with attention deficit disorder; Aklin et al., 2012; Mantyla et al., 2012) show abnormal risk seeking, whereby reward seeking drives behavior in the face of ultimately certain punishment. In three studies using the BART in SZ, patients showed reduced tolerance for risk (fewer pumps), relative to controls (Brown et al., 2015; Cheng et al., 2012; Reddy et al., 2014). That is, they appeared to be abnormally sensitive to the prospect of a punishment and settled for reduced gains.

In sum, findings from the above-described studies point to deficits in multiple processes subserving the successful performance of probabilistic RL in SZ patients, but leave open the questions of how closely these RL deficits relate to motivational impairments in SZ. Recent findings, however, appear to indicate that motivational impairments in SZ relate more closely to PFC-driven rapid/explicit RL processes than BG-driven gradual/procedural RL processes. These observations fit with observations that: 1) rapid/explicit RL processes relate closely to PFC-dependent working memory processes (Collins and Frank, 2012); and 2) strong correlations between performance on RL tasks and standard measures of working memory/executive function (Brown et al., 2015).

4.1.2 Studies of in-the-moment value estimation in schizophrenia

A limited set of findings (Brown et al., 2013; Heerey et al., 2008; Tremeau et al., 2008) points to an impaired ability in schizophrenia patients to estimate expected value on-the-fly, in hypothetical decision-making situations. However, while data support a connection between the instantaneous computation of expected value and intellectual function in schizophrenia (Brown et al., 2013; Heerey et al., 2008), we are not aware of data supporting a link between the instantaneous computation of expected value and motivational deficits in schizophrenia. When the time of reward delivery is a factor in decision making, however, data do support a relationship between decision-making performance and motivational deficits in schizophrenia (Heerey et al., 2007). That is: when the decision-making situation calls for the adjustment of value representation, based on the fact that potential rewards would be delivered at some point in the future, rather than immediately, one sees systematic relationships between decision-making performance and ratings of motivational deficits in SZ patients (Heerey et al., 2007). This effect was revealed in a study of delay discounting, in which Heerey et al. (2007) found that patients discounted more steeply than did comparison participants, and that discounting among patients related to both memory capacity and negative symptoms. Taken as a whole, studies of in-the-moment value estimation in SZ suggest that the online estimation of EV depends heavily on intellectual resources (such as working memory), whereas the discounting of the value of future rewards is a process that relates closely to avolition and anhedonia.

4.1.3 Devaluation and extinction experiments

Another approach to investigating the integrity of value representations in schizophrenia has been to use devaluation experiments, whereby a rewarding stimulus, or a stimulus or action predictive of a reward, become less valued, as a consequence of satiety or an aversive event. Such a paradigm may involve conditioning, whereby a stimulus, or a stimulus-response pairing, becomes associated with an aversive outcome across encounters with the stimulus. Alternatively, a reinforcer (usually a primary reinforcer) may lose value as a consequence of satiety – not because the stimulus has become associated with an aversive event.

Holt et al. (2009) used a contextual fear-conditioning paradigm to examine extinction learning and recall in SZ patients. In this paradigm, individuals were conditioned to expect a finger shock following a CS (a lit lampshade) in one context (e.g., an office), but not in another (e.g., a conference room). Among individuals showing autonomic responsivity, Holt and colleagues (2009) found no group-differences in conditioning. That is, patients learned to associate a CS in context with a US (the finger shock) just as well as controls. However, responders in the SZ group showed impaired extinction recall, relative to controls, in that they exhibited a stronger association between the CS+ and the US in the extinction learning context than controls did. This abnormality in extinction recall in SZs was found to correlate with the severity of psychotic symptoms, but the authors reported no significant relationships with negative symptoms. In a second study, Waltz et al. (2015) used a sensory-specific satiety (SSS) paradigm to examine associations between measures of value-updating in schizophrenia and clinical ratings of negative symptoms. The paradigm involved feeding subjects small squirts of liquid foods (V8 juice and chocolate hazelnut drink), as well as a control solution, in a pseudo-random order, using syringes. In each of 2 sessions, subjects received 16 squirts of each rewarding food and 32 squirts of the control solution. In between the 2 sessions, each subject was instructed to drink one of the foods (determined through counterbalancing) until he/she felt “full, but not uncomfortable”. At ten regular intervals, interspersed throughout the 2 sessions, subjects rated each liquid from 0 to 100, using a Likert-type scale. We observed group differences in SSS effects, such that controls showed an effect of satiety that was sensory-specific, but patients showed an effect of satiety that was not. Furthermore, in SZ patients, we observed correlations between the magnitude of SSS effects and measures of anhedonia and avolition. Both of these results indicate that patients with SZ have an impairment in the ability to flexibly and rapidly update representations of the value of stimuli and actions. Our study of SSS in SZ patients, however, specifically indicates that the ability to flexibly and rapidly update representations of the value of stimuli figures critically in the adaptive motivation of goal-directed behavior.

4.2 Neural evidence for faulty EV signaling in SZ

Neural correlates of reward- (or punishment-) anticipation are often operationalized as brain responses to reward- (or punishment-) predicting cues. Many of these reports are on studies using Monetary Incentive Delay (MID) paradigms, but other studies assessed the expression of preferences among stimuli following conditioning. Some of these conditioning studies involved the application of computational model to behavioral data, in order to estimate expected value on a trial-by-trial basis. In this section we review these different kinds of studies.

4.2.1 Investigating hypothetical decision making in SZ with MRI

We know of a single study of hypothetical decision making in SZ with MRI. In this study, involving the use of a delay discounting (DD) paradigm, Avsar et al. (2013) found that, when compared with controls matched for consistency of preferences, SZ patients showed reduced activation during performance of the DD task in executive function and reward areas, such as the inferior frontal gyrus, dACC, posterior parietal cortex, and VS, thalamus, and midbrain. Furthermore, SZ patients had abnormal activation of lateral and medial frontal regions in relation to trial difficulty.

4.2.2 Making sense of MID results

As noted above, MID paradigms have been used numerous times in MRI studies involving schizophrenia patients, with findings seeming to depend, to some extent, on the medication status of patients (Juckel et al., 2006a; Juckel et al., 2006b; Walter et al., 2009). Studies involving chronic, medicated SZ patients have not always found evidence of blunted reward anticipation responses (most commonly sought in VS) in entire samples of SZ patients, although Walter et al. (2009) found that patients and controls differed in their anticipatory responses in anterior cingulate cortex, consistent with a deficit in action valuation. Several studies (Simon et al., 2010; Waltz et al., 2010) found that the blunting of reward anticipation responses correlated with clinical ratings of avolition and anhedonia in SZ patients. More recently, Mucci et al. (2015) found the reward anticipation signals in the dorsal caudate correlated with avolition in schizophrenia.

In general, one could interpret the results of studies using MID paradigms as pointing to aberrant EV-related signals in patients with schizophrenia. More specifically, however, reports on studies using MID paradigms have revealed correlations between measures of avolition and anhedonia and EV-related signals in both striatum and PFC.

4.2.3 Conditioning experiments

Another set of studies attempted to examine expected value signaling in schizophrenia by using neuroimaging in conjunction with conditioning paradigms, such that stimuli or actions acquired value over the course of the experiment. In classical/Pavlovian conditioning paradigms, stimuli acquired value, because they predict/co-occur with rewards or punishments throughout a learning phase of an experiment. In such experiments, there is nothing that the subject needs to do in order to receive a reward or punishment; they only observe the temporal associations between stimuli and outcomes.

In an fMRI study of classical conditioning in schizophrenia, Jensen et al. (2008) trained participants to expect an aversive outcome after the presentation of a circle of a certain color. Whereas one colored circle (the CS+) was followed by a loud noise on 50% of trials, another colored circle (the CS-) was followed by the visual presentation of a star on 50% of trials. These authors observed that SZ patients showed inappropriately strong activations in the VS in response to the neutral stimulus (CS-) as compared to the healthy controls. That is: patients with SZ activated VS in such a way as to indicate that they had assigned motivational salience to the neutral cue, and not the cue paired with the salient/aversive event. Despite the fact that all patients in the study were stably-medicated, and no associations were observed between VS responses and clinical symptoms, this finding was primarily interpreted by the authors to indicate that the aberrant processing of salient events predisposes patients to psychosis. No group-differences were observed in adaptive salience signaling in the VS.

Romaniuk and colleagues (2010) designed an experiment in which stimuli (a yellow or blue background) were associated with neutral or aversive images from the International Affective Pictures Set (IAPS; Lang et al., 2005). One conditioned stimulus CS (CSav) was paired on 50% of trials with the presentation of an aversive IAPS picture. The second CS (CSneu) was paired on 50% of trials with an emotionally neutral IAPS picture. The authors included parametric regressor for both cues and outcomes to assess brain signals for both expected value and reward prediction errors. Romaniuk et al. (2010) found that patients with schizophrenia showed abnormal activation of the amygdala, midbrain, and VS during conditioning, such that these brain regions failed to distinguish between cues to aversive outcomes and cues to neutral outcomes (as they did in healthy volunteers). Furthermore, activation of the midbrain in response to neutral rather than aversive cues during conditioning was correlated with the severity of delusional symptoms in the patient group.

In a study using two separate classical conditioning paradigms (one involving a primary reinforcer and one involving a symbolic/monetary reinforcer), Dowd and Barch (2012) found that patients with schizophrenia exhibited striatal responses to appetitive cues that were largely intact. However, these authors (Dowd and Barch, 2012) observed that striatal responses to appetitive cues correlated with self-reports of trait anhedonia in both patients and controls. Furthermore, Individual difference analyses in patients revealed an association between physical anhedonia and activity in ventromedial prefrontal cortex during anticipation of reward, such that greater anhedonia severity was associated with reduced activation to money versus no-money cues, in both controls and patients. These findings suggest that attenuated EV signaling may be a marker for anhedonia, regardless of diagnosis, as opposed to motivational deficits in schizophrenia, in particular.

Several groups have used operant conditioning paradigms to assess the intactness or disruption of value signals in individuals with schizophrenia. Koch et al. (2010), for example, used a reinforcement learning paradigm in which participants were presented with three sets of probabilistic contingencies: one highly uncertain condition that did not allow any outcome prediction (i.e. 50% stimulus-outcome contingency), one condition permitting full prediction (i.e. 100% stimulus-outcome contingency), and one condition where prediction was partly possible (i.e. 81% stimulus-outcome contingency). Each probabilistic contingency involved the presentation of a card with a geometrical figure on it (i.e. circle, cross, half-moon, triangle, square, or pentagon), Participants were told that each figure was associated with an unknown value ranging from 1 to 9, and that each figure predicted the respective value (higher or lower than five) with a certain probability. The participant was asked to guess whether the figure on the card predicted a value higher or lower than the number five and told that each correct guess was followed by a monetary reward (+0.50 €), whereas each wrong guess was followed by a punishment (−0.50 €). The whole paradigm consisted of two stimuli in each stimulus category, and 16 trials with each stimulus, for a total of 96 trials. Consistent with the results of many of the studies listed above, Koch et al. (2010) found that patients’ ability to learn contingencies on the basis of feedback and reward was significantly impaired. Furthermore, these researchers observed that the effects of reward probability on neural responses in ACC and DLPFC were modulated by group, such that controls showed steeper (inverse) relationships between BOLD signal activation and reward likelihood. That is: controls showed significantly stronger activation, compared to patients, in association with decreasing predictability, whereas patients showed weaker effects of reward likelihood on the BOLD signal in ACC and DLPFC.

Gradin et al. (2011) used a probabilistic stimulus selection paradigm with a juice reward, in conjunction with computational modeling, to estimate brain responses to both EV and RPE, finding that patients with SZ showed attenuated MRI activity associated with EV in amygdala-hippocampus, as well as posterior hippocampal gyrus, bilaterally. Of note: SZ patients and controls did not differ in EV-associated activity in VS. In fact, EV-associated activity in the left VS was significantly greater in SZ patients than in controls. Furthermore, EV-associated activity in amygdala-hippocampus was found to correlate with psychotic symptoms in SZ patients.

There is also one published report in the literature describing the results of a neuroimaging study using an extinction paradigm in schizophrenia. In this study Holt and colleagues (2012) used a paradigm similar to the one described above in their behavior study (Holt et al., 2009). In their neuroimaging study, Holt et al. (2012) found that, during contextual fear conditioning, SZ patients showed abnormal BOLD responses, relative to control participants, within the posterior cingulate gyrus (PCG), hippocampus, and thalamus, with PCG abnormalities linked to negative symptoms. Although, SZ patients and controls showed comparable neural responses during extinction learning, patients and controls differed in their brain responses 24 hours after the learning phases. Whereas, controls showed increased vmPFC responses in the extinction (safe) context (indicating successful retention of the extinction memory), SZ patients showed blunted vmPFC responses in the safe context. This attenuation was especially pronounced in delusional patients.

Finally, there is one published report in the literature describing the results of a neuroimaging study using a devaluation paradigm in schizophrenia. In this study, Morris et al. (2015) used a Pavlovian-Instrumental Transfer (PIT) paradigm, in order to evoke responding to cues predictive of a food reward. In a later phase of the experiment, the food rewards (crackers) were devalued, by the presentation of images in which cockroaches were crawling on the food rewards. These authors found that, whereas controls developed a strong preference for foods that had not been devalued, SZ patients chose the devalued food just as much as the alternative. This result indicated that SZ patients had a starkly reduced ability to update the values of actions when actions were guided by experienced outcomes. Accordingly, patients in the study showed reduced activity in the caudate, relative to controls, during choices of appetitive stimulus, with the greatest attenuations in caudate activity observed in patients with the most severe negative symptoms.

As shown in Table 1, some neuroimaging studies of conditioning (Dowd and Barch, 2012; Morris et al., 2015) suggest that disruptions in fronto-striatal circuits related to EV-signaling do, in fact, correlate with negative symptoms in SZ patients. In other studies, correspondences between aberrant EV-related fronto-striatal signals and positive symptoms were reported (Holt et al., 2012; Romaniuk et al., 2010). It is not clear whether correspondences between aberrant EV-related fronto-striatal signals and negative symptoms were investigated in these studies. In any case, neuroimaging studies of conditioning provide some of the best evidence that fronto-striatal circuit activity associated with EV-signaling plays a role in schizophrenia psychopathology.

Table 1.

Neuroimaging studies of reward anticipation/expected value in schizophrenia: Effects of group and symptom dimension.

Paradigm	Group-diff. in PFC/HC	Group-diff. in VS/MB	Neg. Sx. Effect in PFC/HC	Neg. Sx. Effect in VS/MB	Pos. Sx. Effect in PFC/HC	Pos. Sx. Effect in VS/MB
Reward anticipation/Expected value
MID	Walter et al. (2009)	Juckel et al. (2006a)*		Simon et al. (2009)	Walter et al. (2009)
		Juckel et al. (2006b)**		Waltz et al. (2010)
				Mucci et al. (2015)
Pavlovian Conditioning	Holt et al. (2012)	Jensen et al. (2008)	Dowd & Barch (2012)	Dowd & Barch (2012)	Holt et al. (2012)	Romaniuk et al. (2010)
		Romaniuk et al. (2010)
Instrumental Conditioning	Koch et al. (2010)				Gradin et al. (2011)
	Gradin et al. (2011)
Pavlovian-Instrumental Transfer		Morris et al. (2015)		Morris et al. (2015)
Delay Discounting	Avsar et al. (2013)	Avsar et al. (2013)
Other
Electrical shock	Linnman et al. (2013; Insula)				Linnman et al. (2013; Insula)
Sternberg Item Recognition Paradigm	Ehrlich et al. (2012)			Ehrlich et al. (2012)

^*= Study in unmedicated psychosis patients

^**= Comparison of patients medicated with first- and second-generation antipsychotics

Abbreviations: diff., difference; PFC, prefrontal cortex; HC, hippocampus; VS, ventral striatum; MB, midbrain; Neg. Sx., negative symptoms; Pos. Sx., positive symptoms; MID, Monetary Incentive Delay

4.2.4 Conclusions regarding evidence for faulty EV signaling in SZ

When we revisit the various ways in which value representations can influence behavior, we conclude that the ability to estimate EV on the fly (by integrating the probabilities and magnitudes of various outcomes) is disrupted in schizophrenia, but does not systematically relate to motivational deficits. By contrasts, there is considerable behavioral and neuroimaging evidence in the literature that abnormalities in the signaling of the acquired incentive value of stimuli in the environment relate systematically to clinical and self-report ratings for motivational deficits, in medicated patients with schizophrenia. It should be noted, however, that abnormalities in the signaling of the acquired incentive value of stimuli in the environment have not been found to relate solely to measures of motivational deficits. A finding common to multiple studies (e.g., Jensen et al., 2008) was that SZ patients activated dopaminergic brain regions to neutral stimuli in a way indicating that inappropriate motivational salience had been assigned to those stimuli. Although not every neuroimaging study of value representation in SZ study reported results bearing on the question of whether the ability to adaptively assign value to stimuli related to motivational deficits (Holt et al., 2012; Romaniuk et al., 2010), some systematic relationships between behavioral/brain signals and clinical measures of motivational deficits in SZ have been reported – particularly in the context of paradigms specifically probing the signaling of the learned value of an action or choice (Morris et al., 2015). Finally, there is behavioral evidence that motivational deficits in SZ are associated with deficits in the ability to estimate the costs of actions (Gold et al., 2013).

5 Avolition and outcome processing in schizophrenia

While a deficit in hedonic experience was long considered a core feature of schizophrenia (Bleuler, 1950/1911; Kraepelin, 1919), this view has been challenged in recent years by researchers who argue that the hedonic deficit in schizophrenia is primary one of pleasure anticipation (Kring and Neale, 1996). As noted in the introduction, the preponderance of behavioral self-report studies support this view, indicating that SZ patients do not differ from controls in their self-reported experience of pleasure (Cohen and Minor, 2008). Nonetheless, more recent studies, especially those using neuroimaging techniques, have painted a more complicated picture. While a deficit in hedonic experience is still not thought to be a core feature of schizophrenia, there is some evidence that measures of consummatory anhedonia and neural correlates thereof, relate to avolition in schizophrenia. Whether a deficit in hedonic experience might be the primary driver of avolition in SZ (rather than a deficit in expected value – the ability to anticipate rewards) is the subject of the following sections.

5.1 Behavioral studies of outcome processing in schizophrenia

The interpretation of the synthesis of self-report studies, by Cohen and Minor (2008), is based on studies using both experience sampling (Gard et al., 2007) and experimental behavioral paradigms. A study from our group (Heerey and Gold, 2007), for example, involved the use of a paradigm where participants were presented with images from the IAPS (Lang et al., 2005), and prompted to respond in multiple ways: 1) by rating the degree to which each slide was experienced as pleasurable and arousing using 9-point Likert scales anchored by “extremely [unpleasant/calm]” and “extremely [pleasant/arousing]”; 2) by repeatedly pressing one set of keys if they wanted to see pleasant images again, or repeatedly pressing another set of keys if they did not want to see unpleasant images again; and 3) by repeatedly pressing one set of keys to extend the display time of pleasant images, or repeatedly pressing another set of keys to reduce the display time of unpleasant images. In this study, we found that, relative to controls, SZ patients showed a weaker correspondence between their hedonic ratings of images and their goal-directed responding, both when it was “representational” (for the purpose of making images return, or stay away) and “evoked” (for the purpose of extending or reducing the display time of images). Additionally, representational responding was predicted by self-reported social anhedonia, confirming our hypothesis that anhedonia might relate to faulty representations of reward value.

It should be noted that the results of experimental studies do not exclusively point to an intact experience of pleasure in schizophrenia. First, a fraction of the studies described in the Cohen and Minor (2010) meta-analysis actually are suggestive of a consummatory hedonic deficit in schizophrenia patients. Second, a number of studies reporting the lack of a group-difference between the entire sample of SZ patients and the entire sample of controls in consummatory hedonic ratings include subsets of patients who do differ from controls in consummatory hedonic ratings. Third, the experience of reinforcers through the gustatory and olfactory modalities may be specifically disrupted in schizophrenia (Crespo-Facorro et al., 2001; Moberg et al., 1999). This may be partially due to degraded discriminatory abilities by these modalities (Turetsky et al., 2003), although, as described below, there is also evidence of attenuated hedonic responses in the presence of intact sensory discrimination in SZ (Plailly et al., 2006).

5.2 Neuroimaging studies of reward and punishment receipt in schizophrenia

Results from neuroimaging studies are perhaps more mixed than those from behavioral studies. In fact, findings from a number of neuroimaging studies suggest that brain responses underlying pleasurable emotional experiences are aberrant in SZ (Paradiso et al., 2003; Plailly et al., 2006; Taylor et al., 2005). Plailly and colleagues (2006), for example, presented subjects with 48 different odorants during 8 Positron Emission Tomography (PET) scans. Subjects had either to detect odor, or to judge odor familiarity or hedonicity. Plailly et al. (2006) found that SZ patients and controls did not differ in their ability to detect suprathreshold odorants, but patients found odors less familiar, and pleasant odors less pleasant than controls. These behavioral results were related to reduced regional cerebral blood flow (rCBF) in patients, in posterior piriform cortex and orbital regions for familiarity judgments, the insular gyrus for hedonicity judgments, and the left inferior frontal gyrus and anterior piriform cortex/putamen region for the three olfactory tasks. In another study, Dowd and Barch (2010) had SZ patients and controls undergo fMRI scanning while making valence and arousal ratings in response to emotional pictures, words, and faces. These authors found that functional activity was largely intact in the sample of patients, as a whole, except for regions in right VS and left putamen, which showed reduced responses to positive stimuli.

Studies using MID paradigms have also been a source of findings regarding brain responses to outcomes in schizophrenia. Note that, in studies using the MID, it is difficult to specifically label outcome responses as RPE signals, because unexpected outcomes are also generally salient, and participants are trained on reward contingencies prior to MRI scanning, and thus there is no learning to be modeled. Group differences in outcome-evoked brain responses have been observed in several of these studies, but these group differences have tended to be localized in cortical areas, such as the anterior insula/VLPFC, medial PFC, lateral temporal cortex, and amygdala (Walter et al., 2009; Waltz et al., 2010).

The main issue, however, for the purpose of this review, is whether reward-related signals in fronto-striatal circuits have been shown to correlate with the severity of symptoms in schizophrenia. Multiple studies have observed significant correlations between outcome responses in the brain and ratings of delusional symptoms (e.g., Schlagenhauf et al., 2009). Multiple studies have observed significant correlations between outcome responses in the brain and ratings of motivational deficits in SZ, as well. Work from our group (Waltz et al., 2009), for example, has tied the experience of a gustatory reinforcer (juice) to motivational deficits in SZ.. We observed that avolition scores from the SANS correlated significantly with activity in the primary gustatory cortex and putamen at the time of unsurprising deliveries of the reinforcer (Waltz et al., 2009). In a study using an MID task, Simon and colleagues (2010) found that depressive symptoms (from the Calgary Depression Scale; Addington et al., 1992; Muller et al., 1999) were predictive of the magnitudes of the [REWARD RECEIVED – REWARD OMITTED] contrast in the VS in individual SZ patients, VS responses in patients with higher ratings for depression distinguished less between reward deliveries and omissions than they did in less depressed patients. In our study using an MID paradigm (Waltz et al., 2010), described above, we also assessed correlations between negative symptoms in SZ and neural responses to outcomes, finding that ratings of avolition/anhedonia (from the SANS) in patients correlated with sensitivity to obtained losses in medial PFC (Waltz et al., 2010). Thus, in at least two studies, neural responses to the experience of pleasant stimuli in the striatum were correlated with measures of motivational deficits in SZ, while in a third, neural responses to the experience of pleasant stimuli in VMPFC were correlated with measures of motivational deficits. In a more recent study, our group also observed a correlation between clinical ratings of avolition/anhedonia and the magnitude of [PUNISHMENT – REWARD] contrasts in the VS (Waltz et al., 2013). Specifically, in the context of the performance of a probabilistic reversal learning task, patients with higher ratings for avolition/anhedonia showed less VS deactivation for punishments patients with lower ratings. Finally, in their study using emotional pictures, words, and faces as stimuli, (Dowd and Barch, 2010) found that higher anhedonia scores were associated with reduced activation to positive versus negative stimuli in bilateral amygdala and right VS in SZ patients.

Findings from at least one recent neuroimaging study provides strong evidence that, in the absence of behavioral response requirements, immediate brain responses to pleasurable visual stimuli are largely intact in medicated individuals with chronic schizophrenia. Ursu and colleagues (2011) examined the brain activity of SZ patients and controls during trials in which they viewed an affective picture and, after a delay, reported their emotional experience while viewing it. These authors (Ursu et al., 2011) found that, in the presence of emotional stimuli, SZ patients and controls exhibited brain activity that was similar. By contrast, patients showed decreased activation, relative to controls, in dorsolateral, medial, and ventrolateral prefrontal cortices, during the delay. Importantly, the delay-related response of the DLPFC to pleasant stimuli correlated negatively with SANS anhedonia scores.

5.3 Neuroimaging studies of RPE-signaling in SZ

The observation of aberrant signals related to both anticipatory and consummatory hedonics in SZ does not preclude the possibility that prediction error computation is also aberrant in SZ (and vice versa). In theory, inaccurate, or maladaptive, EV representations could reflect a reduced ability to translate hedonic experience into the expectation of value. That is: a reduced ability to update representations of expected value could be the result of faulty learning mechanisms (presumably reinforcement learning algorithms driven by reward prediction errors, or RPEs). Establishing, however, that the actual mechanism of computing the mismatch between expected and experienced outcomes is abnormal, despite the frequent input of inaccurate or degraded representations, becomes a more difficult task. Isolating these three processes, again, is made possible through experimental design and computational modeling. While the literature, summarized below, does not point to general sparing of RPE signaling in SZ, modeling work indicates that RL deficits persist in the presence of intact RPE signaling.

Studies of acutely-ill patients have found evidence of disrupted RPE signaling in the midbrain (Murray et al., 2007) and VS (Schlagenhauf et al., 2014), with potentially-important implications for RL and belief-formation (Deserno et al., 2013; Heinz and Schlagenhauf, 2010). Neuroimaging studies with chronically-ill schizophrenia patients clearly point to abnormal outcome-related signals in schizophrenia (Jensen et al., 2008; Walter et al., 2010; Walter et al., 2009; Waltz et al., 2009; Waltz et al., 2010), but, as noted above, it is not entirely certain that those abnormal outcome-related signals definitively represent RPE signals. Walter and colleagues (2010; 2009), for example, have demonstrated that healthy volunteers use anterior insula to signal unsigned prediction errors, in the context of MID task performance. That is, in controls, anterior insula indicates when outcomes are either better or worse than expected. By contrast, patients with schizophrenia fail to show this U-shaped response pattern, as a function of RPE valence and magnitude – a result that can be interpreted as a blunted general salience signal in schizophrenia.

The results of several studies using RL paradigms point specifically to the disruption of RPE-signals in the striatum in chronic, medicated SZ patients. Koch et al. (2010), for example, showed that brain responses scale with the magnitude of appetitive RPEs in healthy volunteers in the putamen, and to a much lesser degree in patients with SZ. In the MRI study of learning from Gradin et al. (2011), SZ patients were found to show abnormal outcome-related brain responses in the midbrain and caudate. In this study, in particular, the authors used model-derived trial-by-trial estimates of RPE valence and magnitude in order to construct amplitude-modulated regressor for fMRI data analysis.

Findings from multiple groups indicate that negative RPE-signals in the striatum may be intact in chronic, medicated SZ patients – at least for worse-than-expected outcomes (Walter et al., 2010; Walter et al., 2009; Waltz et al., 2009; Waltz et al., 2010). In their study of classical conditioning, mentioned above, Dowd and Barch (2012) also examined responses to outcomes, finding that, at the time of receipt, SZ patients showed largely intact responses to receipt of reward vs. nonreward in striatum, midbrain, and frontal cortex. Right anterior insula demonstrated greater activation for nonreward than reward cues in controls, and for reward than nonreward cues in patients. At the time of receipt, robust responses to receipt of reward vs. nonreward were seen in striatum, midbrain, and frontal cortex in both groups. Furthermore, both groups demonstrated enhanced responses to unexpected versus expected outcomes in cortical areas including bilateral dorsolateral prefrontal cortex. Responses to reward receipt in dorsal PFC (BA 6; but not in VS) were correlated with physical anhedonia in both patients and controls.

In short: a large body of evidence indicates that the aberrance of RPE signals relates to the severity of psychotic symptoms in SZ, thereby supporting the idea that the inappropriate signaling of mismatches between expected and experienced outcomes may be a path to abnormal learning and, subsequently the formation of delusions that comprise psychosis (Kapur, 2003). There is some evidence that the aberrance of RPE signaling relates to the severity of negative symptoms in SZ. More worked is needed to determine whether the aberrant outcome signals associated with negative symptoms in SZ can be labeled RPE signals by the strictest measures.

5.4 Do abnormalities in consummatory hedonics or RPE signaling account for abnormalities in EV signaling in SZ?

The results cited above present a complicated picture: more often than not, patients appear to report stimuli as pleasurable or aversive in a similar fashion as healthy controls. Frequently, however, neural responses involved in the actual experience of pleasure are found to be altered in SZ patients. This raises the possibility that abnormal neural responses to pleasurable experiences, at the physiological level, may contribute to the motivational deficits observed in schizophrenia, even if patients report normal hedonic experience. Given evidence of both aberrant EV and aberrant outcome signals in medicated patients with schizophrenia, one would be right to wonder whether the two abnormalities ever occur independently, or if one is only epiphenomenal to the other. Presumably, a deficit in “wanting” would reflect a reduced ability to translate hedonic experience into the expectation of value. This reduced ability to update representations of expected value could, in turn, be the result of faulty learning mechanisms (presumably reinforcement learning algorithms driven by reward prediction errors, or RPEs), or the consequence of reinforcement learning algorithms receiving faulty inputs.

Experimental studies of RL in SZ, where expected value was either specifically manipulated as an independent variable, or modeled on a trial-by-trial basis, or both, have been rare. Especially rare have been studies where expected value, as a condition, was crossed with outcome value, and orthogonal with outcome salience. We sought to dissociate the contributions of EV- and RPE-signaling to RL deficits in SZ through the use of computational modeling of behavioral data from an experimental reinforcement learning paradigm (Gold et al., 2012). In this study (Gold et al., 2012), SZ patients and controls performed a probabilistic reinforcement learning task, requiring subjects to learn to choose the best option from each of four pairs of stimuli. In two pairs, the best option resulted in a frequent monetary gain; in the other two pairs, the best option resulted in the frequent avoidance of a monetary loss. Once the pairs were acquired, stimuli were presented in novel combinations in a test phase to assess preferences among stimuli, based on their valence and reinforcement probability. In order to investigate relationships with negative symptoms, patients were split into two subgroups, based on the median rating for avolition/anhedonia in the sample (from the SANS). We (Gold et al., 2012) found that patients with mild negative symptoms exhibited normal preferences, during both acquisition and test, for gain stimuli, relative to neutral stimuli, and for neutral stimuli, relative to loss stimuli. This result pointed to an intact ability in patients with low negative symptoms to learn from reward prediction errors (RPEs). However, when patients with high negative symptoms were compared to either controls or patients with mild negative symptoms they showed normal acquisition for loss avoidance stimuli, but slower acquisition and lower accuracy for gain stimuli. High negative patients also showed a reduced preference for “frequent-gain” stimuli, relative to “frequent-loss-avoidance” stimuli. Therefore negative symptoms were not associated with a general deficit in learning from all reward prediction errors, but more specifically with a reduced ability to represent or utilize expected positive values to guide decisions – a capacity consistently associated with function in the ventral and medial aspects of PFC (Noonan et al., 2012; Rushworth, 2008).

The behavioral findings were elaborated upon by computational modeling analyses of the data. These analyses made use of a hybrid model integrating an (hypothetically striatally-driven) actor-critic mechanism with a Q-learning component (meant to capture the role of OFC in RL; see Section 2.2). Computational modeling of reinforcement learning performance (Gold et al., 2012) suggested that the mechanisms for integrating expected value estimates with information derived from RPE calculations were disabled in patients with severe negative symptoms. That is, the addition of a Q-learning component to the model did not help to account for the behavior of avolitional SZ patients, beyond that of a simple actor-critic model (Gold et al., 2012). It was as if avolitional SZ patients relied entirely on striatally-driven RL mechanisms, with little contribution from OFC-dependent representations of choice value. This account is consistent with the idea that what is disrupted in avolitional SZ patients, with regard to reinforcement learning, is the ability to signal the expected values of choices, and not the ability to compute RPEs by comparing expected to obtained outcomes.

The results of this study are consistent with the recent findings of a separate group, using a similar task (Reinen et al., 2014). This group (Reinen et al., 2014) found that, when compared to controls, the acquisition performance of SZ patients was impaired, especially in a Gain condition, with learning in this also inversely correlated with negative symptom severity.

6 General Conclusions

Our purpose in discussing the issues pertaining to experimental studies of reinforcement learning was to pinpoint the sort of neural signal that one could say, with confidence, constituted an “expected value”, for purposes of relating that signal to clinical measures. The bulk of evidence from basic neuroscience ties the representation of expected value to neural activity in fronto-striatal networks centered on ventral and medial regions of prefrontal cortex. Surprisingly – while many of the results recounted above are consistent with a contribution of OFC dysfunction to motivational deficits in SZ, by way of degraded representations of value, direct evidence of such a mechanism is limited (Barch and Dowd, 2010). Based on the present state of the literature, we have reached the following conclusions and recommendations, regarding the contributions of neural representations of value to motivational deficits in schizophrenia. Based on behavioral and neuroimaging results from the literature, there is reason to believe that the severity of motivational deficits in schizophrenia relates to the ability to precisely represent value in the brain and flexibly update these representations. Many of the reports cited above describe relationships between clinical ratings of motivational deficits and aspects of reinforcement learning related to updating of the prospective value of stimuli and actions. In our view, observations of systematic relationships between clinical ratings of motivational deficits in SZ and abnormalities in other aspects of reinforcement learning, such as blunted consummatory hedonics and aberrant RPE-signals, have been more rare. We do not mean to imply that other factors (such as blunted consummatory hedonics or aberrant RPE-signals) never contribute to motivational deficits in individuals with schizophrenia. That is, there is considerable imaging evidence that blunted consummatory hedonics and aberrant RPE-signals contribute to motivational deficits in at least some individuals with serious mental illness (including schizophrenia). In our own work, we have found evidence of abnormal RPE signaling in patients with severe negative symptoms (Waltz et al., 2013). Whether abnormalities in RPE signaling in subsets of patients with psychiatric illness mediate relationships between EV signals and avolition is an issue that requires further study to resolve.

1. There are multiple findings, from neuroimaging studies, of abnormal striatal reward anticipation signals in SZ patients (or, at least, avolitional SZ patients). Functional imaging studies from the last two decades have most commonly implicated two regions the representation of the expected value in the brain: ventral striatum and ventral PFC (Kahnt et al., 2010; Knutson et al., 2005). In SZ patients, abnormal representations of expected value have most commonly been tied to dysfunction in ventral striatum (Morris et al., 2015; Mucci et al., 2015; Waltz et al., 2010). This could, to some extent, reflect the difficulty in extracting a strong signal from ventral PFC (Deichmann et al., 2003), and it is possible that neuroimaging studies with SZ patients would be more able to evaluate expected value signals in ventral and medial areas of PFC with newer imaging and analytic techniques (Kahnt et al., 2010). There is at least one instance of an observed relationship between a cue-evoked PFC signal (in the context of a classical conditioning paradigm) and a measure of a motivational deficit in SZ (Dowd and Barch, 2012). While theoretically-attractive, we are not aware of any direct evidence tying a cue-evoked PFC signal in the context of an operant learning task (in the dACC, in particular), and a measure of avolition/anhedonia in SZ.

2. Neural signals related to reward anticipation have been found to correlate with measures of motivational deficits in samples other than schizophrenia patients, including patients with major depressive disorder (MDD) (Remijnse et al., 2009; Smoski et al., 2009) and healthy volunteers (Dowd and Barch, 2012). Thus, it remains an open question, as to whether there are unique mechanisms that contribute to avolition in schizophrenia. A reduced ability to precisely represent and flexibly update value representations is a probable source of motivational deficits in general, and not just in individuals with schizophrenia. For a comprehensive review of the similarities and differences in mechanisms of motivational deficits in schizophrenia and depression, see Barch et al., in this volume.

3. Motivation appears to be driven not just by representations of value, but also by representations of the costs of actions (Treadway et al., 2009). For that reason, motivational deficits in SZ are likely driven not just by a failure to accurately represent the value of actions, but also by a failure to accurately represent the costs of actions. Schizophrenia has been associated with deficits in representing both the costs (Gold et al., 2013) and expected value (Gold et al., 2008) of actions – as well as with attenuated neural signals in dACC (Kerns et al., 2005; Minzenberg et al., 2009; but see Wilmsmeier et al., 2010), the brain region most frequently associated with the representation of the costs and benefits of actions (Kennerley et al., 2006; Walton et al., 2006). There is additional evidence that negative symptoms in schizophrenia are associated with deficits in representing both the costs (Gold et al., 2013) and expected value (Gold et al., 2012) of actions, though not necessarily with neural signals in dACC related to conflict monitoring.

4. Treatments for motivational deficits in SZ should have, as a goal, the enhancement of neural signals related to valuation. Whether this goal can be attained by pharmacological treatment, or non-pharmacological therapy, is not yet clear. The findings recounted in this chapter further suggest that treatments should not only seek to enhance the valuation of rewards, but also the ability to adaptively modulate the value attached to stimuli and actions (based, in part, on the instability of the environment being experienced). Finally, one could also imagine that psychosocial interventions aimed at helping people better weigh the costs and benefits of choices could lead to improvements in clinical status and real-world functioning.

Figure 3.

Aspects of reinforcement learning and expected value signaling. Signals for Expected Value, Experienced Value RPE Computation, and Stimulus Salience (both cue and outcome) are associated with distinct neural substrates. The signaling of expected value can be further parsed along a number of dimensions. Abbreviations: OFC, orbitofrontal cortex; dACC, dorsal anterior cingulate cortex; DLPFC, dorsolateral prefrontal cortex; VS, ventral striatum; Ant., anterior; IPL, inferior parietal lobule; Rec., receptors; V., ventral.