An Analysis of Decision Under Risk in Rats

Summary:

In 1979, Daniel Kahneman and Amos Tversky published a ground-breaking paper titled “Prospect Theory: An Analysis of Decision Under Risk,” which presented a behavioral economic theory that accounted for the ways in which humans deviate from economists’ normative workhorse model, Expected Utility Theory [1,2]. For example, people exhibit probability distortion (they overweight low probabilities), loss aversion (losses loom larger than gains), and reference dependence (outcomes are evaluated as gains or losses relative to an internal reference point). We found that rats exhibited many of these same biases, using a task in which rats chose between guaranteed and probabilistic rewards. However, Prospect Theory assumes stable preferences in the absence of learning, an assumption at odds with alternative frameworks such as animal learning theory and reinforcement learning [3–7]. Rats also exhibited trial history effects, consistent with ongoing learning. A reinforcement learning model in which state-action values were updated by the subjective value of outcomes according to Prospect Theory reproduced rats’ nonlinear utility and probability weighting functions, and also captured trial-by-trial learning dynamics.

Blurb

Constantinople et al. apply Prospect Theory, the predominant economic theory of decision-making under risk, to rats. Rats exhibit signatures of both Prospect Theory and reinforcement learning. The authors present a model that integrates these frameworks, accounting for rats’ nonlinear econometric functions and also trial-by-trial learning.

Results:

Two key components of Prospect Theory are utility (rewards are evaluated by the subjective satisfaction or “utility” they provide) and probability distortion (people often overweight low and underweight high probabilities; Figure 1D). In this theory, subjective value is determined by the shapes of subjects’ utility and probability weighting functions.

Figure 1. Rats choose between guaranteed and probabilistic rewards.

(A) Behavioral task and timing of task events: flashes cue reward probability (p) and click rates convey water volume (x) on each side. Safe and risky sides are not fixed.

(B) Relationship between cues and reward probability/volume in one task version. Alternative versions produced similar results (Figure S2). There were four possible volumes (6, 12, 24, or 48μL), and the risky side offered reward probabilities between 0 and 1 in increments of 0.1.

(C) One rat’s performance for each of the safe side volumes. Axes are probability and volume of risky options.

(D) A behavioral model inferred the utility and probability weighting functions that best explained rats’ choices. We modeled the probability that the rat chose the right side by a logistic function whose argument was the difference between the subjective value of each option (V_R-V_L) plus a trial history-dependent term. Subjective utility was parameterized as:

$$u(x)=\{\begin{array}{ccc}{(x-r)}^{\alpha }& \text{ if }& x>r\\ -\kappa {(r-x)}^{\alpha }& \text{ if }& x<r\end{array},$$ (1)

where α is a free parameter, and x is reward volume. r is the reference point, which determines whether rewards are perceived as gains or losses. We first consider the case where r = 0, so

$$u(x)=x^{\alpha }.$$ (2)

The subjective probability of each option is computed by:

$$w(p)=e^{-\beta {(-ln(p))}^{\delta }},$$ (3)

where β and δ are free parameters and p is the objective probability offered. Combining utility and probability yields the subjective value for each option:

$$V_R=u\left(x_R\right)w\left(p_R\right)$$ (4)

$$V_L=u\left(x_L\right)w\left(p_L\right).$$ (5)

These were normalized by the max, and transformed into choice probabilities via a logistic function:

$$P\left({\mathit{\text{Choose}}}_R\right)=\iota +\frac{1-2\iota }{1+e^{-\lambda \left(V_R-V_L\right)+\mathit{\text{bias}}}},$$ (6)

where ι captures stimulus independent variability (lapse rate) and λ determines the sensitivity of choices to the difference in subjective value (V_R − V_L). The bias term was comprised of three possible parameters, depending on trial history:

$$\mathit{\text{bias}}=\{\begin{array}{ll}+/-h_1\hfill & \text{ if t-}1\text{ was safe L}/\text{R choice }\hfill \\ +/-h_2\hfill & \text{ if t-}1\text{ was risky L}/\text{R rew }\hfill \\ +/-h_3\hfill & \text{ if t-}1\text{ was risky L}/\text{R miss}\hfill \end{array}.$$ (7)

(E) Model prediction for held-out data from one rat, averaged over 5 test sets.

See also Figures S1, S2.

Learning theories provide an alternative account of subjective value. In animal learning theory, Thorndike’s “Law of Effect” described the effect of reinforcers on action selection [8], and Pavlov’s subsequent experiments demonstrated how animals learn to associate stimuli with rewards [9]. The Rescorla-Wagner model of classical conditioning formalized how such learning might occur [3–5]. Although these models described how animals might learn associations between stimuli, they were naturally extended to account for learning values from experience [7,10]. Models of trial-and-error learning from animal learning theory form the basis for reinforcement learning algorithms, including temporal difference learning, which captures temporal relationships between predictors and outcomes [7,11]. Reinforcement learning provides a powerful framework for value-based decision-making in psychology and neuroscience, in which value estimates are learned from experience and updated trial-to-trial based on prediction errors [7,12,13].

Reinforcement learning has had profound impact in part because many of its components have been related to neural substrates [12,14–17]. However, standard reinforcement learning algorithms dictate that agents learn the expected value (volume × probability) of actions or outcomes with experience [6], meaning that they will exhibit linear utility and probability weighting functions. This is incompatible with Prospect Theory. We found that rats exhibited signatures of both Prospect Theory and reinforcement learning, and we present an initial attempt to integrate these frameworks. First, we focus on Prospect Theory.

Most economic studies examine decisions between clearly described lotteries (i.e., “decisions from description”). Studies of risky choice in rodents, however, typically examine decisions between prospects that are learned over time (i.e., “decisions from experience”), which are difficult to reconcile with Prospect Theory [18–20]. We designed a task in which reward probability and amount are communicated by sensory evidence, eliciting decisions from description rather than experience. This enabled behavioral economic approaches, such as estimating utility functions.

Rats initiated a trial by nose-poking in the center port of a three-port wall. Light flashes were presented from left and right side ports, and the number of flashes conveyed the probability of water reward at each port. Simultaneously, auditory clicks were presented from left/right speakers, and click rate conveyed the volume of water reward baited at each port (Figure 1A,B). One port offered a guaranteed/safe reward, and the other offered a risky reward with an explicitly cued probability. The safe and risky ports (left/right) varied randomly. One of four water volumes could be the guaranteed or risky reward (6, 12, 24, 48μL); risky reward probabilities ranged from 0 to 1, in increments of 0.1 (Figure 1A,B).

High-throughput training generated 36 trained rats and many tens of thousands of choices per rat, enabling detailed behavioral quantification. Rats demonstrated they learned the meaning of the cues by frequently “opting-out” of trials offering smaller rewards, leaving the center poke despite incurring a time-out penalty and white-noise sound (Figure S1A–C). This indicated that they associated the click rates with water volumes, instead of relying on a purely perceptual strategy. It is possible that opting-out, which persisted despite longer time-out penalties for low-volume trials (Methods), reflected reward-rate maximizing strategies [21–23]. Rats favored prospects with higher expected value (Figure 1C, S1).

We used a standard choice model [24,25] to estimate each rat’s utility and probability weighting functions according to Prospect Theory (Figure 1D; see legend; Figure S1). The model predicted rats’ choices on held-out data (Figure 1E). It outperformed alternative models, including one imposing linear probability weighting (according to Expected Utility Theory [26]), one that fit linear weights for probabilities and volumes, and several models implementing purely perceptual strategies with sensory noise (Figure S2A–F).

Concave utility (the utility function exponent α<1) produces diminishing marginal sensitivity, in which subjects are less sensitive to differences in larger rewards. Rats’ median α was 0.54, indicating concave utility, like humans [2] (Figure 2A; Figure S1F). To test for diminishing marginal sensitivity, we compared performance on trials offering guaranteed outcomes of 0 or 24μL, and 24 or 48μL (Figure 2B,C). Concave utility implies that 24 and 48μL are less discriminable than 0 and 24μL (Figure 2C). Indeed, the concavity of the utility function was correlated with reduced discriminability on trials offering 24 and 48μL (Figure 2D,E; p = 1.03e-7, Pearson’s correlation). This was true when the guaranteed outcome of 0 included trials offering 24μL with p=0 (Figure 2D,E), or all volumes with p=0 (Figure S2G,H). Our choice set did not permit analysis of trials offering non-zero rewards, as these trials (24 vs. 0, 48 vs. 24) were the only ones with equal reward differences. This suggests that rats, like humans, exhibit diminishing marginal sensitivity.

Figure 2. Non-parametric analyses confirm nonlinear utility and probability weighting and reveal diverse risk attitudes.

(A) Model fits of subjective utility functions for each rat, normalized by the maximum volume (48μL).

(B) Schematic linear utility function: the perceptual distance (or discriminability, d’) between 0μL and 24μL is the same as 24μL and 48μL.

(C) Schematic concave utility function: 24μL and 48μL are less discriminable than 0μL and 24μL.

(D) One rat’s performance on trials with guaranteed outcomes of 0μL vs. 24μL (green), or 24μL vs. 48μL (purple). Performance ratio on these trials (“d’ ratio”) less than 1 indicates diminishing sensitivity.

(E) The concavity of the utility function (α) is significantly correlated with reduced discriminability of larger rewards. Pink circle is rat from D.

(F) Model fits of probability weighting functions.

(G) Weights from logistic regression parameterizing each probability match probability weighting function for one rat. Error bars are s.e.m. for each regression coefficient.

(H) Mean squared error between regression weights and parametric fits for each rat (mean mse=0.006, in units of probability).

(I,J) To obtain “certainty equivalents,” we measured psychometric functions for each probability of receiving 48μL, and estimated the certain volume at which performance = 50%.

(K) Measured (blue) and model-predicted (red) certainty equivalents from one rat indicates systematic undervaluing of the gamble, or risk aversion. Error bars for model-prediction are 95% confidence intervals of parameters from 5-fold cross validation. Data are mean +/− s.e.m. for left-out test sets.

(L) Distribution of CE areas computed using analytic expression from model fits. Measured CEs were similar (Figure S3C). See also Figure S2, S3.

Rats’ probability weighting functions revealed overweighting of probabilities (Figure 2F). A logistic regression model that parameterized each probability to predict choice yielded regression weights mirroring the probability weighting functions (Figure 2G,H). Control experiments indicated that nonlinear utility and probability weighting were not due to perceptual errors in estimating flashes and clicks [27] (Figure S2I–K).

To evaluate rats’ risk attitudes, we measured the certainty equivalents for all gambles of 48μL [2,28,29]. The certainty equivalent is the guaranteed reward the rat deems equal to the gamble (Figure 2I,J). If it is less than the gamble’s expected value, that indicates risk aversion: the subject effectively “undervalues” the gamble, and will accept a smaller reward to avoid risk (Figure 2K). Conversely, if the certainty equivalent is greater than the gamble’s expected value, the subject is risk seeking, and risk neutral if they are equal. Measured certainty equivalents closely matched those predicted from the model, using an analytic expression incorporating utility and probability weighting functions (CE = w(p)^1/α; Methods. Pearson’s correlation 0.96, p=1.58e-11; Figure 2K). This non-parametric assay further validated the model fits and revealed heterogeneous risk preferences across rats (Figure 2L; Figure S3A–C).

Although rats exhibited nonlinear utility and probability weighting, consistent with Prospect Theory, they also exhibited trial-by-trial learning, consistent with reinforcement learning. Fitting the model to trials following rewarded and unrewarded choices revealed systematic shifts in utility and probability weighting functions: utility functions became less concave and probability weighting functions became more elevated to reflect increased likelihood of risky choice following rewards (Figure 3A). This was consistent across rats, as observed in certainty equivalents from the data (p=6.49e-5, paired t-test) and model (Figure 3B; p=2.46e-7).

Figure 3. Rats exhibit evidence of trial-by-trial learning.

(A) Probability weighting function (left) and utility function (right) for one rat from model fit to trials following reward (turquoise) or no reward (black).

(B) CE areas predicted from model fits for all rats following rewarded and unrewarded trials.

(C) ΔProbability of repeating left/right choices (relative to mean probability of repeating), following each reward. Points above the dashed line indicate an increased probability of repeating (“stay”); those below indicate a decreased probability (“switch”). Black curve is average +/− s.e.m. across rats.

(D) A separate cohort of 3 rats was trained with doubled water volumes. They exhibited lose-switch biases following 12 and 24μL.

(E) Win-stay/lose-switch biases for one rat separated by reward history two trials back.

(F) Schematic illustrating that with concave utility, rewards should be more (less) discriminable when the reference point is high (low).

(G) Psychometric performance from one rat when the inferred reference point was low (black) or high (blue). Red curve is ideal performance.

(H) Value function with the median parameters across rats indicates loss aversion (median α=0.6, κ=1.7).

See also Figure S3.

Another feature of human behavior is “reference dependence:” people evaluate rewards as gains or losses relative to an internal reference point. It is unclear what determines the reference point [30]; proposals include status quo wealth [1], reward expectation [31,32], heuristics based on the prospects [33,34], or recent experience [35,36].

Rats demonstrated reference dependence by treating smaller rewards as losses. They exhibited win-stay/lose-switch biases: following unrewarded trials, rats were more likely to switch ports (Figure 3C). Surprisingly, most rats exhibited “switch” biases after receiving 6 or 12μL, consistent with treating these outcomes as losses. The “win/lose” threshold (i.e., reference point) was experience-dependent: a separate cohort of rats (n=3) trained with doubled rewards (12–96μL) exhibited lose-switch biases after receiving 12 or 24μL (Figure 3D).

The win/lose threshold was often reward-history dependent (Figure 3E). Therefore, we parameterized a reference point, r, as taking one of two values depending on whether the previous trial was rewarded (see Methods). Rewards less than r were negative (losses). The relative amplitude of losses versus gains was controlled by the parameter κ (Equation 1; Figure 1D). Subjective value was reparameterized to include the zero outcome of the gamble, which is a loss when r > 0:

$$V_R=u\left(x_R\right)w\left(p_R\right)+u(0)w\left(1-p_R\right)$$ (8)

$$V_L=u\left(x_L\right)w\left(p_L\right)+u(0)w\left(1-p_L\right)$$ (9)

Model comparison (Akaike Information Criteria, AIC) favored the reference point model for all rats (Figure S3D,E). We also parameterized the reference point as reflecting several trials back with an exponential decay, where the time constant was a free parameter (see Methods). For most rats (20/36, 77%), this did not significantly improve model performance compared to the reference point reflecting one trial back, although it was a better fit for a minority of rats with longer integration time constants over trials (Figure S3F–H). For the sake of simplicity and because it generally provided a better fit, we focused on the “one-trial back” reference point model. Interestingly, the reference point from the model was not significantly correlated with average reward rate for each rat [37]; this was true regardless of whether opt-out trials were included in estimates of average reward per trial (Pearson’s correlation, p>0.05).

With concave utility, rats should exhibit sharper psychometric performance when the reference point is high (and rewards are more discriminable; Figure 3F). Indeed, performance was closer to ideal when the reference point was high (Figure 3G; mean mse between psychometric and ideal performance was 0.143 low ref vs. 0.122 high ref, p = 3.4e-5, paired t-test across rats).

A loss parameter κ>1 indicates “loss aversion,” or a greater sensitivity to losses than gains (Equation 1). We observed a median κ of 1.66 (Figure 3H). There was variability across rats: 16/36 rats (44%) were not loss averse but were more sensitive to gains (κ<1). Still, the median κ across rats suggests similarity to humans (Figure 3H).

Prospect Theory does not account for how agents learn subjective values from experience, and we explicitly incorporated trial history parameters to account for trial-by-trial learning (Figure 4A). To examine learning dynamics, we fit the model to the first and second half of all trials, once rats achieved criterion performance (Figure S4A). There was no significant change in the parameters for the utility or probability weighting functions (Figure S4B). Rats showed a significant increase in the softmax parameter with training, indicating increased sensitivity to value differences, and a decrease in one of the trial history parameters, h₁, indicating reduced win-stay biases (Figure S4).

Figure 4. Integrating Prospect Theory and reinforcement learning captures nonlinear subjective functions and learning.

(A) Prospect Theory model predictions for each rat, without the trial history parameters (h₁-h₃, see Figure 1 legend) does not account for win-stay/lose-switch trial history effects. Inclusion of these parameters accounts for these effects.

(B) Prospect Theory model fit to simulated choices from a basic reinforcement learning agent yields linear utility and probability weighting functions over a range of generative learning rates (0.2, 0.4, 0.6, 0.8. 1.0, overlaid).

(C) Schematic of model incorporating Prospect Theory and reinforcement learning.

(D) The hybrid model described in panel C accounts for win-stay/lose-switch affects.

(E) The model recovers nonlinear utility and probability weighting functions.

(F) Model comparison when the error term used in the model was the subjective value (as shown in panel C), or the expected value (probability × reward). Red arrow is mean ΔAIC.

(G) Binned values of rats’ lose-switch biases (measured from the data) plotted against the best-fit learning rate, α_learn. Pearson’s correlation coefficient is −0.37 across rats.

See also Figure S4.

Reinforcement learning describes an adaptive process in which animals learn the value of states and actions. This framework, however, implies linear utility and probability weighting. We simulated choices of a Q-learning agent, which learned the state-action values of each unique trial type. Fitting the Prospect Theory model to these simulated choices recovered linear utility and probability weighting functions (regardless of learning rate; Figure 4B). This is expected: since trial sequences were randomized (i.e., each trial was independent) basic reinforcement learning will learn the expected value of each option [6], and will resolve to linear utility and probability weighting functions. We therefore implemented a reinforcement learning model that could accommodate nonlinear subjective utility and probability weighting, but also learning over trials (Figure 4C; [10,38]). The model assumed that the rats learned the value of left and right choices for each unique combination of probability (p) and reward (μL) according to the following equation:

$$V_{p,\mu L}(t+1)=V_{p,\mu L}(t)+{\alpha }_{\mathit{\text{learn}}}\left(w(p)u(x)-V_{p,\mu L}(t)\right),$$ (10)

where α_learn is the learning rate parameter, and w(p) and u(x) are parameterized as in Equations 2 and 3. The learned values of the right and left prospects on each trial were transformed into choice probabilities via a logistic function (see Equation 6). We also implemented a global update to the value matrices for left and right choices depending on reward history (Figure 4C,D). In this model, utility and probability weighting functions were exclusively used for learning/updating values, whereas choice depended on the learned values on each trial. Although the parameters of the utility and probability weighting functions were free parameters in this model, we recovered parameter values identical to the Prospect Theory model (Figure 4E, Figure S4C). Importantly, a reinforcement learning model in which the expected value (EV) was the error signal driving learning underperformed compared to the model incorporating subjective value according to Prospect Theory (Figure 4F; p = 1.38e-8, paired t-test of AIC). Finally, each rats’ learning rate (α_learn) was negatively correlated with the magnitude of their lose-swich biases, suggesting an inverse relationship between learning dynamics governing gradual improvements in task performance, and trial-by-trial learning which is deleterious to performance when trials are independent [27,39,40]. Rats with slower dynamics (lower learning rate) showed more prominent trial history effects, whereas rats with rapid learning showed reduced trial history biases (Figure 4G).

Discussion:

There is a strong foundation for using animal models to study the cost-benefit calculations underlying economic theory [41]. In foraging studies, animals either exploit a current option for reward or explore a new one, and often maximize their rate of reward [42–45]. Rodents exhibit complex aspects of economic decision-making, including regret [46,47], and sensitivity to sunk costs [48,49]. Here, we applied Prospect Theory to rats. Like humans, rats exhibit nonlinear concave utility for gains, probability distortion, reference dependence, and, frequently, loss aversion. Nearly all rats exhibited concave utility, which produced diminishing marginal sensitivity. In contrast, most studies in monkeys have reported convex utility [24,50–53] (but see [25]). In Expected Utility Theory, concave utility indicates risk aversion [26]. However in Prospect Theory, concave utility can coincide with risk-seeking behavior due to the elevation of the probability weighting function [28,54].

Our rats differ from primates in that they do not appear to underweight moderate and high probabilities [1,24,28]. The “inverted-S” shape of the probability weighting function may reflect diminishing sensitivity relative to two reference points which, in the probability domain, correspond to 0 and 1 [2,28]. The rats, either due to the task or species differences, may not treat certainty as a reference point.

Reward history modified rats’ risk preferences, producing shifts in utility and probability weighting functions. The extent to which risk preferences are stable traits is an area of active research [55,56]. Recent work suggests a general/stable component of risk attitudes, but variability across domains (e.g., finance, recreation [56]). In foraging tasks, risk preferences reflect food availability and/or energy budget in a variety of species [44,57]. Here, we document dynamic risk preferences; these trial-by-trial dynamics are not likely driven by physiological factors (e.g. energy budget) but may reflect dynamic internal/cognitive states mediated by reinforcement learning.

We found evidence for reference dependence, in which rats’ treatment of outcomes as gains/losses reflected their reward history. Studies in several species, including capuchin monkeys [58], have also suggested reference dependence. Starlings and locusts prefer options that previously were encountered under greater hunger, presumably because those rewards were perceived to have greater reference-dependent value [59–61]. Rats modulate their approach speed for rewards depending on previously experienced reward amounts [62,63]. Regret, which reflects a post-decision valuation of a choice relative to an unchosen alternative, may be a reference-dependent computation; for regret, the reference point would be the counterfactual prospect [64].

While variable, the median loss parameter (κ) across rats indicated loss aversion, which has been documented in capuchin monkeys [65]. We note that we did not examine losses by taking reinforcers away from the animal. However, several decision theories [32,66] posit that rewards less than the reference point are losses. Loss aversion in humans is remarkably variable [67] and possibly domain specific [68,69]. The nature of loss aversion is intriguing: is it a constant, a psychological trait similar to risk preferences [56], or an emergent property of constructing preferences [70]?

Prospect Theory, animal learning theory, and reinforcement learning are complementary frameworks for studying decision-making (but see [71]). Reinforcement learning and animal learning theory are principally concerned with how subjects learn values over experience and use those learned values to make decisions. Prospect Theory, in contrast, does not address learning but describes nonlinear distortions that account for the decision. We propose a simple approach for integrating these frameworks, in which animals learn the values of actions associated with task states, but the reward prediction error driving learning is in units of subjective value according to Prospect Theory [10,38]. This hypothesis is consistent with studies of dopamine neurons, which are thought to instantiate reward prediction errors in the brain in a temporal-difference learning algorithm [6,7,13,14]. Conditioned stimuli predicting rewards with different probabilities or magnitudes have been shown to elicit phasic dopamine responses reflecting the value of the expected reward [72–74]. In delay discounting tasks, the phasic dopamine response reflects discounted value of delayed rewards in monkeys and rats [75,76]. Finally, recent work has shown that dopamine reward prediction errors reflect the shape of monkeys’ measured utility functions [77]. The hypothesis of a reward prediction error in units of subjective value (perhaps according to Prospect Theory in the case of explicitly described lotteries) is also conceptually related to studies of homeostatic reinforcement learning, in which internal state influences subjective valuation [78]. This hypothesis bridges animal learning theory, reinforcement learning, and economic concepts of subjective value. A key topic of future research should address how subjective estimates of value arise: are they innate, learned early in life, or constantly evolving over the lifespan?

STAR Methods

Contact for Reagent and Resource Sharing

Further information and requests for resources should be directed to and will be fulfilled by the Lead Contact, Christine Constantinople (constantinople@nyu.edu). Transgenic (Pvalb-iCre)2Ottc rats (n=5) were obtained by an MTA from University of Missouri RRRC.

Experimental Model and Subject Details

Subjects

A total of 39 male rats between the ages of 6 and 24 months were used for this study, including 35 Long-evans and 4 Sprague-Dawley rats (Rattus norvegicus). The Long-evans cohort also included LE-Tg (Pvalb-iCre)2Ottc rats (n=5) made at NIDA/NIMH and obtained from the University of Missouri RRRC (transgenic line 0773). These are BAC transgenic rats expressing Cre recombinase in parvalbumin expressing neurons. Animal use procedures were approved by the Princeton University Institutional Animal Care and Use Committee and carried out in accordance with National Institutes of Health standards.

Rats were typically housed in pairs or singly; rats that trained during the day were housed in a reverse light cycle room. Some rats trained overnight, and were not housed with a reverse light cycle. Access to water was scheduled to within-box training, 2–4 hours per day, usually 7 days a week, and between 0 and 1 hour ad lib following training.

Method Details

Behavioral training

Rats were trained in a high-throughput facility using a computerized training protocol. Rats were trained in operant training boxes with three nose ports. When an LED from the center port was illuminated, the animal could initiate a trial by poking his nose in that port; upon trial initiation the center LED turned off. While in the center port, rats were continuously presented with a train of randomly timed clicks from a left speaker and, simultaneously, a different train of clicks from a right speaker. The click trains were generated by Poisson processes with different underlying rates [79,80]; the rates conveyed the water volume baited at each side port. After a variable pre-flash interval ranging from 0 to 350ms, rats were also presented with light flashes from the left and right side ports; the number of flashes conveyed reward probability at each port. Each flash was 20ms in duration; flashes were presented in fixed bins, spaced every 250ms, to avoid perceptual fusion of consecutive flashes [81]. After a variable post-flash delay period from 0 to 500ms, the end of the trial was cued by a go sound and the center LED turning back on. The animal was then free to choose the left or right center port, and potentially collect reward.

The trials were self-paced: on trials when rats did not receive reward, they were able to initiate another trial immediately. However, if rats terminated center fixation prematurely, they were penalized with a white noise sound and a time out penalty. Since rats disproportionately terminated trials offering low volumes, we scaled the time out penalty based on the minimum reward offered. The time out penalties were adjusted independently for each rat to minimize terminated trials (as an example, several rats were penalized with 6 second time-outs for terminating trials offering a minimum of 6μL, 4.5 seconds for terminating trials offering a minimum of 12μL, 3 seconds for terminating trials offering a minimum of 24μL, and 1.5 seconds for terminating trials offering a minimum of 48μL).

In this task, the rats were required to reveal their preference between safe and risky rewards. To determine when rats were sufficiently trained to understand the meaning of the cues in the task, we evaluated the “efficiency” of their choices as follows. For each training session, we computed the average expected value per trial of an agent that chose randomly, and a perfect expected value maximizer, or an agent that always chose the side with the greater expected value. We compared the expected value per trial from the rat’s choices relative to these lower and upper bounds. Specifically, the efficiency was calculated as follows:

$$\mathit{\text{efficiency}}=0.5\frac{ra{t}_{EV/\mathit{\text{trial}}}-ran{d}_{EV/\mathit{\text{trial}}}}{{\mathit{\text{EVmax}}}_{EV/\mathit{\text{trial}}}-ran{d}_{EV/\mathit{\text{trial}}}}+0.5$$ (11)

The threshold for analysis was the median performance of all sessions minus 1.5 times the interquartile range of performance across the second half of all sessions. Once performance surpassed this threshold, it was typically stable across months. Occasional days with poor performance were usually due to hardware malfunctions in the rig. Days in which performance was below threshold were excluded from analysis.

Quantification and Statistical Analysis

Behavioral model

We fit a behavioral model separately for each rat (see Figure 1 legend for description of the model). We used Matlab’s constrained minimization function fmincon to minimize the sum of the negative log likelihoods with respect to the model parameters. 20 random seeds were used in the maximum likelihood search for each rat; parameter values with the maximum likelihood of these seeds were deemed the best fit parameters. When evaluating model performance (e.g., Figure 1E), we performed 5-fold cross-validation and evaluated the predictive power of the model on the held-out test sets.

We initially evaluated three different parametric forms of the probability weighting function, the one- and two-parameter Prelec models and the linear in log-odds model (see below; [24,28]. We compared the different parametric forms using Akaike Information Criteria (AIC), AIC = 2k + 2nLL, where k is the number of parameters, and nLL is the negative log likelihood of the model. AIC favored the two-parameter Prelec model for nearly all rats, although some rats were equally well-fit by the linear in log-odds model (data not shown). Therefore, we implemented the two-parameter Prelec model.

$$\text{One-parameter Prelec}:w(p)=e^{-{(-ln(p))}^{\delta }},$$ (12)

where p is the true probability, and δ is a free parameter. δ controls the curvature of the weighting function; its crossover point is fixed at 1/e.

$$\text{Two-parameter Prelec}:w(p)=e^{-\beta {(-ln(p))}^{\delta }},$$ (13)

Where p is the true probability, β and δ are free parameters. δ primarily controls the curvature and β primarily controls the elevation of the weighting function.

$$\text{Linear in log-odds}:w(p)=\frac{\delta p^{\gamma }}{\delta p^{\gamma }+{(1-p)}^{\gamma }},$$ (14)

where p is the true probability and γ and δ are free parameters. γ primarily controls the curvature of the weighting function and δ controls the elevation.

Alternative models

We compared the Prospect Theory model to a number of alternative models. The Expected Utility Theory model (EUT) has the same form as the Prospect Theory model, except that the subjective value on each side is the product of objective probability and subjective utility (see Figure S2A):

$$V_R=u\left(x_R\right)p_R$$ (15)

$$V_L=u\left(x_L\right)p_L.$$ (16)

The linear weighting model fit different weights to flashes and clicks, before combining them (see Figure S1G):

$$u(x)=cx$$ (17)

$$w(p)=dp,$$ (18)

where c and d are constants, x is the click rate and p is the number of flashes presented to the animal on each side. The value on each side is the product of linearly weighted flashes and clicks:

$$V=u(x)w(p).$$ (19)

We next included sensory noise as part of a perceptual strategy. Previously, we have used a signal-detection theory (SDT) model to estimate rats’ perceptual variability (noise) in estimating numbers of flashes and clicks; we found they exhibit a property called scalar variability, meaning that the standard deviation in their estimate grows linearly with the mean [27]. We implemented four different signal-detection theory models that instantiated scalar noise, according to this work. The models differ in the decision rules they apply. The models assume that on each trial, the rats’ estimate of the number of flashes (clicks) on each side is a random variable drawn from a normal distribution, the mean of which corresponds to the actual number of flashes (clicks) presented to the animal. According to scalar variability, the standard deviation is linearly related to the number of flashes (clicks). There are two free parameters that define this linear relationship; we fit separate linear scaling relationships to the estimation of clicks and flashes:

$${\sigma }_F=m_{F}x+b_F$$ (20)

$${\sigma }_C=m_{C}x+b_C$$ (21)

Where m_F, m_C, b_F, b_C, are free parameters. x is the number of flashes (clicks) presented to the rat on each side. For the first two SDT models, We compute choice probabilities based on the flash difference (ΔF), and click difference (ΔC) separately, where these choice probabilities are calculated as follows, according to [27]:

$$P(\mathit{\text{went\hspace{0.17em}right}}|\Delta F)={\int }_0^{\infty }N\left(R_F-L_F\right)\sqrt{{\sigma }_{R_F}^2+{\sigma }_{L_F}^2}d\left(R_F-L_F\right)$$ (22)

$$P(\mathit{\text{went\hspace{0.17em}right}}|\Delta C)={\int }_0^{\infty }N\left(R_C-L_C\right)\sqrt{{\sigma }_{R_C}^2+{\sigma }_{L_C}^2}d\left(R_C-L_C\right)$$ (23)

R_F (R_C) and L_F (L_C) are the number of right and left flashes (clicks) presented to the rat on each trial, and the σ terms are the noise terms defined in Equations 20 and 21. One model (SDT₁) assumes that the rat’s choice is given by the average of these probabilities (see Figure S2C). Another model (SDT₂) assumes that the rat’s choice is given by the most informative cue on each trial (the choice probability most different from 0.5; see Figure S2D).

Alternatively, it’s possible that the rats combine the noisy estimates of flashes and clicks on each side. Therefore, we evaluated two additional models parameterized as follows:

$$P(\mathit{\text{went\hspace{0.17em}right}}|\mathit{\text{right\hspace{0.17em}ev}})={\int }_0^{\infty }N\left(R_F+R_C\right)\sqrt{{\sigma }_{R_F}^2+{\sigma }_{R_C}^2}d\left(R_F+R_C\right)$$ (24)

$$P(\mathit{\text{went\hspace{0.17em}right}}|\mathit{\text{left\hspace{0.17em}ev}})=-{\int }_0^{\infty }N\left(L_F+L_C\right)\sqrt{{\sigma }_{L_F}^2+{\sigma }_{L_C}^2}d\left(L_F+L_C\right)$$ (25)

One model (SDT₃) assumes that the rat’s choice is given by the average of these probabilities (Figure S2E), and the other (SDT₄) assumes that the rat’s choice is given by the most informative side on each trial (the choice probability most different from 0.5; Figure S2F).

Psychometric curves

We measured rats’ psychometric performance when choosing between the safe and risky options. For these analyses, we excluded trials where both the left and right side ports offered certain rewards. We binned the data into 11 bins of the difference in the subjective value (inferred from the behavioral model) of the safe minus the risky option. Psychometric plots show the probability that the subjects chose the safe option as a function of this difference (see Figure S1D). We fit a 4-parameter sigmoid of the form:

$$p\left({\mathit{\text{Choose}}}_S\right)=y_0+\frac{1-2a}{\left(1+e^{\left(-b\left(V_S-V_R-x_0\right)\right)}\right)},$$ (26)

where y₀, a, b, and x₀ were free parameters. Parameters were fit using a gradient-descent algorithm to minimize the mean square error between the data and the sigmoid, using the sqp algorithm in Matlab’s constrained optimization function fmincon.

Logistic regression to compare regressors to probability weighting functions

We fit a logistic regression model with a separate regressor for each probability the rat may have been offered (0 to 1 in 0.1 increments), plus a constant term. To compare the regressors to the parametric fits, we normalized the regressors for each probability by subtracting the minimum and dividing by the maximum regressor value, so they ranged from 0 to 1 (Figure 2G). We computed the mean square error between these normalized regressor values and the probability weighting functions (Figure 2H). The model was fit using Matlab’s function glmfit.

Certainty equivalents

Non-parametric estimate

We estimated rat’s certainty equivalents by evaluating their psychometric performance (%Chose risky) for each gamble of 48 μL, and estimating the value of the psychometric curve at which performance was at 50% (Figure 2I). To do this, we fit a line to the two points of the psychometric curve above and below chance level using Matlab’s regress.m function, and interpolated the value of that line that would correspond to 50%.

Analytic expression for CE from the model fits

We compared our estimates of rats’ certainty equivalents from their behavioral data to an analytic expression from the subjective probability and utility functions we obtained from the model. We define the certainty equivalent, $\tilde{x}$, as the guaranteed reward equal to a gamble, x with probability p. In the case of linear probability weighting, we express this as follows:

$$\begin{gathered} p{x}^{\alpha }={\tilde{x}}^{\alpha } \\ ln(p)+\alpha ln(x)=\alpha ln(\tilde{x}) \\ ln(p)=\alpha ln\left(\frac{\tilde{x}}{x}\right) \\ \frac{1}{\alpha }ln(p)=ln\left(\frac{\tilde{x}}{x}\right) \\ {p}^{\frac{1}{\alpha }}=\frac{\tilde{x}}{x} \end{gathered}$$ (27)

For nonlinear probability weighting, substituting w(p) for p yields an analytic expression for the certainty equivalent from the exponent of the utility function (α) and the probability weighting function (also see [29]).

Behavioral model with reference point

The behavioral model with the reference point (see Figure 3) was similar to the behavioral model described above, except for elaborations of the subjective utility function u(x) and subjective value (V_R, V_L). We modified the subjective utility function to include a dynamic reference point, r, below which value was treated negatively (as a loss). The relative amplitude of losses versus gains was controlled by the scale parameter κ.

$$u(x)=\{\begin{array}{cc}{(x-r)}^{\alpha }& \text{if} x\ge r\\ -\kappa {(r-x)}^{\alpha }& \text{ if } x<r\end{array},$$ (28)

where, as before, α is the exponent of the utility function, and x is the offered reward. We also reparameterized subjective value. The risky prospect offers two possible outcomes: x with probability p, and 0 with probability 1 − p. In the absence of a reference point, the zero reward outcome (0, 1 − p) does not influence choice (0^α = 0). However, if r>0, the zero reward outcome can be perceived as a loss. Therefore, in the reference point model, subjective value was reparameterized to incorporate this possible outcome of the gamble:

$$V_R=u\left(x_R\right)w\left(p_R\right)+u(0)w\left(1-p_R\right)$$ (29)

$$V_L=u\left(x_L\right)w\left(p_L\right)+u(0)w\left(1-p_L\right)$$ (30)

We parameterized the reference point, r, to take on two discrete values depending on whether the previous trial was rewarded or not. There were two additional free parameters, y and m that could account for asymmetric effects of rewarded and unrewarded trials:

$$r(t)=\{\begin{array}{ll}m\hfill & \text{ if t-}1\text{ was rewarded }\hfill \\ y\hfill & \text{ if t-}1\text{ was not rewarded }\hfill \end{array}.$$ (31)

We constrained r ≥ 0.

Behavioral model integrating Prospect Theory and Reinforcement Learning

This behavioral model was similar to the Prospect Theory model, except that w(p) and u(x) were used to update the subject’s value of each unique trial type based on experience. There were separate state-action value matrices for left and right choices. The entry of each matrix corresponded to a unique trial type, for each unique probability p and reward volume μL,

$$V_{p,\mu L}(t+1)=V_{p,\mu L}(t)+{\alpha }_{\mathit{\text{learn}}}\left(w(p)u(x)-V_{p,\mu L}(t)\right),$$ (32)

where α_learn is an additional free parameter fit by the model. w(p) and u(x) are parameterized as they were in the Prospect Theory model, according to Equations 2 and 3 in the main text. The difference in value of the right and left prospect is the argument to a logistic function, as parameterized in Equation 6 of the main text.

We also included a global update of the entire left or right value matrix that reflected reward history as follows:

$$V(t+1)=\{\begin{array}{ll}{\beta }_{win}u(x)\hfill & \text{ if t was rewarded }\hfill \\ {\beta }_{\mathit{\text{loss}}}u(x)\hfill & \text{ if t was not rewarded}\hfill \end{array},$$ (33)

where u(x) corresponds to the subjective utility of the chosen reward volume.

Data and Software Availability

Behavioral data are available upon request by contacting the Lead Contact, Christine Constantinople (constantinople@nyu.edu).

Key Resources Table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Experimental Models: Organisms/Strains
Rat: Long Evans	Taconic	RRID: RGD_1566430
Rat: Sprague Dawley	Taconic	RRID: RGD_1566440
Rat: Long Evans	Hilltop	www.hilltoplabs.com
Rat: Long Evans	Harlan	RRID: RGD_5508398
Rat: Pvalb-iCre	University of Missouri RRRC	RRID: RGD_10412329
Software and Algorithms
Matlab	MathWorks	RRID: SCR_001622
Behavioral control software	Bcontrol	http://brodywiki.princeton.edu/bcontrol/index.php/Main_Page

Highlights.

• A novel task enables application of core behavioral economic approaches in rodents.

• Like humans, rats exhibit nonlinear utility and probability weighting.

• Rats also exhibit trial history effects, consistent with ongoing learning.

• A reinforcement learning model incorporating subjective value accounts for the data.