Dopaminergic basis for signaling belief updates, but not surprise, and the link to paranoia

Significance

To survive in changing environments animals must use sensory information to form accurate representations of the world. Surprising sensory information might signal that our current beliefs about the world are inaccurate, motivating a belief update. Here, we investigate the neuroanatomical and neurochemical mechanisms underlying the brain’s ability to update beliefs following informative sensory cues. Using multimodal brain imaging in healthy human participants, we demonstrate that dopamine is strongly related to neural signals encoding belief updates, and that belief updating itself is closely related to the expression of individual differences in paranoid ideation. Our results shed new light on the role of dopamine in making inferences and are relevant for understanding psychotic disorders such as schizophrenia, where dopamine function is disrupted.

Abstract

Distinguishing between meaningful and meaningless sensory information is fundamental to forming accurate representations of the world. Dopamine is thought to play a central role in processing the meaningful information content of observations, which motivates an agent to update their beliefs about the environment. However, direct evidence for dopamine’s role in human belief updating is lacking. We addressed this question in healthy volunteers who performed a model-based fMRI task designed to separate the neural processing of meaningful and meaningless sensory information. We modeled participant behavior using a normative Bayesian observer model and used the magnitude of the model-derived belief update following an observation to quantify its meaningful information content. We also acquired PET imaging measures of dopamine function in the same subjects. We show that the magnitude of belief updates about task structure (meaningful information), but not pure sensory surprise (meaningless information), are encoded in midbrain and ventral striatum activity. Using PET we show that the neural encoding of meaningful information is negatively related to dopamine-2/3 receptor availability in the midbrain and dexamphetamine-induced dopamine release capacity in the striatum. Trial-by-trial analysis of task performance indicated that subclinical paranoid ideation is negatively related to behavioral sensitivity to observations carrying meaningful information about the task structure. The findings provide direct evidence implicating dopamine in model-based belief updating in humans and have implications for understating the pathophysiology of psychotic disorders where dopamine function is disrupted.

To successfully navigate the world we need to exploit sensory information to make inferences about the environment (1). For example, before crossing the road it is sensible to check the traffic lights at a pedestrian crossing to decide whether it is safe to cross or not, drawing on our cognitive model of what traffic lights (the observable information) tell us about the traffic flow (the partially observable, or hidden, environmental state). When the light changes from the “red man” to the “green man” this should cause us to update our belief about the state of the environment to infer it is now safe to cross. Importantly, however, it is also critical to assess the informativeness of any sensory input. For example, although it would be surprising to see both the green and red lights on simultaneously, it is not advisable to update one’s beliefs about traffic flow based on this observation alone. Thus, adaptive behavior depends on an ability to discriminate between observations carrying relevant information for the task at hand (informative or meaningful cues) and observations carrying irrelevant, ambiguous, or no information (noninformative or meaningless cues). The former should induce updates in an agent’s model of the world, whereas the latter should not (2).

Dopamine may play a key role in the processing of meaningful sensory information. Phasic activity in midbrain dopamine neurons is implicated in processing unexpected and salient environmental stimuli (3), including those that are novel (4–6) and associated with reward (7, 8). More recent evidence suggests a role for dopamine in updating a rich internal model of the task environment, necessary for flexible behavior (9–11). Specifically, phasic midbrain dopamine signals can reflect inferences about the identity of hidden task states (12, 13) and encode value-neutral prediction errors (14, 15), as well as support stimulus–stimulus associative learning (10). Here, we test whether dopamine is associated with the processing of meaningful sensory information in humans, so as to allow an agent to make inferences on a sensory input and appropriately update their internal representations of the environment.

Meaningful information can be formally quantified as the degree to which a new observation changes an agent’s prior belief about the current state of the world, given previous observations, to a new (posterior) belief. The magnitude of this belief update from a ‘prior’ belief to a ‘posterior’ belief is usually quantified as the Kullback–Leibler divergence (D_KL) and has been termed “Bayesian surprise” (SI Appendix, Eq. S8) (2, 16).

Belief updates occur after unexpected observations, but unexpectedness alone should be insufficient to motivate change in an agent’s internal representations. As outlined in our example above, unexpected observations that are equally unlikely under all competing hypotheses about the environment contain no meaningful information with respect to the hidden state. The improbability of an observation, given an agent’s prior expectation, is often quantified in terms of information-theoretic surprise (I_S, or “surprisal”), which can be thought of as “counter evidence” to an agent’s representation of the world (SI Appendix, Eq. S9).

The distinction between the pure unexpectedness (information-theoretic surprise) of an observation and its meaningful information content (Bayesian surprise) is central to understanding how new information influences adaptive behavior and may also be of relevance for understanding psychotic symptoms in schizophrenia. One theoretical formulation postulates that stimulus-locked dopamine neural activity is important for processing salient stimuli, and that maladaptive dopaminergic activity in response to ambiguous, unreliable, or behaviorally irrelevant (meaningless) events leads to aberrant attribution of salience to these same events. This in turn is thought to underpin misattributional symptoms such as paranoia (17–23). Of note, the detection of behaviorally salient stimuli involves a number of brain circuits that modulate the firing of dopamine neurons in the midbrain. In particular, the anterior hippocampus has a key role in regulating midbrain dopamine neuron activity depending on the novelty and context of stimuli via a circuit that involves the nucleus accumbens and ventral pallidum (5, 20, 24).

An understanding of the mechanisms underlying belief updating is therefore critical for understanding both the generation of complex goal-directed behaviors and symptoms of certain neuropsychiatric disorders. Recent fMRI studies have begun to investigate the neural correlates of belief updating in humans, showing that encoding of unsigned belief updates (but not simple unexpectedness) is present in dopamine-rich midbrain regions, specifically the ventral tegmental area (VTA) and substantia nigra (SN) (25–27). However, to date, there is no evidence linking direct measures of dopamine function to belief updating in humans.

We investigated a dopaminergic basis for belief updating using a model-based fMRI task, combined with PET imaging of dopamine function. We used a task that separates Bayesian surprise, information-theoretic surprise, and reward prediction errors, on a trial-by-trial basis (Fig. 1) (27). In brief, during the task participants (n = 39) need to track which of two (hidden) task states pertained at every trial, based on imperfectly informative observations about state identity. Specifically, they were tasked to infer whether visual or auditory cues were currently relevant for predicting monetary outcomes, where the relevant modality signaled the sign of the monetary outcome with ∼90% cue validity. The identity of the relevant modality reversed (switched) periodically. Participants were not explicitly informed of the validity of the relevant cue or the reversal probability but were thoroughly trained on the task before scanning.

Fig. 1.

fMRI task showing two example trials, one informative and one noninformative. The task contained two auditory and two visual cues, with one cue from each modality being predictive of a monetary win and the other of a monetary loss (∼90% validity). Trials started with the simultaneous presentation of one visual and one auditory cue, followed by a monetary outcome (gains or losses from 10 to 30 pence). For any given trial only one cue modality was relevant for predicting the outcome, and the identity of the relevant cue switched five to six times in a session of 60 trials. The goal of the task was to correctly track the identity of the relevant cue modality (i.e., the hidden task state) at each trial, using information from cue-outcome observations. At the end of each trial participants reported their belief about the identity of the relevant modality using a rating scale. Half of the trials were noninformative, in that the visual and auditory cues predicted the same (congruent) monetary outcome, while the other half were informative, in that auditory and visual cues predicted incongruent outcomes. Unexpected outcomes in both informative and noninformative trials had positive information-theoretic surprise (I_S), but these events were only associated with positive Bayesian surprise (D_KL) in informative trials.

At the start of each trial, two cues (one auditory, one visual) were presented simultaneously and could either be incongruent or congruent in their monetary predictions. Following cue presentation participants observed a monetary outcome (either a win or a loss) and subsequently indicated their belief about the relevant predictive modality (current environmental state) on a rating bar (Fig. 1). Monetary outcomes that were unexpected under a current prior hypothesis (rendering I_S > 0) could provide either meaningful (D_KL > 0, in incongruent trials) or meaningless (D_KL = 0, in congruent trials) information regarding the identity of the task-relevant modality. This design allows a decorrelation of Bayesian (D_KL) and information-theoretic (I_S) surprise (27, 28), enabling us to identify the neural signature of each construct. We hypothesized that belief updates (correlating with the meaningful information content of an observation), but not sensory unexpectedness, would be encoded in dopamine-rich brain areas, namely the SN/VTA complex and ventral striatum, in line with predictions from previous findings (25–27). Moreover, we tested whether deviations from optimal behavior in this task were related to the presence of subclinical paranoid thoughts, a key prediction of the aberrant salience hypothesis of schizophrenia.

To test directly the role of dopamine in these processes, we used PET with the dopamine-2/3 receptor (D2/3R) agonist ligand [¹¹C]-(+)-4-propyl-9-hydroxy-naphthoxazine ([¹¹C]-(+)-PHNO) at baseline (n = 36) and following 0.5 mg/kg dexamphetamine challenge (n = 17). The baseline [¹¹C]-(+)-PHNO PET scan measures D2/3 autoreceptor availability in the midbrain, which are inhibitory receptors (29–31). We hypothesized that greater midbrain D2/3R availability, reflecting greater tonic inhibitory tone, would be negatively related to phasic midbrain neural response during belief updates (4). Following acute amphetamine challenge there is an increase in dopamine concentration in the striatum, consequent upon blockade of dopamine reuptake (4, 32), and also possibly due to increased dopamine neuron firing (33–35). Greater dexamphetamine-induced dopamine release is thought to be associated with more spontaneous dopamine transients in the drug-free state, indicating a lower signal-to-noise ratio in dopaminergic signaling (17). Consequently, we hypothesized that greater striatal dopamine release capacity would be associated with lower ventral striatal neural response during belief updates. Finally, by measuring the D2/3R availability in the striatum at baseline, we were able to test a hypothesized inverted-U relationship between cognitive flexibility and striatal dopamine function at rest (36).

Results

Task Behavior, Computational Modeling, and Relationship to Baseline Striatal Dopamine Function.

We modeled individual participants’ behavior (belief ratings) by fitting a Bayesian observer model, with two free parameters reflecting participants’ expectations about the cue validity ($\text{ψ}$) and the probability of state transitions (reversals) at any given trial ($1-\delta$) (Fig. 2). Together, these parameters captured individual differences in trial-by-trial belief updating. Specifically, the magnitude of a belief update following an unexpected and informative observation is proportional to participants’ expectations about cue validity ($\text{ψ}$), while their estimate of the state transition probability on a given trial $(1-\delta )$ governs belief uncertainty with each time step. This computational model allowed us to quantify trial-by-trial belief updates as the Kullback–Leibler divergence from prior beliefs to posterior beliefs (SI Appendix, Eq. S8) (Bayesian surprise), as well as the information-theoretic surprise (surprisal) of an observation (SI Appendix, Eq. S9) at the monetary outcome stage of each trial.

Fig. 2.

Behavioral modeling. A Bayesian model with two free parameters ($\text{ψ}$ and $\delta$) was fitted to individual participant belief ratings. The model assumes that beliefs about the relevant cue modality (environmental state at time t), $x_t$, are updated after making a cue-outcome observation $o_t$, in accordance with Bayes’ theorem (1), where $p\left(o_t\text{|}{x}_t\right)$ is the likelihood of the cue-outcome observation given a particular environmental state $x_t$ (visual or auditory cues relevant). This likelihood is determined by the participant’s estimate of cue validity, $\text{ψ}$. $p\left(x_t|o_{1:t-1}\right)$ is the prior belief about the relevant cue modality at the start of trial t, and $p\left(x_t\text{|}{o}_{1:t}\right)$ is the posterior belief after observation $o_t$. For each subsequent trial (e.g., t + 1), the prior belief about the relevant modality is dependent on the posterior belief from the previous trial, $p\left(x_t\text{|}{o}_{1:t}\right)$, and the participant’s belief about the probability of state transitions, $p\left(x_{t+1}\right|x_t)$, reflected by the parameter δ (2). See SI Appendix for further details.

Our fitted model had high accuracy in explaining participants’ behavior (R² = 0.67, 95% confidence intervals = [0.60, 0.73]). Moreover, we found a strong positive correlation between participants’ ratings and those predicted by an ideal Bayesian observer model (an instantiation of our computational model using the true parameters of the task) (r = 0.75, [0.69, 0.80]), supporting the idea that participants performed the task adequately, and that their behavior was closely approximated by a simple Bayesian observer model (Fig. 3A). There was no significant correlation between the two free parameter estimates (averaged over task blocks) within participants (r = 0.07 [−0.25, 0.38], P = 0.65. Mean [SD] for $\text{ψ}$ and $\delta$ were 0.90 [0.09] and 0.92 [0.08], respectively), indicating that each captured different aspects of task performance (see SI Appendix, Table S1 for individual participant parameter estimates).

Fig. 3.

Behavioral results. (A) Behavioral performance of two exemplar participants in two sessions. Gray/white epochs represent periods when auditory/visual cues are relevant, respectively. The blue solid line displays observed belief ratings at each trial [varying between complete certainty that auditory cues are relevant, P(a) = 1, and complete certainty that visual cues are relevant, P(v) = 1]. The orange broken line displays predicted behavior from the fitted model (participant model parameters given above the plots). (B) There was a negative relationship between participants’ self-reported paranoia scores [measured with the Green Paranoid Thoughts Scale (37)] and the degree to which their behavior correlated with the predictions of an ideal Bayesian observer (I.B.O.). (C) There was a direct correlation between overall behavioral performance (correlation with I.B.O.) and behavioral sensitivity to meaningful information (mean difference in belief shift on informative vs. noninformative trials). (D) There was an inverted-U relationship between striatal D2/3R availability and behavioral sensitivity to meaningful information (mean difference in belief shift on informative vs. noninformative trials) (see SI Appendix, Table S2 for parameters of the quadratic model). Broken trendlines (B–D) represent 95% confidence bounds.

We analyzed both overall behavioral performance and trial-by-trial belief updating. We operationalized overall performance as the correlation between participant belief ratings and those of an ideal Bayesian observer (i.e., approximation to normative Bayesian behavior), while our primary measure of trial-by-trial behavior was the mean difference in reported belief update on informative vs. noninformative trials (i.e., behavioral sensitivity to meaningful information). These measures of behavior were directly correlated (rho = 0.56 [0.29, 0.75], P < 0.001) (Fig. 3C). Closer inspection uncovered that poor overall performance was specifically related to the absolute magnitude of reported belief shifts on noninformative trials (rho = −0.60 [−0.77, −0.34], P < 0.001), rather than informative trials (rho = −0.10 [−0.41, 0.23], P = 0.53). This indicates that participants who showed poor overall performance had reduced behavioral sensitivity to the meaningful information content of cues and tended to update their beliefs following noninformative observations.

Interestingly, we found a negative correlation between participants’ subclinical paranoia scores [Green Paranoid Thoughts Scale (37) total] and both overall behavioral performance (rho = −0.60 [−0.77, −0.34], P < 0.001) and the magnitude of belief shifts on informative vs. noninformative trials (rho = −0.32 [−0.59, <0.001], P = 0.04) (Fig. 3B). In other words, participants with elevated subclinical paranoid thoughts showed reduced behavioral sensitivity to the meaningful information content of observations. Crucially, we found no significant relationship between overall performance and working memory capacity, measured with digit span assessment (rho = −0.03 [−0.30, 0.35], P = 0.85), indicating that a lower capacity to retain information per se does not account for our findings. For additional trial-by-trial behavioral results see SI Appendix.

Next, we investigated the relationship between baseline striatal dopamine function and task performance. Previous work has established that there is an inverted-U relationship between resting striatal dopamine levels and cognitive flexibility, thought to be mediated by tonic stimulation of striatal D2/3 receptors (36). Consistent with this hypothesis, we found that baseline D2/3R availability in the whole striatum predicted trial-by-trial sensitivity to meaningful information (mean belief shift on informative vs. noninformative trials) with an inverted quadratic relationship in a simple regression model including linear and quadratic terms for [¹¹C]-(+)-PHNO nondisplaceable binding potential ($B{P}_{ND}$) (model 1: F_2,33 = 3.42, adjusted R² = 0.12, model P = 0.04; P value for linear and negative quadratic term coefficients = 0.26 and 0.04, respectively). As age and body mass index influence striatal baseline [¹¹C]-(+)-PHNO $B{P}_{ND}$ (38, 39), we repeated this analysis including these as covariates in case they were influencing our findings. The inverted quadratic relationship remained significant in this adjusted model (model 2: F_4,31 = 3.40, adjusted R² = 0.22, model P = 0.02; P value for [¹¹C]-(+)-PHNO $B{P}_{ND}$ linear and negative quadratic term coefficients = 0.10 and 0.01, respectively) (Fig. 3D and SI Appendix, Table S2). Importantly, the linear models relating behavioral performance to striatal baseline D2/3R availability (i.e., omitting the quadratic term) were not significant compared with the null model (P > 0.18). Furthermore, the simple linear and quadratic regression models describing the two free parameters ($\text{ψ}$ and $\delta$) as a function of striatal D2/3R availability were also not significant compared with the null model (all P > 0.25).

Belief Updates Are Encoded in the Midbrain and Ventral Striatum.

In the region of interest (ROI) fMRI analysis we found effects for model-derived trial-by-trial estimates of Bayesian surprise (meaningful information) in the bilateral SN/VTA complex and ventral striatum (Fig. 4A), whereas no such effect was evident for information-theoretic surprise (meaningless information), consistent with previous reports (26, 27). A formal comparison of regions preferentially encoding Bayesian surprise vs. information-theoretic surprise (D_KL > I_s) showed significantly greater activation for Bayesian surprise in the SN/VTA and ventral striatum bilaterally (significant activation differences at P_peak < 0.05, with small volume correction for SN/VTA and ventral striatum ROI; SI Appendix, Fig. S1). Importantly, neural encoding of Bayesian surprise in the midbrain and ventral striatum (quantified as the principle eigenvariate of the contrast parameter estimates within each region) did not correlate with individual participant model R² or the free parameters of the model (all P > 0.25), indicating that variation in fMRI parameter estimates in these regions is not driven by differences in model fit.

Fig. 4.

fMRI neural activity encoding belief updates (Bayesian surprise). (A) ROI analysis revealed significant bilateral activation peaks in SN/VTA [right peak Montreal Neurological Institute (MNI) coordinates, 10 −26 −12, P_peak = 0.003, T_peak = 4.94, and left peak MNI, −9 −26 −12, P_peak = 0.006, T_peak = 4.69) and ventral striatum (left peak MNI, −14 15 −4, P_peak < 0.001, T_peak = 6.88, and right peak MNI, 8 14 0, P_peak = 0.001, T_peak = 5.62) after applying small-volume correction for SN/VTA and ventral striatum. Image thresholded at P < 0.005 (uncorrected) with a cluster extent threshold of 25 and an inclusive mask (SN/VTA and ventral striatum) for illustration purposes only. (B) Significant clusters of activation encoding Bayesian surprise across the whole brain (familywise error correction at P_cluster < 0.05. Voxel cutoff P < 0.001 (uncorrected), critical cluster threshold = 290), including ventral midbrain, pre-SMA, and dorsal anterior cingulate cortex (see SI Appendix, Fig. S2 and Table S3). Color bars represent t values.

At the whole-brain level, we also found effects for Bayesian surprise in the presupplementary motor area (pre-SMA), dorsal anterior cingulate cortex, posterior parietal cortex (e.g., supramarginal gyrus), and lateral prefrontal cortex (e.g., middle frontal gyrus) (Fig. 4B and SI Appendix, Fig. S2 and Table S3). Information-theoretic surprise at monetary outcome was encoded in a network of brain regions including the pre-SMA, anterior insula, middle frontal gyrus, angular gyrus, and precuneus, significant at whole-brain P_cluster < 0.05 (SI Appendix, Fig. S3A and Table S4). There was no significant hippocampal activation encoding Bayesian or information-theoretic surprise at monetary outcome, and no activation at cue onset, in a whole-brain analysis.

We found no significant correlation between task performance, model parameters, or paranoia scores and the effect size of the signal encoding Bayesian surprise in the SN/VTA complex or ventral striatum (all P > 0.25), although there was a positive correlation between overall task performance and neural effect size within the pre-SMA encoding information-theoretic surprise (rho = 0.35 [0.03, 0.61], P = 0.03) (SI Appendix, Fig. S3B).

Beyond neural encoding of Bayesian and information-theoretic surprise, there was no evidence for signed reward prediction error in the SN/VTA complex or the ventral striatum in the ROI analysis, nor at whole-brain cluster level at the conventional voxel-level cutoff of P < 0.001 (uncorrected). When this cutoff was reduced to a very liberal threshold of P < 0.05 (uncorrected), we found a single large cluster involving the left striatum (pallidum, caudate, and putamen) significant at the whole-brain level (SI Appendix, Fig. S4).

Finally, belief uncertainty at cue presentation (prior uncertainty) was encoded in a widespread network involving dorsolateral prefrontal cortex, medial prefrontal cortex, and occipitoparietal cortex. These brain regions were implicated in encoding belief uncertainty both when prior uncertainty was defined from the rating bar report on the previous trial (“subjective uncertainty”) and when it was defined as the entropy over the distribution of model-derived prior beliefs on the current trial (“model-derived uncertainty”; SI Appendix, Eq. S10) (SI Appendix, Fig. S5 and Table S5). The recruitment of a widespread frontoparietal network to encode the uncertainty (entropy) of internal models may reflect the necessity to consider competing hypotheses in working memory (40).

Dopaminergic Basis for Neural Signals Encoding Belief Updates in Midbrain and Ventral Striatum.

Having found evidence consistent with belief update (Bayesian surprise) encoding in midbrain and ventral striatum, we next determined whether this neural encoding was related to in vivo measurements of dopamine acquired in the same participants, using [¹¹C]-(+)-PHNO PET. Baseline D2/3R availability (measured as the [¹¹C]-(+)-PHNO $B{P}_{ND}$) was calculated for the SN/VTA complex, where this measure indexes midbrain dopamine autoreceptor availability. In the midbrain, baseline D2/3R availability was negatively related to fMRI-measured neural activation encoding Bayesian surprise (rho = −0.43 [−0.67, −0.11], P = 0.009) (Fig. 5A). There was no significant correlation between SN/VTA baseline D2/3R availability and fMRI activation encoding Bayesian surprise within the ventral striatum (rho = −0.12 [−0.44, 0.23], P = 0.49).

Fig. 5.

Correlation between dopaminergic measures (PET) and neural activity encoding belief updates (fMRI). (A) Negative relationship between effect size of activation encoding of belief updates (principle eigenvariate of parameter estimates for Bayesian surprise contrast) and baseline D2/3R availability ([¹¹C]-(+)-PHNO $B{P}_{ND}$) in the midbrain (SN and VTA complex). (B) Negative relationship between ventral striatum activation encoding belief updates and dexamphetamine-induced dopamine (DA) release (percent decrease in [¹¹C]-(+)-PHNO, $\%\mathrm{\Delta }B{P}_{ND}$) in the whole striatum. Broken trendlines represent 95% confidence interval bounds. Brain masks represent ROIs used for analysis.

We next assessed dexamphetamine-induced dopamine release capacity, calculated as the percentage reduction in $B{P}_{ND}$ from baseline scan to dexamphetamine scan within the whole striatum. This measure is hypothesized to reflect the dopamine system’s tendency toward spontaneous transients at rest, which reduces the signal-to-noise ratio of stimulus-locked dopamine bursts (17). We found a negative correlation between dexamphetamine-induced striatal dopamine release capacity and neural activation within the ventral striatum encoding Bayesian surprise (rho = −0.71 [−0.89, −0.34], P = 0.002) (Fig. 5B). This negative relationship was also present when considering dopamine release capacity within the ventral striatum only (rho = −0.66 [−0.87, −0.24], P = 0.005). Dexamphetamine-induced dopamine release capacity in the whole striatum did not correlate with neural activity encoding Bayesian surprise in the SN/VTA (rho = −0.10 [−0.56, 0.41], P = 0.71).

There was no significant relationship between whole striatal D2/3R availability at baseline and either midbrain or ventral striatal fMRI activation encoding Bayesian surprise (rho = −0.10 [−0.43, 0.24], P = 0.54 and rho = −0.05 [−0.38, 0.30], P = 0.79, respectively).

Testing the Specificity of the Dopamine–fMRI Correlation.

Both Bayesian and information-theoretic surprise were encoded in an overlapping medial prefrontal cortex cluster (at whole-brain corrected P_peak < 0.05; see Fig. 4B and SI Appendix, Fig. S3B), yet there was no significant relationship between the neural activation in the significant voxels for either contrast and midbrain D2/3R availability (rho = −0.05 [−0.38, 0.30], P = 0.79 for Bayesian surprise and rho = −0.27 [−0.56, 0.07], P = 0.11 for information-theoretic surprise), whole striatal dopamine release capacity (rho = 0.22 [−0.31, 0.64], P = 0.40 for Bayesian surprise and rho = −0.45 [−0.77, 0.05], P = 0.07 for information-theoretic surprise), or whole striatum D2/3R availability (rho = −0.06 [−0.39, 0.28], P = 0.73 for Bayesian surprise and rho = 0.09 [−0.25, 0.42], P = 0.57 for information-theoretic surprise).

Finally, we conducted four whole-brain analyses, testing the negative linear relationship between both midbrain D2/3R availability or whole-striatum dopamine release capacity, and neural activation encoding either Bayesian or information-theoretic surprise. These analyses confirmed that neural encoding of Bayesian surprise in the left ventral striatum is negatively correlated with striatal dopamine release capacity (surviving small volume correction at P_peak < 0.05, using SN/VTA and ventral striatum ROI; SI Appendix, Fig. S6). There were no other voxels that showed a significant relationship between neural activation and dopamine measures in any of the four fMRI-PET whole-brain analyses, either at whole-brain or small-volume-corrected P < 0.05.

Discussion

Controlling for the effects of signed reward prediction errors, we show that the SN/VTA and ventral striatum encode meaningful information content in sensory observations. This encoding reflected solely the magnitude of belief updates regarding the current environmental state (Bayesian surprise from prior beliefs to posterior beliefs), but not the simple unexpectedness of an observation (information-theoretic surprise). Using in vivo PET imaging of dopamine we also demonstrate that neural activity encoding belief updates is negatively related to D2/3R availability in the midbrain, and dopamine release capacity in the striatum. These results provide a direct link between belief updating and dopaminergic function, extending observations from previous fMRI studies that implicate SN/VTA in encoding the magnitude of belief update signals on the one hand (25–27) and the assumed role of dopamine in an implementation of probabilistic inference on the other (41, 42). Additionally, we show that participants’ trial-by-trial sensitivity to the meaningful information content of observations has an inverted-U relationship with striatal baseline D2/3R availability, in line with evidence that striatal D2/3R signaling has an inverted-U relationship with cognitive flexibility (36). Our results therefore shed light on the neurochemical basis of belief updating in humans using in vivo quantification of dopamine function.

The [¹¹C]-(+)-PHNO signal in the SN/VTA primarily indexes D3 autoreceptor availability (30, 43, 44) and the signal here is less sensitive to tonic synaptic dopamine levels compared with the striatum (45). D2/3R availability was negatively related to neural activity encoding belief updates in the SN/VTA complex, consistent with evidence that midbrain D3Rs have an inhibitory effect on dopaminergic neurons (29, 31), and in line with the notion that tonic dopamine signaling may regulate the amplitude of stimulus-locked phasic dopamine neuron activity (4). For example, D3R knockout mice have elevated extracellular dopamine levels in the nucleus accumbens (46), while mice treated with D3R-preferring agonists show reduced dopamine concentration in the accumbens (47). In a recent fMRI study, selective antagonism of the D3R enhanced midbrain and ventral striatal fMRI activation during anticipation of monetary reward, providing indirect evidence for an inhibitory role for midbrain D3Rs in humans (48). The behavioral significance of elevated midbrain D2/3R availability has also recently been investigated in rats, where nigral [¹¹C]-(+)-PHNO $B{P}_{ND}$ correlated with impaired reversal learning in a probabilistic reward task (49). Our findings extend this work by showing that natural variation in human midbrain D2/3R availability is associated with altered midbrain activation during belief updating, with lower levels associated with relatively greater activation. Moreover, our task design allowed us to investigate the specific role of dopamine in encoding the meaningful information content of an observation, decorrelating this construct from simple unexpectedness and reward prediction error.

We found that a belief update signal in the ventral striatum was negatively correlated with dexamphetamine-induced striatal dopamine release capacity, providing in vivo human evidence that this signal is related to dopamine function. This complements findings from a recent optogenetic fMRI study in rats, which demonstrated that striatal blood-oxygen-level-dependent (BOLD) activations may be driven by mesolimbic dopamine neuron firing (50). It has been proposed that greater amphetamine-induced dopamine release capacity in vivo corresponds to a greater tendency toward spontaneous dopamine neuron firing in the drug-free (baseline) state, which decreases the signal-to-noise ratio of stimulus-locked dopamine bursts (17). Our finding that striatal dopamine release capacity is negatively correlated with the striatal BOLD response encoding belief updates is therefore consistent with current hypotheses regarding the relationship between amphetamine-induced dopamine release capacity and mesostriatal dopaminergic function at rest. Moreover, this finding extends our understanding by showing a negative relationship between the natural variation in dopamine release capacity in humans and adaptive neural activation in the ventral striatum. However, it is important to note that the relationship between spontaneous dopamine neuron firing and amphetamine-induced dopamine release has yet to be tested, and that, while some studies report that amphetamine’s action is dependent on neuronal firing within the VTA (33, 34), acute amphetamine administration has generally been found to reduce dopamine neuron firing (51–53), as well as having other actions to increase striatal dopamine levels (4, 32, 54). Thus, preclinical studies that combine PET and dopamine neuron recordings would be useful to test the hypothesis that spontaneous dopamine neuron firing in the amphetamine-free state is directly associated with dopamine release induced by amphetamine.

Consistent with a previous study using the same task (27), information-theoretic surprise was encoded in frontal brain areas including pre-SMA. We also replicated the finding that the effect size of this activation positively correlated with task performance, suggesting that surprising events may be imbued with higher salience in participants with a better model of the task (resulting in better performance) (27). Importantly, there was no relationship between the effect size of the neural response in this region and any PET measure of dopamine function, favoring a more specific role for dopamine in encoding meaningful information.

An influential model proposes that the anterior hippocampus regulates midbrain dopamine neuron firing depending on the novelty and context of stimuli through the descending arm of a hippocampal–VTA loop. Activity in projections from the VTA to the hippocampus, constituting the ascending arm of the loop, in turn facilitate the updating of memory by enhancing long-term potentiation in the hippocampus (5, 20). However, we found no evidence for increased hippocampal activity at cue onset, and there was no positive correlation between hippocampal activation and either meaningful (Bayesian) or meaningless (information-theoretic) surprise at monetary outcome. It should be noted, however, that our task was not optimized to detect event-related hippocampal activity relating to novelty processing or learning, as participants had been thoroughly trained on the task stimuli and structure before scanning. Nevertheless, further studies are required to investigate the relationship between prediction error signals (e.g., in the midbrain and orbitofrontal cortex) and hippocampal representations, given the proposed role of the hippocampus in the learning and remapping of internal models (“cognitive maps”) (11, 26, 55–57).

It has also been suggested that a connection from the medial prefrontal cortex to the dopaminergic midbrain may convey information relating to inference about the environment (specifically, inference over possible hidden states of a task) (12). In line with this finding we found that belief updates were encoded in the medial frontal cortex, including dorsal anterior cingulate. This observation is consistent with previous human and nonhuman primate studies (26–28, 58, 59) as well as with suggestions that anterior cingulate cortex is active in novel or volatile environments wherein agents need to refine their internal models in light of new observations (28, 60). Moreover, we also detected activation encoding belief updates in lateral prefrontal and posterior parietal cortical regions, which have been implicated in inference on the nature of the causal relationships between observations (hidden causal structures) (61) and in encoding state prediction errors that support learning an internal model of a task (state–action–state transition probabilities) (62).

The ventral striatum and SN/VTA are implicated in encoding signed reward prediction errors that update action and state values (7, 63, 64). Ventral striatal encoding of these model-free reward prediction errors may be negatively related to ventral striatal dopamine synthesis capacity (65, 66). Consistent with previous studies using similar task designs (27, 67), we did not find strong evidence for effects within these regions for signed reward prediction errors. Previous studies have shown that the processing of reward anticipation and prediction error in the mesolimbic dopamine circuit is sensitive to current task demands, including action planning (68–70). In our task participants were not attempting to maximize reward, and the observation of monetary gains vs. losses was not indicative of task performance. Furthermore, unexpected outcomes were equally informative about changes in relevant cue modality, regardless of whether they took the form of a monetary gain or loss. Thus, the important contribution of our results is to highlight dopamine’s role in signaling belief updates beyond its role in signaling signed reward prediction errors, an observation that hints at a role for dopamine in probabilistic inference and structural learning. Consistent with this interpretation, a recent study employing electrophysiological recordings in behaving rats demonstrated that midbrain dopamine neurons that signal classical signed reward prediction errors also signal value-neutral sensory prediction errors (14). Moreover, in humans the magnitude of value-neutral “stimulus identity” prediction errors in the midbrain is related to updates in state representation in the orbitofrontal cortex (71). The implication here is that dopamine has a wide range of functions that extends to updating a predictive associative model of the world, suggesting phasic dopamine activity signals a more general error signal, where value errors are a special case (10, 14).

The findings of our study are highly relevant for dopaminergic and neurocomputational theories of schizophrenia (59, 72). The aberrant salience hypothesis proposes that symptoms such as paranoia arise when unwarranted meaning and behavioral salience is attributed to ambiguous, irrelevant, or unreliable stimuli (17, 18, 20–23). This is suggested to reflect maladaptive phasic dopamine signaling in a mesostriatal circuit, activity that underpins learning of cue values and associations under normal circumstances (7, 10, 14). Our results speak to this hypothesis in two ways. First, subclinical paranoia was negatively related to behavioral sensitivity to the meaningful information content of an observation, and also to the degree to which a participant’s performance correlated with that of an ideal Bayesian observer. This suggests that maladaptive belief updating (i.e., updating one’s beliefs following ambiguous or meaningless observations) may contribute to the formation of subclinical paranoid beliefs. Second, by dissociating the meaningful information content of an observation from its simple unexpectedness, and showing a dopaminergic relationship with the former, our findings point to the possibility of advances that might accrue from reformulating constructs such as “salience” in a more mathematically rigorous fashion. In fact, one hypothesis from our findings is that the central feature of “aberrant salience” in psychotic disorders is a failure to dissociate between meaningful (task-relevant) and meaningless (task-irrelevant) information, resulting in belief updating arising out of merely surprising inputs (59).

Conclusions

Using model-based fMRI we demonstrate that activity within both the midbrain and ventral striatum correlates with the magnitude of a belief shift following an observation, indicating that these structures encode the meaningful information content of a stimulus, as opposed to its simple unexpectedness (surprise). Moreover, using PET we demonstrate a potential dopaminergic basis for these neural signals. Specifically, neural encoding in the midbrain was negatively related to midbrain D2/3R availability, while encoding in the striatum was negatively related to striatal dopamine release capacity. Finally, we show that participants who displayed the least sensitivity to the meaningful content of observations also reported greater subclinical paranoid ideation. Together, our results suggest that the role of phasic mesolimbic dopamine activity extends beyond its well-established role in signaling signed reward prediction errors and includes updating a rich internal model of the word capable of supporting flexible behavior. Furthermore, our findings have relevance for understanding the pathophysiology of psychotic disorders such as schizophrenia, which are characterized by mesostriatal dopamine abnormalities and symptoms arising from aberrant inferences about the world, as manifested in delusions.

Materials and Methods

Subjects.

The study was approved by the local National Health Service Research Ethics Committee and the Administration of Radioactive Substances Advisory Committee. Thirty-nine healthy volunteers [17 females, mean age 26.2 y (SD 7.0)] were included in the fMRI analysis. Thirty-six subjects also received a baseline [¹¹C]-(+)-PHNO scan to quantify D2/3R availability in the midbrain. Seventeen subjects additionally received a second [¹¹C]-(+)-PHNO PET scan, timed to start 3 h after oral administration of dexamphetamine (0.5 mg/kg), to quantify dexamphetamine-induced dopamine release in the striatum. See SI Appendix for further details. Volunteers provided written informed consent to participate in the study.

Task.

We used a validated task that decorrelated information-theoretic and Bayesian surprise (27). Subjects performed three sessions of the task (60 trials per session) during fMRI after at least 1 h of training on the task before the scan, in which they learned about the task structure and cue valences. For full task details see Fig. 1 and SI Appendix.

Computational Modeling.

We used a simple Hidden Markov Model that captures trial-by-trial belief updating using iterative application of Bayes’ rule (SI Appendix, Eq. S6) (27). The model was fitted to individual subject behavior by varying two free parameters using constrained maximum likelihood estimation: (i) cue validity ($\text{ψ}$) and (ii) the state transition probability ($1-\delta )$. See SI Appendix for further details.

Image Acquisition.

Structural and functional magnetic resonance (MR) images were acquired using a Siemens MAGNETOM Verio 3-T MR scanner. Functional images were acquired with a multiband sequence based on the multiband EPI WIP v012b provided by the University of Minnesota (73–76), using a multiband acceleration factor of 2. We acquired a whole-brain volume consisting of 72 interleaved slices (2-mm thickness), with a repetition time of 2,000 ms, echo time of 30 ms, an in-plane resolution of 3 × 3 mm, flip angle of 62°, and bandwidth of 1,906 Hz per pixel. In each task session 402 volumes were acquired (duration = 13 min, 24 s), totaling 1,206 volumes over three task sessions. An MR-compatible button box recorded right index and middle finger presses to move the cursor on the rating bar. Auditory cues were presented using MR-compatible headphones.

PET images were acquired using a Siemens Biograph HiRez XVI PET scanner. PET acquisition started with the injection of a single i.v. bolus of 0.020–0.029 μg/kg [¹¹C]-(+)-PHNO (77). For dexamphetamine PET scans 0.5 mg/kg dexamphetamine was administered orally 3 h before [¹¹C]-(+)-PHNO administration, so that scan acquisition coincided with the expected time of peak action (78). Across all PET scans the mean [¹¹C]-(+)-PHNO mass administered was 1.5 μg (SD 0.31) and mean injected activity was 177.5 MBq (SD 50.0). After the administration of the radiotracer, dynamic emission data were acquired continuously for 90 min. For further details of MR and PET image acquisition see SI Appendix.

fMRI Analysis.

fMRI analysis was performed using SPM12 (https://www.fil.ion.ucl.ac.uk/spm) and employed standard image preprocessing procedures (outlined in SI Appendix). For first-level analysis we used a mass-univariate approach, using a general linear model (GLM) with separate stick function events for the onset of fixation crosses, cues, monetary outcome presentation, and rating bars (27). At monetary outcome we included parametric regressors defining (i) information-theoretic surprise (I_S, z-scored, SI Appendix, Eq. S9), (ii) Bayesian surprise (D_KL, z-scored, SI Appendix, Eq. S8), (iii) the difference between reported belief shifts (derived from observed changes on the rating bar) and estimated shifts in beliefs from the fitted model (z-scored), (iv) monetary outcome (+1 for win, −1 for loss), and (v) signed reward prediction errors (observed reward minus expected reward, where expected reward is defined as the sum of the valence of the observed auditory and visual cues, weighted by the prior beliefs about their relevance).

To control for possible confounds we included the following parametric regressors at cue onset: (i) current reported beliefs about relevant modality, (ii) subjective uncertainty about these beliefs (derived from the rating bar report on the previous trial), (iii) expected relevant outcome value (defined above), and (iv) expected irrelevant outcome value (sum of the valence of observed cues weighted by the subject’s prior beliefs about their irrelevance). We included the number of button presses as a parametric regressor at rating bar onset. Regressors were not serially orthogonalized, to remove shared variance. The GLM described here (GLM1) was used for the main analysis, including the “subjective uncertainty” fMRI analysis. We defined a second model (GLM2) for the “model-derived uncertainty” fMRI analysis. See SI Appendix for further details regarding both GLM1 and GLM2.

A standard summary statistic approach was used to test for second-level effects of Bayesian surprise (D_KL) and information-theoretic surprise (I_s) at monetary outcome using one-sample t tests on the estimated responses for the first-level analysis and a t-contrast of D_KL > I_s to identify brain regions that showed preferential activation for belief updates compared with sensory unexpectedness. Random field theory was used to correct for multiple comparisons.

Our fMRI analysis focused on an a priori ROI comprising the bilateral midbrain SN/VTA complex [manually delineated using the mean structural image from an independent sample of healthy participants (27, 79)] and the bilateral ventral (limbic) striatum (nucleus accumbens, ventral caudate rostral to the anterior commissure and ventral putamen rostral to the anterior commissure) (80), given these regions are implicated in model-free (habit) and model-based (goal-directed) learning (7, 10, 14, 15, 25–27, 63). We combined these two regions into a single ROI mask, defined in MNI space, to ensure that statistical results were corrected for the total number of voxels across both areas (see SI Appendix, Fig. S7 for an illustration of the ROI).

ROI activations were considered statistically significant at peak-level P < 0.05 familywise error-corrected using a small-volume correction. For PET–fMRI correlations we extracted the principal eigenvariate of BOLD response from the relevant ROI subregion (bilateral SN/VTA or ventral striatum). This measure reflects the “typical” parameter estimate for a given contrast within a region and is more robust to intraregional heterogeneity of parameter estimates compared with the mean (using the mean parameter estimate did not change the nature of the results). For whole-brain analyses, we report BOLD activations that survive familywise error correction at P < 0.05 at the cluster level [with cluster-forming threshold set to P < 0.001 (uncorrected), to ensure a well-behaved family error control] (81, 82).

PET Analysis.

We employed an automatic pipeline to obtain an individual parcellation of the brain into the studied ROIs, implemented in MIAKAT release 4.2.6 (www.miakat.org) (83), SPM12, and FSL (version 5.0.9) (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki). The simplified reference tissue model was used to derive the $B{P}_{ND}$ of [¹¹C]-(+)-PHNO $B{P}_{ND}$ from the regional time activity curves (SI Appendix, Eq. S11) (84, 85), with cerebellar gray matter as the reference region. For each ROI we estimated baseline D2/3R availability ($B{P}_{ND}$) and dexamphetamine-induced dopamine release ($\mathrm{\Delta }B{P}_{ND}$, the percentage reduction in $B{P}_{ND}$ from the baseline to dexamphetamine scan, SI Appendix, Eq. S12). See SI Appendix for further details.

Statistical Analysis of PET–fMRI Relationship.

We tested for the hypothesized PET–fMRI correlations both using an a priori ROI analysis (SN/VTA and ventral striatum) and at the whole-brain voxel level (see SI Appendix for further details). PET outcome measures were $B{P}_{ND}$ and dexamphetamine-induced $\mathrm{\Delta }B{P}_{ND}$. The fMRI outcome measure was the contrast parameter estimate for Bayesian or information-theoretic surprise (voxel parameter estimate for whole-brain analysis; principle eigenvariate of the parameter estimate for ROI analysis). For PET–fMRI ROI correlations we used Spearman’s rank correlation coefficient, as we did not assume linear monotonic relationships (correlations remain significant when using Pearson’s correlation coefficient).