The prefrontal–striatal signatures of reduced model-based learning in depressed patients

Abstract

Background

Anhedonia is the core symptom of major depressive disorder (MDD). Accumulating evidence indicates that an imbalance between model-based (MB) and model-free (MF) reinforcement learning (RL) characterizes MDD, but the underlying neural substrates remain unclear. We examined whether alterations in MB and MF reward prediction error (RPE) neural signature underlie deficits in RL in depressed patients.

Methods

We used a two-stage Markov decision task (MDT) in combination with computational modeling to examine model-based and model-free learning. A total of 49 MDD and 41 matched HC individuals performed the MDT. 19 MDD and 21 HC individuals underwent functional neuroimaging during the MDT. The stress-RL deficits model was tested using a mediation model with MB and MF RL as mediators between stress and depressive/anhedonic symptoms.

Results

Depressed patients showed RL deficits, with less reliance on MB strategies and more reliance on MF strategies. MB and MF RL deficits mediated the relationship between stress and anhedonic symptoms, with specific striatal signatures (i.e., RPE_MF signals in VTA and caudate) mediating stress and anhedonia symptoms across MDD and HC groups.

Conclusions

This study showed deficits in model-based RL for depressed patients, with underlying neural deficits in prefrontal-striatal RPE signals, which would be promising for improving therapeutic practice in depression.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12991-026-00630-z.

Introduction

Major depressive disorder (MDD) is one of the most prevalent mental illnesses and has high comorbidity and mortality worldwide [1, 2]. Anhedonia, defined as “decreased interest and pleasure in most activities most of the day”, is one of the cardinal symptoms of MDD according to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5). Compared to other depressive symptoms, anhedonia is usually the last to be resolved [3]. However, the underlying mechanisms are not well understood. A more fine-grained conceptualization towards behavioral models as well as an understanding of the neural basis of anhedonia would improve the identification of specific targets of treatment [4].

Anhedonia could be conceptualized as the deficits in anticipating, experiencing and learning from reward with underlying neural substrates [5, 6]. Bodgan proposed reward processing as a putative intermediate phenotype of MDD [7]. The reward deficits model proposed that stressful events may disrupt the dopaminergic reward system and impair reward sensitivity, which is closely related to chronic stress-induced depression [8, 9]. Acute stress reduces reward responsiveness, especially for healthy individuals reporting greater anhedonic symptoms in their daily life [10], healthy individuals with genetic stress vulnerability [11] and MDD patients with high anhedonic symptoms [12], while perceived chronic stress predicts reduced reward responsiveness even when controlling for general distress and anxiety [13].

At the behavioral level, learning could be classified as habitual vs. goal-directed behaviors, depending either on reinforcement or cognitive representation of task. Reinforcement learning (RL) computational modeling with model-based (MB) and model-free (MF) dichotomies allows to capture these two components with finer perspective and bridge the gap between subjective experience and the neural substrates of anhedonia. The MF-RL is driven by prediction error (PE) while the MB-RL is driven by prediction error and mental simulation of task structure [14]. The MB-MF dichotomies extend beyond the operationalization of goal-directed and habitual behaviors, for example, the spatial navigation [15]. The work with two-step decision-making task has revealed that humans combine MB and MF computations [16]. When faced with uncertain situations, individuals with depressive symptoms tend to adopt more MF and less MB learning strategies measured with other RL tasks [17, 18], similar with the results of humans [19, 20] and rats [21] under stress. Specifically, acute stress may shift individuals, especially those with higher levels of depression from the MB to MF strategy [22, 23]. Recent work emphasized the prediction error to stress could predict negative affective consequences after stress [24]. Increased MF strategies correlated with elevated depressive symptoms, and more MB strategies was linked to lower anhedonia levels across all patient groups (MDD, OCD and SZ) [25]. To summarize, the blunted ability in RL may result in deficiency in approach behavior, which is characteristic of anhedonia symptoms in depression [26].

At the neural level, dopamine encodes prediction error, which is signaled by phasic burst or functional neuroimaging [27]. Dopamine might enhance model-based control, whereas disruption of the prefrontal cortex could make behavior more habitual [28, 29]. More recently, neuroimaging studies have explicitly modeled RL with probabilistic reward tasks and suggested that MDD is characterized by impaired phasic reward prediction error (RPE) neural signals of reward circuits [30–32]. The blunted RPE responses to naturalistic stimuli was closely related to depression, who showed less emotional benefit from surprisingly good outcome (positive PE), in contrast to normal responses to surprisingly negative outcome (negative PE) [33], a phenomenon of positive emotional blunting regularly reported in depression. Furthermore, altered RPE signals have been correlated with increased anhedonia or depressive symptoms [31, 32]. The subclinical depression was associated with reduced PE in the insula (MB-RL) and caudate (MF-RL) [34]. Stress disrupted both MB (decreased hippocampal activity) and model-free (decreased lateral prefrontal activity) neural computation [35]. Additionally, the RPE signal in ACC could predict anhedonia symptoms one year later [36]. Anhedonia is correlated with resting-state graph theoretical metrics (e.g., Strength Centrality, Eigenvector Centrality and Local Efficiency) of RPE network (e.g., dACC, dlPFC, caudate and ventral striatum) functionally defined according to the Reward Flanker Task (RFT) [37]. To sum up, we speculate that alterations in RPE encoding in prefrontal-striatal circuits might underlie RL deficits in depression.

Therefore, the current study was conducted based on the following hypotheses: (1) depressed patients show a relative shift from model-based to model-free control; (2) RL (MB and/or MF) acts as a mediator between stress and depressive and/or anhedonic symptoms; and (3) impaired RL prefrontal-striatal neural signals (RPE) are correlated with depressive and/or anhedonic symptoms.

Materials and methods

Participants

All participants were recruited via posters in the community or advertisements in psychology classes. All participants provided written informed consent. This study received approval from the Ethical Committee of Army Medical University and complies with the ethical standards of the Helsinki Declaration.

1. Behavioral study. 49 depressed patients and 41 healthy controls were recruited and included in the study to perform the behavioral task. Another subgroup of healthy young adults (n = 183) was recruited for the replication of the results.

The inclusion criteria for both groups were as follows: (1) right handedness and (2) normal or corrected-to-normal vision aged 18–65 years old. Additional criteria for the depressed patients were the current experience of a depressed episode according to the DSM-5 criteria and mild to moderate depressive symptoms (Hamilton Depression Rating Scale (HAMD-17) score ≥ 17; Beck Depression Inventory (BDI) score ≥ 14). The depressed patient group was selected and diagnosed according to the structured clinical interview by psychiatrists in the Mental Health Center of Chongqing (CN) before the recruitment.

The exclusion criteria for both groups were as follows: (1) residual symptoms of or manifest axis-I disorders or (2) a history of learning disabilities, neurological illnesses or physical illnesses that significantly impair psychosocial functioning or brain function. Additional exclusion criteria for the healthy control group were a history of medication with antidepressants, antipsychotics, or tranquilizers or a history of any axis-I disorder.

• (2)Neuroimaging study. Among the participants in study 1, 19 depressed patients and 21 healthy controls participated in study 2 to undergo the neuroimaging while performing the behavioral task.

Clinical measures

All recruited participants were administered clinical instruments in the interview before the experiment.

1. Beck Depression Inventory (BDI-II). The BDI-II is a 21-item self-reported questionnaire measuring the severity of depression in normal and psychiatric populations. Each item is scored from 0 (not at all) to 3 (nearly every day) [38]. The translated Chinese version showed acceptable reliability (Cronbach’s α coefficient = 0.94, test-retest reliability coefficient = 0.55) and validity (criterion validity with HAMD-17: r = 0.67, P < 0.01) [39]. To further separate depressive symptoms from anhedonic symptoms, the total score of the anhedonia items (4, 12, 19, 21) on the BDI was used as the anhedonic symptom score (ASS), and the sum of the scores of the rest of the items was used as the depressive symptom score (DSS). These two scores were used additionally as variables to investigate the correlations between model-based and model-free RL signals and symptoms (Sect. "The prefrontal-striatal neural substrates of RL in depression").

2. Mood and Anxiety Symptom Questionnaire-Short Form (MASQ-SF). The MASQ-SF is used to measure anxious and depressive symptoms [40]. It consists of 62 items which could be divided into 4 subscales. Each item is scored from 1 (not at all) to 5 (severe). The depression subscales include the general distress depression (GDD) and anhedonic depression (AD) subscales. The MASQ_AD subscale is used to assess trait anhedonia. The translated Chinese version of the MASQ-SF showed good reliability (Cronbach’s α coefficient = 0.94, test-retest reliability coefficient = 0.82) and validity (four-factor model: NFI = 0.90, CFI = 0.90, GFI = 0.92, RMSEA = 0.07) [41].

3. Perceived Stress Scale (PSS). The PSS is used to assess the level of unexpected and unmanageable stress that an individual has experienced in the past month [42]. There are 10 items that are rated from 0 (never) to 4 (very common). The translated Chinese version has good reliability (Cronbach’s α coefficient = 0.91, test-retest reliability coefficient = 0.69) and construct validity (two-factor model: NFI = 0.91, CFI = 0.97, GFI = 0.93, RMSEA = 0.04) in depressed, obsessive-compulsive and healthy individuals [43].

Behavioral study

We adopted the two-step Markov sequential decision task [16], which is designed to distinguish model-based and model-free RL. Before the formal task, all participants underwent a self-paced tutorial that introduced the task structure and provided 50 practice trials.

At the start of each trial, participants were shown a pair of abstract colorful stimuli (stage 1) and were asked to choose between them. The participant then reached stage 2 where they again had to choose between two abstract stimuli. The choice of one stage 1 stimulus more frequently led to one of two stage 2 stimulus pairs (P = 0.70 or 0.30), while it rarely led to another stage 2 stimulus pair (P = 0.30 or 0.70). The opposite was true for the other stage 1 stimulus. After choice of stage 2 choice participants received probabilistic feedback (reward in form of xxx currency) according to Gaussian random walks.

To ensure that our participants understood the instructions, we asked all participants about the transition structure (“Which red picture more frequently led to the two green pictures?“). There were 201 trials in total (Fig. 1). After the task was completed, participants were paid extra bonuses upon their performance.

Fig. 1.

Flow chart of the two-stage Markov reward decision task

Neuroimaging study

After entering the 3.0T MRI scanning room, the participants lay supine on the scanning bed, with their head and body kept still. Rubber earplugs and headphones were used to reduce the impact of noise, and foam head pads were used to fix the head to reduce the head motion. T1-weighted structural images were collected from the whole brain with a FSPGR sequence using 3.0T General Electric (GE) Signa EXCITE scanner. The acquisition parameters were FOV = 25.6 cm, layer thickness 1.2 mm, resolution 1 × 1 × 1 mm, TE = 30 ms, turn angle = 15°. T2-weighted functional images were collected by means of a gradient echo pulse sequence with 32 layers (sagittal), FOV = 19.2 cm, matrix size = 64 × 64, spatial resolution = 3 × 3 × 3 mm, flip angle = 90°, TE = 30 ms, TR = 2000 ms.

Data analysis

The behavioral data were analyzed and computationally modeled as reflecting a combination of model-based and model-free reward learning using an adaptation of a previously described hybrid reinforcement learning (RL) model [16, 44]. The MB and MF components of RL were then entered into the mediation model to examine the indirect effects of stress on depressive/anhedonic symptoms with RL components as mediators. The impairments in neural substrates of prefrontal-striatal circuits underlying MB and MF RL of depressed patients were also examined.

Multi-level GLMM model of choice behavior

Generalized linear mixed effects model (GLMM) was used. The hierarchical logistic regression analysis was performed across the HC and MDD group, using the glmer function of lme3 package (Version 3.3.0.1959) in R software (Version 3.4.0). (1) The effects of transition and outcome on stay behavior for each group. The dependent variable (stay behavior) was a dichotomous variable (1 = stay vs. 0 = shift). The independent predictors included transition (−0.5 = rare vs. 0.5 = common), and outcome (0.5 = reward vs. −0.5 = no reward). Subjects were taken as random effect. (2) The effects of transition and outcome on stay behavior across groups. According to previous studies [44, 45], the dependent variable (stay behavior) was a dichotomous variable (1 = stay vs. 0 = shift). The independent predictors included group (HC vs. MDD), transition (−0.5 = rare vs. 0.5 = common), and outcome (0.5 = reward vs. −0.5 = no reward). Group (HC vs. MDD) was taken as fixed effect. Within-subject factors (intercept, the transition, outcome, and their interactions) were taken as random effects. The main effect of outcome (i.e., the model-free term) and transition-by-outcome interaction (i.e., the model-based term) were the effects of interest. The intercept reflects the tendencies to repeat the action from the previous trial [46], using bound-constrained Optimization by Quadratic Approximation (bobyqa) with 1e6 functional evaluations. The model was defined as follows: Stay ~ 1 + transition × outcome × Group + (1 + transition × outcome|Subject). The trials in which a participant missed the option during the first- or second-stage were removed.

Individual-level RL computational model of choice behavior

The modelling was performed with Matlab 2019b (https://www.mathworks.com), with the emfit toolbox developed by Huys et al. [47] within and across the groups, i.e., MDD, healthy control (HC) and young healthy adult (YHA) groups. The task consisted of three states: first stage S_A; second stage S_B and S_C, each resulting from two actions (a_A and a_B). Then the transition between conditions were as follows: (1) common transitions. , , or vice versa, (2) rare transitions. , . The goal of RL modeling is to map each state-action pair to its expected future value. The model-free value was modeled with the State-Action-Reward-State-Action (SARSA) algorithm. This algorithm updates the action value for each state-action pair according to:

1

where (2).

The model-based value was modeled by learning a transition function and immediate reward values for each state and computing cumulative state-action values by iterative prediction of the reward values.

3

The weighted sum of the model-free and model-based values was calculated for the first stage:

4

where w is the weighting parameter, which was assumed to be constant across trials.

In the first and second stages, the probability of action can be represented as the net state-action value Q_net, inverse temperature parameter β₁ and β₂ (which differ between stage 1 and 2), perseverance parameter p and indicator function rep(a):

5

The update of the first-stage action value by the second-stage prediction error at the end of each trial according to eligibility trace parameter λ, . In the first and second stages, different learning rates α₁ and α₂ were adopted to allow for potential differences in learning from state transitions vs. rewards [16].

The estimation was derived from simulations of SARSA and model-based algorithms using the parameters with the best fit to the participants’ data. We estimated the seven free parameters (α_1, α₂, β₁, β₂, λ, ω, p) separately for each subject to maximize the negative log likelihood (LL) of the obtained choices given the previously observed choices and rewards summed over all participants and trials. The model fitting procedures were performed by Expectation-Maximisation (EM) to find group priors and individual (Laplace) approximate posterior distributions for the estimates for each parameter for each participant.

Bayesian model comparison at the group level is to assess the model parsimony by comparing posterior probability of each model given the dataset for all participants. The individual parameters were integrated out by sampling from the fitted priors and computing the posterior log likelihood (LL) of each model given all the data, with Bayesian information criterion (BIC) at the group level (with smaller BIC representing greater model parsimony). We first compared all computational parameters between groups (MDD vs. HC) based on the default hybrid model (7 parameters, MB and MF), and then tested whether the MDD and HC groups showed differences in their best fitting model.

Stress-RL deficits mediation model of depression

The mediating role of RL deficits in the relationship between stress and depression was analyzed in depressed patients. Using SPSS 19.0 (https://www.ibm.com/cn-zh/spss) combined with the PROCESS macro (model 4) [48], stress was taken as the independent variable, depressive symptoms as the dependent variable, model-based and model-free RL were used as the mediating variables to investigate whether stress affected depression and/or anhedonic symptoms through the effects of model-based and/or model-free RL.

The prefrontal-striatal RL BOLD signals

SPM12 (Wellcome Trust Center for Neuroimaging, http://www.fil.ion.ucl.ac.uk/spm/) was used for magnetic resonance imaging data preprocessing and analysis.

Preprocessing. ① Slice timing. The scanning sequence was interlayer scanned (1:2:31, 2:2:32), including 32 layers with the reference layer of 16. TR = 2, TA = 2–2/32. ② Realignment. Head motion correction was performed using least squares method and space transformation with 6 parameters (rigid body model), and the allowable head motion range (translation ≤ 3.0 mm, rotation ≤ 3.0°). ③ Normalization. The functional images were normalized to the Montreal Neurological Institute (MNI) template, with resampling resolution of 2 × 2 × 2 mm³, bounding box [−78 112 − 70; 78 76 85], and other default parameters. ④ Smoothing. The Gaussian kernel of 8 mm Full width at half maximum (FWHM) was selected for smoothing.

Task-related functional image analysis. The general linear model (GLM) was used for ROI-based analysis, the first 7 volumes of each scan were removed, and the head motion parameters were included as covariates. First, the RPE was calculated at the beginning of the second stage ( and at the time of the reward outcome (, with their general form expressed as Eq. (2). Second, the RPE was calculated as the partial derivative with respect to ω, which stands for the difference regressor between the RPEs with respect to model-based and model-free actions. The nuisance regressors included from Eq. (5) and its partial derivative with respect to [16].

Finally, a standard hemodynamic response function (HRF) was constructed to locate the RPE BOLD signals of prefrontal-striatal ROIs within and across groups (HC vs. MDD). The areas of interest (ROIs) were defined using PickAtlas v3.0.5 (https://www.nitrc.org/projects/wfu_pickatlas). The ROIs include: (1) medial (including BA25, 12) and lateral (including BA10, 11, 47) orbitofrontal lobes (i.e., MOFC, LOFC)(IBASPM71); and (2) midbrain/VTA (TD lobes); (3) ventral striatum/nucleus accumbens (i.e., VS/NA) (IBASPM71); (4) dorsal lateral and medial striatum (i.e., putamen, caudate).

The second level analyses for RPE BOLD signals of ROIs within and across groups were conducted. (1) One-sample t test. The model-based and model-free learning weight ω was used as covariate, and the significant RPE_MB and RPE_MF signals within the prefrontal-striatal ROIs in HC and MDD group respectively were examined. (2) Two-sample t test. The differences in RPE_MB and/or RPE_MF signals between HC and MDD groups were examined.

Correlation analyzes. Correlations between RPE ROI signals and symptoms (depression, anhedonia) were explored. The Pearson correlations between RPE_MB and/or RPE_MF ROI signals and depressive (BDI and DSS) symptoms across groups were examined. The significant correlation coefficients (P < 0.05) were reported.

Mediation model. We hypothesized that reward prediction error (RPE) brain signals mediate the relationship between model-based and/or model-free RL and severity of anhedonia. RL was the independent variable, severity of anhedonia the dependent variable, RPE brain signals of ROIs (including lateral and medial OFC, putamen, caudate, VTA and NAc) the mediators. The PROCESS macro in SPSS 19.0 (model 4) (https://www.ibm.com/cn-zh/spss) was utilized to examine the mediating role of RPE BOLD signal of ROIs between RL behavior and anhedonia. A bootstrapping analysis was conducted with 1000 resamples to calculate the bias-corrected 95% confidence intervals (CI) and test the significance of the mediation effect. The 95% CIs that did not include zero were regarded as significant [48].

The significance level is set at 0.05, and the cluster size is set at ≥ 5. The brain function activation map based on the peak voxel coordinates and significant clusters was reported, using xjview96 (http://www.alivelearn.net/xjview/) toolbox and the small volume correction (SVC) was adopted (P < 0.05, extent threshold k = 5 voxels), restricting the correction to the prefrontal ROIs (including lateral and medial OFC, putamen, caudate, VTA and NAc). For display purposes in the supplementary materials, we did whole-brain analyses and rendered activations at an uncorrected threshold of p < 0.0001, in combination with a cluster-based family wise error (FWE) correction of P_FWE<0.05 (i.e.,>336 voxels). MRIcro was used to output brain function activation map which was superimposed on the standard brain structure map (ch2bet).

Results

The individual- and multi-level behavioral analyses of choice behavior confirmed reduced model-based reward learning in depressed patients. The MB and MF components of RL were then entered into the mediation model to examine the indirect effects of stress on depressive/anhedonic symptoms with RL components as mediators. The impairments in neural substrates of prefrontal-striatal circuits underlying MB and MF RL of depressed patients were also examined.

Demographic and clinical data

There was no significant difference between the HC and MDD group (study 1) in the gender ratio (χ² = 1.07, df = 2, P = 0.30) or education level (χ² = 6.48, df = 3, P = 0.09). There were statistically significant differences in depression (BDI) (t=−7.44, P < 0.001) and anhedonia (MASQ_AD) (t=−7.49, P < 0.001) scores between the HC and MDD group. There was a statistically significant difference in stress (PSS) scores between the HC and MDD group (t=−5.21, P < 0.001) (Table 1).

Table 1.

Descriptive statistics of the demographic and clinical information

	Healthy controls (HC, n = 41)	Depressed patients (MDD, n = 49)	Young healthy adults (YHA, n = 183)
Demographics
age (M ± SD)	38.24 ± 7.62	40.10 ± 10.58	22.36 ± 2.88
gender ratio (M/F)	14/27	22/27	159/24
education (M ± SD)	2.91 ± 0.83	2.38 ± 0.95	4.04 ± 0.21
Symptoms
BDI (M ± SD)	7.66 ± 10.09	25.79 ± 13.60	5.74 ± 8.12
MASQ_AD (M ± SD)	49.20 ± 15.31	72.61 ± 13.69	48.87 ± 15.00
RSAS (M ± SD)	8.30 ± 5.98	16.24 ± 8.87	7.34 ± 5.27
Stress
PSS (M ± SD)	23.80 ± 7.63	31.13 ± 5.10	22.96 ± 6.28

Education level is an ordinal variable that is divided into four categories: junior high school (1), senior high school (2), college degree (3) and postgraduate degree (4)

The effect of depression on model-based and model-free RL

The stay probabilities for common vs. rare transition, with reward vs. nonreward outcome across groups were examined (Fig. 2).

Fig. 2.

Stay probability of different groups (HYA, HC and MDD) (Mean ± SEM)

Across the groups, the logistic regression analyses yield that: (1) the main effect of the outcome was statistically significant in the HC group (β = 0.14, P < 0.001) and the depressed patients (β = 0.11, P = 0.007); (2) the main effect of the outcome was statistically significant (β = 0.25, P < 0.001), and the transition-by-outcome interaction was statistically significant (β = 0.08, P < 0.001) in the HYA group. The results suggested that the HC and MDD group mainly adopt model-free RL, while the HYA group adopt both model-free and model-based RL. Across the three groups, the intercept item had a statistically significant main effect (P < 0.001), suggesting that all groups had a tendency to repeat the first stage selection of the previous trial, which was independent of reward outcome or transition condition (Table 2).

Table 2.

Generalized linear mixed-effects model of reward outcome, transition type and anhedonic symptoms on stay behavior across groups

Predictor	Estimates of effects (SE)		p	Model evidence (AIC, BIC, -LL)
HC
Intercept	1.54(0.20)	7.59	< 0.001***	7446.8, 7544.5, −3709.4
Outcome	0.14(0.06)	2.43	0.02*
Transition	0.07(0.04)	1.67	0.09
Outcome×Transition	0.04(0.04)	0.96	0.34
HYA
Intercept	1.18(0.08)	13.98	< 0.001***	36855.0, 36973.5, −18413.5
Outcome	0.25(0.03)	9.54	< 0.001***
Transition	−0.002(0.02)	−0.13	0.89
Outcome×Transition	0.08(0.02)	4.70	< 0.001***
MDD
Intercept	0.44(0.10)	4.46	< 0.001***	9590.9, 9687.5, −4781.5
Outcome	0.11(0.04)	2.70	0.007**
Transition	−0.008(0.03)	−0.27	0.78
Outcome×Transition	0.02(0.03)	0.67	0.50

 *P < 0.05; **P < 0.01; ***P < 0.001. SE = standard error; AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; -LL = negative log-likelihood

Notably, although the main effect of transition within each group was not statistically significant, the main effects of participants’ group, outcome as well as the group-by-transition interaction were statistically significant across groups (MDD and HC) (Ps ≤ 0.05) (Table 3). The rewarded trials tend to induce more stay behaviors across groups, which was similar with within-group results. Furthermore, the main effect of group suggested that depressed patients are less inclined to choose the stay behavior in the first stage, and this tendency persists even when the previous trial is common transition or rewarded.

Table 3.

Generalized linear mixed-effects model of reward outcome, transition type and group on stay behavior

Predictor	Estimates of effects (SE)		p	Model evidence (AIC, BIC, -LL)
Intercept	1.54(0.20)	7.56	< 0.001***	18176.5, 18515.0, −9044.3
Outcome	0.15(0.06)	2.43	0.015*
Transition	0.07(0.04)	1.67	0.09
Outcome×Transition	0.04(0.04)	0.96	0.337
Group	−1.08(0.22)	−4.75	< 0.001***
Group×Outcome	−0.03(0.07)	−0.41	0.68
Group×Transition	−0.10(0.05)	−1.99	0.046*
Group×Outcome×Transition	−0.03(0.05)	−0.67	0.50

*P < 0.05; **P < 0.01; ***P < 0.001. SE = standard error; AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; -LL = negative log-likelihood

Computational RL model fit within and between groups

The default RL model parameters between groups

As default, we first adopted the assumption that the hybrid model is the best fitting model across all participants (which was only supported in the HC group according to Sect. "The RL model comparison within groups", Fig. 2). The Shapiro-Wilk test was used to investigate whether the RL parameters of each group were normally distributed. Those participants with too much missing data were deleted from the analyses (n = 5). For the parameters estimated from the behavioral data, values more than two standard deviations (95% confidence intervals) above the group mean value are marked as outliers and also excluded from the analyses.

The results showed that parameters were not normally distributed (P < 0.05). As a result, the nonparametric Mann-Whitney U test was used to investigate whether there were significant intergroup differences in the RL parameters.

For simplicity and in convenience of comparison, we listed the RL parameters for both groups as the default hybrid model (MB and MF combined, 7 parameters). The depressed patients had a significantly lower learning rate (α₂), lower model-based learning weights (ω) and more frequent transitions of options (ρ) than the healthy control group. The model evidence (Bayesian information criterion, Laplace approximation of Bayes factor, and negative log-likelihoods) of the behavioral data fitted by the seven-parameter hybrid model between the two groups were statistically significant (Ps < 0.001), with better model fit of the hybrid model for HC and YHC group (Table 4).

Table 4.

Hybrid reinforcement learning model parameters for each group

Parameter	MDD (n = 44)			HC (n = 41)			Z (MDD vs. HC)	P	YHC (n = 176)			Z (MDD vs. YHC)	P
	25%	50%	75%	25%	50%	75%			25%	50%	75%
α1	1.698	2.301	2.509	1.706	1.913	2.294	−1.504	0.133	1.577	1.930	2.371	−0.175	0.861
α2	0.726	1.151	1.678	−0.261	0.692	1.456	−2.436	0.015*	0.746	1.195	1.590	−3.352	0.001**
β1	−3.032	−2.307	−0.960	−3.809	−2.822	−1.870	−2.331	0.02*	−2.612	−1.778	−0.271	−4.380	< 0.001***
β2	−3.110	−1.806	−0.162	−3.745	−2.384	−0.167	−1.222	0.222	−2.761	−1.007	0.102	−2.351	0.019
λ	−0.784	0.077	0.634	−1.839	−1.147	−0.449	−3.272	0.001**	−1.038	0.047	0.946	−4.337	< 0.001***
ω	−1.976	−1.301	−0.667	−2.593	−1.806	−1.093	−2.964	0.003**	−1.625	−1.137	−0.536	−4.396	< 0.001***
ρ	0.015	0.057	0.120	−0.002	0.016	0.043	−3.966	< 0.001***	0.025	0.083	0.170	−5.672	< 0.001***
BIC	494.831	451.153	368.533	450.465	349.374	265.124	−3.412	0.001**	461.231	390.547	298.990	3.923	< 0.001***
Lap	269.954	250.702	210.660	252.125	202.020	159.735	−3.085	0.002**	222.453	180.786	145.637	3.835	< 0.001***
-LL	202.828	244.138	265.977	151.124	193.249	243.794	−3.412	0.001**	168.057	213.835	249.177	3.923	< 0.001***

α1, α2, learning rates of stage 1 and stage 2, respectively; β1, β2, choice repetition of stage 1 and stage 2, respectively; λ, reinforcement eligibility trace; ρ, perseverance parameter; ω, model-based versus model-free parameter weights; BIC, Bayesian information criterion; Lap, Laplace approximation of Bayes factor; -LL, negative posterior log-likelihood estimation. Multiple comparison was corrected by the Bonferroni method, and the significance threshold was set at P_Bonferronis<0.007. *** P < 0.001, **P < 0.01, *P < 0.05

Specifically, the parameter (λ) of depressive patients is significantly lower than that of the normal control group, and is close to 0 (Table 2). The parameter (λ) can reflect the relative influence of the action value of the second stage and the final reward outcome on the choice of the first stage. If λ = 1, it means that only the final reward outcome affects the choice of the first stage (model-free learning). If λ = 0, it means that only the action value of the second stage affects the choice of the first stage (model-based learning). The above results suggest that the final reward outcome exerts less influence on the first stage action selection of depressed patients, showing deficits in MF RL.

Depression patients’ model-based RL weight (ω) decreased (Table 2), mainly adopted MF learning strategies, and depend less on changes in environmental state (e.g., transition probability) to modify the action strategy. Additionally, the second stage learning rate (α) decreased (Table 2), which revealed that the degree of reward prediction value updates slower in depressed patients, which then may result in decreased stay behavior for the next trial.

The RL model comparison within groups

Using random-effects Bayesian model selection (BMS) [49] for different groups, the results showed that the hybrid RL model (MB and MF) best fit the behavioral data of the HC group (inside-scanner) (Fig. 3, A1), while the MF RL model best fit the behavioral data of the depressed patients (inside-scanner) (Fig. 3, B1). When all healthy participants, HC (n = 219) (inside & outside-scanner) were included, similar results were found (Fig. 3, A2), suggesting the hybrid RL model fit across healthy individuals. To replicate the results, we also conducted model comparison for all depressed patients (n = 43, from study 2), which yielded similar results of best MF model fit (Fig. 3, B2).

Fig. 3.

Comparison of the degree of model fit of different computational models: the models with the optimal fit for the healthy control group and depressed group were the mixed model and the model-free RL model, respectively. Note: A1: model evidence for the HC group (n = 19, inside-scanner) (-LL, △iBIC); B1: model evidence for the MDD patients (n = 17, inside-scanner) (-LL, △iBIC); A2: model evidence for all healthy participants (HYA and HC) (n = 219, inside & outside-scanner) (-LL, △iBIC); B2: model evidence for all MDD patients (n = 43, inside & outside-scanner) (-LL, △iBIC). The dotted lines represent the estimated parameters of each subject

Collectively, the above results were consistent with the model evidence and intergroup comparisons based on RL weights (Table 2). The model evidence suggested that for the hybrid model used to fit the behavioral data of the participants, the degrees of model fit (BIC, LAP and -LL parameters) of the HC group were better than those of the MDD group.

The mediating role of RL between stress and anhedonia/depression

The results showed that for depressed patients, stress could influence anhedonic symptoms (MASQ_AD) via model-based (β=−37.73, SE = 16.64, P = 0.03, 95% CI=−71.38~−4.08) and model-free (β=−17.73, SE = 16.64, P = 0.03, 95% CI=−71.38~−4.08) RL. However, no significant mediating role was found for either model-based (β=−139.50, SE = 96.25, P = 0.16, 95%CI=−334.18 ~ 55.18) or model-free (β=−20.75, SE = 17.43, P = 0.24, 95%CI=−56.01 ~ 14.52) RL between stress and depressive symptoms.

The prefrontal-striatal neural substrates of RL in depression

For the MDD group, the RPE_MF signals in LPFC, midbrain/VTA and LOFC were less activated than those in the HC group. The RPE_MF signal in LPFC was greater than in the HC group (Table 5; Fig. 4). No significant activation with RPE_MF was found for the between-group comparison of whole-brain analyses (cluster based P_FWE<0.05).

Table 5.

Model-free RPE in PFC-striatal circuits between groups

Region	Side	MNI Coordinates			Cluster size	t score
		x	y	z
HC > MDD
LPFC	L	−52	20	12	33	2.733
Midbrain/VTA		0	−20	−14	25	2.196
LOFC	L	−32	26	−20	7	1.850
MDD > HC
LPFC	R	44	8	28	10	2.036
LPFC	L	−46	8	26	5	1.957

ROI-based analyses, P < 0.05, extent threhold k = 5 voxels, small volume corrected

Fig. 4.

The differences of Model-free PRE in PFC-striatal circuits between groups. (A) Left LPFC, (B) Midbrain, (C) Left LOFC, (C) Right LPFC, (D) Left LPFC

In addition, for the MDD group, the RPE_MB signals in the LPFC, MPFC, LOFC, MOFC, midbrain, and dorsal striatum (putamen, caudate) were less activated than those in the HC group (Table 6; Fig. 5). The RPE_MB signals in the LPFC and MPFC were more activated than those in the HC group (Table 6; Fig. 6). No significant activation with RPE_MB was found for the between-group comparison of whole-brain analyses (cluster based P_FWE<0.05).

Table 6.

Model-based RPE in PFC-striatal circuits between groups

Region	Side	MNI Coordinates			Cluster size	t score
		x	y	z
HC > MDD
LPFC	R	42	38	34	232	3.387
MPFC	L	−2	48	−10	31	2.249
LOFC	L	−30	30	−18	278	3.875
LOFC	R	42	30	−4	15	2.278
MOFC	L	−8	20	−18	25	2.594
MOFC	L	−2	52	−10	5	1.949
Midbrain	R	4	−14	−4	76	2.664
Putamen	L	−14	8	−6	11	2.026
Caudate	L	−12	10	−6	13	2.131
MDD > HC
LPFC	L	−44	42	10	45	2.224
LPFC	R	44	22	42	5	1.886
MPFC	R	2	30	60	45	2.598
MPFC	L	−6	46	46	17	1.984

ROI-based analyses, P < 0.05, extent threhold k = 5 voxels, small volume corrected

Fig. 5.

The differences of model-based PRE in PFC-striatal circuits between groups (HC > MDD). (A) Right LPFC, (B) Left MPFC, (C) Left LOFC, (D) Right LOFC, (E) Left MOFC, (F) Left MOFC, (G) Midbrain, (H) Putamen, (I) Caudate

Fig. 6.

The differences of model-based PRE in PFC-striatal circuits between groups (MDD > HC). (A) Left LPFC, (B) Right LPFC, (C) Right MPFC, (D) Left MOFC

Discussion

The current study examined the effects of depression/anhedonia on model-based and model-free RL. Our results suggested that depressed patients were inclined to search for other possible rather than stay with their original strategies. They showed a relative shift from model-based to model-free control. The decreased activation in prefrontal-striatal circuits (L/MOFC, VTA and DS) was shown in the depressed patients during model-based and model-free RL.

The reduced MB learning in depression

The results revealed that the reward outcome rather than the outcome-by-transition interaction significantly influenced the first-stage choice of depressed patients, suggesting that depressed patients mainly adopted model-free RL strategies, with nonsignificant differences from the healthy controls. These results suggested that the model-free RL of depressed patients was relatively unimpaired. Computational modeling results showed that the hybrid model (MB and MF) best fit the choice behavior of the healthy control group, while the MF model best fit the choice behavior of depressed patients, which supported hypothesis (1). This study also showed that depressed patients’ propensity to switch from previous behavior choices persisted even when participants were faced with positive reinforcement or the presence of common transitions in the environment (more frequent transitions of options (ρ) than the healthy control group). This is consistent with previous findings that when faced with an uncertain environment, individuals with depressive symptoms tend to explore alternative options in a disorderly manner rather than sticking to original action strategies to maximize expected benefits [18]. Presumably, this may reflect the decreased representation of task structure in depressed individuals. Depressed patients may reduce the complexity of state space sampling by reducing the number of differentiable states, shifting the balance between model-free and model-based strategies toward the former [50]. This assumption was supported by the neuroimaging results of depressed patients in which RPE_MB signals of OFC (encoding task state space, see Sect. "The neural deficits in model-based RL for depression") [51] were less activated when compared to HC group.

Stress-RL deficits model of anhedonia

The study found that MB and MF RL played the mediating role between stress and anhedonic symptoms, which confirmed the hypothesis (2) and consistent with previous evidence that physiological and psychosocial stress may affect individual MB RL but not MF RL [20, 22]. In particular, chronic stress combined with acute psychosocial stress reduces MB behavioral control, while acute psychosocial stress alone does not change MB behavioral control [20]. Recent evidence showed that chronic stress may decrease the model-free behaviors (lever press) as well as model-based behaviors (high fixed-rate reinforcement identification) during instrumental conditioning of rats [52]. Previous evidence suggests that stress affects the structure and function of the prefrontal cortex [53], which may result in the shift from model-based to model-free control [19, 54]. Furthermore, model-based RL deficits result in a reduced sense of controllability, with stress-induced serotonin responses and changes in adaptive behavior (such as reduced approach coping and increased avoidance coping), leading to a decrease in contact with positive reinforcement, which may then exacerbate anhedonia [55, 56]. Above all, these studies suggest that the cumulative effect of chronic stress may be a risk factor leading to MB and MF RL deficits.

The neural deficits in RL for depression

Model-based brain function imaging showed that in the healthy control group, there were RPE_MF signals in NAc, putamen, L/MOFC, and RPE_MB signals in LPFC, consistent with previous findings [57–59]. The comparison between groups revealed deficiency in the prefrontal (OFC, MPFC) and striatal (VTA, DS, NAc) regions as well as aberrant functional connectivities underlying model-free and model-based RL, which is consistent with our hypothesis (3).

The neural deficits in model-free RL for depression

VTA↓ and VTA-NAc↑. First, this study found reduced RPE_MF signal of VTA in the depressed patients, which is consistent with previous studies [31, 60]. Additionally, we did not find altered RPE_MF signal of NAc in depressed patients, consistent with previous study [61]. We also found that the RPE signal in NAc is negatively correlated with anhedonia, consistent with previous studies [31]. NAc is involved in transformation of reward value to action [62]. However, for depressed patients, the signal could not be used to update the reward expectancy, which is critical for the transformation into action value. Moreover, this deficiency worsened with increased anhedonia in depressed patients [61]. Second, as a complement, our study found statistically significant positive VTA-NAc FC in the MDD group (r = 0.941, P < 0.001), as compared to no statistically significant FC in the HC group (r = 0.128, P = 0.600). VTA projects dopamine-releasing neurons to target areas such as NAc (the mesolimbic pathway) and MPFC (the mesocortical pathway). Therefore, reduced VTA activation may lead to compensatory increase of VTA-NAc functional connectivity in depressed patients. These results did not support the receiver-end deficit hypothesis (NAc deficit -> VTA upregulation) [63]. The accumulating evidence highlights the impairment in VTA-NAc pathway underlying anhedonia in depressed patients, which merits further examination with circuit-based or network-based metrics.

The neural deficits in model-based RL for depression

DS↓. This study found that the RPE_MB signal of DS (putamen, caudate) was decreased in the depressed patients, consistent with previous studies [31]. DS tracks action-related task components and has bidirectional functional connectivity to the ventral striatum. The caudate is responsible for representing model-based behavior, and the putamen is responsible for representing model-free behavior [64, 65]. Specifically, the putamen represents and keeps track of the goal selection signal until the next goal is selected, while the caudate mainly encodes the response and outcome information and dynamically maintains the representation of the goal selection signal [66]. Therefore, the results suggest that while encoding RPE_MF signals, the representation and maintenance of goals and outcomes may be reduced in the patients with depression, leading to a decline in model-based RL ability.

LOFC↓. Notably, both the RPE_MB and RPE_MF signals of lOFC were decreased in the depressed patients. Furthermore, the mediating role of lOFC’s RPE_MB signal between model-based RL and depressive symptoms was observed. lOFC is involved in both model-based and model-free RPE encoding [67]. OFC encodes a cognitive map of task space which represents the current (especially unobservable) state of the task [51]. LOFC activity is necessary for value updating in a decision-making task which requires model-based control [64]. Moreover, lOFC is involved in salience processing which is sensitive to both positive and negative clues [68]. Therefore, the value updating during model-based as well as model-free RL may be dampened for depressed patients.

MOFC↓. The patients with depression showed decreased RPE_MB signal in MOFC, but no abnormalities of RPE_MF in MOFC were found, which is inconsistent with previous study [60]. However, this study showed that RPE_MF is negatively correlated with depressive symptoms. MOFC is anatomically extensively connected with visceral and motor areas. MOFC is functionally responsible for sensing internal states, encoding the relative value of current actions [69], and associating outcome value with action selection [70]. The results of deficits in RPE_MB signal MOFC suggest that depressed patients fail to estimate and update the action value to make model-based decisions.

Limitations and directions for future research

The total reward of practice was not taken into consideration to assess the effectiveness of practice. Therefore, there is a lack of objective indexes to validate if the subjects really understood the task. We could not parse overall performance from MF vs. MB strategies. Future studies could include more measurement indicators to assess the degree of task understanding (such as a questionnaire on task structure knowledge, and the comparison between the total scores earned in the practice task) and to parse general performance from RL strategies.

The neuromodulation effects (e.g., dopamine and serotonin) were not assessed and taken into consideration. Previous evidence suggests that elevated dopamine (DA) levels can promote model-based selection behavior [29]. In contrast, reduced DA synthesis and conduction can lead individuals to be more prone to habitual behavior [71]. Serotonin (5-HT) levels regulate the balance between model-based and model-free behavior [72]. Therefore, the individual differences in neurotransmitters may cause an imbalance in the effect of the two types of RL. Future studies need to consider the influences of neurotransmitters underlying deficits in RL processes for depression.

Due to the sample size of the current study, the whole-brain analyses based on family wise error (FWE) correction did not yield significant activation of between-group comparison related to reward prediction error, either model-based or model-free. However, the significant ACC activation of model-free RPE for the HC group confirmed its role of error detection which drives behavioral switch [73].

Future intervention studies are needed to investigate the cause-effect relationship between RL deficits and depressive/anhedonic symptoms. For example, behavior activation (BA) therapy is recommended by the WHO for supplementary intervention of moderately severe depression, which theoretically aims to engage patients in goal-directed behavior consistent with their values in life [74, 75]. It is still unclear whether BA intervention alters the RL (especially model-based learning) function underlying prefrontal-striatal circuits, which may be promising targets for the personalized treatment of depression. Finally, due to the modest sample size, future studies should replicate the findings to validate the conclusions.

Conclusions

1. Depressed patients showed reward learning (i.e., model-based RL) deficits compared to healthy controls.

2. Model-based and model-free RL deficits mediated the relationship between stress and anhedonic (or depressive) symptoms for both groups, with underlying specific neural signatures (i.e., RPE_MF signals in VTA and caudate).

All participants provided written informed consent.

Author contributions

W.X. and F.Z. design the study, W.X. drafts the manuscript, W.X., H.J. and F.Z. review and revise the manuscript, Z.X. and W.X. recruit the participants, Z.D. performs imaging data acquisition. W.X., Z.Z., H.J. and G.Y. analyze and visualize the data.

Funding

This research was financially supported by the Nursery Fund for Young Talents in the Army Medical University (410301053421).

Data availability

The datasets in the current study are available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.