Impulse control disorder in Parkinson’s disease is associated with abnormal frontal value signalling

Abstract

Dopaminergic medication is well established to boost reward- versus punishment-based learning in Parkinson’s disease. However, there is tremendous variability in dopaminergic medication effects across different individuals, with some patients exhibiting much greater cognitive sensitivity to medication than others. We aimed to unravel the mechanisms underlying this individual variability in a large heterogeneous sample of early-stage patients with Parkinson’s disease as a function of comorbid neuropsychiatric symptomatology, in particular impulse control disorders and depression.

One hundred and ninety-nine patients with Parkinson’s disease (138 ON medication and 61 OFF medication) and 59 healthy controls were scanned with functional MRI while they performed an established probabilistic instrumental learning task. Reinforcement learning model-based analyses revealed medication group differences in learning from gains versus losses, but only in patients with impulse control disorders. Furthermore, expected-value related brain signalling in the ventromedial prefrontal cortex was increased in patients with impulse control disorders ON medication compared with those OFF medication, while striatal reward prediction error signalling remained unaltered.

These data substantiate the hypothesis that dopamine’s effects on reinforcement learning in Parkinson’s disease vary with individual differences in comorbid impulse control disorder and suggest they reflect deficient computation of value in medial frontal cortex, rather than deficient reward prediction error signalling in striatum.

See Michael Browning (https://doi.org/10.1093/brain/awad248) for a scientific commentary on this article.

Tichelaar et al. examine the effect of dopaminergic medication on reward learning in Parkinson’s disease. Patients with impulse control disorders are particularly sensitive to rewards when on dopaminergic medication, and this reward sensitivity is associated with increased value signalling in the medial prefrontal cortex.

Introduction

Parkinson’s disease (PD) is the most rapidly growing neurodegenerative disease, expected to affect 12 million people worldwide in 2040.¹ The cardinal motor symptoms are bradykinesia, rigidity and tremor, which are related to a severe loss of dopamine in the basal ganglia. However, the disease is also accompanied by various cognitive impairments,^2-4 for example in learning, memory and/or attention. These cognitive impairments are known to contribute to neuropsychiatric symptoms such as depression and impulse control disorder (ICD). Approximately 14% of patients with PD are diagnosed with ICD, manifesting as pathological gambling, eating, shopping and/or hypersexuality.⁵ Depression is also common, occurring in ∼35% of patients with PD.⁶ Interestingly, depression is a risk-factor for developing ICDs,⁷ and there is considerable comorbidity between these symptoms, suggesting that common mechanisms may play a role.⁸ Patients on dopamine receptor agonists are particularly vulnerable to developing ICDs,⁹ and dopaminergic medication can alleviate depressive symptoms. However, there are large interindividual differences in the sensitivity to dopaminergic medication, while certainly not all patients with PD develop these severe side effects. The origins of this variability remain poorly understood, but likely implicate abnormal reward-related activity in cortical-striatal-thalamo-cortical circuitry.^8,10,11 Here we aimed to understand the cerebral mechanisms underlying this variability. To this end, we assessed the effect of dopaminergic medication on reinforcement learning (RL) in a large (n = 199) and heterogeneous sample of early-stage patients with PD (Personalized Parkinson Project, PPP), allowing us to compare patients with and without depression as well as those with and without ICDs.

RL involves the gradual, incremental learning of associations, characteristic of the formation of habits^12,13 and this process is well known to implicate reward prediction error (RPE) signalling in the dopamine-rich striatum.^10,14-22 In line with the canonical striatal dopamine hypothesis of RL,^23-25 many studies have revealed that even mildly affected patients with PD exhibit deficits on tasks that require RL,^12,26-28 and these RL deficits depend on dopaminergic medication.^29-31 Specifically, as predicted by influential Go/Nogo basal ganglia pathway models of RL,^22,32 dopaminergic medication in patients with PD improves performance on tasks requiring learning from gains, while impairing performance on tasks requiring learning from losses.^{16,17,22,30,33-37} This pattern of effects has been replicated many times across different laboratories and has been proposed to contribute to depression, dopamine dysregulation syndrome and ICD in patients with PD.^38-41 However, there are three key outstanding questions regarding the RL impairment in PD.

First, there are large individual differences in the degree to which patients with PD exhibit dopamine-dependent RL deficits. For example, separate studies have shown greater medication-related shifts towards gain versus loss learning in non-tremor patients with PD than in patients with tremor,⁴² in patients with versus without depression¹¹ and in those with versus without ICD.^10,43 For example, Voon et al.¹⁰ have shown that dopaminergic medication boosts gain learning, but only in patients with ICDs (i.e. pathological gamblers or shoppers). This medication effect in patients with ICDs was associated with abnormal striatal RPE signalling as well as abnormal expected value (EV)-related signalling in the frontal cortex.¹⁰ Furthermore, we have shown that patients with PD with higher medication doses exhibit greater impairments in probabilistic reversal learning than patients with lower medication doses, in line with the original dopamine overdose hypothesis,³⁶ but this effect was present only in patients who also suffered from depression.⁴ Moreover, patients with PD with depression exhibited greater medication-related decreases in loss aversion during risky choice on a gambling task than patients without depression.⁴⁴ These findings concur with clinical evidence indicating that PD patients with more severe depressive symptoms are at increased risk for developing medication-related ICD,^7,40,44-46 possibly due to deficits in dopamine autoregulatory mechanisms in fronto-striatal circuitry. This interindividual variability in dopaminergic medication effects may explain why some studies have failed to reveal any (dopamine-related) deficits on RL when collapsing data across all patients with PD.^11,47-49 Here we aimed to more definitively resolve the clinical, cognitive and neural factors that contribute to this individual variability in cognitive medication effects by studying RL in the largest sample of patients with PD to date (n = 199).

A second outstanding issue follows from the first: If there is such large interindividual variability in dopamine-dependent RL impairments in PD, then it is less likely that these impairments reflect the core pathology of PD, i.e. impaired dorsal striatal signalling. Thus, while some prior work with small sample sizes has suggested that RPE signals during RL are impaired in dorsolateral striatum⁵⁰ (but see Cools et al.¹⁵), we ask here in a much larger sample whether depression and ICD-related RL impairments reflect dopamine-dependent changes in other nodes of the cortical-striatal-thalamo-cortical reward network, particularly the prefrontal cortex, which is more variably affected across different patients.^51-54

Finally, a third outstanding question concerns the computational mechanisms underlying the deficient task performance: Does it reflect a change in learning or a change in motivational biases of choice? While various studies implicate a role for aberrant computation of canonical RPE learning signals in the striatum,^10,14-22 there is also evidence for medication-related changes in reward-based choice that cannot be attributed to changes in learning.^32,47,55 Here we investigate in a much larger sample of PD patients whether medication-related changes during RL are accompanied by changes in: (i) striatal RPE learning signals during outcomes; (ii) EV-related signals at the time of choice, more pervasive in the ventromedial prefrontal cortex (vmPFC); or (iii) both.²⁰ As such, the study also contributes to the longstanding debate about dopamine’s contribution to RPE-based learning versus value-based choice.^47,55-57

Based on the canonical RPE hypothesis of striatal dopamine²³ and prior empirical findings,^10,15,29 we hypothesize to replicate previous findings that, relative to patients OFF medication, patients ON dopaminergic medication exhibit a shift from loss towards gain learning and that this behaviour is accompanied by increased RPE signalling during gain versus loss outcomes in the (ventral) striatum.^{10,15,16,19,21,22} We will also test the (non-exclusive) alternative hypothesis, that effects on gain versus loss trials are associated with abnormal EV-related signals at the time of choice in the vmPFC. Furthermore, we expect these valence-specific medication group effects in RL to be particularly pronounced, and perhaps present only in patients with PD with depression¹¹ and/or ICD,¹⁰ because of their overly dynamic ventral reward circuitry.⁸

Materials and methods

Ethics

This study was approved by the local institutional review board (Commissie Mensgebonden Onderzoek Region Arnhem-Nijmegen; reference number 2016–2934; NL59694.091.17) and was conducted in accordance with the Ethical Principles for Medical Research Involving Human Subjects, as defined in the Declaration of Helsinki (version amended in October 2013). All participants gave written informed consent.

General procedure

The current study adopted a between-subject design. All PD patients were part of the PPP, which is a single-centre, longitudinal observational study in 520 PD patients.⁵⁸ Inclusion criteria were a diagnosis of idiopathic PD, a disease duration of <5 years, 18+ years of age and absence of contraindications for MRI (see Supplementary material for a detailed description of participant acquisition). At the time of analysis, 457 patients had been included in the PPP (see Bloem et al.,⁵⁸ for a power analysis), for which they underwent two separate measurement sessions across 2 days. On Day 1 of the PPP, they were tested and scanned using functional MRI (fMRI) while they were ON dopaminergic medication, under their normal medication regime. On Day 2, they were tested OFF dopaminergic medication, after withdrawal from their normal medication regime for at least 12 h prior to the onset of the measurements. This Day 2 OFF session did not include any fMRI but did include clinical and neuropsychological assessments. Prior to this Day 2 OFF session, patients stayed overnight in a local hotel. On Day 2 and partially at home, participants took our clinical and neuropsychological test battery (for details see Supplementary material).

Of the 457 PPP patients, 145 patients completed the RL task during fMRI scanning on Day 1, ON medication, and these are the ON data reported here. The other 312 patients completed a different motor selection task during scanning on Day 1, the data of which will be reported elsewhere. Of the 145 patients who completed the RL task on Day 1 reported here, 138 were measured ON their own medication regime, while seven were de novo patients who had never taken dopaminergic medication. In addition to an fMRI scan during RL task performance, the Day 1 protocol also included a T₁ anatomical scan.

From the subset of 312 patients who completed the motor selection task (but not the RL task) on their first visit, 59 patients were asked to return for another MRI session OFF their normal medication regime. During this second session, they completed the RL task during fMRI. Again, patients refrained from taking medication for 12 h before onset of the session. For these patients the clinical data were obtained from their first PPP visit.

Sixty healthy controls (HCs) were also asked to perform the RL task in the MRI scanner, and to complete a short battery of tests (including the Montreal Cognitive Assessment; Table 1). They also filled out the neuropsychiatric rating scales [i.e. Beck Depression Inventory II (BDI-II), Questionnaire for Impulsive-Compulsive Disorders in Parkinson’s disease Rating Scale (QUIP-rs) and Anxiety Inventory for Adults (STAI)].

Table 1.

Patient characteristics

	ON		OFF		HC	Main effect ICD	Main effect MED	MED × ICD
	PD ICD+	PD ICD−	PD ICD+	PD ICD−
Number of participants, n	36	96	12	46	60	–	–	–
Age, years	57.25 (10.2)	61.08 (9.08)	63.25 (7.07)	60.46 (9.68)	60 (9.61)	n.s.	n.s.	n.s.
Gender, female/male	14/22	55/40	2/10	23/23	27/33	**	n.s.	n.s.
Depression (BDI)	13.22 (8.01)	8.52 (6.28)	12.92 (6.82)	7.39 (4.73)	4.6 (3.51)	***	n.s.	n.s.
ICD (QUIP-rs)	26 (10.13)	3.68 (4.27)	23.08 (11.9)	5.57 (5.07)	21.98 (10.13)	***	n.s.	n.s.
Anxiety (STAI:trait)	42.06 (9.28)	34.73 (9.91)	42.75 (10.64)	32.87 (7.97)	33.76 (6.77)	***	n.s.	n.s.
Disease severity (UPDRS:ON)	28.56 (12.16)	25.73 (11.02)	32.36 (13.43)	26.7 (13.62)	–	n.s.	n.s.	n.s.
Disease severity (UPDRS:OFF)	32.28 (12.52)	31.39 (11.82)	38.08 (13.61)	33.15 (13.45)	–	n.s.	n.s.	n.s.
LEDD	700.64 (480.96)	499.81 (237.45)	535.06 (237.57)	400.22 (267.79)	–	**	*	n.s.
Dopamine receptor agonist use, yes/no	20/16	28/68	6/5	15/24	–	**	n.s.	n.s.
Brixton	14.33 (7.35)	13.44 (6.17)	13.92 (6.53)	15.26 (5.9)	–	n.s.	n.s.	n.s.
Semantic fluency	24.19 (6.26)	25.89 (5.13)	25.08 (6.37)	25.72 (6.64)	–	n.s.	n.s.	n.s.
Symbols and digits	33.81 (9.02)	38.21 (7.83)	35.58 (6.3)	38.54 (5.4)	–	*	n.s.	n.s.
Months since diagnosis	27.53 (18.33)	24.37 (15.44)	34.17 (17.96)	32.52 (15.73)	–	n.s.	*	n.s.
Resting tremor	1.08 (1.34)	1.21 (1.67)	0.83 (1.59)	1.5 (2.13)	–	n.s.	n.s.	n.s.

We subdivided the Parkinson’s disease (PD) population based on impulse control disorder (ICD) by using a clinical cut-off score for the Questionnaire for Impulsive-Compulsive Disorders in Parkinson’s disease Rating Scale (QUIP-rs). For a subdivision for depression, see Supplementary Table 5. To compare groups, we used a 2 × 2 ANOVA (Med × ICD-group). We found no interaction effects between medication (MED) and ICD-group. For gender and dopamine agonist use, we performed a chi-square test for both ICD versus non-ICD and ON versus OFF medication. *P < 0.05, **P < 0.01, ***P < 0.001. n.s. = not significant. BDI = Beck Depression Inventory; HC = healthy controls; LEDD = levodopa equivalent daily dose; UPDRS = Unified Parkinson’s disease Rating Scale; STAI = Anxiety Inventory for Adults.

Participants were reimbursed for a hotel stay on the night before the measurements as well as their travel expenses. Furthermore, participants received 5% (€4.30 on average) of their total winnings from the RL task. We found no effect of medication group on the amount of money received [F(22,49) = 1.28, P = 0.28].

Reinforcement learning task

Participants completed three blocks of a probabilistic instrumental learning paradigm.¹⁸ The first block was a training session, performed outside the scanner with a keyboard. The second and third blocks were performed inside the scanner with a fMRI-compatible button box. Each block consisted of 28 gain trials pseudo-randomly interleaved with 28 loss trials and took approximately 5 min to complete. Intertrial intervals during the training sessions were randomly drawn from a normal distribution with a mean of 1 s. For the fMRI blocks, intertrial intervals were optimized using the optseq2 procedure,⁵⁹ also with a mean of 1 s.

For each block, two unique sets (one per trial type) of two abstract visual cues were randomly selected from eight cue pairs (based on Pessiglione et al.¹⁸). The side of the screen at which each cue was presented was randomized across trials. Participants were instructed to maximize their monetary payoff and to make their decisions ‘within time.’ They were told to press a key corresponding to the side of the screen where they believed the ‘correct’ cue was depicted (full instructions in supplement). Each cue pair stayed on the screen for 2.94 s, but participants had to respond within 2.5 s. As soon as the participant responded, visual feedback was provided indicating the chosen cue with an arrow. After 2.94 s, the monetary outcome of the current trial was depicted for 1 s.

The outcome of a trial depended on both the trial type (gain or loss) and outcome probability. For both trial types, the ratio of reward:punishment was 75:25. In gain trials, the rewarding outcome was +€10 and punishing outcome was €0, while in loss trials the rewarded outcome was €0 and the punished outcome −€10. The outcome was depicted by both text and the image of a €10 euro bill indicating the monetary amount. If the participant gained money, the text was green, if money was lost, the text was red, and a red cross was drawn across the €10 euro bill. The money was replaced with a grey rectangle with a red €0 above, on trials without reinforcement. Participants were paid according to their performance (5% of the average payoff across two blocks). The task is depicted in Fig. 1A.

Figure 1.

Task description and a Win-Stay-Lose-Shift trial explanation. (A) Participants performed a probabilistic instrumental learning task adapted from Pessiglione et al.¹⁸ (B) Example of successful Win-Stay-Lose-Shift (WSLS) behaviour (i.e. Win-Stay and Lose-shift trials) as well as an example of aberrant WSLS behaviour (i.e. Win-Shift trial). WSLS measures the degree to which the outcome of the previous trial influences the switching behaviour of the current trial. RT = reaction time.

Analyses of behavioural performance

We performed a Bayesian mixed effects logistic regression on trial-wise accuracy and stay behaviour [to assess win-stay-lose-shift (WSLS) behaviour]. To do so, we used R version 3.6.1⁶⁰ and the brm function from the brms package.⁶¹ The model space is presented in Supplementary Table 2 and the regressors included in our models are presented in Supplementary Table 3. All continuous variables were z-scored to increased interpretability of the beta values. All variables were added as fixed effects and additionally valence and outcome were added as random slopes for each subject, because their values change over trials. We used the default brms priors (family = Bernoulli, link = logit) and the models were fit using four chains with 6000 iterations and 1500 warmup iterations. To assess WSLS behaviour, we assessed the effect of the outcome (reward or punishment) on the previous trial on the stay behaviour of the current trial. Note that for accuracy and WSLS-behaviour, the coefficients represent the log odds ratio (OR) [i.e. ln(OR)], hence the effect size (as OR), can be calculated by taking to exponential of the coefficients. Trial-wise reaction times were analysed with a normally distributed standard prior (family = Gaussian; link = identity).

Statistical analyses of fMRI image data

All images were preprocessed with fMRIPrep 20.2.1⁶² (Supplementary material). Per subject, we ran a general linear model (GLM) on the preprocessed blood oxygen level-dependent (BOLD) images, to analyse the neural representation of two key classic RL-model-derived parameters: EV and RPE. This method is well established⁶³ and systematically used in fMRI studies.⁶⁴ Per trial, we updated the EV of the chosen cue as well as the RPE according to:

$$EV(t+1)=EV(t)+\alpha \times \mathrm{RPE}(t)$$ (1)

$$\mathrm{RPE}(t)=\mathrm{Actual}\mathrm{reward}-EV(t)$$ (2)

where t is the trial number and α the learning rate. EV on the first trial was 0.5 for all cues. Note that we calculate a signed RPE.

Eight regressors of interest were added to our first-level GLM. Two onset regressors for cue (i.e. one for gain and one for loss trials) were parametrically modulated by EV. Similarly, two onset regressors (again one for gain and one for loss trials) for outcome were modulated by RPE. To avoid between-subject differences on EV and RPE modulation due to between-subject differences in learning rate, we used a fixed learning rate of 0.2 for all subjects, as recommended by Wilson and Niv.⁶⁵ The regressors of interest were convolved with a haemodynamic response function and regressed against the preprocessed BOLD images. To minimize the effects of noise and motion artifacts, we added a subset of the confound regressors calculated by fMRIPrep to our GLM; standardized DVARS, framewise displacement, eight anatomical CompCor regressors, eight cosine regressors to replace high-pass filtering, six motion regressors, their squares, their derivatives and the squares of the derivatives of the six motion regressors. In addition, we added the AROMA noise regressors but excluded all AROMA components that correlated with the task regressors. Lastly, we added two intercept regressors denoting the two task blocks. To assess the activation patterns at the group level, a second-level random effects analysis was performed. To analyse group difference, we used ANOVA or t-tests. To assess continuous effects, we used the relevant continuous variable as a modulator for the t-test. To account for multiple comparisons, we applied threshold-free cluster enhancement (TFCE),⁶⁶ with 5000 permutations. Unless stated otherwise, statistically significant clusters were TFCE adjusted with family-wise error (FWE) P < 0.05.

Data availability

The data that support the findings of this study are available on request only, to ensure the privacy of the subjects. A data acquisition request can be sent to info@parkinsonopmaat.nl. All analysis code used in the present study is available at the Donders Repository: https://doi.org/10.34973/bt7r-p864.

Results

Patient characteristics

Characteristics of all participants are shown in Table 1. PD patients were subtyped based on ICD scores (Table 1) and on depression scores (Supplementary Table 5). More males than females had clinical ICD [χ²(2,247) = 6.86, P = 0.009], and those with ICDs (PD-ICD+) showed higher depressive symptom scores [main effect of ICD group on BDI: F(1,186) = 18.07, P < 0.001], higher anxiety scores [main effect of ICD group on STAI-trait: F(1,186) = 23.39, P < 0.001], by definition higher ICD scores [main effect of ICD group on QUIP-rs: F(1,186) = 256.15, P < 0.001] and reduced performance on the Symbol Digit modalities test [main effect of ICD group: F(1,186) = 6.75, P = 0.01, suggesting reduced psychomotor speed] compared with the PD-ICD− group. There were no interaction effects between dopaminergic medication and ICD groups.

The patients’ regular dose of dopaminergic medication [levodopa equivalent daily dose (LEDD)] was higher in the PD group ON medication than in the PD group OFF medication [main effect of medication group on LEDD: F(1,185) = 5.26, P = 0.02]. Furthermore, PD-ICD+ patients took higher daily doses of dopaminergic medication than PD-ICD− patients [main effect of ICD on LEDD: F(1,185) = 8.43, P = 0.004] and also were more often users of dopamine receptor agonists [χ²(2,182) = 8.16, P = 0.004]. However, there was no Medication (ON versus OFF) × ICD interaction for LEDD [F(1,185) = 0.33, P = 0.57], indicating that in each ICD subgroup, patients who were scanned ON versus OFF dopaminergic medication took approximately the same daily dose. Group differences in LEDD are unlikely to reflect differences in disease severity: the Movement Disorder Society Unified Parkinson’s disease Rating Scale (UPDRS) was similar between the ON and OFF groups, both in the OFF state [F(1,186) = 2.57, P = 0.11] and ON state [F(1,179) = 1.03, P = 0.31]. Note, we do show a strong within-subject improvement in UPDRS scores ON versus OFF medication across all participants [F(1,189) = 103.4, P < 0.001]. We did not find any further relevant differences in (demographic) background variables (Table 1), except for the time since diagnosis which was longer for the OFF than the ON group [main effect of medication group: F(1,178) = 5.96, P = 0.02]. However, this did not interact with ICD.

Patients ON medication are more sensitive to gains versus losses than healthy controls

Participants engaged successfully with the RL task, as indexed by a monotonic increase over trials in choice of the cue with the highest reward probability [Fig. 1; main effect of trial number on P(cue_correct); brms 95% confidence interval (CI) = 0.03, 0.04]. Adequate task engagement is further supported by a WSLS analysis of stay responses, which confirmed that participants exhibited a tendency to stay with a cue (versus shift away from it) after having received a positive outcome for that choice on the previous trial [main effect of previous outcome on P(stay); brms 95% CI = 0.46, 0.59]. This WSLS tendency was modulated by valence (Previous outcome × Valence; brms 95% CI = 0.01, 0.11), due to participants exhibiting greater tendency to WSLS after losses than after gains.

Based on previous findings,²⁹ we hypothesized that patients ON medication would be more sensitive to gains versus losses than HCs, whereas patients OFF medication would be more sensitive to losses versus gains than HCs. Partly in line with these hypotheses, trial-wise Bayesian mixed effects modeling of stay responses revealed that patients ON medication showed increased WSLS behaviour after gains versus losses, compared with HCs [Group (ON versus HC) × Valence × Previous outcome; brms 95% CI = 0.01, 0.11; Group (ON versus HC) × Previous outcome; brms 95% CI = −0.11, 0.05]. Post hoc analysis revealed that patients ON medication were less likely to switch after a punishment (i.e. the outcome is −€10) compared with HC (for post hoc analyses, see Supplementary material).

There was no evidence for differences between the OFF group versus HC, either in accuracy [Group (OFF versus HC) × Valence; brms 95% CI = −0.15, 0.11; main effect of medication; brms 95% CI = −0.28, 0.03] or WSLS [Group (OFF versus HC) × Valence × Previous outcome; brms 95% CI = −0.03, 0.10; Medication × Previous outcome; brms 95% CI = −0.11, 0.07]. In contrast to our hypothesis, there were no differences between the ON and OFF groups, either in accuracy [Group (ON versus OFF) × Valence; brms 95% CI = −0.05, 0.17; main effect of medication; brms 95% CI = −0.14, 0.14] or WSLS [Group (ON versus OFF) × Valence × Previous outcome; brms 95% CI = −0.07, 0.03; Medication × Previous outcome: brms 95% CI = −0.04, 0.11] (Fig. 2).

Figure 2.

Performance on the reinforcement learning task as a function of medication. (A) Learning across all participants. The orange line presents gain trials and green line presents loss trials. Opaque area represents the standard error of the five-trial moving mean of the accuracy of each subject. (B) Win-stay-lose-shift (WSLS) behaviour per participant, grouped by medication status. WSLS behaviour was calculated as the percentage of stay trials after a reward minus the percentage of stay trials after a loss. Parkinson’s disease (PD) ON patients show more WSLS behaviour during gain versus loss trials relative to healthy controls (HC). (C) Average accuracy per patient grouped by medication status. Accuracy during loss trials was subtracted from accuracy during gain trials (for separate gain and loss figures, see Supplementary Fig. 4).

Effects of dopaminergic medication group on reinforcement learning depend on impulse control disorders

We tested the hypothesis that medication group effects on RL are particularly pronounced in patients with clinical ICD and/or depression. In line with our hypothesis, dopaminergic medication group interacted with the presence of ICD, both in terms of accuracy (ICD × Medication × Valence; brms 95% CI = −0.31, −0.04) and WSLS (ICD × Medication × Valence × Previous outcome: brms 95% CI = −0.15, −0.03). These interactions with ICD were due to greater medication group-related increases in accuracy and WSLS tendency on gain versus loss trials in patients with ICD relative to patients without ICD (Fig. 3). The WSLS effect was driven by an increase of win-stay behaviour (and not lose-shift) in the PD-ICD + ON medication group (WIN trials only; Valence × ICD-class × Medication; brms 95% CI = −0.28, −0.04). For the full post hoc analyses of both the WSLS and accuracy effect, see Supplementary material; for specific gain or loss effects, see Supplementary Table 2 and Fig. 4. The accuracy effect remained when total QUIP-rs scores were used as a continuous factor [ICD (continuous) × Medication × Valence; brms 95% CI = 0.01, 0.269]. Post hoc analyses revealed that this effect was driven by an enhanced sensitivity to gains in patients ON versus OFF medication, but only when they have ICD (for post hoc analyses, see Supplementary material). Analysis of reaction times revealed a similar pattern, where patients ON versus OFF medication responded faster for gain versus loss trials, but this was true only for ICD patients (Fig. 3G and Supplementary material). We found no other interactions or main effects for both continuous ICD and depression scores or depression classification.

Figure 3.

Performance on the reinforcement learning task as a function of medication and impulse control disorder. Learning curves for the four different patient groups: (A) PD-ICD+ ON medication, (B) PD-ICD− ON medication, (E) PD-ICD+ OFF medication and (F) PD-ICD− OFF medication. The orange lines represent gain trials and green lines represent loss trials. Opaque areas represents the standard error of the five-trial moving mean of the accuracy of each subject. In particular, PD-ICD+ patients ON medication show reduced punishment learning. (C) Average accuracy per patient, grouped medication status and ICD class. Accuracy during loss trials was subtracted from accuracy during gain trials (for separate gain and loss figures, see Supplementary Fig. 4). (D) Win-stay-lose-shift (WSLS) behaviour per participant, grouped by medication status and ICD-class. WSLS behaviour was calculated as the percentage of stay trials after a reward minus the percentage of stay trials after a loss. (G) Reaction time difference between gain and loss trials (above zero is faster during loss trials compared to gain trials and vice versa). ICD = impulse control disorder; PD = Parkinson’s disease.

Figure 4.

Performance on the reinforcement learning task as a function of medication and depression. (A) Win-stay-lose-shift (WSLS) behaviour per participant, grouped by medication status and impulse control disorder (ICD) class. WSLS behaviour was calculated as the percentage of stay trials after a reward minus the percentage of stay trials after a loss. (B) Average accuracy per patient, grouped medication status and depression status. Accuracy during loss trials was subtracted from accuracy during gain trials (for separate gain and loss figures, see Supplementary Fig. 4). (C) Pearson correlation between depression [Beck Depression Inventory (BDI) scores] and ICD [Questionnaire for Impulsive-Compulsive Disorders in Parkinson’s disease Rating Scale (QUIP-rs) scores]. DEP = depression.

No evidence that medication group effects depend on depression

Despite a significant correlation between QUIP-rs and BDI scores across individual patients (Fig. 4C), analogous effects of clinical depression did not reach significance (accuracy: brms 95% CI = −0.06, 0.22; WSLS: brms 95% CI = −0.02, 0.11; Fig. 4). To investigate the specificity of the role of ICD in medication group-related effects, we added depression status as a covariate of non-interest to our ICD-class models. Effects of ICD-class on WSLS (ICD-class × Medication × Valence × Previous outcome + DEP-class; brms 95% CI = 0.03 0.15) and accuracy (ICD-class × Medication × Valence + DEP-class; brms 95% CI = 0.04 0.31) remained significant, indicating that this effect is driven specifically by ICD, but not by depression. We did not find any continuous effects of BDI.

Confounding factors

Several clinical variables differed between patients with and without ICD (i.e. gender, depression, anxiety, LEDD, dopamine agonist use and psychomotor speed (Symbols and digits modalities test), and between patients ON and OFF medication (i.e. LEDD and months since diagnosis) (Table 1). To assess whether our effect of interest (i.e. ICD × Medication × Valence) can be accounted for by group differences in any of these variables of no interest, we added them to our models as confound regressors. Our effects on accuracy and WSLS remained significant after correcting for each of these variables (for all CIs, see Supplementary Table 3).

Abnormal expected value, but not reward prediction error coding, in impulse control disorders

Next, we assessed the effects of medication group, valence and ICD on neural signalling during RL. To this end, we used a classic RL model to derive, for each trial, an RPE (at the time of outcome) and an EV (at the time of choice). These variables were then inserted as parametric regressors modulating gain and loss outcome events and gain and loss choice (cue) events respectively in a general linear model.⁶⁷

In line with previous work,²⁰ we observed highly significant RPE-related BOLD signal in a network of regions associated with reward, including the nucleus accumbens, when averaged across all participants and across gain and loss trials [17 112 voxels, Montreal Neurological Institute (MNI) local maximum 12, 7, −9; TFCE = 8709.23; P_fwe < 0.001; Fig. 5A]. There was also highly robust EV-related BOLD signal in a network of regions including the vmPFC (336 voxels, MNI local maximum −2, 49, −5; TFCE = 719.09; P_fwe < 0.001; Fig. 5B). Across groups, we found a main effect of valence (gain > loss) on both RPE (11 949 voxels, MNI local maximum −16, −102, 2; TFCE = 94 489.01, P_fwe < 0.001; Fig. 5C) and EV-related signalling (624 voxels, MNI local maximum −2, 49, −2; TFCE = 822.49; P_fwe < 0.001; Fig. 5D), with stronger RPE and EV coding in the gain than the loss condition.

Figure 5.

BOLD response for reward prediction error and expected value across all participants. Main and valence effects for expected value (EV) and reward prediction error (RPE). The variables are calculated by a simple Rescorla–Wagner model and added as parametric regressor to cue (EV) and outcome (RPE) onsets. The whole-brain images were adjusted for family-wise error (fwe) correction with Threshold Free Cluster Enhancement (TFCE) P_fwe < 0.05. Reported images are across all participants. Beta values extracted from all clusters above, plotted separately for PD-ON medication, PD-OFF medication and healthy controls (HC). (A and D) Main effect of RPE. (B and E) Main effect of EV. (C and F) Valence effect of RPE (gain minus loss trials). (D and G) Valence effect of EV (gain minus loss). MNI = Montreal Neurological Institute; PD = Parkinson’s disease.

When we did not consider the presence of ICD, there were no RPE- or EV-related BOLD signal differences between any of the groups (ON versus OFF, ON versus HC or OFF versus HC; Fig. 5), also not as a function of valence or in any of the ROIs. However, whole-brain analysis revealed a significant medication by valence effect, when stratifying the effects by ICD. Specifically, there was an ICD × Medication × Valence interaction on EV-related BOLD signal in the vmPFC, encompassing the subgenual and anterior cingulate cortex. This effect was substantiated by both whole-brain analyses (331 voxels, MNI local maximum 12, 49, 5; TFCE = 319.06; P_fwe < 0.05; Fig. 6A and for a subthreshold map see Supplementary Fig. 5) as well as ROI analyses (vmPFC, 11 voxels, MNI local maximum 8, 32, −5; TFCE = 12.83; P_fwe < 0.05; ROI based on meta-analysis²⁰; Supplementary Fig. 2) and reflects a medication group-related shift towards greater EV-related signal on gain versus loss trials in patients with ICD (ON versus OFF medication) compared with patients without ICD. This pattern of cerebral effects paralleled that of the behavioural effects, showing that, compared with ICD patients OFF medication, ICD patients ON medication were more accurate, faster and showed increased WSLS behaviour on gain versus loss trials compared with patients ON versus OFF medication without ICD. This observation was substantiated by a significant brain-behaviour correlation at the whole brain [r(200) = 0.28, P ≤ 0.001; Fig. 6] and the ROI level [r(200) = 0.21, P ≤ 0.001], so that those patients who exhibited greater EV-related signals for gain versus loss trials also exhibited greater accuracy on gain versus loss trials.

Figure 6.

BOLD response for reward prediction error and expected value per impulse control disorder group. (A) Medication induced a shift towards greater expected value (EV)-related signal on gain versus loss trials in impulse control disorder (ICD) patients versus non-ICD patients. Here the whole-brain analysis is depicted; see Supplementary material for region of interest (ROI). (B) Beta-values from the frontal cluster in A. (C) Brain behaviour correlation; increased activity in the whole-brain ventral medial prefrontal cortex (vmPFC) cluster correlated with increased accuracy during gain trials compared with loss trials across groups. (D) Striatal clusters of the reward prediction error (RPE) ROI by Chase et al.²⁰ (E) Beta values from the ROI, for the prediction error signal during gain minus loss trials. (F) Brain behaviour correlation; increased activity in the RPE ROI did not correlate with task-accuracy.

In contrast to EV-related BOLD signal, RPE-related BOLD signal did not vary between ICD-groups. No clusters survived familywise error correction, at the whole-brain level, or within the ROI, based on a meta-analysis by Chase et al.²⁰ (or within the smaller ROI defined by Piray et al.⁶⁸; Supplementary Fig. 3). There was also no interaction with valence and no correlations between beta values extracted from the ROI and measures of task performance (Fig. 6F).

Discussion

The present findings demonstrate canonical medication group effects on gain- versus loss-based decision making in PD patients with ICDs, but not those without ICDs. This suggests that patients with ICDs are disproportionately sensitive to dopaminergic medication effects on value-based decision making. Moreover, the results show that medicated patients with ICDs also exhibit increased EV signalling during gain versus loss trials in the vmPFC, rather than changes in RPE signalling in the striatum. Together, these findings establish key clinical, computational and neural factors that contribute to the large between-subject variability in value-based decision making in PD.

The finding that medication group effects on value-based choice are observed in only a subset of PD patients, namely those with ICDs, advances our understanding of both the role of dopamine in reinforcement learning and decision making (RLDM) as well as the mechanisms of ICD in PD. Specifically, this result resolves the discrepancy between, on the one hand, the classic observation that PD medication has asymmetric effects on gain versus loss learning, an effect that has become almost canonical due its cross-lab replication,^{16,17,22,29,30,33-37} and on the other hand, the recent non-replications of these asymmetric effects.^11,47-49 The latter non-replications may reflect in part failure to consider key individual variability in the presence of comorbid psychiatric disorders that implicate, among other things, unstable mesolimbic dopamine transmission, as might be the case in ICDs. The finding also has impact on our understanding of ICD mechanisms by supporting the proposal that medication-related increases in the weight on gains versus losses during choice might contribute to the development and/or expression of ICDs in PD. Our neuroimaging results generally concur with the hypothesis that dopaminergic medication in PD ICD acts on mesolimbic reward circuitry including the vmPFC. Specifically, compared with those OFF medication, ICD patients ON medication have boosted EV-related signalling in the vmPFC at the time of choice, for gain versus loss trials, and this effect correlated with the behavioural effect of medication on gain versus loss trial accuracy.

The medication group effect on the vmPFC in PD patients with ICD might reflect altered ventral striatal input to the vmPFC. This is suggested by evidence from PET and SPECT studies, demonstrating that patients at risk of ICDs exhibit reduced negative feedback control over dopamine release in the ventral striatum,⁶⁹ reduced D2/3 receptor availability in the ventral striatum⁷⁰ and reduced dopamine transporter availability in the ventral striatum, suggesting reduced dopamine clearance from the synaptic cleft.^71-73 Hypersensitivity of the mesolimbic reward system has previously been demonstrated, with greater dopaminergic medication^74-76 and gambling task-related dopamine release^74,76 and blood flow⁷⁷ in the ventral striatum in PD ICD. Deficient autoregulation of dopamine transmission in mesolimbic reward circuity might reflect genetic variation associated with predisposing personality characteristics such as trait impulsivity⁷⁸ and novelty seeking,⁴⁰ or environmental factors like neuroinflammation.^79,80 Thus, the impairment on value-based choice tasks in PD ICD might follow from an unstable, hyperdynamic mesolimbic dopamine system.

The observation of medication group differences in EV signal at the time of choice concurs with prior findings that dopaminergic medication in PD acts by altering the expression of learning on choice rather than learning itself.^32,47,55 The suggestion that learning itself was unaffected is supported by the absence of convincing evidence for modulation of RPE signals at the time of outcome in the striatum or elsewhere. This is in contrast to our primary prediction, as well as to an extensive prior literature on RL model-based neuroimaging both in healthy volunteers,^18,81-83 PD patients without ICDs^15,16,50 and those with ICDs.¹⁰ Given that PD medication might affect a tonic rather than only a phasic mode of dopamine transmission, these findings generally support a growing literature on a key role of tonic dopamine in the impact of action value on choice and motivation^56,84-88 (but see Mikhael et al.⁸⁹). In this context, it is at least intriguing to note that ICDs are seen more frequently in PD patients using dopamine receptor agonists,⁹ which simulate action of tonic dopamine, than in patients using levodopa, which also promotes the phasic release of dopamine.^90-92 Clearly, hypotheses regarding the effects of agonist use require future work, in which patients with agonists are compared directly with patients on levodopa.

While the lack of an effect of ICD on striatal RPE signalling might reflect selective abnormality in tonic dopamine transmission, we remain particularly puzzled about the lack of an effect of PD OFF versus controls on striatal RPE signalling (and on RL performance). This contrasts with prior PD studies revealing abnormal striatal RPE signalling in PD and is hard to reconcile with the observation that PD is characterized by severe degeneration of striatal dopamine cells, which must be associated with reduced phasic dopamine release. There may be several reasons for this absence of abnormal striatal RPE signalling.

First, it might be that our PD patients were in relatively early stages of the disease (average disease duration = 2.5 years), where cells projecting to the relevant ventral striatum have not yet degenerated.⁹³ Thus, while the ventral striatum of PD patients with ICDs might exhibit subtle (pre-existing) autoregulatory (presynaptic D2/D3 receptor-related) problems and aberrant tonic dopamine levels, it is less likely that the cells themselves have already degenerated completely, leaving phasic dopamine transmission relatively intact (or even upregulated⁹⁴). Thus, the ventral striatum of PD patients without ICDs might exhibit both intact phasic RPE signalling, as well as intact autoregulatory mechanisms, rendering value coding also insensitive to dopaminergic medication.

Second, the absence of abnormal striatal RPE signalling might be explained by a disproportionate reliance on a working memory strategy commonly associated with the PFC rather than the incremental RPE-based learning strategy that is associated with the striatum. Our task comprised only two gain and two loss cues, thus requiring the working memory of only four cue values, which is well within most people’s working memory capacity. Indeed, it is established that performance in especially the initial learning stage of RL tasks depends on cue set-size.^95-99 Thus, it is possible that PD patients relied on working memory rather than an RL strategy when completing the current task. The hypothesis that the group effects observed here might reflect dopaminergic modulation of higher-order cognitive functioning like working memory is consistent with the locus of the medication group effect in the prefrontal cortex. Indeed, dopamine (receptor stimulation^100,101) in the prefrontal cortex has long been implicated in a wide variety of cognitive control functions, including not just working memory,^102-104 but also other higher-order cognitive functions that might be argued to contribute to performance on the current task, including set-shifting,^105,106 delayed reward discounting,^107,108 temporal control,^109-112 and effort-based decision making.^113,114

Third, a 12-h washout period for dopaminergic medication might not be sufficient. Hence, it is possible that persistent medication effects masked aberrant RPE signalling in the ventral striatum (Supplementary material).

A medication group difference in the vmPFC in PD ICD during value-based choice is reminiscent of findings from previous neuroimaging studies in PD ICD,¹¹⁵ showing aberrant prefrontal signalling during the evaluation of future reward and punishment,^116,117 during risk taking in gambling tasks¹¹⁸ and during speeded decision making in a Stroop task.¹¹⁹ More specifically, the present observation is remarkably consistent with prior work by Voon et al.,¹⁰ who demonstrated a shift towards gain learning away from loss learning in 14 patients with impulsive shopping and/or gambling. Voon et al.,¹⁰ also found PD patients with ICD to exhibit increased EV signalling in the vmPFC, as in our study. However, in contrast to our study, this was not modulated by medication. Instead, Voon et al.¹⁰ reported a medication-related increase in striatal RPE signalling during gain trials (Supplementary material). Similarly, Piray et al.⁴³ reported that increased probabilistic gain versus loss learning in PD ICD was best accounted for by a model that assumed abnormal RPE-based learning of values. How can we account for this apparent discrepancy with the current study showing no RPE signal changes? One factor that might play a role in the discrepancy between these studies on PD ICD is disease duration. Average disease duration of the PD ICD group in the Piray et al.⁴³ study was 9.6 years, whereas the average disease duration of our very early-PD cohort was only 2.5 years. It is thus possible that the presence of abnormal striatal RPE signalling in their studies, but not in the current study, reflects increased degeneration of neurons projecting to the ventral striatum due to longer disease duration in the Piray et al.⁴³ study, also leading to deficient capacity of these neurons to exhibit phasic bursting. In sum, dopaminergic medication-related deficits in ventral striatal RPE signalling might be most readily seen in clinically more advanced PD patients with ICDs, who exhibit not only reduced phasic firing capacity of cells projecting to the ventral striatum, but also reduced ability to rely on a prefrontal working memory strategy. In contrast, ICDs in the more mildly affected PD patients studied here are accompanied by biased gating of action value representations in vmPFC, leading to abnormal expression of learnt values on decision making, an anomaly compounded by their tendency to rely on a working memory strategy.

One might wonder whether the medication group-related increase in gain- versus loss-based choice in PD patients with ICDs versus non-ICDs reflects other factors of no interest that also differed incidentally between the patient groups. Supplementary analyses revealed that the effects of ICDs survived correction for a variety of potentially confounding regressors (Supplementary material). Nevertheless, this study did not involve a within-participant manipulation of medication. Such within-subject designs control for potential confounding factors, but also come with challenges, such as relevant learning effects on cognitive tasks that may prevent a reliable comparison between sessions.⁴² We also did not randomize individuals between OFF or ON medication testing, although the assignment of individuals to either of these sessions was determined by enrollment order, which we assume is a random process. Taken together, while the current data strongly suggest an association between the presence of ICDs and medication-related shifts towards gain- versus loss-based choice in early PD, future placebo-controlled cross-over medication withdrawal studies are required to firmly establish a causal link between the consequences of dopaminergic medication for RL and the presence of ICDs.

In contrast to our prediction that RL and decision making vary as a function of depression, our study did not reveal evidence for an interaction between depression and medication. However, we consider the evidence for no effect of depression weak and refrain from concluding that medication effects on RL do not depend on depression, particularly given that the numerical pattern of effects of depression resembles that of ICD (Supplementary material).

These data substantiate the hypothesis that the effect of dopamine on RL in PD varies with individual differences in comorbid ICDs. Specifically, our findings suggest that the presence of ICD is associated with deficient computation of value in medial frontal cortex, rather than deficient RPE signalling in striatum.

Funding

This study was supported by the Michael J. Fox Foundation for Parkinson's Research (grant ID #15581), Verily Life Sciences and Health ∼ Holland. The Centre of Expertise for Parkinson & Movement Disorders was supported by a centre of excellence grant of the Parkinson’s Foundation. J.T. was supported by internal funds from the Radboudumc. R.C. was supported by an Ammodo award from the Royal Netherlands Academy of Arts and Sciences and a Vici award from the Dutch Research Council (Grant No. 453-14-015). R.H. was supported by a VIDI grant from the Dutch Research Council (Grant No. 09150172010044).

Competing interests

The authors report no competing interests.

Supplementary material

Supplementary material is available at Brain online.