Effects of dopamine on reinforcement learning in Parkinson’s disease depend on motor phenotype

See Manohar (doi:10.1093/brain/awaa363) for a scientific commentary on this article.

Parkinson’s disease is complicated by great variability in symptoms. Van Nuland et al. report that the well-established effects of levodopa on reinforcement learning apply only to patients without tremor symptoms, suggesting a need to revisit neurocognitive models of dopaminergic function in Parkinson’s disease.

Abstract

Parkinson’s disease is clinically defined by bradykinesia, along with rigidity and tremor. However, the severity of these motor signs is greatly variable between individuals, particularly the presence or absence of tremor. This variability in tremor relates to variation in cognitive/motivational impairment, as well as the spatial distribution of neurodegeneration in the midbrain and dopamine depletion in the striatum. Here we ask whether interindividual heterogeneity in tremor symptoms could account for the puzzlingly large variability in the effects of dopaminergic medication on reinforcement learning, a fundamental cognitive function known to rely on dopamine. Given that tremor-dominant and non-tremor Parkinson’s disease patients have different dopaminergic phenotypes, we hypothesized that effects of dopaminergic medication on reinforcement learning differ between tremor-dominant and non-tremor patients. Forty-three tremor-dominant and 20 non-tremor patients with Parkinson’s disease were recruited to be tested both OFF and ON dopaminergic medication (200/50 mg levodopa-benserazide), while 22 age-matched control subjects were recruited to be tested twice OFF medication. Participants performed a reinforcement learning task designed to dissociate effects on learning rate from effects on motivational choice (i.e. the tendency to ‘Go/NoGo’ in the face of reward/threat of punishment). In non-tremor patients, dopaminergic medication improved reward-based choice, replicating previous studies. In contrast, in tremor-dominant patients, dopaminergic medication improved learning from punishment. Formal modelling showed divergent computational effects of dopaminergic medication as a function of Parkinson’s disease motor phenotype, with a modulation of motivational choice bias and learning rate in non-tremor and tremor patients, respectively. This finding establishes a novel cognitive/motivational difference between tremor and non-tremor Parkinson’s disease patients, and highlights the importance of considering motor phenotype in future work.

See Manohar (doi:10.1093/brain/awaa363) for a scientific commentary on this article.

Introduction

Parkinson’s disease is clinically defined by bradykinesia, rigidity, and tremor. However, the severity of motor symptoms differs considerably between patients. Arguably, the main clinical subdivision is between patients with a tremor-dominant phenotype and those with a non-tremor phenotype (Helmich et al., 2012; Marras and Lang, 2013). Compared with tremor-dominant patients with Parkinson’s disease, non-tremor patients suffer more from gait and balance problems, as well as more severe cognitive decline and earlier dementia (Williams-Gray et al., 2007, 2009; Wu et al., 2011), increased motivational dysfunction indicative of impaired impulse control (Wylie et al., 2012), and increased levels of anxiety (Dissanayaka et al., 2010). These clinical differences are mirrored by a variety of neurochemical alterations, predominantly in the dopaminergic system. Specifically, dopamine cell loss has been demonstrated, in both post-mortem and nuclear imaging studies, to be more severe in non-tremor patients than in tremor-dominant patients (Spiegel et al., 2007; Rossi et al., 2010; Helmich et al., 2011). Furthermore, non-tremor patients have more extensive substantia nigra degeneration (Paulus and Jellinger, 1991; Jellinger, 2012), which contains the vast majority of mesencephalic dopaminergic neurons (76% in non-human primates) (François et al., 1999). In contrast, tremor-dominant patients have more extensive retro-rubral area degeneration (Hirsch et al., 1992), which in non-human primates, contains only 10% of mesencephalic dopaminergic neurons (and the remaining 14% in the ventral tegmental area) (François et al., 1999).

Brain dopamine has long been implicated not only in motor behaviour but also in a wide range of cognitive functions. Most pervasive is the role of dopamine in reinforcement learning. Yet, given the well-established importance of dopamine in the neural implementation of reinforcement learning (Schultz et al., 1997; Holroyd and Coles, 2002; Fiorillo et al., 2003; Steinberg et al., 2013), there is a puzzlingly large variability in the effects of dopaminergic medication on reinforcement learning in Parkinson's disease (Frank et al., 2004; Cools et al., 2006; Grogan et al., 2017; Timmer et al., 2017; McCoy et al., 2019). Here we exploit clinically relevant variance in Parkinson’s disease phenotypes, namely the presence or absence of tremor, to characterize this large variability in dopaminergic drug effects on reinforcement learning.

Multiple controlled medication withdrawal studies in Parkinson’s disease have demonstrated that dopaminergic medication enhances learning from reward, while impairing learning from punishment (Frank et al., 2004, , 2007; Cools et al., 2006; Moustafa et al., 2008; Bodi et al., 2009; Palminteri et al., 2009; McCoy et al., 2019). This medication-related shift away from punishment towards reward learning is grounded in neural network modelling work (Frank, 2005). According to this work, medication potentiates reward prediction error-related phasic dopamine bursts, while blocking punishment prediction error-related dopamine dips. These effects are compelling, both theoretically and empirically, but several recent studies have failed to replicate them (Grogan et al., 2017; Timmer et al., 2017). While at first puzzling, this variability in the effects of dopaminergic medication is perhaps not so surprising; it concurs with extensive evidence from pharmacological work demonstrating great variability in dopaminergic drug effects, for example as a function of variation in baseline dopamine levels (Cools and D’Esposito, 2011). Based on this literature, and the different dopaminergic phenotypes of tremor-dominant and non-tremor Parkinson’s disease patients, we hypothesized that the variability in dopaminergic medication effects on reinforcement learning reflects differential effects depending on motor phenotype, with non-tremor patients exhibiting greater medication-related increases in reward versus punishment learning than tremor patients.

A second key open issue about dopamine’s effects on reinforcement learning is the degree to which these effects reflect modulation of learning or, rather, of motivational choice bias (Berridge, 2007). According to the learning hypothesis, dopamine prediction errors drive reward and punishment learning through selective modulation (long term potentiation and depression) of direct ‘Go’ and indirect ‘NoGo’ pathway activity (Frank, 2005). According to the alternative motivational choice biasing hypothesis, dopamine alters only the expression of learning on choice, invigorating action in the face of reward (Berridge, 2007; Robbins and Everitt, 2007) and suppressing action in the face of punishment (Guitart-Masip et al., 2014; Lloyd and Dayan, 2016). Disentangling these hypotheses has been difficult because correctly learned performance on most learning tasks requires responses that are congruent with motivational biases (Go-for-reward or NoGo-to-avoid-punishment) (Frank et al., 2004; Cools et al., 2006; Bodi et al., 2009; McCoy et al., 2019). Thus, some effects that have been attributed to modulation of learning might in fact reflect biasing of motivational choice (Guitart-Masip et al., 2012a; Boer et al., 2019). In keeping with this hypothesis, evidence from recent studies with patients with Parkinson’s disease have revealed that effects of dopaminergic medication on reinforcement learning tasks can be attributed, at least in part, to modulation of choice (Shiner et al., 2012; Smittenaar et al., 2012). However, those studies do not exclude that medication alters both learning and choice, as these could not be assessed simultaneously. Here we address this issue by combining computational reinforcement learning modelling with the use of a reinforcement learning task where Go/NoGo response requirements and motivational valence were manipulated independently (modified from Guitart-Masip et al., 2011). The task capitalizes on the fact that rewards and punishments elicit differential action biases of activation and inhibition of behaviour, respectively. Valence effects on learning would affect learning across action domains, leading to changes in accuracy as a function of valence, but not of the Go/NoGo response requirement. Conversely, effects on motivational choice bias would drive choice accuracy in opposite directions depending on whether the required action was congruent (e.g. Go-for-reward) or incongruent (e.g. NoGo-for-reward) with the valence of the cue. We used this design to compare the effect of dopaminergic medication on reinforcement learning between two carefully selected Parkinson’s disease subtypes, i.e. tremor-dominant and non-tremor patients, as well as healthy control subjects.

Materials and methods

Population and study design

We tested 63 patients with Parkinson’s disease, in addition to 22 healthy controls. This work is part of an overarching study, registered at https://www.trialregister.nl/trial/4940 and previously described in Dirkx et al. (2019). Participants were paid €30 for participation. The study was approved by the local ethics committee (Commissie Mensgebonden Onderzoek MO Arnhem Nijmegen, CMO 2014/014) and written informed consent was obtained prior to inclusion from all participants, according to the Declaration of Helsinki. The overarching study aimed to investigate (i) the differences between tremor and non-tremor Parkinson’s disease, and, within tremor patients; (ii) the differences between patients whose tremor symptoms are responsive to dopaminergic medication, and those that are not. The total study therefore consisted of three Parkinson’s disease subgroups (tremor dopamine responsive, tremor dopamine resistant and non-tremor), with an aim of 20 participants in each group. Sample size was based on previous studies assessing tremor-related activity. It is important to note that the dopamine responsiveness criterion was tremor-specific; all patients were responsive to dopamine with respect to their other symptoms. One tremor-responsive patient dropped out on Day 2 due to claustrophobia. Because of technical errors, behavioural data were not available for one tremor non-responsive patient on Day 1, and one tremor-responsive patient on Day 2. In the non-tremor group, one patient dropped out on Day 2 due to claustrophobia, and one patient did not complete the behavioural task. See Supplementary Table 1 for a summary of the data available.

For patients, inclusion criteria were idiopathic Parkinson’s disease, and match into either clinical phenotype (see below). Exclusion criteria were cognitive dysfunction defined as a Mini-Mental State Examination (MMSE) score <26, a frontal assessment battery (FAB) score <13 (Lima et al., 2008), severe dyskinesias, neurological or psychiatric comorbidity, severe head tremor, known allergy against levodopa-benserazide or domperidone. Complete clinical and demographic information is presented in Table 1. More details regarding the inclusion procedure can be found in Dirkx et al. (2019).

Table 1.

Estimated parameters for all models

	M1	M2	M3	M4	M5a
ρ	2.5 [1.2 6.5]	2.2 [0.7 12.0]	2.1 [0.9 7.5]	3.6 [2.5 6.3]	3.9 [1.3 13.5]
ε0	0.04 [0.01 0.29]	0.03 [0.02 0.06]	0.06 [0.03 0.23]	–	–
b	–	0.37 [0.13 0.59]	0.45 [0.13 0.73]	0.22 [0.05 0.46]	0.61 [0.35 0.85]
π	–	–	1.7 [0.99 2.35]	0.96 [0.66 1.55]	2.04 [1.07 2.65]
ε+pe	–	–	–	0.22 [0.11 0.63]	–
ε−pe	–	–	–	0.02 [0.01 0.04]	–
εwin	–	–	–	–	0.05 [0.04 0.09]
εavoid	–	–	–	–	0.02 [0.01 0.04]

Median and range [25–75 percentile] of subject-level parameter estimates.

^aWinning model.

Tremor-dominant Parkinson’s disease was defined as a history of tremor and a resting tremor score of ≥1 point in at least one arm on item 17 of the Movement Disorders Society Unified Parkinson’s disease Rating Scale (MDS-UPDRS part III). The tremor-dominant phenotype was established using polymyography by a trained neurologist on average 6 months prior to participation (range 46–394 days). Non-tremor Parkinson’s disease was defined as the absence of resting tremor in all limbs (UPDRS resting tremor score of 0). These definitions were used in our previous study (Helmich et al., 2011). Action tremor was not an exclusion criterion for this group, given that action tremor has a different pathophysiology (Dirkx et al., 2018).

Procedure

Both patients and healthy participants were measured on two separate occasions, always in the morning. In both sessions, patients arrived in an OFF state, defined as abstinence from medication for >3 times the drug half-life, i.e. >12 h after their last dose of levodopa, >48–72 h after their last dose of dopamine-agonist. Healthy participants were also measured on two separate sessions to test for task repetition effects. Parkinson’s disease patients were measured in pseudorandomized order with respect to the dopaminergic intervention (see below). All measurements took place over a 3-year period from 2014 to 2017, and for each participant, the two sessions took place within a period of 3 months.

Testing procedure

For patients only, each day started with a measurement of their motor symptoms (using the UPDRS motor scale), followed by administration of medication (or placebo). Next all participants underwent a combination of functional MRI and anatomical scans that lasted ∼2 h (Dirkx et al., 2019;van Nuland et al., 2020, submitted for publication). After a short break, participants performed the behavioural task (outside the scanner). Finally, motor symptoms were measured again, to obtain a measure of symptom severity ‘ON’ medication (or placebo). Cognitive assessment (FAB/MMSE) took place on Day 2 so as to best match overall timing on the first and second day, taking place either in-between UPDRS/MRI and behavioural sessions or after the behavioural session.

Medication regime

In the patient sample, we used a randomized, within-subject, double-blind, cross-over design. During one session, patients received a standardized dose of 200/50 mg dispersible levodopa-benserazide (ON state), dispersed in water. Levodopa dose was on average 70% higher than the patients’ own morning dose [seeTable 1 for average levodopa equivalent daily dose (LEDD) per patient group]. During the other session, patients received a placebo (cellulose dispersed in water, which matches the dispersible levodopa both visually and in terms of taste). In both sessions, all patients received a dose of domperidone (10 mg) 1 h before drug/placebo intake, to increase gastrointestinal absorption and to reduce side effects. Healthy controls did not receive medication in either session.

Motivational Go/NoGo task

Participants performed a reinforcement learning task with four different task conditions to disentangle the separate but interacting axes of motor response requirement (Go/NoGo) and motivational valence (Win/Avoid) (Fig. 1A). Each trial started with the presentation of a cue (3 s). During cue presentation, participants could decide to press a button (Go response) or abstain from responding (NoGo response). One hundred milliseconds after cue offset, participants received feedback based on their response. Valence of the cue was signalled by a coloured edge. Cues with a green edge (Win) were followed by reward (100 points) or neutral feedback (0 points). Cues with a red edge (Avoid) were followed by neutral feedback (0 points) or punishment (−100 points). Subjects were instructed to try to maximize the number of points won while minimizing the total points lost. For each cue, there was one correct response (Go or NoGo), which participants had to learn by trial and error. Feedback was probabilistic, that is, correct responses were followed by the desirable outcome 80% of the time, and incorrect responses 20% of the time. Following from the 2 × 2 factorial design (Valence × RequiredAction), there were four cues in total. Cue presentation order was pseudorandom, with a maximum cue repetition of two cues sequentially. Each cue was presented 45 times. The task lasted ∼30 min, including instructions and two self-paced breaks split evenly between trials.

Figure 1.

Motivational Go/NoGo learning task and performance. (A) Each trial starts with either a Win or an Avoid cue; signalled by the green or red edge of the cue. For each cue, the participant needs to learn the correct response, either press the spacebar (Go) or not (NoGo). Participants have to respond while the cue is on the screen. Outcomes are presented 100 ms after cue offset. In total, four cues are presented, reflecting the 2 × 2 factorial design of response requirement (Go/NoGo) and cue valence (Win/Avoid), such that for each valence there is one cue where Go is correct, and one cue where NoGo is correct. Feedback is probabilistic: correct responses are followed by reward (Win cues) or a neutral outcome (Avoid cues) in 80% of the time, and by a neutral outcome (Win cues) or punishment (Avoid cues) otherwise. For incorrect responses, these probabilities are reversed. (B) Average accuracy per cue type—performance of the cues congruent with the automatic motivational bias (Go2Win, NoGo2Avoid)—is higher than for the incongruent trials (NoGo2Win, Go2Avoid). (C and D) Trial-by-trial proportion of Go responses [± standard error of the mean (SEM)], displayed using a within-subject five-trial average sliding window, for both Parkinson’s disease patients (C) and healthy controls (D). From the first trial onwards, a clear motivational bias is apparent as participants start by making more Go responses for Win cues, and more NoGo responses for Avoid cues. However, during the course of the experiment both participant groups learn to adjust responses towards the correct contingencies. HC = healthy controls; PD = Parkinson’s disease.

Prior to the task, instructions were presented on screen, in which participants were informed about the colour coding of cue valence, probabilistic nature of the feedback, and that each cue had an optimal response. At the end of the task the total number of points won or lost was displayed on the screen. All cues were uniquely shaped and coloured, with colours that were well distinguishable from the edge colours. On each testing day, a unique stimulus set was used, the order of which was counterbalanced across participants and drug conditions.

Statistical analysis

Basic task analysis

The basic analysis design comprised a 2 × 2 repeated-measures ANOVA with proportion correct (accuracy) as the dependent variable and factors RequiredAction (Go/NoGo) and Valence (Win/Avoid). Here, the Valence contrast captures the ability to learn from reward relative to punishment, while the RequiredAction contrast indexes whether individuals are better at learning to make a Go rather than NoGo response. The Valence × RequiredAction interaction quantifies the degree to which motivation and action are coupled, i.e. a motivational action bias, with increased performance on congruent cues (Go-to-Win, NoGo-to-Avoid) relative to decreased performance for incongruent cues (Go-to-Avoid, NoGo-to-Win) (Guitart-Masip et al., 2011). This basic 2 × 2 ANOVA was extended to assess the medication and patient group effects, as our main hypothesis. Furthermore, we verified the robustness of the findings in a set of control analyses, considering test-retest effects and nuisance variables. We describe these below.

Test-retest differences

Learning-dependent behavioural tasks are inherently vulnerable to test-retest differences as performance often increases at second task exposure. We therefore checked whether there were consistent test-retest differences affecting our main factors of interest. We performed a 2 × 2 × 2 ANOVA (Valence × RequiredAction × Testing day) on accuracy, with participant status (control/patient) as a between-participant factor. For patients, we collapsed over medication status. There was a significant interaction of Valence × Testing day, such that people learnt better from Win cues than Avoid cues on Day 1, but vice versa on Day 2 (for details see the ‘Results’ section). Importantly this effect was present in and not significantly different between control subjects and patients (where medication was a potential confound). Given this test-retest difference, we limited our analyses of medication effects to Day 1. This effectively reduced the design to a between-subject medication study. Exclusion of Day 2 data resulted in a final sample of n = 42 tremor patients (23/19 ON/OFF medication), n = 20 non-tremor patients (10/10 ON/OFF medication), and n = 22 healthy controls. For completeness, we report the same main analyses for Day 2 in the Supplementary material.

Medication and patient group effects

Our two main questions centred on (i) effects of dopaminergic medication on valence learning in the context of different required actions; and (ii) potential differences of these dopaminergic effects between tremor and non-tremor patients. Given test-retest differences described above, we restricted this analysis to Day 1. We extended the basic (2 × 2) repeated-measures ANOVA with Accuracy as dependent variable and RequiredAction and Valence as within-participant factors, with medication status (ON/OFF) and patient group [TremorResponsive (TR)/TremorNonResponsive (TNR)/NonTremor (NT)] as between-participant factors. All tests were two-tailed, and using an α-level of significance of P < 0.05. We followed up any patient group effects with a comparison to healthy controls to assess whether medication ‘normalizes’ altered behaviour, or disturbs normal behaviour. For each patient group (tremor/non-tremor) we performed two separate t-tests comparing behaviour ON and OFF medication to healthy controls.

Control analyses

Prior to statistical analyses, several data quality checks were performed. Any responses with a response time <200 ms were removed. This led to exclusion of on average 0.26% of trials in patients [range (0, 4.4)%], and on average on 0.13% of trials in controls [range (0, 1.1)%]. Next, we established that task performance was above chance/improved over time in both congruent and incongruent conditions. For incongruent cues, we tested whether performance during the last block was greater than during the first block (paired samples t-test), indicating learning. As we expected above-chance performance from Block 1 for the congruent cues, we tested for performance chance in Block 1 (one sample t-test against a test value of 0.5).

To quantify relevant clinical or demographic differences between groups (controls versus patients, and tremor versus non-tremor patients) we used a series of two-tailed t-tests (Table 1). When a difference between groups was detected, we followed this up with extra control analyses dedicated to this particular variable (Supplementary material and ‘Results’ section).

Computational modelling

As described above, changes in performance in the motivational Go-NoGo task can result from both altered motivational learning (from reward versus punishment), but also from changes in motivational choice bias, i.e. an increased (or decreased) tendency to invigorate responding in the context of a Win cue, and inhibit responding in the context of an Avoid cue. Increased (or decreased) performance in our task could in principle arise from both mechanisms (Frank et al., 2004; Guitart-Masip et al., 2012b; Swart et al., 2017, 2018; McCoy et al., 2019). To assess which of these mechanisms gave rise to the observed effects of dopaminergic medication, we fitted computational models that allowed us to independently quantify these processes. We fitted six models of increasing complexity to the choice behaviour (Go/Nogo) using a hierarchical Bayesian estimation procedure implemented in RStan. We will describe these models briefly below; full equations and fitting procedure are described in detail in the Supplementary material.

M1 was a simple Rescorla-Wagner model (Rescorla and Wagner, 1972), in which action weights are fully determined by the learned action values (Q-values). These action values are learned through a standard delta-rule learning with two free parameters: a learning rate (ε) which scales the impact of a prediction error on current Q-values, and feedback sensitivity (ρ) scaling the outcome value. In M2 we add a go bias parameter (b) to allow for a differential ‘base rate’ of Go responding—this captures potential differences in the tendency to make a ‘Go’ response, independent of cue valence. In the next models, we implemented various mechanisms through which motivational valence could affect choice. In M3 a motivational bias parameter (π) was added that modulates Go responding according to cue valence; a positive value of π reflects an increased tendency to make more Go responses for Win cues, and more NoGo responses on Avoid cues. Thus, this parameter captures valence effects on choice bias. Models M4 and M5 explore whether motivational valence affects learning rate, to test for previously observed effects of dopaminergic medication on reward versus punishment learning (Frank et al., 2004; Cools et al., 2006). Model M4 include two learning rates that depend on the sign of the prediction error; any outcome that is better than expected (i.e. a neutral outcome in an Avoid trial, or a win in a Win trial) is learnt with a positive learning rate ε_win , while impact of outcomes that are worse than expected will be governed by ε_loss . In model M5, learning is shaped by two learning rates based on cue valence, so that patients may learn differently from outcomes in Win trials relative to Avoid trials, through learning rates ε_win and ε_avoid, respectively. Note that in Win as well as Avoid trials, both positive and negative prediction errors will occur, thus orthogonalizing models M4 and M5.

After fitting these five models, we first assessed which model best described the data, based on comparison of the Watanabe-Akaike Information Criteria (WAIC), an estimate of model evidence, i.e. the model fit corrected for model complexity (Supplementary material). Then, to assess which parameters may capture the observed differential effects of levodopa administration on valence processing as a function of motor phenotype, we compared the parameters of the winning model between patients ON and OFF levodopa, within each motor phenotype (Tremor versus Non-Tremor). To this end we first assessed whether the parameters were normally distributed using a Kolmogorov-Smirnov test. For parameter distributions that differed significantly from normal, we used a non-parametric Mann-Whitney U-test for independent samples, and for the remaining parameters we used simple parametric two-sample t-tests. For further model fitting and comparison procedure details and assessment of independence of parameters, see Supplementary material.

Data availability

All derived and anonymized individual data are available at the Donders Repository https://doi.org/10.34973/cp7s-qy12.

Results

Participants show learning and motivational choice biases

Across participants, subjects exhibited a motivational bias, meaning they were more likely to make a Go response to Win cues and NoGo response to Avoid cues. This was reflected by better performance for bias-congruent Go2Win and NoGo2Avoid cues relative to bias-incongruent NoGo2Win and Go2Avoid cues [Action × Valence F(1,76) = 171.7, η² = 0.69, P < 0.001] (Fig. 1C–E). Participants further exhibited an overall bias towards making Go responses [RequiredAction: F(1,76) = 106.3, η² = 0.58, P < 0.001], but no overall differential performance for Win versus Avoid cues [Valence: F(1,76) = 0.6, η² = 0.01, P = 0.5].

Furthermore, participants successfully learned the task (Fig. 1C and D), indexed particularly by performance changes across blocks in incongruent conditions, where motivational bias and instrumental learning oppose each other. Accuracy in block 3 was significantly higher than in block 1, for both Go2Avoid [Δ(pCorrect): mean (standard deviation, SD) = 0.13 (0.27); t(61) = 3.9, P < 0.001] and NoGo2Win [Δ(pCorrect): mean (SD) = 0.15 (0.19); t(61) = 3.5, P < 0.001]. For congruent cues, accuracy was above chance from the start of the experiment [Block 1: Go2Win: mean (SD) = 0.85 (0.19); accuracy > 0.5: t(61) = 14.8, P < 0.001; NoGo2Avoid: mean (SD) = 0.70 (0.21); accuracy > 0.5: t(61) = 7.7, P < 0.001]. Finally, there was an unexpected test-retest effect in task performance across patients and controls (Supplementary material), which led us to restrict analysis of medication effects to Day 1 only.

Levodopa affects performance as a function of valence and Parkinson’s disease motor phenotype

We did not replicate previous reports that levodopa medication improved performance in the Win versus Avoid trials [F(1,56) = 0.06, η² = 0.001, P = 0.8]. Instead, we found that the interaction of medication and valence was modulated by patient group [Tremor group × Medication × Valence: F(1,56) = 7.3, η² =0.20, P = 0.002]. Pair-wise post hoc comparison of the groups showed that the tremor patient groups did not significantly differ from each other [Tremor group (TR versus TNR) × Medication × Valence: F(1,38) = 0.007, η² < 0.001, P = 0.9], while there was a main effect of medication on valence [Medication × Valence: F(1,38) = 7.1, η² = 0.16, P = 0.01]. In contrast, the non-tremor group was significantly different from each of the tremor groups [Tremor group (NT versus TR) × Medication × Valence: F(1,39) = 9.3, η² < 0.19, P = 0.004; Tremor group (NT versus TNR) × Medication × Valence: F(1,35) = 11.0, η² < 0.24, P = 0.002], while for neither comparison was there a significant effect of medication alone (Medication × Valence, η² < 0.03, P > 0.3). This result provided very strong evidence that responsiveness of tremor to dopaminergic medication was not a relevant factor. Therefore, in further analyses we collapsed across the tremor patient groups.

When assessing the direction of effects within the tremor and non-tremor groups, both groups showed a significant modulation by levodopa of performance in Win versus Avoid cues [Medication × Valence: Tremor: F(1,40) = 7.4, η² = 0.2, P = 0.01; Non-Tremor: F(1,18) = 6.0, η² = 0.2, P = 0.026], but in opposite directions. The non-tremor group replicated the previous literature: patients ON levodopa exhibited higher accuracy in Win versus Avoid trials than those OFF levodopa, with a simple main effect of valence in the ON group [F(1,9) = 8.9, η² = 0.3, P = 0.008], but not the OFF group [F(1,9) = 0.34, η² = 0.02, P = 0.6]. In contrast, in the tremor group, those ON levodopa exhibited higher accuracy in the Avoid versus the Win trials relative to those OFF levodopa, with no simple main effect of valence ON levodopa [F(1,22) = 0.6, η² = 0.02, P = 0.5], but, surprisingly, lower accuracy in Avoid than Win trials OFF levodopa [F(1,18) = 10.2, η² =0.36, P = 0.005].

When comparing each patient group ON and OFF levodopa with healthy control subjects (Fig. 2), there were no significant differences. There were trend level differences between the behaviour of healthy controls and tremor-dominant patients ON medication [HC versus TD-ON, F(1,43) = 3.7, η² = 0.08, P = 0.06], and healthy controls and non-tremor patients OFF medication [HC versus NT-OFF, F(1,30) = 2.9, η² = 0.09, P = 0.10]. The other comparisons were not significant [HC versus TD-OFF, F(1,39) = 0.4, η² = 0.01, P = 0.5] and [HC versus NT-ON, F(1,30) = 2.5, η² = 0.08, P = 0.13]. For healthy controls, there was a trend to perform better in Win than Avoid trials [F(1,21) = 3.6, η² = 0.15, P = 0.07]. Thus, qualitatively non-tremor patients performed more like controls when they were ON medication, while tremor patients performed more like controls when they were OFF levodopa. However, note that we cannot draw strong conclusions from this comparison given its relatively low power, particularly for the non-tremor group.

Figure 2.

Effects of levodopa on performance accuracy. All panels display the average probability of choosing the correct action; data are from testing Day 1. (A) Accuracy as a function of all experimental conditions: Valence, RequiredAction, Medication status and Group. Note that healthy controls did not receive medication. In all groups and conditions there is a significant Valence × RequiredAction interaction. Error bars represent SEM; dots represent individuals. (B) Accuracy (±SEM) for Win versus Avoid cues, collapsed across Required Action, to illustrate the Valence effects of medication. For both Parkinson’s disease motor phenotypes there is a significant Medication × Valence interaction, but in opposite directions. Patients with tremor performed worse for Avoid relative to Win cues when OFF relative to ON levodopa. In contrast, non-tremor patients performed better for Win cues relative to Avoid cues ON relative to OFF levodopa. To enable visual comparison with healthy controls, the average performance of the healthy controls was plotted as green (Win) and red (Avoid) dashed lines (±SEM) in the background of the two patient groups. (C) Differential performance associated with levodopa administration (ON > OFF, positive score relates to better performance ON levodopa) for each cue condition (Valence × RequiredAction) for tremor and non-tremor patients. Error bars represent standard error of the differences. **P < 0.01, *P < 0.05.

Across patient populations, there was a trend for a weaker motivational choice bias ON levodopa than OFF [RequiredAction × Valence × Medication: F(1,56) = 3.0, η² = 0.05, P = 0.09] (Fig. 2B). There was also a trend interaction with Parkinson’s disease motor phenotype [RequiredAction × Valence × Medication × Group: F(1,56) = 2.5, η² = 0.08, P = 0.09]. Finally, there was no significant group difference in terms of motivational bias [Valence × RequiredAction × Patient group: F(1,56) = 0.9, η² = 0.03, P = 0.4], nor did medication affect differential learning to Go or NoGo [Medication × RequiredAction: F(1,56) = 0.4, η² = 0.007, P = 0.5], or as a function of motor phenotype [RequiredAction × Medication × Group: F(1,56) = 1.3, η² = 0.05, P = 0.3].

Computational modelling

We used computational modelling to differentiate between a number of algorithms that could account for behaviour, and to assess which latent variables, i.e. computational mechanisms, may mediate the effects of medication on performance in the two patient groups. Specifically, we aimed to assess the relative contribution of motivational choice bias and reinforcement learning, and whether these mechanisms differed between groups. We started by considering a Rescorla-Wagner model (M1) (Rescorla and Wagner, 1972). Stepwise addition of a Go bias (M2), motivational choice bias (M3), and separate valence-based learning rates (M4, M5) improved the WAIC estimate of model evidence (Fig. 3A). Model M5 with separate learning rates for Win cues versus Avoid cues significantly outperformed M4, which had separate learning rates for ‘positive’ prediction errors (following a win or avoided punishment) versus ‘negative’ prediction errors (following a punishment or failure to win).

Figure 3.

Model and parameter inference. (A) Estimated negative log model evidence (WAIC), relative to model M1. Lower (i.e. more negative) WAIC indicates better model evidence. The simplest model M1 contains a feedback sensitivity (ρ) and learning rate (ε) parameter. Model evidence improves with addition of a Go-bias (b) (M2), motivational bias (π) (M3), and a separation of learning rates either by the sign of the prediction error (positive prediction error: ε_+pe; negative prediction error: ε_−pe; M4), or cue valence (Win: ε_win; Avoid: ε_avoid; M5). M5 performs best. (B) One-step-ahead predictions and posterior predictive model simulations of winning base model M5. This shows how the winning model captures the behavioural data (grey lines). Both methods use the fitted model parameters to compute the choice probabilities. The one-step-ahead predictions compute probabilities based on the history of each participant’s actual choices and outcomes, whereas the simulation method generates new choices and outcomes based on the response probabilities. (C) Effect of levodopa on parameter estimates generated from M5: (Difference ON − OFF levodopa in each Parkinson motor phenotype); we found that the tremor-dominant group showed a significant increase in punishment learning, while the non-tremor groups shows a significant decrease in motivational bias. Error bars represent standard error of the differences. *P < 0.05.

Given that the effects of medication depended on tremor phenotype, we next assessed how parameters differed as a function of patient group and medication status, analysing parameter estimates from the winning model (M5). We focused on both the motivational bias parameter, and the cue valence dependent learning rates, for each group, because modulation of (only) these parameters can account for the observed valence-based performance differences. Because the learning rate parameter distributions differed significantly from normal (Kolmogorov-Smirnov test, ε_win: P = 4×10⁻⁵; ε_avoid: P = 0.005; π: P = 0.9) (Supplementary Fig. 2A), we report non-parametric Mann-Whitney U-test statistics for these parameters. In the non-tremor group, there was a significant reduction in motivational bias [π: F(1,18) = 4.7, η² = 0.206, P = 0.04], but no changes in learning rates (ε_win: P = 0.5; ε_avoid: P = 0.4). In contrast, in the tremor group, the Avoid learning rate was higher in patients ON than those OFF levodopa (ε_avoid: P = 0.019), but no changes in reward learning rate (ε_win: P = 0.9), or motivational bias [π: F(1,40) = 0.1, η² = 0.001, P = 0.8]. This change in punishment-learning rate can easily explain the raw performance effects, i.e. relatively better performance for Avoid cues in tremor patients ON versus OFF medication.

For the non-tremor group, the change in the motivational bias parameter is puzzling at first, because in this group, the main effect of interest was an increase in performance on Win cues ON medication. The current observation suggests that the increased performance does not originate from an increase in reward learning (as is often assumed), but rather from reducing the automatic influence of reward cues on action invigoration, thereby allowing for a relatively greater impact of adaptively learnt instrumental values on the final choice, surfacing primarily in NoGo2Win trials.

Given this observation of reduced motivational bias in non-tremor patients ON medication, we performed a post hoc ANOVA to assess a change in motivational bias as a function of medication in the raw choice data, specifically for the non-tremor patients. Here we observe a significant interaction between Action × Valence × Medication [F(1,18) = 4.5, η² = 0.20, P = 0.048] (Fig. 2C), due to a disproportionate levodopa-related increase in accuracy in NoGo2Win trials. For completeness, this interaction was not present in the tremor group [Tremor: F(1,40) = 0.1, η² = 0.003, P = 0.7]. While this result should be interpreted cautiously given the absence of a significant four-way interaction [Action × Valence × Medication × Group: F(1,58) = 2.7, η² = 0.045, P = 0.10], it illustrates why the effect of medication on performance for non-tremor patients is captured by the parameter indexing the motivational (choice) bias, rather than a (differential) effect of valence learning.

Finally, there is an overestimation of the number of ‘Go’ responses for the initial trials, particularly for the NoGo2Avoid condition. This may reflect an underestimation of the initial motivational bias, which interestingly hints that the motivational bias could change over time. For example, when we gain confidence in instrumentally learnt action values, we may adaptively ‘dial down’ the bias (Fig 3B in Swart et al., 2017). However, such putative dynamics cannot be captured by the fitted constant parameter. Instead, it will take on the value that best reflects the motivational bias across all trials, thereby likely resulting in the observed mis-estimation in the initial trials. To capture such a change in motivational bias, we would need to add more free parameters to the model. Because the mis-estimation affects only the first few trials, power is too limited to fit such an effect. We are exploring putatively adaptive changes in motivational bias in future studies. 

Control analyses

We assessed (clinical) differences between patient groups as well as control groups to verify a successful inclusion procedure. There were no differences between patient groups and controls in gender balance, age, FAB and MMSE scores. There were also no differences between Parkinson’s disease phenotypes in terms of age, FAB, MMSE and MDS-UPDRS non-tremor score motor scores (Table 2). Tremor scores differed significantly between patient groups [t(54) = 9.79, P < 1.5 × 10⁻¹⁶], reflecting our inclusion procedure. There was a difference in average ‘task-delay’ (representing the delay between medicine administration and the onset of the behavioural task) of ∼30 min, reflecting the finding that non-tremor patients were consistently faster going through the experimental procedure preceding the behavioural Go/NoGo task. In the Supplementary material, we present follow-up analyses showing that this task-delay difference did not affect our findings. There, we also present the absence of any drug-related effects of confound variables age, gender and LEDD.

Table 2.

Disease characteristics of participants

	Range	Tremor	Non-tremor	P-value	Patients	Controls	P-value
Sample size		43	20		63	22
Gender, male:female		18:25	9:11	P = 0.9	27:36	9:13	P = 0.9
Age, years		61.4 (11.1)	60.2 (9.2)	P = 0.7	61.0 (10.5)	64.3 (9.1)	P = 0.19
FAB	0–18	17.3 (0.9)	16.5 (2.1)	P = 0.12	17.0 (1.4)	17.5 (0.9)	P = 0.8
MMSE	0–30	29.2 (1.3)	29.2 (1.3)	P = 0.9	29.2 (1.3)	29.3 (0.9)	P = 0.16
LEDD	0–2255	449 (302)	645 (503)	P = 0.052
UPDRS non-tremor	0–108	49.7 (24.2)	53.6 (23.1)	P = 0.6
UPDRSrest tremor	0–16	9.4 (3.9)	0.0 (0.0)	P < 2 × 10−18
Drug-delay, min		164 (23)	134 (16)	P < 0.001
Delay-selection, min		139 (17)	134 (16)	P = 0.4

Values are presented as mean (SD). For disease scores, min-max possible range is reported. Patients were successfully matched for gender, age, cognitive function [Frontal Assessment Battery (FAB) and Mini-Mental State Examination (MMSE)] and non-tremor disease severity (MDS-UPDRS part III, items 1–23), although there was a trend level difference in LEDD, with higher medication for non-tremor patients. Subjects were successfully differentially included based on resting tremor scores (MDS-UPDRS items 28-33). P-values are reported for the difference between tremor and non-tremor groups. Two-tailed t-tests were used for continuous variables, and χ² test for the categorical variable ‘gender’.

Discussion

In this study, we aimed to understand whether and how differences in the cognitive effects of dopaminergic medication relate to the presence or absence of tremor, a fundamental clinical variation in the Parkinson’s disease phenotype. Building on established neural differences in the dopaminergic phenotypes of tremor-dominant and non-tremor Parkinson’s disease patients, we investigated whether these two patient groups have different dopamine-dependent reinforcement learning deficits. We tested this hypothesis by disambiguating effects of dopaminergic medication on motivational choice from effects on learning. Our results provide evidence for different cognitive effects of dopamine-enhancing medication in tremor-dominant and non-tremor Parkinson’s disease patients. Dopaminergic medication enhances performance when cues signal a potential win, but this well-known dopaminergic effect on motivational choice bias (Frank et al., 2004, , 2007; Cools et al., 2006; Bodi et al., 2009; Palminteri et al., 2009) is restricted to non-tremor Parkinson’s disease patients. In contrast, tremor-dominant patients under levodopa learned faster during trials when punishment needed to be avoided. This finding suggests a novel correspondence, with a dopaminergic basis, between motor and cognitive phenotypes in Parkinson’s disease patients.

Different cognitive effects of dopamine in different motor Parkinson’s disease phenotypes

It has been argued that dopaminergic modulations of valence-dependent learning in fact reflect biasing of motivational choice (Shiner et al., 2012; Smittenaar et al., 2012). The current findings add to that debate by showing that patients with different motor phenotypes selectively change learning and choice-related computations when receiving levodopa. In non-tremor patients, levodopa decreases motivational choice bias, which matches the decrease in motivational bias evoked by levodopa in healthy participants performing the same task (Guitart-Masip et al., 2014). In contrast, in tremor-dominant patients, dopaminergic medication increases learning in Avoid trials. This effect is indeed opposite to previous reports where levodopa led to a reduction in learning to avoid punishment (Frank et al., 2004, 2007; Cools et al., 2006; Bodi et al., 2009; McCoy et al., 2019). However, this observation that dopamine enhances avoidance learning in tremor-dominant patients does fit with rodent studies showing a stimulus-locked dopaminergic surge observed during Go-to-avoid trials (Oleson et al., 2012; Gentry et al., 2016), as well as enhanced functional MRI blood oxygen level-dependent responses in the dopaminergic midbrain during successful active avoidance (Rigoli et al., 2016). This could be an instance of the suggested role of dopamine in ‘safety learning’, i.e. the active avoidance of an unpleasant stimulus (Mowrer, 1951; Dinsmoor, 1977; Lloyd and Dayan, 2016).

Taken together, the current findings provide further evidence for the notion that dopamine can modulate different computations contributing to value-based choice. There are a number of possible neural accounts of these differential effects in the two parkinsonian motor phenotypes. One possibility is that they reflect distinct functional anatomical alterations in tremor-dominant and non-tremor Parkinson’s disease patients, such as the different spatial distribution of dopaminergic degeneration in the midbrain of those two Parkinson’s disease phenotypes (Hirsch et al., 1992; Jellinger, 2012). This differential spatial distribution may provide an explanation for the observed differential effects of levodopa on win versus avoid performance, particularly given recent evidence for a spatial differentiation of midbrain dopaminergic neurons that reinforce avoidance of threatening stimuli versus rewarding stimuli (Menegas et al., 2018). A second possibility is that the severity of dopaminergic depletion, besides its spatial distribution, plays a role. However, this interpretation is less likely given the absence of effects of LEDD, an indirect marker of dopamine depletion. Other monoamines may also contribute, given that resting tremor in Parkinson’s disease has been associated with abnormalities in the noradrenergic and serotonergic system (Isaias et al., 2011; Qamhawi et al., 2015; Pasquini et al., 2018), dopaminergic medication in Parkinson’s disease has been shown to alter serotonin transmission (Everett and Borcherding, 1970; Miguelez et al., 2014, 2016), and serotonin is well known to be implicated in punishment learning (Soubrie, 1986; Chamberlain et al., 2006; Dayan and Huys, 2008; Crockett et al., 2009; Deakin, 2013).

Interpretational issues

This study involved a relatively large number of Parkinson’s disease patients (n = 63), all of whom were measured both ON and OFF dopaminergic medication, and was designed to assess behaviour in each group under both medication conditions. Unfortunately, we found test-retest differences in task performance across patients and healthy controls. Therefore, we had to limit analyses to Day 1 and shift towards a between-subject group design. Nonetheless, particularly the novel finding of dopamine-driven enhanced punishment learning in tremor-dominant patients is supported by a sizeable sample (20 ON dopamine and 23 OFF dopamine tremor-dominant patients).

We used a standardized levodopa dose instead of the patients’ own dopaminergic medication, to avoid heterogeneity in the effects of (different) dopamine agonists and different regimes of levodopa. The dose used here (200/50 mg levodopa-benserazide) was higher than the normal dose for most patients, as quantified using their LEDD. The difference in LEDD between patient groups was close to the statistical threshold, raising the possibility that tremor-dominant patients were overdosed relative to non-tremor patients. However, control analyses indicate that LEDD did not predict performance, and the findings did not change when including LEDD as a covariate.

Conclusion

Our key finding is that often-replicated effects of dopaminergic medication in Parkinson’s disease hold only for a minority of patients, namely patients without tremor (20–30% of the patient population) (Hughes et al., 1993; Helmich et al., 2012; Simuni et al., 2016). In line with the previous literature, non-tremor patients ON dopaminergic medication showed better performance in the reward domain than did patients OFF medication. In line with recent work in healthy subjects (Guitart-Masip et al., 2014), this effect reflected a decrease in motivational choice bias. In stark contrast to non-tremor patients, patients with tremor symptoms ON medication showed better Go-to-avoid learning in the punishment domain than did patients OFF medication. This improvement could putatively reflect dopamine-related changes in safety learning. These divergent effects of dopaminergic medication during reinforcement learning as a function of motor phenotype are especially relevant in light of established clinical cognitive/motivational differences between these patient groups (Dissanayaka et al., 2010; Wu et al., 2011; Wylie et al., 2012), and associated differences in degeneration of dopaminergic nuclei such as substantia nigra and retro-rubral area. Finally, our findings may have brought to light a structural testing bias in earlier studies, and underline the importance of increased awareness of interpatient diversity in Parkinson’s disease.

Glossary

LEDD =

levodopa equivalent daily dose