Reinforcement and Reversal Learning in First-Episode Psychosis

Abstract

Background: Abnormalities in reinforcement learning and reversal learning have been reported in psychosis, possibly secondary to subcortical dopamine abnormalities. Methods: We studied simple discrimination (SD) learning and reversal learning in a sample of 119 first-episode psychosis patients from the Cambridge early psychosis service (CAMEO) and 107 control participants. We used data on reinforcement learning and reversal learning extracted from the Cambridge Neuropsychological Test Automated Battery Intradimensional-Extradimensional shift task, which measures cognitive flexibility but also involves simple reinforcement learning (SD learning) and reversal learning stages. We also gathered diagnostic information to examine whether there were any differences between patients ultimately diagnosed with schizophrenia-spectrum disorders and those diagnosed with affective psychosis. Results: Psychosis patients demonstrated deficits in simple reinforcement learning (SD learning) and in reversal learning, with no differences between affective psychosis and schizophrenia-spectrum psychosis. There was a significant modest correlation between reversal errors and negative symptoms (Spearman ρ = 0.3, P = .02). Conclusions: There are reinforcement learning abnormalities in first-episode psychosis, which correlate with negative symptoms, suggesting a possible role for orbitofrontal cortex and ventral striatal pathology in the pathogenesis of motivational deficits in psychosis.

Introduction

It has long been proposed that there are associative and reinforcement learning deficits in schizophrenia and other psychoses.^1–7 Such theories have been strengthened by documentation of dopamine deficits in the pathophysiology and treatment of schizophrenia,^8,9 given evidence from animal^10,11 and human¹² studies implicating a critical role for dopamine transmission in reinforcement learning. Until recently, reinforcement learning in psychosis has received little laboratory study. However, initial evidence has recently emerged demonstrating abnormal reinforcement learning, coupled with disrupted activity of dopamine innervated brain regions in schizophrenia and other psychoses.^13–15

Reinforcement learning, in its general sense, involves a subject learning by trial-and-error feedback to select actions that maximize reward over time. One variant of reinforcement learning, termed reversal learning, examines the ability to flexibly adapt the response to a change in learning contingencies. Reversal learning is of particular interest in psychotic illness as a marker of functioning in orbitofrontal cortex–ventral striatal circuitry. Other orbitofrontal cortex–related paradigms, such as the Iowa Gambling Test, have yielded equivocal results in schizophrenia.^16,17 Three previous studies have specifically examined reversal learning in psychosis, each finding deficits, but these few existing studies have sampled only patients with chronic schizophrenia.^18–20 It remains unclear whether reinforcement and reversal learning are dysfunctional in the early phase of schizophrenia and whether these processes are differentially impaired in schizophreniform psychoses and affective psychoses. We therefore examined simple reinforcement and reversal learning in a large sample of first-episode psychosis patients who undertook the Intra-Extra Dimensional (ID/ED) Set Shift task from the Cambridge Neuropsychological Test Automated Battery (CANTAB) cognitive test battery.²¹ The ID/ED test involves various stages of rule learning and rule reversal, as well as the formation of attentional set. Because of the highly structured progression of the task, it is possible to examine specific cognitive-behavioral components involved at different stages of the experiment. We specifically focused on performance of those parts of the test involving reversal learning and compared performance in psychosis patients with matched controls. Furthermore, we followed up patients presenting with first-episode psychosis, and stratified their performance according to formal diagnostic status after 12 months. In this report, we do not focus on ED set shifting performance, as performance there has previously been well studied in early psychosis,²² including in a partially overlapping sample.²³

Methods

Participants

One hundred nineteen individuals (mean age 23.4 years; 88 men) with first-episode psychosis were recruited from the Cambridge first-episode psychosis service, Cambridge early psychosis service (CAMEO; www.cameo.nhs.uk), for the study. Inclusion criteria for CAMEO is age between 17 and 35 years, suffering from a first episode of psychosis as defined by the Melbourne criteria of the presence of psychotic symptoms for at least a week,²⁴ and duration of antipsychotic treatment of under 6 months at time of initial assessment. One hundred seven healthy volunteers (mean age 24.5 years, 72 men) were recruited from the general population by advertisement to act as a control group. The majority of the patients were taking second-generation antipsychotic medication. Most patients in this study were assessed within 6 weeks of their referral to CAMEO and had mild symptoms at the time of the experiment. Twelve months after the experiment, a psychiatrist (G.K.M.) utilized all available clinical information including case history, ongoing clinical assessments, structured clinical interview for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition,²⁵ and operational criteria diagnostic system ²⁶ to classify the cases as affective psychosis or nonaffective psychosis. The research was approved by the Local Research Ethics Committee; all participants provided written informed consent. The National Adult Reading Test was used to estimate IQ of patients and controls.

ID/ED Shift Test

The ID/ED Shift test was derived from the Wisconsin Card Sort Test.²⁷ It involves visual discrimination and attentional set formation and tests the maintenance, shifting, and flexibility of attention (see figure 1). Two dimensions are used in the test: color-filled shapes and white lines. Simple stimuli are made up of just one of these dimensions, whereas compound stimuli are made up of both, namely white lines overlying color-filled shapes. At the first stage, simple discrimination (SD), the subject starts by seeing 2 simple color-filled shapes, and must learn which one is correct by touching it. Feedback teaches the subject which stimulus is correct, and after 6 correct responses, the contingencies change and the previously correct element becomes incorrect (simple reversal stage). At this point distracting stimuli (lines) are added in order to provide compound discrimination stages (CD1 and CD2), followed by compound discrimination reversal. After this stage has been learnt, there is an intradimensional shift (IDS), where new exemplars of the 2 dimensions “line” and “shape” are introduced, but the relevant dimension is unchanged, eg, color-filled shapes remain the only relevant dimension. After an intradimensional reversal (IDR), there follows an extradimensional shift (EDS, eg, white lines become the relevant dimension) and a final reversal stage (EDR). Subjects progress through the test by satisfying a set criterion of learning at each stage (6 consecutive correct responses). We extracted information on all stages but focus our analysis on the basic acquisition and reversal trials. In cognitive assessments, the ID/ED test is sometimes preceded by a preliminary test called Big/Little Circle, which is a screening test in the CANTAB test battery; we did not use the Big/Little Circle test in our study.

Fig. 1.

Schematic of the Intradimensional-Extradimensional Shift Task From the Cambridge Neuropsychological Test Automated Battery (CANTAB). The correct choice for each stage is marked with a green box. Copyright 2008 Cambridge Cognition, Ltd. All rights reserved.

Data Analysis

Demographic characteristics of the 2 groups, and number of subjects failing test stages, were compared using independent samples t tests and chi-squared tests. Error scores were skewed and could not be normalized by mathematical transformation, and were therefore analysed with Mann-Whitney U Tests. Because error scores are count data, the relationship between error scores and symptoms scores were implemented using Poisson regression in intercooled Stata 8.2. Separate analyses with error score as the dependent variable were employed for each predictor variable, and the results confirmed with correlational analyses using Spearman ρ.

Results

Group demographics and diagnostic information:

The psychosis group and the control group were matched on age, gender, and NART estimated verbal IQ (Table 1). When all available information was utilized to apply diagnostic categories after 12 months, 81 patients were classified as schizophrenia-spectrum psychosis and 31 as affective psychosis, with missing information on 6 cases. PANSS scores from initial assessment were available on 78 patients.

Table 1.

Demographic Information on Psychosis Cases and Controls and on Psychosis Patients Stratified by Diagnosis of Nonaffective or Affective Psychosis (Diagnostic Information Was Missing on 6 Patients)

	Control, Mean (SD)	All Psychosis, Mean (SD)	Nonaffective Psychosis, Mean (SD)	Affective Psychosis, Mean (SD)	T or χ2	df	P
Age (y)	24.5 (4.7)	23.3 (5.4)	23.6 (5.6)	24.2 (1.1)	1.6	224	.1
NART	112 (13)	109 (8)	108 (8.2)	111 (7.8)	1.9	175	.1
Gender	72 Men, 35 women	88 Men, 31 women	63 Men, 19 women	20 Men, 11 women	1.2	1	.3
PANSS positive	N/A	16.9 (6.2)	18.4 (6.2)	13.6 (5.2)
PANSS negative	N/A	14.5 (6.6)	15.4 (6.7)	12.2 (6.0)

Note: Also shown are statistical comparisons of psychosis patients vs controls in age, National Adult Reading Test Score, and gender.

Learning analysis: stages passed and failed

In the control group, 1 participant failed at the IDS stage, 2 failed at the IDR stage, 5 failed at the EDS stage, and 1 failed at the final reversal stage (see figure 2). In patients, 2 failed at the compound discrimination stage, 1 at the compound reversal stage, 27 failed at the EDS stage, and 11 failed at the final reversal stage. Thus, in terms of stage failures, significantly more patients failed the EDS stage (χ = 16.5, P < .001), and the final reversal stage (χ = 9.6, P < .01), than controls.

Fig. 2.

Attrition at Each Stage of the Intradimensional-Extradimensional Test. SD, simple discrimination; SR, simple reversal; CD1, compound discrimination 1; CD2, compound discrimination 2; CDR, compound discrimination reversal; IDS, intradimensional shift; IDR, intradimensional reversal; EDS, extradimensional shift; EDR, extradimensional reversal. Asterisk indicates group difference P< .01.

Learning analysis: error analysis

For a more sensitive analysis, we examined error scores at each stage. We compared all patients (n = 119) vs all controls (n = 107) on initial discrimination learning (figure 3). By examining the number of errors using the Mann-Whitney U test, we confirmed that psychosis patients made more errors than controls during initial discrimination learning (z = 2.5, P = .01); in contrast, there were no differences between patients and controls on initial reversal learning, compound discrimination learning, ID set shifting, compound reversal, or IDR. However, there were deficits in ED set shifting (z = −5.1, P < .001) and final reversal stages (z = −3.7, P < .001). Next, we examined the total number of reversal errors over the course of the experiment in participants who attempted all stages of the IDED test: ie, those who completed at least the EDS stage and so could attempt the final reversal stage (99 controls and 89 patients, see table 2 and figure 4). Although some psychosis patients showed good performance, as a group patients made more total reversal errors than controls (z = −2.4, P = .02).

Fig. 3.

Simple Discrimination Errors in Psychosis Cases and Controls.

Table 2.

Comparison of Psychosis Cases and Control Error Scores (Mann-Whitney U Test) in Those Participants Who Attempted All Stages of the IDED Test: ie, Those Who Completed At Least the Extradimensional Shift Stage and Who Therefore Were Able to At Least Attempt the Final Reversal Stage

	Psychosis N = 89 Mean (SD)	Control N = 98 Mean (SD)	z	P
Simple discrimination	0.84 (0.62)	0.68 (0.75)	−2.33	0.02
Simple reversal	1.27 (0.72)	1.27 (0.75)	−0.04	1
Compound discrimination 1	1.06 (1.41)	1.04 (1.61)	−0.03	1
Compound discrimination 2	0.22 (0.56)	0.15 (0.48)	−0.92	0.4
Compound reversal	1.64 (1.71)	1.48 (1.51)	−0.62	0.5
Intradimensional shift	0.67 (1.19)	0.67 (1.37)	−0.22	0.8
Intradimensional reversal	1.30 (0.86)	1.32 (0.97)	−0.14	0.9
Extradimensional shift	6.55 (5.66)	4.36 (4.45)	−3.44	<0.001
Extradimensional reversal	5.31 (8.39)	1.40 (1.20)	−3.71	<0.001
Total reversal errors	9.60 (9.11)	5.72 (3.62)	−2.41	0.02

Fig. 4.

Total Reversal Learning Errors in Psychosis Cases and Controls.

We next examined whether, within the patient group, total reversal errors could be explained, at least in part, by SD errors. Utilizing Poisson regression, we found that SD errors did not predict total reversal errors (z = 0.7, P = .5). In contrast, SD errors were a significant predictor of EDS errors (z = 5.4, P < .001).

Having established the differences between first-episode psychosis patients and controls, we proceeded to examine whether patients with schizophrenia-spectrum psychoses differed from patients with affective psychosis. Patients with affective psychosis made fewer ID shifting errors than patients with schizophrenia-spectrum psychosis (z = −2.3, P = .026), but there were no differences in any other stages of the ID/ED or in total reversal errors (z = −0.539, P = .6; table 3).

Table 3.

Comparison of Nonaffective Psychosis and Affective Psychosis Error Scores (Mann-Whitney U Test) in Those Cases Who Attempted All Stages of the IDED Test: ie, Those Who Completed At Least the Extradimensional Shift Stage and Who Therefore Were Able to At Least Attempt the Final Reversal Stage

	Nonaffective Psychosis, N = 62, Mean (SD)	Affective Psychosis, N = 24, Mean (SD)	z	P
Simple discrimination	0.85 (0.62)	0.88 (0.61)	−0.30	.77
Simple reversal	1.32 (0.81)	1.17 (0.48)	−0.77	.44
Compound discrimination 1	1.13 (1.48)	0.88 (1.30)	−0.89	.38
Compound discrimination 2	0.23 (0.58)	0.25 (0.53)	−0.47	.64
Compound reversal	1.65 (1.64)	1.58 (1.93)	−0.91	.36
Intradimensional shift	0.82 (1.29)	0.33 (0.92)	−2.58	.01
Intradimensional reversal	1.32 (0.84)	1.29 (0.95)	−0.57	.57
Extradimensional shift	6.26 (4.91)	8.16 (6.54)	−0.97	.33
Extradimensional reversal	5.06 (7.83)	6.46 (10.15)	−0.16	.88
Total reversal errors	9.35 (8.47)	20.82 (11.15)	−0.54	.59

Finally, we examined whether symptom scores correlated with performance on simple and reversal trials in the 56 first-episode psychosis patients who completed at least the ED stage of the ID/ED test and for whom PANSS scores were available. Poisson regression revealed that total reversal errors were predicted by negative symptoms (z = 3.72, P < .001), but not positive symptoms (z = −0.6, P = .6), which is consistent with results from a correlational analysis: total reversal errors correlated significantly with negative symptoms (Spearman ρ = 0.3, P = .02) but not with positive symptoms (Spearman ρ = 0.2, P = .20). There was no association between SD errors and psychopathology either on correlational analysis (positive symptoms Spearman ρ = 0.06, P = .6; negative symptoms Spearman ρ = 0.02, P = .98) or on Poisson regression (positive symptoms z = 0.52, P = .6, negative symptoms z = 0.35, P = .7).

Discussion

As expected, we found that psychosis patients had deficits in ED set shifting. This deficit has been previously documented in chronic schizophrenia^19,28–30 and in first-episode psychosis,²³ although some studies in first-episode psychosis suggest that there may either be no deficit in this domain³¹ or that the deficit may be slight.^22,32 Given that a number of previous studies have investigated ED set shifting in schizophrenia and early psychosis, here we focus our interpretation on the results concerning simple reinforcement learning and reversal learning.

Elliott et al¹⁸ and Pantelis et al²⁸ previously demonstrated reversal learning deficits in chronic schizophrenia. Both these groups extracted reversal learning performance from a version of the ID/ED test: the same approach that we employ in this study. Waltz and Gold²⁰ showed profound reversal deficits in 34 patients with chronic schizophrenia using a different method; they employed a probabilistic reversal task adapted from Robbins and colleagues.^33,34 Our results demonstrate that reversal learning deficits are also present in many patients near the time of initial presentation to psychiatric services. We note that these deficits were not universal however, and many patients performed at comparable levels to controls (see figure 4).

In contrast to Waltz and Gold,²⁰ who argued that patients performed adequately on rule acquisition, we were able to detect subtle abnormalities in simple reinforcement learning (SD learning), possibly because of our large sample size. Our study thus provides further support to long-held contentions that there are reinforcement learning abnormalities in psychosis. Discrimination learning has been shown to be impaired by caudate tail lesions³⁵; previous data supports caudate dysfunction in psychosis.^13,36,37 Specifically, the tail of the caudate is itself connected to the medial temporal lobe, an area that is strongly implicated in the pathogenesis of psychotic illness³⁸ as well as playing a role in discrimination learning. Research in rhesus monkeys has shown that lesions to the medial temporal lobe rhinal cortex, and to the inferior temporal cortex, result in mild and severe deficits, respectively, in discrimination learning, possibly through an inferior temporal-frontal-thalamic network.^39–43 Thus, the SD deficit we note is also consistent with previous evidence for disrupted frontotemporal connectivity in psychosis.^44,45

Patients with affective and nonaffective psychosis did not differ significantly in reversal learning errors (or indeed in EDS errors). Previous research has identified deficits in reversal learning to be present in bipolar mania,⁴⁶ consistent with other recent research implicating orbitofrontal cortex dysfunction in mania (including manic psychosis), such as the presence of impairment on the Iowa Gambling Test.⁴⁷ Interestingly, reversal learning is intact in euthymic bipolar disorder without a history of psychosis, suggesting a state-dependent deficit in nonpsychotic bipolar disorder.⁴⁸

Lesion studies in rodents and nonhuman primates have demonstrated a key role for the orbitofrontal cotrex and ventral striatum in reversal learning.^49–53 Moreover, this evidence is corroborated from human functional imaging studies^33,54,55 and from studies of human patients with orbitofrontal lesions.^56,57 These regions are critical for motivational and goal-directed processing¹⁰; thus, the present study suggests that there is dysfunction of orbitofrontal/ventral striatal circuitry in psychosis. This contention is consistent with the findings of our correlational analysis in patients, which demonstrates that the greater the reversal impairment, the more severe the negative symptoms (ie, the greater the impairment in motivational and goal-directed behavior). We note that the specificity of this correlation should be viewed with caution because the magnitude of the significant correlation coefficient between negative symptoms and reversal errors (ρ = 0.3) differed only slightly from the nonsignificant correlation between positive symptoms and reversal errors (ρ = 0.2).

Interestingly, we found that the patient group made few errors at the compound discrimination stage, which is in contrast with recent results reported by Jazbec et al.²⁹ They studied 34 patients with chronic schizophrenia and found pronounced deficits in compound discrimination. It is possible that this process may deteriorate with disease progression, though longitudinal research will be required to examine this conjecture.

Our study does have a number of limitations. Although we found deficits in SD learning, the ID/ED test is not solely or primarily a test of this cognitive domain. Given that the test starts with SD learning, it is conceivable that some psychosis patients might have had trouble adjusting to the task environment in general, leading to an apparent specific deficit in this domain. In addition, there was only a small range in scores in SD learning, which limits the power of correlation and regression analyses to detect associations with clinical variables. For this reason, the failure to detect association between SD errors and clinical variables should not be overinterpreted. SD learning, and its association with clinical variables, merits further investigation in early psychosis in other cognitive paradigms that focus on SD learning in more detail.

Another limitation of the current study is that the majority of patients were taking second-generation antipsychotic medications. Such medications act on dopaminergic and serotonergic systems, and ascending serotonin and dopamine neurotransmitter systems are known to play a modulatory role in reinforcement learning processes.^51,52,58 There are, however, a number of reasons why our current results are unlikely to be secondary to medication effects. First, we note that in a recent functional magnetic resonance imaging study in healthy volunteers, a low dose of the dopamine D2/D3 receptor antagonist, sulpiride, did not modulate brain activations during reversal learning or impair behavioral reversal performance.⁵⁹ Secondly, we observed a correlation between the level of negative symptoms and reversal errors, consistent with the theory that both these measures are secondary to one underlying pathological process. Finally, we have, in recent studies, demonstrated behavioral and physiological abnormalities during tests of reinforcement learning and motivational modulations in unmedicated first-episode psychosis patients.^13,15 Future studies should examine reversal learning in unmedicated patients with psychosis, its relation to symptoms, and the extent to which reinforcement and reversal learning deficits can be modulated by pharmacological interventions. The relationship between reinforcement learning and reversal deficits and functional impairments also merits investigation.

Funding

UK Department of Health National Institute of Health Research RCD Award (to GKM); NARSAD Young Investigator Award (to GKM); Bernard Wolfe Health Neuroscience Fund (to PCF); Wellcome Trust Senior Research Fellowship in Clinical Science (to PCF); CAMEO received pump priming funding from the Stanley Medical Research Institute and GlaxoSmithKline, and is now funded by the National Health Service; study completed within the University of Cambridge Behavioural and Clinical Neuroscience Institute, supported by a joint award from the Medical Research Council and the Wellcome Trust.