Identifying Cognitive Remediation Change Through Computational Modelling—Effects on Reinforcement Learning in Schizophrenia

Abstract

Objective: Converging research suggests that individuals with schizophrenia show a marked impairment in reinforcement learning, particularly in tasks requiring flexibility and adaptation. The problem has been associated with dopamine reward systems. This study explores, for the first time, the characteristics of this impairment and how it is affected by a behavioral intervention—cognitive remediation. Method: Using computational modelling, 3 reinforcement learning parameters based on the Wisconsin Card Sorting Test (WCST) trial-by-trial performance were estimated: R (reward sensitivity), P (punishment sensitivity), and D (choice consistency). In Study 1 the parameters were compared between a group of individuals with schizophrenia (n = 100) and a healthy control group (n = 50). In Study 2 the effect of cognitive remediation therapy (CRT) on these parameters was assessed in 2 groups of individuals with schizophrenia, one receiving CRT (n = 37) and the other receiving treatment as usual (TAU, n = 34). Results: In Study 1 individuals with schizophrenia showed impairment in the R and P parameters compared with healthy controls. Study 2 demonstrated that sensitivity to negative feedback (P) and reward (R) improved in the CRT group after therapy compared with the TAU group. R and P parameter change correlated with WCST outputs. Improvements in R and P after CRT were associated with working memory gains and reduction of negative symptoms, respectively. Conclusion: Schizophrenia reinforcement learning difficulties negatively influence performance in shift learning tasks. CRT can improve sensitivity to reward and punishment. Identifying parameters that show change may be useful in experimental medicine studies to identify cognitive domains susceptible to improvement.

Introduction

Converging literature suggests that people with schizophrenia have difficulties in using feedback to guide their behavior.^1,2 These difficulties are thought to underlie and maintain core symptoms associated with the diagnosis of schizophrenia and may be relevant in predicting poor prognosis.³ Despite a wealth of congruent findings, very limited research has explored interventions specifically targeting reinforcement learning. A possible reason is the poor alignment of reward sensitivity measures with measures employed in intervention trials. Intervention studies traditionally measure symptoms and/or functioning as their outcomes, while basic research studies measure outcomes associated with biological phenomena, structures, and processes. We argue that in a time where research efforts are focused onto closing the gap between the brain and behavior, efforts should be made to increase our understanding of processes that may help bridge these 2 perspectives. This view is further strengthened by recent initiatives to characterize new research paradigms in mental health such as the Research Domain Criteria (RDoCs) from the National Institute of Mental Health⁴ and the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) and Cognitive Neuroscience Test Reliability and Clinical applications for Schizophrenia (CNTRACS) consortia.^5,6

In this landscape, reinforcement learning seems a perfect candidate as a substantial amount of basic research has studied this phenomenon and linked it to specific neural areas and functions. The next step would be to explore this process in the context of complex interventions and assess whether it can change. In the current study we focus on the effect of cognitive remediation on reinforcement learning as this intervention primarily targets impaired cognitive domains, and some of its training components (eg, errorless learning and guided feedback) are likely to alter the sensitivity to reward and punishment.^7,8

Basic science measures are rarely used in clinical trials and this has limited the translational value of basic science findings to outcomes valued by patients and clinicians. In the field of medicine computational modelling is increasingly used to extract and simplify complex information from process data.⁹ The main advantage of this approach is in managing complexity and allowing the use of composite information.

One task assumed to require a number of different processes on which individuals with schizophrenia consistently underperform is the Wisconsin Card Sorting Test (WCST).¹⁰ This test requires trial and error learning as participants try to match response cards to 4 key cards according to a hidden sorting rule. This is the task we focus on in our computational modelling efforts as its performance depends on being sensitive to positive or negative feedback.

WCST performance in individuals with schizophrenia seems to be similarly impaired across the different task outputs including the number of correct responses and categories achieved but is possibly even more impaired on perseverative errors.^11,12 Poor performance seems to be stable over time and minimally influenced by repeated exposure to the task.^13,14 Performance on the test depends on a number of different cognitive domains including abstract thinking, working memory, set-shifting, and self-monitoring.¹⁵ Research conducted on individuals with brain injury, and brain imaging studies, show that the WCST performance is largely associated with activity in the prefrontal cortex regions.^16–19

Recent literature has focused on the role of reward processing to explain deficits in tasks requiring trial-by-trial learning, such as the WCST, in individuals with schizophrenia.^1,2,20,21 One of the key advantages in using a more molecular framework, such as reward processing sensitivity, to account for poor performance is the possibility of linking biological with behavioral abnormalities. One prominent example is the key role played by dopamine in the detection, evaluation, and prediction of reward.^21–24 People with schizophrenia show reward processing abnormalities in learning environments, and this has been associated with deficient dopamine receptor functionality.^21,25,26

The standard WCST scoring measures (eg, number of categories completed and the number of perseverative errors) can provide a general indication of the impaired domains. However, these measures have not always been able to pinpoint the underlying processing impairments in individuals with schizophrenia.^15,27,28 The standard scoring measures have also not helped differentiate schizophrenia from other clinical groups.²⁹,³⁰ It seems unlikely that diverse patient populations perform poorly on the WCST for the same reasons and that the same standard scores result from different patterns of information processing. There may also be biological differences. For instance, individuals with schizophrenia and Huntington’s disease, despite having similar WCST scores, show significantly different cerebral blood flow patterns during the task³¹ suggesting that multiple processes are recruited to perform the WCST, but such processes are not distinguished by the standard scoring measures alone.

Computational modelling provides an alternative means to disentangle processes in the task. In particular, a recent study validated a sequential learning model specifically designed for the WCST.³² According to this model, choices on the task depend on the perceived relevance of different dimensions (ie, color, form, and number), which increases or decreases across trials (see figure 1). The rate at which a participant adaptively modifies relevance weights depends on 2 free parameters: R and P. Respectively, these parameters reflect the degree to which a participant adaptively adjusts relevance weights based on reinforcing (“CORRECT”) and punishing (“WRONG”) feedback. Higher values on these parameters indicate quicker, more adaptive adjustment of relevance weights based on feedback. A third free parameter in the model, D, represents the degree to which a participant’s choices are consistent with the current relevance weights. Higher values on this parameter indicate that responses are more deterministic and lower values indicate more exploratory responses. These 3 parameters are estimated separately for each individual participant based on trial-by-trial choices in the WCST.

Fig. 1.

(a) If a participant sorts to match Number and is rewarded (right), the relevance weight for Number will increase. The reward sensitivity (R) parameter controls the rate of this increase. R ranges from .01 to 1, with higher values leading to faster updating of relevance weights following rewarded trials. In this example, an R of .1 leads to a subtle adjustment in favor of Number, whereas an R of .4 leads to a larger adjustment. The punishment sensitivity (P) parameter (not shown here) works similarly but controls the rate of updating following punished trials. P also ranges from .01 to 1, with higher values leading to faster (better) updating. (b) The probability of choosing to place a card into each pile is determined not only by the current relevance weights but also by the choice consistency (D) parameter. The D parameter ranges from .1 to 5.0, with a higher values leading to more deterministic choice probabilities. If D = 1, the probabilities of sorting based on color, form, and number are proportional to the current relevance weights, and so, in this example, there is a .60 probability of sorting on the basis of Number (ie, sorting into the 2 green stars pile). However, if D = 2, the decision is more deterministic, and the probability of sorting on the basis of Number is approximately .82.

Our overall research aim is to apply the cognitive modelling framework to characterize the effects of cognitive remediation therapy (CRT) on reinforcement learning. In order to formulate an empirically based hypothesis we first carried out a preliminary exploratory study to assess the parameters’ fit and to identify which parameters indicate poor performance in people with schizophrenia. Our next study (Study 2) then examines the effects of CRT on these reinforcement learning parameters. We hypothesized that a course of CRT would improve those parameters associated with difficulties on the WCST in schizophrenia as highlighted in Study 1.

The use of WCST cognitive modelling may offer some significant insights. Firstly, the parameters could reflect more accurately the cognitive and behavioral phenomena characteristic of the idiosyncratic learning patterns typical of people with schizophrenia. Secondly, characterising changes in more molecular phenomena such as reward and punishment sensitivity may help to map remediation effects against hypothesized biological mechanisms affecting reward sensitivity in schizophrenia.

Study 1

Method

Design and Procedure.

This study is a cross-sectional design comparing WCST scores and computational modelling parameters in a group of individuals with schizophrenia to a group of healthy controls (HC). In addition, participants with schizophrenia were assessed on clinical and social behavior characteristics and they carried out additional neuropsychological tests.

Participants.

The sample of patients with a diagnosis of schizophrenia (N = 100) was recruited as part of a larger ongoing randomized controlled trial (CIRCuiTS—Computerized Interactive Remediation of Cognition—Training for Schizophrenia).

Inclusion criteria for the study were age between 18–65 years, a Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) diagnosis of schizophrenia, and cognitive impairment of 1 SD below the population average in at least 1 of 4 cognitive domains including memory, working memory, processing speed, and executive function (as measured by cognitive assessment, see below). Poor social functioning, as evidenced by a score of 2 or above on a measure of social behavior (the Social Behavior Schedule, see below), was also an inclusion criterion. Participants were excluded if they had a history of learning disability/developmental disorder, history of organic brain disorder/head trauma, a diagnosis of substance dependence or if they required the use of an interpreter. Recruitment was conducted from clinical teams within the South London and Maudsley National Health Service (NHS) Foundation Trust and Sussex Partnership NHS Foundation Trust in the UK.

Participants in the control group (N = 50) were healthy individuals recruited with advertisement from the local community to match the clinical group for age and gender. Inclusion criteria were set to allow appropriate group matching: gender ratio male/female 3/1; age between 18–55 years, no history of DSM-IV diagnosis of schizophrenia or other mental disorder, no evidence of head injury/organic brain disorder and no diagnosis of substance dependence. Participants were also screened for prior knowledge of the WCST and were considered ineligible if they were familiar with or had already been exposed to the task.

Measures.

General demographic details were collected from both groups.

Wisconsin Card Sorting Task

All participants were assessed with the computerized version of the WCST adapted from Heaton.¹⁰ The task requires individuals to sort a deck of cards based on a periodically changing rule. After each card is sorted participants are provided with feedback (ie, right or wrong). Positive feedback is provided only if the sorting matches the current sorting rule. Three sorting rules are possible according to the attributes of the cards (ie, color, form, and number). Participants are encouraged to “discover” the sorting rule, however after 10 consecutive cards are correctly sorted (ie, category achievement criteria) the correct sorting rule changes without notification. From this point, only selections that match the new sorting rule will receive positive feedback. The task ends either after 128 trials or after 6 categories have been achieved. The WCST produces a number of outputs among which the most traditionally used are: number or percentage of error trials, number or percentage of correct trials, number of categories achieved, and number of preservative errors.^33,34 The WCST is among the most widely used neuropsychological tasks and features applications in almost all the fields of neurology, psychiatry, and cognitive science.^35–37 Clinical studies initially reported poor functioning on the task related to damage or dysfunction in the prefrontal regions,³⁸ more recently functional brain scanning studies reported more precisely the cortical and subcortical areas involved in the task including the ventrolateral and dorsolateral prefrontal cortex, the anterior cingulate cortex, and cerebellum.^16,39

Patient-only Data

• (i) Positive and Negative Syndrome Scale (PANSS). The score from the positive and negative symptom subscales were used in the analysis. A higher score on either scale indicates greater symptom severity.⁴⁰

• (ii) Social Behavio r Schedule (SBS). The SBS was used as a measure of social functioning and was rated by a member of each participant’s care team on their functioning over the last month. A higher total score suggests greater difficulty across the behavioral domains.⁴¹

• (iii) Neuropsychological Assessment. Premorbid intelligence quotient (IQ) was estimated with the Wechsler Test of Adult Reading.⁴² Current IQ was assessed with a short form of the Wechsler Adult Intelligence Scale Revised (WAIS-R).⁴³ The Rey Complex Figure,⁴⁴ the Hayling Sentence Completion Test,⁴⁵ the Digit Span and the Digit Symbol Coding⁴² were also used to assess memory, executive function, working memory, and processing speed, respectively.

WCST Parameter Estimation

Model parameters were estimated through Maximum Likelihood Estimation using previously published methods.³² Specifically, parameters were sought that maximized the likelihood of a participant’s next actual choice given all previous choices made and feedback received by that participant. Parameter search was implemented in the R programming language,⁴⁶ and relied on a robust combination of an iterative simplex search routine⁴⁷ combined with 100 quasirandom starting parameter sets. The search was conducted separately for each participant, resulting in separate estimates of R, P, and D for each individual. Parameters were constrained in range such that .01 < R, P < 1, and .1 < D < 5. The lower bounds of the parameters were set to be slightly higher than 0 because extremely low values on one or more parameters (especially D) can cause estimates of the other parameters to become unstable and arbitrary. Further details of the modelling estimation method and model fits can be found in the Supplementary Materials (see Appendix).

Analysis

Distribution characteristics were inspected for variance inequalities and normality violations. Analysis of variances and t tests were used to compare the 2 groups (ie, healthy controls and people with schizophrenia) on the modelling parameters and the WCST output scores. In case of distributional inequalities between groups approaching significance we supplement parametric with nonparametric analyses (Mann-Whitney U Test) to increase confidence in our results. Binary logistic regression was used to assess the discriminatory value of the WCST scores and computational modelling parameters in predicting membership to the clinical group. Beta scores were used to assess the contribution of individual predictors in the logistic regression. Kendall’s Tau_b correlation was used to explore whether the parameter estimates were related to medication, clinical, and/or neuropsychological variables in the participants with schizophrenia. This method was preferred to Spearman rho as it is considered more robust to tied data, which were common here.⁴⁸ To explore whether we can detect differences in neuropsychological and clinical variables associated with the reward learning parameters we compared participants grouped using a median split for each of the parameters as in Gold et al.⁴⁹ This method was preferred to control for skewed distribution and reduce heterogeneity. Significance threshold was set at P < .01.

Results

Table 1 shows the demographic and neuropsychological characteristics of the participants as well as additional descriptors of the clinical sample. There were no differences in gender distribution and age between the groups. Consistent with previous research, participants with schizophrenia showed more overall incorrect WCST trials, t(148) = 9.8, P < .0001, perseverative errors, t(148) = 6.9, P < .0001, and fewer categories achieved, t(148) = −8.8, P < .0001, compared with participants in the healthy control group. The schizophrenia group had lower scores on R, t(148) = −7.4, P < .0001; and P, t(148) = −7.9, P < .0001; but not on D, t(148) = −0.8, P = .44 (see figure 2). Mann-Whitney U tests to control for potential variance inequalities showed the same pattern of significant and nonsignificant results.

Table 1.

Study 1 Demographic and Clinical Characteristics of the Study Participants

Demographics	Schizophrenia	Control
Female (%)	35	32
Mean age (SD)	38.8 (11.2)	35.8 (8.9)
Mean education in years (SD)	13.5 (2.7)	15.3 (3.25)
Employment status(%)
Paid	7	80
Voluntary	20	0
Unemployed	50	2
Student	23	18
Time since first admission (%)
Less than 1 year	3	—
1–5 years	17	—
5–10 years	20	—
More than 10 years	60	—
Clinical characteristics
Mean SBS total score (SD)	10.12 (8.1)	—
Mean PANSS total (SD)	54.30 (14.8)	—
Mean PANSS positive (SD)	12.40 (5.1)	—
Mean PANSS negative (SD)	12.90 (5.8)	—
Mean estimated premorbid IQ (SD)	93.90 (11.1)	—
Mean current IQ (SD)	87.40 (13.9)	—
Mean chlorpromazine equivalent daily dose (SD)	517.40 (420.14)	—

Note: IQ, intelligence quotient; PANSS, Positive and Negative Syndrome Scale.

Fig. 2.

Study 1

(a) Mean and median percentage of Wisconsin Card Sorting Test (WCST) errors (b) Mean and median percentage of WCTS perseverative errors (c) Mean and median number of WCST categories achieved (d) Mean and median for R parameter (e) Mean and median for P parameter (f) Mean and median for D parameter.

Results of the logistic regression performed to predict group membership (HC vs schizophrenia) showed that R, P, and WCST number of perseverative errors were the only significant predictors (all P values < .01). The overall model explained approximately 60% of the variance (Nagelkerke presudo R ² = .59; P < .01) with P being the strongest predictor (β = −2.6) followed by WCST perseverative errors (β = −0.21) and R (β = −0.13).

Model parameters were expected to be related to traditional measures from the WCST. However, a perfect relationship would indicate that model parameters are redundant. Correlations between the parameters and the WCST scores for the schizophrenia group are reported in table 2. As expected, R, P, and, to a lesser extent, D were positively correlated with traditional measures of good performance (eg, total trials correct) and negatively correlated with traditional measures of bad performance (eg, total trials administered).

Table 2.

Study 1 Reinforcement Learning Parameters and WCST Outcome Correlations for Schizophrenia Group

	R parameter	P parameter	D parameter
P parameter	.346**	—	—
D parameter	.012	.086	—
WCST total trials administered	−.387**	−.500**	−.114
WCST total trials correct	.403**	.406**	.176**
WCST % errors	−.444**	−.500**	−.227**
WCST % perseverative errors	−.214**	−.582**	−.023
WCST number of categories completed	.546**	.491**	.280**
WCST number of failures to maintain set	.089	.093	.143

Note: P, punishment sensitivity; R, reward sensitivity; D, choice consistency.

*P < .05, **P < .01

Table 3 shows the comparisons between individuals with high and low reinforcement learning parameters on a range of neuropsychological and clinical measures. Standard WCST scores were significantly lower in those participants with poor sensitivity to feedback. Individuals with higher R and P scores also had significantly lower IQ loss following illness onset. Higher R scores were associated with better memory while those more responsive to negative feedback had lower negative symptoms.

Table 3.

Mean and Standard Deviation of Neuropsychological and Clinical Assessment for Low and High Parameters Groups Defined by Median Split

	R		P		D
	Low	High	Low	High	Low	High
WCST correct %	41.4 (10.7)	58.3 (14.3)	41.7 (13.2)	58.1 (15.2)	43.3 (13.5)	55.9 (16.8)
WCST error %	57 (10.9)	41.4 (17.7)	58.3 (11.8)	40.7 (16.6)	56.3 (17.2)	43 (13.6)
WCST perseverative errors %	27.9 (10)	24.8 (18.9)	31.1 (14.2)	21.5 (14.7)	28.7 (18)	24 (11.4)
WCST categories completed	1.1 (0.9)	3.8 (1.9)	1.3 (1.3)	3.6 (2.2)	1.6 (2.2)	3.3 (1.7)
Premorbid IQ	93.3 (10.9)	94.4 (11.3)	93.1 (10.7)	94.5 (11.5)	94.3 (10.8)	93.4 (11.3)
IQ loss	8.6 (9.7)	6.9 (9.5)	9.4 (10.1)	4.2 (8.5)	7.6 (8.8)	6.1 (10.4)
Ray recall	9.1 (5.1)	13.2 (6.8)	10.9 (6.8)	11.3 (5.9)	10.2 (6)	11.4 (6.5)
Hayling	5.6 (3.7)	4.6 (3.6)	5.7 (3.7)	4.5 (3.6)	4.9 (3.1)	5.3 (4.1)
Digit span	12.8 (3.4)	14.6 (3.4)	13.1 (3.2)	14.4 (3.7)	13.5 (3.7)	13.8 (3.9)
Digit symbol	42.9 (16.8)	47.2 (16.9)	42.7 (16.8)	47.4 (16.9)	43.9 (17.9)	45.2 (15.9)
PANSS positive	12.6 (4.9)	12.1 (5.2)	12.7 (5.4)	11.9 (4.7)	12.1 (4.5)	12.7 (5.6)
PANSS negative	13.3 (5.5)	12.5 (6.1)	15.1 (6.4)	10.7 (4.7)	13.9 (6.4)	11.9 (4.9)
SBS	10.3 (7.5)	9.9 (8.5)	10.8 (7.8)	9.4 (8.2)	9.9 (7)	10.3 (8.5)

Note: WCST, Wisconsin Card Sorting Test; SBS, social behavior schedule.

Significant differences in bold (P < .01).

In light of the logistic regression results, showing that D does not discriminate between people with schizophrenia and controls, we do not report this parameter in Study 2.

Study 2

Method

Design and Procedure.

This study is a nonrandomized controlled design comparing a group of patients with schizophrenia who received CRT with a similar group of patients receiving treatment as usual (TAU). Participants were assessed at 2 time points: baseline and 3 month (ie, post-therapy for the active group). For the purpose of this study the main outcomes are the model parameters that distinguished individuals with schizophrenia from HCs (ie, R and P).

Participants

Cognitive Remediation Group.

Participants in this group were recruited as part of a larger single arm cognitive remediation study (CRT, N = 37).⁵⁰ Participants were selected for this study if they had completed a WCST at each assessment point. There were no significant differences in demographic and clinical characteristics between those who completed the WCST (ie, 86% of the participants) and those who did not. Inclusion criteria to enter the trial were: no evidence of head injury or organic disorder, no substance abuse, no history of learning disability/developmental disorder, DSM-IV diagnosis of schizophrenia or schizoaffective disorder verified. Recruitment was conducted from clinical teams within the South London and Maudsley NHS Foundation Trust in the UK. Criteria were checked from a potential participant list from each team and following referral participants were approached for written informed consent. Participants in this group received cognitive remediation therapy.

Control Group.

Participants in this group were recruited as part of the treatment-as-usual arm of the CIRCuiTS trial (TAU; N = 34). This is an ongoing study recruiting patients from the same geographical area of the CRT group. Inclusion criteria, screening, assessment and consent procedure were the same. Participants in this group received standard care or treatment as usual from public mental health clinics as part of the National Health Service (NHS) in the UK.

Therapy

CRT is an individualized psychological therapy using a variety of training techniques, including massed practice, errorless learning, self-monitoring, and strategy-based learning, to improve cognitive performance.^51,52 CRT is a manual-based intervention provided in individual sessions by trained graduate psychologists. Individuals enrolled in the program complete tasks engaging predominantly memory, cognitive flexibility, and planning. Tasks vary in level of complexity, starting from the simplest possible and progressing to more difficult levels. Feedback is provided by the therapist during and at the end of each task and is structured and contextualized. Participants are also encouraged to reflect on mistakes that occurred during the task and prompted to think of strategies that may reduce future mistakes (eg, chunk information in more manageable portions) and the context in which the mistake has occurred (eg, noticing that in the presence of distractions we are more likely to miss the relevant information).

Each therapy session lasts up to an hour, and there is a minimum frequency of 3 days per week for approximately 12 weeks. Participant and therapist set goals and revise strategies regularly to aid transfer of cognitive skills to real life. Therapist fidelity was supported by clinical supervision and assessed by independent ratings of 50 session tapes on the CRT Fidelity Scale.⁵³ All ratings confirmed that the key therapy components were administered to protocol.

Measures

Assessment measures were the same as employed in Study 1 with the exception of the SBS.

Analysis

Reinforcement learning parameters were calculated as in Study 1. Similar to Study 1 distributional parameters and deviation from normality were inspected, and parametric testing was supplemented by post hoc nonparametric analysis to increase confidence in the findings. Analysis of covariance was used to explore the effect of therapy on each of the parameters. Levels of the parameters at post-therapy were entered as a dependent variable, therapy group was entered as the independent variable, and baseline parameter levels, levels of chlorpromazine equivalents and variables significantly different between the 2 groups at baseline were entered in the model as covariates. Kendall’s Tau_b correlation was used to investigate whether change in the parameters was associated with any other changes. In this exploratory analysis WCST scores, negative symptoms, and memory variables were chosen as they showed associations in Study 1 with R and P.

Results

General demographic, clinical, and neuropsychological characteristics of the groups recruited for Study 2 are presented in table 4. Participants did not differ, at initial assessment, in any of the variables considered (P values in all comparisons exceed the threshold of .05) apart from gender, χ²(1) = 8.7, P = .003. Gender was therefore controlled in all the analyses reported below.

Table 4.

Study 2 Demographic and Clinical Characteristics of the Study Participants

Demographics (at baseline)	CRT (N = 37)	TAU (N = 34)
Female (%)	73	39
Mean age (SD)	39.46 (9.81)	36.44 (10.13)
Mean education in years (SD)	13.03 (2.01)	13.32 (2.24)
Employment (%)
Employed (paid / voluntary)	89.2	44.1
Unemployed	0	38.2
Student	10.8	17.6
Mean hours in employment per week (SD)	14.28 (10.83)	6.64 (8.60)
Mean hours in education per week (SD)	9.20 (7.66)	5.36 (4.77)
Time since first contact with mental health services (%)
1–5 years	18.9	26.5
5–10 years	21.6	35.3
More than 10 years	56.8	38.2
Time since first admission (%)
Never admitted	0	11.8
1–5 years	29.7	23.5
5–10 years	24.3	29.4
More than 10 years	43.2	35.3
Clinical characteristics
Mean PANSS total (SD)	48.05 (9.94)	53.29 (14.56)
Mean PANSS positive (SD)	10.86 (4.15)	12.09 (5.14)
Mean PANSS negative (SD)	11.59 (4.12)	12.97 (5.40)
Mean estimated premorbid IQ (SD)	97.56 (10.33)	94.41 (11.80)
Mean current IQ (SD)	92.60 (18.03)	87.53 (16.98)
Mean chlorpromazine equivalent daily dose (SD)	514.7 (435.8)	575.8 (441.6)
Therapy
Mean number of CRT sessions	29.95 (4.97)	—

Note: CRT, cognitive remediation therapy; TAU, treatment as usual.

Two independent analyses were conducted to compare changes in the model parameters R and P in the 2 groups.

Reward Sensitivity

Analysis conducted on the reward sensitivity (R parameter) showed that there was a significant effect of CRT on post-treatment R levels, F(1, 71) = 4.4, P = .04, partial η² = 0.07. Mann-Whitney U test also showed that the CRT group had significantly higher R values at post-therapy, z = 2.04, P = .04 (see figure 3a).

Fig. 3.

Study 2

(a) Mean R parameter values before and after therapy for Cognitive Remediation (CRT) and Treatment as Usual (TAU) group.

(b) Mean P parameter values before and after therapy for Cognitive Remediation (CRT) and Treatment as Usual (TAU) group.

Punishment Sensitivity

The same analysis conducted on the punishment sensitivity (P parameter) revealed a significant effect of CRT on post-treatment P levels, F(1, 71) = 5.2, P = .02, partial η² = 0.08. Mann-Whitney U test also showed significantly higher P values in the CRT group at post-therapy, z = 2.41, P = .01(see figure 3b).

Correlates of Parameter Improvement

Improvement in R was associated with reduced WCST total errors (r = −.38, P < .0001) and more WCST categories (r = .59, P < .0001). There was a positive correlation trend between R improvements and digit span improvements (r = .279, P = .02). A similar analysis showed that improvements in P were related to improvements in the number of WCST correct choices (r = .31, P < .001), a trend for WCST categories (r = .41, P < .01). There was also a negative association trend for improvement in P and reduction in negative symptoms (r = −.35, P < .05).

Discussion

In these studies we apply, for the first time, reinforcement learning computational modelling to behavior during performance of the WCST by people with a diagnosis of schizophrenia. Our 2 molecular subprocesses, critical for learning in a changing environment, provided useful insights into the difficulty people with schizophrenia have with positive and negative feedback sensitivity and also their malleability following cognitive remediation.

Do People With Schizophrenia Differ on Their Sensitivity to Feedback?

Study 1 showed that learning from positive and negative feedback is impaired in schizophrenia, with HCs showing considerably higher scores on both the R and P parameters. This result is in keeping with recent literature highlighting difficulties of patients with schizophrenia in reward-based learning.^2,25

Participants with higher R scores had a better performance on both verbal and nonverbal memory tests. This association is consistently found in the literature and lends support to the validity of the R parameter.^2,54

The results of Study 1 also highlight the association between the parameters and the traditional WCST outcomes. Interestingly both R and P showed an association with the estimated IQ loss after illness onset. This suggests that the parameters may be indicative of overall cognitive loss resulting from the illness linking further a molecular aspect such as feedback sensitivity to more macroscopic indicators of cognitive function.

Participants with low P levels had more severe negative symptoms. This finding is in partial contrast to some recent findings suggesting that negative symptoms are preferentially associated with poor reward learning.^21,49 In the context of this study it is possible that this result may reflect the more behavioral assessment of negative symptoms as indexed by the PANSS and may be associated with lack of motivation and consequent poor use of negative feedback.

The choice consistency parameter D did not differ between individuals with schizophrenia and HCs which suggests, unlike for other neuropsychiatric problems, it is not a key parameter in determining performance.³²

Can Reward Sensitivity Change?

In Study 2 both the P and R parameters showed a significant change after therapy. This change suggests that CRT modifies both reward and punishment sensitivity in patients with schizophrenia.

Two limitations should be noted when considering the results of Study 2. Firstly, the clinical samples recruited were not randomized and one characteristic emerged as different between the groups—gender distributions. Although gender has not been implicated as having a differential effect on reward processes we nevertheless controlled statistically for unequal gender distribution in all the Study 2 analyses. Secondly, our cognitive remediation program was provided by therapists, and it is possible that the effects of CRT on reward sensitivity in participants with schizophrenia were mediated by expectations and social contact. Recent research suggests that social reward acts on the same brain reward networks of other types of reward (eg, money).⁵⁵ It is therefore possible that the therapist role might have strengthened the feedback provided as part of the CRT program.

Change in R and P following CRT correlated with WCST improvements in the numbers of categories achieved. Also change in R correlated with reduced number of errors and change in P with larger number of correct choices. Despite participants in the CRT group showed only a trend towards improvement in the WCST traditional measures the correlation suggests that improvement in the reward learning parameters is associated with changes at the more global behavioral level as accessed by the WCST output measures. It is of interest that our modelling parameters were able to detect some subtle aspects of performance that are not indexed by the traditional WCST scores suggesting that these parameters might be precursors, of interest to those wanting to understand the first responses to therapy.

Consistent with the results of Study 1 there was an association trend for improvement in R and memory, and improvements in P and reduced negative symptoms. This suggests that improvements in reward sensitivity may have a preferential positive effect on cognition while changes on negative feedback sensitivity may have a more targeted effect on negative symptoms. These are themes that would deserve further research to elucidate the mechanisms so that interventions can be adapted accordingly.

It is possible that the unstimulating environments in which individuals with schizophrenia live may contribute to the reward insensitivity. However, difficulties in reinforcement learning are present both in the premorbid and early clinical stages of schizophrenia^24,56 suggesting that problems in this domain may be specific to psychosis. Notwithstanding this consideration, it is likely, in particular for those individuals with a long illness, that a negative environment might exacerbate the poor sensitivity to feedback.

What Are the Implications of the Findings for Future Research?

Reward processing abnormalities are increasingly suggested as an important feature of schizophrenia contributing to positive and negative symptoms.^1,2 A recent study from Nielsen et al.²⁴ linked abnormalities in the reward system with dopamine dysregulation (ie, D2/D3 receptors) in the ventral striatum and suggested an association between improvements in positive symptoms, after initial antipsychotic psychotherapy, and increased sensitivity to reward in patients with schizophrenia. Studies conducted on patients with major depression also suggested that reward sensitivity improves, after pharmacological treatment with antidepressants.⁵⁷ Our results indicated no interaction between medication level and change in reinforcement learning parameters. However, unlike the results reported by Nielsen et al.,²⁴ and Stoy et al.,⁵⁷ our study only considered individuals on a stable level of medication (ie, over 75% were medicated with antipsychotic medications for 5 years or more). Medication changes may influence reward sensitivity. However our results suggest that at a stable dose medications do not deter the effect of cognitive remediation.

The interaction between cognitive remediation and pharmacological interventions targeting dopamine receptors is still unclear largely because the putative targets of these interventions are considered different (eg, cognition and positive symptoms). Our parameter measures might be useful behavioral indicators of change both for psychological intervention and cognitive enhancement drug studies.⁷

Conclusions

Our study, using behavioral and computational modelling methods, showed for the first time that reinforcement learning can change as a result of nonpharmacological treatments targeting cognition. Achieving, supplementing or boosting reinforcement learning sensitivity using nonpharmacological interventions can be extremely beneficial for patients and may lead to improvements in social functioning or in learning related tasks such as work. The association between reinforcement learning and changes in dopamine levels remain highly speculative and in need of further research. Nonetheless, these types of studies align well to the ethos of recent initiatives (eg, RDoC, CNTRICS, and CNTRACs) and ultimately aim to provide hypotheses for linking the biology with the behavior.

Supplementary Material

Supplementary material is available at http://schizophreniabulletin.oxfordjournals.org.