Probabilistic Reinforcement Learning in Adults with Autism Spectrum Disorders

Abstract

Background

Autism spectrum disorders (ASDs) can be conceptualized as disorders of learning, however there have been few experimental studies taking this perspective.

Methods

We examined the probabilistic reinforcement learning performance of 28 adults with ASDs and 30 typically developing adults on a task requiring learning relationships between three stimulus pairs consisting of Japanese characters with feedback that was valid with different probabilities (80%, 70%, and 60%). Both univariate and Bayesian state–space data analytic methods were employed. Hypotheses were based on the extant literature as well as on neurobiological and computational models of reinforcement learning.

Results

Both groups learned the task after training. However, there were group differences in early learning in the first task block where individuals with ASDs acquired the most frequently accurately reinforced stimulus pair (80%) comparably to typically developing individuals; exhibited poorer acquisition of the less frequently reinforced 70% pair as assessed by state–space learning curves; and outperformed typically developing individuals on the near chance (60%) pair. Individuals with ASDs also demonstrated deficits in using positive feedback to exploit rewarded choices.

Conclusions

Results support the contention that individuals with ASDs are slower learners. Based on neurobiology and on the results of computational modeling, one interpretation of this pattern of findings is that impairments are related to deficits in flexible updating of reinforcement history as mediated by the orbito-frontal cortex, with spared functioning of the basal ganglia. This hypothesis about the pathophysiology of learning in ASDs can be tested using functional magnetic resonance imaging.

Introduction

Autism spectrum disorders (ASDs) are characterized by impairments in social functioning and language, and by the presence of restricted interests and repetitive behaviors. Neurocognitive research has attempted to explain ASDs from the perspective of affect recognition [Hobson, 1996], theory of mind [Baron-Cohen, 1995], and executive functions [Hill, 2004; Pennington & Ozonoff, 1996]. A complimentary approach, which has not been widely investigated, is to conceptualize ASDs as disorders of learning.

In recent years, substantial progress has been made in understanding the cognitive and neural underpinnings of learning. Reinforcement learning describes how organisms acquire the ability to map situations with actions that maximize resulting rewards [Sutton & Barto, 1998]. It involves extracting reinforcement history implicitly from the environment [Cleeremans & McClelland, 1991; Curran, 2001; Knowlton, Mangels, & Squire, 1996; Reber & Squire, 1998] and adopting the optimal balance of “exploration” and “exploitation” of behavioral options [Sutton & Barto, 1998].

Both animal and computational models, as well as human behavioral and neuro imaging studies, suggest that reinforcement learning is supported by basal ganglia based neural circuits, and the neuromodulator dopamine (DA) [Brown, Bullock, & Grossberg, 2004; Waltz, Frank, Robinson, & Gold, 2007]. One influential primate model [Schultz, 1998] holds that DA bursts in the striatum act as a temporal difference reinforcement learning signal. Choices that lead to unexpected rewards produce transient bursting of dopaminergic cells. Conversely, choices that do not yield expected rewards produce dips in DA firing. This process trains the basal ganglia about the reward value of given actions.

While simple associative and habit learning is thought to be supported primarily by the basal ganglia [Graybiel, 2008; Jog, Kubota, Connolly, Hillegaart, & Graybiel, 1999]. Higher-level goal-directed behavior is thought to involve mediation by regions of the prefrontal cortex (PFC) [Aizenstein et al., 2004; Balleinea & Dickinson, 1998; Daw, Niv, & Dayan, 2005; Doll, Jacobs, Sanfey, & Frank, 2009; Graybiel, 2008]. The orbito-frontal cortex (OFC) is specialized for rapid and flexible updating of representations of expected value [Rolls, 2004; Schoenbaum & Roesch, 2005]. In one prominent systems level computational model of reinforcement learning [Frank & Claus, 2006], the OFC is thought to receive signals from the basal ganglia, and to store very short term “working memories” of the reward value of actions. This reward-related information exerts a top-down biasing effect on the more primitive and slower-to-train basal ganglia system. Consistent with findings about neurobiology, computational modeling work has shown that early in learning, reward-related information represented in the OFC has a greater influence on behavior. After training, the contribution of the basal ganglia predominates as habits form [Frank & Claus, 2006].

Simply put, implicit learning is learning that takes place outside of conscious awareness [Reber, 1967]. There has been recent interest in studying various forms of implicit learning children and adults with ASDs including motor procedural learning [Barnes et al., 2008; Brown, Aczel, Jimeacutenez, Kaufman, & Grant, 2010; Gordon & Stark, 2007; Mostofsky, Goldberg, Landa, & Denckla, 2000; Nemeth et al., 2010; Travers, Klinger, Mussey, & Klinger, 2010]; artificial grammar [Brown et al., 2010; Klinger, Klinger, & Pohlig, 2010], and other aspects of language learning [Scott-Van Zeeland, Dapretto, Ghahremani, Poldrack, & Bookheimer, 2010]; prototype learning [Brown et al., 2010; Soulieres, Mottron, Giguere, & Larochelle, 2010; Vladusich, Olu-Lafe, Kim, Tager-Flusberg, & Grossberg, 2010]; and social and non-social probabilistic implicit learning [Scott-Van Zeeland et al., 2010]. Motor procedural learning has been assessed using serial reaction time tasks which involve training on specified botton-push sequences. Serial reaction time studies have demonstrated learning impairments in higher-functioning adolescents with ASDs [Mostofsky et al., 2000], as well as lower-functioning individuals [Gordon & Stark, 2007]. However, using improved IQ matching, less “deterministic” or predictable sequences, and tasks with shorter response intervals, several newer studies have failed to find such differences [Brown et al., 2010; Nemeth et al., 2010] or have documented only subtle ones, suggesting that while learning does occur in these individuals, it develops over a prolonged time course [Barnes et al., 2008]. Findings of learning studies relevant to language have failed to find behavioral differences in artificial grammar learning in adolescents [Brown et al., 2010], or the learning of artificial speech streams in young adults; however, functional magnetic resonance imaging (fMRI) of the fronto-temproal-parietal networks for the ASD group during the speech stream study did show the ASD group did not manifest the “typical” pattern where there was a facilitatory effect of cues to word boundaries [Scott-Van Zeeland et al., 2010]. Finally, findings about implicit prototype and category learning have been mixed. One of the first studies of this form of learning in lower-functioning individuals with autism showed an absence of facilitiation by prototypes [Klinger & Dawson, 2001]. However, a study of individuals with IQs in the typical range did not reveal group differences in the prototype effect [Molesworth, Bowler, & Hampton, 2005]. The Brown et al. [2010] study also failed to show these group differences. Some studies, which offer a bridge to reconciling these disparate findings, suggest that while prototype learning is possible, category acquisition is slower [Bott, Brock, Brockdorff, Boucher, & Lamberts, 2006; Soulieres et al., 2010; Vladusich et al., 2010]. A study of probabilistic reinforcement learning with probabilities of 0, 50, and 100% showed that individuals with ASDs exhibited impairments relative to typically developing individuals in an implicit learning task with social and monetary reward conditions [Scott-Van Zeeland et al., 2010].

Some of these behavioral findings [e.g. Barnes et al., 2008] have been interpreted as suggesting that striatal and medial temporal lobe structures are relatively spared in individuals with ASDs. Indeed, structural [Bauman & Kemper, 2005; Hardan, Kilpatrick, Keshavan, & Minshew, 2003] and functional [Luna et al., 2002] studies have failed to establish the pathology of the basal ganglia and other striatal structures as central to ASDs. However, there have been more consistent reports about impairments in the functioning of the OFC as assessed by behavioral [Dawson, 2008] and neuroimaging studies [Loveland, Bachevalier, Pearson, & Lane, 2008; Schultz et al., 2000] of the disorders. Furthermore, studies of functional connectivity between PFC and other brain regions, which likely is involved in flexible updating of reinforcement history, have demonstrated less frontal neural network integration [Just, Cherkassky, Keller, & Minshew, 2004; Minshew & Williams, 2007; Solomon et al., 2009].

To the best of our knowledge, the current study is one of the first to investigate probabilistic reinforcement learning in adults with ASDs and the first to use univariate and Bayesian state–space methods, which are more sensitive to learning related changes, better model the shapes of learning curves, and commonly are used in animal learning studies. Based on other findings about learning in ASDs, our over-arching hypothesis was that learning in individuals with ASDs would be possible, but that this group would show atypicalities in early learning, and that such impairments would be suggestive of primarily frontal as opposed to striatal pathology. The first hypothesis was that, while both groups would exhibit comparable performance by test, in early learning (block 1), individuals with ASDs would demonstrate deficits relative to typically developing adults, and that this would be most pronounced for stimuli with more reliable (i.e. greater) reinforcement probabilities because performance in these simple cases benefits from maintenance of recent reinforcement history (i.e. “reward working memories”), which are dependent on an intact PFC/OFC. Based on this line of reasoning, our second hypothesis was that in early learning (block 1), individuals with ASDs would outperform typically developing individuals on less frequently validly reinforced stimulus pairs where reliance on reward working memories could actually hurt performance, and reliance on more habit-like basal ganglia-mediated responding, would produce better integration of probabilistic outcomes across trials [see Frank, Moustafa, Haughey, Curran, & Hutchison, 2007]. This type of strategy, however, could make one more subject to maladaptive and inflexible “win–stay” (sticking with the last correct response) or “lose–shift” (shifting away from the last incorrect response) behaviors. Thus, our third hypothesis was that individuals with ASDs would show impairments in win–stay and lose–shift behavior during early learning.

Methods and Materials

Participants

Twenty-eight adults with ASDs (Mean age = 23.5; (SD) = 5.50) and thirty adults with typical development (Mean age = 24.4 (5.08)) enrolled in this study, and were able to perform the task at levels better than chance. See Table I. Based on the male to female gender ratio of approximately 4:1 in the population [Nyden, Hjelmquist, & Gillberg, 2000], five women were enrolled in each group. Participants were recruited through psychiatrists, speech and language pathologists, advocacy groups, psychologists, state-funded centers for persons with developmental disabilities, and M.I.N.D. Institute’s Subject Tracking System database. All participants had a Full Scale IQ of at least 70 on the Wechsler Abbreviated Scales of Intelligence (WASI) for Children [Wechsler, 1999]. Given that stimuli consisted of Japanese Hiragana, we enrolled no subjects that could read or speak Japanese. Of the 28 enrolled participants with an ASD, 10 were diagnosed with high-functioning autism, 15 were diagnosed with Asperger syndrome, and 3 with PDD-NOS, according to criteria set by the DSM-IV-TR [American Psychiatric Association, 2000], ADOS-G [Lord et al., 2000], and a DSM-IV-R checklist. The decision to include individuals with both high-functioning autism and Asperger’s syndrome derives from studies showing that it is difficult to reliably distinguish between the two disorders, especially by adolescence [e.g. Howlin, 2003; Macintosh & Dissanayake, 2004; Ozonoff & Griffith, 2000; Szatmari, Bryson, Boyle, Streiner, & Duku, 2003]. Exclusion criteria for ASD subjects included diagnoses of autism with known genetic etiologies (i.e. fragile X syndrome, tuberous sclerosis), and known psychiatric diagnoses. Participants taking antipsychotic medications, which are known to interact with the DA system, were excluded. Individuals taking stimulants (two in the ASD group) were asked to stop taking these medications for 48 hr prior to the study. Four remaining subjects in the autism group were taking SSRIs.

Table I.

Participant Characteristics

	ASD group (n = 28)		TYP group (n = 30)
	Mean (SD)	Range	Mean (SD)	Range
Age (Years)	23.5 (5.50)	18–40	24.4 (5.08)	18–40
VIQ	110.7 (15.55)	86–145	112.8 (11.31)	91–128
PIQ	108.6 (16.36)	80–134	112.3 (12.78)	86–129
FSIQ	111 (16.04)	85–140	115.8 (13.00)	87–136
ADOS communication	3.8 (1.55)	2–8	–	–
ADOS social interaction	7.2 (1.83)	4–12	–	–
ADOS restricted interest	1.1 (1.03)	0–3	–	–
ADOS comm+social	10.9 (2.70)	7–18	–	–
Male/Female ratio	23:5	–	26:5	–
Asperger’s syndrome	15	–	–	–
High functioning autism	10	–	–	–
PDD-NOS	3	–	–	–

All subjects gave written assent along with consent from their legal guardians to participate in this study, which was approved by the University of California, Davis’ Institutional Review Board.

Measures

Qualification

WASI [Wechsler, 1999] was developed to provide a short and reliable means of assessing intelligence in individuals aged 6–89. The WASI produces the three traditional Verbal, Performance, and Full Scale IQ scores. It consists of four subtests: Vocabulary, Block Design, Similarities, and Matrix Reasoning. The WASI is nationally standardized, and exhibits strong psychometric properties. It has exhibited acceptable levels of internal consistency, test–retest reliability, and validity.

Autism Diagnostic Observation Schedule—Generic [ADOS-G; Lord et al., 2000]

Once qualification based on the WASI was established, participants with ASDs were administered Module 3 or 4 of the ADOS-G, a semi-structured interactive session and interview protocol that offers a standardized observation of current social-communication behavior. Participants are rated based on their responses to standardized social “presses”. An algorithm score, that combines ratings for communication and reciprocal social interaction, is the basis for diagnostic classification. The ADOS-G has demonstrated high levels of inter-rater reliability, test–retest reliability, and internal consistency reliability, and inter-rater agreement in diagnostic classification [Lord et al., 2000].

Learning

The probabilistic learning task was administered on a laptop computer with a 15-inch monitor. Participants were instructed to press a key corresponding to the side of the stimulus pair they believed to be correct. Visual feedback was provided following each choice as either the word “Correct!” printed in blue or the word “Incorrect” printed in red. If no response was made after four seconds, “no response detected” was displayed printed in red.

Probabilistic Selection Task [Frank, Seeberger, & O’Reilly, 2004; Frank, Rudy, Levy, & O’Reilly, 2005; Frank, O’Reilly, & Curran, 2006]

Three stimulus pairs, AB, CD, and EF, consisting of two Japanese characters (Hiragana) were presented. Given that poor randomization could induce response bias, the order of the trials (i.e. AB, CD, EF) in the experiment was randomized with the constraints that there had to be equal numbers of each trial type and that they had to appear sequentially (i.e. one of each type every three trials). The side that the truly correct character (i.e. A, C, and E) appeared on started the same for all participants (A was on the right and C and E were on the left). These positions alternated. Thus, given that trial order was randomized, there was no set pattern or side that the truly correct character appeared on. Participants learned to choose one of the two stimuli based on probabilistic feedback following each trial. They were instructed that one of the stimuli was “correct” and that one was “incorrect,” and that they were supposed to guess the “correct” figures as quickly and accurately as possible. They also were told there was no absolute right answer, but that some symbols had a higher chance of being correct than others and that it was their job to pick the symbol they thought had the higher chance of being correct. For AB trials, a choice of stimulus A led to valid positive feedback 80% of the time, while a B choice led to valid negative feedback in these trials. In the remaining 20% of AB pairs, invalid feedback was given. For CD trials, valid feedback was given 70% of the time, and in EF trials valid feedback was given on 60% of trials. The probability of valid or invalid feedback (i.e. cue-outcome contingencies) was determined based on the set percentage for each trial type (i.e. 80%, 70%, and 60%) calculated at each individual trial. Terminal percentages were checked to ensure that they did not deviate significantly from these benchmarks. Criteria for passing on to the test block were 65%, 60%, and 40%, respectively, for AB, CD, and EF trials. These criteria were selected to ensure that all subjects were at roughly the same performance level on the basic discriminations before advancing to the test phase. The 65% criterion on AB pair ensures that participants have learned, but is not so strict to induce overtraining. For the purposes of assessing positive and negative learning (which is what is most often probed with this task), it is less critical that robust preferences are exhibited in the other pairs, which have on average 50% value and are separately paired with A and B in the test phase. The main reason to impose any criterion at all on them is to ensure that there is no strong bias to prefer the less reinforced stimulus. Given that subjects perform less robustly on the lower probability pairs, we impose a more liberal criterion on these pairs. Participants who were not able to complete the AB and CD and EF trials at levels greater than these levels after six training blocks were omitted from the analysis during the test block. Participants were instructed to use “gut instinct” when uncertain. After training, participants were tested with familiar and novel combinations of stimulus pairs with either an A (AC, AD, AE, AF) or a B (BC, BD, BE, BF). No feedback was provided during testing. Each test pair was presented six times. See Figure 1 for a schematic diagram of the PS task.

Figure 1.

The PS task. Example stimulus pairs for the probabilistic stimulus selection (PS) task, which minimize explicit verbal encoding. The task consists of two phases. During the training phase, subjects are presented with three stimulus pairs (AB, CD, and EF). Each pair is presented separately in different trials in random order, and participants have to select among the two stimuli; correct choices are determined probabilistically. The frequency of positive and negative feedback for each stimulus is shown. Once a subject was able to score better than chance on AB and CD trials or completed 360 total trials, they proceeded to the test phase. In the test phase, 12 new pairs (only eight are shown) created from all unused combinations of training stimuli, are introduced and tested along with the three training pairs.

Data Analysis

Given prior findings and study hypotheses, our focus was on early learning as demonstrated in block 1, which was quantified in two ways. First, we examined overall error rates for each trial type in the first block using univariate analyses. Second, we employed a Bayesian state–space model. This type of model relies on the assumption that trial-by-trial observations of task performance are a noisy approximation of an underlying smooth cognitive state and that consideration of trial-by-trial performance within the context of this state provides a more sensitive means of determining whether learning has occurred [Smith et al., 2004]. The question answered by state–space models is whether the probability of a group or subject’s performance is above chance at either a given trial or over the state. Such models can also be used to compare performance between individuals or groups to answer the question of whether the probability of one individual’s or groups’s performance being greater than chance is greater than the probability of the other’s being greater than chance.

The state–space model consisted of two underlying equations (1) a state equation and (2) an observation equation. The state equation defines the temporal evolution of task learning, and was assumed to follow a Gaussian random walk. A binomial observation equation related the state to the observations [Kitagawa & Gersch, 1996]. The model was estimated using Markov-Chain Monte-Carlo methods as described previously [Smith, Wirth, Suzuki, & Brown, 2007]. It is referred to as an “ideal observer” approach because it computes the learning curve fit to all the data over all time in contrast to a causal filter approach. Given its sensitivity, this method has become a widely accepted way to conceptualize animal and human learning [e.g. Kumaran, Summerfield, Hassabis, & Maguire, 2009; Singer & Frank, 2009]. Note that the learning curves computed this state–space model are smoothed and their slopes may not be as steep as the raw data.

Trial-to-trial behavior as a function of error feedback during early learning (the first block of training for each trial type) also was investigated in a “win–stay” (exploitation of rewarded stimuli) and “lose–shift” (shifting to the alternative choice when stimuli are not rewarded) analysis. Win–stay behavior was defined as the percentage of trials following positive feedback in which the participants chose the same stimulus, and lose–shift behavior was defined as the percentage of trials following negative feedback in which participants avoided choosing the same stimulus. Learning in the test block also was examined using univariate and state–space measures.

Prior to all statistical analyses, data for each participant were checked to ensure there were no apparent perseverative patterns of responses (i.e. responses on one side only or responses that alternated from sided to side), which would constitute response biases. Univariate analyses were completed in SPSS version 16.0. The state–space analysis was performed using Monte Carlo Markov Chain software [Lunn, Thomas, Best, & Spiegelhalter, 2000] interfaced to Matlab [Natick, 2000] using Matbugs [Murphy & Mahdaviani, 2005].

Results

All participants were able to complete the trials at levels greater than chance by the end of training, and 86% of ASD and 87% of TYP participants met criteria to go on to the test block, and a chi square test revealed this percentage was not significantly different (χ² = 1.23, P = 0.268). Furthermore, the groups did not differ significantly with respect to the mean blocks to achieve performance criteria t(56) = 0.25, P = 0.80). See Table II, which also illustrates why the use of the first block for analyses presents the most unbiased view of the data, given that the sample size reduces by about one third by the second block based on training criteria, and continues to decline in subsequent blocks.

Table II.

Participants Reaching Criteria by Block

	1 Block	2 Blocks	3 Blocks	4 Blocks	5 Blocks	6 Blocks	Mean
Number of training blocks completed before test block
ASD (N = 28)	10	6	2	1	0	9	3.071
TYP (N = 30)	11	5	2	3	1	8	3.067
	Block 1	Block 2	Block 3	Block 4	Block 5	Block 6	Total
Percentage of subjects that achieve criterion/block
ASD (N = 28)	35.7%	21.4%	7.1%	3.6%	0.0%	17.9%	85.7%
TYP (N = 30)	36.7%	16.7%	6.7%	10.0%	3.3%	13.3%	86.7%

Early Learning on PS Task

A 3×2 analysis of variance (ANOVA) examined error rates by trial type (AB, CD, EF) by group (ASD and typically developing) in the first block of the task. There was a main effect of trial type (F(2, 112) = 10.22, P = 0.001, ${\mathrm{\eta }}_p^2=0.154$). Paired samples t-tests showed that performance on EF trials was significantly different than that on CD (t(57) = 2.92, P = 0.005) and AB (t(57) = 4.72, P = 0.001) trials. There was no main effect of group (F(1, 56) = 0.05, P = 0.824). However, the group by trial type interaction was significant (F(2, 112) = 4.30, P = 0.016, ${\mathrm{\eta }}_p^2=0.071$). Planned comparisons showed that the group with ASDs outperformed the group with typical development on the EF trials (t(57) = 2.78, P = 0.007). There were no performance advantages for the typical group on CD trials using univariate methods (t(57) = 1.68, P = 0.125). See Figure 2. Given the significant group by trial type interaction, we also examined within group patterns of responding. For the TYP group, error rates on AB trials differed significantly from those on EF trials (t(58) = 4.19, P<0.001); as did those on CD and EF trials (t(58) = 3.87, P<0.001). There were no significant differences between error rates on AB and CD trials (t(85) = 0.182, P = 0.86). For the ASD group, there were no significant differences in error rates across the trial types, although the difference between AB and CD trials approached significance (t(54) = 1.48, P=0.14, as did the difference between AB and EF trials (t(54) = 1.6, P<0.11).

Figure 2.

Early learning on the PS task-univariate analysis. Univariate analysis performance of 58 subjects (28 ASDs and 30 TYPs) during the first training block of the PSS task. There was no significant difference between the two groups for the AB and CD training pairs, but the ASDs performed significantly better (P = 0.017) than TYPs on the EF pair which is only correctly reinforced 60% of the time.

We then applied the state–space analysis to each group’s pooled responses across the first 20 trials for each stimulus pair. At each trial, the raw data for the typically developing and ASD groups consists of the proportion of correct respondents from that group. The state–space analysis yields median learning curves and 95% credible intervals for each stimulus trial type. Performance is judged to be above chance for any trial where the 95% lower credible interval is above 0.5. We illustrate the group performance on all the trials in Figure 3. Raw data is marked by open circles. A bar at the origin signals that results on the first trial should be at chance, although the smoothing inherent in state–space modeling which takes performance over the entire block into account, can make the curve appear flattened and shifted upwards from the origin for initial trials. For the AB and CD trials, both groups performed above chance (P = 0.5) for at least part of the first 20-trial block. For the EF trials, typically developing participants performed at the chance level for all 20 trials whereas the ASD participants were able to perform above chance for 5 of the last 6 trials.

Figure 3.

State–space learning curves for all trial types for ASD and TYP in block 1. The state–space model showing the performance on the three training pairs (AB, CD, and EF) for 58 subjects (28 ASDs and 30 TYPs) during the first training block. The bottom panel shows the exact trials for which performance was significantly different for the groups as places where the gray region is above or below the x axis. There was a greater overall probability of having better performance on CD trials if one was in the typically developing group, and an overall probability of having better performance on EF trials if one was in the ASD group.

In Panels C, F, and I, we show the trial-by-trial probability that the typically developing group’s performance was better than the ASD group’s performance. This is a probability distribution estimated by subtracting the ASD learning curve distribution from the typically developing learning curve distribution. Note that the 95% credible bounds on this computed difference are broader than the credible intervals on the learning curves as expected when subtracting two distributions. These distributions allowed us to show when performance at a specific trial was different for the groups. For example, the typically developing group’s performance was better than the ASD performance when the lower 95% credible interval was above zero. Similarly, when the ASD performance was better than the typically developing performance the upper 95% credible interval was below zero. From comparison curves in Figure 3 it is clear that there were no performance differences between groups for the AB trials. For the CD trials the performance of typically developing participants was better than the ASD performance at one trial (trial 11), whereas for the EF trials the ASD performance was better than typically developing individuals at several trials (trials 7, 10, 15, and 16). In addition to the trial-by-trial measures one can also ask (using Monte Carlo sampling techniques) more general questions such as whether over all 20 trials the typically developing participants outperformed the ASD participants by using this subtraction methodology. Overall, the results show no between-group performance differences on AB. However, during early learning, typically developing individuals outperform individuals with ASD on CD (P<0.001). On EF trials this effect was reversed: individuals with ASD outperformed typically developing individuals (P<0.001). Thus, the probability that these groups performed better than chance over the “state” of the first block differed significantly on these tasks, even at the Bonferroni-corrected 0.01 significance level of 0.01/3 = 0.003. Wilcoxon two-sided signed-rank tests, with Bonferroni-correction (AB difference = ns; CD difference<0.001; EF difference<0.001) also confirmed these results.

T-tests were used to examine win–stay and lose–shift behavior for the first block. Individuals with ASD were significantly worse at winning and staying on trials t(54) = 2.512, P = 0.015, (Cohen’s d = 0.41, indicative of a medium effect size), although they did not differ from typically developing individuals on losing and shifting. While the ASD group was worse at winning and staying for all trial types, the significant overall difference was driven by the CD trials (t(56) = 2.67, P<0.01). See Figure 4.

Figure 4.

Win–stay and lose shift behavior on the PS task in block 1. The win–stay and lose–shift percentages for 58 subjects (28 ASDs and 30 TYPs) during the first training block of the PSS task. The win–stay percentages were calculated by summing all incidents in which a subject chose the same stimulus (“stayed”) after receiving positive feedback (“winning”) for a given train pair and dividing it by the total number of times they received positive feedback, regardless of whether the feedback was accurate. Likewise, the lose–shift percentages were calculated by summing all incidents in which a subject chose a different stimulus (“shifted”) after receiving negative feedback (“losing”) for a given training pair and dividing it by the total number of times they received negative feedback. In Block 1, TYPs were significantly more likely than ASDs to win and stay; however, lose–shift performance was equivalent.

Test Block

To examine performance upon completion of training, a 3×2 ANOVA examined error rates by trial type and by group for the test block. There was a main effect of trial type (F(2, 112) = 3.214, P = 0.044, ${\mathrm{\eta }}_p^2=0.054$). Paired samples t-tests showed that there was a significant difference between EF and CD trials (t(57) = 2.30, P = 0.025), and EF and AB trials (t(57) = 2.03, P = 0.047). There was no main effect of group (F(1, 56) = 0.083, P = 0.77). The group by trial type interaction also was not significant (F(2, 112) = 0.488, P = 0.62). This suggests that there were no differences between the groups in learning after training. The state–space model produced similar non-significant results all across all three trial types.

Discussion

This study confirmed our hypothesis that there would be subtle, but clear early learning differences in individuals with ASDs, although they would be able to achieve typical performance levels over time. Contrary to expectations, both groups were able to perform the simplest and most consistently accurately reinforced pair at comparable levels from the outset. As shown by the more sensitive state–space model, however, the probability of learning the CD pair in this first block was poorer in individuals with ASDs. As hypothesized, both univariate and state–space methods confirmed that individuals with ASDs were better at acquiring the EF pair. It is also interesting to note that the TYP group performed similarly on AB and CD trials, but their performance differed significantly from EF trials, whereas the ASD group showed no significant performance differences between the trials although CD and EF performance was most similar for them. This may suggest that the groups detect when feedback is “valid” with different sensitivities, and/or that the ASD group is less sensitive to feedback across all trial types. The ASD group also showed early deficits in using positive feedback to “exploit” correct feed back by winning and staying, although the percentage of times they shifted to away from choices accompanied by negative feedback was comparable to TYPs.

Contrary to our first hypotheses, during the first block of the task, the ASD groups’ performance on the most reliably reinforced AB pair was comparable to the typically developing group. This runs counter to the supposition that reliable reinforcement information mediated by an intact OFC is necessary to complete the task [Frank & Claus, 2006; Graybiel, 2008]. In hindsight, however, we would argue that this close to accurately reinforced pair was relatively simple, and could be learned through rote memorization or even explicit as opposed to implicit strategies. Indeed, declarative and recognition memory, which are involved in rote learning, are thought to be intact or superior in autism [Bowler, Gaigg, & Gardiner, 2008]. This finding also is consistent with one prominent cognitive theory of autism which posits that individuals with ASDs showed spared or facilitated simple information processing (including declarative and recognition memory) and impaired complex information processing [Minshew, Goldstein, & Siegel, 1997]. Furthermore, it has been suggested that explicit strategies can be used to bootstrap implicit ones in either or both groups [Brown et al., 2010]. The degree to which explicit strategy use may have affect AB performance in the ASD group remains to be tested.

The hypothesis that individuals with ASDs would perform better than typically developing individuals on the EF trials, since information provided by rapid updating of OFC of representations of reinforcement contingencies using frequently incorrect feedback would lead to poorer performance, was confirmed using both univariate and state–space methods. This adds to a body of findings about islands of spared or superior abilities such as declarative memory [Walenski, Mostofsky, Gidley-Larson, & Ullman, 2008], and visual perception [Plaisted, O’Riordan, & Baron-Cohen, 1998] in individuals with ASDs. Assuming our suggestions about the neurobiology underlying such performance deficits, which includes enhanced basal ganglia functioning and impairments in the PFC/OFC is accurate, this could be conceptualized as a case of “paradoxical functional facilitation” [Kapur, 1996], which is said to occur when an important neural process is inhibited and leads to compensatory plasticity in another brain region. Such facilitation has been reported for other disorders including schizophrenia where patients demonstrate increased word reading and reaction time facilitation in the incongruent condition of the Stroop task, due to their inherent context processing deficits [Barch, Carter, & Cohen, 2004]. Our findings also are important because it is critical to identify relative strengths in patients alongside their impairments to address the concern that specific findings are purely the consequence of generalized performance deficits [Chapman & Chapman, 1973; Knight & Silverstein, 2001; Pennington, 2002].

Our findings of impairments in win and stay behavior has clinical face validity. Individuals with ASDs frequently do demonstrate atypical patterns of motivation [Chen, 2005], including the inability to initiate goal directed behavior [Hughes, 2001; Ozonoff & Jensen, 1999; Ruble & Scott, 2002]. This could be a symptom of their difficulty representing positive reward-related feedback in the OFC, which would undercut their ability to act adaptively by exploiting rewarding opportunities. This type of deficit has been observed in disorders thought to involve DA dysregulation [see Juckel et al., 2006 for an example in schizophrenia]. Our findings are consistent with those of Johnson, Yechiam, Murphy, Queller, and Stout [2006], who found that young adults with ASDs were less efficient in extracting the motivational significance of the various decks used in a Gambling task paradigm. Although they may be at odds with those of Minassian, Paulus, Lincoln, and Perry [2007] who found no group differences in win–stay or lose–shift behavior in a decision-making task with valid probabilistic feedback 20, 50, and 80% of the time, although this study did not look specifically at win–stay and lose–shift behavior in early learning.

Impairments in win and stay behavior along with a lack of impairment in lose and shift behavior could be interpreted as inconsistent with the perseverative responding and restricted and repetitive behaviors found in ASDs. This raises the question of whether win and stay and lose and shift percentages are good measures of perseveration. Given that perseveration involves the bias to continue selecting a preferred stimulus even when it is not reinforcing, win and stay behavior (or its absence) does not assess perseveration per se because winning and staying is an adaptive and lawful means of “exploitation” of the environment. Although at first blush, it appears that intact losing and staying performance makes the case there is no perseveration, there are two additional points to consider here. First, over time, it is true that participants need to learn to lose and shift in response to negative feedback, but they also have to learn not to do this once they discover that a particular stimulus is good on average. Examining the first block only is a way to try to get at the early basic process but it is hard to know to what degree the second counteracting factor (learning not to do this) plays a role. Ultimately, there might be group differences in lose–shift if there were a reversal in reinforcement contingencies after the probabilities were well learned. This remains to be tested.

There is, however, a relationship between performance on the PS task and cognitive and behavioral flexibility, although it cannot be summed up using only win and stay or lose and shift data. As we have argued, flexible/non-repetitive behavior can be conceptualized using the computationally based model [i.e. Frank & Claus, 2006], in which perseveration would be seen as basal ganglia-based learning that is not under good control of the PFC/OFC, and/or that there are deficits in representation of motivational context in PFC/OFC due to poor signaling/connectivity between the striatum and PFC. Such a problem would result in inflexible and perseverative behavior that it is not responsive to reward context (i.e. reward-based working memory). According to such a model, the impairment in win and stay behavior may reflect a problem in one form of signaling that influences reward-based working memory leading to faulty updating of the context buffer that drives appropriate approach behavior. In sum, the signal inherent in win and stay or lose and shift is only part of the system. Obviously, this model will have to be tested using fMRI.

Findings of this study have implications for clinical practice. Structured behavioral learning therapies comprise the majority of empirically supported autism treatments because persons with ASDs learn many things under the right conditions. In fact, best practices intervention and teaching methods for children and adults with autism stress breaking tasks into small units, using highly structured teaching methods, and providing reliable reinforcement [National Research Council, 2001]. Our findings of intact rote learning provide indirect support for the assertion that simple stimulus response and habit learning is intact in these individuals. Our findings also suggest that when provision of positive feedback is not strong or consistent or is rapidly changing, individuals with ASDs will be slower to acquire new skills.

Paradoxical findings that individuals with ASDs were better able to extract the faint “signal” of the EF pair are reminiscent of “stimulus overselectivity” [Koegel & Lovaas, 1978; Lovaas, Koegel, & Schreibman, 1979], whereby persons with autism are thought to learn through the use of somewhat idiosyncratic cues. Thus, it is essential to make sure that individuals with ASDs are aware of task cues and remain focused on true feedback while receiving instruction. It also has been suggested that in cases of paradoxical functional facilitation, one way to normalize performance is to inhibit the facilitated function to induce plasticity in the deficient one [Fecteau, Pascual-Leone, & Théoret, 2006; Kapur, 1996]. This presents an interesting potential avenue for future treatment research. Some of our own intervention work that involves training children to play while prohibiting them from engaging in their special interests represents an example of how this approach can be implemented successfully [Solomon, Ono, Timmer, & Goodlin-Jones, 2008].

This current study has several limitations. First, in order to improve the homogeneity of our sample and to avoid confounds associated with the use of medications that act on the DA system, we recruited only individuals not taking anti-psychotics. This may limit the generalizability of our results, as does the decision to recruit only persons with cognitive abilities in the average range or above. The sample also included four persons taking SSRIs, and consistent with current guidelines [Carter, Heckers, Nichols, Pine, & Strother, 2008] we believed it was not ethical to exclude them. Although we recruited only individuals without co-morbid diagnoses of attention problems, anxiety, or depression, several participants manifested these symptoms on questionnaires. We completed all analyses excluding these individuals, as well as those taking SSRIs. While reducing the power of the study to find statistically significant results, the pattern of results remained identical even without individuals with clinically significant attention symptoms and/or depression, and without subjects taking SSRIs. A third limitation of the study is that the computational model within which we embedded this study is not developmental. Given that ASDs are neurodevelopmental disorders, and that mature adult functioning may not be achieved by the early 20s for these individuals, additional insights likely could be obtained through investigation of developmental interactions between the basal ganglia, hippocampus, OFC, and PFC as has been considered by others [Bachevalier & Loveland, 2006; Ernst & Fudge, 2009]. Fourth, although we attempted to control for response bias through careful randomization that took into account presentation order, side of stimulus presentation, and side on which valid and invalid feedback was presented; through ensuring that the practice block was the same for all participants and did not contain items used in the task; and by checking for obvious perserverative responding, we cannot be sure whether unidentified and subtle biases were present. Finally, results of this work would have been strengthened if we had collected other measures thought to tap the functioning of the basal ganglia and the OFC that could have provided additional convergent validity for interpretation of our findings.

The mechanistic neurobiological and computational models used to interpret our findings provide a biologically plausible interpretive framework; however, this interpretation awaits direct testing. It bears mention that findings from one of the few existing fMRI studies of social and non-social probabilistic reinforcement learning, a relatively small study in preadolescents and adolescents, were not entirely consistent with our predictions about the relationship between fronto-striatal functioning and implicit learning [Scott-Van Zeeland et al., 2010], although this study focused on ventral as opposed to dorsal striatum, included patients on medications, and used a different task with reward probabilities of 0%, 50%, and 100%. A natural future direction for our work would be to use fMRI to the model we propose against others, as well as to examine development during the transition from adolescence to adulthood. Such an investigation holds the promise to shed light on the relative roles of the OFC, PFC, and basal ganglia in early and later learning in individuals with typical development and ASDs as well as the pathophysiology of group differences in the representation of positive feedback.