Reinforcement Learning in Young Adults with Developmental Language Impairment

Abstract

The aim of the study was to examine reinforcement learning (RL) in young adults with developmental language impairment (DLI) within the context of a neurocomputational model of the basal ganglia-dopamine system (Frank et al., 2004). Two groups of young adults, one with DLI and the other without, were recruited. A probabilistic selection task was used to assess how participants implicitly extracted reinforcement history from the environment based on probabilistic positive/negative feedback. The findings showed impaired RL in individuals with DLI, indicating an altered gating function of the striatum in testing. However, they exploited similar learning strategies as comparison participants at the beginning of training, reflecting relatively intact functions of the prefrontal cortex to rapidly update reinforcement information. Within the context of Frank’s model, these results can be interpreted as evidence for alterations in the basal ganglia of individuals with DLI.

1. Introduction

1.1. Linguistic and Non-Linguistic Deficits in Developmental Language Impairment (DLI)

Individuals with DLI represent a heterogeneous group of people who have substantial difficulty acquiring language despite normal hearing, normal non-verbal intelligence, appropriate social functioning, and no obvious signs of brain injury (Bishop, 1997). One of the defining characteristics of DLI¹ is the impaired acquisition of rule- or pattern-based components in language, such as morphology, syntax, and some aspects of phonology (Leonard, 1997). This commonly used definition leads to early hypotheses regarding the etiology of DLI that an impaired language-specific learning mechanism underlies language development and disorders (Clahsen, 1989; Rice, Wexler, & Cleave, 1995; van der Lely, 2005).

However, recent studies have shown that deficits in individuals with DLI are not limited to language but include general cognitive functioning, such as phonological working memory (Archibald & Gathercole, 2006), long-term memory (Lum, Conti-Ramsden, Page, & Ullman, in press; Tomblin, Mainela-Arnold, & Zhang, 2007), speed of processing (Miller, Kail, Leonard, & Tomblin, 2001), music processing (Jentschke, Koelsch, Sallat, & Friederici, 2008), and statistical learning (Evans, Saffran, & Robe-Torres, 2009). These findings indicate that poor language learning in general and morphosyntax in particular shown in individuals with DLI may be a manifestation of impaired domain-general cognitive mechanisms that go beyond the language system. As noted above, some of the studies have pointed to a domain-general impairment in statistical learning (e.g., Evans et al., 2009). In this research tradition, neither the nature of the learning system nor the underlying neural mechanisms have been clearly elucidated. The current study expands the line of inquiry concerning statistical learning deficits in DLI to consider reinforcement learning (RL) as a potential learning mechanism supporting language development, due to its reliance on the corticostriatal loops, the basal ganglia in particular.

1.2. A Neurocognitive Approach to the Understanding of DLI

Procedural learning has been suggested to be important for language learning (Gupta & Dell, 1999; Gupta & Cohen, 2002; Nicolson & Fawcett, 2007; Ullman, 2001, 2004). More recently, Ullman & Pierpont (2005) proposed the Procedural Deficit Hypothesis to explain a wide array of behavioral and neurophysiological findings for individuals with DLI. According to this hypothesis, a fundamental, but not exclusive, cause of DLI, particularly the grammatical deficits, can be attributed to impaired or less efficient procedural memory. Procedural memory is mediated, at least in part, by the corticostriatal loops connecting the basal ganglia with the cerebral cortex (Alexander, & Crutcher, 1990; Alexander et al., 1986; Eichenbaum & Cohen, 2001; Gabrieli, 1998; Seger, 2006). The basal ganglia play a particularly important role in this system for two reasons. First, anatomically, the basal ganglia connect to almost all regions of the cortex, and therefore are in an ideal position to influence a wide range of behaviors mediated by the corticostriatal loops. Second, it is becoming well understood that the basal ganglia, especially the striatum, not only play an important role in motor function, but they also support a broad array of incremental and implicit cognitive learning, most notably procedural learning (Poldrack et al., 2005; Seger, 2006, 2009; Shohamy et al., 2005, 2007) and RL (Doya, 1999; Frank, Seeberger, & O’Reilly, 2004; O’Doherty, Dayan, Friston, Critchley, & Dolan, 2003; Niv, 2009).

In the literature, several studies have shown poor procedural learning in individuals with DLI (Kemedy & Lukacs, 2009; Hedenius et al., 2011; Lee & Tomblin, submitted; Lum et al., in press; Lum, Gelgec, & Conti-Ramsden, 2010; Tomblin et al., 2007; but see Gabriel et al., 2011). These findings provide empirical support for the Procedural Deficit Hypothesis. Because RL also shares the same basal ganglia system as procedural learning, we might predict that performance on RL would also be relatively poorer in individuals with DLI than in comparison participants; however, no studies have examined RL in DLI. Therefore, in the current study, we used an RL paradigm developed by Frank et al. (2004) to further test the Procedural Deficit Hypothesis by examining how individuals with DLI implicitly extract reinforcement history from the environment by trial and error. We believe that the findings can shed light on the neurocognitive underpinnings of DLI, and by extension the role of RL and the basal ganglia in language development in general.

1.3. RL in a Nutshell: The Past and the Present

The beginning of RL research originated from the area of artificial intelligence and machine learning that deals with how an agent (e.g., a robot) learns to make decisions through an incremental, trial-and-error process (Sutton & Barton, 1998). The ultimate goal of RL is to maximize the likelihood of rewards while minimizing the occurrence of punishments without explicit instructions in the learning process (c.f., supervised learning). In the early 1960s when the contemporary study of language development began, RL was rejected as a plausible mechanism for language learning (Chomsky, 1959). This state of affairs has remained so, despite considerable reconsideration of Chomsky’s original argument.

During the last 30 years, the reinforcement theory has been substantially revised. One of the key features of this advancement has been the discovery that the striatal dopaminergic systems play a role in calculating probability of future reward and punishment that come from an agent’s current actions in an environment (e.g., Dayan & Niv, 2008; Frank, Moustafa, Haughey, Curran, & Hutchison, 2007a; Pizzagalli et al., 2008). Researchers found that phasic changes in dopamine levels are based upon prior experiences of interacting with the environment. On the one hand, phasic dopamine increase (a.k.a. dopamine burst) primarily results from positive reinforcing experiences and occurrence of unanticipated rewards, which strengthens synaptic plasticity in D1 dopamine receptors in the “Go” neural pathway and therefore supports learning of the behavior. On the other hand, phasic dopamine decrease (a.k.a. dopamine dip) is the result of negative reinforcing experiences and omission of an expected or predicted reward, which strengthens synaptic plasticity in D2 dopamine receptors in the “NoGo” neural pathway and therefore leads to avoidance of this behavior in the future. All of the reinforcement-related values converge to the substantia nigra (SN), the dopamine synthesizing region within the basal ganglia. As a result, while the cortical regions are responsible for information processing and representation storage, the basal ganglia select, via thalamic pathways, which of the numerous possible representations in the cortex are appropriate to execute under different circumstances (Frank, Loughry, & O’Reilly, 2001; Redgrave, Prescott, & Gurney, 1999).

Recent research in the role of the dopaminergic system in RL invokes interest among cognitive scientists for at least two reasons. First, RL algorithms can be instantiated in biologically plausible mechanisms, and therefore provide a direct means to uncover important insights on human decision-making behaviors (Dayan & Niv, 2008). Second, findings regarding the role of dopamine in RL bring about clinical implications in both degenerative neurological disorders (e.g., Parkinson’s disease and Huntington’s disease) and developmental disorders (e.g., attention deficit/hyperactivity disorder (ADHD) and autism spectrum disorder (ASD)) (Bradshaw, 2001). In addition, understanding the nature of these disorders helps reconstruct the path from the brain to individual differences in behaviors. The corticostriatal loops, in particular the striatum of the basal ganglia, are one of the primary recipients of dopaminergic projections, and therefore is considered as an obvious candidate neural system for RL (Niv, 2009).

1.4. The Probabilistic Selection Task: An RL Paradigm Based on a Biologically Grounded Computational Model

Recently, Frank and his colleagues developed a neurocomputational model of the basal ganglia-dopamine system, which was built upon a large body of earlier theoretical work on the role of dopamine in RL (see Cohen & Frank, 2009, for a review). According to their model, the basal ganglia are conceptualized as a gating system, which strengthens the pattern of neural firing in the frontal cortex that is related to appropriate actions, while suppressing those that are less appropriate. The dopaminergic system, the D1 and D2 dopamine receptors to be more specific, regulate the gating function of the basal ganglia (Wickens & Arbuthnott, 2010). Both types of dopamine receptors are highly concentrated in the caudate nucleus and the putamen, while D1 receptors have much higher density in the prefrontal cortex (PFC) than D2 receptors (Hall et al., 1994; Meador-Woodruff, 1994). This dopaminergic modulation of cortical input to the basal ganglia is conceived to be the primary mechanism of RL: it leads to correct feedback or reward by sending positive learning signals via D1 dopamine receptors, while simultaneously reducing the probability of incorrect or non-rewarding behaviors by sending negative learning signals via D2 dopamine receptors (Frank, 2005, 2011; Frank et al., 2004; Frank et al., 2007a; Frank, Santamaria, O’Reilly, & Willcutt, 2007b).

To further test this biologically plausible model, Frank et al. (2004) designed an RL paradigm, and made several verifiable predictions with respect to human behavioral responses during RL. The RL paradigm is a probabilistic selection task. During the acquisition phase of the task, participants learn to choose the most frequently reinforced stimulus from each training pair (e.g., choosing A from the AB pair) based on either positive or negative feedback on their decisions. It should be noted that the feedback is probabilistic: the feedback received is not always the same for each choice, and therefore it is impossible to always make the right decision. During the acquisition phase, all participants are expected to learn to choose Stimulus A over B, given that Stimulus A is the most frequently rewarded symbol whereas Stimulus B is the least. However, learning to choose A over B can be achieved by learning that 1) Stimulus A leads to positive feedback, 2) Stimulus B leads to negative feedback, or 3) both. Therefore, to distinguish the choose-A and avoid-B learning strategies, a test phase follows immediately. During the test phase, participants are presented with the original stimuli in novel pairings in order to evaluate whether participants have a bias for choosing more reinforced stimuli and/or for avoiding less reinforced stimuli. This sheds light on individual differences in basal ganglia dopamine function (i.e., preference over choose-A in novel pairs via D1 receptors and/or avoid-B in novel pairs via D2 receptors).

Based on the simulations by Frank and his colleagues (Frank, 2005; Frank & Claus, 2006), performance during the early acquisition phase of the task mainly relies upon the PFC and in particular the orbitofrontal cortex (OFC), which represents and integrates feedback information online in order to rapidly update probabilistic reinforcement contingencies in the brain. In contrast, performance during the test phase of the probabilistic selection task reflects the gating function of the basal ganglia modulated by the dopaminergic system, which selects the stimulus with the highest reinforcement history and/or avoids the one with the lowest reinforcement history.

The probabilistic selection task was designed after a biologically grounded computational model, and it has been used with a variety of patient groups and healthy individuals to examine both frontal and striatal processes in RL (e.g., Chase et al., 2009; Endrass et al., 2011; Frank et al., 2004; Frank et al., 2007a, 2007b; Simon, Howard, & Howard, 2010; Solomon et al., 2011; Waltz, Frank, Robinson, & Gold, 2007). Therefore, data from this task allow for interpretations with respect to a well-specified model of RL that can provide insight into component mechanisms.

1.5. The Current Study

In the current study, we adopted the experimental methods of Frank et al. (2004) to investigate probabilistic RL in young adults with DLI. According to what we know so far, the current study is the first study to examine how individuals with DLI perform on RL. If the Procedural Deficit Hypothesis were to be supported, we would expect to see poor RL in individuals with DLI, and the impaired performance should be more pronounced in the test phase than in the early acquisition phase of the probabilistic selection task due to the greater involvement of the basal ganglia. We posed three questions based on Frank et al.’s neurocomputational framework. First, do individuals with DLI demonstrate poor overall learning of the original training pairs, which is reflective of impairments in the D1 and/or D2 dopamine receptors (i.e., inefficient in sending positive and/or negative learning signals during the acquisition phase)? Second, do individuals with DLI demonstrate impaired RL in the test phase and thus, provide evidence of abnormal gating functions of the basal ganglia? Third, do individuals with DLI demonstrate equivalent learning during the early acquisition phase of the task, and thus providing evidence of relatively intact functions of the PFC?

2. Methods

2.1 Participants

Two groups of young adults, one with DLI and the other without, were drawn from the Midwest Collaboration of Specific Language Impairment. These participants were originally assessed in kindergarten and validated as having either normal language development or DLI by using the diagnostic standards and measurement tools (Tomblin, Records, & Zhang, 1996), and they do not have any reported history of ADHD or ASD. At the point of being tested, they were within the age range of 19 to 25 years.

For the current study, screening questions were asked to make sure that potential participants had not sustained brain injuries since the last time they had participated in the longitudinal study. In addition, two performance IQ measures and three language tasks were used to assess their current nonverbal IQ and language skills respectively. The two nonverbal IQ measures included the Block Design and Matrix Reasoning subtests from Wechsler Abbreviated Scale of Intelligence (WASI, Wechsler, 1999). The three language tasks were: 1) Word Derivations, a subtest from The Test of Adolescent and Adult Language, Fourth Edition (TOAL-4; Hammill, Brown, Larsen, & Wiederholt, 2007) to assess knowledge of derivational morphology, 2) Peabody Picture Vocabulary Test, Fourth Edition (PPVT-4; Dunn & Dunn, 2007) to assess receptive vocabulary, and 3) a modified version of the Token Test (de Renzi & Faglioni, 1978; Morice & McNicol, 1985) to assess sentence comprehension. Individuals whose language composite scores were at least 1.5 standard deviations (SD) below the mean were considered as having DLI; otherwise, they were assigned to the comparison group. If there is a contradiction between the current diagnosis and the previous ones, the authors discussed about the possibility of reassignment based on participants’ performance along multiple time points in the longitudinal database. The decisions were made based on mutual consensus before the two authors looked into task scores. All participants were compensated for their time. Table 1 summarizes the current demographic information and measure scores for the participants.

Table 1.

Age, language scores, and nonverbal IQ scores for the two groups of participants.

	DLI Group (n = 25)		Comparison Group (n = 23)
	M	(SD)	M	(SD)	p
Age (months)	265.64	15.73	266.78	6.48	n.s.
PPVT-4	84.40	7.27	101.09	13.24	<.001
Token Test	52.32	32.55	101.30	13.86	<.001
Word Derivations	72.00	7.91	93.48	13.27	<.001
Nonverbal IQ	89.68	12.41	110.52	9.89	<.001

Note. Test scores were reported in standard scores with a mean of 100 and SD of 15. These scores were converted from z scores based on local norms (the Token Test) or national norms (PPVT-4, Word Derivations, Nonverbal IQ). The nonverbal IQ composite was based on two subtests of the WASI: Block Design and Matrix Reasoning.

It should be noted that participants with DLI were not required to have nonverbal IQ levels above 85 (i.e., the traditional diagnostic cutoff for SLI) for two theoretical reasons. First, longitudinal studies showed that the patterns of language deficits were not different between language-impaired children with nonverbal IQ above and below 85 (Tomblin & Zhang, 1999). Second, researchers found a significant fall of over 20 nonverbal IQ points from kindergarten to adolescence in individuals with SLI, challenging the discrepancy-based approach to SLI research (Botting, 2005; Rice et al., 2004). Therefore, we believe that matching IQ scores will end up creating unrepresentative groups: either the language-impaired group will have higher nonverbal IQs than the population with DLI, or the healthy comparison group will have nonverbal IQs below normative expectations (Dennis et al., 2009). Further analyses regarding the association between RL and nonverbal IQ differences between the two groups were shown in the Result section (see Section 4.4).

2.2 Materials

The probabilistic selection task, the instructions, the procedure, and the stimuli were created by and taken from Frank et al. (2004, 2007b). More details of the task could be found in the two papers cited above. To summarize, the task protocol contained one acquisition phase and one test phase (see Figure 1). In the acquisition phase, three training pairs (AB, CD, EF) were created by combining six different visual stimuli chosen among Japanese Hiragana characters. In each case, one member of the pair would have a higher probability of reinforcement than the other. The three training pairs were randomly presented one at a time, and participants had to choose one of the two stimuli in a pair during the acquisition phase. Immediate feedback followed participants’ choice to indicate whether it was correct or incorrect; however, the feedback was probabilistic. For example, choosing stimulus A leads to positive feedback in 80% of AB trials, whereas choosing stimulus B leads to positive feedback in 20% of AB trials only. The probability of reinforcement contingencies was differently assigned to each pair: Stimulus A was correct in 80% of AB trials, stimulus C in 70% of CD trials, and stimulus E in 60% of EF trials. Like Frank and his colleagues, we used the 60% criterion for the training pair EF to establish a baseline for learning, and to ensure that participants learn stimulus A over B more reliably due to the high probabilistic structure of the AB training pair. Because of the different reinforcement contingencies, participants were expected to choose stimulus A, C, E over stimulus B, D, F at the end of the acquisition phase.

Figure 1.

The acquisition phase and the test phase in the probabilistic selection task. The number in parentheses was the probability of positive feedback given for each response choice (After Frank et al., 2007b).

The acquisition phase included a maximum of six training blocks comprised of sixty trials each (i.e., 20 trials per training pair). The extent of training was determined by the participant’s performance during the training. To avoid over-learning of training pairs so as to minimize developing explicit knowledge at the time of test, performance criteria were set respectively: 65% correct in the AB training pair, 60% in CD, and 50% in EF. The different performance criteria were due to different reinforcement contingencies of the three training pairs. The 50% criterion for the EF training pair was quite liberal because stimulus E was only correct 60% of the time during the acquisition phase. Therefore, it was only used to ensure that participants did not have a bias toward stimulus F. Performance was evaluated after each training block of 60 trials, and participants could move to the test phase if all these criteria were met, or after six blocks of training (i.e., 360 trials).

In the test phase, participants were presented with 1) the same training pairs (AB CD EF), and 2) the original stimuli in novel pairings involving either stimulus A (AC AD AE AF) or stimulus B (BC BD BE BF). Performance on the former was used to assess learning of training pairs in the acquisition phase, whereas performance on the latter was used to evaluate the influence of reinforcing feedback (i.e., positive and negative feedback) on novel pairings. It should be noted that stimulus A was the most frequently reinforced stimulus whereas stimulus B was the least frequently reinforced one during the acquisition phase.

Each test pair was presented six times during the test phase (i.e., 66 test trials in total), and no feedback was provided. The primary goal of the test phase was to assess whether participants had a bias for positive RL (i.e., consistently choosing stimulus A in novel test pairs wherein it was present) or for negative RL (i.e., consistently choosing stimulus B in novel test pairs wherein it was present).

3. Procedure

Participants sat in front of a laptop screen, on which the visual paired stimuli were presented in black on a white background. They were instructed to press keys (i.e., either the f or j key on the keyboard) to indicate which of the paired stimuli they thought was correct. Visual feedback was immediately provided after participants’ choice, and it lasted for 1.5 seconds. If the computer did not detect responses from participants, the words No Response Detected printed in red would appear on the center of the screen. Participants could take a brief break between two consecutive blocks if needed. The task lasted approximately 15 to 25 minutes, depending on the number of blocks participants went through in the acquisition phase.

4. Results

4.1. Acquisition of Training Pairs

First, we examined whether learning of training pairs during the acquisition phase was significantly different between the two groups. Two primary statistical analyses were performed: 1) an independent t-test to compare the number of training blocks that the individuals in each group required before moving to the test phase, and 2) a two-way ANOVA to test for group differences in performance on each training pair (i.e., AB CD EF) in the test phase.

The independent t-test revealed that participants with DLI required significantly more training blocks (M = 5.48, SD = 1.48, range: 1-6) than comparison participants (M = 3.83, SD = 2.12, range: 1-6) before moving to the test phase, t(46) = 3.15, p = .003. There was no significant difference in “No Response Detected” trials between the DLI group (M = 1.56, SD = 2.80, range: 0-11), and the comparison group (M = .70, SD = 1.11, range: 0-4), t(46) = 1.38, p = .17.

To examine whether the DLI group and the comparison group were significantly different in discriminating among the three training pairs with disparate reinforcement contingencies, we conducted a 3 (Training Pair: AB, CD, EF) × 2 (Group: DLI, Normal) mixed-design ANOVA. The results showed a significant main effect of Training Pair, F(2, 92) = 9.02, p < .001, η_p² = .16, and of Group, F(1, 46) = 5.50, p = .02, η_p² = .11. The interaction effect was non-significant, F(2, 92) = .95, p = .39, η_p² = .02 (see Figure 2). These results indicate that participants with DLI had poor overall learning but a similar level of sensitivity to different reinforcement contingencies when compared with the normal participants.

Figure 2.

Performance on the three training pairs (AB, CD, EF) in the test phase. Data were presented as the mean with one standard error.

4.2. Performance on the Test Phase

In the test phase, positive RL was reflected in the percentage of responses in choosing stimulus A in all novel pairings in which it was present (i.e., choose-A test trials), whereas negative RL was measured by the percentage of responses in avoiding stimulus B (i.e., the alternative in a pair was selected) in all novel pairings in which it was present (i.e., avoid-B test trials). Given that the basal ganglia select actions based on reinforcement history (i.e., to facilitate or avoid certain actions for the current state based on past reinforcing experiences), the degree to which participants chose stimulus A or avoided stimulus B is viewed as a reflection of the gating function of the basal ganglia via the dopaminergic system (i.e., the D1 and D2 dopamine receptors) on the PFC.

Given that the performance data were proportional, an arcsine transformation was applied to normalize the distribution for analysis (Kirk, 1968). A 2 (Learning Strategy: Choose-A, Avoid-B) × 2 (Group: DLI, Normal) ANOVA revealed a significant main effect of Learning Strategy, F(1, 46) = 19.21, p < .001, η_p² = .18, with the percentage of positive RL (M = 76.61, SD = 20.37) significantly higher than that of negative RL (M = 59.23, SD = 19.35). There was a significant main effect of Group, F(1, 46) = 10.11, p = .002, η_p² = .11, showing that the DLI group was globally impaired at RL. The interaction between Group and Learning Strategy did not reach significance, F(1, 46) = .04, p = .85, η_p² < .001. Figure 3 illustrates these results in box plots with percent accuracy (%) as the dependent variable instead of arcsine transformed values for easy data interpretation.

Figure 3.

Box plots of group performance on positive RL (i.e., choose A in novel pairs) and negative RL (i.e., avoid B in novel pairs) in the test phase. Performance on the AB training pair in the test phase was also included for comparison.

In addition, one-sample t-tests were performed to examine whether the two groups showed positive and negative RL that exceeded chance. Comparison participants showed above-chance positive RL, t(22) = 7.56, p < .001, as well as negative RL, t(22) = 3.75, p < .001. In contrast, participants with DLI exhibited above-chance positive RL, t(24) = 5.62, p < .001, but non-significant negative RL, t(24) = 1.01, p = .33.

It should be noted that all analyses were carried out with and without participants who failed to meet the performance criterion on the AB training pair (i.e., 65%). However, given that the number of excluded participants did not differ between the DLI group (n = 7) and the comparison group (n = 2), χ(1, N = 48) = 2.93, p = .09, and the pattern of arcsine transformed results remained unchanged after subject exclusion, the findings reported above were generated based on the full data set.

Thus, these results showed generalized deficits in performance of participants with DLI in the test phase that probed whether the two groups differed with respect to a dependence on learning from the most rewarded stimulus (i.e., Choose A) or learning from the least rewarded stimulus (i.e., Avoid B). Both groups showed a greater effect of the Choose-A strategy over the Avoid-B; however, the DLI group showed lower levels of learning from both.

4.3. Impact of Reinforcing Feedback on Rapid Early Acquisition

According to Frank’s framework, RL involves both the PFC and the basal ganglia within the corticostriatal loops. The former is responsible for representing, integrating, and updating feedback information online during the early phase of the reinforcement learning process, whereas the latter modulates the selection of actions considered in the PFC and therefore serves as a gating mechanism at the end of the learning process (Frank & Claus, 2006). To further evaluate whether the PFC is a possible source of impaired RL in individuals with DLI, we computed “win-stay” and “lose-shift” scores for each trial in the first block of the acquisition phase. These scores were used to reflect participants’ ability to update reinforcement values online during early acquisition of training pairs, and were therefore considered to rely upon the PFC primarily (e.g., Waltz et al., 2007).

The win-stay scores were defined by computing the proportion of repeated stimulus selections in a given condition that followed reinforced choices, whereas the lose-shift scores were defined by computing the proportion of switched stimulus selections in a given condition that followed non-reinforced choices. Independent t-tests revealed that following positive feedback, participants with DLI were as likely as comparison participants to choose the same rewarded stimulus in the next trial where it appeared (i.e., win-stay condition), t(46) = 1.69, p = .10, and also were equivalent in switching choices in the subsequent trial of the same type after receiving negative feedback (i.e., lose-shift condition), t(46) = .509, p = .61 (see Figure 4).

Figure 4.

Different response types to positive versus negative feedback during the first block of the acquisition phase. Data were presented as the mean with one standard error.

These results indicated that the DLI group and the comparison group used similar PFC-dependent learning strategies to integrate reinforcement contingencies from the environment during the first block of the acquisition phase.

4.4. Relationship among RL, Language, and Nonverbal IQ

The design of the current study differentiated the RL performance in two groups that contrasted on their language ability. The evidence above provides preliminary support for an association between RL and language. However, as shown in Table 1, the two groups were also significantly different in nonverbal intelligence, despite that both nonverbal IQ scores were within the normal range. The average effect size across the language measures for the two groups was 1.94 whereas the effect size contrasting the two groups for nonverbal IQ was 1.39. Thus, although the two groups contrasted more on language than on nonverbal IQ, it cannot be said that language was the sole basis for the group difference in RL. Indeed, the theoretical approach we are taking stipulates that a general-purpose learning and memory system contributes to individual differences in language, but is not unique to language. Furthermore, we do not assume that language abilities are completely isolated from nonlinguistic cognitive abilities (see also Baldo et al., 2005; Botting, 2005; Dethorne & Watkins, 2006; Swisher, Plante, & Lowell, 1994). Indeed, in the current study, the correlation between nonverbal IQ and language composite scores among our participants was r = .71, p < .001.

Although we are not claiming that there is a specific and largely encapsulated effect of RL on language, it is important to determine whether the association of RL with group differences could be due solely to the nonverbal IQ differences between the two groups. The interactions among RL, language, and nonverbal IQ are complex and difficult to tease apart. By using mediation analysis (Baron & Kenny, 1986; MacKinnon, Lockwood, Hoffman, West, & Sheets, 2002), we can ask whether the association between RL and nonverbal IQ requires language as a mediator. For this analysis, the independent variable of RL was defined as the average percent accuracy of positive and negative probabilistic selection performance during the test phase. The dependent variable of Nonverbal IQ was the standard scores derived from the nonverbal subtests (i.e., Block Design and Matrix Reasoning) of the WASI. The mediating variable of Language represented average standard scores of PPVT-4, Word Derivation, and the Token Test. Results from the simple mediation analysis showed that RL was significantly associated with Language, β = .37, p = .03, and Language was significantly associated with Nonverbal IQ, β = .57, p < .001. The association between RL and Nonverbal IQ was marginally significant, β = .27, p = .05. However, the relation between RL and Nonverbal IQ was no longer significant after the effect of Language was controlled, β = .07, p = .54. The summary of the mediation analysis was depicted in Figure 5. These data support the notion that the association between RL and nonverbal IQ can hardly exist without the mediation of language, and more importantly, that the group effects in the current study are unlikely to only reflect the differences in the nonverbal general cognitive abilities of the participants.

Figure 5.

Summary of mediation analysis. The number in parenthesis represented the relation between the independent variable (IV) and the dependent variable (DV) before controlling for the mediator. IV: Reinforcement Learning, DV: Nonverbal IQ, Mediator: Language.

5. Discussion

5.1. Summary and Implication of Current Findings

In the Introduction, we posed three questions based on Frank et al.’s neurocomputational framework. By using the probabilistic selection task, the current study sought evidence that RL was associated with language ability, and therefore could be a viable mechanism, among others, that supports language development. In this section, we summarized the findings for each question, and provided plausible interpretations drawn on the literature of the striatal and frontal processes in RL. It should be noted that all the data interpretations were within the context of the predominant accounts that have been well supported in the literature. We certainly acknowledge that the existing theoretical models of the brain bases of RL remain open to debate. However, resolving this problem is beyond the scope of the current study.

5.1.1. Do individuals with DLI demonstrate poor overall learning of the original training pairs, which is reflective of impairments in the D1 and/or D2 dopamine receptors (i.e., inefficient in sending positive and/or negative learning signals during the acquisition phase)?

To answer this question, we analyzed 1) the number of training blocks participants required before moving to the test phase, which reflects participants’ overall learning ability, and 2) between-group performance on the three training pairs (i.e., AB CD EF) in the test phase, which reflects the learners’ sensitivity to different reinforcement values. The findings showed that compared to normal participants, participants with DLI needed a greater amount of training blocks before moving to the test phase. Nevertheless, they still demonstrated depressed overall learning of training pairs, despite of a relatively similar level of sensitivity to reinforcement values with comparison participants. These results indicate an abnormality of the striatal dopaminergic system in individuals with DLI.

A person’s prior experiences of interacting with the environment may change the phasic levels of dopamine via 1) D1 striatal dopamine receptors, which leads to positive RL, as well as 2) D2 striatal dopamine receptors, which leads to negative RL (e.g., Cohen & Frank, 2009; Wickens & Arbuthnott, 2010). In other words, the phasic changes of dopamine in the basal ganglia serve as a critical neural mechanism for learning, so that some actions can be reinforced whereas others avoided or suppressed. The current findings of poor overall learning during the acquisition phase in the DLI group suggest reduced phasic changes in dopamine signals in the basal ganglia. That is, learning signals sent via the D1 and/or D2 dopaminergic systems are reduced in participants with DLI, and therefore, the weakened synaptic connections in the basal ganglia may lead to poor learning from probabilistic positive/negative feedback.

5.1.2. Do individuals with DLI demonstrate impaired RL in the test phase and thus, provide evidence of abnormal gating functions of the basal ganglia?

To answer this question, we tested the novel combinations of AC, AD, AE, AF (i.e., novel pairs in the Choose-A condition) and BC, BD, BE, BF (i.e., novel pairs in the Avoid-B condition) to see if participants have a bias for choosing a more reinforced stimulus and/or for avoiding a less reinforced stimulus. This analysis provides information regarding the differential contribution of the “Go” (i.e., strengthening of synaptic plasticity in D1 dopamine receptors) versus “NoGo” (i.e., strengthening of synaptic plasticity in D2 dopamine receptors) neural pathways to the learning processes. That is, choosing A in all test pairs where it was present represents evidence of the “Go” learning bias, whereas avoiding B in all test pairs where it was present represents evidence of a “NoGo” learning bias.

The results showed that individuals with DLI were impaired at both positive and negative RL, indicating a poor gating function of the basal ganglia in individuals with DLI. Given that the basal ganglia carry out the gating function by requiring a normal range of dopamine bursts and dips to support “Go” and “NoGo” learning respectively, these findings, again, suggest that individuals with DLI have reduced phasic changes of dopamine signals in the basal ganglia. This performance pattern of our research participants is consistent with that of young adults with ADHD off medication (see Frank et al, 2007b, for details).

Our findings showed a significant positive learning bias in both groups, which is in contrast with previous studies reporting equivalent performance on positive and negative RL (e.g., Frank et al., 2004, 2007b, Solomon et al., 2011; but see Endrass et al., 2011; Simon et al., 2010). We do not have a definite explanation to this discrepancy. One of the possibilities may be, at least partially, attributed to a genetic effect on individual differences in RL. Frank et al. (2007a) found that although on average participants demonstrated equivalent choose-A and avoid-B performance, individual differences in RL occurred (i.e., some people are better at choosing positively reinforced stimuli, whereas others are better at avoiding negatively reinforced stimuli). According to Frank et al.’s genetic analyses, individual genes that control basal ganglia dopamine efficiency (e.g., DARPP-32) were able to predict the extent to which participants learned from positive or negative feedback. Therefore, it is possible that the participant sample in the current study is more homogeneous, and therefore is better reflective of the positive learner phenotype. Future research is necessary to examine possible sources of learning bias in our group sample.

5.1.3. Do individuals with DLI demonstrate equivalent learning during the early acquisition phase of the task, and thus provide evidence of relatively intact functions of the PFC?

To answer this question, we analyzed the “win-stay” and “lose-shift” performance of participants with and without DLI in the first block of the acquisition phase. The results are generally interpreted as reflective of the function of the PFC, and in particular the OFC (e.g., Endrass et al., 2011; Solomon et al., 2011; Waltz et al., 2007). The representations in the PFC can be rapidly updated, but they are also fragile and are overwritten quickly, which stands in contrast with the slowly built but stable representations in the basal ganglia.

The results showed similar patterns of “win-stay” and “lose-shift” performance between the two groups. In other words, individuals with DLI have a functionally compatible OFC system when compared to those without DLI, and thus this aspect of early learning in the probabilistic selection task can be seen as spared or perhaps unrelated to group differences in language learning.

5.2. Basal Ganglia and DLI

The possibility of basal ganglia abnormalities in DLI has been raised from different directions of research, including imaging studies (Hwang et al., 2006; Ors et al., 2005; Vargha-Khadem et al., 1998), genetic studies (Fisher & Scharff, 2009; Reimers-Kipping, Hevers, Paabo, & Enard, 2011), motor development studies (Bishop, 2002; Hill, 2001, 2010), and procedural learning studies (Kemedy & Lukacs, 2009; Hedenius et al., 2011; Lum et al., 2010, in press; Tomblin et al., 2007). Our current findings are in line with the recent trend of research, suggesting alterations in the corticostriatal loops, particularly the basal ganglia, of individuals with DLI, and therefore, provide further support for the Procedural Deficit Hypothesis. Because the basal ganglia have been implicated in a broad array of motor and cognitive functions, including movement planning, procedural memory, RL, and language processing (e.g., Conway & Pisoni, 2008; Frank, 2011; Seger, 2006, 2009; Shohamy et al., 2007), it is not surprising to observe heterogeneous profiles of non-linguistic deficits along with language difficulty in individuals with DLI.

While growing evidence showed a strong association between the basal ganglia and language, different perspectives are held on how the basal ganglia influence language processing. Some researchers adopt a domain-general view, proposing that the striatum indirectly influences language learning via general functioning processes, such as executive functioning (Lieberman, 2002; Longworth, Keenan, Barker, Marslen-Wilson, & Tyler, 2005), attention and intention (Crosson, 1992), or working memory (Grossman et al., 2000). Other researchers adopt a domain-specific view, arguing for a language-specific role for the basal ganglia (Teichmann et al., 2005, 2008). These views on the relationship between the basal ganglia and language mainly come from patient studies (e.g., Parkinson’s disease and Huntington’s disease). Although patient studies are helpful for understanding the basal ganglia mediation in language, findings should be interpreted with caution because degenerative disease processes tend to affect multiple brain systems, and are usually confounded with aging. Therefore, in place of disease processes, future research is suggested to investigate the role of the basal ganglia in people with developmental language disorders.

5.3. RL and Language

In the current study, we found a significant relationship between performance on RL and on language tasks. Such an association may be, in part, explained by way of a shared underlying mechanism, and we hypothesize that the corticostriatal loops, particularly the basal ganglia, play an important role in supporting RL as well as language acquisition.

RL and language acquisition seem very different on the surface. RL, on the one hand, is a fundamental learning process, by which animals and humans learn from trial and error to predict future events and act upon the environment without explicit instructions (Niv, 2009). On the other hand, language is unique to humans, comprising a small number of elements (e.g., grammatical inflections and words) combined together in a rule- or pattern-based manner (Saffran, 2003). Language acquisition emerges from complex interactions among biological and environmental factors, and most important, does not require continuous reinforcing stimulation from the environment (e.g., direct feedback for correct or incorrect responses from caregivers) (Marcus, 1993; Ramscar & Yarlett, 2007).

Despite no apparent similarities, these two learning processes share several commonalities, all of which involve the basal ganglia. First, both processes involve incremental learning of associations with continuous exposures to stimuli, such as a chess player learning to make a move that has better chance of winning, or a two-year-old acquiring his first fifty words. Second, neither of them requires conscious awareness during learning processes, and well-learned behaviors can be executed automatically. That is why people feel that they make decisions by “intuition” (Frank, O’Reilly, & Curran, 2006), or young children are able to acquire grammar easily without effort or formal teaching. Third, experiences in the past are important for both processes. In the reinforcement learning process, learners rely upon previous experiences about what actions in the past led to desired or undesired outcomes in order to make better decisions in the future (Sutton & Barton, 1998). Similarly, language learners take advantage of past experiences to modify their current use of language (e.g., Wells et al., 2009). Last, both RL and language acquisition involve anticipatory behaviors. Anticipation is defined as “a process, or behavior, that does not only depend on past and present but also on predictions, expectations, or beliefs about the future” (Butz, Sigaud, & Gerard, 2003, p.3). Anticipation is important for RL because agents (e.g., animals or humans) learn to predict the most desirable reward that they will receive in the future in order to make decision about actions in the current state. By comparison, certain aspects of language processing also involve anticipation (Swarup & Gasser, 2006). For example, Otten and Berkum (2008) showed that people tried to predict specific upcoming words based on discourse contents during reading. Misyak, Christiansen and Tomblin (2010) found that prediction-based processes play a critical role in individuals’ processing of complex sentences.

However, it remains difficult to explain the absence of direct reinforcement in the process of language acquisition (i.e., direct feedback for correct/incorrect responses on children’s language production). We propose that the reinforcement learning process not only occurs in the external environment (e.g., feedback from interactions with caregivers), but it can be internal within an individual. For example, a mismatch between a young child’s word production (e.g., /dæ/) and word comprehension (e.g., previous experiences about the word dad) can serve as a prediction error to drive language acquisition. This idea remains to be tested, and we hope that studying the role of RL in language can provide new insights into fundamental processes that affect individual differences in language, including DLI.

5.4. Performance of DLI, ADHD, and ASD on RL: What Do Commonalities and Differences Tell Us about the Nature of Underlying Mechanisms?

Traditional diagnostic frameworks tend to suggest a clear-cut distinction between ADHD and DLI, as well as between ASD and DLI. However, researchers have found that ADHD and ASD are often comorbid with DLI, and therefore raised a question whether a common or overlapping genetic or neurobiological mechanism underlies these neurodevelopmental disorders (Bishop, 2010; Nicolson & Fawcett, 2007; Tomblin, 2011; Williams et al., 2000). Recently, growing evidence suggests abnormal corticostriatal loops in individuals with ADHD (e.g., Durston, van Belle, & Zeeuw, 2010; Frank et al., 2007b), with ASD (e.g., Cheung et al., 2010; Solomon et al., 2011), and with DLI (e.g., Fisher & Scharff, 2009; Ullman & Pierpont, 2005); however, the commonalities and differences among these disorders are rarely discussed together within the context of the corticostriatal system in the brain.

The RL paradigm developed by Frank et al. (2004) provides a way to look at the question from a different perspective, because it is grounded in a biologically plausible computational model. Solomon et al. (2011) adopted this paradigm to study RL in adults with ASD. It was found that participants with ASD showed equivalent RL performance in the test phase; however, they revealed different learning strategies and a different pattern of sensitivity to reinforcement contingencies during early learning (i.e., the first block of the acquisition phase). Given that the PFC mediates rapid updating of reinforcement values in the corticostriatal loops, Solomon et al. (2011) suggested that individuals with ASD have deficits in the PFC, with relatively intact functions of the basal ganglia.

In contrast, Frank et al. (2007b) recruited a group of individuals with ADHD to test their computational model. They found impaired RL in participants with ADHD, and this impairment could be attributed to deficits in learning of the training pairs during the acquisition phase. These findings are interpreted as reduced phasic changes (i.e., bursts and/or dips) of dopamine levels in the basal ganglia in young adults in ADHD, which is in line with recent studies regarding dopamine as one of the core etiologies in ADHD. For example, Tripp and Wickens (2008) proposed the dopamine transfer deficit in ADHD, suggesting that the magnitude of dopamine signals received by people with ADHD might be diminished due to a disrupted time course of dopamine cell firing, and thus they are not able to predict or anticipate future rewards adequately.

With the exception of a general “Go” bias in the test phase, performance of our participants with DLI was quite similar to that of young adults with unmedicated ADHD in Frank et al. (2007b)’s study: both groups were less capable of learning from either positive or negative feedback. Given that phasic changes in dopamine levels can in part account for ADHD, this same mechanism may be also operative within individuals with poor language abilities, such as our participants with DLI. However, due to different methodologies (e.g., use of stimulant medications) and statistical analyses, it is hard to compare these different studies in a systematic way. ADHD, ASD, and DLI are all considered as developmental disorders; therefore, a possible avenue for future research is to examine the developmental role of the dopaminergic systems in the basal ganglia of people with ADHD, ASD, and DLI, which will shed light on the possible overlapping mechanism(s) underlying these neurodevelopmental disorders. In addition, we hope that by considering the complicated interactions between predisposing factors (e.g., abnormal structures or neurochemical processes in the brain) and precipitating factors (e.g., environmental influence) in the developmental process, we will have a better understanding of the high comorbidity of these disorders as well as how each disorder manifests itself in a unique way.

5.5. Future Work

The current study extends the Procedural Deficit Hypothesis by investigating RL in DLI. However, the corticostriatal loops, as well as the dopaminergic systems, involve a wide range of human behaviors, and there is no gold standard for testing each of these. Therefore, stronger tests of the Procedural Deficit Hypothesis will come from a combination of analysis at different levels, such as the behavioral level, the genetic level, or the cellular level, to examine the role of the corticostriatal loops in individual differences in language.

To further examine the etiology of DLI at the neural level, two research directions are proposed. First, imaging techniques are one of the most direct approaches to examine the basal ganglia in the human brain. In previous research, there were a few studies looking at abnormal structures and functions of the brain in individuals with DLI (Badcock et al., 2012; Friederici, 2006; Hugdahl et al., 2004; Hwang et al., 2006; Im et al., 2007; Ors et al., 2005); however, none of the imaging studies of DLI looked at the structures of the basal ganglia based on an established hypothesis. Therefore, further studies are needed to explore the role of the basal ganglia regions in individuals with DLI. Second, computational modeling is a promising way to link empirical evidence to theoretical explanations in terms of possible underlying mechanisms. The neurocomputational model used in the current study provides a biologically plausible way to interpret our behavioral findings. This interpretation awaits further investigation on direct model testing.

Highlights.

• Impaired reinforcement learning found in developmental language impairment (DLI)

• Intact early response strategies to positive and negative feedback in DLI

• Corticostriatal system likely to influence individual differences in language