A role for adaptive developmental plasticity in learning and decision making

Abstract

From both a medical and educational perspective, there is enormous value to understanding the environmental factors that sculpt learning and decision making. These questions are often approached from proximate levels of analysis, but may be further informed by the adaptive developmental plasticity framework used in evolutionary biology. The basic adaptive developmental plasticity framework posits that biological sensitive periods evolved to use information from the environment to sculpt emerging phenotypes. Here we lay out how we can apply this framework to learning and decision making in the mammalian brain and propose a working model in which dopamine neurons and their activity may serve to inform downstream circuits about environmental statistics. More widespread use of this evolutionary framework and its associated models can help inform and guide basic research and intervention science.

Imagine you are a young mammal on the day you disperse from the burrow in which you were born. You enter an unfamiliar world and must discover what you can eat and where it is safe. You must decide where you should explore and when to seek shelter. Consider next, there may be different challenges present in the environment into which you were born. What if you find yourself in a world that is very dangerous or in which food is scarce? How should your behavior differ from a situation in which food is plentiful and the world relatively safe or stable? In these two different environments, it is likely you should behave differently in order to survive. You ought to store information differently, weigh risks and benefits differently, and use different foraging strategies. In other words, you should learn and make decisions differently in these two different environments. Learning and decision making are core concerns of behavioral neuroscience, but we have rarely asked how developmental experiences and evolution prepare young mammals for these different scenarios.

In many organisms, multiple phenotypes can emerge from a single genotype. This flexibility in phenotypic expression creates opportunity for an organism to adapt to the environment in which it is born and matures. This ‘built in’ strategy called adaptive developmental plasticity is thought to have evolved in response to environmental fluctuation over millennia to confer fitness advantages [1–5]. Adaptive developmental plasticity is most easily observed when it involves changes in the physical appearance of an organism (for example its size or the development of external armor in daphnia [6], Figure 1) but it can also include properties of the nervous system and through the nervous system, behavioral patterns and tendencies [2,4]. This means that for many species, there are likely multiple developmental sensitive periods, such as prenatal, early postnatal, and adolescent periods, during which life experience can have a strong and long-lasting influence on phenotypic expression [4].

Figure 1. The adaptive developmental plasticity framework.

Animals with a single genotype may develop different phenotypes based upon their experiences during development. A. The freshwater crustacean Daphnia longicephala develops a defensive helmet when chemical predator cues are present in the juvenile environment but does not invest in this armor if the cues are not detected [6]. B. Here, we propose that cues in the juvenile environment may alter phenotypic expression in behavioral domains related to reward processing, learning, and decision making. In this example, a food scarce versus a food rich environment (food represented by yellow “cheese” triangles) during development may lead to differences in reward sensitivity, cognitive flexibility, and explore/exploit balance.

Using the adaptive developmental plasticity framework, we propose that cues which indicate a harsh or adverse environment may have strong effects on the development of systems that support learning and decision making in the mammalian brain (Figure 1). We propose that the effects of adaptive developmental plasticity may be particularly apparent in the dopamine system and its striatal and cortical targets. Basic aspects of these circuits have been conserved for 500 million years [7]. These systems support foraging, dispersal, decision making, and potentially play a role in mating and reproduction, making them targets of selective pressures [8–10]. Also, in the mammalian brain, these circuits show protracted development into adolescence and early adulthood. Late development of these circuits may be timed to incorporate information about the statistics and causal structure of the foraging or social environment (e.g. richness/scarcity and stability/volatility) into developmental programs that shape circuit function and behavior of the emerging adult.

The goal of this review is to lay out how the adaptive developmental plasticity framework may be applied to understand how the environment influences learning and decision making across the lifespan. To this end, we lay out how dopaminergic projections that target striatal and cortical regions are likely to support behavioral variations in learning and decision making. Then we propose a working model for how cumulative experience in response to differences in the environment may drive differences in circuit development. Finally, we briefly reference the growing literature that demonstrate these neurobiological and behavioral variables are sensitive to developmental experience.

What systems are likely to underlie developmentally sensitive phenotypic variations in learning and decision making?

Dopamine systems, basal ganglia, and cortico-striatal circuits together are known to play a critical role in learning and decision-making processes, such as integrating and making associations between sensory and reward information from the external world, assigning values to available options, and choosing the optimal option [8,9,11,12]. It is known that dopamine supports these processes through its effect on neurons in many brain regions. One of core regions is the striatum, which is the input structure of the basal ganglia and can be divided into ventral striatum (which includes the nucleus accumbens) and the medial and lateral dorsal striatum (also known as the caudate and putamen in the primate brain) [13].

The striatum receives incoming information from cortical and thalamic regions as well as dopaminergic projections from the ventral tegmental area (VTA) and substantia nigra (SN) [9,14] (Figure 2). Recording and manipulation data suggest the pattern of cortical and thalamic inputs to the two hemispheres of the striatum generate action selection in left/right orienting tasks [8,15] and also reaching tasks [16] in rodents and saccade tasks in non-human primates [17]. Inputs to the ventral striatum are thought to participate in outcome evaluation, reinforcement of past actions, motivation and salience [8,9,12,14,18].

Figure 2. Working model in which differences in cumulative patterns of dopamine signaling during development leads to changes in cortico-striatal circuit function and behavior.

Potential sites of adaptive developmental plasticity: 1) Changes in dopamine (DA) neuron activity and release may occur in response to environmental cues and experience. This could have downstream effects on 2) SPN intrinsic excitability; 3) pre- and postsynaptic D1R and D2R expression; and 4) cortical input weights and activity. The sum of these changes can influence the weight of positive and negative outcomes, explore versus exploit behavior and other variables contributing to learning and decision making through changes in D1R SPN and D2R SPN output.

The main neurons of the striatum are spiny projection neurons (SPNs). SPNs express two different major subtypes of dopamine receptors, D1 receptors (D1R) and D2 receptors (D2R). In the dorsal striatum, these two populations of SPNs form two distinct output pathways [19] (Figure 2) whereas in the ventral striatum there is more overlap [20]. The D1R expressing SPNs and D2R expressing SPNs of the striatum are thought to participate in action selection and outcome evaluation and represent the benefit and cost information of different choices, respectively, in learning and decision making [8,21–23]. Computational models suggest that changes in dopamine levels and/or D1R and D2R expression on SPNs can “tune” how activity in the striatal output pathways are integrated to mediate action selection and outcome evaluation [21] (Figure 2). Greater activity in D1R expressing SPNs may lead to greater weight given to benefits while greater activity in D2R expressing SPNs may lead to greater weight given to costs. This cost/benefit balance could be tipped by differences in the level of dopamine present in the striatum, D1R and D2R expression on SPNs, excitability of the SPNs, and differences in the activity and/or weights of inputs to these two SPN populations by incoming glutamatergic synapses [8,18,21,24].

Importantly, these interacting systems (dopamine axons, basal ganglia neurons, and cortico-striatal circuits) show protracted development into adolescence or early adulthood [25–28] (with some elements changing gradually and others more exhibiting more punctuated change). We propose that this protracted development may exist to enable adaptive developmental plasticity in learning and decision making.

The learning and decision making phenotypes resulting from adaptive developmental plasticity can be subdivided into multiple measures, each with its own range or spectrum of phenotypes. For example, offspring with different developmental experiences might differ in the weight they give to positive or negative feedback or the rate of incorporating this feedback when learning [29–32]. Also, when making choices, the rate at which individuals explore the environment or exploit what they know may differ based on past experience [30,33–35]. In another example relevant to addiction vulnerability, individuals with different experience may later respond to cues associated with rewards differently [36,37]. In some individuals, these cues may gain powerful incentive salience while in others with different developmental history these cues may remain more neutral. An individuals’ phenotype for each of these learning and decision making variables (updating in response to feedback, explore/exploit tradeoff and incentive salience) likely driven by different mechanisms, could sum to alter cognitive and affective behavior broadly. In humans, differences in learning and decision making phenotypes defined by these basic dimensions could affect educational performance, social interactions, mental health and even physical health across the lifespan.

A working model of the interactive development of the cortico-striatal circuit and its modulation by cumulative patterns of dopamine signaling

We propose a three-step working model in which cumulative differences in dopamine neuron activity in response to early life experience may shift the developmental trajectory of downstream striatal and cortical circuits.

1). The acute and long-lasting changes in dopamine release and function in response to fluctuations in feeding states, uncertainty, reward, and punishment during development may combine to drive cumulative downstream differences.

Dopamine neuron activity is known to be sensitive to feeding state, uncertainty in the environment, and shows phasic modulation in response to rewards, punishments, and prediction errors [12,38]. The cumulative activity profile of the dopamine system is therefore predicted to differ in terms of amplitude, temporal patterning, and total activity in scarce vs. rich environments as well as volatile vs. stable environments. This environmental variation could be driven by food cues and foraging outcomes as well as social interactions or predation threats.

It is likely that environmental variables affect the brain throughout the lifespan. However, environmental impacts on dopamine system and downstream circuits may be more dramatic during the juvenile and adolescent period due to the greater plasticity in these circuits during development. In studies of rodents, dopamine neurons show protracted outgrowth of axons during adolescence with innervation the prefrontal cortex continuing to grow until young adulthood [25,28,39–41]. Within this long period of development there maybe periods of punctuated change and possibly greater plasticity, but more research is needed to know when specific sensitive periods may occur. We do know for example, within the striatum the amount of dopamine available to be evoked by stimulation is low in the early juvenile period but increases with development [25,42, 43], and that the increase from the early juvenile period to late juvenile period is critical for down stream targets [43] (more on this below). VTA axons that are present in the prefrontal cortex show greater activity dependent structural plasticity during adolescence compared to adulthood [44]. Expression of dopamine receptors also peaks in dopamine target regions during the peripubertal period [45–48]. In sum, experiments with higher temporal resolution of development suggest week-by-week changes are occurring throughout the juvenile and adolescent period. Greater temporal resolution will be needed to identify exactly when specific sensitive periods with specific mechanisms may occur [49].

A large number of studies have found developmental experience with adversity can have effects on later dopamine neuron function, axonal anatomy and dopamine release in striatum measured in adulthood [50–54, are some examples]. Within this literature, some studies find harsh environments such as those involving maternal care disruption or social isolation are associated with facilitation of dopamine signals (via multiple measures) while others find dopamine signals are weakened. This variability may be driven by the timing, duration, and form of adversity. The adaptive developmental plasticity framework may help to clarify reasons for these different outcomes and provide an organizing framework for more comprehensive integration of the literature.

2). Dopaminergic neurotransmission can regulate the acute and long-term excitability of basal ganglia spiny projection neurons and cortical neurons

Dopamine release and binding onto D1R or D2R can affect intracellular signaling cascades, including a major cascade involving protein kinase A (PKA). Dopamine binding at D1Rs activates the cAMP-PKA cascades through excitatory G proteins while binding at D2Rs inhibits these same pathways through inhibitory G proteins. Changes in expression of D1Rs and D2Rs thus regulate SPN and cortical neuron responsiveness to changes in extracellular dopamine levels. An increase in dopamine will enhance excitability and connectivity (via long term potentiation of synapses) in SPNs and other neurons expressing D1Rs and reduce excitability and connectivity in SPNs and other neurons expressing D2Rs. These effects will be opposite in response to a dip in extracellular dopamine after a negative prediction error [21,55–57].

In our working model, we propose that more subtle, developmental experience-driven differences in dopamine activity could tune/program D1R and D2R expression and intrinsic excitability lasting into adulthood. In development, dopamine release and signaling may have acute and long-term impacts on the excitability of D1R and D2R-expressing SPNs [43,57–59] and dopamine receptor-expressing pyramidal neurons and interneurons of the prefrontal cortex [60,61]. Indeed, Lieberman et al [43] found that a developmental increase in dopamine release in the striatum is necessary for maturation of the excitability of D1R-expressing SPNs of the dorsal striatum. This regulation of excitability was linked to changes in potassium channel currents in the D1R-expressing SPNs. In this same study, the authors showed that mutant mice which had lower levels of dopamine release in striatum during development showed blunted change in D1R-expressing SPN excitability [43].

There is considerable literature in laboratory rodents showing that rearing in different environments can affect striatal and prefrontal cortex dopamine receptor expression [52,53,62–64]. Differences in dopamine release and receptor expression are thought to sculpt learning and decision making in rodents, non-human primates, and human subjects. While many studies focus on dopamine signaling and learning in terms of reinforcement of behavior [11,18], other studies have found relationships between dopamine receptors and aspects of decision making such as exploratory behavior and the explore-exploit tradeoff [64,65].

3). SPN excitability and history of activity will affect the weight or strength of cortical inputs

The excitability and activity of SPNs (sculpted by dopamine and dopamine receptor expression), may in turn influence the development of cortico-striatal inputs. This may be regulated by Hebbian and spike timing dependent plasticity [66] plus changes in intracellular signaling downstream of G protein coupled receptors (GPCRs) and activity dependent signals [55,56]. Recent data suggest genetic mutations in SPNs that affect excitability also affect cortical input onto these specific SPNs and can enhance learning [67]. Manipulation of cortical inputs to the striatum has also been shown to alter decision making [68] and drug related compulsive behavior [69,70].

In our working model, we suggest cumulative differences in experience may similarly affect cortical inputs to the striatum. Experiments in lab rodents support the basic premise that early experience can sculpt cortical inputs to the striatum. Cortical axons in the striatum continue to refine their synapses through development, and early life maternal care has been shown to affect the density of synapses made by frontal cortical axons in the dorsal striatum in adulthood [27]. Chronic stress during the juvenile period can also impact cortico-accumbal function [71,72]. These same manipulations have also been shown to affect cognitive flexibility and addiction related behaviors such as alcohol consumption [27→ 29,71]. There is also a rich literature describing the effects of developmental adversity on reward processing [36,37,72–74], learning [29,62,71] and decision making [35].

In sum, data from a broad literature suggest mechanisms by which accumulated experience affecting the dopamine system may have iterative downstream effects on the neurobiology of the striatum and its cortical inputs. We propose these effects, serving as a form of adaptive developmental plasticity, combine to alter how environmental stimuli engage learning and decision making processes in the brain.

Conclusions and applications

Going forward, new research needs to be conducted to decipher the input:output rules governing the relationship between environment and phenotype. To map these relationships, we may need to use more ethological models of experience in the laboratory that better mimic conditions from the wild. These studies can build on studies of maternal separation, social isolation, and exposure to high fat diet that are motivated to model modern human adversity. However, they may need to take into account the cognitive experiences of an animal engaged in active exploration and foraging [75]. Intensive sampling also needs to be used to understand when there may be a sensitive period in development for specific behaviors and to power examination of sex and species differences. Finally, computational methods will be needed to supplement traditional behavior metrics to isolate the latent variables under selection pressure.

By introducing concepts of adaptive developmental plasticity and proposing a working model for circuit development we hope to build a stronger foundation for understanding how developmental adversity can affect learning and decision making. Typically, differences in learning and decision making associated with adversity are considered behavioral deficits from a psychology perspective. From an evolutionary perspective, we may reframe some differences as adaptive strategies that emerge from developmental sensitive periods. This in turn may help to reframe how we think about issues like the achievement gap and inform how we motivate, design, and time interventions.

Highlights.

• Dopamine neurons and their striatal and cortical circuit targets support learning and decision making.

• These circuits and behaviors show protracted development into adolescence and early adulthood

• Cumulative activity in dopamine neurons will reflect the environmental statistics

• Evolved developmental programs could use this cumulative activity to sculpt behavioral phenotypes