Neural substrates of habit

Abstract

Active reward pursuit is supported by the balance between the cognitive and habitual control of behavior. The cognitive, goal-directed strategy relies on prospective evaluation of anticipated consequences, which allows behavior to readily adapt when circumstances change. Repetition of successful actions promotes less cognitively taxing habits, in which behavior is automatically executed without prospective consideration. Disruption in either of these behavioral regulatory systems contributes to the symptoms that underlie many psychiatric disorders. Here, I review recently identified neural substrates, at multiple neural levels, that contribute to habits and outline gaps in knowledge that must be addressed to fully understand the neural mechanisms of behavioral control.

Two behavioral control systems contribute to active reward pursuit and decision making, one cognitive and one habitual. The cognitive, goal-directed system mediates behavior that relies on consideration of the learned relationship between actions and their consequences (Adams and Dickinson, 1981), allowing for flexible decision making. The habitual system allows behavior to be relatively automatically executed on the basis of prior success. Thus habits are elicited by antecedent events and do not involve prospective consideration of expected outcomes (Adams and Dickinson, 1981; Dickinson, 1985). Failure in either of these behavioral regulatory systems is characteristic of myriad psychiatric and neurodegenerative diseases. An overreliance on habit is especially associated with the compulsivity that manifests in several conditions (Voon et al., 2015; Gillan et al., 2016), including obsessive-compulsive disorder (Gillan et al., 2011; Gillan et al., 2014), schizophrenia (Griffiths et al., 2014; Morris et al., 2015), addiction (Belin et al., 2013), alcoholism (Corbit and Janak, 2016), and compulsive overeating (Corbit, 2016). On the other hand, failures in the acquisition and execution of habits is characteristic of Huntington’s (Heindel et al., 1988) and Parkinson’s disease (Agostino et al., 1996; Knowlton et al., 1996; Cohen and Pourcher, 2007). Therefore, it is necessary to understand the neural substrates of these behavioral regulatory systems and how these two, seemingly opposing, systems coordinate behavioral control.

The aim of this review is to highlight recently identified neural substrates, from the circuit level to the cellular level and down to the epigenetic level, that contribute to habits and to identify the gaps in knowledge that must be addressed to understand how behavioral control arises. Understanding how, for example, cortical input to the striatum alters not only striatal cell activity, but also the molecular and epigenetic mechanisms within those cells, will reveal how neuronal function is altered to give rise to the network activity that may direct behavioral output.

Identifying goal-directed actions and habits

Although instrumental behavior, in which voluntary actions are performed to cause a particular outcome, may start out goal directed, with repeated practice behavioral control can shift to the more automatic, habitual system to perform the exact same behavior (e.g., pressing a lever for a food pellet) more efficiently. Consequently, simply observing instrumental responding is insufficient to determine whether performance is being achieved via a goal-directed or habit strategy because the instrumental response itself is not indicative of whether prospective consideration of the outcome is taking place. A common approach to untangle these behavioral control systems is to devalue the outcome and subsequently examine behavioral responding (Dickinson, 1985). Behavioral responding is assessed in a non-reinforced probe test to ensure instrumental performance relies on the anticipated, rather than experienced, outcome value to guide behavior. This will demonstrate whether the goal (i.e. the outcome) is part of the representational structure guiding behavior. Goal-directed behaviors rely on the learned association between an action and the anticipated outcome. Therefore, responding will diminish if the anticipated outcome is no longer valued. Conversely, habits, which are defined in terms of their autonomy from goal representation, will continue regardless of whether the outcome is devalued. The presence of goal-directed control can also be revealed by degrading the contingency between an action and its outcome by freely delivering the outcome whether or not a response is performed, or by explicitly omitting the outcome following a response, both of which will decrease responding if behavior is goal-directed. Behaviors such as learned motor responses, motor-skills, procedural, and action-sequence learning have the overlapping feature of automaticity and share neural circuitry with habits as defined above, and thus may be related to, and sometimes defined as habits. Goal-directed behavior shares features with model-based reinforcement learning, where prospective consideration of potential outcomes drives flexible behavior, while habits have been proposed to be akin to model-free reinforcement learning, in which behavior is driven by retrospective events (Daw et al., 2005; Daw et al., 2011; Voon et al., 2015; Groman et al., 2019).

Neural circuits supporting goal-directed and habit behavior

Early studies demonstrating that animals, under some conditions (e.g., limited training), showed knowledge of anticipated outcomes, and under other conditions (e.g., extended training) showed evidence of habit (e.g., (Tolman and Gleitman, 1949)), led to the idea that these two forms of learning might be mediated in parallel by distinct neural mechanisms. Packard and McGaugh (1996) showed that following extended training on a cross-maze to reach a food outcome, in which a response habit strategy (i.e., body turn) eventually dominates over the goal-oriented, place-strategy (e.g., learned location within the room), inactivating the dorsolateral striatum at test prevented expression of the response habit strategy and exposed the conservation of the goal-oriented, place-strategy. These findings were among the first to establish that habits are learned concurrently with other types of goal-directed behaviors and mediated by different neural systems. This dissociability between the behavioral control systems has since been extended specifically to goal-directed and habitual instrumental behavior. There is now considerable converging evidence (reviewed previously (Balleine and O’Doherty, 2009; Burton et al., 2015; Malvaez and Wassum, 2018)) demonstrating the dorsolateral striatum (DLS) is engaged in and necessary for habit (Yin et al., 2004; Smith and Graybiel, 2013), while the dorsomedial striatum (DMS) is engaged in and necessary for goal-directed behavior (Yin et al., 2005a; Yin et al., 2005b; Thorn et al., 2010). Notably, the data supporting this dissociability has been collected across a variety of assays—lesion (Yin et al., 2004; Yin et al., 2005b), inactivation (Yin et al., 2005a, 2006; Corbit and Janak, 2010) and neural recording studies (Jog et al., 1999; Barnes et al., 2005; Jin and Costa, 2010; Thorn et al., 2010; Smith and Graybiel, 2013; Jin et al., 2014; Regier et al., 2015; Rueda-Orozco and Robbe, 2015)—and across species— mice (Hilario et al., 2012), rats (Yin et al., 2004, 2005a; Yin et al., 2005b; Corbit and Janak, 2010), non-human primates (Miyachi et al., 1997; Miyachi et al., 2002), and humans (Tricomi et al., 2009; Liljeholm et al., 2011; McNamee et al., 2015), but see (de Wit et al., 2018). This body of literature has, generally, set up a dichotomy between the DLS and DMS in the competition for behavioral control. However, more recent evidence suggests these systems may be engaged simultaneously throughout instrumental learning, suggesting more cooperative function (Li et al., 2016; Kupferschmidt et al., 2017; Malvaez et al., 2018). This section highlights neuroanatomical structures recently identified to play a crucial role in goal-directed and habit behavior.

Evidence suggests several cortical regions may influence behavioral control strategy. The DMS receives several inputs from limbic and prefrontal cortical regions (McGeorge and Faull, 1989) that have themselves been implicated in goal-directed behavior (Corbit and Balleine, 2003; Murray and Izquierdo, 2007; Ostlund and Balleine, 2007a, b; Corbit et al., 2013; Wilson et al., 2014; Bradfield et al., 2015; Schuck et al., 2016; Wikenheiser and Schoenbaum, 2016). Often, the neuronal activity or gray matter volumes of these regions are compromised in diseases characterized by an overreliance on habit (Voon et al., 2015; Fettes et al., 2017). The orbitofrontal cortex (OFC) displays the greatest change in neuronal firing rate during goal-directed control of behavior, but not habits (Gremel and Costa, 2013). Furthermore, new data has revealed that updating goal expectations triggers plasticity of OFC dendritic spines, the principle sites of excitatory synapses, and this plasticity is necessary for goal-directed behavior (Whyte et al., 2019). Inhibiting OFC excitatory activity (Gremel and Costa, 2013) or dendritic spine plasticity (Whyte et al., 2019) disrupts goal-directed behavior, together indicating the OFC is important for maintaining goal-related information.

Using a within-subject task that promotes the use of either habits or goal-directed behavior, depending on the training context, Gremel and colleagues (2016) showed that OFC projections specifically to the DMS are vital for goal-directed control. Inhibiting the activity of OFC terminals in the DMS prevents goal-directed learning and forces reliance on the habit strategy, even in the training context where the goal-directed strategy should dominate behavioral control (Gremel et al., 2016). Consistent with the disruption of goal-directed behavior as a consequence of reduced OFC output activity, chronic ethanol exposure decreases the excitability of OFC projection neurons, specifically reduces neurotransmitter release onto DMS neurons, and disrupts goal directed control of behavior (Renteria et al., 2018). That excitatory OFC output activity is essential for goal-directed behavior is in accordance with findings implicating the OFC in representing potential outcomes (Ostlund and Balleine, 2007a, b), including those needed for goal-directed instrumental behavior (Zimmermann et al., 2017; Zimmermann et al., 2018; Malvaez et al., 2019), and indicates that the OFC must communicate with the DMS for successful goal-directed behavior, otherwise habits emerge. Moreover, human subjects with lower gray matter volume in the medial OFC and striatum are biased toward model-free (habit) learning in the two-step sequential learning task, which measures whether choices are made using the predicted outcome of each choice (model-based, goal-directed learning) or the retrospective success of the preceding choice (model-free, habit learning) (Voon et al., 2015). However, this is not to imply that goal-directed behavior only needs OFC-DMS interactions. Indeed, DMS inputs from the basolateral amygdala (Corbit et al., 2013) and thalamus (Bradfield et al., 2013) are also necessary for the acquisition and expression of goal-directed behavior.

Relative to all we know about the various neural substrates of goal-directed action (for additional review see (Balleine and O’Doherty, 2010; Griffiths et al., 2014)), there are far fewer defined neural substrates of habits. This is largely due to the fact that neural substrates contributing to behavioral control have been typically identified through their necessity for goal-directed behavior; when a brain structure necessary for goal-directed behavior is compromised, habit strategies usurp behavioral control. Moreover, very little is known whether or how goal-directed circuit activity changes to allow adaptive habits to emerge.

Only a handful of studies have identified cortical structures that contribute to habit behavior. Some evidence suggests the infralimbic cortex (IL) is required not only for the acquisition of habit responses (Smith and Graybiel, 2013), but also for preserving habitual behavior. When habits are established, inhibiting the IL restores goal-directed behavior (Coutureau and Killcross, 2003; Smith et al., 2012). Using a within-session dual reinforcement schedule, one promoting goal-directed behavior and another promoting habit responding, it was demonstrated that, following unreinforced lever presses during the outcome devaluation test, IL firing activity is suppressed during goal-directed actions, but is sustained while responding on the habit-promoting reinforcement schedule (Barker et al., 2017). Selectively inhibiting IL activity immediately after each lever press when animals are using a habit strategy yields sensitivity to contingency degradation, essentially reverting animals back to goal-directed control of behavior (Barker et al., 2017). These findings suggest the IL is involved in the expression of habits, but it is unclear whether IL activity is achieving this role by suppressing goal-directed behavior, sustaining habitual responding, or some other strategy-selection process. Indeed, the IL is thought to suppress previously learned contingencies to promote habits (Coutureau and Killcross, 2003; Killcross and Coutureau, 2003; Smith et al., 2012; Barker et al., 2014). Considering the IL does not project to the DLS (Mailly et al., 2013), the canonical brain region supporting habits, it will be interesting to explore how the IL interacts with other striatal and/or cortical regions to direct behavioral control. It is unknown, for example, if IL projections to limbic regions implicated in behavioral inhibition are involved in the maintenance of habit behavior by suppressing goal-directed behaviors or if IL projections to the ventral striatum are influencing the incentive processes that modulate instrumental performance (Corbit et al., 2001).

Cell-type specific contributions to goal-directed and habit behavior

In addition to identifying the neuroanatomical structures contributing to behavioral control, some effort has been made to identify the cell-type specific contributions to instrumental control. Within each striatal subregion there are two classes of spiny projection neurons (SPNs), which are a type of GABAergic inhibitory cell that comprise ~95% of striatal neurons, that directly (dSPNs) or indirectly (iSPNs) project to basal ganglia output nuclei (Freeze et al., 2013). These pathways have traditionally been thought to work in opposition (Albin et al., 1989; Graybiel, 2000). Indeed, post-synaptic strength is augmented in dSPNs, but attenuated in iSPNs, in the DMS following the acquisition of goal-directed behavior (Shan et al., 2014). Furthermore, habit learning induces post-synaptic depression of iSPNs in the DLS (Shan et al., 2015). However, there is also evidence of coordinated pathway-specific activity. During the execution of well-trained actions, DLS dSPNs and iSPNs are both active (Cui et al., 2013) and necessary for the initiation and execution of well-trained action sequences (Tecuapetla et al., 2016). Moreover, both DLS pathways display strengthened responses to cortical input following habit training (O’Hare et al., 2016). These conflicting findings on pathway-specific activity (for review, see (Malvaez and Wassum, 2018)) may be reconciled by more comprehensive studies examining the subregion-specific contributions of each striatal pathway in habits.

Only recently, have investigations focused on the role of discrete striatal interneurons in the context of goal-directed versus habitual behavior (Figure 1). Striatal fast-spiking interneurons (FSIs), most prominent in the DLS (Gerfen, 1985), display increased excitability following habit training, firing more readily in response to excitatory input, and FSI activity in the DLS is necessary for the expression of habits (O’Hare et al., 2017). Moreover, DLS FSI activity was found to be necessary for establishing the DLS SPN properties that predict habit behavior (O’Hare et al., 2017). In the DMS, a reduction in FSIs has been implicated in the development of pathological repetitive behaviors via dysregulation of SPN baseline firing rates (Burguière et al., 2013). In a mouse model of compulsive behavior, OFC mediated activation of DMS FSIs restores inhibition of DMS SPN activity and alleviates repetitive behavior (Burguière et al., 2013). Together, these studies indicate FSIs may differentially regulate striatal microcircuits in a subregion-specific manner.

Figure 1. Striatal cell-types and potential interactions regulating behavioral control.

The dorsomedial (DMS) and dorsolateral striatum (DLS) both contain spiny projection neurons (SPNs), that directly (dSPNs) or indirectly (iSPNs) project to basal ganglia output nuclei, as well as various interneurons such as fast-spiking interneurons (FSIs), tyrosine-hydroxylase expressing interneurons (THINs), and cholinergic interneurons (CINs). These interneurons can modulate the activity of SPNs to differentially affect goal-directed and habitual behavior, however, several of the interactions between the specific cell-types have not been delineated in the context of goal-directed and habit behaviors.

Striatal microcircuits are also regulated by tyrosine-hydroxylase expressing interneurons (THINs) that receive excitatory thalamic and cortical input, as well as integrate dopaminergic and cholinergic signaling (Ibáñez-Sandoval et al., 2010; Ibáñez-Sandoval et al., 2015). Kaminer et al., (2019) found that THIN lesions throughout the dorsal striatum do not impair motivation to lever press for a food reward in the progressive ratio test, in which the response requirement increases for each successive reward, but do produce insensitivity to outcome devaluation (Kaminer et al., 2019). This impairment in goal-directed behavior was found to correlate with THIN lesions restricted to the more posterior striatum (Kaminer et al., 2019), which corresponds with the necessity of the posterior DMS, but not the anterior DMS, for goal-directed behavior (Yin et al., 2005b). A potential means by which THINs affect behavioral control is via integration of cholinergic signaling. Indeed, the activity of cholinergic interneurons in the DMS has previously been found to be necessary for updating action-outcome contingencies for effective goal-directed behavior (Bradfield et al., 2013). However, it is still unclear how striatal THIN activity is modulated by cortical and/or thalamic inputs during instrumental learning and whether this differs between dorsomedial and dorsolateral striatal subregions to coordinate behavioral control. Furthermore, given the known role of dopamine signaling in habit learning (Faure et al., 2005; Nelson and Killcross, 2006; Hernandez et al., 2013), it will be useful to investigate whether the effects of dopaminergic input to the striatum are mediated via striatal THINs.

Cellular signaling mechanisms of habits

Long-lasting changes in neuronal excitability are required to support instrumental learning and likely also behavioral control strategies. Recent studies have begun to elucidate the intracellular signaling mechanisms that contribute to such plasticity. Both dSPNs and iSPNs activity is altered by the activation of G-protein coupled receptors (GPCRs) (Wang et al., 2006; Shen et al., 2008), which relay incoming signals by either stimulating or inhibiting the activity of intracellular targets (Figure 2). Dopamine D1 and D2 receptors, located on dSPNs and iSPNS, respectively, are GPCRs that have opposing effects on the activity of the catalytic enzyme adenylyl cyclase and have both been implicated in habits. Antagonism of D1 receptors, which effectively reduces dSPN output, prevents the acceleration of habits, while antagonism of D2 receptors, which facilitates iSPN output, exacerbates habit formation (Nelson and Killcross, 2013). Adenosine A2A receptors (A2ARs) are GPCRs exclusively expressed on iSPNs that have also been implicated in behavioral control (Yu et al., 2009; Li et al., 2016). A2ARs contain the activating Gs α subunit, therefore, A2AR activation will lead to an increase of iSPN GABAergic output (Shen et al., 2008). In the DMS, activating A2AR signaling, effectively increasing iSPN output, during outcome delivery renders animals habitual (Li et al., 2016), while deleting A2ARs, effectively decreasing iSPN output, specifically in the DMS enhances goal-directed behavior (Li et al., 2016). This suggests A2AR-mediated signaling during the time of reward exerts control of instrumental behavior (Figure 2). Correspondingly, loss of goal-directed behavior induced by psychostimulant exposure can be restored by suppressing A2AR-mediated signaling in the DMS during goal-updating (Furlong et al., 2015).

Figure 2. Striatal output activity is regulated by G-protein coupled receptors and contributes to behavioral control.

In the dorsomedial striatum (DMS), G-protein coupled receptors are known to modulate spiny projection neuron (SPN) activity. (a) Adenosine A2A receptors (A2ARs), exclusively expressed in indirect output pathway spiny projection neurons (iSPNs), contain the activating G-protein (Gs). A2AR activation in the DMS increases the inhibitory activity of the indirect output pathway (iSPN), ultimately inhibiting goal-directed behavior. Conversely, inhibiting A2ARs in the DMS prevents the inhibitory iSPN output activity, thereby enhancing goal-directed behavior. (b) Excitatory input from the orbitofrontal cortex (OFC) increases DMS direct pathway spiny projection neuron (dSPN) output activity and is necessary for goal-directed behavior. (c) OFC pre-synaptic activity in the DMS is negatively regulated by the cannabinoid type 1 (CB1) that is coupled to inhibitory G-proteins (Gi), which causes reduced dSPN activity and suppresses goal-directed behavior, rendering animals habitual.

Beyond GPCRs located directly on SPNs, the cannabinoid type 1 (CB1) GPCR has also been implicated in behavioral control (Figure 2). CB1 receptors are located pre-synaptically and contain the Gi α subunit, therefore, CB1 receptor activation reduces the probability of neurotransmitter release onto post-synaptic cells (Gerdeman and Lovinger, 2001). Genetic deletion of the cannabinoid type 1 (CB1) GPCR impairs habit learning (Hilário et al., 2007) and activation of CB1 on DMS-projecting OFC neurons is necessary to suppress goal-directed behavior (Gremel et al., 2016). Interestingly, chronic activation of CB1 by THC accelerates the transition to habits and impairs iSPN long-term depression and synaptic depotentiation (Nazzaro et al., 2012)— a potential pathway whereby chronic cannabinoid use could lead to maladaptive habitual behavior. A crucial open question is how the intracellular signaling cascades that are activated by these receptors contribute to the plasticity underlying habits.

Striatal neuroplasticity depends on the activation of intracellular signaling cascades such as the extracellular signal-relate kinase (ERK) pathway (Mazzucchelli et al., 2002; Shiflett and Balleine, 2011; Hutton et al., 2017). Indeed, striatal ERK signaling has been implicated in motor-skill learning (Mazzucchelli et al., 2002; Bureau et al., 2010; Hutton et al., 2017) and has been proposed as a candidate molecular regulator of habit (Shiflett et al., 2010; Shiflett and Balleine, 2011). Recent studies have revealed mechanisms upstream of ERK that may be involved in regulating the shift in behavioral control (Figure 3). Activation of the tyrosine receptor kinase B (TrkB), the receptor for Brain-derived Neurotrophic Factor (BDNF), leads to the activation of several small G-proteins that ultimately activate ERK (Huang and Reichardt, 2003). Pitts et al., (2018) reported that TrkB mediates plasticity necessary for both actions and habits (Pitts et al., 2018). Overexpressing a truncated form of TrkB, which cannot activate intracellular signaling pathways, in the DMS prevented goal-directed behavior. In contrast, expressing the inactive TrkB in the DLS impaired the acquisition of habit learning (Pitts et al., 2018). It is noteworthy that the stress hormone corticosterone reduces TrkB levels in the brain (Gourley et al., 2012), causes a shift from full length TrkB to the truncated, inactive from of TrkB (Barfield et al., 2017), and biases behavioral control toward habits (Gourley et al., 2012; Barfield et al., 2017), similar to the ability of stress to promote habit formation (Dias-Ferreira et al., 2009).

Figure 3. Striatal subregion-specific activation of intracellular signaling mechanisms differentially regulates behavioral control.

The extracellular signal-related kinase (ERK) pathway is activated in the dorsal striatum during instrumental learning and may potentially contribute to the transcription necessary for synaptic plasticity. (a,b) The tyrosine receptor kinase B (TrkB), the receptor for brain-derived neurotrophic factor (BDNF), is a potential upstream mechanism of ERK activation in both the dorsomedial striatum (DMS) and dorsolateral striatum (DLS). (a) In the DMS, TrkB activation is necessary for goal-directed behavior, (b) whereas in the DLS TrkB activation contributes to habits. In the DLS, (b) the proton-gated acid-sensing ion channel isoform ASIC1a activates ERK signaling and is necessary for motor learning. It is unknown whether ASIC1a contributes specifically to instrumental habit learning.

Another potential component regulating habits, recently gleaned from motor learning studies, is the regulation of ERK by proton-gated acid-sensing ion channels (ASICs), in particular isoform ASIC1a. Genetic deletion or acute inhibition of ASIC1a in the dorsal striatum impairs activation of ERK signaling and impairs motor learning, without affecting initial motor performance (Yu et al., 2018). Reintroducing ASIC1a in the DLS of Asic1a knockout mice is sufficient to restore motor learning (Yu et al., 2018). Although high-frequency stimulation can trigger substantial release of neurotransmitter along with sufficient protons to activate ASICs (Du et al., 2014), the endogenous striatal inputs that activate striatal ASIC1a during instrumental learning are unknown. Additional studies are also necessary to clarify the contribution of ASICs in a subregion- and pathway-specific manner.

Establishing long-term changes in habit-related plasticity

The long-lasting neuroplasticity— from changes in synaptic strength to altered dendritic structure— that has been reported in striatal SPNs following habit learning suggest stable changes in gene expression occur to alter neuronal function. Epigenetic mechanisms alter accessibility to DNA for the transcriptional machinery to coordinate gene expression and are, therefore, fundamental regulators of the transcriptional processes mediating stable changes in neuronal function and memory (Kouzarides, 2007; Day and Sweatt, 2011; Peixoto and Abel, 2013). One of the best studied epigenetic regulators are histone deacetylases (HDACs), which remove acetyl groups from histone tails, creating a repressive chromatin state that prevents active gene transcription (Renthal and Nestler, 2009; McQuown and Wood, 2011; Robison and Nestler, 2011; Vogel-Ciernia and Wood, 2012). With sufficient activity-dependent signaling, such as that induced by a salient learning event, HDACs, particularly HDAC3 (McQuown et al., 2011; McQuown and Wood, 2011; Malvaez et al., 2013; Rogge et al., 2013; Alaghband et al., 2017; Kwapis et al., 2017), can be removed temporarily to allow for histone acetyltransferases (HATs) to acetylate histones. Histone acetylation neutralizes the positive charge of the lysine residues on the histone’s N-terminal tails, relaxing the chromatin structure to, generally, promote the active gene expression that subserves long-lasting synaptic plasticity and learning (Renthal and Nestler, 2009; McQuown and Wood, 2011; Robison and Nestler, 2011; Vogel-Ciernia and Wood, 2012).

Recently, the role of this epigenetic mechanism in the development of habit learning was examined. Using a multifaceted approach, including pharmacological HDAC inhibition, novel viral constructs to bi-directionally manipulate HDAC3 in the dorsal striatum, and chromatin immunoprecipitation, it was revealed that in both the DLS or DMS the transcriptional repressor HDAC3 is removed from the promoters of key neuronal activity-related genes (Figure 4) only following extended training, when conditions are ripe for habits to form (Malvaez et al., 2018). Furthermore, compromised HDAC3 activity in either the DLS or DMS promotes habits at a point when behavior should be goal-directed (Malvaez et al., 2018), indicating HDAC3 functions as a critical negative regulator of the gene transcription underlying habit formation in both the DLS and the DMS. It remains unknown what cortical or thalamic inputs are involved in activating the appropriate intracellular signaling mechanisms to modulate the repressive function of HDAC3 as behavior shifts toward habit. Furthermore, future studies must examine how HDAC3 regulates the transcription necessary for establishing changes in the signaling dynamics of the individual cell-types in the DMS and DLS to coordinate the development of habits.

Figure 4. HDAC3 negatively regulates habits in the DMS and DLS.

Histone modifications regulate the access to DNA for the transcriptional machinery and are fundamental in neuronal function and memory. (a) Early in instrumental training, when behavior is goal-directed, the transcriptional repressor histone deacetylase 3 (HDAC3) is engaged near gene promoters, removing acetyl groups from the histone’s N-terminal tails, creating a transcriptionally repressive chromatin state where DNA is less accessible to the transcriptional machinery. (b) With sufficient training, when animals become habitual, HDAC3 is disengaged from gene promoters by allowing the addition of acetyl groups (Ac) to the histone tails, which neutralized the interaction between the DNA and histone tails and leads to a relaxed chromatin state. This allows for the transcription of plasticity-related genes. It is unknown which histone acetyltransferase (HAT) contributes to this process and how these mechanisms are activated.

Exposure to cocaine causes an abnormally rapid development of habits and dysregulates striatal synaptic plasticity (Corbit et al., 2014). Interestingly, cocaine experience also induces a specific histone acetylation profile in the striatum (Kumar et al., 2005), possibly establishing an epigenetic landscape ripe for potentiating maladaptive habit learning. Indeed, very recent work has demonstrated that acetylation of histone H3, along with the expression of the plasticity-related gene Bdnf, was upregulated in the DLS following cocaine experience, and this was associated with a bias toward use of a response habit spatial navigation strategy (Harvey et al., 2019). This was also associated with decreased acetylation of histone H3 in the hippocampus (Harvey et al., 2019), possibly disrupting the acquisition of the goal-oriented strategy. These findings demonstrate that histone acetylation is differentially regulated in distinct brain regions, possibly altering the neuronal function of these two systems in an opposing manner to contribute to behavioral control. It remains unknown how the activity of the HATs and HDACs that regulate histone acetylation are altered as habits develop in the cortical structures highlighted above that contribute to behavioral control.

Beyond chromatin modifications (e.g., histone acetylation, methylation, phosphorylation), transcription is also epigenetically regulated via chromatin remodeling, which refers to the restructuring of nucleosomes (Saha et al., 2006), and via DNA methylation (Day and Sweatt, 2010), mechanisms that are both pivotal for memory formation (Day and Sweatt, 2010; Vogel-Ciernia et al., 2013). Interestingly, mutations in components of the chromatin remodeling complex have been implicated in autism (Neale et al., 2012; O’Roak et al., 2012) and schizophrenia (Loe-Mie et al., 2010), both of which have symptoms of maladaptive habits (Griffiths et al., 2014; Alvares et al., 2016). DNA methylation is also interesting in the context of habit learning given that it works in concert with histone acetylation to regulate memory processes (Miller et al., 2008). It is currently unknown whether these mechanisms are also involved in instrumental learning, and if so, what the gene targets are, and do these mechanism constrain the acquisition and/or expression of goal-directed or habitual behavior.

Significant open questions

An overreliance on habit is associated with the various forms of compulsivity that manifest across a range of conditions (Voon et al., 2015; Gillan et al., 2016) where maladaptive reward-seeking is pursued at the expense of goal-directed action strategies, including addiction (Belin et al., 2013; Sjoerds et al., 2013; Everitt and Robbins, 2016; Vandaele and Janak, 2017), obsessive-compulsive disorder (Gillan et al., 2011; Gillan et al., 2014), and schizophrenia (Griffiths et al., 2014; Morris et al., 2015). Although significant progress has been made in identifying the neural mechanisms involved in goal-directed and habit behavior, several gaps in knowledge remain. A lack of goal-directed behavior often manifests as habitual control of behavior, but this can emerge due to various deficits in specific elements of goal-directed behavior, such as the inability to update the value of the prospective outcome, failures in retrieving the current outcome value, or the inability to decipher distinct features of an outcome, and/or the direct strengthening of habits. Although several putative habit substrates have been identified, as discussed above, the unique function of these substrates in behavioral control, as well as the mechanisms involved in arbitrating these regulatory systems, have not been clearly defined. It remains to be understood how habits are encoded in the ensemble activity of direct and indirect pathway projection neurons. How does the cell-type specific ensemble activity change in each striatal subregion as habits develop? And importantly, how do these pathways interact? From the studies highlighted here, it is clear that behavioral output is regulated at multiple levels—from molecular, to cellular, to brain network interactions. Adaptive behavioral control does not arise from any single brain region or epigenetic mechanism in isolation, but from the coordinated interactions of all of these regulatory mechanisms. To understand how behavior transitions between the goal-directed and habit systems, it is critical to understand how each neural level influences the other. Changes at the molecular level will give rise to functional changes in cell activity, which in turn may alter neural responses to incoming signals (e.g., environmental stimuli), and ultimately influence behavioral output. This approach may facilitate our understanding of how the engagement of one cortical-striatal network versus another is controlled.

Significance.

Habits are an adaptive component of our daily lives that allow us to efficiently perform routine tasks, and their disruption contributes to the symptoms that underlie many psychiatric disorders. This review highlights the neural substrates, from brain networks, to cellular and epigenetic mechanisms, recently identified to play a crucial role in goal-directed and habit behavior.