Regulation of habit formation in the dorsal striatum

Abstract

Habits are an essential and pervasive component of our daily lives that allow us to efficiently perform routine tasks. But their disruption contributes to the symptoms that underlie many psychiatric diseases. Emerging data are revealing the cellular and molecular mechanisms of habit formation in the dorsal striatum. New data suggest that in both the dorsolateral and dorsomedial striatum histone deacetylase (HDAC) activity acts as a critical negative regulator of the transcriptional processes underlying habit formation. In this review, we discuss this recent work and draw conclusions relevant to the treatment of diseases marked by maladaptive habits.

When making a decision one prospectively considers potential actions and their anticipated consequences [1,2]. This allows behavior to readily adapt when circumstances change, but is cognitively taxing. The brain has another, more resource efficient, strategy for routine behaviors: habits. Habits permit common behaviors to be executed more automatically on the basis of their past success, without thought of their consequences, freeing attention to be focused elsewhere [3,4]. The balance between the reflective, goal-directed, and reflexive, habit, systems promotes maximally adaptive and efficient behavior [5,6], but when it is disrupted can lead to the symptoms that underlie myriad psychiatric and neurodegenerative diseases [7–11]. Indeed, deficits in the acquisition and execution of behavioral habits are symptoms of both Huntington’s [12] and Parkinson’s disease [13–15]. An overreliance on habit is associated with the various forms of compulsivity that manifest across a range of conditions [8,16], including obsessive-compulsive disorder [17,18], schizophrenia [19,20], addiction [11], alcoholism [21], and compulsive overeating [22]. Research over the past two decades has exposed the largely dissociable cortico-striatal-limbic brain circuits vital for goal-directed actions and habits (for review see: [23–29]). New information is emerging on how habits are acquired and stored within these networks. This will be the focus of this review.

Diagnosing and understanding habits

Instrumental behaviors can be modeled by training subjects to perform an action (e.g., lever press, maze run, button push) to receive a reward (e.g., food). Behavioral strategy cannot, however, be determined from simple observation of performance. Outcome-specific devaluation provides one diagnostic tool. Following training, non-reinforced instrumental performance is assessed after devaluation of the earned reward, usually achieved by sensory-specific satiation or pairing with a nausea-inducing agent. If subjects are prospectively evaluating the outcome of their actions, they will reduce action performance following devaluation [1,5]. Insensitivity to devaluation is a mark of habits [4,5]. Habits are also resistant to degradation of the action-outcome contingency, e.g., reward omission [30]. Initially, instrumental behavior is under control of the cognitively-taxing, but less error-prone, goal-directed system [1,5]. With repeated practice (e.g., overtraining), habits will slowly form and will come to dominate behavioral control when enough successful repetition has proceeded to ensure sufficient accuracy of the habit [3–5]. For the purpose of this review, habits will be defined as behaviors that are demonstrably insensitive to outcome devaluation and/or action-outcome contingency manipulation. In some cases, we will refer to data on motor-skill, procedural, or action-sequence learning, performance of all of which have been argued to be reflective of behavioral habits.

Habits have long been proposed to rely on a stimulus-response (S-R) associative architecture [5,31–34]. In this view, a reinforcing event functions to stamp in an association between the behavioral response that preceded it and the stimuli present when the response was executed, such that those stimuli become capable of automatically triggering the response. S-R associations are, however, inferred from a lack of evidence for action-outcome control. Although certainly a possibility, habits need not require S-R learning. Another, not necessarily mutually-exclusive, view is that habits result from the ‘chunking’ of commonly-performed action sequences into automatic, stereotyped routines [3,35,36]. One extension of this proposes that, rather than deliberating at each individual step, sequences of movements are concatenated into chunks that, once selected by the decision-maker, run off automatically [37–39]. Insensitivity to devaluation results from mistakes in planning that will systematically arise when devaluation occurs offline and from slips in action [38]. Recording studies lend support for this view [36]. Data from our laboratory show that dopamine release in the nucleus accumbens (NAc) core relates to hierarchical motivational control over a large chunk of stereotyped action sequences [40]. Following overtraining, dopamine release will backpropagate away from individual sequence performance and, instead, be released in response to the earliest unexpected reward predictor, where its magnitude inversely correlates with the average speed of subsequent stereotyped action sequence performance. The hierarchical view, that deliberate choices are executed through habitual action sequences, implies that goal-directed and habit processes cooperate more than compete [6,41] to control behavior. This has important implications for the neural mechanisms of actions and habit, suggesting they might not be as discrete as originally thought.

Dorsal striatal encoding of habit

The goal-directed and habit systems have been demonstrated in mice [42], rats [43–46], non-human primates [47,48], and humans [49–51] to rely on separate cortico-striatal-limbic circuits centered on the dorsomedial (DMS) ‘associative’ striatum and dorsolateral (DLS) ‘sensorimotor’ striatum, respectively. DMS lesions or inactivation prevent subjects from acquiring or using action-outcome associations, forcing reliance on the habit strategy [43–45]. Conversely, DLS lesions prevent the formation and execution of habits, preserving goal-directed control even after overtraining [46]. Neural recordings further support DMS v. DLS encoding of actions and habits. Whereas DMS medium spiny projection neuron (MSN) activity can relate to the deliberation required for goal-directed control [52,53], DLS activity is recruited with repeated training and inversely correlates with deliberation [54]. DLS, but not DMS, MSN activity has been shown to occur at the initiation and termination of instrumental action sequences [55–57] and this ‘chunking’ activity pattern strengthens with training and persists with overtraining [52,54,55,58], suggesting a relationship to habit [54]. Continuous DLS activity has also been reported during execution of well-trained motor sequences and such activity has been shown to represent the contextual and kinematic information needed to consistently execute a behavioral habit [59].

Striatal MSNs either directly (dMSNs) or indirectly (iMSNs) project to basal ganglia output nuclei and these pathways have traditionally been thought to promote and oppose, respectively, both motor output and reinforcement [60–63]. Evidence of opposing activity for instrumental behavioral control has been identified in the DMS, where augmented dMSN and attenuated iMSN post-synaptic strength has been associated with action-outcome learning [64]. In the DLS, depression of spontaneous excitatory events in iMSNs has been implicated in habit formation [65]. Interestingly however, there is also evidence of more coordinated dMSN/iMSN activity in the DLS [66,67]. Both DLS dMSNs and iMSNs are transiently active during execution of a well-trained instrumental behavior [66]. Correspondingly, strengthened ex vivo responses to cortical input in both DLS pathways has been associated with habit expression [68] and the synaptic strength of both projections is enhanced following extended training of a motor skill [69]. Activity in both DLS projections is also needed for well-trained behavior; optical inhibition of either DLS dMSNs or iMSNs will disrupt the initiation and execution of well-trained action sequences [67]. One interpretation of these coordinated activity data is that dMSN activity might facilitate output of desired habits, while iMSN activation inhibits competing motor programs [70]. In support of this, the disruption of well-trained sequence performance by DLS dMSN inactivation is caused by slowing of initiating movements, whereas DLS iMSN inactivation prompts a switch to other competing behaviors. The balance and timing of dMSN v. iMSN activation has also been suggested to be critical for habit execution, given evidence that faster dMSN relative to iMSN ex vivo activation correlates with habit [68]. More work is needed to clarify the unique and interacting functions of each subregion-specific striatal projection pathway in habit. It also remains to be well understood how habits are encoded in ensembles of DLS direct and indirect pathway projection neurons.

Molecular mechanisms of habit

Emerging evidence is indicating that striatal, especially DLS, MSN activity and neuroplasticity is associated with habit learning. Such long-lasting changes in neuronal excitability that support learning require changes in gene transcription and protein synthesis [71]. This has been borne out throughout the brain, including in the NAc, where considerable efforts have identified the intracellular signaling-transcriptional-translational pathways underlying the experience-dependent plasticity associated with reward learning, especially in the context of drug reward (for review see [72–76]). Evidence of the dorsal striatal molecular mechanisms that might regulate the neuroplasticity underlying habit is, relatively, more limited, but developing.

The extracellular signal-relate kinase (ERK) pathway has been demonstrated to be central to synaptic plasticity and learning in several different brain regions [77–79], including the dorsal striatum. In the striatum, ERK activation is regulated by glutamatergic and dopaminergic receptor signaling [80–83], making this pathway ideally suited to initiate intracellular signaling based on coincident or patterned activity of the two major striatal inputs and, therefore, to mediate MSN corticostriatal plasticity [84–87] and striatal-dependent learning. Indeed, dorsal striatal ERK signaling has been implicated in motor-skill learning and [86–88] and DLS ERK has been suggested as a candidate molecular regulator of habit [85,89], though direct evidence of the latter is needed. Similarly, cyclin-dependent kinase 5 (Cdk5), which is both down- and upstream of ERK [90], has been demonstrated to regulate MSN excitability, and, in the DLS, to be necessary for motor-skill learning [91]. ERK signaling is thought to contribute to striatal plasticity via regulation of gene transcription. In support of this, deletion of ERK from striatal iMSNs suppresses activity-associated gene expression and attenuates functional and structural plasticity [87].

One way ERK signaling regulates gene transcription is through phosphorylation of transcription factors, such as cyclic AMP response element-binding protein (CREB) [92]. CREB activates the transcription of target genes in response to a diverse array of stimuli mediated via activation of several different signal transduction pathways, including PKA and Ca²⁺, in addition to MAPK/ERK [93]. CREB-mediated gene expression is required for synaptic plasticity and memory processes in many brain regions [93,94], including both the ventral [95] and dorsal striatum [96]. In the dorsal striatum, CREB is required for both long-term potentiation and depression [96]. CREB is activated in the DLS, in particular, as habits come to dominate behavioral control [97]. Correspondingly, disruption of striatal CREB function interferes with both procedural learning [96] and the development of a habit-like navigation strategies [98,99], supporting the possibility that CREB-mediated transcription is critical for habit formation. Which specific gene transcription CREB regulates to mediate habit is an avenue ripe for investigation.

Epigenetic mechanisms of habit

Gene transcription occurs in the context of chromatin, the DNA-histone protein complex that packages genomic DNA. Epigenetic mechanisms alter accessibility to DNA for the transcriptional machinery to coordinate gene expression and are, therefore, fundamental regulators of the transcriptional processes mediating neuronal plasticity and memory [100–102]. Histone deacetylases (HDACs) are particularly interesting. Under basal conditions, HDACs, and their associated co-repressor complexes, are present at the promoter regions of actively transcribed genes. These HDACs remove acetyl groups from the histone tails, creating a repressive chromatin state that prevents active gene transcription [75,103–105]. HDACs have, therefore, been proposed to act as a ‘molecular brake’ to maintain specific genes in a silent state [103,104,106]. This brake can, however, be released; strong activity-dependent signaling, such as that induced by a salient learning event, can trigger a cascade that temporarily removes HDACs from gene promotors. This allows histone acetylation, which neutralizes the positive charge of the lysine residues on the histone’s N-terminal tails, loosening the DNA-histone interaction to, generally, promote the active gene expression that can subserve synaptic plasticity and learning [75,103–105]. Indeed, disrupting HDAC activity can produce hyperacetylation and has been shown to enhance hippocampal synaptic plasticity, transforming what would otherwise be transient, early-phase potentiation, into more persistent, transcription-dependent, long-term potentiation [107] and converting a subthreshold learning event into a long-term, transcription-dependent memory [107–112]. HDACs have also been shown to regulate sensory cortical plasticity [113] and amygdala long-term potentiation [114]. In the NAc, considerable evidence has implicated HDACs in drug memory (for review see [76,115]). Until recent work from our laboratory, the function of such mechanisms for instrumental learning or in the dorsal striatum was not known.

We recently investigated this [116]. Systemic inhibition of class I HDACs following instrumental conditioning sessions was found to increase histone acetylation in the dorsal striatum and to accelerate habitual control of behavior, suggesting that HDACs might normally be engaged in the dorsal striatum to restrain the gene expression underlying habit formation. In support of this, early in training, when habits did not yet dominate behavioral control, occupancy of HDAC3, the most highly-expressed class I HDAC in the brain [110,117], was enriched at specific learning-related and CREB-regulated gene promoters in the DLS and then returned to baseline levels with the repeated training that promoted habit. Correspondingly, decreasing HDAC3 function in the DLS was found to accelerate habit formation, while DLS HDAC3 overexpression prevented habit. These data suggest that HDAC3 is normally initially engaged to constrain the CREB-regulated plasticity necessary for habits and under suitable conditions (e.g., repeated success), an instrumental learning opportunity can trigger activity-dependent signaling that removes HDAC3 to create a state permissive to CREB-mediated transcription in the DLS that may allow the neuroplasticity necessary for habits to strengthen and, eventually, come to control behavior.

Based on the canonical role of the DMS in goal-directed learning, it would be logical to predict that disrupting transcriptionally-repressive HDAC3 activity in the DMS would enhance the transcription necessary for action-outcome encoding, and thereby, enhance goal-directed behavioral control, and, congruently, that increasing DMS HDAC3 function would attenuate action-outcome learning and force reliance on habit. But, surprisingly, we found the opposite; HDAC3 activity was found to constrain habit learning in the DMS, similar to its function in the DLS [116]. Disrupting HDAC3 function specifically in the DMS accelerated habit formation, while DMS HDAC3 overexpression prevented the normal transition to habit that would otherwise occur with overtraining. This was supported by evidence that HDAC3 occupancy was engaged early in training and removed with overtraining in the DMS, though at different (but still CREB-regulated) learning-related gene promoters than in the DLS. Therefore, the repressive enzyme HDAC3 acts as a molecular ‘brake’ on the gene transcription underlying habit formation in both the DLS and the DMS.

The precise upstream mechanisms that regulate the HDAC3 brake during instrumental learning are unknown. Because its catalytic activity can be regulated by its phosphorylation state [118], HDAC3 might be directly regulated by presynaptic inputs that activate intracellular signaling cascades. To form an active enzyme complex, HDAC3 must be associated with a co-repressor, either nuclear receptor corepressor (NCoR) or silencing mediator of retinoic acid and thyroid hormone receptor (SMRT) [119]. The ERK pathway might also, therefore, modulate HDAC3 function by phosphorylating and facilitating the nuclear export of SMRT [120], which leads to its dissociation from HDAC3, thereby reducing HDAC3’s transcriptional repression [121]. Future investigation of HDAC3 regulation, its targeting to specific genes, and how this might differ depending on striatal subregion and projection pathway will be pivotal.

Update of medial and lateral dorsal striatal function in actions and habits

Bidirectional subregion-specific manipulations reveal that the epigenetic repressor HDAC3 activity normally functions to constrain habit formation in not only the DLS, but also in the DMS [116]. This implies that transcriptional and likely also structural and/or functional changes in the DMS must occur for habits to develop. These data, therefore, challenge the canonical strict dissociation between DMS and DLS function in goal-directed and habitual control of behavior. Previous data suggest that DLS circuits might store habit-related information [54,59,65,68], with memories vital for goal-directed control stored in the DMS [64,122,123]. One possibility to reconcile these results with the traditional DLS/DMS functional dichotomy is, therefore, that HDAC3 may regulate the formation and storage of habit memories in the DLS, while in the DMS it depotentiates the action-outcome memories underlying goal-directed control as deliberation becomes less required. This might occur specifically in DMS dMSNs, because enhanced plasticity in these cells has been associated with goal-directed learning [64]. In support of this possibility, recent data suggest attenuation of DMS activity is needed to permit the transition to habit [123]. A second possibility is that DMS HDAC3 activity promotes the concatenation of actions into stereotyped chunks. Third, it is possible that HDAC3 regulates activity changes in DMS indirect-pathway projections, activity in which might oppose goal-directed behavioral control and thereby promote habit [124]. These three non-mutually-exclusive hypotheses are ripe for future investigation.

Implications for disease

Dorsal striatal HDAC3 functions as a molecular brake over habit, remaining in place to slow the transition to habit and being removed when the conditions are ripe for habits to dominate [116]. HDAC3 dysfunction could, therefore, lead to the inability to form habits or the maladaptive compulsive habits that mark many neurodegenerative and psychiatric diseases. HDACs might, therefore, be a promising therapeutic target for a variety of conditions [104,125–129]. Indeed, the aforementioned data support recent proposals of the utility of HDAC inhibitors for the treatment of Huntington’s and Parkinson’s disease [130], suggesting such treatment targeted to dorsal striatal HDAC3 might alleviate some of the deficits in habit that mark these diseases.

An overreliance on habit learning strategies is associated with a compulsive phenotype [8,16], including that found in patients diagnosed with addiction [8,131]. Similarly, chronic exposure to cocaine [132–135], amphetamine [136], or alcohol [137,138], as well as binge-like consumption of highly palatable foods [139] can potentiate habit formation, resulting in reward-seeking behavior that is insensitive to its consequences, even when those consequences are negative. Chronic exposure to the most commonly abused substances also increases dorsal striatal histone acetylation [140–142], creating a chromatin state ripe for potential maladaptive learning. Nicotine’s ability to do this has been proposed as a potential mechanism for it as a gateway drug [142,143]. Moreover, stress, a predisposing factor to mental illness, can lead to abnormal HDAC activity [144,145] and can potentiate habits [146–148] via opposing structural changes in DMS and DLS circuits [146]. If dorsal striatal HDAC3 functions as a molecular brake on habit, then chronic stress and exposure to addictive substances might remove this brake, creating an epigenetic ‘tag’ that biases future behavioral strategy towards habit, even with this is not adaptive, producing the compulsivity that marks many mental illnesses. If this is true, then dorsal striatal HDAC3 might be a viable target to prevent or reverse maladaptive compulsive behavior.

Highlights.

• Actions and habits rely on dorsal medial and lateral striatal activity, respectively.

• The epigenetic repressor HDAC3 is a negative regulator of habit in both subregions.

• Stress and addictive substances can alter striatal HDAC activity.

• Disrupted dorsal striatal HDAC3 activity might underlie maladaptive habits.