Striatal dopamine signals are region specific and temporally stable across action-sequence habit formation

Summary

Habits are automatic, inflexible behaviors that develop slowly with repeated performance. Striatal dopamine signaling instantiates this habit-formation process, presumably region specifically and via ventral-to-dorsal and medial-to-lateral signal shifts. Here, we quantify dopamine release in regions implicated in these presumed shifts (ventromedial striatum [VMS], dorsomedial striatum [DMS], and dorsolateral striatum [DLS]) in rats performing an action-sequence task and characterize habit development throughout a 10-week training. Surprisingly, all regions exhibited stable dopamine dynamics throughout habit development. VMS and DLS signals did not differ between habitual and non-habitual animals, but DMS dopamine release increased during action-sequence initiation and decreased during action-sequence completion in habitual rats, whereas non-habitual rats showed opposite effects. Consistently, optogenetic stimulation of DMS dopamine release accelerated habit formation. Thus, we demonstrate that dopamine signals do not shift regionally during habit formation and that dopamine in DMS, but not VMS or DLS, determines habit bias, attributing “habit functions” to a region previously associated exclusively with non-habitual behavior.

Graphical abstract

Highlights

• •Validation of a novel test that monitors habit development individually across time
• •Dopamine release during habit development is stable across relevant striatal regions
• •Only dopamine release in dorsomedial striatum correlates with habit development
• •Optogenetic stimulation of dorsomedial striatal dopamine accelerates habit formation

Van Elzelingen et al. simultaneously track striatal dopamine signaling and development of habitual behavior. In the dorsomedial striatum, previously associated with non-habitual behavior, dopamine release increases during action initiation in habitual rats and decreases during action completion, whereas non-habitual rats show opposite effects.

Introduction

Our daily lives are governed by learned routines, exemplified by many actions we perform automatically without paying attention, such as pulling out our smartphone to seek rewarding input by perpetually scrolling down the screen. Such habits are thought to gradually develop from initially goal-directed, conscious actions (e.g., using our smartphone to contact a friend) that reliably resulted in desired outcomes. Habit formation is often tied to detection of stimuli predictive of these outcomes and is facilitated by both extended high-rate action repetition (referred to as “overtraining”¹) and relative uncertainty as to when precisely the desired outcome will be achieved.^2,3

The striatum has long been implicated in habit formation,^{4, 5, 6} and its subregions are assumed to fulfill distinct functions in this process. The dorsolateral striatum (DLS) is hypothesized to actively mediate habit learning, supported by ample evidence from rodent experiments which link DLS activity with (overtraining-induced) habit formation and DLS lesion or inactivation with attenuated habitual responding.^{7, 8, 9, 10, 11, 12, 13, 14} The (posterior) dorsomedial striatum (DMS) is hypothesized to control goal-directed behavior and, thus, oppose habitual strategies.^15,16 A third functional domain heavily investigated in the context of reward learning is the ventromedial striatum (VMS), which is involved in motivation and pre-programmed appetitive behaviors.^17,18 The behavioral impact of these three regions is thought to evolve with the gradual development of a habit, reflecting an inter-regional shift of the locus of behavioral control: a dorsal shift away from VMS after early stages of instrumental learning^{9,17, 18, 19, 20} and a relative transfer from DMS to DLS as behavior becomes more ingrained and habitual.^{7,9,13,16,21, 22, 23, 24, 25, 26, 27}

These presumed intra-striatal shifts of the locus of behavioral control are thought to depend on striatal-dopaminergic neurotransmission, which is in line with the anatomy of the dopamine system²⁸ and is corroborated by studies using dopaminergic drugs as reinforcers.^{29, 30, 31} Consistently, ample evidence implicates striatal dopamine in habit formation,^6,9,23,24,32 and performance of ingrained action sequences is reflected in action-chunking activity patterns both in striatal and dopamine neurons.³³ Furthermore, genetic deletion of dopamine-neuron NMDA receptors impairs habit learning,³⁴ as does lesion of nigrostriatal dopamine projections,³⁵ whereas pharmacological enhancement of dopamine transmission biases behavior toward habit formation.³⁶ In humans, dopamine-depleted Parkinson’s patients exhibit deficits in habitual control.^6,37,38 However, although dopamine is indubitably involved in habit formation, its regional specificity and its involvement in the widely assumed regional shift of behavioral control during the transition from goal-directed to habitual behavior is understudied. Conclusions were advanced on the grounds of anatomical studies and sparse functional evidence: one study tracked regionally shifting striatal dopamine signaling across development of a presumed habit reinforced by cocaine,³¹ whereas another study reported no such shift using a natural reinforcer.³⁹ Thus, the central questions that remain are how region-specific and regionally shifting dopamine signals regulate habit formation.

In experimental animals, habit formation is best achieved by either overtraining or employing variable-interval (VI) reinforcement schedules that lead to high response rates and reward-timing uncertainty.^{1, 2, 3,40, 41, 42, 43} Dickinson and collaborators developed a widely applied operational definition that became the gold standard for habit classification:^9,40,41,44 an insensitivity to (short-term) changes in value of action outcomes and/or action-outcome contingency.⁴² However, this definition identifies habits on the failure to satisfy criteria for goal-directed action (negative definition), including the possibility that unforeseen factors compromise integration of changes in action-outcome conditioning.^45,46 Besides, traditionally used habit measures can only be applied once in a given individual and are not suited to detect individual differences, both of which are critical features, since habits develop slowly and recruitment of habitual strategies vary significantly between individuals. To address these limitations, we set out to develop a paradigm that “positively” identifies habits, accounts for individual differences, and may be repeated within a subject to track gradual habit development.

We delineate the differential contributions of relevant striatal subregions in habit formation by measuring real-time fluctuations in extracellular dopamine concentrations in VMS, DMS, and DLS of rats performing in an action-sequence reinforcement task (a “seeking-taking” chain task, which combines VI and fixed-ratio (FR) schedules and overtraining) and subsequently validating the role of region-specific dopamine signaling using optogenetic stimulation. Our findings challenge the traditionally assumed shifting of regional dopamine dynamics and surprisingly indicate that DMS, instead of DLS, dopamine determines the transition to habitual behavior, thereby questioning the idea that DMS opposes habit formation.

Results

Habitual behavior is induced by excessive behavioral repetition (overtraining)^9,11,24,47 or by reinforcing behavior on a VI schedule.^{3,48, 49, 50} Here, we overtrained animals under a chained VI60-FR1 reinforcement schedule, thereby combining the two most prominent paradigms. Rats learned to press a seeking lever (distal to reward) on a VI60 schedule (followed by a 3-s delay in which no levers were extended) and, subsequently, an FR1 taking lever (proximal to reward) that produced a food pellet (Figure 1A). The number of seeking presses under VI reinforcement increased over 10 weeks of training (week 1 versus week 10; t₃₅ = 12.21, p < 0.0001; Figure 1B, left), whereas the number of food-magazine head entries did not (n = 36, Z = −0.440, p = 0.660; Figure 1B, middle). We assessed the probability with which rats checked the magazine for food during the 3-s delay period between retraction of the seeking lever and extension of the taking lever. This probability decreased dramatically across weeks (week 1 versus week 10: n = 36, Z = −5.185, p < 0.0001; Figure 1B, right), demonstrating that animals learned the task structure quickly.

Figure 1.

A seeking-taking chain task to monitor dopamine signaling during habit formation

(A) A VI60:FR1 trial begins with seeking-lever extension. Seeking-lever presses (“distal”) are registered until the final VI60 press (“intermediate”), which triggers seeking-lever retraction, followed by taking-lever extension 3 s later. Approximately 1 s later, animals press the taking lever (“proximal”), it is retracted, and the food-pellet reward is delivered 3 s later, marking the start of the variable inter-trial interval (ITI).

(B) Number of seeking-lever presses (left) and food-magazine head entries (middle) during the VI60 segment. The probability of a food-magazine head entry during the 3-s period following seeking-lever retraction and prior to taking-lever extension (right) decreases rapidly with training, demonstrating task acquisition. Data are mean ± SEM (black) with individual animals (gray).

(C) Rats that were trained for 10 weeks underwent outcome devaluation via sensory-specific satiety (pre-feeding), in which, on two separate days, rats had 1-h ad libitum access to either regular pellets (bottom) or alternative (grain-based) pellets (top) before they were exposed to the seeking-lever (extinction) test. Seeking was unaffected by outcome devaluation, demonstrating that a habit was induced. Subsequently, rats were exposed to both pellets (“choice test”) to demonstrate that outcome-specific devaluation was successful (non-pre-fed pellets preferred; p < 0.001).

(D) Coronal brain sections with recording sites in VMS (blue), DMS (green), and DLS (red), relative to bregma.⁵¹

(E) Representative verification of electrode-tip placement in VMS (blue circle), DMS (green circle), and DLS (red circle) and corresponding electrode tracks (arrows).

(F) Dopamine release to an unpredicted food pellet demonstrates stable electrode sensitivity. Data are median with range in quartiles and mean (+); ns, not significant.

(G) DMS dopamine following seeking-lever extension (distal) was modestly diminished after 10 weeks. No significant differences (ns) were observed in VMS or DLS.

(H) During proximal actions, no significant differences (ns) were detected in dopamine release between weeks 1 and 10. ^∗p < 0.05, ^∗∗∗p < 0.001.

See also Figure S1.

We tested whether behavior persists despite reward devaluation by sensory-specific satiety via a 1-h pre-feeding of regular training pellets (devalued condition) or alternative pellets (valued condition) after 10 weeks of seeking-taking training (Figure 1C, left). Subsequent presentation of the seeking lever under extinction conditions generated no significant difference in the number of seeking-lever presses between valued and devalued conditions (n = 14, Z = −0.785, p = 0.433; Figure 1C, middle), demonstrating that seeking-taking training induced a habit. A “pellet choice test” verified that satiety was specific to the pre-fed pellet type, i.e., each pellet type was devalued effectively (n = 14, valued condition: Z = −3.296, p < 0.001; devalued condition: Z = −3.297, p < 0.001; Figure 1C, right).

We characterized dopamine release longitudinally across weeks, using fast-scan cyclic voltammetry in VMS, DMS, and DLS. Electrode implantation induced minimal brain damage, and placement was verified (Figure 1E). Dopamine release in response to unpredicted delivery of a food pellet was measured as a proxy for electrode sensitivity, which remained stable across weeks in all regions (VMS: t₂₄ = 0.7162, p = 0.4808; DMS: t₁₀ = 1.355, p = 0.2052; DLS: t₁₅ = 1.046, p = 0.3121; Figures 1F and S1). Besides modestly diminished DMS dopamine release upon seeking-lever extension (distal; t₁₀ = 2.267, p = 0.0468), no longitudinal differences were found (distal VMS: t₂₄ = 1.186, p = 0.2473; distal DLS: t₁₈ = 0.1541, p = 0.8793; proximal VMS: t₂₄ = 1.798, p = 0.0848; proximal DMS: t₁₀ = 1.041, p = 0.3223; proximal DLS: t₁₈ = 0.3994, p = 0.6943; Figures 1G and 1H). Thus, regional dopamine release is relatively stable across training, with the only task-relevant changes occurring in DMS.

To overcome limitations of traditional habit testing, we developed a test during which both seeking and taking levers were extended simultaneously for 1 min under extinction conditions (Figure 2A, left), allowing rats the choice to engage with the lever proximal to reward delivery (taking; goal-directed choice) or the lever at the distal to reward delivery (seeking; habitual choice). In week 1, rats exhibited a preference for the taking over the seeking lever, but by week 10, this preference had shifted to the seeking lever (n = 36; week 1: Z = −3.540, p = 0.0004; week 10: Z = −4.357, p < 0.0001; Figure 2A, right). This temporal evolution of lever preference was highly reliable between different rat cohorts (Figure S2A). Extinction effects due to repeated testing are unlikely because (1) the time spent in regular training exceeds preference testing almost 500 times and (2) the frequency of preference tests did not influence the habit readout (Figures S2A and S2B).

Figure 2.

A novel preference test to measure the development of habits

(A) Habit formation was tested by presenting both levers simultaneously for 1 min under extinction conditions (left). In week 1 of VI60:FR1 seeking-taking training, rats preferentially pressed the taking lever, but by week 10, preference had shifted to the seeking lever (right).

(B) In the habit group, the number of food-magazine head entries remained stable (top) across weeks, but rats switched from a taking- to a strong seeking-lever preference (middle) and exhibited more seeking than taking presses in week 10 (bottom).

(C) Food-magazine head entries remained stable in the no-habit group (top), but rats switched from a taking-lever preference to no preference (middle) and exhibited no significant difference between seeking and taking presses in week 10 (bottom).

(D and E) Habitual rats performed more seeking-lever presses (D) and fewer taking-lever presses (E) than non-habitual rats in week 10, but not in week 1.

(F) The seeking-taking preference index demonstrates a higher seeking-lever preference in habitual rats compared to non-habitual rats in week 10, but not in week 1.

(G) The conditioning “logic” translated from training to preference test: food-magazine head entries were executed more frequently after taking-lever presses than after seeking-lever presses, both in habitual and non-habitual rats. Histogrammed data with mean ± SEM shown separately.

(H and I) Linear regression analyses demonstrate associations of the seeking-lever index with other habit-like behavior during the preference test, i.e., (H) a positive correlation with seeking-lever stickiness and (I) a negative correlation with food-magazine head-entry probability. Data are mean ± SEM, in some panels combined with individual animals (open circles). ^∗p < 0.05, ^∗∗∗p < 0.001.

See also Figures S2 and S3.

To capitalize on individual differences, we classified animals (same data as in Figures 1C–1H) into two equal-sized groups based on their relative seeking-taking preference in week 10 (i.e., number of seeking minus number of taking presses, divided by total number of presses, as shown in Figure 2F), which is identical to a median split. There were no significant changes in number of head entries into the food magazine over time in either group (Figures 2B and 2C [top]). Both groups displayed a taking-lever preference in week 1 (habit: n = 18, Z = −1.810, p = 0.0370; no habit: n = 18, Z = −3.312, p = 0.0002; Figure 2C), but only the habitual animals developed a steadily increasing seeking-lever preference (habit group: lever, F_1,44 = 58.17, p < 0.0001; lever x week interaction, F_4,98 = 42.82, p < 0.0001; no-habit group: lever, F_1,46 = 1.206, p = 0.2779; Figures 2B and 2C [middle]) and exhibited this preference in week 10 (habit: n = 18, Z = −3.724, p < 0.0001; no habit: n = 18, Z = −1.491, p = 0.1454; Figures 2B and 2C [bottom]). Habitual animals showed more seeking presses (U(n_habit = 18, n_no-habit = 18) = 11.500, Z = −4.765, p < 0.0001; Figure 2D) and fewer taking presses (U(n_habit = 18, n_no-habit = 18) = 1.500, Z = −5.098, p < 0.0001; Figure 2E) compared to non-habitual rats in week 10, but not week 1 (seeking: U(n_habit = 18, n_no-habit = 18) = 106.500, Z = −1.758, p = 0.0790; Figure 2D; taking: U(n_habit = 18, n_no-habit = 18) = 153.500, Z = −0.269, p = 1.0000; Figure 2E). Consistently, grouping the rats based on week-10 performance (Figure 2F) separated habit from no-habit animals in week 10 (U(n_habit = 18, n_no-habit = 18) = 0.000, Z = −5.144, p < 0.0001), but not week 1 (U(n_habit = 18, n_no-habit = 18) = 115.000, Z = −1.487, p = 0.1370;). Across weeks 1 and 10, seeking-lever presses (n = 18, Z = −3.724, p = 0.0006; Figure 2D) and seeking-preference index (n = 18, Z = −3.724, p = 0.0006; Figure 2F) increased in habitual animals, whereas taking responses decreased (n = 18, Z = −3.725, p = 0.0006; Figure 2E). In non-habitual animals, seeking responses (n = 18, Z = −3.680, p = 0.0005; Figure 2D) and seeking-preference index (n = 18, Z = −3.636, p = 0.0005; Figure 2F) also increased, albeit to a lesser extent than in habitual rats, whereas taking responses did not change (n = 18, Z = −0.104, p = 0.9175; Figure 2E).

During lever-preference tests, habitual and non-habitual rats made more head entries into the food magazine after pressing the taking lever than the seeking lever (Figure 2G) in weeks 1 (habit group: t₃₂ = 7.542, p < 0.0001; non-habit group: t₃₄ = 8.488, p < 0.0001) and 10 (habit group: t₂₇ = 2.510, p = 0.0184; non-habit group: t₃₄ = 6.202, p < 0.0001). Thus, both groups learned the task structure and translated it from training sessions (levers presented sequentially) to the test situation (levers presented simultaneously). Habit and no-habit rats did not differ in number of head entries that followed seeking- or taking-lever presses in weeks 1 (seeking: t₃₂ = 1.506, p = 0.4257; taking: t₃₄ = 1.430, p = 0.3238; Figure 2G, left) or 10 (seeking: t₃₄ = 0.5440, p = 0.5900; taking: t₂₇ = 2.393, p = 0.0956; Figure 2G, right). Notably, repeated presses on the same lever were much more prevalent than seeking-taking action sequences in both habit and no-habit animals in both weeks 1 and 10 (Figures S3A–S3C). Thus, simultaneous lever presentation during the preference test is marked by rats “getting stuck” on one of the levers (no-habit rats on the taking lever [Figure S3B] and habit rats on the seeking lever [Figure S3C]) rather than performing action sequences involving both levers (Figure S3A).

We validated the preference test by carrying out regression analyses, testing for a relationship between the seeking-taking preference index (shown in Figure 2F) and other “habit-like” behaviors displayed during the test. The preference index correlated positively with the probability to repeat seeking-lever presses (R value = 0.6797, p < 0.0001; Figure 2H) and negatively with the probability to make food-magazine head entries (R value = −0.4301, p = 0.0088; Figure 2I).

Classifying rats as habitual or non-habitual based on week-10 preference-test performance, revealed that the two groups perform differently during seeking-taking training (Figures 3A and S4): the frequency of “distal” seeking lever presses across VI training weeks was higher in habitual rats (group, F_5,34 = 18.48, p = 0.0001; group x week interaction, F_5,170 = 3.154, p = 0.0095). Conversely, the frequency of distal food-magazine head entries during VI was lower in habitual rats (group, F_1,32 = 10.66, p = 0.0026). In contrast, we observed no differences in head-entry frequency outside VI when levers were retracted (group, F_1,32 = 0.0003168, p = 0.9859) and in “intermediate” head-entry probability during the 3-s period following seeking-lever retraction, prior to taking-lever extension (group, F_1,34 = 1.463, p = 0.2348). Together, these data suggest that habit animals exhibit a greater affinity to the seeking lever (habit-like behavior) paired with less outcome checking (goal-directed behavior) and that group differences are specific to task-relevant actions and do not reflect discrepancies in general activity.

Figure 3.

Diametrically opposing DMS dopamine signals in habitual and non-habitual rats during distal and proximal reward seeking

(A) Habitual animals showed a higher frequency of distal seeking-lever presses during the VI60 trial segment compared to non-habitual rats (middle left). Conversely, habitual animals exhibited fewer food-magazine head entries during VI60 (middle right). However, this difference in head entries was not present outside VI60, when no levers were extended (left). Similarly, no group differences were observed for the head-entry probability during the 3-s period following seeking-lever retraction and prior to taking-lever extension (right). Data are mean ± SEM.

(B) Dopamine release to an unpredicted food-pellet delivery (proxy for electrode sensitivity) did not differ between groups. Data are median with range in quartiles and mean (+); ns, not signifcant.

(C) During distal actions (start of VI60), no significant dopamine group differences were observed in VMS or DLS. However, DMS dopamine was greater in habitual rats in both weeks 1 and 10. Data are mean + SEM; traces are aligned to seeking-lever extension.

(D) During proximal actions, no significant group differences in dopamine were observed in VMS or DLS. However, again, DMS dopamine differed between habitual and non-habitual rats, but in contrast to distal actions, proximal events significantly decreased dopamine in habitual animals in both weeks 1 and 10. Data are mean + SEM; gray-shaded areas indicate approximate final seeking-press timing.

(E) To evaluate the association of dopamine with reward seeking (distal actions: lever presses early in the VI60 segment), trial-by-trial correlations were calculated for week 1 (top) and week 10 (bottom). Gray bars display the animals’ R values distributed over given histogram brackets. Each circle represents a single animal (habitual in orange, non-habitual in brown), and significant correlations are depicted as filled circles and non-significant correlations as empty circles. VMS demonstrated the closest relationship between dopamine and seeking-lever presses. ^∗p < 0.05, ^∗∗p < 0.01, ^∗∗∗p < 0.001.

See also Figure S4.

Dopamine release to an unpredicted food pellet (proxy for electrode sensitivity) did not differ significantly between habitual and non-habitual animals (Figure 3B) in VMS (week 1: t₂₃ = 0.3175, p = 0.7537; week 10: t₂₆ = 1.146, p = 0.2623), DMS (week 1: t₁₀ = 0.7898, p = 0.4480; week 10: t₁₁ = 0.5837, p = 0.5712), and DLS (week 1: t₁₇ = 1.541, p = 0.1417; week 10: t₁₈ = 0.4060, p = 0.6895). Dopamine measurements throughout the 10-week training (Figures 3C and 3D) did not reveal any differences between habitual and non-habitual rats during “distal,” “intermediate,” or “proximal” actions in VMS (distal week 1: t₂₃ = 0.8278, p = 0.4163; distal week 10: t₂₆ = 1.648, p = 0.2228; proximal week 1: t₂₃ = 0.6261, p = 1.0000; proximal week 10: t₂₆ = 0.3088, p = 0.7600) and DLS (distal week 1: t₂₀ = 0.6727, p = 1.0000; distal week 10: t₁₇ = 0.09775, p = 0.9233; proximal week 1: t₂₀ = 0.6011, p = 0.5545; proximal week 10: t₁₈ = 1.922, p = 0.1412). However, distal DMS dopamine concentration was higher in habitual animals in week 10 (t₁₁ = 5.213, p = 0.0006), a surprising effect that was already detectable in week 1 (t₁₀ = 2.438, p = 0.0350). Remarkably, this effect was reversed during intermediate and proximal actions: habitual animals had lower DMS dopamine in both week 1 (t₁₀ = 5.428, p = 0.0006) and week 10 (t₁₁ = 3.663, p = 0.0037). Within-animal trial-by-trial correlations (Figure 3E) revealed that reward seeking in itself, irrespective of training amount, was strongly associated with VMS, but not DMS or DLS dopamine; seeking-lever pressing in itself, therefore, does not bring about the changes in DMS dopamine we observe during habit development. In summary, habitual behavior was correlated with increased DMS dopamine release during distal behavioral responses, which was decreased during responses proximal to reward delivery. Moreover, DMS dopamine differences were already apparent early in training (before habits developed) and were not involved in (motor) performance of distal seeking.

To determine whether augmented DMS dopamine signaling is sufficient in itself to induce habitual behavior, we bilaterally optogenetically stimulated the terminals of DMS-projecting dopamine neurons during seeking presses (Figures 4A and 4B). We controlled for possible reinforcing effects by exposing rats to an open-field arena where presence of the animal in a specific quadrant triggered photo-stimulation (real-time place preference [RT-PP] assay) and observed no significant difference between ChR2 and control rats (expressing EYFP only) in time spent in the active quadrant (t₁₅ = 1.745, p = 0.1014; Figure 4C). In comparison, other studies applying RT-PP via dopamine photo stimulation have reported animals spending about 80% of their time in the photo-stimulated position,^{52, 53, 54, 55, 56} whereas we observed only 45%. Similarly, when rats were given the opportunity to self-stimulate DMS dopamine terminals optogenetically by poking their noses into a device (over the course of 3 days), there was no significant difference in the number of active or inactive nose pokes between ChR2 and EYFP rats (active-port group, F_1,9 = 4.476, p = 0.0635, time x group interaction: F_2,18 = 2.566, p = 0.105; inactive-port group, F_1,9 = 2.127, p = 0.1787, time x group interaction: F_2,18 = 1.046, p = 0.372; Figure 4D), and there was no difference in active and inactive nose pokes within either of these groups (ChR2: poke-port, F_1,12 = 4.514, p = 0.0551, time x poke-port interaction: F_2,24 = 1.395, p = 0.267; EYFP: poke-port, F_1,6 = 0.06194, p = 0.8118, time x poke-port interaction: F_2,12 = 0.016, p = 0.984; Figure 4D). Furthermore, for ChR2 animals, no difference between active and inactive pokes was detected on day 3 (t₉ = 1.576, p = 0.150). In comparison, overall, other studies applying optogenetic self-stimulation of dopamine have reported about 25–40 responses/min,^52,55,57,58 whereas we observed only about 1 response/min on day 1 and 0.15/min on day 3. Taken together, stimulation of dopaminergic DMS terminals was without significant reinforcing effects.

Figure 4.

Optogenetic stimulation of DMS dopamine during seeking accelerates habit formation

(A) AAV injection into substantia-nigra pars-compacta (SNpc) of Th::Cre rats induced ChR2-EYFP or EYFP (control) expression in dopamine neurons. DMS dopamine terminals were optogenetically stimulated bilaterally.

(B) Immunostaining for tyrosine hydroxylase (TH) and EYFP indicates selective virus expression in midbrain dopamine neurons (bottom) and their striatal terminals (top). SNR, substantia nigra pars reticulata; VTA, ventral tegmental area.

(C) Percentage of time spent in the DMS-photo-stimulated (active) quadrant of an open-field arena. No significant difference (ns) was observed between ChR2 and EYFP rats.

(D) Nose pokes into the active port triggered DMS dopamine photo stimulation. We observed no significant differences (ns) within or between ChR2 and EYFP animals in the number of active and inactive nose pokes.

(E) During seeking-taking training, seeking presses triggered DMS dopamine photo stimulation. The conditioning “logic” translated from training to preference test: food-magazine head entries were executed more frequently after taking-lever presses than after seeking-lever presses, both in habitual and non-habitual rats. Histogrammed data with mean ± SEM shown separately.

(F) No significant differences were found in frequency of food-magazine head entries outside VI60, when no levers were extended.

(G) ChR2 animals made more distal VI60 seeking-lever presses (left). No significant difference was found for food-magazine head entries during VI60 (right).

(H) Similarly, no group differences were observed for the head-entry probability during the 3-s period following seeking-lever retraction and prior to taking-lever extension (right).

(I) In week 1, both ChR2 and EYFP rats preferentially pressed the taking lever during the habit test. By week 3, this preference had shifted to the seeking lever for the ChR2 rats, but not the EYFP rats.

(J) The ChR2 group made significantly more seeking-lever responses, but groups did not differ in taking-lever presses. Data in (E) and (F) are mean ± SEM. ^∗p < 0.05, ^∗∗p < 0.01.

During seeking-taking training, each (distal) seeking press triggered a 1 s DMS photo stimulation. The lever-preference test showed that both groups made more food-magazine head entries after taking compared to seeking responses (ChR2: lever, F_1,18 = 22.49, p = 0.0002; EYFP: lever, F_1,12 = 20.10, p = 0.0007; Figure 4E). Furthermore, we detected no differences over time (between weeks 1 or 3) between the number of ChR2 and EYFP rats that performed taking-lever-to-head-entry sequences (group, F_1,15 = 0.003275, p = 0.9551) or seeking-lever-to-head-entry sequences (group, F_1,15 = 0.1307, p = 0.7228), indicating that task-structure representation was stable across training sessions.

During seeking-taking training, ChR2 animals increased their seeking-lever presses during VI60 compared to EYFP controls (group, F_1,15 = 18.27, p = 0.0007, group x week interaction, F_4,60 = 15.20, p < 0.0001; Figure 4G, left). Post hoc analysis revealed that the number of seeking-lever presses was higher in weeks 1–3 (respectively: t₁₅ = 4.546, p = 0.0012; t₁₅ = 4.521, p = 0.0016; t₁₅ = 4.654, p = 0.0015), indicating that optogenetic DMS dopamine stimulation promoted habit formation. In contrast, no differences in frequency of magazine head entries were found between ChR2 and EYFP rats, either during VI (group, F_1,15 = 0.3029, p = 0.5901; Figure 4G, right) or outside VI (group, F_1,15 = 1.088, p = 0.3134; Figure 4F). Therefore, DMS dopamine did not simply invigorate general responding but specifically enhanced habitual responding. No group differences were found for head-entry probability during the 3-s period following seeking-lever retraction and prior to taking-lever extension (group, F_1,15 = 0.4022, p = 0.5355; Figure 4H), indicating that photo stimulation did not affect task-structure comprehension. In comparison, optogenetic effects are not robust in the self-stimulation condition (Figure 4D) but are very robust when applied in conjunction with seeking-lever presses during seeking-taking training (Figure 4G, left). Thus, we conclude that optogenetic effects on seeking-lever preference cannot be explained by reinforcing properties of DMS photo stimulation alone.

DMS photo-stimulation effects also translated into behavior during the lever-preference test. ChR2 animals pressed the seeking lever more than the taking lever across 3 weeks of training (lever, F_1,15 = 7.840, p = 0.0135; Figure 4I, left), whereas EYFP animals did not (lever, F_1,12 = 0.6623, p = 0.4316; Figure 4I, right). When compared to taking-lever presses, post hoc analysis revealed that both groups made fewer seeking-lever presses in week 1 (EYFP: n = 7, Z = −2.366, p = 0.036; ChR2: n = 10, Z = −2.397, p = 0.017), but only ChR2 rats made significantly more seeking- than taking-lever presses in week 3 (EYFP: n = 7, Z = −1.609, p = 0.1077; ChR2:n = 10, Z = −2.805, p = 0.010; Figure 4I). The number of taking-lever presses did not differ between groups (group, F_1,15 = 0.2355, p = 0.6345; Figure 4J, right), but ChR2 rats made more seeking-lever responses than EYFP animals (group, F_1,15 = 7.840, p = 0.0135; Figure 4J, left), where post hoc analysis revealed that the number of seeking-lever presses was higher in week 3 (U(n_ChR2 = 10, n_EYFP = 7) = 1.500, Z = −3.273, p = 0.001). This suggests that augmented DMS dopamine during training contributes to (1) an increased occurrence of seeking, but not taking responses and (2) accelerating habit formation assessed in the lever-preference test.

Discussion

We set out to delineate the role of the striatal dopamine system in habit formation (i.e., automation of reward seeking) and assessed dopamine release in three striatal regions implicated heavily in acquisition and execution of reward seeking: VMS, DMS, and DLS. A 10-week (over)training in a seeking-taking chain task resulted in habitual seeking, as evidenced by reward devaluation via sensory-specific satiety, traditionally employed to identify habits. The sequential nature of the task enabled us to dissect dopamine signaling during distal (seeking initiation) and proximal (seeking completion and reward taking) epochs. Surprisingly, dopamine release was stable between training weeks 1 and 10, the only exception being a modestly decreased distal DMS signal in week 10. To better interrogate the developing habit, we designed a novel lever-preference test. In week 1, animals displayed a taking-lever preference (goal-directed), but with continued training, rats began either to sample both levers equally (no-habit rats) or to develop a seeking-lever preference (habit rats). Habit rats displayed increased DMS dopamine during distal reward seeking and decreased DMS dopamine during actions more proximal to reward delivery (compared to no-habit rats), whereas VMS and DLS dopamine did not differ between groups. Although this DMS “habit signature” was not related to seeking actions themselves (these were more associated with VMS dopamine), we hypothesized it to be causally related to the gradually developing seeking-lever bias. Thus, we optogenetically stimulated DMS dopamine neuron terminals during seeking (during training), which resulted in a nearly instant selective promotion of seeking behavior during training, and accelerated the shift toward seeking-lever preference (in the test). Together, our findings demonstrate that dopamine release in DMS, but not VMS or DLS, predicts future reward-seeking strategies, where augmented distal DMS dopamine promotes the development of habitual reward-seeking (and more pronounced proximal DMS dopamine reflects the propensity toward a non-habitual strategy).

To study neural mechanisms of a gradually developing habit, we introduced a novel (lever-preference) test that, unlike traditionally employed indicators of habits (invoked via satiety-specific devaluation, contingency degradation, or taste aversion), was designed to allow for repeated testing and analysis of individual differences. We validated that seeking-taking training conditions translated to preference-test conditions by demonstrating that taking-lever presses are followed more often by reward-magazine entries than seeking-lever presses, indicating that rats expect reward after taking-lever, but not seeking-lever, presses, and thus learned and generalized the task structure. That training-induced seeking-lever preference is a valid indicator of habit formation was underlined by the fact that it correlated positively with seeking-lever stickiness (another habit-like feature) and negatively with food-magazine entries (presumably a goal-directed action). Our test enables positive identification of habitual performance by measuring an active-choice preference, as opposed to the “traditional” negative identification via exclusion of a goal-directed strategy. Notably, the test consistently and reliably exposed the gradual nature of the change in lever preference across training weeks in several cohorts of animals (Figure S2A). The slow speed at which the preference shift develops is another validating feature (as reported in other seeking-taking studies^59,60), as habits in humans are thought to form slowly via extensive repetition.^3,42,61,62 Moreover, the test demonstrated that within the same animals, via overtraining, habit-like performance can arise out of a behavior that was initially under goal-directed control;^61,62 overtraining has been shown to effectively produce habits, both in single-action and action-sequence tasks, but aspects of action sequences (e.g., initiation or termination) may remain under goal-directed or mixed control (the latter may explain that no-habit rats do not exhibit a lever preference in week 10).^{1,63, 64, 65} Crucially, by chaining two operant behaviors, we temporally separated initiation (distal) and termination (proximal) of seeking (followed by reward consumption: taking), which allowed clear distinction between underlying neuronal mechanisms. Finally, our approach offers additional measures (reward-magazine head entries and different action sequences) that further qualify behavioral strategy (goal-directed or habitual). Taken together, we propose that “habits” are not a unitary concept and can be detected in different ways (see STAR Methods for more discussion). Here, we introduce a valuable addition to traditionally employed habit paradigms.

Although dopamine neurons are well known to contribute to habit formation,^{6,9,10,23,32, 33, 34, 35, 36,66, 67, 68} little is known about their projection-specific activity of the dopamine system therein. Based on the presumed functions of VMS (motivation/Pavlovian behavior), DMS (goal-directed behavioral control), and DLS (sensorimotor/habit learning), we recorded dopamine transients in these striatal domains, hypothesizing that regional dopamine signals throughout habit development would align to these functions. Additionally, based on the generally presumed dopamine-dependent inter-regional shift in locus of behavioral control during habit development (see introduction), we expected task-relevant dopamine signals to shift from VMS to the dorsal striatum and/or from DMS to DLS. An alternative hypothesis suggests that the influence of dopamine subsides altogether with increasing training/automaticity.^23,69,70 However, surprisingly, our results demonstrate that dopamine signaling (1) does not shift inter-regionally (neither across the animal population as a whole nor between individual habit and non-habitual animals) and (2) does not decrease with increasing behavioral automaticity in any of the three regions. In fact, the stability of dopamine signals across 10 weeks of training is remarkable given dopamine’s role as a “learning/teaching” signal (the task is fully acquired within first few weeks). Even more surprisingly, DMS, a region traditionally associated with the control of goal-directed reward seeking, exhibited the most critical features, where an increase in dopamine during reward-seeking initiation was predictive of future habit-like behavior. This habitual distal DMS dopamine surge was paralleled by a depression of proximal DMS dopamine. In contrast, DMS dopamine signals in non-habitual animals were not simply absent but opposite to those of habitual animals: they decreased distally and increased proximally. This result also underlined that DMS dopamine signaling was not compromised in non-habitual rats. Furthermore, although “action vigor” is a potential contributor to DMS dopamine signals, the number of seeking presses in training (i.e., a proxy for vigor) increases over time in both habit and no-habit animals (Figure 3A). Furthermore, DMS dopamine signals are opposite between these groups, which is the case from the beginning of training, and the signals do not correlate with seeking-press frequency (Figure 3E). Thus, it seems unlikely that vigor is itself a major contributor. Together, our data indicate that the temporal distribution (distal/proximal) of dopamine signaling in DMS, but not VMS or DLS, governs whether a behavior becomes habitual or not.

Since the DMS is traditionally not associated with habit formation directly,^{8,13,16,25,27,29} the question is what function dopamine released into the DMS fulfills during habit formation. From the beginning of training, DMS dopamine signals were present during seeking initiation in animals that displayed an affinity to the seeking lever, reflecting a predisposition toward habit formation, which manifested in the preference test only after weeks of training. This suggests that DMS dopamine signals may strengthen the seeking preference incrementally over many repetitions, which then translates into other situations (i.e., the lever-preference test) later. This is corroborated by our optogenetic-stimulation study: optogenetic DMS dopamine stimulation applied during training is reflected in a significant seeking-lever preference only after 2 weeks. Notably, repetition of seeking presses (overtraining) alone was not sufficient to establish a lever preference in itself, since non-habitual rats were also overtrained, suggesting that DMS dopamine is essential to habit formation.

Enhanced engagement with the seeking lever is reminiscent of “sign tracking,” but sign tracking is a Pavlovian trait that does not develop gradually and, thus, does not explain the gradual shift in lever preference across weeks. Furthermore, sign tracking is strongly linked to VMS dopamine. Consistently, VMS dopamine was correlated with seeking-press frequency, whereas DMS dopamine was less so. This suggests that DMS dopamine release does not directly induce seeking behavior (although it does exacerbate it, as evidenced by our optogenetic study), which is supported by the finding that optogenetic manipulation of DMS dopamine (outside of seeking-taking training) did not suffice to robustly reinforce behavior during intracranial self-stimulation and RT-PP.

An intriguing, yet speculative, idea is that in habit animals, the dopamine response to the earliest predictor of reward “propagated back in time” to the extension of the seeking lever (distal), whereas in no-habit animals, this dopamine response “remained” at seeking-lever retraction (a more conservative and precise predictor of reward [proximal]). This way, DMS dopamine stimulates development of the habit-like reward pursuit, since DMS dopamine increased in habit rats and decreased in no-habit rats at the time of seeking-lever presentation and coincided with seeking initiation without causing this behavior directly. In other words, DMS dopamine may be the effector that, during repeated performance, slowly strengthens the association between seeking stimulus/lever and response and thereby may reflect a proclivity to habit formation, consistent with our finding that such DMS dopamine was already enhanced prior to habit development.

Our results challenge the reigning hypotheses of regionally shifting dopamine involvement during habit formation, dopamine independence of habits, and exclusive DLS governance of habits and thereby break with the idea that the DMS simply counterbalances a DLS habit system. Notably, without our in-depth analysis of individual animal behavior across habit development, our dopamine data (averaged across the entire population) would have supported the idea of a weakened goal-directed DMS system that biases animals toward habitual performance and masked the intricate contribution of the DMS dopamine system. Instead, we were able to delineate two distinct rat populations and demonstrate that DMS-dopamine release during seeking initiation critically contributes to the formation of a habit. Intriguingly, DMS dopamine exhibited a double dissociation, where proximal events were accompanied by a signal opposite to distal events (in both habit and no-habit rats). In consideration of the fact that dopamine signaling did not vanish in, or shift between, any of the measured striatal regions, we suggest that habit formation is a decentralized, concerted effort of many collaborating striatal domains including the DMS.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Antibodies
Rabbit anti-GFP (primary antibodies)	Thermo Fisher Scientific	Cat#A-6455; RRID: AB_221570
Chicken anti-TH (primary antibodies)	Aves Labs	RRID: AB_10013440
Donkey anti-rabbit Alexa 488	Jackson Immunoresearch	RRID: AB_2340619; 711-546-152
Donkey anti-chicken Alexa 647	Jackson Immunoresearch	RRID: AB_2340380; 703-606-155
Bacterial and virus strains
AAV5.EF1α -DIO-ChR2-EYFP	Viral Vector Facility, University of Zurich	N/A
AAV5.EF1α -DIO-EYFP	Viral Vector Facility, University of Zurich	N/A
Chemicals, peptides, and recombinant proteins
Phosphate-buffered saline	Thermo Fisher Scientific	Cat#18912014
Normal donkey serum	Jackson ImmunoResearch	RRID: AB_2337258
Bovine serum albumin	Sigma-Aldrich	Cat#A9647-1kg
Sodium azide	Sigma-Aldrich	Cat#S2002-100 g
Triton X-100	Sigma-Aldrich	Cat#X-100-500ml
4% Paraformaldehyde (37% formalin)	VWR	Cat#97064-606
Deposited data
Code	This paper	https://osf.io/j3zaq/
Experimental models: Organisms/strains
Long-Evans rats	Janvier Labs, France	RjOrl:LE
TH-Cre rats on Long-Evans background	Rat Resource & Research Center https://www.rrrc.us/	LE-Tg(TH-Cre)3.1Deis / RRID:RRRC_00659
Software and algorithms
MATLAB version R2018b	MathWorks	https://www.mathworks.com/products/new_products/release2018b.html
IBM SPSS statistics 25	IBM	https://www.ibm.com/nl-en/products/spss-statistics
Prism version 9.2.0	GraphPad	https://www.graphpad.com/scientific-software/prism/
LabVIEW version 2014	National Instruments	https://www.ni.com/nl-nl/shop/labview.html
Med-PC version IV	Med-associates	https://www.med-associates.com/med-pc-v/
QuPath version 0.2.3	Open source	https://qupath.github.io/
Other
45mg Dustless Precision Pellets Rodent, Purified	Bio-Serv	Cat#: F0021
45mg Dustless Precision Pellets Rodent, Grain-Based	Bio-Serv	Cat#: F0165

Resource availability

Lead contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Ingo Willuhn (i.willuhn@nin.knaw.nl).

Materials availability

This study did not generate new unique reagents.

Experimental model and subject details

Animals

Adult male Long-Evans rats from Janvier Labs were used for voltammetry and behavior-only studies. For the optogenetic study, adult male TH-Cre rats (on Long-Evans background⁷¹) were used. Rats (320-410 g) were housed individually and kept on a reversed 12 h light/dark cycle (lights off from 08:00 to 20:00) with controlled temperature and humidity. Rats were food-restricted to 85% of their free-feeding body weight, and water was provided ad libitum. Regular laboratory chow was available in the home cage two h after the end of daily behavioral training (executed during the dark phase), supplementing food intake during training. All animal procedures were conducted in accordance with Dutch and European law and approved by the Animal Experimentation Committee of the Royal Netherlands Academy of Arts and Sciences. For dopamine recordings, 35 animals had at least one functional and histologically verified recording electrode. 16 rats were used to verify whether reward-driven behavior was habitual after 10 weeks of training by traditional satiety-specific outcome devaluation, of which two rats were excluded from analysis, because outcome devaluation failed. 17 rats were used for optogenetic stimulation of dopaminergic neuron-terminals in DMS that had histologically verified virus expression in both hemispheres and placement of optical fibers bilaterally in DMS.

Method details

Stereotactic surgery procedures

Stereotactic surgery was executed as described previously.^31,72 Before surgery, all surgical equipment was sterilized and the immediate surroundings were wiped with 70% ethanol. Rats were anesthetized with 1%–3% isoflurane and placed in a stereotactic frame. Heart rate and breathing frequency where monitored to assess surgical anesthetic depth. Body temperature was monitored and maintained with a heating pad. Following subcutaneous injection with the analgesic Metacam (0.2mg meloxicam/100 g), the scalp was shaved, disinfected with 70% alcohol, and incised to expose the cranium. The site of incision was treated with lidocaine (100mg/mL). For the voltammetric experiment, holes were drilled for three anchor surgical screws, a Ag/AgCl reference electrode, and two custom-made carbon-fiber micro-electrodes⁷³ unilaterally targeting two of three regions: VMS (1.2mm rostral, 1.5mm lateral, and −7.1mm ventral from Bregma⁵¹), DMS (−0.2mm rostral, 2.5mm lateral, and −4.5mm ventral), and DLS (1.2mm rostral, 3.6mm lateral, and −4.5mm ventral). Electrodes were secured to the skull with dental acrylic cement anchored to the surgical screws.

For the optogenetic experiment, holes were drilled for three anchor surgical screws, two optic fibers targeting the DMS bilaterally (−0.2mm rostral, +/−2.7mm lateral, and −3.7mm ventral), and four virus injections in the ventral midbrain across both hemispheres, targeting the substantia nigra pars compacta (−5.2mm rostral, +/−2.4mm lateral, and −7.8mm ventral, and −6.0mm rostral, +/−2.4mm lateral, and −7.6mm ventral). After slowly lowering the microinjector into position, virus was infused at a rate of 0.1μl/min using a microsyringe pump. After infusion, injectors were left in place for 10min and then removed slowly. Optic fibers (diameter of 230μm, custom-made) targeted at the DMS (the dopamine-neuron projection target of interest) were secured to the skull with dental acrylic cement anchored to the surgical screws.

Following surgery, rats received 0.5ml of saline s.c for rehydration and were placed in an incubator for at least 1 h. To monitor any possibility of discomfort or pain and to make sure that the animals properly recovered, we monitored their overall appearance, weight, behavior, and the state of the incision (wound healing) daily for 3 days post-surgery. Surgery and anesthesia duration was shorter than 2 h.

Viral vectors and optogenetic stimulation

For optogenetic stimulation, AAV5.Ef1α-DIO-ChR2-EYFP (0.9μl/injection site, for a total of 3.6μl/rat, titer ∼3x10¹² particles/mL) was injected into the ventral midbrain for Cre-dependent expression of channelrhodopsin in dopamine neurons of TH-Cre rats. AAV5.Ef1α-DIO-EYFP (0.9μl/injection site, for a total of 3.6μl/rat, titer ∼3x10¹² particles/mL) was used as a Cre-dependent fluorophore control. For optogenetic stimulation, we used laser light with a wavelength of 470 nm, a pulse width of 10ms, a frequency of 20Hz, and an intensity of 20mW at the tip of the optic-fiber patch cable (constant illumination).

Operant chambers

All behavioral experiments were conducted in modified modular operant chambers (32x30x29cm, Med Associates Inc., VT, USA), equipped with a food magazine with an integrated light (connected to an automated food-pellet dispenser) flanked by two retractable operant levers, a house light, multiple tone generators, a white-noise generator, and metal grid floors. The wall opposite the food magazine was outfitted with two nose-poke response devices (port with integrated cue lights) located on adjacent panels. Each operant box was surveilled by a video camera. The boxes were housed in metal Faraday cages, that were insulated with sound-absorbing polyurethane foam and ventilated by a fan.

Training on seeking-taking chain schedule of reinforcement

The seeking-taking chain schedule of reinforcement was adapted from Olmstead and colleagues⁷⁴ and Balleine and colleagues.⁷⁵ After rats were placed in the operant chamber, the start of each daily behavioral-training session was marked by illumination of the house light and the sound of the white noise, which both remained on for the duration of the session. During each trial, rats could earn one “purified” food pellet (Bio. Serv. Inc., USA), delivered into the tray of the food magazine, accompanied by a 2 s-illumination of the magazine light. Behavioral training commenced when rats reached 85% of free-feeding body weight. On the first day, animals were habituated to the operant chamber and 30 food pellets were delivered on a variable-interval (VI) schedule of 90 s. On the second day, rats learned to press the taking lever, the proximal link of the chain (i.e., proximal to the food reward), on a fixed ratio 1 (FR1) schedule for an immediate delivery of a pellet, with a maximum of 40 pellets. The following three days, taking-lever presses triggered pellet delivery (FR1) plus lever retraction, and an inter-trial interval (ITI) of 25 s was imposed, followed by re-extension of the lever into the operant box. In the next session, rats learned to press the seeking lever, the distal link of the chain (i.e., distal to the food reward), in order to gain access to the taking lever. Following its extension into the box (marking the start of each trial), one press on the seeking lever (FR1) triggered retraction of this lever, and after a delay of 3 s, the taking lever was extended. One press on the taking lever (FR1) triggered retraction of the taking lever and delivery of a food pellet 3 s thereafter (FR1:FR1 schedule). During the variable ITI of 25 s, both levers remained retracted. In the following sessions, rats were trained on increasing VIs (ranging from 2 s to eventually 60 s) for the seeking lever: VI2:FR1, VI7:FR1, VI15:FR1, VI30:FR1, VI60:FR1. Each VI started with the rat’s first response on the seeking lever, once extended. Subsequent lever presses went without programmed consequences, until the first press after the VI ended, which triggered seeking-lever retraction and taking-lever extension 3 s thereafter. Figure 2A depicts a schematic overview of a VI60:FR1 trial. Sessions ended once rats earned 40 rewards. Rats were trained daily on the VI60:FR1 for 10 weeks. Lever position (seeking/taking) was counterbalanced between rats.

Lever-preference test

Behavioral automaticity was probed regularly (no more than once per week) immediately preceding the daily training session, using a novel “lever-preference” test: After a variable interval of 25 s with only house-light and white noise turned on, animals were exposed to both seeking and taking levers simultaneously for one minute under extinction conditions (recording lever presses and head entries into the food magazine without programmed consequences). During this minute, a preference for the taking lever (i.e., more presses on the taking than seeking lever) was interpreted as a goal-directed behavioral strategy, whereas a seeking-lever preference was taken as evidence for habitual responding.

We aim to establish the lever-preference test as a new way to assess the development of habitual behavior. Our intention is not to emulate another version of an outcome-devaluation procedure or to design a proxy for it. Instead, we introduce an alternative way to track progressive automation of behavior and present a number of points that validate and/or support this test as a tool that identifies a habit:

• 1)Across the 10 weeks of training, the animals’ initial test preference to interact with the taking lever (that is associated with immediate (FR1) food delivery) switches to the seeking lever (that never directly leads to food delivery and never leads to food quickly, but instead only leads to the extension of the taking lever).
• 2)We demonstrate that this preference switch occurs within individual animals.
• 3)The taking preference at the beginning of training is present in animals from both groups (habit and no-habit; Figures 2B and 2C).
• 4)This switch only occurs after a very high rate of action and session repetition (1000s of repetitions in 10 weeks of daily training sessions; Figure 1B). It is widely accepted that habit learning materializes through high repetition of responding. Our paradigm requires many repetitions of the behavior before the habit is detectable (long after the motor skill to lever press had been acquired).
• 5)The slow speed of this switch is comparable to the speed of habit development in human everyday life (Figures 2B and 2C), establishing face validity, in contrast to some other habit tasks that report habits developing as quickly as after 1-4 days of behavioral training.
• 6)Induction of sensory-specific satiety after 10 weeks of seeking-taking training demonstrates that our behavioral training schedule induces a habit as defined and tested by traditional outcome devaluation (and in the traditional way: as a singular group, as opposed to considering individual differences; Figure 1C), establishing construct validity and fulfilling the traditional requirement to classify a behavior as habitual.
• 7)Habitual animals exhibit a higher probability to repeat seeking-lever presses (lever stickiness) during the preference test (Figure 2H), a measure associated with habitual behavior.
• 8)Habitual animals commit fewer head entries into the food magazine during the preference test (Figure 2I), a measure associated with goal-oriented behavior (non-habitual).
• 9)The measures mentioned under points 7 and 8 correlate with seeking-lever preference (Figure 2H and 2I).
• 10)The preference test detects a similar preference progression across weeks independent of the frequency with which the test is applied (Figure S2).
• 11)Rats do not perform head entries into the food magazine during the 3-s waiting period before extension of the taking lever, indicating that the animals learn the “task logic” (i.e., the taking lever needs to be pressed in order to receive food (Figure 1B (right)).
• 12)The “task logic” translates well from the training situation (VI60/FR1) to the preference-test situation (both levers out), as animals in both groups (habit and no-habit) perform substantially more food-magazine entries after taking-lever presses (the lever associated with food delivery; Figure 2G).

We do not provide an extensive cross-validation of our preference test with the sensory-specific satiety data presented in Figure 1C for several reasons: 1) We employed behavioral training under a VI60 reinforcement schedule, which has in fact already been widely validated and widely accepted to induce habits, for both single-action and action-sequence tasks. 2) Data acquired using the reward-devaluation approach is relatively “noisy” and provides little clustering of the population data, as exemplified in Figure 1C (center). 3) We believe that what is commonly referred to as a “habit” is not a unitary concept and has a number of different aspects and that, therefore, there are different ways to approach its detection. An approach that focuses on devaluation-insensitivity criteria is a widely accepted approach, originally put forward by Dickinson and colleagues, and has led to invaluable research findings. However, it does not capture all aspects of a (human) habit. For example, using its underlying logic, a habit can be induced in a single, relatively short behavioral training session in humans, and no more than 4-5 sessions in rodents. Yet, the human behavior to be modeled by this research, is behavior that has slowly developed across weeks or months, or even years. Our paradigm requires weeks of daily training to induce a habit. 4) To our knowledge, sensory-specific satiety devaluation has never been reported as a way to assess individual differences in habit formation. In addition to this type of devaluation being very sensitive to disturbances in general, we think that this approach is actually not suitable to assess individual differences: Following the ad-lib pre-feeding, rats were exposed to the seeking lever for five minutes under extinction conditions during which lever presses were recorded. Each animal was subjected to both valued (pre-feeding with pellets different from training pellets) and devalued (pre-feeding with training pellets) conditions (on separate days); the sequence of conditions was counterbalanced between animals. An important observation is that during the first pre-feeding session, animals eat more than after the second. Thus, whichever condition comes first, valued or devalued, the first session elicits more lever presses relative to the same condition when it occurs second in temporal sequence (in other animals). We have observed this phenomenon reliably in several cohorts of animals, and believe it to be a general phenomenon, despite the fact that, to our knowledge, it has not been reported in the literature. This phenomenon complicates looking at individual differences, as individuals are strongly affected by the sequence of pre-feeding, which restricts the utility of pre-feeding devaluation to group comparisons, where an equal number of animals have undergone each condition first (due to counterbalancing).

Experimental procedures

Experiment 1: Week 10 validation of habit induction by seeking-taking paradigm

One group of rats (n = 14) was tested for habitual responding using a traditional satiety-specific outcome devaluation, after 10 weeks of daily training on the VI60:FR1 seeking-taking chain schedule of reinforcement. Outcome devaluation (sensory-specific satiety) was achieved by giving rats one-h ad-libitum access to either grain-based pellets (used to remove hunger as a motivating factor for lever pressing; the pellets that are used for behavioral training retain their “rewarding” value, this is the “valued condition”), or regular (purified) pellets (used to decrease both hunger and value of the behavioral-training pellets, this is the “devalued condition”) prior to behavioral testing. Food pellets were placed in custom-made metal cups in an empty, otherwise unused home cage (“feeding cage”; water freely available). Following one h of this pre-feeding, rats were exposed to the seeking lever in the operant chamber for five minutes under extinction conditions during which lever presses were recorded. This was subsequently followed by a “pellet-choice” test to determine the efficacy of the pre-feeding procedure, measured by the preference for either pellet type during two minutes of simultaneous ad-libitum access to both types in the feeding cage. Animals were excluded from analyses if they consumed any of the prefed-type pellets during this test (insufficient sensory-specific satiety). Each animal was subjected to both valued and devalued conditions (on separate days); the sequence of conditions was counterbalanced between animals.

Experiment 2: Dopamine measurement during seeking-taking chain training

In a second group of rats (n = 35), we employed fast-scan cyclic voltammetry (FSCV) using chronically implanted carbon-fiber micro-electrodes⁷³ to record rapid changes in dopamine release in different domains of the striatum, during week 1 and week 10 of training on the seeking-taking chain of reinforcement schedule. On recording days, before the start of behavioral training, electrodes were connected to a head-mounted voltammetric amplifier that was interfaced through an electrical swivel above the test chamber (allowing animals free movement; Crist Instrument, MD, USA) with a PC-driven data-acquisition and analysis system (National Instruments, TX, USA).⁷³

Voltammetric scans were repeated every 100ms (10Hz sampling rate). During each scan, the alternating potential at the carbon-fiber electrode tip was ramped linearly from –0.4V versus Ag/AgCl to +1.3V (anodic sweep) and back to −0.4V (cathodic sweep) at 400V/s (total scan time of 8.5ms) and held at −0.4V between scans. Waveform generation, data acquisition, and analysis were carried out on a PC-based system using two PCI multifunction data acquisition cards and software written in LabVIEW (National Instruments). Dopamine is oxidized during the anodic sweep, if present at the surface of the electrode, forming dopamine-o-quinone (peak reaction detected around +0.7V), which is reduced back to dopamine in the cathodic sweep (peak reaction detected around −0.3V). The ensuing flux of electrons is measured as current and is directly proportional to the number of molecules that undergo electrolysis. The background-subtracted, time-resolved current obtained from each scan provides a chemical signature characteristic of the analyte, allowing resolution of dopamine from other substances.⁷⁶

Experiment 3: Optogenetic stimulation of DMS dopamine-neuron terminals

In a third group of rats (n = 17), we optogenetically stimulated DMS dopamine-neuron terminals (bilaterally) in response to seeking-lever presses for three weeks during daily seeking-taking chain training sessions starting under VI30:FR1 and progressing to VI60:FR1. A seeking-lever press triggered a 1 s laser photo-stimulation (20mW, 20Hz, 20x10ms light pulses) followed by a 1 s timeout. Starting 10 s after the first seeking-lever press, only every third seeking-lever press triggered stimulation. Rats were exposed to the lever-preference test (without optogenetic stimulation) regularly to assess whether an approach habit was developing. Laser-stimulation visibility was masked using a custom-made LED mounted on the wall of the operant box emitting blue light pulses (20Hz, 20x10ms).

Experiment 4: Real-time place preference (RT-PP) and intracranial self-stimulation

Prior to experiment 3 (after 8-9 week post-surgery period to allow for sufficient virus expression), rats were exposed to an open-field arena in which an RT-PP assay was implemented. Following a 5min baseline period without photo-stimulation, the presence of the rat in one of the quadrants of the open-field (i.e., active quadrant) triggered photo-stimulation (20mW, 20Hz, 20x10ms light pulses) of DMS dopamine-neuron terminals during a 15min assay. The position of the active quadrant was assigned randomly. Time spent in the active quadrant was recorded. Laser-stimulation visibility was masked using a custom-made LED mounted on the wall of the open-field emitting blue light pulses (20Hz, 20x10ms).

After Experiment 3, rats were tested in the same operant box for three days in an operant self-stimulation experiment, where nose pokes into the active nose-poke device triggered a 1 s photo-stimulation (20mW, 20Hz, 20x10ms light pulses) of DMS dopamine-neuron terminals, followed by a 1 s timeout. Nose pokes into the inactive nose-poke device were recorded without programmed consequence. The location of active and inactive nose-poke devices was counterbalanced between rats. Laser-stimulation visibility was masked using a custom-made LED mounted on the wall of the operant box emitting blue light pulses (20Hz, 20x10ms).

Histological verification of recording sites, optic-fiber placement, and virus expression

After completion of experiments, rats were deeply anesthetized using a lethal dose of pentobarbital (14mg/100 g), and transcardially perfused with saline followed by 4%-paraformaldehyde (PFA). In animals with electrode implants, recording sites were marked with an electrolytic lesion before perfusion. Brains were removed and post-fixed in PFA for one day after which they were placed in 30% sucrose for cryoprotection. The brains were rapidly frozen using an isopentane bath, sliced on a cryostat (50-μm coronal sections, −20°C), and stained with Cresyl violet to aid visualization of anatomical structures, electrode-induced lesion or virus-infusion sites, and optic-fiber placement.

For the optogenetic experiment, sliced serial coronal sections (50μm) were stored in phosphate-buffered saline (PBS) with 0.02%-sodium azide at 4°C until further use. For immunohistochemical stainings, sections were incubated in blocking buffer (5% bovine serum albumin (BSA), 5% normal donkey serum (NDS), and 0.2%-triton-X in PBS) for 1 h, followed by overnight incubation in primary antibodies rabbit anti-GFP (1:1000, Invitrogen A6455, Thermo Fisher) and chicken anti-TH (1:1000, Aves TYH) in blocking buffer. All sections were washed 4x (10min each) in PBS and then incubated for 1 h in blocking buffer containing fluorescent Alexa-conjugated secondary antibodies (1:1000, donkey anti-chicken Alexa 647 and donkey anti-rabbit Alexa 488; Jackson Immunoresearch 711-546-152 and 703-606-155). Sections were washed again 4x in PBS, with the second wash containing DAPI (nuclei stain; Sigma Aldrich D9542), then mounted onto slides and coverslipped with Mowiol mounting medium. Slides were imaged using three fluorescent channels of the ZEISS Axio Scan.z1 slide-scanner at 10x magnification for qualitative expression and targeting verification. Sections were subsequently imaged in z stacks with a confocal microscope (Leica TCS SP5) to verify specificity of viral expression.

Quantification and statistical analysis

Data analysis

Dopamine-related current was isolated from the voltammetric signal by chemometric analysis using a standard training set, based on electrically stimulated dopamine release detected with chronically implanted electrodes; resulting dopamine concentration was estimated based on average post-implantation sensitivity of electrodes.⁷³ Resulting dopamine traces were smoothed with a moving 10-point median filter prior to analysis of average concentration. Dopamine concentration was averaged over 10 s (approximate duration of the observed phasic signal) following seeking-lever extension (distal link of the chain) and over 7.5 s before and 2.5 s after reward delivery (proximal link of the chain); resulting averages were compared to the average concentration over the 2 s before each trial (baseline). Prior to each FSCV recording session, two unpredicted food-pellet deliveries (spaced apart by two minutes) confirmed electrode viability to detect dopamine, as well as electrode sensitivity over time. Data analysis was performed using MATLAB R2018b (The Mathworks, Inc., MA, USA).

Statistical analysis

Individual electrochemical signals were averaged across sessions and then across animals. For comparison of electrochemical data, individual behavioral data were binned into weeks and then averaged across animals. Corresponding electrochemical and behavioral data from week 1 were compared to week 10 using paired Student’s t test. When separated into groups, behavioral data were analyzed using multivariate ANOVAs with group, lever, nose-poke port, and week as factors. When main effects or interactions were significant, post hoc analyses were carried out and P values were adjusted according to the Holm-Bonferroni correction method for multiple comparisons. For the optogenetic self-stimulation experiment (Figure 4D), we used two-way ANOVAs, with nose-poke port (active versus inactive) or group (ChR2 versus EYFP) as factors, and time (day) as repeated-measurement. Non-parametric testing was conducted when data was not normally distributed. Graphical representations were made using Prism (GraphPad Software, La Jolla, CA, USA) and MATLAB. Statistical analysis was carried out using Prism and SPSS (version 25; Chicago, IL, USA) and MATLAB. Data collection and analysis were not performed blind to the conditions of the experiments. Statistical significance was set to p < 0.05. All data are presented as mean or median ± SEM.

Sample size was 35 rats for dopamine recordings (Figures 1 and 3), 16 for testing habitual behavior by traditional outcome devaluation via sensory-specific satiety (Figure 1), and 17 for optogenetic stimulation of dopaminergic neuron-terminals in the DMS (Figure 4), as described under “Animals.” To demonstrate reliability of the lever-preference test, we used seven cohorts of rats with different sample sizes (ncohort1 = 12, ncohort2 = 12, ncohort3 = 13, ncohort4 = 16, ncohort5 = 6, ncohort6 = 14 ncohort7 = 6; Figure S2).

Published: February 7, 2022

Data and code availability

• •All data reported in this paper will be shared by the lead contact upon request.
• •The code used for this study is available at https://osf.io/j3zaq/ and, if necessary, more detailed information is available from the corresponding author upon request.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.