Examining a punishment-related brain circuit with miniature fluorescence microscopes and deep learning

Abstract

In humans experiencing substance use disorder (SUD), abstinence from drug use is often motivated by a desire to avoid some undesirable consequence of further use: health effects, legal ramifications, etc. This process can be experimentally modeled in rodents by training and subsequently punishing an operant response in a context-induced reinstatement procedure. Understanding the biobehavioral mechanisms underlying punishment learning is critical to understanding both abstinence and relapse in individuals with SUD. To date, most investigations into the neural mechanisms of context-induced reinstatement following punishment have utilized discrete loss-of-function manipulations that do not capture ongoing changes in neural circuitry related to punishment-induced behavior change. Here, we describe a two-pronged approach to analyzing the biobehavioral mechanisms of punishment learning using miniature fluorescence microscopes and deep learning algorithms. We review recent advancements in both techniques and consider a target neural circuit.

1.1. Introduction

Rodent models of operant conditioning offer theoretical insight into the mechanisms underlying substance use disorder (SUD) in humans [1]. Laboratory rats can be trained to perform an operant response (e.g., pressing a lever) to earn a reward such as food, social interaction, or intravenous drug infusion [2–5]. An important aspect of operant conditioning procedures is how animals subsequently learn to stop this learned responding when some contingency in the environment changes [6]. One such scenario is punishment, in which the learned response now earns an aversive stimulus—such as electrical shock—in addition to the previously-conditioned reinforcer [7–9]. In many punishment procedures, responding declines to near-zero after only a few sessions [9–11].

Punishment has recently generated interest as a model of learning to stop responding that is relevant to the study of maladaptive behaviors involved in SUD [7,12]. Although punishment is not feasible as a therapeutic intervention, punishment-based contingencies often motivate attempts to abstain from problematic behavior. For instance, humans may abstain from drug use due to a desire to avoid various negative financial, legal, social, and health consequences [13,14]. Thus, studying punishment is important to understand how organisms learn to abstain from maladaptive behaviors, as well as the circumstances under which abstinence fails and maladaptive behaviors return.

Punishment is a component of many operant conditioning procedures modeling risky decision-making or compulsive reward-seeking under reward/punishment conflict. Various punishment-based procedures involve concurrent responses with different reward magnitudes and punishment probabilities [15,16], responding for optogenetic stimulation of midbrain dopamine neurons [17], consecutive responses in a reward-seeking/taking chain [18–21], or distinct within-session trial blocks of different punishment probabilities [22,23]. These procedures have been reviewed elsewhere [24].

The present review concerns mechanisms of renewal or context-induced reinstatement following punishment. In a typical procedure, an operant response is conditioned in one context, subsequently punished in another context, then tested in both contexts (Figure 1). Responding is suppressed in the punishment context, but robustly recovers in the original training context [8,9,12,25–29]. Context-induced reinstatement is a useful framework for studying punishment learning because the underlying associative structure is well-characterized. Although much of this work is based on extinction [2,30–32], recent research indicates that punishment is governed by a similar associative structure [9,11,25,33]; both punishment and extinction are forms of inhibitory operant learning that are specific to the context in which they are acquired [6,8,9,25,33]. It is worth noting that “context” can be more than a physical environment, and that similar processes support learning about specific cues [11,34] as well as specific actions in a heterogenous chain [35]. Finally, context-induced reinstatement has been reliably demonstrated with both food- and drug-reinforced responses [2,4,9,12,25–27], and in both male and female rats [9,25] (however, sex differences have been reported in other punishment-based procedures [22,36–38]).

Figure 1.

Schematic of context-induced reinstatement following punishment.

Note. R = response, -- = nothing, ITI = intertrial interval. Filled arrows represent reinforced responses, empty arrows represent non-reinforced responses. a) Example trial structure from response training in Context A. For an animal responding on a random interval (RI) 30-s schedule, responses can earn food (and an associated reward cue) or nothing. b) Example trial structure from punishment training in Context B. Responses may earn food, shock, both, or neither.

Two parallel topics characterize research on context-induced reinstatement following punishment: 1) how punishment learning influences behavioral suppression in the punishment context, and 2) how responding recovers outside the punishment context. Various brain regions have been implicated in both punishment-induced response suppression and context-induced reinstatement following punishment, including basolateral amygdala, nucleus accumbens (NAc), orbitofrontal cortex, lateral hypothalamus, ventral subiculum, and the prelimbic (PL) and infralimbic (IL) cortices of the medial prefrontal cortex [10,12,13,20,26,27,29,33,39,40]. However, much of this research has utilized loss-of-function manipulations that do not necessarily capture ongoing changes in neural circuitry related to punishment learning. Current understanding of the biobehavioral mechanisms of punishment learning would benefit from simultaneous analysis of moment-to-moment changes in both behavior and the brain. Miniature fluorescence microscopes (miniscopes) can be used to record neuronal activity in freely moving animals [41–44], and this imaging data can be combined with deep learning approaches to behavior analysis to identify behavioral microstates correlating with neuronal activity [45–51]. This two-pronged approach is a powerful tool for studying punishment learning. Here, we describe a punishment-relevant corticostriatal brain circuit, review recent advancements in both miniscope technology and deep learning behavior analysis, and discuss application of these techniques to the study of punishment.

2.1. A relevant corticostriatal circuit

Much of the research on the neural bases of punishment learning is based on the context-induced reinstatement framework described above [12,27–29,33]. Context-induced reinstatement is also a fundamental characteristic of operant extinction [2,4,52]. In extinction, an animal learns to stop responding because the response is no longer reinforced. Recent research suggests that punishment and extinction involve similar behavioral mechanisms [8,11,13,25], and may be similar manifestations of a common process of learning to inhibit an operant response [6,13]. Accordingly, there is evidence that punishment and extinction share overlapping neural substrates [33] (but see [29]). As the neural bases of extinction have received more study than those of punishment [52], it can be useful to refer to the extinction literature when anticipating the roles of certain brain regions in punishment.

Infralimbic cortex and nucleus accumbens have both been implicated in studies of context-induced reinstatement following punishment [27,33] as well as extinction [33,53–56]. Broomer [33] reported parallel involvement of IL in the context-specific suppression of responding following either punishment or extinction training. Rats learned to lever-press in one context (A), then underwent either punishment or extinction training in a second context (B). During subsequent tests in each context, IL inactivation increased responding in Context B in both groups (punished and extinguished), suggesting impaired retrieval of context-specific punishment or extinction learning. These results aligned with prior evidence that IL is necessary for the consolidation and retrieval of operant extinction learning [53,54,57], but contrasted with those of Jean-Richard-dit-Bressel and McNally [10], who reported no effect of IL inactivation on punishment learning. The reason for this discrepancy remains unclear but may involve a context switch; Broomer conducted training and punishment in separate contexts whereas Jean-Richard-dit-Bressel and McNally did not.

In both punishment and extinction-based procedures, IL is thought to influence activity in various subcortical brain regions to control whether or not an animal responds [52]. One such region is NAc, which is comprised of shell and core subregions. In many cases, shell and core likely work in concert; it is theorized that limbic signals become refined through laterally spiraling projections ascending from shell toward core [58]. NAc core is thus considered “downstream” of NAc shell and has been proposed as a major “motor output” with projections toward motor areas of thalamus and premotor/motor cortex [58,59]. Therefore, it is thought that NAc shell may have greater influence on higher order motivated cue processing whereas NAc core may be involved as a final “gatekeeper” for the motoric expression of motivated behaviors [59–61].

Whereas IL activity is typically associated with response suppression following punishment or extinction, NAc activity is frequently associated with response recovery in renewal/context-induced reinstatement procedures [27,55,56,62] (but see [63]). This function depends on activity at D1-family dopamine receptors [27,55]. The degree to which the shell and core subregions differ in this role is unclear. Some studies suggest that shell and core are similarly involved in context-induced reinstatement following extinction of alcohol seeking [56] and cocaine seeking [64,65]. Others, however, have identified such a role for shell only. Bossert et al. [55] found that D1 antagonism in NAc shell reduced context-induced reinstatement after extinction of heroin seeking, but D1 antagonism in NAc core did not. Similarly, Cruz et al. [62] reported that although neuronal ensembles in both shell and core were activated during context-induced reinstatement, only inactivation of those ensembles in shell reduced responding during a subsequent test.

The functional distinction between NAc shell and core in punishment-based procedures is similarly unclear. Marchant and Kaganovsky [27] reported similar effects of shell and core D1 antagonism on context-induced reinstatement of punished alcohol seeking but suggested that core involvement may have corresponded to processing of discrete reward cues (alcohol infusions were accompanied by a compound light/tone cue). Others have suggested that the functional distinction between D1-related signaling in core and shell involves cue-based processing in the former and context-based processing in the latter [56,66]. Piantadosi et al. [23] found that NAc shell inactivation selectively increased responding during punished trials in a reward/punishment conflict task—consistent with the results of Marchant and Kaganovsky [27]—whereas NAc core inactivation reduced responding on all trial types, suggesting that core activity may have been generally involved in the cue-based discrimination of different trial types (rewarded vs. punished). Thus, although evidence implicates both NAc shell and NAc core in context-induced reinstatement following punishment or extinction, the evidence for NAc shell involvement is more consistent (see also [28,40]).

As noted in the introduction, two parallel processes underlie context-induced reinstatement following punishment: 1) the suppression of responding in the punishment context; and 2) the renewal of responding in the training context. Separately, IL and NAc shell appear to be involved in response suppression and response recovery, respectively. Together, they may form a circuit that mediates both processes [67]. Quiroz et al. [68] reported that electrical stimulation of IL increased extracellular concentrations of glutamate and dopamine in the posterior medial NAc shell, suggesting that IL activity is capable of influencing dopamine signaling in NAc shell. LaLumiere et al. [69] reported that intra-IL AMPA activation reduced cue-induced cocaine seeking following extinction, whereas intra-NAc shell AMPA antagonism reversed this suppression. Similarly, Nett et al. [70] identified a role for IL-to-NAc shell projections in the encoding of extinction. During extinction training, closed loop optogenetic inhibition [57] of IL-to-NAc shell projections immediately after an unreinforced lever press increased responding overall, suggesting impaired encoding of extinction learning. Taken with the evidence reviewed above, these results support the perspective that IL-to-NAc shell projections may be a critical component of a circuit mediating response suppression versus response recovery following punishment.

Context-induced reinstatement is a useful framework for examining the behavioral and neural mechanisms of abstinence and relapse [1,71]. However, as noted in the Introduction, experimental applications of this framework can be narrow in focus, utilizing discrete neural manipulations and largely focusing on behavior during brief test sessions at the end of the experiment. In other words, this procedure has traditionally offered limited insight into ongoing changes in neural circuitry throughout acquisition and punishment of operant conditioning. However, recent advancements in both in vivo calcium imaging with miniscopes and behavior analysis with deep learning algorithms offer significant promise in characterizing ongoing learning and behavior.

3.1. Recent advances in miniscope imaging

Miniscopes function similarly to typical fluorescence microscopes and can be used to longitudinally image the activity of hundreds of neurons in a single subject. An animal is injected with an adeno-associated virus containing a genetically encoded calcium indicator (GECI) and implanted with a gradient index (GRIN) lens in the brain region of interest. Although superficial structures can be imaged directly via cranial windows [72], a GRIN lens provides access to deep brain structures that would otherwise be obscured by brain tissue [73–75]. When neurons in that region are active, the resulting intracellular release of calcium produces a fluorescent signal when illuminated with light of a specific frequency. This fluorescence is then refracted to the miniscope via the GRIN lens [73]. Current GECIs such as GCaMP8 have fast half-rise times (approximately 2 ms) that make them well-suited for synchronization with video data collected at a sub-second timescale (see section 4.1) [76] Early investigations with miniscopes utilized mice [45,75,77–82] due in part to their transgenic flexibility [83–85]. This allowed researchers to image from genetically distinct cell types but offered little opportunity for imaging during complex learning and memory paradigms, where rats are favored [86–88]. However, genetic engineering advancements have generated readily available rat transgenic lines [89–92]. This has led to the adaptation of miniscopes to rats. Rats are approximately 10 times larger than mice and can handle heavier head-mounted equipment without interference with behavior. Thus, the same basic miniscope developed in mice can be augmented with additional optical components for optogenetics [93], multiple LEDs for observing multiple genetic subtypes [94], or an onboard “liquid lens” [95] that can be used for multi-plane imaging [96].

The development of custom and commercial miniscopes for rat subjects has allowed researchers to examine activity in a variety of brain regions during complex behavioral tasks. Here, single-photon miniscopes are typically favored due to their lightweight design, low cost, and potential for large field of view (FOV). Both custom and commercial options allow for in vivo calcium imaging and data analysis using open source (e.g., EXTRACT or CaImAn [97,98]) or purchased software (for Inscopix [99]), respectively. Custom miniscope preparations are cost-effective, allowing for investment in additional miniscopes and equipment (see [100]), but require expertise and training. Commercial packages offer customer service and “off-the-shelf” components and software at increased cost. Recently, Hart et al. [96] used the UCLA miniscope (~1 mm² FOV) to record neural activity in anterior cingulate cortex (ACC) during effort-based decision making and found that ACC activity related to a high-value/high-effort reward was attenuated in the presence of a low-value/low effort alternative. The UCLA miniscope has also been used to image from rat hippocampus: Wirtshafter and Disterhoft [44] detected changes in CA1 hippocampal place cells as rats changed positions on a linear track, and observed a greater percentage of active place cells in rats compared to mice. Guo et al. [41] adapted the UCLA miniscope to provide a larger (~2.54 mm²) FOV, and used this “MiniLFOV” to record from hippocampal place cells in freely moving rats running on a track. Similarly, the single-photon Inscopix nVista miniscope (< 1 mm² FOV) has also been used to record from CA1 place cells [101], and both the Inscopix nVista and UCLA miniscopes have successfully recorded from other subcortical targets including ventral posteromedial thalamic nuclei [102] and NAc [103].

Although single-photon miniscopes are lightweight and cost effective, they are limited by lower resolution and higher background fluorescence than two- and three-photon miniscopes, which offer reduced photobleaching and background fluorescence, and improved cell viability, and axial resolution [104]. Although multi-photon miniscopes often sacrifice FOV and wireless capability [78,105], various miniscopes have been developed that increase FOV. Zong et al. [105] recorded activity from mouse visual cortex, entorhinal cortex, and hippocampus using a two-photon miniscope with an enlarged FOV of approximately 0.25 mm² (relative to 0.17 mm² in a previous version), and Zhao et al. developed a two-photon miniscope with a ~1 mm² FOV similarly capable of imaging from dorsal hippocampal CA1 neurons in combination with an implanted GRIN lens in freely moving mice [106]. Zhao and colleagues have also developed a three-photon miniscope, extending imaging depth to record from hippocampal CA1 neurons without an implanted GRIN lens [107]. Other three photon miniscopes have been used to record from layer 4 and 6 cortical neurons in freely moving mice [108].

In summary, single-photon miniscopes combined with GRIN lenses remain the favored approach to imaging in rat subjects thanks to their light weight, low cost, wireless capability, and large FOV. However, multi-photon miniscopes are being developed, and have already produced imaging data from freely moving mice.

4.1. Deep learning methods for behavior analysis

The ability to record both cortical and subcortical neuronal activity longitudinally during complex behavioral tasks makes miniscope imaging useful for addressing the questions about punishment learning posed in the Introduction: How does punishment learning influence behavior in the punishment context? How does responding recover outside the punishment context? As an animal learns and its behavior changes, miniscopes can image corresponding changes in neural activity in real-time. Traditionally, imaging data is associated with metrics of animal behavior such as freezing, lever-pressing, cue presentation, reward delivery, and food cup entry via TTL signals or even manually scored behavior (e.g., [109]). However, recent advances in deep learning approaches have expanded the temporal scope and resolution of behavior analysis and may better characterize corresponding neural activity.

Deep learning methods for behavior analysis derive behavioral motifs from pose estimation data generated by algorithms such as DeepLabCut (DLC) [49] and SLEAP [47]. These algorithms use large volumes of behavioral video data to train pose estimation models that refer to a series of experimenter-labelled points (e.g., the animal’s head, tail ears, trunk, etc.) to identify where in the operant chamber the animal is and how it is positioned. Various deep learning methodologies can then use this pose estimation data to identify behavioral motifs in subsequent video data. Supervised approaches identify specific experimenter-defined behavior labels, unsupervised approaches such as HUB-DT [110] identify behavioral “clusters” with little experimenter input, and self-supervised approaches such as Deep Behavior Mapping (DBM) [45] and VAME [50] generate labels from the data itself. These classifications have been reviewed in more detail elsewhere [46,110].

Deep learning approaches have been applied to a variety of behavioral analyses. For example, VAME has been used to assess behavioral differences between wildtype mice and APP/PS1 transgenic mice (a transgenic model of Alzheimer’s disease) [50]. Lindsay et al. [111] applied HUB-DT to a “3-valence” behavioral task [112], in which different “emotional contexts” were produced by trial blocks of distinct cue-outcome pairings. HUB-DT identified behaviors specific to appetitive and aversive emotional contexts. In a separate application, HUB-DT was also able to identify morphine-specific behaviors from video data alone, without experimenter input [110]. Our lab has used DBM to identify discrete behavioral “microstates” contained within greater behavioral sequences corresponding to experimental events in a mouse operant conditioning paradigm, such as lever-press, reward retrieval, and reward consumption [45]. Many of these microstates identified behaviors such as lever approach and orientation toward the food cup, as well as “non-task” behaviors related to rearing, locomotion, and grooming, that are typically not characterized in operant conditioning procedures.

The above approaches can be combined with in vivo calcium imaging or electrophysiological recording to provide a fine-grained correlation between behavior and brain activity in freely moving animals, with minimal disruption to an animal’s behavior (Figure 1). Typically, video recording and neural recording framerates are synchronized, and statistical analyses are performed to assess the degree to which behavioral output predicts single-cell or ensemble activity and vice-versa. For example, Zhang et al. [45] used DBM to identify behavior associated with miniscope calcium imaging data from PL in a mouse operant conditioning task, and determined that DBM-identified behavioral states significantly predicted single-neuron activity across trials. Lindsay et al. [111] applied a similar analysis to their combination HUB-DT/in vivo electrophysiology recording in ACC during their 3-valence conditioning task, and determined that ensemble-level activity could reliably predict both specific behaviors and emotional contexts.

The use of deep learning for behavioral/neural analyses provides some clear advantages over traditional approaches. Compared with manual behavior scoring, methods such as DBM, VAME, and HUB-DT are less labor-intensive and less error-prone, and provide greater temporal resolution for comparison with neural recording and imaging data. Furthermore, deep learning approaches can categorize spontaneous behavior that cannot be reliably timestamped by TTL signals (i.e., freezing, head-jerking, lever approach, grooming, rearing, and any number of other pre- or post-response behaviors occurring within or between conditioning trials). Deep learning approaches can also outperform human observers: VAME was capable of distinguishing between wildtype and APP/PS1 transgenic mice when expert human observers could not [50]. HUB-DT identified behaviors that were more context-specific than those identified via manual scoring and was able to disambiguate neural activity related to specific behaviors from activity more generally related to emotional context [111].

Deep learning algorithms may be particularly useful for parsing both behavioral and neural data in emotionally complex procedures such as punishment. Punishment learning involves conflict between appetitive and aversive conditioning manifested as approach and avoidance behavior, respectively [24,113]. In many punishment-based procedures, these emotional states—as well as the effects of various brain activity manipulations—have been inferred from changes in response rate [8–11,25,33]. Deep learning approaches have the potential to expand behavioral analyses beyond these metrics. As demonstrated by Lindsay et al. [111], deep learning can separate behavioral and neural data related to specific behaviors, emotional states, and contexts. Thus, deep learning algorithms may also be used to identify behavioral and neural signatures of operant punishment learning and distinguish them from signatures related to Pavlovian fear and/or pain processing. This is a crucial aspect of punishment procedures that has typically been inferred from null results in yoking or two-response procedures [9,10,25,39] (but see [114]). Deep learning approaches may also be useful for identifying and characterizing sex differences in punishment sensitivity [22,36–38] and in response patterns to aversive stimuli in general [115]. In summary, deep learning approaches stand to add substantial nuance to behavioral and neural analyses of punishment learning and could potentially identify novel signatures of motivational conflict.

5.1. Conclusions

Studying the mechanisms of context-induced reinstatement following punishment of an operant response will increase understanding of how punishment learning reduces performance of a response and how a change in context can cause responding to renew. Examining these basic operant mechanisms is a necessary step toward understanding pathologies such as SUD, which may be characterized by punishment insensitivity [18,114,116–118]. Here, we review a brain circuit involving IL and NAc shell that may be involved in punishment-induced behavioral suppression. Indeed, altered function in these regions (or in analogous structures) has been linked to addiction in humans [119–123].

The recent advancements in miniscope imaging and deep learning behavior analysis reviewed above allow for detailed and longitudinal characterization of punishment-related ensembles in brain regions such as IL and NAc shell. This technology is developing rapidly and is constantly being refined. Future directions include the use of dual-color imaging from both D1-expressing neurons [73] and glutamatergic projection terminals from IL in NAc shell. Such an experiment could image the activity of each cell type during punishment-induced response suppression in one context, and context-induced reinstatement in the other. deep learning algorithms such as DBM could parse punishment-related neural activity and behavioral microstates in each context. Collectively, miniscope imaging and deep learning approaches to behavior analysis stand to significantly increase our understanding of the biobehavioral mechanisms of punishment learning.