Avoidance Problems Reconsidered

Abstract

Active avoidance is the prototypical paradigm for studying aversively-motivated instrumental behavior. However, avoidance research stalled amid heated theoretical debates and the hypothesis that active avoidance is essentially Pavlovian flight. Here I reconsider key “avoidance problems” and review neurobehavioral data collected with modern tools. Although the picture remains incomplete, these studies strongly suggest that avoidance has an instrumental component and is mediated by brain circuits that resemble appetitive instrumental actions more than Pavlovian fear reactions. Rapid progress may be possible if investigators consider important factors like safety signals, response-competition, goal-directed vs. habitual control and threat imminence in avoidance study design. Since avoidance responses likely contribute to active coping, this research has important implications for understanding human resilience and disorders of control.

Introduction.

Active avoidance responses (ARs) are learned defensive behaviors that escape or prevent aversive stimuli. “Avoidance” has a negative connotation, but AR learning may be a powerful coping mechanism, giving subjects the flexibility to suppress innate defensive reactions like fleeing, fighting or freezing in favor of more adaptive actions. AR learning may underlie basic protective behaviors to distant or uncertain threats, like wearing a helmet to prevent head injury or paying taxes to prevent an audit. It may also help people deal more effectively with intense threats. For instance, first responders and soldiers undergo extensive training to suppress fear reactions and select actions that produce a better outcome in lifethreatening situations. However, dysfunctional ARs may also contribute to disorders of control. In OCD and addiction, compulsive actions that escape anxiety or withdrawal symptoms are repeated despite punishing consequences and even cognitive awareness that they are maladaptive [1–4].

Active avoidance was the dominant paradigm for studying aversive conditioning and defensive behavior for much of the last century. The field lost momentum in 1980s, however, when simpler Pavlovian paradigms rapidly advanced psychological and neurobiological explanations of defensive behavior. Avoidance has experienced something of a revival recently as model paradigm for studying aversively-motivated instrumental behavior [5] (but see Figure 1). However, these efforts have been dogged by unresolved theoretical debates and a general notion that avoidance brain research was unproductive in the past and likely will be again. Shifting focus to Pavlovian processes and the topography of defensive responding was the right move for the field at the time. Today an impressive body of literature details the brain systems, microcircuits, and physiological/molecular mechanisms of Pavlovian defensive responding. However, it is increasingly clear that Pavlovian processes alone cannot explain avoidance and many basic questions relevant to human psychopathology and resilience remain unanswered.

Figure 1. Publications containing Fear Conditioning vs. Active Avoidance.

(or related terms) in Title/Abstract (PubMed 6/25/2018). Note that avoidance research plateaued after Bolles’ SSDR Theory. Key behavioral and neurobiological tools that could resolve seemingly intractable “avoidance problems” are represented by numbers: (1) instrumental outcomedevaluation, (2) in vivo Fast-scan cyclic voltammetry measurement of dopamine, (3) instrumental contingency degradation, (4) optogenetic and (5) chemogenetic manipulation of neurons.

In this essay, I will briefly describe the avoidance paradigm and some key theoretical considerations. I will then focus on recent neurobiological data that may help reveal the “natural fracture lines” of avoidance behavior [6]. These studies suggest that rapid progress is possible if researchers apply modern neurobiological and behavioral tools to longstanding avoidance problems.

Active Avoidance Behavior.

In signaled active avoidance (SigAA), subjects receive warning signals (WSs; e.g. tone) that predict an aversive unconditioned stimulus (US; e.g. footshock; Figure 2). However, unlike Pavlovian conditioning, subjects can exert control over events in the SigAA paradigm: they can escape the WS and prevent the US by emitting a specific action determined by the experimenter (e.g. shuttle or barpress). WS-US pairings occur early in training, before the subject gains exposure to instrumental contingencies, and Pavlovian reactions like freezing dominate. As training progresses and subjects gain accidental exposure to instrumental contingencies, ARs emerge and Pavlovian reactions are drastically diminished [7–13].

Figure 2. A. Relationship between stimuli and target response for a typical avoidance protocol.

Failure to perform an AR during the WS leads to delivery of a painful US. Performing the AR leads to immediate WS termination, exposure to feedback (FB; external & internal stimuli not present before the response) and cancelation of the scheduled US. B. Hypothetical associations acquired during SigAA training. Failed trials transform the WS into a Pavlovian threat that predicts the US. Exposure to instrumental contingencies likely produces both hierarchical goal-directed associations, where the WS indicates that response-produced outcomes are possible, and habitual associations, where the WS directly triggers the AR because of negative and/or positive reinforcement. Note that FB stimuli are initially neutral and must develop into safety signals (via negative correlation with aversive WS and painful shock) before they can influence AR learning. Tricolored asterisk denotes outcome(s)/reinforcer(s) since it remains unclear how WS-termination, US-omission and Safety Signals relate to stimuli and ARs during goal-directed vs. habitual avoidance learning.

Theoretical Considerations.

From the beginning, avoidance posed a problem for theoretical accounts of learned behavior, mainly because learning occurred on trials where the US was omitted. It was difficult to understand how a nonevent could support learning, especially since US omission does not coincide in time with the AR (at the time, cognitive expectancies were not widely accepted). The most influential account of SigAA was two-factor fear theory [14,15]. According to fear theory, rats first acquired conditional fear through Pavlovian conditioning (Factor 1). Then, on later trials, an instrumental stimulus-response (WS-AR) association was reinforced by “fear reduction” associated with WS-termination (Factor 2). Conditional fear was an acquired drive and (negative) reinforcement occurred because of drive reduction. Fear theory also hypothesized a motivational role for the WS: once learning occurred, WS presentations triggered fear and motivated ARs that reduce fear. Importantly, fear theory solved the problem of explaining how a response could be reinforced by the nonoccurrence of an aversive US. As Seligman and Johnston [16] explain it: “rather than responding to ward off a shock in the future, an “avoider” is actually escaping from fear evoking stimuli in the present”, which puts the AR in contact with the reinforcing event (WS-termination).

Although enormously influential, fear theory eventually ran into significant difficulty on several fronts. First, several groups demonstrated avoidance learning with no explicit WS [17]. Other experiments found that both WS-termination and US-omission contribute to learning, and the relative contribution depended on the task [18]. It was also criticized for its reliance on WSinduced “fear”, mainly because researchers found little correspondence between the level of fear (i.e. defensive and autonomic reactions) and ARs [7]. Evidence for fear-reduction at WStermination was also lacking [19]. Lastly, avoidance is extremely resistant to extinction; when shock is turned off ARs may continue to be emitted for thousands of trials [20].

The most damaging blow to active avoidance research came with Robert Bolles’ interpretation of ARs through the lens of SSDR (Species-Specific Defense Reactions) theory. Bolles was bothered that the most important determinant of SigAA learning was the response choice [18]. Initially, he suggested that aversive stimuli restrict behavior to innate SSDRs, and thus only SSDRs or responses compatible with SSDRs can be reinforced. However, subsequent experiments found that the most prominent rodent SSDRs, freezing and flight, were insensitive to reinforcement [18, 21]. Compared to yoked controls, freezing and locomotion are not increased when they delay or prevent shock delivery. Similarly, freezing is not decreased by punishment. Bolles then reasoned: if aversive stimuli restrict behavior to SSDRs, and SSDRs are insensitive to reinforcement, then response learning is not possible in the avoidance situation. Instead, apparent ARs likely represent SSDRs that come under stimulus control through Pavlovian learning, where rats flee threats to the safest spot available and then freeze [22]. Note that safety stimuli may include internally perceived stimuli such as proprioceptive feedback – so animals may approach (or flee towards) feedback stimuli by emitting the response. Although this explanation cannot easily account for “avoidance” with ratio schedules or arbitrary responses like bar-pressing [23,24], Bolles and others argue that such ARs require extensive training and are not a major component of defense (since animals could not have evolved a defense mechanism that allows for many unsuccessful encounters with a predator) [22].

Reconsidering Major Avoidance Problems.

Since the decline of avoidance research, there has been remarkable progress in the understanding of Pavlovian conditioning and instrumental learning with appetitive outcomes. Newer behavioral and neurobiological tools are already resolving longstanding questions about avoidance. Here I will reconsider past reasons for abandoning the avoidance paradigm in light of new data and models of learning, behavior and brain function. Although many brain regions have been implicated in avoidance, I will focus on the amygdala, striatum and medial prefrontal cortex because of their known roles in fear conditioning, instrumental learning and response selection. Unless otherwise noted, studies refer to active avoidance in rodents.

Is the avoidance paradigm useful for studying defensive brain circuits?

Early research on the role of the amygdala in aversive conditioning illustrates why neuroscientists favored the Pavlovian paradigm over avoidance. Conflicting reports suggested that amygdala damage could impair, facilitate or have no effect on ARs [25,26]. This contrasts with initial studies of fear conditioning. Independent laboratories, using different conditioned stimuli (CSs) and measuring different responses, provided converging evidence that basolateral (BLA) and central (CeA) amygdala were essential for the learning, storage and expression of Pavlovian fear memories [27]. However, recent work with more precise manipulations suggests that clarity is possible if experimenters account for competing Pavlovian responses and the amount of training.

Specificity of brain manipulations.

Many early studies investigating avoidance brain circuits targeted the entire amygdala with large lesions that inconsistently damaged amygdala subregions. More recent work evaluating specific amygdala nuclei paints a clearer picture. For instance, muscimol inactivation of BLA blocks wheel-turning ARs in a rabbit discrimination task [28] and platform-mediated avoidance in rats [13]. Consistent with this, selective lesions of lateral amygdala (LA) or basal amygdala (BA) severely impair the learning and performance of shuttlebox ARs [8,9]. Measures of neural activity also support a role for BLA; plasticity of WSevoked responses occurs rapidly with SigAA training and correlates with ARs [29]. Interestingly, lesions of CeA produce a very different profile: they have no effect in good avoiders (rats who acquired the AR without difficulty) but facilitate ARs in poor avoiders (rats emitting few ARs even after significant training) [8,9]. Considered with results from Pavlovian studies, this suggests that threat memories stored in LA can be used by to generate Pavlovian reactions (via CeA), or instrumental actions (via BA; Figure 3). They also illustrate how different labs using different tasks and subjects can provide converging neurobehavioral evidence when more precise methods are used.

Figure 3. Neurobehavioral model of active avoidance.

(A) Pavlovian SSDRs like freezing dominate early in avoidance training and are controlled by LA and CeA. (B) With exposure to instrumental contingencies, IL is recruited to suppress competing CeA-dependent SSDRs and threat processing is diverted to BA and striatal regions important for the learning and performance of goal-directed ARs. At this stage, VTA responses in striatum also contribute to response-selection (SSDR vs. AR, via suppression or facilitation of WS-evoked dopamine) and encode safety (via facilitation of feedback-evoked dopamine). (C) With continued training, IL suppression of Pavlovian SSDRs is no longer necessary and PL becomes important for AR performance. This may reflect the transition of the WS from a CS that predicts harm to a DS that signals the opportunity to avoid harm (attain safety). (D) After overtraining, habit circuits gain control of ARs and performance no longer depends on amygdala, VTA or goal-directed corticostriatal circuits. Dopaminergic inputs from SNc support reinforcement of habitual associations. Note that LA, BA, IL, NAc-shell and dorsal striatum are also implicated in safety signal learning [50,54,78–81]. Both direct and indirect striatal output pathways contribute to AR performance [69,82]. Note also that latter stages of this model remain to be tested, especially the roles of DMS, DLS and IL. Abbreviations defined in text.

Competition between incompatible responses.

Although early researchers recognized that incompatible responses like freezing interfered with avoidance, they treated this mostly as a nuisance and engineered chambers and tasks to minimize freezing. Bolles considered the freezing hypothesis of poor avoidance something of a lazy explanation, remarking “…the real issue is whether the animal fails to [avoid] because it is freezing, or freezes because it has not learned to [avoid]”, clearly favoring the latter description [18]. However, the CeA lesion data discussed above suggest that behavior in the avoidance chamber should be viewed as a competition between incompatible Pavlovian reactions and instrumental actions. In support of this, Pavlovian reactions like freezing are 1) inversely correlated with ARs, 2) unusually strong in poor avoiders, and 3) abolished by CeA lesions that reverse poor avoidance [8,9,11–13,30]. Note that CeA lesions in poor avoiders produce near-asymptotic avoidance behavior in the first post-lesion session, indicating that these rats acquired instrumental associations but failed to express them [8,9]. Thus, at least in shuttlebox tasks, rats fail to avoid because they freeze.

More recent studies examining response competition in avoidance have focused on infralimbic (IL) prefrontal cortex and nucleus accumbens (NAc). c-Fos in IL [12] and NAc [31] correlates positively with ARs and negatively with freezing after moderate avoidance training. To test whether IL suppresses CeA-mediated reactions during SigAA, Moscarello and LeDoux [11] evaluated permanent and temporary lesions in a rat shuttlebox task. Disrupting IL early in training impaired SigAA and produced corresponding increases in freezing. An opposite pattern was observed with CeA inactivation. Subsequent work by Greg Quirk’s group evaluated IL in a novel platform-mediated avoidance task [13]. Here, rats lever-pressing for food must step onto a safe platform to avoid shocks signaled by a tone. ARs reach asymptote in session 2, and Pavlovian fear reactions gradually disappear by session 10. Using muscimol inactivation of IL during session 11, they also observed increased fear reacitons, but these weren’t so strong as to prevent ARs. Together, these data suggest that IL is most important early in avoidance training to suppress incompatible Pavlovian reactions like freezing.

Two recent SigAA studies examined ARs and Pavlovian reactions after NAc inactivation. The first, using the platform-mediated avoidance task, found that inactivation of the entire NAc impaired ARs after 10d of training [13]. The second, using a disconnection strategy and shuttlebox avoidance, found that BA and NAc-shell operate in series to motivate AR expression after three training sessions [32]. In both studies, disrupting ARs led to corresponding increases in Pavlovian freezing. Interestingly, NAc-shell inactivation had no effect on freezing when the opportunity to avoid was removed by blocking the shuttlebox door, consistent with the responsecompetition model and many earlier studies showing that response prevention leads to a rapid return of Pavlovian fear reactions [20].

Amount of training.

Some early researchers recognized that the amount of avoidance training could impact brain lesion effects, but this was not widespread. Recent studies show a dramatic shift in the brain circuits mediating avoidance across training. First, manipulations of BLA that impair avoidance early in training are ineffective after overtraining [9,28]. Second, ventral striatum, which receives direct projections from BLA, appears necessary for moderately trained, but not overtrained, ARs [13,32,33]. Finally, in dopamine-deficient mice with severe impairments in SigAA learning, replacing dopamine in amygdala and striatum is sufficient to enable learning [34]. However, overtrained mice require dopamine only in striatum to perform the AR. This dramatic shift from amygdala-dependent to amygdala-independent avoidance is reminiscent of the shift from goal-directed to habitual responding in appetitive conditioning [35]. In appetitive studies, instrumental behavior that is sensitive to outcome-devaluation requires dorsomedial striatum (DMS) but not dorsolateral striatum (DLS). However, with overtraining, behavior is DLS-dependent and insensitive to devaluation. It is unclear if the shift to amygdalaindependent avoidance behavior reflects a transition to S-R control with overtraining. Outcome manipulations after moderate training vs. overtraining have not been examined in avoidance and early attempts to address the role of dorsal striatum in habitual avoidance have been inconclusive [36–38]. But these studies demonstrate that the amount of training is a critical factor for neurobehavioral analyses of avoidance.

Are ARs merely Pavlovian flight reactions?

SSDR theory generated little new avoidance research as it argued that avoidance was a poor method for studying defense. Nevertheless, several findings cast doubt on the conditioned flight interpretation of ARs. First, Pavlovian flight is rarely observed in the laboratory, even when rats are familiar with escape routes [21]. This makes sense, as the CS functions to prepare the animal to react to the US, not substitute for it. One study attempted to show conditioned flight to a familiar enclosure during a shock-free test [39]. They observed CS-elicited movement to the enclosure, but these responses were so slow (~0.25cm/s) that they would not have avoided the shock had it been present. Perhaps the best demonstration of conditioned flight was recently published using a serial compound CS in mice (tone→noise→shock; [40]). Mice froze to the tone but exhibited rapid (~8cm/s) movement to the noise. However, even here conditioned flight was severely constrained; flight was only observed in the context where shock had previously occurred. Second, although some ARs resemble flight, the notion that rats flee to a safe spot to freeze is not supported by the data. Once rats reach a high level of avoidance responding, Pavlovian reactions including freezing approach zero [8,11–13]. Finally, if ARs are Pavlovian flight reactions one would expect the core underlying brain circuitry to match shared circuitry for other well-known fear reactions. But there are stark differences. CeA is required for fear conditioning in a variety of tasks but is unnecessary for avoidance [8,9,11,41]. The opposite is true for BA [8,9,42,43]. Most strikingly, the BLA-dependence of avoidance is transient. With overtraining, lesions of BLA have no effect on ARs [9,28] and WS-elicited neural activity declines [29]. This contrasts with fear conditioning, where LA is required for the lifetime of the memory [44] and remains essential even after overtraining [45–47]. And notably, when Pavlovian flight can be demonstrated it depends on threat processing in the LA→CeA pathway, unlike avoidance [40].

Fear and reinforcement problems.

Although not fatal to the paradigm, avoidance research was severely bogged down by theoretical debates about reinforcement and the mediating role of fear. Neurobiological studies are beginning to shed light on these important questions. For instance, outcome revaluation procedures that paired morphine injections with shock or safety signals suggest that US-omission is important for goal-directed ARs and safety signals are important for habitual ARs [48–50]. Both effects appear to depend on NAc-shell. In other “escape from fear” studies, aversive CS-termination supports new response learning in chambers where shock and explicit safety signals are never presented [51]. Amygdala lesions show that the underlying neurocircuitry mirrors AR learning; LA and BA are required, but CeA is not [52]. Appropriately-timed US-omission signals have not yet been identified in the brain, however, the lateral habenula, a region implicated in negative prediction errors in appetitive tasks inhibits AR learning [53]. Dopaminergic ventral tegmental area (VTA) projections to NAcshell may play a role, as these have been implicated in learning via omission of an expected aversive US [54]. These are important first steps towards understanding reinforcement in avoidance, but many questions remain. For instance, are WS-termination and US-omission involved in AR learning directly or indirectly via the generation of response-produced safety signals? And to what degree do these events behave as response-linked outcomes vs. S-R reinforcers?

One intriguing possibility that could explain the lack of correspondence between ARs and fear is that WSs come to behave as discriminative stimuli (DSs) or reflex triggers with extended training [55]. A DS would signal the opportunity to avoid harm (or obtain safety) and a reflex trigger would directly elicit the AR. Neither should elicit fear reactions. NAc and prelimbic (PL) prefrontal cortex, regions known to mediate goal-directed responding in appetitive tasks [35], may mediate DS-elicited ARs after moderate training [13,56,57]. VTA dopamine transients in NAc-core indicate that WSs predict harm early in training, predict safety later in training and ultimately behave as reflex triggers after overtraining [33,58]. WS-elicited dopamine responses are negative on early trials when freezing is high. However, on trials where the AR is emitted, positive dopamine responses are observed during the WS and especially during the safety interval. Further, avoidance can be bi-directionally modulated by manipulating these dopamine signals after moderate training, but not after overtraining. A very recent study observed inhibitory WS responses in rostral PL during avoidance conditioning, but not Pavlovian conditioning [59]. Although these didn’t correlate with ARs and were observed even on failed trials, optogenetically blocking PL inhibitory responses severely impaired AR expression. Thus, recruitment of PL appears to track the transition of the WS from a CS to a DS, as residual Pavlovian responses are decoupled from AR performance [31]. This NAc- and PL-dependent stage may bridge threat-motivated avoidance and habitual avoidance.

Together these findings indicate that fear as an intervening variable in avoidance may be an outdated, unnecessary and counterproductive concept [60–62]. Fear-mediation was introduced to explain S-R reinforcement via a nonevent (US-omission) without appealing to cognitive expectancies. However, behavioral and autonomic reactions thought to reflect a central fear state are anticorrelated with ARs and evidence for fear-reduction at WS-termination is lacking. Today it is also clear that animals form associations with outcome representations and neurally register prediction errors when expected outcomes are omitted. Safety conditioning, which appears important for avoidance, depends on such violated expectations [63]. A modern conception of AR learning based on appetitive conditioning has been more productive, where ARs are generated by Pavlovian and instrumental prediction errors and controlled by different brain circuits as subjects gain experience with associative contingencies.

Insensitivity of SSDRs to reinforcement.

Flight and freezing SSDRs are insensitive to reinforcement [21,64,65]. Arbitrary responses like bar-pressing can be exceedingly difficult to train with an avoidance protocol, often requiring procedural tricks like shaping, long ITI safety cues, rest days and extended periods of intermittent shock. Locomotor responses are acquired more readily. However, even in these tasks, ~25% of subjects never acquire the AR despite significant exposure to instrumental contingencies (poor avoiders). Phenomena like “warm-up”, where even good avoiders won’t avoid until receiving a shock, can also be a major nuisance [66]. If animals can learn about the consequences of their actions when threatened, why is this so difficult to reliably demonstrate in the laboratory?

One possibility is that avoidance researchers have been operating at the wrong end of the defensive behavior continuum (Figure 4). Avoidance is typically trained in small chambers with few behavioral options using high density shock protocols. This may maximally activate survival circuits, thoroughly suppress non-defensive behavior and restrict responding to a few SSDRs that are insensitive to reinforcement. These “post-encounter” and “circa-strike” behaviors likely evolved as rapid, inflexible reactions to predators that are detected and/or attacking [21].

Figure 4. Defensive behavior continuum.

(based on [21]). Red represents SSDRs and white represents intermixed non-defensive behaviors.

Although dominant when predatory imminence is high, these behaviors represent a small fraction of the defensive behavioral repertoire. Animals spend most of their time under lower threat levels engaging in anxiety-related “pre-encounter” behaviors, intermixed with nondefensive behaviors like foraging. Perhaps these evolved to be sensitive to reinforcement, since trial-and-error learning would be much less likely to result in death. Unfortunately, very little is known about pre-encounter defensive behavior, especially in the context of avoidance. However, there is renewed interest in more ecologically valid models that evaluate defensive behavior in conflict situations where threats are less certain, safety signals modulate responding and rewards are available [30,59,67–73]. It will be important to evaluate instrumental contingencies in these paradigms to determine if there is an inverse relationship between US imminence and the effectiveness of reinforcement for available ARs [74].

Conclusions.

Converging evidence from studies using modern neurobiological and behavioral tools strongly suggests that ARs are established via instrumental response learning. Conditioned threats and safety signals may be necessary for avoidance, however, generating ARs with purely Pavlovian contingencies is exceedingly difficult. Further, the neural circuits mediating ARs have more in common with appetitive instrumental behavior than Pavlovian SSDRs. Recent advances indicate that avoidance problems identified in the 1970s can be resolved using modern tools, especially if investigators consider important factors like safety signal reinforcement, response-competition, goal-directed vs. habitual control of ARs and threat imminence in the design of avoidance studies [61,63]. This latter factor represents the greatest hurdle for progress in understanding avoidance, and more work is needed to determine if preencounter defensive behaviors (or co-occurring non-defensive behaviors) are more sensitive to reinforcement than circa-strike and post-encounter responses like flight and freezing. Research in this vein may shed light on a neglected class of learned defensive behavior and advance our understanding of human disorders of control. Indeed, recent studies in humans have confirmed, to a first approximation, the neural circuits of avoidance identified in rodent studies [10,75,76]. They also demonstrate that adaptive ARs can be effective active coping strategies [10], whereas maladaptive or habitual ARs may contribute to disorders ranging from OCD [4,62] to addiction [77].

Highlights.

• Avoidance research stalled amid debates about fear, reinforcement and Pavlovian control

• Recent neurobehavioral data are inconsistent with a Pavlovian flight account of avoidance

• Avoidance brain circuits resemble those of appetitive instrumental actions more than Pavlovian SSDRs

• Feedback, response-competition, goal-directedness and threat-imminence are important factors in avoidance

• Warning signals may behave as conditioned stimuli, discriminative stimuli and habit triggers across training