Identifying serotonergic mechanisms underlying the corticolimbic response to threat in humans

Abstract

A corticolimbic circuit including the amygdala and medial prefrontal cortex (mPFC) plays an important role in regulating sensitivity to threat, which is heightened in mood and anxiety disorders. Serotonin is a potent neuromodulator of this circuit; however, specific serotonergic mechanisms mediating these effects are not fully understood. Recent studies have evaluated molecular mechanisms mediating the effects of serotonin signalling on corticolimbic circuit function using a multi-modal neuroimaging strategy incorporating positron emission tomography and blood oxygen level-dependent functional magnetic resonance imaging. This multi-modal neuroimaging strategy can be integrated with additional techniques including imaging genetics and pharmacological challenge paradigms to more clearly understand how serotonin signalling modulates neural pathways underlying sensitivity to threat. Integrating these methodological approaches offers novel opportunities to identify mechanisms through which serotonin signalling contributes to differences in brain function and behaviour, which in turn can illuminate factors that confer risk for illness and inform the development of more effective treatment strategies.

1. Introduction

Major depressive disorder (MDD) is an affective disorder characterized by depressed mood, increased feelings of sadness and diminished interest or pleasure in general activities [1]. Within a twelve-month period, approximately 6–7% of the population experiences a depressive episode [2]. Thus, MDD represents a prevalent disorder with substantial burdens on public health that contribute to emotional and financial pressures on affected individuals, their families and society as a whole [3]. A study evaluating treatment efficacy in a large population reported less than 50 per cent response rate and even lower rates of remission, indicating that treatment efficacy can be improved dramatically [4]. As such, a clearer understanding of factors that contribute to risk for and the pathophysiology of MDD is critical for (i) identifying at-risk populations, (ii) developing novel therapeutics targeting specific molecular mechanisms, and (iii) identifying biomarkers predictive of treatment response. Though the precise mechanisms that precipitate a depressive state are not fully understood, trait-like behaviours such as anxiety and neuroticism have been identified as risk factors for MDD and other affective disorders [5–8]. Thus, evaluating neurobiological mechanisms related to these aspects of personality may in turn be informative of individual differences in risk for clinical illness [9].

2. A threat-related corticolimbic circuit

The amygdala is a subcortical brain structure integral for identifying novel and biologically relevant stimuli within the environment. The amygdala exhibits a particular sensitivity for threat-related cues (e.g. facial expressions of fear and anger) and plays an important role in learning associations between stimuli and events that predict threat [10–13]. Numerous neuroimaging studies in humans, most commonly using blood oxygen level-dependent functional magnetic resonance imaging (BOLD fMRI), have identified a positive association between threat-related amygdala reactivity and trait anxiety or related constructs [14–17]. A distributed corticolimbic circuit including the amygdala and prefrontal cortical regions, namely medial prefrontal cortex (mPFC) and anterior cingulate cortex (Brodmann areas: 24/25/32), plays a key role in multiple facets of emotional behaviour; most notably in the generation, regulation and expression of behavioural and physiological arousal [18–24]. Effective communication within this corticolimbic circuit is thought to play a critical role in integrating salient information and generating adaptive responses to environmental challenges [20]. Neuroimaging studies have also identified an association between functional and structural indices of this corticolimbic circuit and personality measures associated with anxiety [25–30]. Similarly, neuroimaging studies in depressed patients have identified alterations in both threat-related amygdala reactivity and broader corticolimbic circuit function [18,31–34]. Linking discrete molecular mechanisms with individual differences in threat-related corticolimbic circuit function would allow for a more detailed understanding of how brain chemistry contributes to circuit function and disease liability.

3. Serotonin signalling and the corticolimbic response to threat

Serotonin (5-hydroxytryptamine, 5-HT) is a neuromodulator with significant effects on emotional behaviour, including anxiety and sensitivity to threat [35–37]. Serotonergic neurons, derived primarily from the dorsal and median raphe nuclei, innervate this corticolimbic circuit [38]. Direct modulation of this circuitry may underlie the effects of serotonin on emotional behaviour [39–41]. Consistent with its role in regulating mood, a convergence of evidence suggests that serotonin may play a role in the pathophysiology of depression. Human neuroimaging studies have provided novel insight into how serotonin signalling modulates underlying corticolimbic circuit function and individual variability in personality traits such as anxiety, which are related to risk for depression and other affective disorders. Most notably, imaging genetics has repeatedly identified links between threat-related corticolimbic circuit function and genetic variants, which putatively impact serotonin signalling [17,25,42–46]. Pharmacological challenge paradigms have identified an effect of selective serotonin reuptake inhibitors (SSRIs) on corticolimbic circuit function both in healthy controls and depressed patients, suggesting that antidepressant treatment response may in part depend on modulation of this corticolimbic circuit [47–52].

The serotonin system consists of multiple receptor classes (e.g. 5-HT₁, 5-HT₂) and subtypes within these classes (e.g. 5-HT_1A, 5-HT_1B, 5-HT_1D,). Thus, an important aspect in understanding how serotonin contributes to inter-individual variability in personality and related risk for MDD, and other affective disorders, is identifying molecular mechanisms (i.e. receptor pathways) mediating the effects of serotonin signalling on threat-related corticolimbic circuit function. Positron emission tomography (PET) can be used to quantify the availability of a particular molecular substrate, including receptors, in humans. Such neuroreceptor PET offers a unique opportunity to model capacity for receptor-related function, in vivo. Thus, a PET–fMRI multi-modal neuroimaging strategy evaluating the association between threat-related brain function, assessed using BOLD fMRI, and serotonin receptor binding, assessed using PET, can be used to evaluate the effects of serotonin receptor pathways on brain function, behaviour and psychopathology.

4. Multi-modal neuroimaging studies

(a). 5-HT_1A autoreceptor

The inhibitory 5-HT_1A receptor is expressed as both an autoreceptor and post-synaptic receptor [53,54]. Through negative feedback inhibition on serotonergic neurons, the 5-HT_1A somatodendritic autoreceptor plays a critical role in regulating 5-HT release at downstream targets [55]. Alterations in 5-HT_1A availability have been previously associated with depression, and therapeutic efficacy of many antidepressants may depend on modulation of 5-HT_1A autoreceptor signalling [56–58]. Using a PET–fMRI multi-modal neuroimaging approach, we examined the association between individual variability in 5-HT_1A autoreceptor binding, assessed with [¹¹C]WAY100635 PET, and threat-related amygdala reactivity, assessed with BOLD fMRI. By evaluating the association between 5-HT_1A autoreceptor binding and threat-related amygdala reactivity within a single cohort, this novel multi-modal neuroimaging strategy offered the opportunity to identify specific molecular mechanisms through which serotonin signalling may contribute to inter-individual variability in threat-related amygdala reactivity.

Within a cohort of 20 individuals we found that 5-HT_1A autoreceptor binding was significantly inversely correlated with threat-related amygdala reactivity [59]. Remarkably, 30–44% of the variability in threat-related amygdala reactivity was predicted by variability in 5-HT_1A binding within the dorsal raphe, suggesting that a greater capacity to regulate serotonin release (i.e. greater 5-HT_1A autoreceptor binding) was associated with reduced amygdala response to threat-related stimuli. These findings provide evidence for a molecular mechanism through which serotonin signalling modulates the brain's response to emotionally salient, threat-related stimuli. Intriguingly, our findings link a molecular mechanism (i.e. 5-HT_1A autoreceptors) and an aspect of brain function (i.e. amygdala sensitivity to threat), which independently has been identified in previous studies as altered in depressed cohorts [50,58]. Considering studies in animal models indicating that the 5-HT_1A autoreceptor may be a critical mechanism through which SSRIs exert their antidepressant effect, our findings suggest amygdala sensitivity to threat may reflect a neural pathway contributing to antidepressant treatment response.

(b). Serotonin transporter

Reuptake of serotonin via the serotonin transporter represents the primary mechanism for active clearance of extracellular serotonin following release [60]. In a multi-modal neuroimaging study using the same threat-related faces matching BOLD fMRI paradigm as was used in our 5-HT_1A autoreceptor study, Rhodes et al. [61] evaluated the association between 5-HTT binding in the amygdala, assessed with [¹¹C]DASB PET, and threat-related amygdala reactivity. The authors found that amygdala 5-HTT binding was significantly inversely associated with threat-related amygdala reactivity, with up to 40 per cent of the variability in threat-related amygdala reactivity predicted by 5-HTT binding levels. These findings suggest that inter-individual variability in the capacity to regulate serotonin signalling locally within the amygdala via serotonin reuptake is related to the response of the amygdala to threat. A recent study reported an inverse correlation between midbrain (i.e. raphe) 5-HTT binding and threat-related amygdala reactivity [62]. Although no association was then observed between amygdala 5-HTT binding and amygdala reactivity, this finding provides additional evidence for a link between the capacity to regulate 5-HT signalling and threat-related amygdala reactivity.

Taken together with our findings, these studies provide strong support for serotonin as a key modulator of threat-related amygdala reactivity. Perhaps more interestingly, these findings indicate that regulation via autoreceptor feedback and reuptake represents a critical molecular mechanism through which serotonin signalling modulates neural sensitivity to threat. Thus, a compromised capacity to regulate serotonin signalling, resulting in a diminished capacity to regulate amygdala reactivity to threat, may in turn contribute to heightened risk for affective disorders such as depression. These findings support the capacity for this multi-modal neuroimaging framework to link serotonin signalling mechanisms with brain function, and more effectively model how molecular mechanisms may contribute to variability in behaviour and psychopathology.

(c). Prefrontal 5-HT_2A receptor

Within the prefrontal cortex, the excitatory 5-HT_2A receptor is predominantly localized on glutamatergic neurons [63–66]. More specifically, the 5-HT_2A receptor is localized to proximal portions of the apical dendrite, representing a ‘hot spot’ of 5-HT_2A receptor localization coincident with relatively dense 5-HT innervation [63,67]. Previous studies in animal models and humans implicate serotonin signalling and the 5-HT_2A receptor in modulating prefrontal function in the context of fear- or anxiety-related behaviours and depression [39,68–71]. Using this multi-modal neuroimaging strategy in a cohort of 35 healthy individuals, we evaluated the association between mPFC 5-HT_2A binding, assessed with [¹⁸F]altanserin PET, and threat-related corticolimbic circuit function [72]. Based on its localization, we hypothesized that 5-HT_2A binding within mPFC would facilitate this prefrontal regulatory circuitry and be negatively correlated with threat-related amygdala reactivity. Consistent with this model, we found that 5-HT_2A binding was significantly inversely correlated with threat-related amygdala reactivity, such that 25–37% of the variability in amygdala reactivity was explained by mPFC 5-HT_2A binding. Additionally, we observed that mPFC 5-HT_2A binding was positively correlated with the magnitude of amygdala habituation over time. Finally, we observed that 5-HT_2A binding was positively correlated with functional connectivity between the amygdala and mPFC. Studies in both animal models and humans suggest that habituation of the amygdala response to threat is likely dependent upon prefrontal regulation [28,30,73]. Thus, our findings that amygdala habituation and mPFC–amygdala functional connectivity were correlated with mPFC 5-HT_2A binding suggests that these receptors are an important molecular mechanism mediating the effects of serotonin signalling on threat-related corticolimbic circuit function.

(d). Interaction between prefrontal 5-HT_1A and 5-HT_2A receptors

In addition to the 5-HT_2A receptor, the post-synaptic 5-HT_1A receptor is also localized to glutamatergic neurons within prefrontal cortex [74–76]. Intriguingly, the 5-HT_1A and 5-HT_2A receptors appear to be highly co-localized, with both receptors situated proximal to the cell body and thus potential mediators of serotonin signalling on glutamatergic neuronal excitability [75]. Based on this co-localization, the inhibitory 5-HT_1A receptor appears to be localized to moderate or ‘gate’ the capacity of the excitatory 5-HT_2A receptor to facilitate regulation of threat-related amygdala reactivity, as was observed in the previously described study. More specifically, this co-localization suggests that the inverse correlation between 5-HT_2A binding and threat-related amygdala reactivity should be most pronounced in the context of low 5-HT_1A binding, reflecting a reduced capacity for 5-HT_1A receptors to gate the negative effect of mPFC 5-HT_2A binding on threat-related amygdala reactivity.

Within a cohort of 39 healthy volunteers, we determined the association between threat-related amygdala reactivity and the interaction between 5-HT_1A and 5-HT_2A binding, assessed with [¹¹C]WAY100635 and [¹⁸F]altanserin PET, respectively [77]. Consistent with the co-localization of these receptors, we found that 5-HT_1A binding significantly moderated the negative association between 5-HT_2A binding and threat-related amygdala reactivity, such that mPFC 5-HT_2A binding was significantly inversely correlated with amygdala reactivity, but only when mPFC 5-HT_1A binding was relatively low. These findings indicate that molecular interactions between mPFC 5-HT_1A and 5-HT_2A receptors may play an important role in mediating the effects of serotonin signalling on threat-related corticolimbic circuit function. Interestingly, they suggest that acquiring multiple neuroreceptor PET scans within a PET–fMRI framework can be used to evaluate the impact of interacting receptor mechanisms on brain function.

5. Future directions

The studies presented here highlight how a multi-modal neuroimaging strategy using BOLD fMRI and PET can inform our understanding of serotonergic mechanisms that contribute to individual variability in threat-related corticolimbic circuit function. Together with studies implicating an association between corticolimbic circuit function, anxious traits and psychopathology, findings from these multi-modal neuroimaging studies implicate specific molecular mechanisms in mediating the effects of serotonin signalling on brain function, personality and risk for depression. As mentioned previously, however, serotonin signalling is mediated through a complex system with multiple receptor subtypes [78]. Additional serotonin receptors including 5-HT_1B, 5-HT_2C, 5-HT₄ and 5-HT₇ have been implicated in the function of this corticolimbic circuit as well as anxiety- and depression-related behaviours [40,41,78,79]. Future studies evaluating the impact of these additional receptors on brain function will provide further opportunities to effectively model the impact of serotonin signalling on sensitivity to threat through specific molecular pathways.

6. Integration with other neuroimaging approaches

Advancing our understanding of how genetic and molecular mechanisms shape underlying neural circuits that give rise to complex behaviours and confer risk for illness is critically dependent upon effectively integrating complimentary methodological approaches.

(a). Limitations

Even though this multi-modal neuroimaging strategy represents a powerful approach for evaluating how molecular mechanisms modulate underlying neural circuitry and related behaviours, it has its limitations. Neuroreceptor PET is not a direct measure of receptor function, but rather a measure related to quantity of receptors available for binding to the radioligand. The impact of inactive or internalized receptors on radioligand binding is difficult to quantify and likely varies between receptor systems and radioligands. Exposure to radioactivity and the invasiveness of intravenous or arterial sampling creates additional limitations. Currently, there is no radioligand that has been fully validated in humans to measure endogenous serotonin release. This limits the features of the serotonin system that can be measured in the context of PET. fMRI is an indirect measure of brain function that is based on signal relative to a baseline or control task. Changes in fMRI signal do not directly reflect neural activity and may more closely correspond to changes in local field potential, a signal which is thought to reflect incoming neural signalling [80]. Age-related changes have been reported for many PET radioligands and fMRI paradigms. Collecting the imaging measures within close temporal proximity to one another is important for avoiding potential age-related confounds. Correlations between neuroreceptor PET binding and fMRI brain function must be interpreted as such. Associations between these measures should be evaluated cautiously and paired with strong evidence supporting circuit dynamics. Despite these shortcomings, these two methodologies represent the most effective methods currently available for assaying brain chemistry and brain function. The use of well-documented fMRI paradigms that have been applied across multiple cohorts and repeatedly linked to relevant aspects of personality and behaviour, and well-validated PET radioligands benefit the application of this technique because identified associations can be considered in the context of a broader literature.

(b). Imaging genetics

Over the past decade, imaging genetics has become a commonly used approach for evaluating the impact of common genetic variants on underlying brain function, personality and risk for illness [9,81]. As genes play a fundamental role in our biology, genetic variation plays a critical role in biological sources of individual variability. Developing our understanding of how genetic polymorphisms map onto neurobiological mechanisms benefits our capacity to leverage genetic information to model aspects of underlying brain chemistry and brain function. Imaging genetics with BOLD fMRI has provided substantial insight into how polymorphisms within serotonin-related genes (e.g. 5-HTTLPR) predict inter-individual variability in threat-related corticolimbic circuit function and other neural pathways related to risk for illness [36]. Molecular mechanisms mediating these associations, however, are often based on putative effects described using in vitro models, which are susceptible to being too narrowly focused on specific molecular processes. Imaging genetics with neuroreceptor PET, however, offers a possible compliment through the capacity to link common genetic polymorphisms with variation in serotonin receptor binding in vivo [82]. For example, although the 5-HTTLPR putatively affects the expression of 5-HTT, it has been associated with alterations in 5-HT_1A binding in vivo, suggesting its effects may extend to additional serotonin signalling mechanisms [83].

Future studies integrating PET–fMRI multi-modal neuroimaging and imaging genetics through sophisticated statistical modelling techniques, such as structural equation modelling or mediation analysis, may provide novel insight into serotonergic mechanisms mediating the effects of genetic variation on threat-related corticolimbic circuit function. For example, 5-HTTLPR short allele carriers show heightened threat-related amygdala reactivity and decreased 5-HT_1A receptor binding relative to LL individuals [46,83]. Taken together with our observation that 5-HT_1A autoreceptor binding is inversely correlated with threat-related amygdala reactivity, differences in 5-HT_1A autoreceptor levels may be an important molecular mechanism mediating 5-HTTLPR effects on threat-related amygdala reactivity. Alternatively, common polymorphisms (e.g. 5-HTTLPR) can be used to model differences in serotonin signalling and associations between specific serotonin receptor binding and brain function can be evaluated against this genetic background. For example, a bias towards greater prefrontal drive and reduced amygdala reactivity via mPFC 5-HT_2A receptors would be predicted in individuals possessing genetic variants associated with increased 5-HT neurotransmission (e.g. 5-HTTLPR short allele carriers). Leveraging genetic information and imaging approaches through integrated multi-modal neuroimaging strategies such as PET–fMRI are crucial for further developing models of how serotonergic mechanisms may confer risk for depression through effects on underlying neural circuitry. In the case of treatment, such information can be used to apply models of underlying brain chemistry and brain function based on genetic variants, which may benefit the stratification of clinical subgroups according to how likely they are to benefit from particular treatments. Identifying specific receptor mechanisms that modulate relevant brain function would also benefit the development of novel therapeutic targets.

(c). Pharmacological challenge paradigms

Pharmacological challenge paradigms in the context of functional neuroimaging (i.e. pharmaco-fMRI) can be an effective methodological approach for evaluating biological mechanisms and neural circuits underlying behaviour, psychopathology and treatment response. Pharmaco-fMRI paradigms can be used to determine the impact of treatment strategies (e.g. antidepressant treatment) on specific neural circuits, which may mediate antidepressant treatment response. Recent studies have evaluated the impact of SSRI exposure on threat-related amygdala reactivity in healthy cohorts. Interestingly, Bigos et al. [48] found that threat-related amygdala reactivity increased following acute intravenous citalopram administration, whereas Harmer et al. [49] found that threat-related amygdala reactivity decreased following a 7-day oral administration protocol in healthy adults. These seemingly opposing findings may in fact depend on time-dependent effects of SSRI exposure on brain function, reflecting the temporal dynamics of SSRI treatment on the serotonin system observed in animal models and thought to underlie the behavioural response to treatment [35,84].

Future studies incorporating pharmacological challenge of specific serotonin receptors or reuptake blockade within a multi-modal neuroimaging framework would provide more direct evidence implicating specific receptor mechanisms in mediating the effects of serotonin signalling on corticolimbic circuit function. For example, experimentally increasing 5-HT neurotransmission (via pharmacologic challenge with a selective serotonin reuptake inhibitor) may lead to an increase in volume transmission and bias serotonin signalling towards 5-HT_2A receptors, resulting in greater prefrontal drive and subsequently diminished threat-related amygdala reactivity. Additionally, threat-related amygdala reactivity has been previously associated with 5-HTT binding, the primary target of SSRIs [61]. Individuals with higher 5-HTT binding may be more sensitive to disruption of 5-HTT function via SSRI exposure, and thus likely to exhibit a more pronounced sensitivity to SSRIs in the form of greater change in threat-related amygdala reactivity.

(d). Additional neuroimaging methodologies

The current review has focused on the benefits of PET–fMRI multi-modal neuroimaging, however, the general point holds that complementary neuroimaging modalities collected within a single cohort offer a unique opportunity to evaluate how specific molecular mechanisms affect underlying neural circuitry, which cannot be determined through the use of a single neuroimaging technique. A wealth of neuroimaging studies in humans, primarily via imaging genetics and pharmacological challenge paradigms, have implicated serotonin signalling in modulating the neural pathways underlying threat-related behaviour, which is associated with risk for depression. However, as was mentioned previously, the serotonin receptor family and the signalling mechanisms it affects is large and complex. Thus, a nuanced understanding of how these receptors interact and mediate specific aspects of serotonin signalling is critical for more completely understanding the role of serotonin in the pathophysiology of affective disorders including depression. For example, serotonin is known to be a neutrophic factor. Beyond linking molecular mechanisms with functional aspects of neural circuits, study designs evaluating the association between neuroreceptor PET, or perhaps single-photon emission tomography (SPECT), and structural measures (e.g. voxel-based morphometry or diffusion tensor imaging) would provide additional insight into how serotonergic mechanisms might be related to structural characteristics of this corticolimbic circuit [29,62].

(e). Translational perspective

The foundation for interpreting findings from a PET–fMRI multi-modal neuroimaging strategy is an understanding of how serotonergic mechanisms affect similar neural circuits within animal models. Animal models and related studies are critical for the ability to place in context findings from related neuroimaging studies. To gain a more complete understanding of how serotonergic mechanisms modulate underlying neural circuits, translational animal models are ideally situated to evaluate the effect of individual molecular signalling pathways on neural activity and behaviour more directly than neuroimaging paradigms. For example, one of the multi-modal neuroimaging studies described here identified an inverse association between prefrontal 5-HT_2A binding and threat-related amygdala reactivity that was moderated by prefrontal 5-HT_1A binding. Future studies in animal models evaluating the effects of prefrontal 5-HT_2A signalling (via local infusion of a 5-HT_2A agonist) on excitability of amygdala neurons and whether prefrontal 5-HT_1A signalling modulates this effect would provide additional support for such a model. This type of model could be further extended to determine whether such signalling mechanisms affect anxiety-related behavioural phenotypes.

(f). Measuring endogenous serotonin release

Identifying PET radiotracers that can effectively model serotonin release in vivo would bolster the usefulness of PET–fMRI multi-modal neuroimaging as a tool for evaluating how serotonin signalling plays a critical role in biasing threat-related corticolimbic circuit function. A PET radiotracer sensitive to in vivo serotonin levels, analogous to the usefulness of [¹¹C]raclopride for measuring endogenous dopamine release, is not currently available [85]. However, candidates for measuring endogenous serotonin release with promising results in both animals and humans are currently being evaluated [86–88]. Looking forward, the application of a PET radiotracer for modelling endogenous serotonin release in the context of a dual PET–MRI scanner offers the very exciting opportunity to evaluate the effects of threat-related corticolimbic brain function on serotonin release in real-time, offering still more effective ways of understanding how serotonin signalling modulates underlying neurobiological pathways.

7. Summary

A wealth of evidence implicates serotonin signalling in modulating emotional behaviour through its effects on threat-related corticolimbic circuit function and other neural pathways. The effects of serotonin on these neural pathways potentially underlie its role in the pathophysiology of mood and anxiety disorders such as depression. Neuroimaging in humans represents a valuable tool for evaluating biological sources of inter-individual variability in brain function, behaviour and psychopathology. An emerging multi-modal neuroimaging approach using PET/fMRI offers a unique opportunity to identify molecular mechanisms (e.g. receptor pathways) that mediate the effects of serotonin signalling on underlying neural circuitry. Future studies aimed at integrating this multi-modal neuroimaging strategy with genetic information, pharmacological challenge paradigms and additional imaging modalities are critical for building on our current understanding of how serotonin modulates neurobiological mechanisms that contribute to the emergence of individual differences in complex behavioural traits and related risk for psychopathology. These insights may in turn inform the development of novel therapeutics aimed at specific molecular mechanisms with improved treatment outcomes.