Serotonin and beyond: therapeutics for major depression

Abstract

The serotonin (5-HT, 5-hydroxytryptamine) system has been implicated in the pathogenesis of major depressive disorder (MDD). The case for its contribution to the therapeutic efficacy of a wide variety of antidepressant treatments is, however, much stronger. All antidepressant strategies have been shown to enhance 5-HT transmission in the brain of laboratory animals. Catecholamines, norepinephrine (NE) and dopamine (DA) can also play a pivotal role in the mechanism of action of certain antidepressant strategies. The enhancement of 5-HT transmission by selective serotonin reuptake inhibitors, which leads to a dampening of the activity of NE and DA neurons, may account in part for the low remission rate achieved with these medications and/or the residuals symptoms after remission is achieved. The functional connectivity between the 5-HT, NE and DA systems can be used to understand the mechanism of action of a wide variety of augmentation strategies in treatment-resistant MDD. Proof-of-concept studies have shown that antidepressant medications with complementary mechanisms of action on monoaminergic systems can double the remission rate achieved in a trial of standard duration. Novel approaches are also being used to treat MDD, which also appear to involve the monoaminergic system(s) to a varying extent.

1. Introduction

Major depressive disorder (MDD) is the most predominant illness among mental, neurological and substance-use disorders [1]. According to the World Health Organization [2], MDD is currently the leading cause of disability globally in middle- to high-income countries, measured in disability-adjusted life years. In addition, when MDD is present with other chronic medical disorders, such as asthma, epilepsy and diabetes, it worsens the morbidity of such chronic medical conditions [3]. Furthermore, mortality rates are increased several fold if MDD is present following a myocardial infarct [4]. These facts therefore make depression research a priority in the medical field.

There are numerous causal hypotheses for MDD, and they are not necessarily exclusive. The serotonin (5-HT, 5-hydroxytryptamine) hypothesis for depression has been intensely investigated since the 1967 seminal paper of Alec Coppen, laying down the foundation for such a hypothesis [5]. This exposé encompassed both aetiological and therapeutic considerations. These two types of issues need not, however, be linked. One possibility is that there may not be any anomaly in the 5-HT system in MDD, while a therapeutic approach may still work by enhancing 5-HT transmission above normal. An example for this therapeutic principle is the use of diuretics acting on kidneys to help treat chronic heart failure. Another possibility is that a deficient 5-HT system may indeed contribute to the manifestation of MDD, and an enhancement of 5-HT transmission by antidepressant strategies may restore euthymia. An example for this therapeutic principle is the use of dopamine (DA) precursors or agonists in the presence of DA neuronal cell loss in Parkinson's disease. The latter hypothesis for a causal role of 5-HT in MDD has often been dismissed over the years, because of a lack of consistency of observations on a decrease of 5-HT concentrations in post-mortem brain samples, and the fact that acute tryptophan depletion does not produce depression in healthy controls, tryptophan being the essential amino acid precursor of 5-HT [6]. Nevertheless, in unaffected individuals with a family history of depression, mild dysphoric symptoms can be triggered by this acute 5-HT-lowering challenge [7]. One can thus wonder whether a prolonged 5-HT lowering might not lead to MDD.

Although there is no single anomaly in the 5-HT system that is common to the majority of patients with MDD, one has to take into account that numerous neuronal factors control the synaptic concentration of 5-HT and its postsynaptic responses. These include tryptophan availability, genetic variants of tryptophan hydroxylase 2, cell body 5-HT_1A receptors, terminal 5-HT_1B autoreceptors, 5-HT transporters (5-HTT), different levels of monoamine oxidase (MAO) that inactivates 5-HT, and the polymorphism of a variety of postsynaptic 5-HT receptors (figures 1 and 2). If a single anomaly is present, it may only confer a slightly higher risk for developing MDD. For instance, the risk for MDD is directly proportional to the number of short alleles for the 5-HTT, conferring a lower 5-HTT efficiency, and the number of major life stressors [10]. Although this claim has been challenged [11], the analysis of all 54 studies, instead of subsamples, confirmed this important link [12]. It is possible that examining several of the abovementioned factors controlling 5-HT transmission would strengthen the direct link between a deficiency of the 5-HT system and MDD.

Figure 1.

Electrophysiological paradigm used to study in vivo the 5-HT system in the brain of laboratory animals. The firing activity of 5-HT neurons is recorded from the dorsal raphe nucleus either with single glass electrodes or microiontophoretic pipettes to test the sensitivity of 5-HT_1A autoreceptors (yellow rectangles) with 5-HT or selective agonists. Serotonin axons are electrically stimulated in the ventromedial tegmentum where 5-HT fibres originating from both the dorsal and median raphe nuclei course. The responsiveness of postsynaptic 5-HT receptors (red and orange rectangles), as well as the effectiveness of the stimulations, can be assessed from recording neurons in the pyramidal layers of the hippocampus. The responsiveness of terminal 5-HT_1B autoreceptors (blue rectangles) can be evaluated by varying the frequency of the stimulations [8]. The tonic activation of the postsynaptic 5-HT_1A receptor following various antidepressant treatments can be evaluated in unstimulated conditions by injecting the selective 5-HT_1A receptor WAY100635 and observing the increased firing rate of pyramidal neurons, which will be proportional to the degree of enhancement of 5-HT transmission [9].

Figure 2.

Functional interactions between the 5-HT, NE and DA systems and their postsynaptic targets. The circles crossed by an arrow represent reuptake transporters. The small circles with+and – signs represent the excitatory and inhibitory effects, respectively, of the receptors on the firing rate of the neurons. Note the presence of α₂-adrenoceptors on 5-HT terminals. (Online version in colour.)

2. Commonality of antidepressant strategies on the 5-HT system

Extensive in vivo electrophysiological studies of various antidepressant strategies carried out in the rat brain have revealed a striking commonality of action on the 5-HT system (table 1). The approach taken has basically been the one mentioned above: rather than looking at a single parameter controlling 5-HT transmission possibly altered by several types of treatments, a variety of neuronal elements have been examined. Most importantly, overall synaptic transmission has been assessed to determine whether the net effects of such alteration(s) led to increased transmission (figure 1). Initially, it was observed that long-term administration of tricyclic antidepressants (TCAs) with various action(s) on 5-HT and norepinephrine (NE) reuptake sensitized postsynaptic 5-HT receptor responsiveness in forebrain structures [13–16]. Such a possible unifying theory gained ground when it was observed that repeated, but not a single electroconvulsive shock (ECS) produced the same effect in the hippocampus [17]. These two distinct treatments were subsequently shown to enhance net 5-HT transmission by stimulating the 5-HT pathway at physiological firing frequencies for 5-HT neurons and increasing the response on postsynaptic neurons in the hippocampus [8,18]. In 1983, it was first reported that the selective serotonin reuptake inhibitor (SSRI) zimelidine initially decreased the firing rate of 5-HT neurons with repeated injections, but that the discharge frequency returned to normal after 14 daily injections due to the desensitization of the cell body 5-HT autoreceptor. The stimulation of the 5-HT pathway led to a greater effect in the hippocampus after a two-week zimelidine regimen [19]. It was also concluded that this enhancement was not due to mere reuptake inhibition, because acute injection of the SSRI citalopram did not produce this effect, but that the terminal 5-HT_1B autoreceptor controlling 5-HT release was desensitized like its cell body counterpart that controls firing activity [20]. Identical results were obtained with the SSRIs fluoxetine, paroxetine and fluvoxamine [8,21,22]. SSRIs therefore seemed to act by enhancing the function of 5-HT neurons (after they regain their normal firing rate), while leaving intact the sensitivity of postsynaptic neurons in the hippocampus, unlike TCAs and ECS. The mechanism of action of MAO inhibitors on the 5-HT system is similar, in some aspects, to that of SSRIs. Initially, they produce a decreased firing of 5-HT neurons, followed by a recovery after a three-week regimen [23]. In contrast, MAO inhibitors do not desensitize terminal 5-HT autoreceptors, but they still lead to enhanced 5-HT release [9]. By the mid-1980s, there was direct evidence that long-term, but not short-term, administration of TCAs, ECS, SSRIs and MAO inhibitors enhanced 5-HT transmission in the rat brain.

Table 1.

Summary of the effects of various antidepressant strategies on the 5-HT system as assessed in the brain of laboratory animals. SSRI, selective serotonin reuptake inhibitor; SNRI, serotonin and norepinephrine reuptake inhibitor; MAOI, monoamine oxidase inhibitor; ECS, electroconvulsive shocks. Arrows pointing down represent decreased responsiveness, arrows pointing upward represent an increase in function, open circles represent no change, and n.d. not determined. The net effect on transmission was assessed either by stimulating the 5-HT pathway or by determining the degree of activation of postsynaptic 5-HT_1A receptors using the 5-HT_1A receptor antagonist WAY100635. All regimens were administered for periods of 14–21 days.

	cell body 5-HT1A autoreceptor responsiveness	terminal 5-HT1B autoreceptor responsiveness	terminal α2-adrenoceptor responsiveness on 5-HT terminals	postsynaptic 5-HT1A receptor responsiveness	net effect on 5-HT transmission
SSRI	↓	↓	○	○	↑
SNRI	↓	○	↓	○	↑
MAOI	↓	○	↓	○ or ↓	↑
5-HT1A agonists	↓	○	n.d.	○	↑
tricyclics	○	○	↓	↑	↑
ECS	○	○	○	↑	↑
mirtazapinea	↓	○	↓	○	↑
bupropiona	↓	○	↓	○	↑
vagus nerve st.a	○	n.d.	n.d.	○	↑
agomelatinea	○	n.d.	n.d.	○	↑

^aThese treatments increase the firing of 5-HT neurons above their normal baseline levels.

Over the years, further electrical and pharmacological treatments with antidepressant properties have been studied and all shown to enhance overall 5-HT transmission in the rat hippocampus (table 1). These include the 5-HT_1A receptor agonist gepirone (for which there are three positive placebo-controlled studies in MDD), the α₂-adrenoceptor antagonist mirtazapine, the catecholamine releaser bupropion, the melatonin receptor agonist/5-HT_2C antagonist agomelatine and vagus nerve stimulation [9,24–26]. The DA D3/D2 receptor agonist pramipexole, which has been shown to be an antidepressant in patients with Parkinson's disease and in unaffected individuals with MDD, leads to delayed enhancement of the firing of 5-HT and a desensitization of the 5-HT_1A autoreceptor and a net increase in 5-HT transmission [27,28]. The atypical antipsychotic quetiapine, which is an antidepressant in monotherapy at subtherapeutic doses for schizophrenia, increases net 5-HT and NE transmission in the rat hippocampus [29]. It is important to mention that the nomenclature/denomination of certain psychotropic medications usually reflects their index indications (i.e. anticonvulsants and antipsychotics) rather than their distinct properties or mechanisms of action, contributing to further confusion and stigmatization.

3. The 5-HT system and the antidepressant response: important but not exclusive

All antidepressant strategies thus far tested enhance 5-HT transmission at least in normal rats (table 1). Nevertheless, it is obvious that other neuronal systems can also play a role in the antidepressant response. For instance, TCAs also sensitize postsynaptic α₁- and/or α₂-adrenoceptors in regions such as the thalamus, the amygdala and the facial motor nucleus, and most TCAs block NE reuptake [14–17,30]. The clinical parallel to these basic observations is that catecholamine depletion, but not 5-HT depletion, worsens depressive symptoms in patients who had responded to an NE reuptake inhibitor [31]. Similarly, tryptophan depletion does not cause a re-emergence of depressive symptoms in patients who responded to ECS [32]. In contrast, patients who responded to mirtazapine were observed to have a recurrence of their depressive symptoms with both tryptophan depletion and catecholamine depletion [33].

4. Enhancement of 5-HT and the antidepressant response: sometimes a limiting factor

Long-term administration of SSRIs enhances 5-HT transmission in the locus coeruleus [34]. As 5-HT exerts an inhibitory action on NE neurons through enhanced γ-aminobutyric acid (GABA) release, as a result of increased activation of excitatory 5-HT_2A receptors on GABA neurons, the firing activity of NE neurons is dampened by sustained regimens of the SSRIs such as citalopram, escitalopram, paroxetine and even a low dose of venlafaxine that inhibits 5-HT but not NE reuptake [34–37]. It was subsequently shown that the same regimen of citalopram that attenuates NE firing also decreased basal and evoked extracellular levels of NE in the amygdala [38]. It is thus conceivable that the lack of therapeutic benefits of SSRIs in some depressed patients may result from an attenuation of NE transmission in the presence of enhanced 5-HT levels (figure 3).

Figure 3.

Summary of two double-blind controlled studies of six weeks duration comparing the effects of single drugs versus their combination from treatment initiation. The SSRIs used were paroxetine 20–30 mg d⁻¹ and fluoxetine 20 mg d⁻¹. The numbers within the histograms represent the number of patients. The remission rates are expressed as patients achieving a Montgomery Asberg Depression Rating Scale score of 10 or less in Blier et al. [39] and a Hamilton Depression-17 item score of 7 or less in Blier et al. [40]. (Online version in colour.)

Serotonin neurons exert an inhibitory influence on DA neurons in the ventral tegmental area (VTA), which give rise to mesolimbic and mesocortical projections [41]. The SSRI escitalopram suppresses the mean firing rate of VTA DA neurons through enhanced activation of 5-HT_2C receptors, likely on GABA neurons [42]. It is thus also conceivable that the lack of therapeutic benefits of SSRIs in some depressed patients may result from an attenuation of DA transmission in the presence of enhanced 5-HT levels. The inhibitory action of SSRIs on catecholamine neurons may also account for residual symptoms often observed after remission has been achieved with SSRIs.

Attempts to reverse the dampening action of enhanced 5-HT levels produced by SSRIs on catecholamine neurons have led to effective augmentation strategies for MDD. For instance, adding an NE reuptake inhibitor to the regimen of SSRI-resistant patients, or adding DA agonists in treatment-resistant patients [43,44] can be beneficial. This basic approach underlies the common enhancing action of low doses of atypical antipsychotics in patients not responding to SSRIs. Indeed, at subtherapeutic doses of atypical antipsychotics for schizophrenia, 5-HT_2A and 5-HT_2C receptors are potently blocked. In the rat brain, 5-HT_2A and 5-HT_2C receptor antagonists reverse the inhibitory effect of SSRIs on NE and DA neurons, respectively, including atypical antipsychotics [29,35,44,45].

5. The forebrain structures involved in the antidepressant response

Patients with MDD generally exhibit hyperactivity of the hippocampus, the amygdala and the subgenual anterior cingulate cortex; when they respond to antidepressant strategies, this hyperactivity is diminished [46–49]. In such structures, or their equivalent in the rat brain, 5-HT, NE and DA generally exert an inhibitory action on neuronal firing as shown by their direct microiontophoretic application in vivo [13,16,28]. Consequently, the antidepressant response may be triggered by medications potentiating inhibitory transmission in limbic structures through one or more of these monoaminergic systems. The effectiveness of combination of antidepressant medications in treatment-resistant patients may thus be better understood through a greater potentiation of inhibitory transmission in such structures.

In the dorsolateral frontal cortex, the work of Mayberg documented an increase in activity in MDD patients responding to antidepressants [48,49]. Interestingly, 5-HT classically exerts a net excitatory effect on pyramidal neurons in this area of the frontal cortex [50]. It is thus conceivable that enhanced 5-HT transmission at such sites may contribute to the therapeutic action of some antidepressant treatments.

6. Combination of 5-HT targets to improve effectiveness of treatments

The first-line treatment for MDD is now selective reuptake inhibition. This approach was obtained by extracting and enhancing one of the properties of the prototypical agent imipramine. In so doing, the SSRIs have become a class of drugs that can no longer be used to commit suicide, in contrast to TCAs, and are much more tolerable. They can still produce cumbersome side effects, like sexual dysfunctions, which is often a significant hurdle for their long-term use. One strategy to minimize the latter problem has been to use compounds with low potency at the 5-HTT and acting on other 5-HT receptor targets. Trazodone is such a low-potency 5-HTT inhibitor, a 5-HT_2A/2C antagonist and a 5-HT_1A agonist; furthermore, in its new slow-release preparation daytime sedation has been minimized [51,52]. Vortioxetine (Lu AA21004) is another low-potency 5-HTT inhibitor that acts as an agonist on 5-HT_1A and 5-HT_1B receptors, and as an antagonist of 5-HT₃ and 5-HT₇ receptors. Interestingly, vortioxetine has been shown to be as effective as a regimen of venlafaxine that blocks both 5-HT and NE reuptake [53,54].

Such new and old medications have not, however, significantly enhanced the standard remission rate that hovers around 30–40%. Even the rapidly acting ketamine infusions still produce such a low success rate (see below; [55–57]). Drug combinations at adequate regimens have nevertheless pushed up the remission rate when used from treatment initiation in proof of concept studies. These include combinations of SSRIs or the serotonin and norepinephrine reuptake inhibitor venlafaxine with the noradrenergic TCA desipramine or mirtazapine [39,40,43]. While such combinations act on more than one 5-HT neuronal target, they also exert noradrenergic effects, such as blocking the NE reuptake or presynaptic α₂-adrenoceptors. In the case of the mirtazapine combinations, the remission rate after six weeks was approximately doubled compared with SSRIs used alone [39,40] (figure 3). In this context, it is important to emphasize that clinical studies in MDD may be flawed by trial design, the selection of patients with reliable symptomatology and high dropout rates. Nevertheless, clinical results using combinations of drugs clearly indicate that when multiple targets are solicited, a greater proportion of patients with MDD can be brought to remission [58].

7. Beyond serotonin and other monoamines to an antidepressant response

There are several strategies that may not directly target monoamine systems and trigger an antidepressant response. Sleep deprivation, for instance, may produce a rapid antidepressant response, albeit very transient [59]. This strategy, like several others, can indirectly affect the function of monoamine systems. In the case of sleep deprivation, it was reported that in freely moving cats, the firing frequency of 5-HT neurons is enhanced one day after the deprivation, a time when the antidepressant benefits are seen in patients with MDD [60].

A novel antidepressant strategy producing strikingly rapid therapeutic benefits in MDD has provided credence to the theory that neuronal hyperactivity in limbic structures may account in large part for the pathology of MDD. The injection of a subanaesthetic dose of the glutamate N-methyl-d-aspartate (NMDA) antagonist ketamine can produce, within hours, a robust antidepressant response, presumably in part by blocking excitatory NMDA receptors [55–57]. In other words, within hours ketamine may produce a dampening of the excessive neuronal activity in limbic structures that antidepressants take weeks to build up by enhancing inhibitory monoaminergic transmission. It is also noteworthy that even before the first controlled observations of the antidepressant action of ketamine infusion, it had been reported that a variety of classical antidepressants induce a decreased function of NMDA receptors, probably through their monoaminergic properties [61].

Deep brain stimulation is another novel, yet still experimental, approach to treat refractory MDD. Thus far, stimulations at high frequencies of the subgenual anterior cingulate cortex have been the most studied [62]. Consistent with the attenuated hyperactivity that decreases with effective antidepressant treatments, deep brain stimulation also produces the same phenomenon. Similar stimulations applied to the equivalent region in the rat brain leads to an enhancement of the firing rate of 5-HT neurons in the dorsal raphe nucleus and an increase in the extracellular levels of 5-HT in the hippocampus [63,64]. Furthermore, the antidepressant-like action of deep brain stimulation is abolished in rats in which 5-HT neurons have been lesioned [64]. This is yet another example of the important connectivity between various brain structures involved in mood disorders.

8. Conclusion

In closing, the 5-HT system still appears to play an important role both in the pathophysiology and in the treatment of MDD. Above all, it has been a remarkable target for the development of antidepressant strategies. Clinically meaningful progress has been achieved by exploiting the capacity of various 5-HT neuronal elements to modulate overall synaptic 5-HT transmission, and more is still to come. Nevertheless, novel targets have been identified and more are eagerly awaited so that more patients with MDD can be brought to remission and more rapidly. Given the functional connectivity between monoamine systems and other neurotransmitters, it appears unlikely that even novel antidepressant strategies will exert their antidepressant activity totally independent of these monoamine systems.