Modulation of Midbrain Dopamine Neurotransmission by Serotonin, a Versatile Interaction Between Neurotransmitters and Significance for Antipsychotic Drug Action

Abstract

Schizophrenia has been associated with a dysfunction of brain dopamine (DA). This, so called, DA hypothesis has been refined as new insights into the pathophysiology of schizophrenia have emerged. Currently, dysfunction of prefrontocortical glutamatergic and GABAergic projections and dysfunction of serotonin (5-HT) systems are also thought to play a role in the pathophysiology of schizophrenia. Refinements of the DA hypothesis have lead to the emergence of new pharmacological targets for antipsychotic drug development. It was shown that effective antipsychotic drugs with a low liability for inducing extra-pyramidal side-effects have affinities for a range of neurotransmitter receptors in addition to DA receptors, suggesting that a combination of neurotransmitter receptor affinities may be favorable for treatment outcome.

This review focuses on the interaction between DA and 5-HT, as most antipsychotics display affinity for 5-HT receptors. We will discuss DA/5-HT interactions at the level of receptors and G protein-coupled potassium channels and consequences for induction of depolarization blockade with specific attention to DA neurons in the ventral tegmental area (VTA) and the substantia nigra zona compacta (SN), neurons implicated in treatment efficacy and the side-effects of schizophrenia, respectively. Moreover, it has been reported that electrophysiological interactions between DA and 5-HT show subtle, but important, differences between the SN and the VTA which could explain (in part) the effectiveness and lower propensity to induce side-effects of the newer atypical antipsychotic drugs. In that respect the functional implications of DA/5-HT interactions for schizophrenia will be discussed.

DOPAMINE AND SCHIZOPHRENIA

The mesocortical pathway, the mesolimbic pathway, the nigrostriatal pathway and the tuberoinfundibular pathway have all been postulated to be involved in the pathophysiology of schizophrenia and the propensity of antipsychotic drugs to induce side-effects [144]. Hypofunction of the mesocortical pathway and hyperfunction of the mesolimbic pathway [44, 55, 144] are thought to be responsible for the symptoms that can be observed (see also Sesack and Carr, 2002 [138])) and points to one of the many difficulties for effective treatment: increasing dopamine (DA) activity in the mesocortical pathway, while concomitantly decreasing DA activity in the mesolimbic pathway. The nigrostriatal and tuberoinfundibular pathways are involved in side-effects of antipsychotic drug treatment, such as extra-pyramidal side effects and hyperprolactinemia, respectively [10, 30] which are related to changes in firing activity of neurons in these pathways, especially the DA neurons. What lies at the root of this mesocortical mesolimbic dysfunction is unclear but loss of cholinergic interneurons in the striatum, hypoglutamatergia or "miswiring" of glutamatergic and _-amino butyric acid (GABA)-ergic projections from the prefrontal cortex (PFC) have all been proposed [5, 22, 23, 34, 66, 154]. The glutamatergic and GABAergic PFC projections synapse directly and indirectly {via e.g. the nucleus accumbens) to VTA DA neurons (Fig. (1), partly based on [138]). The GABAergic inputs together with local GABAergic interneurons in the VTA and SN (Fig. (1)) reduce DA neuronal firing activity by exerting an inhibitory tone on DA neuronal activity through GABA_A and GABA_B receptors present on DA neurons [26, 43, 151]. Thus, dysfunction of the PFC pathway could disinhibit VTA DA neurons leading to hyperactivity of VTA neurons.

Fig. (1).

A schematic view of SN and VTA connections. DA neurons receive input from the local GABAergic interneurons and input from other brain areas. Both areas receive serotonergic input from the dorsal raphe nucleus (DEN). The dotted line indicates the subdivision of the SN: the SN pars reticulata (SN_r) and the SN pars compacta (SN_C). Neurons receiving a collective input are grouped (dotted circles). GABAergic, glutamatergic and serotonergic inputs are indicated by black, white and gray symbols, respectively. HIPP, hippocampus; NAC, nucleus accumbens; PFC, prefrontal cortex; STN, subthalamic nucleus.

THE DA RECEPTOR AND ANTIPSYCHOTIC DRUGS

The discovery of DA in the brain [8] and subsequent discovery in the late 1950s that schizophrenia could be treated with antipsychotic drugs, which antagonize DA D₂ receptors, reducing mesolimbic DA neuronal hyperactivity, has led to the DA hypothesis for schizophrenia. This hypothesis has been refined recently to incorporate other neurotransmitters such as glutamate, GABA and serotonin (5-hydroxytryptamine; 5-HT) [11, 23, 24, 136, 155]. However, attempts to develop an effective antipsychotic drug that lacks DA D₂ receptor antagonism have been, generally, unsuccessful. For example selective 5-HT₂A receptor antagonists [75, 92] and DA D₄ receptor antagonists [12, 169] failed to show efficacy in schizophrenia reinforcing a pivotal role for DA D₂ receptors in the treatment efficacy of schizophrenia. However, recent studies have suggested that the neurokinin (NK)₃ receptor antagonist does have effects in schizophrenia [100].

DA D₂ RECEPTORS AND DA NEURON FIRING ACTIVITY

DA D₂-like receptors are present as auto-receptors on the DA neurons in SN and VTA and play an important role in the regulation of DA neuronal firing activity by means of auto-inhibition (Fig. (1). These G protein-coupled receptors are activated by somatodendritically released DA [7] and their activation opens G protein-coupled inward rectifying potassium channels (GIRKs) (Fig. (2)). Opening of the GIRK channels leads to hyperpolarization of the cell membrane and consequently a decrease in firing activity [81]. Furthermore, the hyperpolarization mediated via the DA D₂ auto-receptor-activated GIRK channels decreases somatodendritical release of DA [141], allowing the DA neuron to depolarize via voltage-dependent calcium current activation and the non-selective cation current I_h [139, 168]. This in turn leads to increased DA release, increased DA D₂ auto-receptor-mediated GIRK current and so on, contributing to the maintainance of a spontaneous, pace-maker-like firing pattern.

Fig. (2).

Model scheme for atypical antipsychotic drug action. The two schemes above represent the situation under normal conditions, where activation of 5-HT receptors (an unknown (5-HT£) receptor on SN and the 5-HT₂ receptor on VTA DA neurons) can enhance the DA D₂ receptor-mediated auto-inhibitory process. The two schemes below represent the situation in the presence of an atypical antipsychotic drug. This drug antagonizes a portion of the DA D₂ receptors, thereby reducing GIRK channel function, resulting in depolarization and increased firing activity of the DA neurons. Prolonged depolarization could, theoretically, lead to the induction of depolarization blockade and subsequently therapeutic efficacy and extra-pyramidal side-effects (EPS). In VTA DA neurons an atypical antipsychotic drug will, by blocking the 5-HT₂ receptors, prevent 5-HT₂-mediated enhancement of GIRK channels, and thus auto-inhibition. This would permit a further depolarization, and ultimately depolarization blockade. In SN DA neurons "normal" enhancement of auto-inhibition can occur through the 5-HT_x receptor (which is not affected by the antipsychotic drug), a process that will reduce the chance that depolarization blockade will develop.

DEPOLARIZATION BLOCKADE THEORY AND ANTIPSYCHOTIC DRUG EFFICACY

In vitro studies demonstrate that DA neurons usually fire at low pacemaker-like frequencies of 1-8 Hz [61, 162] and this pacemaker-like firing is also observed in vivo, although DA neurons in vivo also display irregular and burst firing activity [18, 47, 140]. In general, it is thought that the tonic release of DA (via regular firing rates) serves to maintain a steady-state level of DA in the brain, while the phasic DA release through bursting activity gives rise to high but transient DA levels which convey discrete signals [42, 46]. Switching the firing pattern from regular to bursting or back is thought to be dependent on coinciding glutamatergic and cholinergic inputs from the subthalamic and pedunculo-pontine nuclei, respectively [80, 87, 130, 159]. In the absence of coinciding glutamate and acetylcholine signaling, switching to a bursting firing pattern is unlikely, although the firing rate can be increased through depolarization of the DA neuron. It has been shown that upon depolarization (e.g. through GIRK channel inhibition) DA neurons cannot sustain increases in the firing rates above a plateau level [170]. At depolarization levels above that producing maximum frequency, DA neuronal firing ceases; such a state is referred to as acute depolarization blockade, a mechanism that could function to curb neuronal excitation [59, 65, 145, 170]. Although difficult to prove in vivo, it is believed that antipsychotic drugs induce a more or less chronic depolarization blockade in VTA DA neurons, thereby effectively reducing mesolimbic hyperactivity [60]. The development of depolarization blockade by antipsychotic drug treatment probably involves the blockade of the DA D₂-like autoreceptors on the VTA DA neurons and activation of excitatory feedback systems, a process that develops over time (i.e. during chronic antipsychotic drug treatment) and is often characterized by an initial increase in DA neuronal activity. However, this mechanism of depolarization blockade will also be effective in SN DA neurons. This is reflected by one of the side-effects of the classical antipsychotic drugs: extra-pyramidal side effects (EPS), a collective name for symptoms such as dystonia, akinesia, dyskinesia, and akathisia [19, 91]. Due to the high liability for EPS induction the search for better antipsychotics continued. The introduction of clozapine was a large step-forward in achieving this goal [9, 17, 29, 49, 94, 111, 142] and mesolimbic hyperactivity could be reduced without inducing EPS. Clozapine, and other so-called atypical antipsychotics known as such for their low propensity to induce EPS [31, 67, 150], has a much lower tendency to induce EPS compared to the older, classical antipsychotic drugs [149]. However, it has been demonstrated that a selective depolarization blockade of only VTA DA neurons may be involved in the underlying mechanism [14]. In vivo studies suggest that atypical antipsychotic drugs appear to preferentially modulate VTA DA neuronal firing activity [27,28,58, 167]. Pharmacological studies have demonstrated that most atypical antipsychotic drugs are not only DA D₂ antagonists, but also have affinities for a range of other neurotransmitter receptors such as 5-HT₂, adrenergic, muscarinic and histamine receptors [20, 21]. These additional receptor affinities could contribute to the selective development of depolarization blockade of only VTA DA neurons and consequently the lower incidence of EPS and improved efficacy compared to the classical antipsychotic drugs.

Besides the development of depolarization blockade of VTA DA neurons, other mechanisms, for instance at the level of the prefrontal cortex, are also likely to improve antipsychotic drug efficacy. It is reported that clozapine facilitates NMDA-mediated neurotransmission in prefrontal cortical neurons [3] and atypical antipsychotic drugs, but not classical antipsychotic drugs, increase acetylcholine release in the forebrain [69]. The release of DA and noradrenaline is also increased by antipsychotic drugs [126, 166]. For a comprehensive review on these mechanisms in the forebrain and their implications for schizophrenia treatment see Moore et al., 1999 ([108]).

Not all atypical antipsychotic drugs combine DA D₂ receptor antagonism with affinities for other receptors [107], and it has been suggested that multi-receptor affinity is not the key to atypical antipsychotic effectiveness. It has been proposed that their effectiveness is related to selective affinities for DA receptors in specific areas of the brain or receptors in specific conformational states [107]. Also a fast dissociation theory has been proposed [73, 74, 109]. This theory suggests a "hit-and-run" action of the antipsychotic drug at the DA receptor insofar as the drug rapidly binds and then dissociates from the receptor, thereby not "rigidly" blocking all DA neurotransmission. This mechanism might explain the clinical atypical antipsychotic profile of amisulpride, a pure D₂/D₃ receptor antagonist, although it has been suggested that the possible selective preference for mesolimbic versus nigrostriatal DA D₂-like receptors by amisulpride or its D₃ receptor antagonism might account for the atypicality [86, 107, 122]. So far, not all the evidence for the “hit-and-run” theory is conclusive and the hypothesis that multi-receptor affinities underlie atypicality is still very much favored [101]. From the large variety in receptor affinities, besides DA D₂ receptors that are displayed by atypical antipsychotic drugs, especially affinity for the 5-HT(₂) receptor has received considerable attention for its potential role in atypicality.

5-HT AND DA PATHWAYS

In vivo DA-dependent behaviors can be modulated by 5-HT receptor activation, as demonstrated using 5-HT₁ 5-HT₂ and 5-HT₄ receptor agonists. For example synergistic enhancement of the acoustic startle reflex by 5-HT_1A receptor agonists [98] or 5-HT₂ and 5-HT₄ receptor-mediated modulation of cocaine-induced locomotor activity [95–97]. In addition, DA release in the medial prefrontal cortex (mPFC), dorsal and ventral striatum can be increased by (cortical) 5-HT_2A receptor activation, while 5-HT_2C receptors suppress DA release (likely via effects on DA cell bodies) [33, 56, 57, 119, 120]. Cortical 5-HT_1A receptor activation appears to increase DA release in the mPFC, possibly in concert with mesolimbic 5-HT₂ receptor antagonism [70]. Others have also extensively demonstrated that 5-HT facilitates DA release and neurotransmission via 5-HT receptors in the forebrain, with specific roles for 5-HT₁, 5-HT₂ and 5-HT₄ receptors [6, 76] and secondary messengers, such as protein kinase A [164] or nitric oxide [165]. Moreover, the modulation of DA synthesis and release is under control of different mechanisms [48, 77, 165], often depending on the projection field [127, 156]. These findings show that 5-HT influences and shapes DA neurotransmission by actions at the somatic and terminal-field level. The findings that DA neurotransmission to the PFC is influenced by 5-HT suggests a role for 5-HT receptors in antipsychotic modulation of mesolimbic hyperactivity (for a comprehensive review see Werkman et al., in press ([161])).

5-HT AND SN AND VTA DA NEURON ACTIVITY

VTA and SN DA neurons (as well as the local GABAergic interneurons) express a range of 5-HT receptor types that can be activated by 5-HT input from the dorsal raphe nucleus (DRN) [62] (Fig. (1)). In Table (1) a summary is provided of the 5-HT receptors and the effect their activation has on DA neuronal firing activity. Some of these receptors have been found on GABAergic interneurons, such as the 5-HT_1B and 5-HT_2C receptor, although the latter might also be present on the DA neurons [128]. The activation of 5-HT_1B receptors inhibits GABA release [71, 171], while 5-HT_2C receptor activation excites GABAergic interneurons [36] and stimulation of the 5-HT_1A receptor has been reported to excite DA neurons [84]. 5-HT₃ receptors are not well studied yet in the SN and VTA, but based on findings in other brain areas they are expected to be located presynaptically [110]. In vivo, i.v. administration of 5-HT₃ receptor antagonists is reported to increase DA neuronal firing, although the mechanism underlying this remains unclear [105, 117]. 5-HT₄ receptor activation increases nigral DA firing activity, but it is not clear if the receptors are present on the DA neurons or GABAergic interneurons [13, 90, 118]. The 5-HT₆ receptor is present on interneurons, and possibly also on the terminals of GABAergic and possibly cholinergic projections to DA neurons [50] but is not thought to be directly involved in modulating DA neurotransmission, but likely indirectly via the regulation of cholinergic neurotransmission [15]. Little is known about the presence or function of the 5-HT₇ receptor in SN and VTA areas, although it has been proposed as an interesting candidate for future antipsychotic drugs, as a number of atypical antipsychotics bind to this 5-HT receptor subtype [99, 133].

Table 1.

5-HT Receptor Types in SN and VTA. An Overview of 5-HT Receptor Types with Their Locations in SN and VTA Areas and Reported (Indirect) Effects on DA Neuron Firing Activity. n.d.=not Determined

	Location	DA neuron firing activity	references
5-HT<sub>1A</sub>	DA neurons Tresynaptic processes	increase or pattern change	[2,41,85, 106, 123]
5-HT<sub>1B</sub>	GABAergic interneuron	increase	[71, 171]
5-HT<SUB>M</SUB>	DA neurons	increase	[40, 114, 123]
5-HT<sub>2C</sub>	GABAergic interneuron DA neuron?	decrease	[38,57, 157]
5-HTj	n.d.	increase	[104, 110, 117]
<bold>5-HT<sub>4</sub></bold>	n.d.	increase	[152]
<bold>5-HT<sub>6</sub></bold>	GABAergic interneuron DA neuron?	n.d.	[50, 132]

These findings, and the reports that 5-HT receptor-induced changes in DA neuronal activity are dependent on the activation state of the DA neurons, i.e. when the DA impulse flow is activated [37, 90, 129, 143], show that 5-HT regulates DA neuronal firing activity both in a tonic and phasic manner. Furthermore, differential roles appear to exist for some 5-HT receptor types, such as the 5-HT₂ receptors, i.e. 5-HT_2C receptors change DA neuronal firing activity and bursting in VTA, but not in SN DA neurons [35, 39, 116], while 5-HT₄ receptors play a more pronounced role in SN DA neurons [90].

The existence of differential roles for a 5-HT receptor type can be related to the diversity of second messenger pathways that 5-HT receptors utilize [4]. Most 5-HT receptors are G protein-coupled receptors (GPCRs) (with the ligand-gated 5-HT₃ receptor being the exception) and they can activate second messenger systems such as phospholipase C, and ion channels linked to specific G protein subunits, including GIRK channels [54, 115] (for a comprehensive review see Barnes and Sharp, 1999 ([4])). 5-HT receptor-mediated activation of second messenger pathways and ion channels can directly increase or decrease DA neuronal firing activity as well as influence the neuronal response to activation of other receptors, such as DA D₂ receptors [16, 51, 88, 112, 116, 124].

5-HT MODULATION OF DA D₂ RECEPTOR-MEDIATED INHIBITION

As mentioned previously, midbrain DA neurons tonically regulate their own firing activity via a feedback auto-inhibition. Extracellular recordings of SN and VTA DA neurons in vitro have shown that concentrations of 5-HT that do not change the firing rate when applied alone enhanced DA D₂ receptor-mediated auto-inhibition [16, 116]. It was demonstrated that this enhancement of auto-inhibition was mediated via different 5-HT receptors in the SN and VTA, with 5-HT₂ receptors being critically involved in the VTA, in line with in vivo findings that 5-HT₂ receptors differentially affect DA neuronal activity in SN and VTA [35, 39]. Various mechanisms underlying this DA/5-HT interaction have been suggested. One proposed mechanism is the inhibition of the Ih current [88]. However, 5-HT concentrations higher than the concentrations used to enhance auto-inhibition were necessary to inhibit Ih [16]. This finding and previous reports that inhibition of Ih current occurs secondary to GIRK channel activation by DA or GABA [25, 102, 160] suggest that additional mechanisms may also involved. The close link between GIRK channel and Ih channel activities [147, 148] is not fully understood, but as GIRK channels play a prominent role in the regulation of DA neuronal firing by translating DA D₂ receptor activation into inhibition of firing activity, 5-HT receptor-mediated modulation of GIRK channels may underlie 5-HT-enhanced auto-inhibition (Fig. (2)).

GIRK CHANNEL MODULATION

GIRK channels have up to four binding sites for βγ G protein subunits. Upon activation of a GPCR such as the DA D₂ receptor, the attached G protein dissociates and separates into the α and the βγ subunit. Binding of one βγ subunit only partially activates the channel and additional binding (to 3-4 subunits) results in full activation of the channel [32, 113, 134, 135]. It has been established that this graded activation is a form of positive co-operativity [134]. Besides the capability of binding multiple βγ subunits that contribute in a graded manner to channel (in)activation, it has been suggested recently that GIRK channels can be inhibited by binding of the G protein α subunit to the channel [83, 121, 137, 172].

Thus, different GPCRs can influence their respective effects on GIRK channel activation directly at the level of the channel. For instance, GABA_B and DA D₂ receptors activate the same GIRK channel. Submaximal activation of both receptors results in an additive effect on the GIRK current [81], probably due to the positive co-operativity of the βγ subunits. Such effects at the level of the GIRK channel itself can also lead to inhibitory actions. Neurotensin receptors are coupled to G proteins that inhibit GIRK function; simultaneous activation of neurotensin and DA D₂ receptors in midbrain DA neurons results in a decreased GIRK current [45] and consequently an attenuation of DA D₂ receptor-mediated auto-inhibition [163]. Furthermore, receptors that activate second messenger pathways (such as 5-HT receptors) can modulate GPCR-activated GIRK channels, as it has been shown that intracellular components such as protein kinases and phosphatidyl inositol diphosphate (PIP₂) are able to stabilize the channel in the open state or facilitate βγ subunit binding [63, 64, 68, 82, 89, 125, 135].

Altogether, GPCR stimulation and intracellular mechanisms that influence GPCR and G protein coupling can affect GIRK channel activation, and thus the effect of GPCR activation on DA neuronal firing. Such interactions have been demonstrated for estrogen receptors in hypothalamic neurons, which upon activation, disrupt the coupling of GABA_B receptors to GIRK channels through an action on the G protein [78, 79]. As the relative size of the G protein pools that GPCRs have access to differs [115], it is possible to attain a weighted effect on GIRK channel activity upon receptor activation through different input systems.

FUNCTIONAL MECHANISM OF 5-HT MODULATION AND IMPLICATIONS FOR ANTIPSYCHOTIC DRUG ACTIVITY

5-HT₂ receptors differentially affect SN and VTA DA neuronal activity, both in vitro and in vivo. In animal models of schizophrenia, limited as they are, it appears that 5-HT₂ receptor antagonism indeed plays a role in a differential modulation of DA neuron firing activity by antipsychotic drugs [52, 53, 72, 146, 153] and this may underlie the reduced EPS liability in drugs possessing mixed D₂ / 5-HT_2A receptor modalities.

How are the affinities of atypical antipsychotic drugs for DA D₂ and 5-HT₂ receptors translated to functional changes in mesolimbic (and possibly mesocortical) DA neuronal firing? DA D₂ receptor-mediated auto-inhibition can be enhanced by 5-HT₂ receptors in the VTA, but in the SN probably another 5-HT receptor type is involved [116] (Fig. (2)). Antagonism of part of the DA D₂ receptor population increases the firing activity of DA neurons in SN and in VTA [103, 131, 162]. However, in the presence of 5-HT, the inhibitory effect remaining through DA D₂ receptor activation will be enhanced, thus (partly) counteracting the effect of the DA D₂ receptor antagonism. This can occur in both SN and VTA DA neurons. In the case of treatment of schizophrenia, the preferential effect of the atypical antipsychotic drugs is the induction of depolarization blockade in VTA DA neurons, while leaving the firing activity of SN DA neurons largely unchanged. Classical antipsychotic drugs, as discussed before, will induce depolarization blockade in both areas [28]. In contrast, atypical antipsychotic drugs have an additional feature that could assist in achieving selective depolarization blockade [1, 60]. The atypical antipsychotic drugs also block the DA D₂ receptors in SN and VTA DA neurons. However, the 5-HT₂ receptor antagonistic action of these drugs [20] results in a blockade of the 5-HT-mediated enhancement of the remaining DA auto-inhibition in VTA DA neurons, allowing a further depolarization and increase in firing rate (Fig (2)), ultimately leading to depolarization blockade. In SN DA neurons however, since another 5-HT receptor subtype may be responsible for the enhancement of auto-inhibition, the remaining DA D₂ receptor mediated auto-inhibition can still be enhanced. In general, it is thought that 5-HT₂ receptor antagonism increases the likely-hood that DA D₂ receptor antagonism induces depolarization blockade of VTA DA neurons and not of SN DA neurons, thereby decreasing the liability for EPS induction. This mechanism is supported by the observations that elevated 5-HT level seems related to clinical efficacy of some atypical antipsychotics, and that atypical antipsychotics can increase 5-HT levels in the brain [93, 158], but this appears very brain area dependent.

In conclusion, 5-HT receptor-mediated modulation of DA neuron physiological function involves direct effects on DA neuron firing activity and indirect effects such as the enhancement of DA D₂ receptor-mediated auto-inhibition. Moreover, 5-HT₂ receptors have been established to be differentially involved in shaping DA D₂ receptor-mediated auto-inhibition, possibly through actions on the GIRK channel via second messenger pathways. This indicates that this channel plays a central role in determining SN and VTA DA neuron activity and possibly the development of depolarization blockade. Differences in control mechanisms of the GIRK channel by DA and 5-HT receptors in both SN and VTA areas allows compounds like atypical antipsychotic drugs to have differential effects on the electrical activity of SN and DA neurons. A pivotal role for the modulation by 5-HT on DA neurotransmission has been established, not only in SN and VTA, but also in areas such as the mPFC. This points to the relevance of 5-HT₂ receptor modulation of DA neurotransmission in schizophrenia, either in understanding the pathophysiology of the disease or, perhaps more pronounced, in the treatment of the disease with antipsychotic drugs.