Sensing Positive versus Negative Reward Signals through Adenylyl Cyclase-Coupled GPCRs in Direct and Indirect Pathway Striatal Medium Spiny Neurons

Abstract

Transient changes in striatal dopamine (DA) concentration are considered to encode a reward prediction error (RPE) in reinforcement learning tasks. Often, a phasic DA change occurs concomitantly with a dip in striatal acetylcholine (ACh), whereas other neuromodulators, such as adenosine (Adn), change slowly. There are abundant adenylyl cyclase (AC) coupled GPCRs for these neuromodulators in striatal medium spiny neurons (MSNs), which play important roles in plasticity. However, little is known about the interaction between these neuromodulators via GPCRs. The interaction between these transient neuromodulator changes and the effect on cAMP/PKA signaling via G_olf- and G_i/o-coupled GPCR are studied here using quantitative kinetic modeling. The simulations suggest that, under basal conditions, cAMP/PKA signaling could be significantly inhibited in D1R+ MSNs via ACh/M4R/G_i/o and an ACh dip is required to gate a subset of D1R/G_olf-dependent PKA activation. Furthermore, the interaction between ACh dip and DA peak, via D1R and M4R, is synergistic. In a similar fashion, PKA signaling in D2+ MSNs is under basal inhibition via D2R/G_i/o and a DA dip leads to a PKA increase by disinhibiting A2aR/G_olf, but D2+ MSNs could also respond to the DA peak via other intracellular pathways. This study highlights the similarity between the two types of MSNs in terms of high basal AC inhibition by G_i/o and the importance of interactions between G_i/o and G_olf signaling, but at the same time predicts differences between them with regard to the sign of RPE responsible for PKA activation.

SIGNIFICANCE STATEMENT Dopamine transients are considered to carry reward-related signal in reinforcement learning. An increase in dopamine concentration is associated with an unexpected reward or salient stimuli, whereas a decrease is produced by omission of an expected reward. Often dopamine transients are accompanied by other neuromodulatory signals, such as acetylcholine and adenosine. We highlight the importance of interaction between acetylcholine, dopamine, and adenosine signals via adenylyl-cyclase coupled GPCRs in shaping the dopamine-dependent cAMP/PKA signaling in striatal neurons. Specifically, a dopamine peak and an acetylcholine dip must interact, via D1 and M4 receptor, and a dopamine dip must interact with adenosine tone, via D2 and A2a receptor, in direct and indirect pathway neurons, respectively, to have any significant downstream PKA activation.

Introduction

Reinforcement learning is a kind of association building where an organism, based on its prior experiences, associates an environmental stimulus to either a reward-producing response or to the reward itself by updating the salience of the stimulus. Basal ganglia play a crucial role in reinforcement learning, and the input nucleus of basal ganglia, striatum, receives sensorimotor and reward-related signals (Liljeholm and O'Doherty, 2012). The reward-related signal, encoded as a reward prediction error (RPE), which is the difference between the expected and the obtained reward, is typically represented by the activity of midbrain dopaminergic neurons (DAN) projecting onto striatum (Schultz, 1998; Steinberg et al., 2013). A salient cue or an unexpected reward produces burst activity in DANs (positive RPE), whereas omission of an expected reward leads to a depression in DAN activity (negative RPE) (Schultz, 1998). The positive RPE signal increases the striatal dopamine (DA) concentration, whereas the negative RPE signal decreases it (Hart et al., 2014). The increase in DAN activity is often accompanied by a simultaneous pause in the activity of striatal cholinergic interneurons, which largely constitute a population of tonically active neurons (TANs), resulting in a dip in the striatal acetylcholine (ACh) level (Aosaki et al., 1995; Goldberg and Reynolds, 2011). These striatal DA and ACh signals are fast, but other neuromodulators, such as adenosine (Adn), seem to change more slowly (Cechova and Venton, 2008; Aosaki et al., 2010; Pajski and Venton, 2010). There are several adenylyl cyclase (AC5)-coupled GPCRs for these neuromodulators in striatal neurons (Lovinger, 2010). However, little is known, quantitatively, about the way in which the aforementioned transient neuromodulatory signals might interact and affect the AC activity via the G_olf and G_i/o type G-proteins.

Striatum is mainly populated by the medium sized spiny projection neurons (MSNs). There are two types of MSNs, striatonigral (D1+ MSNs) and striatopallidal (D2+ MSNs), each of which expresses a unique dopamine receptor class; D1 receptors (D1R) coupled to G_olf in D1+ MSNs and D2 receptors (D2R) coupled to G_i/o in D2+ MSNs. Striatal D1R is known to be expressed with the muscarinic M4 acetylcholine receptor (M4R), which is coupled to G_i/o, whereas D2R with the adenosine A2a receptors (A2aR), which are coupled to G_olf (Fig. 1) (Fink et al., 1992; Ferré et al., 1993; Hersch et al., 1994, 1995; Ince et al., 1997). These receptors show functional interaction and regulate the AC5/cAMP/PKA signaling, which in turn is important for the long-term potentiation of the corticostriatal synapses and motor learning (Brandon et al., 1998; Price et al., 1999; Svenningsson et al., 2000; Onali and Olianas, 2002; Augustin et al., 2014). In this study, we explored interactions between the aforementioned transient neuromodulatory signals via G_olf- and G_i/o-coupled receptors in MSNs and how they affect the downstream cAMP/PKA signaling using quantitative kinetic modeling. Our results suggest that, in D1+ MSNs, AC5 is tonically inhibited by the ACh/M4R/G_i/o pathway under the basal condition; and a DA peak, in response to a positive RPE, is not able to activate the AC5/cAMP/PKA signaling significantly unless it is accompanied by an ACh dip. The ACh dip leads to a reduction in the M4R-mediated inhibition of AC5 and thus acts as a gate for the PKA activation by the DA peak. On the other hand, in D2+ MSNs, a DA dip stimulates cAMP/PKA by disinhibiting the A2aR signaling, which is tonically inhibited by D2R under basal conditions. These results highlight the important role of M4R and A2aR interaction with DA receptors in MSNs along with the common theme that a dip in G_i/o activity is required to gate G_olf-dependent PKA responses in these neurons. The basal inhibition of AC by G_i/o and gating of G_olf signaling by a decrease in G_i/o could be viewed as a general mechanism to filter out spurious G_olf signals.

Figure 1.

An overview of the biochemical network modeled in this study. A, The biochemical network for the D1+ MSN. B, The biochemical network for the D2+ MSN. The signaling core is similar, but the receptors coupled to the Gi (G_i/o) and Golf (G_olf) proteins are different, and this opens up for alternative processing of changes in neurotransmitter signaling in the two neuron types.

Materials and Methods

Biochemical networks.

Here, we study the functional interactions between the following receptor pairs: (1) interaction between D1R and M4R in the striatal direct pathways, D1+ MSNs; and (2) interaction between D2R and A2aR in striatal indirect pathways, D2+ MSNs.

To this end, we developed two related model variants: one for D1+ MSN and the second for D2+ MSN. The biochemical networks modeled for both the cell types focus on the intracellular signaling events related to the cAMP/PKA cascade (Fig. 1). The biochemical reactions in each network are modeled as ordinary differential equations according to the mass-action kinetics with a well-mixed approximation. The resulting system of ordinary differential equations is solved using ode15s solver in the Simbiology toolbox of MATLAB (The MathWorks). The models are deposited in BioModels database (Chelliah et al., 2014) in SBML format along with tabular description (Model id: MODEL1502200000 for D1+ MSN; Model id: MODEL1502200001 for D2+ MSN).

D1+ MSN model.

The reaction network modeled for the D1+ MSN takes into account a subset of intracellular signaling triggered by the activation of the D1R and M4R in these neurons. The signaling cascades triggered by the two receptors converge on AC5, which is the predominant adenylate cyclase in MSNs (Glatt and Snyder, 1993), where M4R-activated G_i/o inhibits the D1R/G_olf-induced AC5 activation. The downstream signaling module is a reparameterized version of the AC5/cAMP/PKA axis of previously developed signaling models (Lindskog et al., 2006; Oliveira et al., 2012; Gutierrez-Arenas et al., 2014). The M4R/G_i/o/AC5 signaling branch is a crucial addition in the current model. Figure 1 shows the major interactions involved in this signaling cascade. The parameter values for the AC5/cAMP/PKA signaling module were further improved on the basis of additional recent experimental evidences related to the cAMP and the PKA dynamics (Castro et al., 2013). Castro et al. (2013) have used AKAR3 to measure the PKA dynamics. Thus, we also included AKAR3 into this model to fit the experimental data.

In this model, the AC5 is assumed to be under a high basal inhibition due to a high activity of M4R resulting from the tonically active TANs. Previously published experiments show that the M4R activation at saturating concentration of agonist is sufficient to fully inhibit the D1R-induced AC5 activity in the striatal membrane preparation (Onali and Olianas, 2002; Jeon et al., 2010). This suggests a high concentration of M4R and associated G_i/o protein assembly. There are reports of nanomolar affinity (25–50 nm) of striatal M4R receptors for ACh (Quirion et al., 1989). The basal level of extracellular ACh in striatum is estimated to be ∼20–30 nm by microdialysis (Rada et al., 2010; Farrar et al., 2011), and it may even be higher as suggested by the large underestimation recently disclosed through electrochemical measurements in prefrontal cortex (Mattinson et al., 2011). Thus, the combination of high concentration of M4R and a basal concentration of ACh around its half-occupancy level suggest a high basal activity of these receptors, which would lead to a high basal inhibition of AC5.

The inputs to the D1+ MSN model were the DA and ACh concentrations. DA could either be represented as a low tonic basal level input or a transient increase in the DA concentration due to burst firing in DANs (e.g., in response to a salient reward-producing stimulus). The possible patterns for ACh release are either a tonic basal level or else a dip in ACh concentration assumed to result from the pause in TANs (e.g., concomitant with DAN bursting as seen in response to a salient stimulus) (Aosaki et al., 2010). The values for the basal level of ACh and DA used in the model are 100 and 10 nm, respectively (Heien et al., 2005). A value of 1500 nm is used for the peak amplitude of the DA transient, and ACh is assumed to be cleared out completely during a TAN pause unless stated otherwise.

D2+ MSN model.

The biochemical network modeled for the D2+ MSN also considers the AC5/cAMP/PKA signaling, but in this case the downstream signaling events are triggered by the D2R and A2aR. The signaling by D2R and A2aR also converges onto the AC5 where the D2R, via G_i/o, inhibits the A2aR/G_olf triggered signaling. The signaling downstream of D2R and A2aR is similar to the D1+ MSN case and is modeled by the same core model (Fig. 1). However, few of the parameter values have been modified to fit the previously published experimental results specific to D2+ MSNs neurons (Nishi et al., 1997, 2000; Bibb et al., 1999; Lindskog et al., 2002; Bateup et al., 2008). AC5 is considered to be under a high basal inhibition in this model, too, but here due to the activity of D2R. Traditionally, it has been thought that a considerable fraction of the D2R population in these neurons is present in the high-affinity state (Cumming, 2011). However, the reported percentage of the high-affinity fraction has a high variability because the receptor affinity state critically depends on the experimental conditions (Skinbjerg et al., 2012). However, in vivo experiments using PET provide indirect evidence that ∼20%-30% of the D2R population are occupied under the basal condition (Laruelle et al., 1997), which could be approximated as the fraction of high-affinity receptors because the basal DA level is reported to be in the nanomolar range (Abi-Dargham et al., 2000; Venton et al., 2003; Dreyer et al., 2010; Spanos et al., 2013). Moreover, several FRET measurements have demonstrated that the G_i/o protein has a higher tendency to precouple with its receptors, including D2R (Bünemann et al., 2003; Nobles et al., 2005). Such kind of precoupling increases the affinity between the receptor and its ligand (Hulme and Trevethick, 2010). In our D2+ MSN model, the fraction corresponding to the high-affinity D2R is assumed to be 30% of the total D2R population, which produces a good fit to experimental data as indicated in the following sections, and this number also falls into the range of the high affinity fraction estimated in vivo (Laruelle et al., 1997). The basal concentration of DA and Adn were set at 10 and 150 nm, respectively (Ballarín et al., 1991).

The inputs to the D2+ MSN model are DA and Adn. Two possible patterns for the DA input are used. First, DA input pattern is an increase in the extracellular DA concentration due to a burst firing in DANs (e.g., in response to an unexpected reward). The other input pattern is a dip in the extracellular DA concentration due to a depression in the DAN activity (e.g., in response to the omission of an expected reward) (Schultz, 1998). The duration of the DA dip is assumed to be of similar duration as the DA peak (Roitman et al., 2008; Hart et al., 2014). These DA inputs are combined with the Adn inputs. We have taken two different types of Adn inputs into consideration: the tonic level of basal Adn (Håkansson et al., 2006) concentration, 150 nm, and an activity-dependent elevation in the Adn concentration produced in response to a high neuronal activity (Sciotti et al., 1993). The activity-dependent Adn elevation reaches its maximum ∼10 s after the stimulus and then slowly returns to the basal level (Cechova and Venton, 2008; Pajski and Venton, 2010; Sims and Dale, 2014).

Molecular phenotypes for model constraining.

The models were constrained using quantitative molecular/subcellular phenotypes collected from previously published results (Table 1). In the context of this study, the word “phenotype” is used in a broader sense to refer a set of markers/observables measured in an experimental sample, such as striatal slices or heterologous expression system, in response to a specific stimulus, such as receptor agonist or ligand uncaging (Gutierrez-Arenas et al., 2014). We used three target phenotypes (refer to “Phenotype Name” in Table 1) specific for the D1+ MSNs. The DARPP-32 phenotype, “DAsliceD32D1,” is the same as that used in a previous version of the D1+ MSN model (Nishi et al., 1997, 2000; Bibb et al., 1999; Bateup et al., 2008; Gutierrez-Arenas et al., 2014). The new phenotypes include the time-series data for receptor-G-protein and PKA activity. The time-series data were parameterized, and these parameters are considered as the phenotypic variables for the time-series data. For example, the data from “ReceptorGprotein” phenotype, which corresponds to the activation of G-proteins, were parameterized in terms of time constants obtained from fitting a monoexponential (Gutierrez-Arenas et al., 2014). Similarly, the PKA activation dynamics (in terms of AKAR3 signal) upon addition of D1 agonist from “DAsliceAKAR3D1” was also parameterized in terms of time constant and amplitude obtained from fitting a monoexponential (Eq. 1) as follows:

The PKA dynamics produced by transient D1R agonist application and DA uncaging were parameterized using the difference of two decaying exponentials (Eq. 2), resulting in three quantities: amplitude of the total response, B_max, and two decay rates for each exponential, k₁ and k₂. This equation represents a quantity that has a finite source and a sink (Gutierrez-Arenas et al., 2014).

The data for “ReceptorGprotein” were inferred from the activation of G-protein-dependent processes in neurons. For example, the activation of D2R could be in between 10 ms in cholinergic interneurons (Chuhma et al., 2014) to a little >100 ms in D2+ MSN GIRK currents (Ford et al., 2009; Marcott et al., 2014); and similarly, PKA activation in D1+ MSN and hippocampal neurons is also significantly faster than other cellular systems (Nikolaev et al., 2004; Castro et al., 2013). These G-protein activation data are considerably faster than the activation reported for heterologous expression system (Bünemann et al., 2003; Hein et al., 2005, 2006; Lohse et al., 2008; Vilardaga, 2012). We assume that the G-protein activation time constant range is between 10 and 100 ms. The “ReceptorGprotein” phenotype is used for constraining both D1+ MSN and D2+ MSN model. Apart from “ReceptorGprotein,” the D2+ MSN model is also constrained by D2+ MSN specific data, listed in Table 1.

Table 1.

Quantitative target phenotypes used to constrain the D1+ MSN and D2+ MSN models

Neuron type	Phenotype name	Treatment	Marker	Phenotypic variable and values	References
D1+ MSN, D2+ MSN	ReceptorGprotein	Receptor + saturating concentration of ligand	G-protein activation (range of activation time constant = 10–100 ms; parameterized with Eq. 1)	Bmax = 100, c = 1.0, k = 10–100	Chuhma et al., 2014; Marcott et al., 2014
D1+ MSN	BasalD1	—	cAMP	30–100 nm	Bacskai et al., 1993; Mironov et al., 2009
			D32p34	0.4 μm	Nishi et al., 1997; Bateup et al., 2008
			D32p75	12 μm	Bibb et al., 1999; Bateup et al., 2008
	DAsliceD32D1	Striatal slices + DA ≥ 10 μm; sampled at 5 min	D32p34	11 × basal	Nishi et al., 2000; Bateup et al., 2008
			D32p75	0.5 × basal
	DAsliceAKAR3D1	Striatal slices + high D1R agonist (equivalent to 10 μm DA)	cAMP	2.5 μm	Castro et al., 2013
			AKAR3 phosphorylation (parameterized with Eq. 1)	Bmax = 100, c = 1.04, k = 0.118
		Striatal slices + high D1R agonist (equivalent to 10 μm DA) for 10 s	AKAR3 phosphorylation and dephosphylation (parameterized with Eq. 2)	Bmax = 94.73, k1 = 0.172, k2 = 2.317
		Striatal slices + transient DA uncaging (for 1.0 s)	AKAR3 phosphorylation and dephosphylation (parameterized with Eq. 2)	Bmax = 92.34, k1 = 0.736, k2 = 2.726
		Striatal slices + transient DA uncaging (for 0.3 s)	AKAR3 phosphorylation and dephosphylation (parameterized with Eq. 2)	Bmax = 95.23, k1 = 0.698, k2 = 3.006
		Striatal slices + transient DA uncaging (for 0.18 s)	AKAR3 phosphorylation and dephosphylation (parameterized with Eq. 2)	Bmax = 45.111, k1 = 0.731, k2 = 6.1
D2+ MSN	BasalD2	—	D32p34	0.7 μm	Nishi et al., 1997; Bateup et al., 2008
			D32p75	13 μm	Bibb et al., 1999; Bateup et al., 2008
	DAsliceD32D2	Striatal slices + DA ≥ 10 μm; sampled at 5 min	D32p34	0.4 × basal	Bateup et al., 2008
			D32p75	1.5 × basal
	D2AntagonistSliceD32	Striatal slices + D2R antagonist; sampled at 15 min	D32p34	2.0 × basal	Bateup et al., 2008
			D32p75	0.9 × basal
	A2aAntagonistSliceD32	Striatal slices + A2aR antagonist; sampled at 15 min	D32p75	2 × basal	Lindskog et al., 2002

The models also take into account that the G-proteins, both G_i/o and G_olf, are acted upon by GTPase-activating proteins (GAPs). GAPs, for instance, RGS (Regulator of G-protein signaling), increase the GTPase activity of heterotrimeric G-protein 20–40 times resulting in their fast deactivation (Doupnik et al., 1997). RGS9-2 is an RGS enriched in striatum and is known to produce faster deactivation of G_i/o proteins (Xie et al., 2012). AC5 also acts as a GAP for the G_s proteins increasing the G_s GTPase activity at least fivefold (Scholich et al., 1999); and by following the argument of homology between G_s and G_olf, we assume that it acts as a GAP for G_olf.

Computing the interaction between two input signals.

The strength of interaction between two different input signals, Signal 1 and Signal 2, affecting the same marker (e.g., active PKA) at each time step of a simulation is quantified with synergy(t) which is defined as follows:

where synergy(t) is the synergy between Signal 1 and Signal 2 at the level of PKA calculated at time t, PKA_stimulated (Signal 1, Signal 2, t) is the PKA activation level in the presence of both Signal 1 and Signal 2 at time t, PKA_stimulated (Signal 1,t) and PKA_stimulated (Signal 2,t) are the PKA activation level in response to either Signal 1 or Signal 2 at time t, and PKA_basal is the basal PKA level at t = 0. A similar synergy(t) calculation is done for another effector whenever required, like the ratio of PKA and PP1.

We also use mean synergy (μ[synergy]) as a single quantity to represent the overall synergy of interaction for a simulation. μ[synergy] is the mean of synergy values calculated every time-step (0.05 s) between two time points t₁ and t₂. We calculate the μ[synergy] for the first hundred seconds of the simulation. Thus, in our case, t₁ = 0 and t₂ = 100.

According to this phenomenological metric, the interaction is additive if μ[synergy] = 1, synergistic for μ[synergy] > 1, and antagonistic for μ[synergy] < 1.

Results

The constrained models reproduce a range of experimental data

In this study, we developed quantitative models for interaction between neuromodulatory signals and their effects on AC5/cAMP/PKA signaling for the two MSN types, D1+ MSN and D2+ MSN (Fig. 1). The D1+ MSN model is an updated version of previously published models (Lindskog et al., 2006; Oliveira et al., 2012; Gutierrez-Arenas et al., 2014). The current D1+ MSN model includes an additional signaling branch for M4R activation by ACh and the further downstream inhibition of AC5. Similar to the D1+ MSN model, the D2+ MSN model also features the AC5/cAMP/PKA signaling. However, the receptors involved in the D2+ MSN model are D2R and A2aR, which converge onto AC5. Both models have been constrained to data from the respective MSN type and reproduce various kinetic and steady-state experimental data listed in Table 1. The receptor and the G-protein reaction parameters in each of the models are constrained within a range of previously published observable activation kinetics in neurons. Both the models closely follow the experimentally observed G-protein activation range (Table 1, “ReceptorGprotein”) (Fig. 2A1,B1,A2,B2).

Figure 2.

Model fitting to experimental data mentioned in Table 1. A, G_i/o activation rate for (A1) M4R and (A2) D2R in the presence of saturating concentration of their respective ligands (phenotype “ReceptorGprotein”). Black curve represents the simulated activation. Red dotted curve represents a best fitted monoexponential with time constant in the range of neuronal G-protein activation mentioned in ReceptorGprotein. B, G_olf activation rate for (B1) D1R and (B2) A2aR in the presence of saturating concentration of their respective ligands (phenotype “ReceptorGprotein”). Black curve represents the simulated activation. Red dotted curve represents a best fitted monoexponential with time constant in the range of neuronal G-protein activation mentioned in ReceptorGprotein. C, Percentage of AKAR3 phosphorylation in response to 10 μm tonic DA (phenotype “DASliceAKAR3D1”). D, Percentage of AKAR3 phosphorylation in response to different duration of DA pulse (phenotype “DASliceAKAR3D1”). y-axis is normalized with respect to the maximum phosphorylation produced by 10 μm tonic DA. E, Changes in the DARPP32 Thr-34 and Thr-75 phosphorylation level measured 5′ after the application of 10 μm DA in the D1+ MSN and D2+ MSN (phenotype “DAsliceD32D1” and “DAsliceD32D2”). F, Experimental versus Simulated values for all the phenotypic variables. Blue lines indicate 20% divergence from experimental values. Red lines indicate 50% divergence from experimental values. Black lines indicate 80% divergence from experimental values. Patches represent the range for the phenotypic variables cAMP basal level and G-protein decay constants. The legends represent various phenotype names listed in Table 1. Each phenotype contains several points corresponding to different phenotypic variables. The “ReceptorGproteinD1” and “ReceptorGproteinD2” correspond to the “ReceptorGprotein” phenotype for the D1+ MSN and D2+ MSN model, respectively.

The models are furthermore constrained to match the basal and activated levels of a number of different signaling molecules. In the case of D1+ MSNs, the basal activity of the D1R is low. Thus, the basal level of cAMP is also expected to be low. There is no basal cAMP measurement for MSNs. However, the cAMP level has been estimated for other neuron types having low basal adenylyl cyclase activity (Bacskai et al., 1993; Mironov et al., 2009), and the current D1+ MSN model is tuned to reproduce a similar cAMP level. The steady-state level of cAMP following a high D1R specific agonist (SKF38393) has been previously reported to be >2.4 μm for MSNs using Epac1-camp biosensor (Castro et al., 2013), and this is captured by the D1+ MSN model presented here. The model also reproduces the previously reported fast PKA dynamics measured in striatal slices using the AKAR3 biosensor in response to sustained application of SKF38393 or a brief (1.0, 0.3, and 0.1–0.2 s) DA uncaging (Fig. 2C,D) (Castro et al., 2013). The basal and the stimulated levels of the phosphorylated forms of DARPP-32 have been used to constrain both the D1+ MSN and D2+ MSN models (Table 1). Figure 2E shows an example of how both models match the DARPP-32 phosphorylation changes after 5 min incubation of striatal slices with a D1R agonist in D1+ MSNs and a D2R agonist in D2+ MSNs. In general, the D1+ MSN and the D2+ MSN model versions have been constrained with respective neuron-specific data, and there is good quality of fit (i.e., <20% divergence) for most of the phenotypic variables for both models (Fig. 2F). However, there are few points that diverge 80%, which are mostly the dephosphorylation phase of the AKAR3 (Fig. 2D).

A DA peak together with an ACh dip in the D1+ MSNs synergistically activate PKA

We used the D1+ MSN model to study the potential integration of the transient DA peak with the concomitant ACh dip via the D1R and M4R. A transient increase in DA results in an increased activation of D1R in D1+ MSNs leading to an increased level of cAMP and PKA activation. Simulation experiments with the D1+ MSN model suggest that the level of PKA activation by a DA peak (1 s in our simulations) depends on the extracellular concentration of striatal ACh (Fig. 3A). It is known that the striatal ACh level could vary depending upon the activity of other brain regions (Rada et al., 2010). The DA peak produces low PKA activation if accompanied by high tonic ACh levels. This is due to the activation of M4R, which inhibits the cAMP/PKA pathway.

Figure 3.

PKA response in D1+ MSNs following ACh and DA transients. A, PKA activation produced by a 1 s DA 1.5 μm elevation as a function of varying basal level of tonic ACh; 100% ACh represents the assumed basal level, which is 100 nm. y-axis is normalized to steady-state level of active PKA produced by saturating D1R agonist in the presence of basal ACh. B, Active PKA levels in response to the different combinations of inputs. The DA and ACh signals have the durations of 1 s and 0.4 s, respectively. y-axis is normalized to steady-state level of active PKA produced by saturating D1R agonist in the presence of basal ACh. ↑, Peak; ↓, dip. C, Synergy at the level of PKA activation following a DA peak and ACh dip starting at the same time. Synergy is calculated at every time-step of the simulation using Equation 3. Mean synergy, μ[synergy], is 2.4. D, μ[synergy] (top) and maximum PKA level (bottom) as a function of the ACh dip amplitude. ACh dip amplitude is represented in percentage of maximum dip amplitude: 100% corresponds to full extracellular ACh depletion, and 0% means no dip. E, PKA increase for varying DA amplitude in the presence and absence of an ACh dip. Vertical line indicates the assumed basal concentration of DA, 10 nm. Amplitudes >10 nm represent a DA peak, and <10 nm represent a DA dip. Transient DA dips are not detected by D1+ MSNs. F, PKA fold increase (maximum/basal) as a function of DA peak amplitude and DA peak duration in the presence of (F1) a 0.4-s-long ACh dip and (F2) a 0.5-s-long ACh dip. Zero percentage in y-axis represents no peak, and 100% represents the 1500 nm amplitude of the peak. x-axis and y-axis labels are shared for both the plots.

Often a DA increase resulting from a salient stimulus is accompanied with pausing activity in TANs and thus an ACh dip. The ACh dip is known to be sensed by various receptors, such as nAChRs (Drenan et al., 2010). However, the capability of adenylyl cyclase-coupled M4R in sensing this ACh dip is not known. Using the D1+ MSN model, we therefore explored whether, and under which circumstances, the ACh dip could be sensed by M4R, and how the ACh signaling is integrated with the DA transients. As mentioned previously, a DA transient alone, in the presence of tonic ACh, is not able to produce high levels of PKA activation. However, the same DA transient is able to produce high PKA activation when accompanied with an ACh dip (Fig. 3B). Thus, the simulations suggest that D1+ MSNs are able to sense the ACh dip via M4R, and the TAN pause thus would gate the D1R-induced PKA activation. However, the ACh dip itself is not able to produce any significant PKA activation (Fig. 3B). Even though the ACh dip itself leads to the disinhibition of AC5, a further D1R-dependent AC5 activation is required to produce downstream cAMP/PKA because the basal AC5 activity in D1+ MSN is not high enough to produce significant cAMP/PKA activation by itself for such as small duration. Because the basal DA level is not enough to produce any significant D1R activation, a DA peak, therefore, with an ACh dip is required for significant PKA activation.

The synergy of interaction between a DA peak and an ACh dip for every time step computed according to Equation 3 is shown in Figure 3C. The mean synergy value, μ[synergy], for the interaction is 2.4 (see Materials and Methods). This highly synergistic response is the way in which a brief DA peak above a low basal level and a brief ACh dip from a high basal level could interact at the AC5 signaling axis for colocalized D1R and M4R. A high synergy for the interaction between a DA peak and an ACh dip implies that the system acts as an AND gate for brief DA “on” and ACh “off” signals. Thus, a dip in ACh gates the effect of DA transients in D1+ MSNs. As the synergy increases, the ACh gate thus becomes more important. One consequence is that the basal ACh tone produced by TAN activity has a potential to filter out a brief DA pulse that is not accompanied by an ACh dip. The synergy of interaction and the PKA activation produced by the concomitant DA peak and ACh dip critically depend on the amplitude of the ACh dip (Fig. 3D). As the amplitude of the ACh dip reduces, the synergy and the PKA level decrease.

The pause in TAN activity is at times preceded or followed by a burst (Morris et al., 2004). Even though the physiological role of the associated TAN burst is not clear, this phenomenon seems to be region-dependent (Chuhma et al., 2014). We also looked at how an ACh peak could affect the DA-dependent cAMP/PKA activation. The PKA activation produced by a DA peak is slightly reduced in the presence of an ACh peak compared with the DA peak alone, but the difference is small (Fig. 3B). The effect of an ACh peak is considerably low because AC5 is already under high inhibition even at the basal level of ACh and there is little room for further inhibition with increasing ACh. Thus, an ACh dip rather than an ACh peak is sensed via M4R in D1+ MSNs, and there are no significant differences in PKA response between the multiphasic ACh dip and a normal dip.

DA signals encode the error in the reward prediction. A higher RPE leads to a higher DAN activity (Fiorillo et al., 2003). The ACh tone keeps a strong inhibitory control of the PKA activation produced even at high DA levels and the ACh dip is important to produce high PKA changes (Fig. 3E). There is no effect on PKA level in D1+ MSNs in response to negative DA amplitudes (i.e., amplitudes less than the basal DA level of 10 nm) (Fig. 3E). Such negative DA amplitudes are thought to represent omission of an expected reward. Thus, the cAMP/PKA signaling axis in D1+ MSN is responsible for sensing only the positive RPE signals. However, we wondered whether there is any role of the peak duration in coding the magnitude of positive RPE. Thus, we looked at how the PKA activation is affected by the duration of the DA peak compared with the DA amplitude. The simulation results suggest that the PKA activation is mostly affected by the DA peak amplitude rather than the peak duration (Fig. 3F1). There is certainly a variation in the PKA activation produced by varying the peak duration, but it is significantly lower than the change produced by varying the peak amplitude. Thus, the amplitude of DA peak rather than its duration seems to be more relevant to encode the graded level of PKA activation. However, whether a DA peak of given amplitude produces a significant PKA activation depends on the ACh dip (e.g., some values of amplitude for the DA peak may not produce PKA activation in the presence of a shorter ACh dip, but the same peak could produce PKA when coupled to a slightly longer dip) (Fig. 3F1,F2). This is especially true for lower values of the amplitude. Thus, whether a D1-dependent PKA signaling occurs in response to a DA peak is flagged by the ACh dip, which goes along with the previously suggested idea that the ACh dip sets the temporal frame for DA peak response (Morris et al., 2004).

Synergy between a DA peak and an ACh dip depends on signal duration

The pause in the TANs in response to a reward signal mirrors the burst in the DANs (Aosaki et al., 1995; Morris et al., 2004). A similar relation may hold for the duration of the peak and the dip in extracellular DA and ACh, respectively. As there are not direct experimental measurements of the duration of the dip in ACh concentration, and the duration of the DA peak also varies according to the mismatch between the expected and the delivered reward, we explored the dependence of the μ[synergy] of PKA activation on the DA peak and the ACh dip duration. The results show that the gating potential of the ACh dip depends critically on its duration as a dip longer than 250 ms is necessary to generate significant synergy values (Fig. 4A). A DA peak in the presence of a very brief (<100 ms) ACh dip does not produce any significant synergy because such a dip does not lead to sufficient disinhibition of AC5 to gate the DA peak-induced AC5·G_olf complex formation. This threshold-like relation between synergy and dip duration has the potential to act as a filter for DA noise if it is not accompanied by an ACh dip longer than a certain duration. In addition to synergy, the actual PKA activation too follows such a threshold-like behavior (Fig. 4B).

Figure 4.

Temporal dependence of synergy in D1+ MSNs. A, μ[synergy] for different combinations of the duration of the DA peak and the ACh dip. In these calculations, the start of the peak and dip coincide with each other. B, Maximum activated PKA level for different combinations of the duration of the DA peak and the ACh dip. The PKA activation values are normalized to steady-state level of active PKA produced by saturating D1R agonist in the presence of basal ACh. C1, μ[synergy] as a function of delay between the ACh dip and the DA peak. The delay in the x-axis represents the time of onset of the ACh dip minus the time of DA peak (t_{Ach dip} − t_{DA peak}). A negative value means that the ACh dip is starting before the DA peak and vice-versa. C2, Maximum activated PKA level as a function of delay between the ACh dip and the DA peak. The activated PKA level is normalized to steady-state level of active PKA produced by saturating level of D1R agonist in the presence of basal ACh. D, Maximum activation level of PKA in the absence of an ACh dip and with varying DA signal duration. y-axis is normalized to steady-state level of active PKA produced by saturating level of D1R agonist (10 μm).

The D1+ MSN model also suggests that the delay between the DA peak and ACh dip affects the synergy produced (Fig. 4C1). If the two signals arrive simultaneously, then the synergy is high. However, subsecond delay between these signals still produces synergy, suggesting that slight variations in the timing of the arrival of DA peak and ACh dip may not impair PKA activation (Fig. 4C2).

Interestingly, the relative gating potential of the ACh dip decreases at longer DA peaks of 5–10 s, which could produce significant PKA activation even in the absence of an ACh dip (Fig. 4D). Recently, such long-duration DA elevations are shown to be physiologically relevant and are involved in providing a drive toward distant goals (Howe et al., 2013). In this work, however, we restrict the analysis to transients no longer than 1 s.

DA dip, and not DA peak, affects the PKA level in D2+ MSNs

Similar to the D1+ MSNs, the D2+ MSNs also detect extracellular DA changes. In this case, the response to DA is mediated via the D2R, which is coupled to G_i/o, inhibiting the cAMP production. However, the D2R-mediated PKA changes produced by transient DA signal remain largely unknown. Here, we used the D2+ MSN model to explore the PKA activation produced by DA transients in D2+ MSNs. In our simulations, we find that the effect of a DA peak on the PKA activation is very little in the D2+ MSNs (Fig. 5A). The fraction of high-affinity D2R in the model is ∼30% (see Materials and Methods), and the activity of basal level of DA on this fraction is sufficient to keep the basal PKA activation at a low level despite the tonic A2aR activity. Thus, a further increase in the D2R activation by a transient DA peak does not lead to a further decrease in the PKA level. Because the DA peak had no significant effect on the PKA level in D2+ MSNs, we further tested the possible effect of a DA dip on these neurons. DA dips are often associated with the omission of an expected reward (Schultz, 1998). The D2+ MSN model detects a DA dip (1 s in our simulations); and in response to this DA dip, there is a significant increase in the PKA activation (Fig. 5A). In the basal state of the D2+ MSN, the inhibition posed by the basal D2R activity keeps the PKA level low despite the basal A2a activity (Fig. 5B), and a DA dip removes the D2R inhibition, resulting in an increased PKA activation. This PKA activation trend produced by the D2+ MSN model (i.e., a DA dip is sensed by D2+ MSNs) is in accordance with some of the hypotheses in field (Frank, 2005; Dreyer et al., 2010).

Figure 5.

Response of D2+ MSNs following changes in DA and Adn stimulation. A, PKA activation in response to a DA peak (1500 nm) and a DA dip (0 nm) of 1 s duration. The y-axis is normalized to steady-state level of active PKA produced by saturating A2aR agonist in the presence of basal DA. ↑, Peak; ↓, dip. B, PKA activation produced by the basal Adn as a function of varying DA level. The y-axis is normalized to steady-state level of active PKA produced by saturating A2aR agonist in the presence of basal DA. The x-axis represents the basal level of DA as a factor of assumed basal level of 10 nm. C, PKA activation in response to a DA dip, Adn elevation, and DA dip in the presence of Adn elevation. The y-axis is normalized to steady-state level of active PKA produced by saturating A2aR agonist in the presence of basal DA. ↑, Increase; ↓, dip. D, The activation half-time for PKA response in response to DA dip, Adn elevation, and DA dip in the presence of Adn elevation. E, PKA fold change (maximum/basal) as a function of DA dip duration and DA dip amplitude. DA dip amplitude is represented in percentage DA dip amplitude: 100% corresponds to full DA depletion upon a dip, and 0% corresponds to no dip at all. F, Free G_i/o-βγ level in response to a DA peak and a DA dip (1 s). The y-axis is normalized to steady-state level produced by high A2aR agonist. ↑, Peak; ↓, dip.

It has previously been observed that dopaminergic transmission is often accompanied with Adn release in the striatum (Cechova and Venton, 2008). This Adn elevation has been attributed to the level of neuronal activity such that high activity leads to increased Adn elevation (Pajski and Venton, 2010). This increase in the Adn concentration is rather slow and reaches its maximum in ∼10 s and then slowly decays to the basal level (Cechova and Venton, 2008; Pajski and Venton, 2010). We looked at how this Adn change interacts with the DA dip and find that the DA dip-induced PKA activation in D2+ MSNs becomes more pronounced in the presence of the activity-dependent Adn increase (Fig. 5C). Thus, in these neurons, similar to the D1+ MSN, the change in the activity of the colocalized receptors potentiates the DA-dependent PKA response. However, unlike the D1+ MSNs where the interaction between the DA peak and ACh dip is synergistic, the interaction between the DA dip and the Adn increase in D2+ MSN is not strongly synergistic because DA dip could produce significant PKA activation alone (Fig. 5C). The Adn elevation alone could also produce PKA activation even though the level of PKA is much lower than the level produced by a DA dip. However, the PKA activation onset kinetics is significantly different between a DA dip response and an Adn elevation response (Fig. 5C,D). The PKA response produced by the Adn elevation alone is considerably slower than the response produced by the DA dip. The difference between response time is not due to any receptor-specific kinetics because the kinetics of all the receptors in this study is constrained by the same set of data (Table 1). Rather, this difference emerges at the level of AC5. In the case of “Adn elevation alone,” there is a higher competition between D2R-activated G_i/o and A2aR-activated G_olf, which slows down the rate of AC5 activation and thus PKA activation.

Whether the amplitude of a dip is sufficient enough to encode the negative RPE information is still not clear, and there exist observations either way (Bayer and Glimcher, 2005; Hart et al., 2014). There have also been suggestions that the magnitude information could be encoded with the duration of the dip (Bayer et al., 2007). Thus, we looked at how the PKA activation is affected by the dip duration and amplitude. The PKA activation in response to a DA dip depends on both amplitude and duration of the dip (Fig. 5E) unlike the activation produced by DA peak in D1+ MSNs where the peak amplitude is much more relevant than the duration. It required a 70%–100% reduction in the DA concentration to produce any significant PKA change (Fig. 5E). The dip amplitude has only a very small dynamic range (70%–100% of dip amplitude) to encode the whole range of PKA activation. In terms of actual DA concentration, assuming 10 nm as basal level, this would translate into a very small range (0–3 nm). In such a steep system, a concentration noise of even 1 nm could significantly affect the PKA activation, thus making the system less robust. On the other hand, the dip duration (for deeper dips) shows a more graded range to encode the whole PKA activation (e.g., PKA changes in response to different dip durations for a 100% dip amplitude in Figure 5E). Thus, it could be inferred that a role of dip duration in coding the magnitude information of negative RPE could make the downstream signaling, at least at the level of PKA activation, more robust compared with when it is coded only by DA amplitude.

The PKA activation produced by the DA dip in D2+ MSNs suggests that the cAMP/PKA signaling axis in this neuronal type mainly detects the negative RPE signals, associated with reward omission, depending upon basal state of the system. This is in contrast with the D1+ MSNs where the cAMP/PKA cascade seems to be responsible for sensing the positive RPE signals. cAMP/PKA signaling is not the only downstream effects of D2R activity. D2R activation modulates L-type calcium channels and GIRK channels via the G-βγ subunit (Hernandez-Lopez et al., 2000; Ford et al., 2009). In our model, because the occupancy of D2R is not saturated at basal DA level, a DA peak could lead to an increase in the G_i/o activation; thus, free G- βγ is produced (Fig. 5F). This increase in free G-βγ could lead to the activation of respective effector, like GIRK2. G-βγ has a low affinity to GIRK2 (Wang et al., 2014); thus, GIRK2 modulation might require a higher level of G-βγ, which is produced by a DA peak. This aligns with the known observations about D2-dependent modulation of GIRK in response to a DA peak (Marcott et al., 2014). The DA peak also increases the α subunit of G_i/o, but this increase does not affect the cAMP/PKA levels significantly because the cAMP/PKA signaling axis is already highly inhibited by the basal level of DA and a further increase in G_i/o-α has a little effect as previously mentioned in this section. Thus, the effects of DA transients on the D2+ MSNs are effector specific (i.e., D2R is able to sense both a DA peak and a dip, but whether a peak or dip produces a downstream effect could significantly depend on the effector).

DARPP-32 can significantly amplify the effect of neuromodulators on the PKA to PP1 activity ratio (PKA:PP1)

PP1 is an important phosphatase in MSNs, which dephosphorylates several PKA targets, such as GluR1. Thus, the PKA-to-PP1 ratio, $\left(\mathrm{PKA}:\mathrm{PP}1=\frac{\text{Level\hspace{0.17em}of\hspace{0.17em}Active\hspace{0.17em}PKA}}{\text{Level\hspace{0.17em}of\hspace{0.17em}Active\hspace{0.17em}PP}1}\right)$, is an important variable affecting the phosphorylation level of downstream signaling proteins, which are substrates to both PKA and PP1. Thus, we looked at the changes in PKA:PP1 in response to the DA transients in D1+ MSNs and D2+ MSNs. In both D1+ and D2+ MSNs, the PKA:PP1 follows the same trend as the PKA activation. The PKA:PP1 increase produced by a concomitant DA peak and an ACh dip in D1+ MSNs is significantly higher than the PKA:PP1 produced by the DA peak or ACh dip alone (Fig. 6A1). Similarly, in the D2+ MSNs, there is an increase in PKA:PP1 in response to a DA dip; and in the presence of an Adn elevation, the effect produced by the DA dip becomes even more pronounced. Like PKA activation, PKA:PP1 increases in response to DA transients, but the amplitude of PKA:PP1 change is significantly higher than PKA (Figs. 3B, 5C, 6A1,A2). The higher level of PKA:PP1 compared with the PKA activation is produced by DARPP-32 via the PKA-dependent inhibition of PP1 and via the phosphorylation of the Thr-34 residue. Simulations of the D1+ MSN and D2+ MSN models with the mutation of Thr-34, by switching off the phosphorylation of this residue, indeed confirm this. The Thr-34 mutant version of the D1+ MSN and D2+ MSN models produces PKA:PP1 similar to the PKA activation level (Fig. 6B1,B2,C1,C2). Thus, the upstream cAMP/PKA signals in both MSNs are amplified at the PKA:PP1 level via DARPP-32 Thr-34 phosphorylation. In D1+ MSNs, the synergy between the DA peak and ACh dip is also enhanced at the level of PKA:PP1 compared with the synergy at the level of PKA (Fig. 6D shows synergy(t) calculated for every time step using Eq. 3), suggesting that the downstream effect of the DA peak and ACh dip integration stays on for a longer duration. This illustrates the importance of DARPP-32 in translating a transient neuromodulator signal into a longer-lasting effector response. As mentioned above, the DA amplitude correlates to the error in the reward prediction. In the case of D1+ MSN, the PKA:PP1 increased with an increase in DA amplitude. At higher amplitudes (>1000 nm), the PKA:PP1 change is significantly higher than the PKA change. The difference between PKA:PP1 and PKA is abolished in the Thr-34 mutant, thus indicating that DARPP-32 is critical in the process of amplifying the positive RPE signal encoded by DA amplitude further downstream (Fig. 6E1). In contrast, the PKA:PP1 in D2+ MSN increased in response to a dip in DA level (Fig. 6E2). The basal DA amplitude for the model is 10 nm, and the DA amplitudes below this value in Figure 6E2 represent a decrease in the DA level, which is often associated with omission of an expected reward.

Figure 6.

Amplification of signal downstream by DARPP-32. A, PKA:PP1 ratio in response to various input combinations in (A1) D1+ MSN and (A2) D2+ MSN models. B, PKA:PP1 ratio in response to various input combinations for T34A mutant in (B1) D1+ MSN and (B2) D2+ MSN models. C, Comparison of the change in PKA:PP1 ratio and PKA in response to various input for the wild-type (WT) and T34A mutant in (C1) D1+ MSN and (C2) D2+ MSN. D, Comparison of the synergy(t) at PKA:PP1 level and the synergy(t) produced at PKA level in D1+ MSN. E, PKA and PKA:PP1 in both wild-type (WT) and D32T34A (T34A) as a function of varying dopamine amplitude. E1, D1+ MSN model. DA amplitude in x-axis represents the amplitude of a DA peak. E2, D2+ MSN model. The basal level of DA is 10 nm, and the amplitude in x-axis represents the decreased level of DA for a dip.

Discussion

PKA activation in MSNs is crucial for long-term plasticity events in corticostriatal synapses (Shen et al., 2008a; Wickens, 2009; Lerner and Kreitzer, 2012). In this study, we investigated, using kinetic ordinary differential equation modeling, the interaction between the G_olf- and G_i/o-coupled GPCRs (M4R and D1R in D1+ MSNs and D2R and A2aR in D2+ MSNs) on the AC5/cAMP/PKA signaling for detecting brief reward-related neuromodulator transients and how these interactions could shape the downstream PKA signaling.

AND gate in D1+ MSNs: PKA activation upon convergent ACh dip AND DA peak

Mice knock-out of M4R in D1+ MSNs shows a hyperlocomotion phenotype characteristic of sustained D1R stimulation (Jeon et al., 2010). M2/M4 antagonists were among the earliest anti-parkinsonian medication (Olianas and Onali, 1996) possibly because they activate the direct pathway, and M4 allosteric activators are also under study for the treatment of schizophrenia (Leach et al., 2010) as they shift the balance of activity toward the indirect pathway, just as D2R antagonists, such as haloperidol. However, despite these facts and high expression levels of M4R in D1+ MSNs, very little is known about its role in processing striatal ACh reward-related transients, such as an ACh dip, which co-occurs with a peak in DA. In this study, we advance the prediction that M4R could sense this reward-related ACh dip through a synergistic interaction with the concomitant DA peak acting through D1R, resulting in a significant PKA activation not achievable with any of the transients independently. This coincidence detection capability may seem redundant if the ACh dip and DA peak are tightly correlated (Graybiel et al., 1994; Threlfell et al., 2012); however, the picture is more diverse. In addition to dopaminergic control, the interruption of the autonomous tonic firing of TANs by brief pauses and bursts is also controlled by cortical and thalamic innervations (Doig et al., 2014), GABAergic innervation from VTA (Brown et al., 2012), and possibly other circuits (Lim et al., 2014). Furthermore, in striatal slices, the induction of a pause in TANs by a DAN burst is highly region-dependent (Chuhma et al., 2014). In the face of these various relations between ACh and DA transients, the synergistic interaction between M4R and D1R on the AC5/cAMP/PKA could be a signaling module specialized in detecting a subset of them. In addition, the coincidence detection capability predicted here would act as a filter of very brief local changes in DA and ACh concentrations arising either from random fluctuations (Faisal et al., 2008) in the mechanisms regulating these transients or from the pace-making activity of these neurons (Grace and Bunney, 1984; Zhang et al., 2009).

A requirement for this synergistic interaction to occur is the high and low occupancies in basal conditions of M4R and D1R, respectively. If the basal activation of M4R decreases due to local and stable changes in the level of ACh, this would open up the possibility for an isolated DA peak to produce PKA even without an ACh dip and thus reducing the potential for synergy. However, there seem to be adaptations in M4R density to counter the changes in ACh level (Bernard et al., 1999), which could be interpreted as the basal inhibition is maintained even on the face of long-term changes in the ACh level. This could mean that the M4R-dependent basal inhibition, and thus synergy, is less variable and might be physiologically important. The second important requirement for synergy is the colocalization of these receptors. The M4R and D1R are both found in the soma and distal dendritic spines (Hersch et al., 1994, 1995); albeit little is known about the colocalization of these receptors with regard to nanoscale resolution.

ACh transients could still be directly detected by D1+ MSNs by other mechanisms (e.g., isolated ACh pauses), such as the one arising on conditioning stimulus presentation in overtrained animals (Aosaki et al., 1994), and could be processed by M1R; and indeed, pauses in M1R activation have been suggested to promote LTD in both MSN types (Wang et al., 2006).

Gate opening to tonic Adn: PKA activation upon DA dip on D2+ MSNs

Unlike D1+ MSNs, the results from the D2+ MSN model suggest that the DA peak has no significant effect on the active PKA levels of D2+ MSNs. However, a DA dip, which represents an omission of expected reward, produced a significant increase in the D2+ MSN PKA level. Systemic administration of the antipsychotic haloperidol, a D2R antagonist, activates PKA in D2+ MSNs of mice (Bateup et al., 2008). The PKA activation in response to the DA dip is due to the disinhibition of tonic A2aR activity in the D2+ MSNs, which, under the basal condition, is effectively opposed by a basal D2R activation. However, this may not mean that D2R sites are saturated under the basal condition, as previously thought (Dreyer et al., 2010), rather a small fraction of the high affinity D2R sufficiently inhibit cAMP production. The PKA triggered by a DA dip might facilitate LTP induction, which aligns with the effect of A2aR agonists and D2R antagonist on D2+ MSNs (Shen et al., 2008b; Lerner and Kreitzer, 2012). A hypothesis about such an effect of a DA dip already exists (Frank, 2005; Cox et al., 2015), and our results support them with subcellular details and highlight the importance of receptor and neuromodulator interaction in the process. The PKA response resulting from a DA dip is even more pronounced in the presence of striatal Adn elevation. There are several reasons for the Adn elevation, but the physiological importance of such elevation is currently not clear (Cechova and Venton, 2008; Pajski and Venton, 2010; Sims and Dale, 2014). A hypothesis could be that, in a series of repeated trials, if the adenosine level increases with each passing trial due to increased striatal activity, then this could make the system more prone to extinction. This, of course, assumes that the intertrial duration is less than the Adn decay duration and the extinction process is related to the cAMP/PKA signaling in D2+ MSN.

Even though the cAMP/PKA signaling in D2+ MSNs is largely insensitive to a DA peak, this does not necessarily mean that a peak does not hold any information for this neuron type (e.g., a DA peak significantly increases the level of G_i/o-βγ in these neurons) (Fig. 5F), and such an increase might have physiological roles considering the Gβγ-mediated regulation of ion channels (Hernandez-Lopez et al., 2000). Such a Gβγ-dependent GIRK activation in response to a peak is observed for D2+ MSNs (Ford et al., 2009; Marcott et al., 2014). Thus, D2+ MSNs could sense both dips and peaks of DA at the receptor level, but the downstream effects of these signals are effector specific.

The G_olf/AC5/G_i/o core, a common signaling motif for opposing functions

The cAMP/PKA signaling is highly inhibited by the basal activity of G_i/o in both types of MSNs even though the neuromodulators responsible for this basal activity are different. Such a basal inhibition appears due to a combination of tonic neuromodulator level. The parameters, which are most critical for the current models, are related to G_i/o-coupled GPCR activation–deactivation cycle. These are responsible for maintaining the basal level of AC5 inhibition. To a lesser extent, the model results could also be affected by the G_olf-coupled GPCR activation kinetic parameters and the AC5-related parameters. These parameters affect the efficacy of receptor activation on AC5. Apart from the kinetic parameters, the total amounts of chemical species, which control the tonic inhibition of AC5, are also important. A transient dip in the levels of the neuromodulator coupled to G_i/o operates as a highly dynamic gate for the G_olf-dependent activation of AC5/cAMP/PKA signaling in these neurons. Fast inactivation of the G_i/o protein is a critical requirement for the transient dip to act as an effective gate-opener, which suggests an important role of GAP proteins, which are known to be expressed in MSNs (Gold et al., 1997; Xie et al., 2012).

Even though the configuration of the G-protein system is similar between the two neuronal types (i.e., high basal activity of G_i/o and a gating of G_olf signaling by a dip), they serve opposite function in the context of reward learning. In D1+ MSN, the PKA activation is produced by a DA peak; whereas in D2+ MSNs, a DA dip affects PKA (Figure 7), which aligns with the notion that positive RPE signals lead to the potentiation of corticostriatal synapses onto D1+ MSNs, whereas negative signals rather potentiate D2+ MSN (Frank, 2005). It is thought that the DA amplitude is sufficient to encode both positive and negative RPE (Hart et al., 2014). However, the PKA activation suggests that amplitude might be sufficient in the case of positive RPE; whereas for negative RPE, the duration might also play an important role (Figs. 3F1, 5E) (Bayer et al., 2007). Outside the context of reinforcement learning, the basal inhibition of AC by G_i/o and gating of G_olf (G_s) signaling by a decrease in G_i/o could be viewed as a general mechanism to filter out spurious G_olf (G_s) signals and should be further explored and extended for cell types other than MSNs.

Figure 7.

Overview of the interaction between ACh dip and DA peak in D1+ MSN and DA dip and Adn in D2+ MSN.