Inactive and active state structures template selective tools for the human 5-HT_5A receptor

Abstract

Serotonin receptors are important targets for established therapeutics and drug development as they are expressed throughout the human body and play key roles in cell signaling. There are 12 serotonergic G protein-coupled receptor members in the human genome, of which the 5-HT_5AR is the least understood and lacks selective tool compounds. Here, we report four high-resolution (2.73-2.80 Å) structures of human 5-HT_5A receptors, including an inactive state bound to an antagonist AS2674723 by crystallization, and active states bound to a partial agonist lisuride, and two full agonists 5-carboxamidotryptamine (5-CT) and methylergometrine by cryo-EM. Leveraging the new structures, we developed a highly selective and potent antagonist for 5-HT_5AR. Collectively, these findings both enhance our understanding of this enigmatic receptor and provide a roadmap for structure-based drug discovery at 5-HT_5AR.

Editor summary:

This comprehensive study of the most enigmatic serotonin receptor 5-HT5AR includes lots of pharmacological investigations, inactive and active state structures with antagonist, partial agonist, and full agonists. Also, a highly potent and selective antagonist was developed.

Introduction

Serotonin (5-hydroxytryptamine; 5-HT) is a neurotransmitter essential for the regulation of mood, perception, feeding, anxiety, gastrointestinal and cardiac functions as well as the actions of many drugs useful in treating neuropsychiatric and other disorders^1,2. To mediate these actions, a large family of 5-HT receptors has evolved including 12 G protein-coupled receptors (GPCRs) divided into 6 major families (viz. 5-HT1, 5-HT2, 5-HT4, 5-HT5, 5-HT6, and 5-HT7)^1,2. Of these, the human 5-HT_5AR is the most enigmatic and least understood. Based on gene knock-out and expression analysis as well as studies with relatively non-selective antagonists, the 5-HT_5A receptor has been implicated in the actions of the psychedelic drug lysergic acid diethylamide (LSD)³, the modulation of pain⁴, antidepressant drug actions⁵ and anxiety¹. These effects are consistent with the wide distribution of 5-HT_5A receptors in the human brain, with the highest expression in cortex and limbic areas⁶. Although there are potential tool compounds available to interrogate the 5-HT_5A function, the most widely used compounds SB-699551, ASP5736, and AS2674723 are non-selective and interact with various 5-HT and related biogenic amine receptors^7-9, making the pharmacological exploration of the role of the 5-HT_5A receptor problematic. More recently, an arrestin-biased partial agonist UCSF678 at the 5-HT_5AR was developed based on the homolog model of 5-HT_5AR, which displayed a more restricted, albeit still considerable off-target profile¹⁰.

The 5-HT_5A receptor also has an unusual pharmacological profile with the highest affinity for ergots and ergolines and the synthetic agonist 5-carboxamidotryptamine (5-CT)^11,12 and low affinity for most other agonists and antagonists of 5-HT receptors. In addition, the signaling pathways downstream of the 5-HT_5A receptor are not fully elucidated, although coupling to a pertussis toxin-sensitive G protein has been demonstrated¹³. Finally, the structural basis for 5-HT_5AR’s unique pharmacology is unknown as given its relatively low sequence identity with other 5-HT receptors it is likely that the ligand recognition site is distinct (Extended Data Fig. 1a). Given this lack of fundamental knowledge regarding the structure, function, and pharmacology of the 5-HT_5A receptor as well as its clear importance for brain function, we determined the structures of the inactive state and active G protein-coupled state by x-ray crystallography and single-particle cryo-EM, respectively. We further used the inactive state structure to template the discovery of the first potent and selective 5-HT_5AR antagonist. Taken together, these studies will accelerate the search for selective 5-HT_5AR agonists and antagonists as potential therapeutics for a number of neuropsychiatric disorders.

Results

Structures of antagonist and agonists bound-5-HT_5AR complexes

We initially obtained an inactive state structure of the 5-HT_5A receptor in complex with the antagonist AS2674723. To achieve this three modifications of 5-HT_5A were applied: (1) the N-terminus 22 residues was truncated; (2) protein PGS (the catalytic domain of Pyrococcus abyssi glycogen synthase)¹⁴ was fused to the third intracellular loop (ICL3), and (3) two thermostabilizing mutations (D86^2x50N and I299^6x49A, superscript is generic residue number in GPCRdb¹⁵) were also introduced into the construct based on both extensive computational predictions and experimental validation¹⁶ (Supplementary Fig. 1). Eventually, the structure of the 5-HT_5AR-AS2674723 complex was solved using molecular replacement (MR) and refined to 2.8 Å resolution (Table 1). There is one 5-HT_5AR-AS2674723 complex in the asymmetric unit, in which the ligand and most residues of the receptor, except intracellular loop 1 (ICL1) and helix 8 (H8), are well defined (Fig. 1a).

Table 1.

Data collection and refinement statistics

	5-HT5AR-AS2674723a (PDB: 7UM4)
Data Collection
Space group	P21212
Cell dimensions
a, b, c (Å)	135.3, 42.5, 94.8
α, β, γ (°)	90, 90, 90
Resolution (Å)	50.0 – 2.8 (2.90-2.80)b
Rmerge (%)	30.7 (102.9)
I/σ(I)	5.8 (1.1)
CC1/2 (%)	95.4 (55.4)
Completeness (%)	99.2 (97.2)
Redundancy	6.6 (3.6)
Refinement
Resolution (Å)	50.0 - 2.8 (2.91-2.81)
No. reflections	13,696 (1,108)
Rwork / Rfree (%)	22.9 / 27.1
No. atoms	3,633
Protein	3,539
Lipids	60
Water	8
B-factors (Å2)	67.6
Protein	67.6
AS2674723	62.5
Lipids	71.4
Water	56.5
R.m.s. deviations
Bond lengths (Å)	0.006
Bond angles (°)	0.78

^aThe number of crystals is 17.

^bValues in parentheses are for the highest-resolution shell.

Fig. 1: Structures of the 5-HT_5AR complexes.

a, Crystal structure of the inactive state 5-HT_5AR in complex with an antagonist AS2674723. b, Superposition of three cyroEM structures of 5-HT_5AR-miniGo complexes in complex with 5-CT, lisuride, and methylergometrine, respectively. The ligands in complexes are shown in spheres within cartoon diagrams of receptors and G proteins. Next to the structural models of 5-HT_5AR complexes, ligands are shown in stick models surrounded by electron density for antagonist AS2674723 or EM maps for agonists 5-CT, Lisuride, and Methylergometrine, respectively. Density maps are shown as gray meshes. c, BRET2 Gi1-activation assay of 5-HT_5AR stimulated by the agonists 5-HT, 5-CT, lisuride, and methylergometrine. Data has been normalized to % of 5-HT. See Supplementary Table 1 for fitted parameter values that represent mean ± SEM of N = 4 biological replicates.

To get insight into the activation mechanism of 5-HT_5AR, we also attempted to determine the active state structure of 5-HT_5AR. Firstly, by using the TRUPATH biosensor platform¹⁷, we found that 5-HT_5AR effectively couples to every member of the G_i/o family (G_i1-i3 and G_oA-B) with the exception of Gz and Gust. We also found that 5-HT_5AR can activate the G₁₅, a subunit in the G_q family when these are co-expressed in HEK-293T cells (Extended Data Fig. 1b). As the activation potencies between G_i1-i3 and G_oA-B subfamilies for 5-HT_5AR are comparable and the miniGo has been used in the structural determination of active state 5-HT_1BR¹⁸ successfully, thus, the miniGo was also applied for the active state structural studies of 5-HT_5AR. Additionally, the mutation H146^34x50P was introduced into the 5-HT_5A receptor construct to improve the stability of the complex (see discussion below). The 5-HT_5AR-miniGo complexes were assembled by co-expression of the 5-HT_5A receptor with miniGo, Gβ₁, and Gγ₂ subunits via the NanoBit tethering system¹⁹. Eventually, the structures of 5-HT_5AR-miniGo bound to 5-CT, lisuride, and methylergometrine were determined at a resolution of 2.73 Å, 2,79 Å, and 2.75 Å, respectively (Table 2 and Extended Data Fig. 2). The overall structures of these three agonists-bound 5-HT_5AR complexes are similar to each other with root-mean-square deviation (RMSD) values lower than 0.3 Å for the whole complex comparisons between each other (Fig. 1b and Extended Data Fig. 3a-c).

Table 2.

Cryo-EM data collection, refinement, and validation statistics.

Structures	5-HT5AR-miniGo 5-CT (EMDB-26597) (PDB: 7UM5)	5-HT5AR-miniGo Lisuride (EMDB-26598) (PDB: 7UM6)	5-HT5AR-miniGo Methylergometrine (EMDB-26599) (PDB: 7UM7)
Data collection and processing
Magnification	45,000	45,000	45,000
Voltage (kV)	200	200	200
Electron exposure (e–/Å2)	29.07	43.8	47.4
Defocus Mean ± SD (μm)1	1.2 (0.3)	1.1 (0.3)	1.2 (0.4)
Defocus range (μm)1	0.3-2.3	0.1-2.2	0.1-2.9
Pixel size (Å)	0.91	0.91	0.91
Symmetry imposed	C1	C1	C1
Initial particle images (no.)	947,828	562,323	1,580,364
Final particle images (no.)	304,291	226,599	634,138
Map resolution (Å)	2.73	2.79	2.75
FSC threshold	0.143	0.143	0.143
Map resolution range (Å)	2.4-5.6	2.4-5.3	2.4-5.1
Refinement
Initial model used (PDB code)	6G79	7UM5	7UM5
Model resolution (Å)2	2.94	2.93	2.96
FSC threshold	0.5	0.5	0.5
Model resolution range (Å)	2.4-5.6	2.4-5.3	2.4-5.1
Map sharpening B factor (Å2)	85.5	82.5	91.6
Model composition
Non-hydrogen atoms	8,276	8,178	8,167
Protein residues	1,105	1,094	1,089
Ligands	1	1	1
B factors (Å2)
Protein	48.83	52.27	49.56
Ligand	40.77	52.22	46.42
Water	43.77	-	-
R.m.s. deviations
Bond length (Å)	0.002	0.003	0.003
Bond angle (°)	0.475	0.515	0.516
Validation
MolProbity score	1.12	1.14	1.29
Clash score	2.10	1.82	2.44
Poor rotamer (%)	0.00	0.00	0.00
Ramachandran Plot
Favored (%)	97.24	96.74	96.16
Allowed (%)	2.76	3.26	3.84
Disallowed (%)	0.00	0.00	0.00

¹underfocus positive

²Resolution estimates from cryoSPARC auto-corrected GSFSC

AS2674723 recognition at the 5-HT_5A receptor

Overall, the recognition of AS2674723 at the 5-HT_5A receptor is primarily based on its three chemical moieties: a tetrahydroisoquinoline ring, a positively charged guanidine group, and a trifluorophenyl ring, which occupies the orthosteric binding pocket (OBP), extended binding pocket 1 (EBP1), and extended binding pocket 2 (EBP2), respectively (Fig. 2a and Extended Data Fig. 4a). The tetrahydroisoquinoline ring mainly interacts with F301^6x51 and F302^6x52 by π–π interactions, and V122^3x33 and C125^3x36 through hydrophobic contacts (Fig. 2a). And the guanidine group forms a salt bridge with E101^2x64, which is accompanied by a rotation and shift of the extracellular end of TM2 in comparison with the active state 5-HT_5AR structures (Figs. 2a, 5d). Besides E101^2x64, D121^3x32, Q193^ECL2, and Y328^7x42 also have hydrophilic interactions with this guanidine group. Additionally, hydrophobic interactions between the side chain of L324^7x38 and the guanidine group are also observed (Fig. 2a). Lastly, the trifluorophenyl ring forms extensive contacts with surrounding residues of EBP2, including V194^ECL2, R196^ECL2, Y200^5x39, S204^5x43, T205^5x44, and E305^6x55 (Fig. 2a and Extended Data Fig. 4a).

Fig. 2: Ligand-specific interactions with 5-HT_5AR.

a-d, Specific residues in the binding pockets that interact with AS2674723 (a), 5-CT (b), Lisuride (c), and Methylergometrine (d), respectively. The salt bridge and hydrogen bond interactions are shown as black dashed lines. e, Heatmap of the mutagenesis studies showing the effects of orthosteric-site residues on Gi1-activation assay. Boxes marked with “X” indicate “Not Detected” as no appreciable signal was detected. See Supplementary Table 2 for fitted parameter values that represent mean ± SEM of N= 3 biological replicates. The surface expression levels of the WT and related mutant 5-HT_5ARs have been determined by the ELISA assay. The mutants W117A, W298A, and Y328A display significantly low surface expression levels (less than 50% of WT). see Extended Data Fig. 10 for details. f, Mutagenesis study of residue S204 of 5-HT_5AR stimulated by either 5-HT or 5-CT. Data has been normalized to % of WT 5-HT_5AR. See Supplementary Table 3 for fitted parameter values that represent mean ± SEM of at least 3 biological replicates.

Fig. 5: Activation of 5-HT_5A receptor.

Structural comparison of inactive state 5-HT_5AR structure, active state 5-HT_5AR structure, and the prediction 5-HT_5AR structure from AlphaFold2 server. a, Side view. The ligand-binding pocket is indicated by a yellow shaded oval. b, Extracellular view. c, Intracellular view. The displacement direction of TM5, TM6, and TM7 are indicated by red arrows. d, Ligand binding pocket. The rotation of E101 from the active state to the inactive state is indicated by a red arrow. e, CWxP motif. f, PIF motif. The A299 and I299 are from inactive and active state structures at position 6x49, respectively. g, Sodium binding site. The N86 and D86 are from inactive and active state structures at position 2x50, respectively. The hydrogen bonds are indicated by black dashed lines. h, DRY and NPxxY motifs. The conformational changes from the inactive state to the active state are indicated as black arrows.

To gain insight into the selectivity of AS2674723, we determined its binding affinities across the aminergic receptors and carried out mutagenesis studies on the residues interacting with the AS2674723. These screening results showed that 5-HT_5AR (0.6 nM) and 5-HT₇R (1.3 nM) were the primary targets of AS2674723. Additionally, 5-HT_1AR (33 nM), 5-HT_1BR (15 nM), 5-HT_1DR (50 nM), α_2AAR (25 nM), and α_2CAR (52 nM) also showed high affinities for AS2674723 (Extended Data Fig. 5a). In the alanine mutagenesis study: (1) D121^3x32A, F301^6x51A, and Y328^7x42A lost the binding ability to the radioligand ³H-LSD and thus no binding affinities were determined for AS2674723; (2) V122^3x33A and L324^7x38A severely reduced the binding affinities by 99-fold and 66-fold, respectively; (3) E101^2x64A, C125^3x36A, Y200^5x39A, T205^5x44A, and A208^5x461G moderately affected the binding affinities by above 5-fold; and, (4) the others (Q193^ECL2A, V194^ECL2A, R196^ECL2A, S204^5x43A, F302^6x52A, and E305^6x55A) had modest effects on binding affinities (Extended Data Fig. 5b-c).

Interestingly, combining the above mutagenesis data with sequence analysis, we found that L324^7x38 was a less conserved residue across the serotonin receptor family with the exception of 5-HT₄R and 5-HT₇R, and L324^7x38A dramatically affected the binding affinity of AS2674723, which might suggest a crucial role for its selectivity (Extended Data Figs. 5b and 6a). The L324^7x38N (5-HT_1AR) and L324^7x38T (5-HT_1BR, 5-HT_1DR, and 5-HT_1ER) mutants displayed similar effects as L324^7x38A, both of which reduced the binding affinities of AS2674723 by more than 50-fold (Extended Data Figs. 5b and 6a). Additionally, E101^2x64 which forms a specific salt bridge with the guanidine group of AS2674723 is another non-conserved residue, suggesting a potential role in the selectivity of this ligand. Three mutations (E101^2x64Q, E101^2x64T, and E101^2x64I) mimicking other members of the 5-HT receptor family displayed similar effects as E101^2x64A reducing the binding affinity of AS2674723 from 7.4- to 13.2-fold (Extended Data Figs. 5b and 6a). Importantly, 5-HT₇R, the primary off-target of AS2674723, has an aspartic acid instead of glutamic acid at position 2x64, which could potentially form similar electrostatic interactions with the guanidine group of AS2674723 (Extended Data Fig. 6a). This was also consistent with our mutagenesis data, as E101^2x64D did not alter the AS2674723 binding affinity (Extended Data Fig. 5b). Taken together, these data indicated that the interactions between E101^2x64 and AS2674723 are unique and potentially useful for selective drug development against 5-HT_5AR. Lastly, mutations C125^3x36S and S204^5x43G, both of which were mutated to the feature residues of the 5-HT2 subfamily, also showed moderate effects on the binding affinities (Extended Data Figs. 5b and 6a). Thus, these differences further strengthened the selectivity of AS2674723 to 5-HT_5AR over the 5-HT2 receptors. In conclusion, these data together revealed the molecule mechanism of the selectivity of AS2674723 against 5-HT_5AR and provided the basis for further selective ligands development, which is exploited below.

Mode of agonist recognition at the 5-HT_5A receptor

In the 5-CT-bound 5-HT_5AR structure, a featured water molecule interacting with the amino group of the 5-CT and D121^3x32 was revealed by clear density (Fig. 1b). However, the position was not matched to the water molecules in a prior apo 5-HT_1AR structure with the closest one being around 1.6 Å away (Extended Data Fig. 4c). Overall, 5-CT adopts a similar binding pose in the orthosteric pocket as 5-HT does in the structure of the 5-HT_1AR-5-HT complex²⁰. Notably, the amide group of 5-CT extends to the EBP2 and forms hydrogen bonds with S204^5x43 and E305^6x55. In a previous report, E6x55 was thought to block the binding of 5-CT to the pocket of 5-HT_1ER or 5-HT_1FR via its relatively larger sidechain²⁰. In our 5-HT_5AR-5-CT structure, we found that the density map for the carboxyl group of E305^6x55 was not evident, suggesting that the interaction between the amide group with E305^6x55 was unstable (Extended Data Fig. 4d). Our mutagenesis data also showed that mutation E305^6x55A failed to show a large effect on the activation of Gi1 signaling (Fig. 2e). Together, these data indicate that E305^6x55 does not play an important role in 5-HT_5A receptor activation by 5-CT.

Compared to E305^6x55, residue S204^5x43 might play a key role in the 5-CT selectivity, as it has a specific hydrogen bond with the amide group of 5-CT in our 5-HT_5AR-5-CT structure but no observed interactions with 5-HT in the 5-HT_1AR-5-HT structure²⁰. Moreover, sequence alignment showed that it was a serine conserved in 5-HT1Rs, 5-HT_5AR, and 5-HT₇R at position 5x43, whereas a glycine in the 5-HT2Rs, cysteine in the 5-HT₄R, and alanine in 5-HT₆R. As all of these three residues (Gly, Cys, and Ala) could not form a hydrogen bond with the amide group of 5-CT, this may lead to lower binding affinities of 5-CT in 5-HT2, 5-HT4, and 5-HT6 subfamilies (Extended Data Figs. 4i and 6a). To validate the role of S204^5x43 in the 5-CT selectivity, three related mutants (S204^5x43A, S204^5x43G, and S204^5x43C) were generated and tested in Gi1-activation assay by 5-HT and 5-CT simultaneously. The result showed that mutations S204^5x43A and S204^5x43G showed modest effects on both the potency and efficacy of the Gi1 activation by 5-HT, but dramatically reduced the potency of the Gi1 activation by 5-CT (Fig. 2f). And the mutation S204^5x43C severely affected the Gi1 activation by both 5-HT and 5-CT, which was also consistent with the relatively low affinity of 5-HT to 5-HT₄R²¹ (Fig. 2f and Extended Data Fig. 4i). Our structural and functional data together suggest a crucial role of S204^5x43 in the selectivity of 5-CT in the 5-HT family.

Lisuride has been reported to have a high affinity for 5-HT_5AR²² and we find it is a partial agonist for Gi signaling activating (Fig. 1c). Overall, lisuride adopts a largely similar pose to the full agonist methylergometrine, another ergoline scaffold compound, both of which occupy the orthosteric pocket by the ergoline ring and EBP1 by their substitutions (Fig. 2c-d and Extended Data Fig. 4a). The key feature of the binding mode of lisuride is the extensive and strong interactions with D121^3x32, which leads to several unique properties of lisuride for Gi1 activation (Fig. 2c, e). For example, residue F301^6x51 is a strickly conserved residue across the serotonin family and is essential for ligand binding and receptor activation²³ (Extended Data Fig. 6a). Our mutagenesis data showed that the mutation F301^6x51A did abolish the Gi1 activation abilities of 5-HT, 5-CT, and methylergometrine, but only slightly affected the Gi1 activation by lisuride, which indicated that the activation by lisuride was not highly dependent on the π–π interaction between F301^6x51 and the ergoline ring (Fig. 2e). In other words, these weaker interactions between lisuride and TM6 might also result in the weaker activation ability of lisuride for 5-HT_5AR. Another difference is that, compared with 5-CT and methylergometrine, the indole ring of lisuride is shifted upward and leading to a weaker hydrogen bond with T126^3x37, a crucial residue in activating the 5-HT_5AR (Fig. 2c-e and Extended Data Fig. 4a, e-g). Taken together, compared with the full agonists, the weaker interactions between lisuride and TM3 and TM6 might lead to its partial agonism.

Inactive-active state 5-HT_5A receptor transitions

Structural alignment of the inactive 5-HT_5AR structure with 5-HT_1BR-ergotamine (RMSD, 1.01 Å for 172 Cα) and 5-HT_2AR-methiothepin (RMSD, 1.72 Å for 168 Cα) structures reveals that significant structural differences are evident in the TM3, TM4, and TM5 regions (Fig. 3a-b). Specifically, in comparison with 5-HT_1BR, the TM3 of 5-HT_5AR shifts to the TM5 side by 1.3 Å (Cα of T144^3x55), and the TM4 of 5-HT_5AR shifts downward by 0.8 Å (Cα of V175^4x51) and bends to the TM3 side around 22° (Cα of K/C^4x40 to the Cα of L172^4x48) at the intracellular side (Fig. 3a). In the TM5 region, notably, the intracellular end of TM5 of 5-HT_5AR shows a kinked structure (around 24° rotation compared with 5-HT_1BR, Cα of K/R^5x66 to the Cα of W221^5x59) and forms close interactions with the intracellular end of TM6 (Fig. 3b). In particular, this hydrophobic interface is composed of three alanines (A226^5x64, A227^5x65, and A284^6x34, namely 3A cluster) (Fig. 3b). Meanwhile, sequence alignment analysis shows that this 3A cluster is nonconserved in the 5-HT receptor family, with only the 5-HT_1ER, 5-HT_1FR, and 5-HT₇R having the same pattern (Extended Data Fig. 6b). However, none of these three receptors have published inactive state structures. This kinked TM5 may block the outward displacement of TM6 upon transducer coupling and thus increase the activation energy barrier (Figs. 3b, 5c).

Fig. 3: Structural comparison between inactive state 5-HT_5AR and 5-HT_1BR and 5-HT_2AR.

a-b, Superpositions of the 5-HT_5AR-AS2674723 structure with 5-HT_1BR-ergotamine (4IAR) and 5-HT_2AR-methiothepin (6WH4) structures. The bending of TM4 (a) and kinked structure of TM5 (b) of inactive state 5-HT_5AR at the intracellular side. The inset shows the 3A cluster (A226, A227, and A284; shown in stick model and colored in magenta) on the interface between the TM5 and TM6. c-d, BRET2 Gi1-activation assay of WT and mutant 5-HT_5ARs stimulated by 5-HT (c) and lisuride (d). See Supplementary Table 4 for fitted parameter values that represent mean ± SEM of N = 3 biological replicates.

To test this hypothesis, we mutated the alanine to a bigger-sidechain residue and determined if the Gi activation signaling is altered. Combined with the sequence and structural analysis, we made three single mutants (A226^5x64V, A227^5x65L, and A284^6X34R) and one double mutant (A226^5x64V and A227^5x65L), and analyzed their Gi1 signaling activities (Extended Data Figs. 3d-e, 6b). The results showed that A227^5x65L increased the efficacy of 5-HT stimulated Gi1 activation by 20%, while A226^5x64V, A284^6X34R, and the double mutation greatly improved the potency by more than 30-fold (Fig. 3c). More interestingly, all the mutations improved the efficacy of lisuride by 1.36- to 1.88-fold, making it a full agonist (Fig. 3d). Furthermore, we tested this double mutant in the Gi1 signaling activation and β-arrestin2 recruitment assays with multiple ligands. The results showed that the efficacies could be dramatically improved to the full agonist level for the partial agonists in the Gi1 activation assay and all the efficacies were significantly improved by at least 3-fold in the β-arrestin2 recruitment assay (Extended Data Fig. 7). Thus, this double mutation significantly potentiated G protein and β-arrestin signaling pathways and would be useful in the drug screening for 5-HT_5AR in the future. These structural and functional data together might suggest a self-inhibitory role of the TM5 kink in the 5-HT_5AR signaling activation.

The 5-HT_5AR-miniGo complex interface

The overall binding mode of G protein in 5-HT_5AR is similar to other 5-HT1 subfamily active-state structures with modest differences observed in the α5 helix of miniGo and the H8 of the receptor. Specifically, the α5 helix of the miniGo in the 5-CT-bound 5-HT_5AR complex is outward shifted by around 0.9 Å (Cα of H^5.11, to 5-HT_1AR-Gi complex) and 1.9 Å (to the other 5-HT1Rs complexes) (Fig. 4a and Extended Data Fig. 3f). Additionally, H8 of 5-HT_5AR is displaced outward by 3.5 Å (Cα of F341^7x56) from the 5-HT_1BR structure (Cα of S372^7x56) (Fig. 4a). Of note, a similar bending of TM4 is also observed in the comparison of active state 5-HT _5AR structures with 5-HT1Rs structures, as does in the comparison of the inactive state structures (Fig. 4b). We found that the R34x57 in ICL2 forms a salt bridge with D3x49 of the DRY motif in all 5-HT1R-G protein complexes while it is a threonine instead of arginine at position 34x57 in 5-HT_5AR (Extended Data Fig. 3g). However, the adjacent R154^4x38 was observed to engage in an equivalent salt bridge with D3x49 (Extended Data Fig. 3g-h). Thus, this unique sequence composition of ICL2 of 5-HT_5AR causes a distinct TM4 bending. Moreover, this salt bridge interaction between R154^4x38 and D138^3x49 was also essential for the agonist-activated Gi signaling for 5-HT_5AR (Fig. 4e).

Fig. 4: 5-HT_5AR-miniGo interface.

a-b, Superposition of the 5-CT-bound 5-HT_5AR structure with the active state 5-HT1 structures. (a) Structural comparison of the G protein binding mode between 5-HT_5AR and 5-HT1 receptors. (b) The TM4 bending was also observed in the active 5-HT_5AR structure. c, Detailed interactions between the 5-HT_5AR and the α5 helix of miniGo. Salt bridge interactions and hydrogen bonds are depicted as black dashed lines. d, The interface between the ICL2 of 5-HT_5AR and miniGo protein. H/P146^34x50 represents a histidine or proline in the AS2674723-bound 5-HT_5AR structure or 5-CT-bound 5-HT_5AR structure, respectively. Inactive and active 5-HT_5ARs are shown as ribbon models and the miniGo shows the electrostatic surface. e, BRET2 validation of the interface residues of 5-HT_5AR. See Supplementary Table 5 for fitted parameter values that represent the mean ± SEM of at least 3 biological replicates. f-g, Mutagenesis analysis of ICL2 residues H146 and M147 of 5-HT_5AR by BRET1 recruitment assay (f) and BRET2 dissociation assay (g). Definitions of curves to the right of each graph indicate mutations, for example, HM_AM indicates a single mutation of H146A while HM_AA indicates a double mutation of H146A and M147A. See Supplementary Table 6 for fitted parameter values that represent the mean ± SEM of at least 3 biological replicates. The surface expression levels of the WT and related mutant 5-HT_5ARs have been determined by the ELISA assay. Of these mutants, the R154^4x38 and V287^6x37 are significantly lower than the WT 5-HT_5A receptor (less than 50%). see Extended Data Fig. 10 for details.

The major interface between the 5-HT_5AR and miniGo could be divided into two regions, one is dominantly mediated by the α5 helix of miniGo and the other by the ICL2 of the 5-HT_5A receptor (Fig. 4c-d). In the α5 helix region, there are several hydrophilic interactions observed, including a salt bridge interaction between R230^5x68 and D212^H5.13, and hydrogen bonds between S142^3x53 (main chain) and N342^8x47 of receptor and N218^H5.19 and G221^H5.22 of G protein, respectively (Fig. 4c). Additionally, L219^H5.20 and L224^H5.25 of miniGo bind into a hydrophobic cavity formed by residues I143^3x54, I223^5x61, A227^5x65, M286^6x36, and V287^6x37 of 5-HT_5AR (Fig. 4c). Surprisingly, unlike the strong π-cation interaction between R^6x29 and Y^H5.26 in the 5-HT1Rs-G protein structures, it is a lysine instead of the arginine in 5-HT_5AR at position 6x29 and does not form interactions with Y225^H5.26. Consistent with this, both the K279^6x29 and Y225^H5.26 show poor density maps in the 5-HT_5AR-miniGo structures (Extended Data Fig. 3i). On the ICL2 side, residue M147^34x51 inserts into the hydrophobic pocket formed by the α5 helix and αN helix junction of miniGo (Fig. 4d). In the following mutagenesis study, the S142^3x53A and M147^34x51A showed no effect or slight effect on the Gi1 activation; the R230^5x68A and M286^6x36A moderately affected the signaling; the I143^3x54A, I223^5x61A, R282^6x32A, and N342^8x47A dramatically reduced Gi1 activation; and the R139^3x50, R154^4x38, and V287^6x37 abolished the activation of Gi1 (Fig. 4e). Moreover, we found that all the mutations of arginines in the cavity affected the Gi1 activation, which indicated that a positively charged environment was important for the Gi1 signaling activity (Extended Data Fig. 3k).

Intriguingly, although M147^34x51 of ICL2 makes extensive hydrophobic interactions with the miniGo protein, the M147^34x51A does not alter Gi1 signaling activation (Fig. 4c-d). Therefore, the sequence alignment of ICL2s of 5-HT receptors was performed and we found that the residue H146^34x50, adjacent to M147^34x51, made the 5-HT_5AR unique from the other 5-HT receptors, as it was an alanine or proline in other members at position 34x50 (Extended Data Fig. 6b). Accordingly, to explore the roles of H146^34x50 on the stability of the 5-HT_5AR-Gi complex, we carried out the miniGi recruitment assay²⁴. Surprisingly, the wild type 5-HT_5AR was unable to efficiently recruit miniGi under the stimulation of full agonist 5-CT, whereas both mutations H146^34x50A and H146^34x50P could dramatically improve the ability of miniGi recruitment (Fig. 4f). Therefore, the mutation H146^34x50P was also introduced to the construct for the active state structure determination as mentioned above. Structural comparison of the inactive and active state 5-HT_5AR structures showed that H146^34x50, a positively charged aromatic residue, could not which might induce the instability of the complex (Fig. 4d). Based on the mutant H146^34x50A or H146^34x50P, an extra point mutation M147^34x51L, M147^34x51V, or M147^34x51A was introduced to look into the role of M147^34x51 in the G protein complex assembling and signaling of 5-HT_5AR. Compared with two single mutations H146^34x50A and H146^34x50P, the P^34x50L^34x51 (short for the double mutation H146^34x50P and M147^34x51L, similar as below) slightly enhanced the recruitment, A^34x50V^34x51 and P^34x50V^34x51 dramatically reduced the recruitment, and A^34x50A^34x51 and P^34x50A^34x51 could not recruit miniGi as the WT did (Fig. 4f). In contrast, the results of the same mutants in the Gi1 activation assay didn’t show significant differences between the WT and mutant 5-HT_5ARs in both potency and efficacy (Fig. 4g). Taken these data together, we found that the M147^34x51 was critical for the stability of the receptor-G protein complex but didn’t play an important role in the activation of Gi protein signaling.

Structural features of 5-HT_5A receptor activation

As we determined the structures of both inactive and active state 5-HT_5ARs, a structural comparison was performed to investigate the molecular mechanism for 5-HT_5AR activation. Consistent with the previously reported GPCR-G protein complexes^23,25,26, the intracellular end of TM6 undergoes the largest displacement and is tilted outward by 14.0 Å (Cα of E277^6x27), while TM7 shifts inward by 3.2 Å (Cα of Y338^7x53) during the activation of 5-HT_5AR coupling the G protein (Fig. 5a-c). As TM5 forms a kink at the intracellular end in the inactive state, the movement direction is perpendicular to the TM6’s and the distance is 7.2 Å (Cα of K228^5x66) (Fig. 5c).

Accompanied by the helical movements, the microswitches also undergo extensive conformational changes. The W298^6x48 of the CWxP motif rotates downward and pushes against residue F294^6x44 of the PIF motif causing it to move out by 2 Å (Cα) and the plane of its phenyl ring to rotate by 60° ²⁷ (Fig. 5e-f). In the sodium binding site, as a result of the rotamer state changing of S128^3x39 and S331^7x46, the hydrogen bond between N86^2x50 and S331^7x46 is broken and reformed between D86^2x50 and S128^3x39, thus leading to the downstream helical movements^28-30 (Fig. 5g). In agreement with our prediction, the D86^2x50N mutation stabilized the inactive state in low-to-medium salt conditions used for crystallization²⁹. Moreover, the sidechains of R139^3x50 of the DRY motif, Y338^7x53 of the NPxxY motif, and a conserved tyrosine Y338^7x53 are also rearranged upon the activation of the 5-HT_5A receptor (Fig. 5h).

We also explored the 5-HT_5AR structure model from the alphafold2 server³¹ (referred to as AF2 structure) and compared it with our inactive and active state structures. In comparison with the inactive structure, TM6 of the AF2 structure displaces outward and TM5 also doesn't form a kink (Fig. 5a-c). However, the outward movement of TM6 in the AF2 structure is less than in the active structure, which indicates that the AF2 structure may represent an intermediate state of the 5-HT_5A receptor (Fig. 5c). In the orthosteric binding pocket of the AF2 structure, the E305^6x55 rotates inside, V194^ECL2 shifts downward, and F301^6x51 and F302^6x52 move inward to the pocket, which results in a relatively shrunken pocket (Fig. 5d). Among the microswitches, the alphafold2 structure largely adopts the active conformation excepting the R139^3x50 of the DRY motif and S331^7x46 in the sodium pocket are in the inactive conformation (Fig. 5e-h). As the 5-HT_5AR structure modeled by the alphafold2 server is ligand-free, this model might represent the apo-state 5-HT_5AR, thus, together with our inactive and active state structures, could provide a more comprehensive understanding of the activation of 5-HT_5AR.

Structure-guided design of a selective 5-HT_5A receptor antagonist

As described above, compound AS2674723 has six off-target receptors, of which 5-HT₇R is the main one, across the aminergic receptors (Extended Data Fig. 5a). To guide the selectivity optimization, we aligned the AS2674723 binding involved residues of 5-HT_5AR with corresponding residues of 5-HT₇R and mapped the unconserved residues on the 5-HT_5AR-AS2674723 structure (Fig. 6a and Extended Data Fig. 6a). As expected, the pocket residues of 5-HT_5AR and 5-HT₇R are highly similar with an overall similarity of 83% over 18 residues. In particular, two critical residues E101^2x64 and L324^7x38 which are vital for the selectivity of AS2674723 to 5-HT_5AR are highly conserved in 5-HT₇R (Extended Data Fig. 6a). Of the residues that do differ, all are located at the upper pocket (EBP2), thus, modifying the substitutions on the phenyl ring might increase the selectivity over 5-HT₇R (Fig. 6a-b).

Fig. 6: Selective antagonist development of 5-HT_5A receptor.

a, Distribution of the different residues interacting with AS2674723 of 5-HT_5AR from 5-HT₇R on the 5-HT_5AR-AS2674723 structure. The different residues are shown as the stick model and colored in magenta. The residues of 5-HT_5AR and 5-HT₇R are labeled in black and red, respectively. b, Chemical structure of the compound AS2674723. The trifluorophenyl ring binding in the EBP2, tetrahydroisoquinoline ring in the orthosteric pocket, and fluorine atom in the lower pocket are colored in blue, red, and pink, respectively. c-e, Chemical structures of the first (c), second (d), and third (e) rounds of SAR-designed compounds. The variations in different moieties from AS2674723 are colored in the corresponding color used in (b). And the best candidate compound in each round is highlighted in a shaded yellow box. f, The binding affinities of the SAR-designed compounds at 5-HT_5AR and 5-HT_7AR. The compounds from the first, second, and third rounds of SAR optimization are colored in yellow, red, and blue, respectively. The starting compound AS2674723 and featured compound MS112 are shown as black and red asterisks, respectively. All the chemical structures of the ligands are illustrated by ChemDraw (20.0). See Supplementary Tables 7-10 for fitted parameter values that represent mean ± SEM of N = 3 biological replicates.

In the first round of testing, we selected four high-affinity compounds to 5-HT_5AR from the patent literature³², which varied the fluorine substitutions on the upper phenyl ring and introduced a cyclopropyl ring at the tetrahydroisoquinoline ring (Fig. 6c). We also selected compound ASP5736, an AS2674723 analog with an isoquinoline core, for evaluation, due to a better selectivity over other 5-HT receptors⁸ (Fig. 6c). Our results indicated that four cyclopropyl-substituted tetrahydroisoquinoline ring derivates (C1, C2, C3, and C4) have improved affinities for 5-HT_5AR, as well as other off-target receptors, resulting in overall decreased selectivity compared to AS2674723 (Fig. 6f and Supplementary Table 7). Moreover, our results did suggest a better selectivity of ASP5736 over 5-HT₇R (from 2.6-fold to 4.8-fold) and other receptors, albeit with a weaker binding affinity to 5-HT_5AR (10.7 nM) versus AS2674723 (1.5 nM) (Fig. 6f and Supplementary Table 7). Accordingly, we selected sixteen analogs of ASP5736 from patent WO2011016504³³ to explore the effect of upper pyridinyl moiety and the isoquinoline core region on the selectivity (Extended Data Fig. 8a). To bias toward the high affinities to the 5-HT_5AR firstly, we used the structure-based free energy perturbation (FEP)³⁴ via the program FEP+³⁵ to evaluate the affinity changing of the designed compounds from the starting compound. Based on the FEP predictions, the top three scoring analogs were synthesized and tested (Fig. 6d and Extended Data Fig. 8a). Of these three analogs, the compound C7 displayed increased selectivity over 5-HT₇R (from 4.8-fold to 21.7-fold) and other off-target receptors compared to ASP5736 (Fig. 6f and Supplementary Table 8). The chemical structure of C7 features a methyl-substituted isoquinoline core with an upper trifluorophenyl ring (Fig. 6d).

To further improve the selectivity of C7, we explored directed substitutions on the upper phenyl ring and more substituent groups on the isoquinoline core. In total, we selected nine compounds from the patent WO2011016504³³ and designed fourteen novel compounds (Extended Data Fig. 8b). Based on the FEP analysis, ten compounds were selected for synthesis and biological evaluation. Of these, six compounds demonstrated good selectivity to 5-HT₇R (> 20-fold; Fig. 6e-f and Supplementary Table 9). Notably, the new compound MS112, introducing a cyano group on the upper phenyl moiety and keeping the methyl group on the isoquinoline ring, maintained a high affinity for 5-HT_5AR (5.0 nM) with improved selectivity over 5-HT₇R (82.6-fold) and other tested aminergic receptors (more than 73.5-fold; 5-HT_1AR, 5-HT_1BR, 5-HT_1DR, α_2AAR, and α_2CAR) (Fig. 6f and Supplementary Table 10). And the MS112 still displays the antagonist activity as the same as compounds ASP5736 and AS2674723 (Extended Data Fig. 9a-b).

We sought to evaluate the FEP predictions quantitatively, as the method may be broadly useful for ligand discovery against GPCRs. Accordingly, we analyzed the correlations between the ΔΔG values predicted by the FEP (ΔΔG_FEP) and those calculated from the experimental result (ΔΔG_exp). For three compounds (C5, C6, and C7) in the second round of optimization, their ΔΔG_FEP and ΔΔG_exp values are significantly correlated with differences lower than 5-fold (Extended Data Fig. 9c). In the last round of optimization, according to the FEP predicted ΔΔG values, ten compounds could be classified into three classes: better (ΔΔG_FEP < −0.5 kcal/mol; C16), similar (0 < ΔΔG_FEP < 0.5 kcal/mol; C8, C9, C10, MS112, C12, C13, and C15), and worse (ΔΔG_FEP > 2.0 kcal/mol; C11 and C14) potencies versus the lead compound C7 (Extended Data Fig. 8b). On experimental testing, compound C16 was found to display a quite similar potency to compound C7 with a ΔΔG_exp value of 0.01 kcal/mol. (Extended Data Fig. 9d). In the similar potencies class, six compounds display ΔΔG_exp values within 0.5 kcal/mol, with exception of the C13 with a ΔΔG_exp value of 1.1 kcal/mol. Finally, the ΔΔG_exp values of C11 and C14 are 0.7 kcal/mol and 0.3 kcal/mol, which are again off by 4- and 8-fold from their ΔΔG_FEP values, respectively (Extended Data Fig. 9d). Thus, quantitatively, the correlation between the predicted and measured affinity changes was modest. However, by less stringent qualitative criteria, the predictions were more accurate: molecules predicted to have improved affinity did maintain the affinity, those predicted to be relatively unchanged typically did only change modestly, while molecules predicted to bind worse did bind worse. Whereas the field continues to strive for chemical accuracy (+/− 1 kcal/mol RMSE in predicted versus measured energies), even qualitative correlation between prediction and experiment is more than helpful in ligand optimization. This study supports the use of this technique in structure-activity relationship campaigns for optimization of affinity and selectivity for GPCR ligands.

Conclusion

In this paper, we determined the structure of the inactive state 5-HT_5AR stabilized by the antagonist AS2674723 and three agonist-activated structures coupled to its canonical transducer Go in complex with a partial agonist lisuride and two full agonists 5-CT and methylergometrine. These structures provided molecular insights into ligand binding modes of 5-HT_5AR and illuminated the selectivities of AS2674723 and 5-CT. Additionally, several unique structural features of 5-HT_5AR from other 5-HT receptors were revealed, including a bending TM4 and an unexpected kinked TM5 at the intracellular side. Together with the mutagenesis and functional studies, key determinants essential for agonist actions and receptor-Go coupling were also identified. Finally, based on the high-resolution structure and combined with FEP and SAR optimization, we obtained a new and highly selective 5-HT_5AR ligand with nanomolar affinity, which will be a powerful probe to investigate this understudied receptor. In conclusion, these observations have wide implications for the mechanistic understanding of 5-HT_5AR signaling and drug discovery.

Methods

Generation of constructs of 5-HT_5AR for crystallization and cryoEM

For crystallization, the human HTR5A gene (UniProt ID: P47898) was cloned into a modified pFastBac1 vector (Invitrogen) containing a hemagglutinin (HA) signal peptide followed by FLAG-tag, His₁₀-tag, and a TEV protease site at the N terminus. The construct was further optimized by fusing the protein PGS (UniProt ID: Q9V2J8) to the third intracellular loop (ICL3) at V231 and E277 of the HTR5A gene, truncation of N-terminal residues 1-22, and introducing two thermostabilizing mutations (D86^2x50N and I229^6x49A).

For cryoEM, to facilitate protein expression and subsequent purification, thermostabilized apocytochrome b562RIL (BRIL) and HRV3C protease site were introduced at the N-terminal and a LgBit fragment at the C-terminal of the receptor, respectively. For the miniGo heterotrimeric complex, we replaced the miniGsqiN with our miniGo in the pFastBac dual expression system which has been used in the structural determination of the 5-HT_2AR–Gq complex²³. A HiBit fragment was fused to the C-terminus of the beta subunit to enhance complex formation with the receptor.

Expression for 5-HT_5AR-XTAL and -cryoEM constructs

The Bac-to-Bac baculovirus expression system (Invitrogen) was used to generate the recombinant baculovirus for protein expression. Before infection, viral titers were determined by flow-cytometric analysis of cells stained with gp64-PE antibody (Expression systems, 97-201, diluted 1:200 in PBS). For the inactive state 5-HT_5AR, expression of 5-HT_5AR was carried out by infection of sf9 cells (Expression systems, 94-001S) at a cell density of 2 x 10⁶ cells/ml in ESF921 medium (Expression systems, 96-001) with P1 virus at a multiplicity of infection (MOI) of 3. Cells were harvested by centrifugation at 48 h −80°C until use. For the 5-HT_5AR-MiniGo complex, baculoviruses of 5-HT_5AR, miniGo-Gβ1γ2, and scFv16 were co-expressed by infecting Sf9 cells at a density of 2x10⁶ cells per ml with P1 baculovirus at MOI ratio of 3:1:1, respectively. Cells were harvested by centrifugation 48 h post-infection and stored at −80°C for future use.

Inactive state 5-HT_5AR purification

The purification of the inactive state 5-HT_5AR is similar to the purifaction of inactive state 5-HT_2AR receptor²³. Thawed insect cell membranes were disrupted in a hypotonic buffer containing 10 mM HEPES (pH7.5), 10 mM MgCl₂, and 20 mM KCl and protease inhibitors containing 500 μM AEBSF, 1 μM E-64, 1 μM Leupeptin, and 0.15 μM Aprotinin. Subsequently, soluble and membrane-associated proteins were removed in a high osmotic buffer containing 10 mM HEPES (pH7.5), 1 M NaCl, 10 mM MgCl₂, and 20 mM KCl. Purified membranes were incubated in the presence of 10 μM AS2674723 and protease inhibitor cocktail at 4°C for 2 h. The membranes were incubated with 2.0 mg/ml iodoacetamide (Sigma, I1149) for 30 min and were solubilized in the buffer containing 50 mM HEPES (pH 7.5), 1% (w/v) n-dodecyl-beta-D-maltopyranoside (DDM, Anatrace, D310), 0.2% (w/v) cholesterol hemisuccinate (CHS, Sigma, C6512) and 150 mM NaCl, at 4°C for 2h. The solubilized 5-HT_5AR were isolated by ultra-centrifugation at 185,000 g at 4°C for 50 min, and then incubated at 4°C overnight with TALON IMAC resin (Clontech, 635507), 800 mM NaCl and 20 mM imidazole as the final buffer concentration. The resin was washed with 10 column volumes of washing buffer I containing 50 mM HEPES (pH 7.5), 0.1% (w/v) DDM, 0.02% (w/v) CHS, 800 mM NaCl,10% (v/v) glycerol, 20 mM imidazole and 1 μM AS2674723 and 10 column volumes of washing buffer II 50 mM HEPES (pH 7.5), 0.05% (w/v) DDM, 0.01% (w/v) CHS, 500 mM NaCl, 10% (v/v) glycerol and 1 μM AS2674723 without imidazole. The protein was eluted using 3 column volumes of the elution buffer containing 50 mM HEPES (pH 7.5), 0.05% (w/v) DDM, 0.01% (w/v) CHS, 500 mM NaCl, 10% (v/v) glycerol, 250 mM imidazole, and 10 μM AS2674723 and concentrated in a Vivaspin 20 concentrator with a molecular weight cutoff of 100 kDa (Sartorius Stedim, VS2042) to 500 μL and was applied to PD MiniTrap G-25 columns (GE Healthcare, 28-9180-07) to remove imidazole. The fusion and 10x His tag were removed by the addition of His-tagged PreScission protease (GeneScript, Z0392-500) and incubation overnight at 4°C. The protease, cleaved His tag, and uncleaved protein were trapped by equilibrated TALON IMAC resin and collected the flow-through. The 5-HT_5AR-AS2674723 complex was concentrated to around 30 mg/ml using a Vivaspin 500 centrifuge concentrator with a molecular weight cutoff of 100 kDa (Sartorius Stedim, VS0142). Protein purity and monodispersity were tested by analytical size-exclusion chromatography column, SRT-300 (Sepax scientific), and Ultimate 3000 UHPLC systems (Thermo Scientific).

Lipidic cubic phase crystallization

The purified 5-HT_5AR protein in complex with AS2674723 was screened for crystallization in lipidic cubic phase (LCP) with mixed molten lipid (90% (w/v) monoolein and 10% (w/v) cholesterol) at a protein: lipid ratio of 1:1.5 (v/v) using a mechanical syringe mixer^23,36. Crystallization was done on 96-well glass sandwich plates (Marienfeld, 0890003) in 50 nL LCP drops that were dispersed from a 10 μL gas-tight pipette (Hamilton) using a handheld dispenser (Art Robbins Instruments) and overlaid with 1 μL of precipitant solution. After optimization, crystals were obtained in several conditions: 0.1 M Sodium chloride, 0.1 M Lithium sulfate, 0.1 M DL-Malic acid pH 5.9, 30 % v/v PEG 400; 0.2 M Sodium chloride, 0.4 M Lithium sulfate, 0.1 M Sodium citrate pH 5.0, 30 % v/v PEG MME 500; and 0.4 M Potassium phosphate monobasic, 0.1 M Tris pH 8.0, 30 % v/v PEG 400. Crystals grew to a maximum size of ~30 mm X 20 mm X 20 mm within 2 weeks and were harvested directly from the LCP plates using MiTeGen micromounts and stored in liquid nitrogen.

X-ray data collection, structure determination and refinement

Data collection was performed using the JbluIce-EPICS data acquisition software at the GM/CA-CAT (23-ID-D) beamline at the Advanced Photon Source of the Argonne National Laboratory using a 10–20 μm mini-beam at a wavelength of 1.0332 Å and a Pilatus 3 6M detector. Crystals within the loops were located by diffraction using the automated rastering module of JbluIce-EPICS. Partial data sets (wedges of 5-10 degrees) were collected from crystals exposed to the non-attenuated mini-beam for 0.2 – 0.3 sec and 0.2 – 0.3 degree oscillation per exposure, a 99.2% complete data set at 2.8 Å resolution was obtained by indexing, integrating, scaling, and merging partial data sets from 17 crystals using HKL3000. Structure determination was performed by molecular replacement using the Phaser module in the Phenix suite. Two search models, one containing the PGS domain and one containing the receptor seven-helix bundle, were created using the structures of OX₂R (PDB, 4S0V) and 5-HT_1BR-ergotamine (PDB, 4IAR), respectively. Subsequent refinement was performed using the MR solution with rounds of PHENIX. refine with simulated annealing, and Refmac5 using translation, libration, and screw-rotation (TLS) parameters along with overall setting for b-factor refinement. Manual examination and rebuilding of refined coordinates were accomplished using COOT. The quality of the model was checked using the Molprobity server. The Ramachandran statistics analyzed using MolProbity are 93.9% favored, 6.1% allowed, and 0.0% outliers for this structure. A summary of data collection and refinement statistics is reported in Table 1. 2Fo-Fc maps imported into PyMOL were generated using the FFT program of the CCP4 suite. Figures were prepared in PyMOL (Schrödinger).

Receptor-G protein complex purification.

The purification of the active state 5-HT_5AR is similar to the purifaction of MRGPRX2-Gq-scFv16 complex³⁷. The cell pellet of 5-HT_5AR-miniGo complex was thawed on ice and incubated with a buffer containing 20 mM HEPES pH 7.5, 50 mM NaCl, 1 mM MgCl₂, proteinase inhibitor, and agonist (5-CT, lisuride, or methylergometrine) at room temperature. After 1.5 h, the cell suspension was homogenized. Membrane was collected by centrifugation at 70,000 g for 30 min using a Ti45 rotor (Beckman) and solubilized using 40 mM HEPES pH 7.5, 100 mM NaCl, 5% (w/v) glycerol, 0.5% (w/v) Lauryl maltose neopentyl glycol (LMNG, Anatrace, NG310), 0.05% (w/v) CHS for 5 h at 4°C. The solubilized proteins in the supernatants were isolated by ultra-centrifugation at 160,000 g for 30 min using a Ti70 rotor and then incubate overnight at 4°C with TALON IMAC resin and 20 mM imidazole. The resin was collected next day and washed with 25 column volumes 20 mM HEPES pH 7.5, 100 mM NaCl, 30 mM imidazole, 0.01% (w/v) LMNG, 0.001% (w/v) CHS and 5-HT_5A agonist (5-CT, lisuride, or methylergometrine). The protein was then eluted using the same buffer supplemented with 250 mM imidazole. Eluted protein was concentrated and subjected to size-exclusion chromatography on a Superdex 200 Increase 10/300 column (GE Healthcare, 289909944) that was pre-equilibrated with 20 mM HEPES pH 7.5, 100 mM NaCl, 1 μM agonist, 0.00075% (w/v) LMNG, 0.00025 (w/v), glyco-diosgenin (GDN, Anatrace, GDN101), and 0.00075% (w/v) CHS. Peak fractions were collected and incubated with 15 μl of His-tagged PreScission protease and 2 μl PNGase F (NEB, P0708S) at 4°C overnight to remove the N-terminal BRIL and potential glycosylation. The protein was concentrated and further purified the next day by size-exclusion chromatography using the same buffer. Peak fractions were collected and concentrated to around 5 mg/ml.

CryoEM data collection, 3D reconstitution, model building and refinement

The cryoEM data collection and processing of 5-HT_5AR-miniGo complex follows a published protocol³⁸. The samples (3.2 μl) were applied to glow discharged Quantifoil R1.2/1.3 Au300 holey carbon grids (Ted Pella) individually and were flash-frozen in a liquid ethane/propane (40/60) mixture using a Vitrobot Mark IV (FEI) set at 4°C and 100% humidity with a blot time range from 2.5 to 5 s. Images were collected using a 200 keV Talos Artica with a Gatan K3 direct electron detector at a physical pixel size of 0.91 Å. Micrograph recorded movies were automatically collected using SerialEM using a multishot array³⁹. Data were collected at an exposure dose rate of ~15 electrons/pixel/second as recorded from counting mode. Images were recorded for ~1.7-2.7 seconds in 60 subframes to give a total exposure dose of ~29-48 electrons per Ų. Following manual inspection and curation of the micrographs, particles from each dataset were selected using the Blob particle picker and initial 2D classification yielded templates for subsequent template picking. After one round of two-dimensional classification and selection in cryoSPARC, a subset of the selected particles was used as a training set for Topaz and the particles were re-picked from the micrographs using Topaz⁴⁰ and subjected to two-dimensional classification and three-dimensional classification. The select classified picked particle coordinates from the three sets were next merged yielding a subset of unique particles that survived 2D classification (i.e. duplicates were removed with a radius of 75 pixels). All subsequent three-dimensional classification and refinement steps were performed within cryoSPARC^41,42. Multiple rounds of multi-reference refinement resolved the final stack of particles that produced a map with a resolution reported in Table 2 (by FSC using the 0.143 Å cut-off criterion)⁴³ after Global CTF refinement and post-processing including soft masking, B-factor sharpening in cryoSPARC, and filtering by local resolution⁴⁴ to generate the post-processed sharpened map. Alternative post sharpening was performed on the two half-maps using deepEMhancer⁴⁵. For more details see Table 2 and Extended Data Fig. 2.

Maps from deepEMhancer were used for map building, refinement, and subsequent structural interpretation. The 5-HT_1BR-miniGo structure (PDB code: 6G79) was used as the initial model and docked into the cryoEM map using Chimera⁴⁶ followed by iterative manual adjustment in COOT⁴⁷, and Phenix.real_space_refine in Phenix⁴⁸. The model statistics were validated using Molprobity⁴⁹. Structural figures were prepared by Chimera or Pymol (https://pymol.org/2/).

Bioluminescence Resonance Energy Transfer (BRET1 and BRET2)

To measure G protein recruitment (BRET1) of 5-HT_5AR, HEK-293T cells (ATCC CRL-11268) maintained in DMEM containing 10% (v/v) dialyzed FBS, 1 IU/ml Penicillin G, and 100 μg/ ml Streptomycin were passed to 10 cm dishes and co-transfected using Transit (Mirus Bio) in an approximate 1:3 ratio with WT or mutant 5-HT_5AR containing C-terminal Renilla luciferase (Rluc) and Venus-tagged N-terminal miniGi protein containing an N-terminal nuclear export signal, respectively²⁴. After at least 24 hours, transfected cells were plated in poly-lysine coated 96-well white clear bottom cell culture plates in plating media (DMEM containing 1% (v/v) dialyzed FBS, 1 IU/ml Penicillin G, and 100 μg/ml Streptomycin) at a density of 40,000 cells in 200 μL per well and incubated overnight. The following day, media was aspirated and cells were washed once with 60 μL of drug buffer (1x HBSS, 20 mM HEPES, pH 7.4). Then 60 μL of Rluc substrate, coelenterazine h (Promega) at 5 μM final concentration was added per well and drug stimulation was performed with the addition of 30 μL of 3x drug dilution in drug dilution buffer (1x HBSS, 20 mM HEPES, 0.3% (w/v) BSA, 0.03% (w/v) ascorbic acid, pH 7.4) per well and incubated at room temperature. After 10 minutes of incubation, plates were read for both luminescence at 485 nm and fluorescent eYFP emission at 530 nm for 1 second per well using a Mithras LB940 (Berthold Technologies). The BRET ratio of eYFP/Rluc was calculated per well and plotted as a function of drug concentration using Graphpad Prism 9.2 (Graphpad Software Inc., San Diego, CA).

To measure 5-HT_5A-mediated G protein dissociation (BRET2), procedures were similar to 5-HT_5A recruitment, except HEK293T cells were co-transfected in a 2:1:1:2 ratio of Gi1-Rluc, Gβ1, GFP₂-Gγ2, and 5-HT_5AR, respectively. G protein dissociation BRET2 assays utilized 10 μL of the Rluc substrate Coelenterazine 400a (Nanolight, 5 μM final concentration), incubated for 10 minutes, and read for luminescence at 400 nm and fluorescent GFP₂ emission at 515 nm for 1 second per well using a Mithras LB940. The ratio of GFP₂/Rluc was calculated per well and plotted as a function of drug concentration using Graphpad Prism 9.2 (Graphpad Software Inc., San Diego, CA).

Radioligand binding assays

Competitive binding assays were performed using membrane preparations from HEK293 T cells transiently expressing WT or mutant 5-HT_5AR. Binding assays were set up in 96-well plates in the standard binding buffer (50 mM Tris, 0.1mM EDTA, 10mM MgCl₂, 0.1% BSA, 0.01% ascorbic acid, pH 7.40). For the competition binding, 50 μL each of ³H-LSD, drug solution (3x), and homogeneous wt or mutant 5-HT_5AR membrane solution were incubated in 96-well plates in the standard binding buffer. Reactions were incubated for 2 h at room temperature in the dark and terminated by rapid vacuum filtration onto chilled 0.3% PEI-soaked GF/A filters followed by three quick washes with cold washing buffer (50 mM Tris HCl, pH 7.40) and read. Results were analyzed using the equation ‘one-site fit Ki’ in GraphPad Prism 9.2.

FEP+ calculations.

All calculations were conducted using v2020-4 of the Schrödinger modeling suite³⁵. The X-ray crystal structure of 5-HT_5AR-AS2674723 was prepared with Protein Preparation Wizard with the default protocol, and protonation states of histidine, glutamic, and aspartic acid residues were predicted by PROPKa⁵⁰ at pH 7.4. A membrane of pre-equilibrated (300K) POPC lipids based on OPM database alignment was added to the system together with SPC solvent. Counterions were added to neutralize the system with additional Na⁺ and Cl⁻ ions added to a 0.15 mM concentration. The MD systems were equilibrated for 100 ps of Brownian NVT at 10 K with restraints on all solute’s heavy atoms, followed by 100 ps of Brownian NPT at 50 K and 200 ps of NPγT at 50 K. Then the system was heated from 100 K to 300 K over 300 ps of NPγT ensemble and gradual release of restraints. A final NVT unrestrained was run for 200 ps. The production runs were sampled for 50ns of unrestrained NPγT ensemble at 298 K.

Glide SP docking with tight core constraint was used to align ligands to AS2674723 of the equilibrated system. Missing torsional parameters of ligands were added by fitting additional quantum mechanical (QM) calculations with Force Field Builder⁵¹. Two rounds of FEP were performed using FEP+⁵². Only heavy atoms of the ligand were included in the REST region. A total of 12 λ windows were used for all calculations and replica exchanges between neighboring λ windows were attempted every 1.2 ps.

The first perturbation map was built with ASP5736 as the reference compound and the default protocol. Three (C5, C6, and C7) out of 16 compounds with predicted ΔΔG < −0.5 kcal/mol were recommended for synthesis and testing. C7 emerged from the first round FEP was used as the reference compound for the second perturbation map. Allowing for free rotation of benzene ring, isoquinoline was used as the custom core of the perturbation map. To test the FEP method prospectively, 10 compounds were synthesized where C16 with predicted ΔΔG < −0.5 kcal/mol; C8, C9, C10, C12, C13, MS112, and C15 with predicted ΔΔG < 0.5 kcal/mol; C11 and C14 with predicted ΔΔG >2 kcal/mol.

For the calculations of the experimental ΔΔG values, we applied the equation ΔG = –RTlnK_i based on the Ki affinities (M) of the compounds. In the equation, the R is the gas constant (1.987x10⁻³ kcal K⁻¹ mol⁻¹) and T is the temperature in Kelvin (here is 298.15 K).

Surface expression

Cell-surface expressions of WT and mutant 5-HT_5ARs were measured using ELISA chemiluminescence³⁷. In brief, 48-h post-transfected cells plated in 384 white-well plates were fixed with 20 μl per well 4% (v/v) paraformaldehyde for 10 min at room temperature. The cells were then washed with 40 μl per well of PBS twice and then incubated with 20 μl per well 5% (v/v) BSA in PBS for 1 h. Cells are incubated with an anti-Flag–horseradish peroxidase-conjugated antibody (Sigma-Aldrich, A8592) diluted 1:10,000 in 5% (v/v) BSA in PBS for 1 h at room temperature. After washing 5 times with 80 μl PBS per well, 20 μl per well Super Signal Enzyme-Linked Immunosorbent Assay Pico Substrate (Thermo Fisher, 37070) was added to the well for the development of signal, and the luminescence was counted using a PHERAstar FSX (BMG Labtech). The luminescence signal was analyzed in GraphPad Prism 9.2 and data were normalized to the signal of WT 5-HT_5AR.

Extended Data

Extended Data Fig. 1. Sequence alignment and transducerome profiling of 5-HT_5AR.

a, Sequencing alignment matrix of the 12 GPCR members in the 5-HT receptor family. Similarities are shown on the lower-left side of the table and identities on the upper-right. b, Transducerome screening of 5-HT_5AR using TRUPATH platform by the endogenous agonist 5-HT. Net BRET values of 5-HT_5AR together with positive controls of either neurotensin receptor 1 (NTSR1, agonist NT1-13) or β₂AR (agonist isoproterenol) are shown in each panel. Data represent mean ± SEM of N = 3 biological replicates.

Extended Data Fig. 2. CryoEM analysis for 5-HT_5AR-miniGo bound to 5-CT, Lisuride, and Methylergometrine.

For each of the respective agonist bound 5-HT5AR heterotrimeric complexes are shown: a, Histograms of defocus values for micrographs used in the single-particle analysis (see Table 2 for more details). b. Representative frame aligned micrograph. The experiment was repeated three times with similar results. c, Orientational distribution heat map. d, 2D plots of the gold-standard Fourier shell correlation (GSFSC) between half maps (black) and FSC between model and the B-factor sharpened map for respective refined model (red) as calculated by Phenix.mitrage. e, Local resolution heat-map calculated using the local windowed FSC method.

Extended Data Fig. 3. Comparisons of 5-HT_5AR structures and the interface between 5-HT_5AR and miniGo protein.

a-c, Extracellular and intracellular views of receptors (a-b) and G proteins (c) from the superpositions of the three agonist-bound 5-HT_5AR complexes. d-e, two views of the displacements of three alanines in the 3A cluster upon the receptor activation. The movements of the residues are indicated by the red arrows. f. superposition of the 5-CT-bound 5-HT_5AR structure with the active state 5-HT_1AR structures. The Ptdlns4P molecule is shown in sticks. g-h, Interactions between the R4x38 or R34x57 and D3x49 in the active (g) and inactive (h) 5-HT_5AR structures. i, Representative density maps of Y225^H5.26 and K279^6x29. j, Interactions between the V287^6x37 and surrounding residues in the inactive 5-HT_5AR structure. k, The electrostatic potential surface of 5-HT_5AR from the intracellular side and interaction with the α5 helix of miniGo.

Extended Data Fig. 4. Ligand binding pocket of 5-HT_5AR.

a-b, Two views of the superposition of ligands AS2674723, 5-CT, lisuride, and methylergometrine in the 5-HT_5AR binding site. The hydrogen bonds between T126 and three agonists are shown as dashed lines in the same color as the corresponding ligand, respectively. The positions of EBP1 and EBP2 are also indicated by the purple- and salmon-shaded ovals, respectively. c, Displacement of waters in the binding pocket of the apo 5-HT_1AR-Gi complex structure (PDB: 7E2X) compared with the water molecule in the 5-CT bound structure. d, The representative density map of E305 in the 5-CT bound 5-HT_5AR structure. e-g, cryo-EM maps of T126 and three agonists. The distances are indicated by the dashed lines. h. Structural alignment of the ligand-binding pockets of three agonist-bound 5-HT_5AR complexes. i. Binding affinities of 5-HT and 5-CT to all the members of the human serotonin receptor family from the Ki database of PDSP (https://pdsp.unc.edu/databases/kidb.php). Data represent mean ± SEM of N = 1-10 replicates based on the source from the Ki database.

Extended Data Fig. 5. Graphical representation of binding affinities.

a, Binding affinities of AS2674723 against the aminergic receptors. Data represent mean ± SEM of N = 3-10 replicates (N=1 for muscarinic receptors (M1-M5)). b-c, Binding affinities of AS2674723 and LSD for WT and mutant 5-HT_5ARs using 3H-LSD. See Supplementary Table 11 for fitted parameter values that represent mean ± SEM of N = 3 biological replicates.

Extended Data Fig. 6. Sequence alignments of 12 GPCR members of the 5-HT receptor family.

a, Alignment of the residues in the ligand-binding pocket. b, Alignment of the residues in ICL2, 3A cluster, and receptor-G protein interface.

Extended Data Fig. 7. Functional validation of the double mutant (A226V and A227L).

a-b, BRET2 Gi1-activation assays and c-d, BRET1 β-arrestin2 recruitment of WT and double mutant (A226V and A227L) 5-HT_5AR stimulated by eight ligands. See Supplementary Table 12 for fitted parameter values that represent mean ± SEM of N = 3 biological replicates.

Extended Data Fig. 8. Drug development of 5-HT_5AR.

a-b, Chemical structures of all the designed compounds in the second (a) and third rounds (b) of structure-based drug development of 5-HT_5AR. Compounds with the ID colored in black or pink are selected from the patent or newly designed, respectively. The starting compound in each cycle is highlighted in a yellow shaded box. The varied moieties are colored in the corresponding color used in the starting compound. The predicted ΔΔG values by the FEP method are also labeled under the corresponding compound.

Extended Data Fig. 9. Functional validation of the new selective compound MS112 and correlation analysis of FEP prediction.

a-b, Agonist and antagonist activities tests of the compound MS112 towards 5-HT_5AR. See Supplementary Table 13 for fitted parameter values that represent mean ± SEM of N = 4 biological replicates. c-d, Correlation analysis between the predicted ΔΔG and experimental ΔΔG values. The dotted lines in both panels represent the ±0.5 kcal/mol between the FEP predicted ΔΔG and the experimental ΔΔG, respectively. And the +0.5 kcal/mol and −0.5 kcal/mol dotted lines are colored in magenta and blue, respectively. See Supplementary Table 14 for details.

Extended Data Fig. 10. Surface expression level of wt and mutant 5-HT_5ARs.

Data represent mean ± SEM of N = 3 biological replicates.

Data Availability

The structures of the 5-HT_5AR-AS2674723, 5-HT_5AR-miniGo-5-CT, 5-HT_5AR-miniGo-Lisuride, and 5-HT_5AR-miniGo-Methylergomertrine have been deposited with the PDB (EMDB) under accession code 7UM4, 7UM5 (EMD-26597), 7UM6 (EMD-26598), and 7UM7 (EMD-26599). The cryoEM micrographs of 5-HT_5AR-miniGo-5-CT, 5-HT_5AR-miniGo-Lisuride, and 5-HT_5AR-miniGo-Methylergomertrine have been deposited in the EMPIAR database (https://www.ebi.ac.uk/empiar/) with accession numbers EMPIAR-11033, EMPIAR-11036, and EMPIAR-11039, respectively.