PocketGen: Generating Full-Atom Ligand-Binding Protein Pockets

Abstract

Designing small-molecule-binding proteins, such as enzymes and biosensors, is crucial in protein biology and bioengineering. Generating high-fidelity protein pockets—areas where proteins interact with ligand molecules—is challenging due to complex interactions between ligand molecules and proteins, flexibility of ligand molecules and amino acid side chains, and complex sequence-structure dependencies. Here, we introduce PocketGen, a deep generative method for generating the residue sequence and the full-atom structure within the protein pocket region that leverages sequence-structure consistency. PocketGen consists of a bilevel graph transformer for structural encoding and a sequence refinement module that uses a protein language model (pLM) for sequence prediction. The bilevel graph transformer captures interactions at multiple granularities (atom-level and residue/ligand-level) and aspects (intra-protein and protein-ligand) with bilevel attention mechanisms. For sequence refinement, a structural adapter using cross-attention is integrated into a pLM to ensure structure-sequence consistency. During training, only the adapter is fine-tuned, while the other layers of the pLM remain unchanged. Experiments show that PocketGen can efficiently generate protein pockets with higher binding affinity and validity than state-of-the-art methods. PocketGen is ten times faster than physics-based methods and achieves a 95% success rate (percentage of generated pockets with higher binding affinity than reference pockets) with over 64% amino acid recovery rate.

Introduction

A primary method for modulating protein functions involves the interaction between proteins and small molecule ligands^1–4. These interactions play a critical role in biological processes, such as enzymatic catalysis, signal transduction, and regulatory mechanisms within cells. The binding of small molecules to specific sites on proteins can induce conformational changes, modulate activity, or inhibit functions. This mechanism serves as a valuable tool for studying protein functions and designing small molecule-binding proteins with customized properties for therapeutic and industrial applications, including designing enzymes to catalyze reactions that do not have natural catalysts^5–8 and developing biosensors that can detect compounds in the environment by transducing signals which can be used for environmental monitoring, clinical diagnostics, pathogen detection, drug delivery systems, and applications in the food industry^9–12. These designs often involve modifying existing ligand-binding protein pockets to facilitate more precise interactions with specific ligands^13–15. The complexity of ligand molecule-protein interactions, ligand and side chain flexibility, and sequence-structure relationships, however, pose significant challenges for computational generation of high-validity, ligand-binding protein pockets^3,15,16.

Traditional methods for pocket design focus on physics modeling or template-matching^{10,11,13,17,18}. For example, PocketOptimizer^18–20 develops a pipeline that predicts mutations in protein pockets to increase binding affinity based on physics-inspired energy functions and search algorithms. Initiated with a bound protein-ligand complex, PocketOptimizer explores side chain structures and residue types. These mutants are evaluated using energy functions and ordered through integer linear programming techniques. Another line of research uses template matching and enumeration methods^11,13,14,17. For instance, Polizzi et al.,¹³ employ a two-step strategy for pocket design. They first identify and assemble disconnected protein motifs (van der Mer (vdM) structural units) surrounding the target molecule to build protein-ligand interactions (e.g., Hydrogen bonds). Subsequently, they graft these residues onto the protein scaffold and select the best combinations of protein-ligand pairs using scoring functions. This template-matching strategy enabled the de novo design of proteins binding the drug apixaban²¹. However, methods based on physics modeling and template-matching can be time-consuming, often requiring several hours for a single protein pocket design. In practical protein engineering, evaluating thousands to millions of designed protein candidates to identify a successful candidate significantly increases the total time required for protein pocket design. The focus on specific fold types (e.g., four-helix bundle¹³ or NTF2¹⁴) by these methods further restricts their widespread application.

Recent advancements in protein pocket design have been facilitated by deep learning-based approaches^{3,8,16,22–24}. For instance, RFDiffusion²⁵ employs denoising diffusion probabilistic models²⁶ in conjunction with RoseTTAFold²⁷ for de novo protein structure generation. Despite its capability to design pockets targeting binding with specific ligands, the approach’s auxiliary guiding potentials lack precision in modeling protein-ligand interactions. RFdiffusion All-Atom (RFdiffusionAA)¹⁶ represents a significant step forward, enabling the direct generation of binding proteins around small molecules through iterative denoising. This advance is attributed to modifications in model architecture to consider protein structures and ligand molecules concurrently. Nevertheless, the derivation of residue sequences in RFDiffusion and RFdiffusionAA involves post-processing with ProteinMPNN²⁸, potentially leading to inconsistencies in modeling sequence and structure protein modalities. Conversely, our prior work FAIR²³ simultaneously designs the complete atom pocket structure and sequence using a two-stage refinement approach, employing a coarse-to-fine approach that initially updates the backbone protein structure before addressing the full atom structure, including side chains. FAIR iteratively refines atom coordinates and residue sequence predictions until convergence is achieved. However, a gap between the two refinement steps occasionally results in instability and limited generation performance, indicating the need for an end-to-end generative approach for pocket design. Related research focused on the sequence and structure co-design of complementarity determining regions (CDRs) in antibodies^29–33. These methods, specifically devised for antibodies, face challenges when adapting to pocket designs conditioned on target ligand molecules. Hybrid approaches that combine deep learning models with traditional methods are also being explored^3,8. For example, Yeh et al.⁸ developed a novel Luciferase by employing a combination of protein hallucination³⁴, the trRosetta structure prediction neural network³⁵, hydrogen bonding networks, and RifDock³⁶, generating a multitude of idealized protein structures with diverse pocket shapes for subsequent filtering. Despite its success, this method’s applicability is confined to specific protein scaffolds and substrates, lacking a generalized solution. Similarly, Lee et al.³ merge deep learning with physics-based methods to create proteins featuring diverse and designable pocket geometries. This integration involves backbone generation via trRosetta hallucination, sequence design through ProteinMPNN²⁸ and LigandMPNN³⁷, and filtering with AlphaFold³⁸. The challenges facing current deep learning models for pocket generation include achieving sequence-structure consistency, accurately modeling complex protein-ligand interactions, and realizing generalized end-to-end pocket generation.

Here, we introduce PocketGen, a deep generative method for full-atom protein pocket generation that addresses existing limitations. PocketGen implements a co-design scheme (Figure 1(a)), where the model concurrently predicts the sequence and structure of the protein pocket based on the ligand molecule and the protein scaffold (excluding the pocket). PocketGen comprises two modules: the bilevel graph transformer (Figure 1(b)) and the sequence refinement module (Figure 1(c)). To achieve end-to-end molecular generation, PocketGen models the protein-ligand complex as a geometric graph of blocks to handle variable atom counts across different residues and ligands. Initially, the pocket residues are assigned the maximum possible number of atoms (14 atoms) and later are mapped back to specific residue types post-generation. The bilevel graph transformer captures interactions at multiple granularities (atom-level and residue/ligand-level) and aspects (intra-protein and protein-ligand) using bilevel attention mechanisms. To account for the influence of the redesigned pocket on the ligand, the ligand structure is updated during refinement to reflect potential changes in binding pose. To ensure consistency across protein sequence and structure domains and incorporate evolutionary information encoded in protein language models (the ESM series models^39,40), PocketGen implements a structural adapter into protein sequence updates. This adapter facilitates cross-attention between sequence and structure features, promoting information flow and achieving sequence-structure consistency. During training, only the adapter is fine-tuned, while the remaining layers of the protein language models remain unchanged. Experimental results demonstrate that PocketGen significantly outperforms current state-of-the-art methods in protein pocket generation across two popular benchmarks. PocketGen achieves an average amino acid recovery rate of 63.40% and a Vina score of −9.655 for top-1 generated protein pockets on the CrossDocked dataset. Comprehensive analyses indicate that PocketGen can produce diverse and high-affinity protein pockets for therapeutically useful molecules, showcasing its efficacy and potential in designing small-molecule binders and enzymes.

Figure 1. Overview of PocketGen generative model for the design of full-atom ligand-binding protein pockets.

a, Conditioned on the binding ligand molecule and the rest part of the protein except the pocket region (i.e., scaffold), PocketGen aims to generate the full atom pocket structure (backbone and sidechain atoms) and the residue type sequence with iterative equivariant refinement. The ligand structure is also adjusted during the protein pocket refinement. b, Bilevel graph transformer is leveraged in PocketGen for all-atom structural encoding and update. The bilevel level attention captures both the residue/ligand and atom-level interactions. Both the protein pocket structure and the ligand molecule structure are updated in the refinement. c, Sequence refinement module adds lightweight structural adapter layers into pLMs for sequence prediction. Only the adapter’s parameters are fine-tuned during training, and the other layers are fixed. In the adapter, the cross-attention between sequence and structure features is performed to achieve sequence-structure consistency.

Results

Benchmarking generated ligand-binding protein pockets across molecular quality dimensions

PocketGen is benchmarked on two widely used datasets, following previous studies^23,41. The CrossDocked dataset⁴² comprises protein-molecule pairs generated through cross-docking and is divided into train, validation, and test sets based on a sequence identity threshold of 30%. The Binding MOAD dataset⁴³ includes experimentally determined protein-ligand complexes that are split into train, validation, and test sets according to the proteins’ enzyme commission numbers⁴⁴. Given the range of distances relevant to protein-ligand interactions⁴⁵, our experimental setting considers all residues with atoms within 3.5 °A of any binding ligand atoms as part of the protein pocket, averaging about eight residues per pocket. We also investigate the capability of PocketGen to design larger pocket areas (e.g., 5.5 °A) and include more residues. Methodological details on data processing are provided in the Methods section.

To comprehensively evaluate the quality of protein pockets generated by PocketGen, we use the following three sets of metrics. Firstly, AutoDock Vina score⁴⁶ measures the affinity between the generated pocket and the target ligand molecule. Secondly, we assess the structural validity of the generated pocket using RMSD, pLDDT, and scTM. RMSD quantifies the Root Mean Square Deviation between the backbone of the generated pocket and the reference structure in the test set, with a lower RMSD indicating a more accurate backbone conformation. The predicted Local Distance Difference Test (pLDDT) score, derived from AlphaFold2³⁸, reflects the confidence in structural predictions on a scale from 0 to 100, with higher scores indicating greater confidence. For efficiency, we input the designed protein sequence into ESMFold⁴⁷ and report the average pLDDT score across pocket residues. scTM, the self-consistency template modeling score, evaluates the designability of the generated pocket structure by determining how likely it is that the predicted residue sequence will realize the pocket structure. This process involves generating eight sequences per input structure by feeding the generated structure into ProteinMPNN²⁸, then using ESMFold⁴⁷ to predict the structure of each putative sequence, following established methodology^48,49. We calculate scTM by comparing the TM-scores⁵⁰ between the ESMFold-predicted structure and the original structure, with scores ranging from 0 to 1, where higher values indicate greater designability. Finally, we determine the Amino Acid Recovery (AAR), which is the percentage of correctly predicted pocket residue types, to evaluate the designed sequence’s accuracy. A higher AAR signifies a deeper understanding of sequence-structure relationships.

The baselines in this study encompass a wide spectrum of models, from recently developed deep learning-based methods such as RFDiffusion²⁵, FAIR²³, and dyMEAN²⁴, to the template-matching method DEPACT¹⁷ and the traditional physics modeling method PocketOpt¹⁸ (refer to Methods for detailed descriptions of baselines). This paper limits comparisons to RFDiffusion due to the unavailability of the source code for RFDiffusionAA. In Table 2, PocketGen and other methods are asked to generate 100 sequences and structures for each protein-ligand complex in the CrossDocked and Binding MOAD dataset test set. We present the mean and standard deviation values over three different runs. PocketOpt is not compared in Table 2 because it focuses on mutating existing pockets for optimization and may be time-consuming for generating a large number of protein pockets. Table 2 shows that PocketGen surpasses all the baselines in RMSD and Vina scores (by around 0.10 and 0.20 on CrossDocked respectively), reflecting its effectiveness in generating structurally valid pockets with high binding affinities. This superior performance stems from PocketGen’s capability to capture interactions at multiple granularities (atom-level and residue/ligand-level) and aspects (intra-protein and protein-ligand). Additionally, PocketGen achieves a significant improvement by around 20% in AAR, benefiting from the incorporation of a protein language model that encompasses evolutionary sequence information. In protein engineering, the practice often involves mutating several key residues to optimize properties while keeping most residues unchanged to preserve protein folding stability^51,52. A high AAR achieved by generated protein pockets aligns with this practice. Moreover, in Table 1, the top-1, 3, 5, and 10 protein pockets generated by PocketGen demonstrate the lowest Vina scores and competitive RMSD, pLDDT, and scTM scores, showcasing PocketGen’s ability to produce high-affinity pockets that maintain structural validity and sequence-structure consistency. With a 97% success rate in creating pockets with higher affinity than the reference cases (the success rate of the strongest baseline method is 92%) on CrossDocked, PocketGen’s effectiveness and broad applicability across various ligand molecules are clearly demonstrated.

Table 2.

Benchmarking PocketGen and other approaches for pocket generation on two datasets. Reported are average and standard deviation values across three independent runs. The best results are bolded.

Model	CrossDocked			Binding MOAD
	AAR (↑)	RMSD (↓)	Vina (↓)	AAR (↑)	RMSD (↓)	Vina (↓)
Test set	-	-	−7.016	-	-	−8.076
DEPACT	31.52±3.26%	1.59±0.13	−6.632±0.18	35.30±2.19%	1.52±0.12	−7.571±0.15
dyMEAN	38.71±2.16%	1.57±0.09	−6.855±0.06	41.22±1.40%	1.53±0.08	−7.675±0.09
FAIR	40.16±1.17%	1.46±0.04	−7.015±0.12	43.68±0.92%	1.37±0.07	−7.930±0.15
RFDiffusion	46.57±2.07%	1.44±0.07	−6.936±0.07	45.31±2.73%	1.45±0.10	−7.942±0.14
PocketGen	63.40±1.64%	1.36±0.05	−7.135±0.08	64.43±2.35%	1.32±0.05	−8.112±0.14

Table 1.

The top 1/3/5/10 generated protein pocket (based on Vina Score) properties on the CrossDocked dataset. The RMSD, pLDDT, and scTM scores for PocketOpt are not reported as PocketOpt keeps protein backbone structures fixed.

	PocketOpt	DEPACT	dyMEAN	FAIR	RFDiffusion	PocketGen
Top-1 generated protein pocket
Vina score (↓)	−9.216	−8.527	−8.540	−8.792	−9.037	−9.655
Success Rate (↑)	0.92	0.75	0.76	0.80	0.89	0.97
RMSD (↓)	-	1.47	1.44	1.39	1.13	1.21
pLDDT (↑)	-	82.1	83.3	83.2	84.5	86.7
scTM (↑)	-	0.901	0.906	0.899	0.924	0.937
Top-3 generated protein pockets
Vina score (↓)	−8.878	−8.131	−8.196	−8.321	−8.876	−9.353
RMSD (↓)	-	1.45	1.43	1.40	1.18	1.24
pLDDT (↑)	-	81.9	82.8	83.1	84.6	86.2
scTM (↑)	-	0.896	0.892	0.897	0.929	0.934
Top-5 generated protein pockets
Vina score (↓)	−8.702	−7.786	−7.974	−7.943	−8.510	−9.239
RMSD (↓)	-	1.46	1.45	1.42	1.25	1.22
pLDDT (↑)	-	82.2	82.9	83.3	84.3	86.1
scTM (↑)	-	0.892	0.903	0.886	0.926	0.935
Top-10 generated protein pockets
Vina score (↓)	−8.556	−7.681	−7.690	−7.785	−8.352	−9.065
RMSD (↓)	-	1.53	1.44	1.41	1.26	1.28
pLDDT (↑)	-	81.5	82.7	83.0	84.2	85.9
scTM (↑)	-	0.895	0.896	0.884	0.924	0.931

To investigate the substructure validity and consistency with reference datasets, we conduct a qualitative substructure analysis (Table S2 and Figure S1). This analysis considers three covalent bonds in the residue backbone (C-N, C=O, and C-C), three dihedral angles in the backbone ($\varphi ,\psi ,\omega$⁵³), and four dihedral angles in the side chains (${\chi }_1,{\chi }_2,{\chi }_3,{\chi }_4$ ⁵⁴). Following previous research^55,56, we collect the bond length and angle distributions in the generated pockets and the test dataset and compute the Kullback-Leibler (KL) divergence to quantify the distribution distances. Lower KL divergence scores for PocketGen indicate its effectiveness in replicating the geometric features present in the data.

Probing generative capabilities of PocketGen

Next, we explore PocketGen’s generative capabilities. In addition to generating high-quality protein pockets, an important capability is to generate high-fidelity pocket candidates that can be directly fed into biochemical screening assays and maximize the yield of downstream experiments. Figure 2(a) offers a comparative analysis of average runtime across different methods. Traditional approaches, such as physics-based modeling (i.e., PocketOpt) and template-matching (i.e., DEPACT), require over 1,000 seconds to generate 100 pockets. Similarly, the runtime for the advanced protein backbone generation model RFDiffusion is significant due to its diffusion architecture. In contrast, recent methods employing iterative refinement, including PocketGen, show a notable decrease in generation time.

Figure 2. Exploring the capabilities of PocketGen.

a, The average runtime of different methods for generating 100 protein pockets for a ligand molecule on the two benchmarks. b, The trade-off between quality (measured by Vina score) and diversity (1 − average pairwise sequence similarity) of PocketGen. We can balance the trade-off by tuning the temperate hyperparameter τ. c, The influence of the design pocket size on the metrics. d, Performance w.r.t. model scales of pLMs using ESM series on CrossDocked dataset. The green dots represent PocketGen models with different ESMs. The bubble size is proportional to the number of trainable parameters. e, PocketGen tends to generate pockets with higher affinity for larger ligand molecules (Pearson Correlation −0.61, shadow indicates 95% confidence interval). f, The top molecular functional groups leading to high affinity.

While recent methods for pocket generation prioritize maximizing binding affinity with target molecules, this strategy may not always align with practical needs, where pocket diversity also plays a critical role. To improve the success rate of pocket design, examining a batch of designed pockets rather than a single design is beneficial. Thus, we examine the relationship between binding affinity and diversity of generated protein pockets in Figure 2(b). Diversity is quantified as (1 − average pairwise pocket residue sequence similarity) and can be adjusted by altering the sampling temperature $\tau$ (with higher $\tau$ resulting in greater diversity). Figure 2(b) illustrates that binding affinity and diversity present a trade-off. PocketGen is capable of producing protein pockets with higher affinity at equivalent levels of diversity, showcasing its utility in practical applications.

In Figure 2(c), the impact of changes in the redesigned pocket size on PocketGen’s performance is examined. The redesign process targets all residues with atoms within 3.5 °A, 4.5 °A, and 5.5 °A of any binding ligand atoms. The average AAR, RMSD, and Vina scores show a slight decline for larger protein pockets. This trend is attributed to the increased complexity and reduced contextual information associated with expanding the redesigned protein pocket area. Furthermore, an observation of note is that larger pockets tend to enable the exploration of structures with potentially higher affinity, as indicated by the lowest Vina scores, reaching −20 kcal/mol for designs with a 5.5 °A radius. This phenomenon could be due to the enhanced structural complementarity achievable in larger pocket designs.

Incorporating protein Language Models (pLMs) distinguishes PocketGen from prior pocket generation models. Beyond employing ESM-2 650M⁴⁷ in the default configuration, various versions of ESM with parameter counts ranging from 8M to 15B were also evaluated. As illustrated in Figure 2(d), PocketGen’s performance improves with the scaling of pLMs. Specifically, a logarithmic scaling law is observed, aligning with trends noted in large language models⁵⁷. Furthermore, PocketGen achieves efficient training with large pLMs by fine-tuning adapter layers while keeping the majority of pLM layers fixed. Consequently, PocketGen has significantly fewer trainable parameters compared to RFDiffusion (7.9 M versus 59.8 M parameters), leading to an efficient and scalable approach.

The performance of PocketGen in generating binding pockets is influenced by the characteristics of the ligand molecule. Figure 2(e) illustrates the relationship between the average Vina score of generated pockets and the number of ligand atoms, indicating that PocketGen tends to generate pockets with higher affinity for larger ligand molecules. This trend may be due to the increased surface area for interaction, the presence of additional functional groups, and enhanced flexibility in the conformation of larger molecules^58,59. Additionally, functional groups in ligand molecules that contribute to high binding affinity were identified using IFG⁶⁰. Figure 2(f) displays the top 10 molecular functional groups, which include hydrogen bond donors and acceptors (e.g., carbonyl groups), aromatic rings, sulfhydryl groups, and halogens. These groups are capable of forming favorable interactions with protein pockets, thereby increasing binding affinity.

In the supplementary material, ablation studies (Table S3) and hyperparameter analysis (Figure S2) were conducted to evaluate the contribution of each module in PocketGen and the impact of loss function hyperparameters on model performance. The results indicate that the bilevel attention module and the integration of pLM into PocketGen substantially improve performance. PocketGen exhibits robustness to variations in hyperparameters, consistently producing competitive results.

Generating protein pockets for therapeutic small molecule ligands

Next, we illustrate the capacity of PocketGen to redesign the pockets of antibodies, enzymes, and biosensors for target ligand molecules, drawing upon previous research^3,10,16. Specifically, we consider the following molecules. Cortisol (HCY)⁶¹ is a primary stress hormone that elevates glucose levels in the bloodstream and serves as a biomarker for stress and other conditions. We redesign the pocket of a cortisol-specific antibody (PDB ID 8cby) to potentially aid in the development of immunoassays. Apixaban (APX)⁶² is an oral anticoagulant approved by the FDA in 2012 for patients with non-valvular atrial fibrillation to reduce the risk of stroke and blood clots⁶³. Apixaban targets Factor Xa (fXa) (PDB ID 2p16), a crucial enzyme in blood coagulation that transforms prothrombin into thrombin for clot formation. Redesigning the pocket of fXa could therefore have therapeutic significance. Fentanyl (7V7)⁶⁴ has become a widely abused drug contributing to the opioid crisis. Computational design of fentanyl-binding proteins (biosensors) can facilitate detection and neutralization of the toxin¹⁰. For example, Baker et al.¹⁰ developed a biosensor (PDB ID 5tzo) for detecting fentanyl in plants. In Figure 3, PLIP⁶⁵ is employed to describe the interactions between the designed protein pocket and ligands, comparing these predicted interactions to the original patterns.

Figure 3. Using PocketGen to design protein pockets for binding with important ligands.

a, b, c Illustrations of protein-ligand interaction analysis for three target molecules (HCY, APX, and 7V7, respectively). ‘PocketGen’ refers to the protein pocket designed by PocketGen, and ‘Original’ denotes the original protein-ligand structure. ‘HP’ indicates hydrophobic interactions, ‘HB’ signifies hydrogen bonds, and ‘π’ denotes the π-stacking/cation interactions. In the residue sequences, red ones denote the designed residues that differ from the original pocket. d, e, f The pocket binding affinity distributions of PocketGen and baseline methods for three target molecules (HCY, APX, and 7V7, respectively). We mark the Vina Score of the original pocket with the vertical dotted lines. For each method, we sample 100 pockets for each target ligand. The ratio of generated pockets by PocketGen with higher affinity than the corresponding reference pocket are 11%, 40%, and 45%, respectively.

The pockets produced by PocketGen replicate most non-bonded interactions observed in experimentally measured protein-ligand complexes (e.g., achieving a 13/15 match for HCY), and introduce additional physically plausible interaction patterns not present in the original complexes. For example, HCY, APX, and 7V7 form 2, 3, and 4 extra interactions, respectively. Specifically, for HCY, PocketGen maintains essential interaction patterns such as hydrophobic interactions (TRP47, PHE50, TYR59, and TYR104) and hydrogen bonds (TYR59), and introduces two new hydrogen bond-mediated interactions with the pocket. For protein pockets designed to bind APX and 7V7 ligands, while retaining key interaction patterns including hydrophobic interactions, hydrogen bonds, and $\pi -\pi$ stacking, PocketGen-generated molecules establish additional interactions (e.g., $\pi$-Cation interaction with LYS192 for APX and hydrogen bonds with ASN35 for 7V7), thus enhancing binding affinity with the target ligands. In conclusion, PocketGen demonstrates the capability to establish non-covalent interactions derived from protein-ligand structure data.

With the ability to establish favorable protein-ligand interactions, PocketGen generates high-affinity pockets for these drug ligands. In Figure 3 (d), (e), and (f), we further show the affinity distributions of the generated pockets by PocketGen and baseline methods. The ratio of generated pockets with higher affinity than the reference pocket is 11 %, 40%, and 45%, respectively, for PocketGen. On the contrary, the best runner-up method, RFDiffusion, only has 0%, 8 %, and 9%, respectively.

Protein stability is a critical factor in protein design, ensuring the designed protein can fold into and maintain its specific three-dimensional structure⁶⁶. Stability is typically quantified by the change in Gibbs free energy $\left(\mathrm{\Delta }\mathrm{\Delta }G\right)$ of folding between the redesigned protein and the wild-type (original) protein, where $\mathrm{\Delta }\mathrm{\Delta }G=\mathrm{\Delta }{G}_{\text{orig}}-\mathrm{\Delta }{G}_{\text{redesign}}$. A positive $\mathrm{\Delta }\mathrm{\Delta }G$ value indicates an increase in protein stability, while a negative value suggests a decrease. DDMut⁶⁷ is used to predict the change in stability for the three cases in Figure 3, with $\mathrm{\Delta }\mathrm{\Delta }G$ values of 0.09, 0.92, and 0.13, respectively. These results indicate that PocketGen can generate protein structures with enhanced stability, essential for functional activities such as binding with target ligand molecules.

Exploring interactions between atoms in protein and ligand molecules learned by PocketGen

Lastly, we analyze the attention matrices visualized in Fig. 4 to understand what PocketGen has learned, selecting a generated pocket for the ligand APX as a case study. In Figure 4(a), a 2D interaction plot is presented, drawn using Schrödinger Maestro software. To assess PocketGen’s recognition of meaningful protein-ligand interactions, we plot the heatmap of attention weights produced by the final layer in PocketGen’s neural architecture model. In Figure 4(b), two attention heads are chosen for illustration, with each row and column representing a protein residue or a ligand atom, respectively. The attention heatmaps are sparse, reflecting the use of sparse attention in PocketGen (refer to Methods for more details). We find that the attention heads display diverse patterns, focusing on different aspects. For instance, it is hypothesized that the first head emphasizes hydrogen bonds, assigning high weights between residue THR146, ASP220, and ligand atom 7. The second attention head appears to capture $\pi -\pi$ stacking and $\pi$-Cation interactions, specifically between residue TYR99 and ligand atoms 15, 21, 23, 25, 29, and 33; and residue LYS192 and ligand atoms 1, 14, 17, 19, and 20. These observations suggest that PocketGen, albeit trained via a data-driven approach, has acquired some biochemical insights to form beneficial interactions.

Figure 4. Attention maps in PocketGen capture interactions between atoms in protein and ligand molecules.

a, The 2D interaction plot of the designed pocket by PocketGen for APX. b, The heatmap of attention matrices between residues and ligand atoms from the last layer of PocketGen. We show two selected attention heads with notable attention patterns marked with red rectangles. We notice that each head emphasizes different interactions. For example, PocketGen recognizes the hydrogen bond interaction and assigns a strong attention weight between residue ① THR146, ② ASP220, and ligand atom 7 in the first head. The $\pi -\pi$ stacking and $\pi$-Cation interactions of ③ TYR99 and ④ LYS192 are well captured in the second head. The values are normalized by the maximum value $\left(v_{\mathrm{m}\mathrm{a}\mathrm{x}}\right)$ and the minimum value $\left(v_{\mathrm{m}\mathrm{i}\mathrm{n}}\right)$ in each heatmap (i.e., $v^{\prime }=\frac{v-v_{\mathrm{m}\mathrm{i}\mathrm{n}}}{v_{\mathrm{m}\mathrm{a}\mathrm{x}}-v_{\mathrm{m}\mathrm{i}\mathrm{n}}}$).

Discussion

Protein-ligand binding plays a critical role in enzyme catalysis, immune recognition, cellular signal transduction, gene expression control, and other biological processes. Recent developments include deep generative models designed to study protein-ligand binding, like Lingo3DMol⁶⁸ and ResGen⁶⁹, which generate de novo drug-like ligand molecules for fixed protein targets; NeuralPLexer⁴ can generate the structure of protein-ligand complexes given the protein sequence and ligand molecular graph. However, these models do not facilitate the de novo generation of protein pockets, the interfaces that bind with the ligand molecule for targeted ligand binding, critical in enzyme and biosensor engineering. We developed PocketGen, a deep generative method capable of generating both the residue sequence and the full atom structures of the protein pocket region for binding with the target ligand molecule. PocketGen includes two main modules: a bilevel graph transformer for structural encoding and updates, and a sequence refinement module that uses protein language models (pLMs) for sequence prediction. To achieve sequence-structure consistency and effectively leverage evolutionary knowledge from pLMs, a structural adapter is integrated into protein language models for sequence updates. This adapter employs cross-attention between sequence and structure features to promote information flow and ensure sequence-structure consistency. Extensive experiments across benchmarks and case studies involving therapeutic ligand molecules illustrate PocketGen’s ability to generate high-fidelity pocket structures with high binding affinity and favorable interactions with target ligands. Analysis of PocketGen’s performance across various settings reveals its proficiency in balancing diversity and affinity and generalizing across different pocket sizes. Additionally, PocketGen offers computational efficiency, significantly reducing runtime compared to traditional physics-based methods, making it feasible to sample large quantities of pocket candidates. PocketGen surpasses existing methods in generating high-affinity protein pockets for target ligand molecules in an efficient manner, finding important interactions between atoms on protein and ligand molecules, and attaining consistency in sequence and structure domains.

There are several extensions of PocketGen for future work. PocketGen could be expanded to design larger areas of the protein beyond the pocket area. While PocketGen has been evaluated on larger pocket designs, modifications will be required to enhance scalability and robustness for generating larger protein areas. Another fruitful future direction involves incorporating additional biochemical priors, such as subpockets⁷⁰ and interaction templates¹⁷, to improve generalizability and success rates. For instance, despite overall dissimilarity, two protein pockets might still bind the same fragment if they share similar subpockets⁷¹. Moreover, conducting wet lab experiments could provide empirical validation of PocketGen’s effectiveness. Approaches such as PocketGen have the potential to advance areas of machine learning and bioengineering and help with the design of small molecule binders and enzymes.

Methods

Overview of PocketGen

Unlike previous methods focusing on protein sequence or structure generation, we aim to co-design both residue types (sequences) and 3D structures of the protein pocket that can fit and bind with target ligand molecules. Inspired by previous works on structure-based drug design^69,70 and protein generation^32,33, we formulate pocket generation in PocketGen as a conditional generation problem that generates the sequences and structures of pocket conditioned on the protein scaffold (other parts of the protein except the pocket region) and the binding ligand. To be specific, let $𝒜={\mathit{a}}_1\cdots {\mathit{a}}_{N_s}$ denote the whole protein sequence of residues, where $N_s$ is the length of the sequence. The 3D structure of the protein can be described as a point cloud of protein atoms ${\left\{{\mathit{a}}_{i,j}\right\}}_{1\le i\le N_s,1\le j\le n_i}$ and let $\mathit{x}\left({\mathit{a}}_{i,j}\right)\in {\mathbb{R}}^3$ denote the 3D coordinate of protein atoms. $n_i$ is the number of atoms in a residue determined by the residue types. The first four atoms in any residue correspond to its backbone atoms $\left(C_{\alpha },N,C,O\right)$, and the rest are the side-chain atoms. The ligand molecule can also be represented as a 3D point cloud $ℳ={\left\{{\mathit{v}}_k\right\}}_{k=1}^{N_l}$ where ${\mathit{v}}_k$ denotes the atom feature. Let $\mathit{x}\left({\mathit{v}}_k\right)$ denotes the 3D coordinates of atom ${\mathit{v}}_k$. Our work defines the protein pocket as a set of residues in the protein closest to the binding ligand molecule: $ℬ={\mathit{b}}_1\cdots {\mathit{b}}_m$. The pocket $ℬ$ can thus be represented as an amino acid subsequence of a protein: $ℬ={\mathit{a}}_{e_1}\cdots {\mathit{a}}_{e_m}$ where $\mathit{e}=\left\{e_1,\cdots ,e_m\right\}$ is the index of the pocket residues in the whole protein. The index $\mathit{e}$ can be formally given as: $\mathit{e}=\{i\mid \underset{1\le j\le n_i,1\le k\le N_l}{\text{min}}{\Vert \mathit{x}\left({\mathit{a}}_{i,j}\right)-\mathit{x}\left({\mathit{v}}_k\right)\Vert }_2\le \delta \}$, where $\parallel \cdot {\parallel }_2$ is the $L_2$ distance norm and $\delta$ is the distance threshold. According to the distance range of pocket-ligand interactions⁴⁵, we set $\delta =3.5\mathrm{Å}$ in the default setting. With the above-defined notations, PocketGen aims to learn a conditional generative model formally defined as :

$$P(ℬ\mid 𝒜\backslash ℬ,ℳ),$$ (1)

where $𝒜\backslash ℬ$ denotes the other parts of the protein except the pocket region. We also adjust the structure ligand molecule $ℳ$ in PocketGen to encourage protein-ligand interactions and reduce steric clashes.

To effectively generate the structure and the sequence of the protein pocket $ℬ$, the equivariant bilevel graph transformer and the sequence refinement module with pretrained protein language models and adapters are proposed, which will be discussed in the following paragraphs. The illustrative workflow is depicted in Fig. 1.

Equivariant bilevel graph transformer

It is critical to model the complex interactions in the protein pocket-ligand complexes for pocket generation. However, the multi-granularity (e.g., atom-level and residue-level) and multi-aspect (intra-protein and protein-ligand) nature of interactions brings a lot of challenges. Inspired by recent works on hierarchical graph transformer⁷⁰ and generalist equivariant transformer⁷², we propose a novel equivariant bilevel graph transformer to well model the multi-granularity and multi-aspect interactions. Each residue or ligand is represented as a block (i.e., a set of atoms) for the conciseness of representation and ease of computation. Then the protein-ligand complex can be abstracted as a geometric graph of sets $𝒢=(𝒱,ℰ)$ where $𝒱=\left\{{\mathit{H}}_i,{\mathit{X}}_i\mid 1\le i\le B\right\}$ denotes the blocks and $ℰ=\left\{e_{ij}\mid 1\le i,j\le B\right\}$ include all the edges between blocks. We added self-loops to the edges to capture interactions within the block (e.g., the interactions between ligand atoms). Our model adaptively assigns different numbers of channels to ${\mathit{H}}_i$ and ${\mathit{X}}_i$ to accommodate different numbers of atoms in residues and ligands. For example, given a block with $n_i$ atoms, the corresponding block has ${\mathit{H}}_i\in {\mathbb{R}}^{n_i\times d_h}$ indicating the atom features ($d_h$ is the feature dimension size) and ${\mathit{X}}_i\in {\mathbb{R}}^{n_i\times 3}$ denoting the atom coordinates. Specifically, the $p$-th row of $H_i$ and ${\mathit{X}}_i$ corresponds to the $p$-th atom’s trainable feature (i.e., ${\mathit{H}}_i\left[p\right]$) and coordinates (i.e., ${\mathit{X}}_i\left[p\right]$) respectively. The trainable feature ${\mathit{H}}_i\left[p\right]$ is first initialized with the concatenation of atom type embedding, residue/ligand embeddings, and the atom positional embeddings. To build $ℰ$, we connect the $k$-nearest neighboring residues according to the pairwise $C_{\alpha }$ distances. To reflect the interactions between the protein pocket and ligand, we add edges between all the pocket residue and the ligand block. We continue by describing the modules in PocketGen’s equivariant bilevel graph transformer, bilevel attention module, and equivariant feed-forward networks.

Bilevel attention module.

Our model captures both atom-level and residue/ligand-level interactions with the bilevel attention module. Firstly, given two block $i$ and $j$ connected by an edge $e_{ij}$, we obtain the query, the key, and the value matrices with the following transformations:

$${\mathit{Q}}_i={\mathit{H}}_i{\mathit{W}}_Q,{\mathit{K}}_j={\mathit{H}}_j{\mathit{W}}_K,{\mathit{V}}_j={\mathit{H}}_j{\mathit{W}}_V,$$ (2)

Where ${\mathit{W}}_Q,{\mathit{W}}_K,{\mathit{W}}_V\in {\mathbb{R}}^{d_h\times d_r}$ are trainable parameters.

To calculate the atom-level attention across the $i$-th and $j$-th block, we denote ${\mathit{X}}_{ij}\in {\mathbb{R}}^{n_i\times n_j\times 3}$ and ${\mathit{D}}_{ij}\in {\mathbb{R}}^{n_i\times n_j}$ as the relative coordinates and distances between atom pairs in block $i$ and $j$, namely, ${\mathit{X}}_{ij}[p,q]={\mathit{X}}_i\left[p\right]-{\mathit{X}}_j\left[q\right],{\mathit{D}}_{ij}[p,q]={∥{\mathit{X}}_{ij}[p,q]∥}_2$.

Then we have:

$${\mathit{R}}_{ij}=\frac{1}{\sqrt{d_r}}\left({\mathit{Q}}_i{\mathit{K}}_j^{\mathrm{\top }}\right)+{\sigma }_D\left(\mathrm{R}\mathrm{B}\mathrm{F}\left({\mathit{D}}_{ij}\right)\right),$$ (3)

$${\mathit{\alpha }}_{ij}=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\mathit{R}}_{ij}\right),$$ (4)

where ${\sigma }_D(\cdot )$ is a Multi-Layer Perceptron (MLP) that adds distance bias to the attention calculation. RBF embeds the distance with radial basis functions. ${\mathit{\alpha }}_{ij}\in {\mathbb{R}}^{n_i\times n_j}$ is the atom-level attention matrix obtained by applying row-wise Softmax on ${\mathit{R}}_{ij}\in {\mathbb{R}}^{n_i\times n_j}$. To encourage sparsity in the attention matrix, we keep the top-$k^{\prime }$ elements of each row in ${\mathit{\alpha }}_{ij}$ and set the others as zeros.

The residue/ligand-level attention from the $j$-th block to the $i$-th block is calculated as:

$$r_{ij}=\frac{{\mathbf{1}}^{\mathrm{\top }}{\mathit{R}}_{ij}\mathbf{1}}{n_i{n}_j},$$ (5)

$${\beta }_{ij}=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(r_{ij}\right)}{\sum _{j\in 𝒩\left(i\right)} \mathrm{e}\mathrm{x}\mathrm{p}\left(r_{ij}\right)},$$ (6)

where $\mathbf{1}$ refers to the column vector with all elements set as ones and $𝒩\left(i\right)$ denotes the neighboring blocks of $i.r_{ij}$ sums up all values in ${\mathit{R}}_{ij}$ to represent the overall correlation between blocks $i$ and $j$. Subsequently, ${\beta }_{ij}$ denotes the attention across blocks at the block level.

We can update the representations and coordinates using the above atom-level and the residue/ligand-level attentions. PocketGen only updates the coordinates of the residues in the pocket and the ligand molecule. The other protein residues are fixed. Specifically, for the $p$-th atom in block $i$:

$$m_{ij,p}={\beta }_{ij}\left({\mathit{\alpha }}_{ij}\left[p\right]\odot {\varphi }_x\left({\mathit{Q}}_i\left[p\right]∥{\mathit{K}}_j∥\mathrm{R}\mathrm{B}\mathrm{F}\left({\mathit{D}}_{ij}\left[p\right]\right)\right)\right),$$ (7)

$${\mathit{H}}_i^{\text{'}}\left[p\right]={\mathit{H}}_i\left[p\right]+\sum _{j\in 𝒩\left(i\right)}{\beta }_{ij}{\varphi }_h\left({\alpha }_{ij}\left[p\right]\cdot {\mathit{V}}_j\right),$$ (8)

$${\mathit{X}}_i^{\prime }[p]={\mathit{X}}_i[p]+\left\{\begin{array}{ll}\sum _{j\in 𝒩\left(i\right)} m_{ij,p}\cdot {\mathit{X}}_{ij}[p],& \text{if}i\text{belongs}\text{to}\text{ligand}\text{or}\text{pocket}\text{residues}\\ 0,& \text{if}i\text{belongs}\text{to}\text{other}\text{protein}\text{residues}\end{array}\right.$$ (9)

where ${\varphi }_h$ and ${\varphi }_x$ are MLPs with concatenated representations as input (concatenation along the second dimension and ${\mathit{Q}}_i\left[p\right]$ is repeated along rows). $\odot$ computes the element-wise multiplication. ${\mathit{H}}_i^{\prime }$ and ${\mathit{X}}_i^{\prime }$ denote the updated representation and coordinate matrices, and we can verify that the dimension size of ${\mathit{H}}_i^{\prime }$ and ${\mathit{X}}_i^{\prime }$ remains the same regardless of the neighboring block size $n_j$. Furthermore, as the attention coefficients ${\mathit{\alpha }}_{ij}$ and ${\beta }_{ij}$ are invariant under E(3) transformations, the modification of ${\mathit{X}}_i^{\prime }$ adheres to E(3)-equivariance. Additionally, it’s noted that this update process is not affected by the permutation of atoms within each block.

Equivariant Feed-Forward Network.

We adjusted the Feed-Forward Networks (FFN) in the vanilla transformer⁷³ to update ${\mathit{H}}_i$ and ${\mathit{X}}_i$. Specifically, the representation and coordinates of atoms are updated to consider the feature/geometric centroids (means) of the block. The centroids are denoted as:

$${\mathit{h}}_c=\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{i}\mathrm{d}\left({\mathit{H}}_i\right),{\mathit{x}}_c=\text{centroid}\left({\mathit{X}}_i\right),$$ (10)

Then we obtain the relative coordinate $\mathrm{\Delta }{\mathit{x}}_p$ and the relative distance representation ${\mathit{r}}_p$ based on the L2 norm of $\mathrm{\Delta }{\mathit{x}}_p$:

$$\mathrm{\Delta }{\mathit{x}}_p={\mathit{X}}_i\left[p\right]-{\mathit{x}}_c,{\mathit{r}}_p=\mathrm{R}\mathrm{B}\mathrm{F}\left({∥\mathrm{\Delta }{\mathit{x}}_p∥}_2\right),$$ (11)

The representation and coordinates of atoms are updated with MLPs ${\sigma }_h$ and ${\sigma }_x$. The centroids are integrated to inform of the context of the block:

$${\mathit{H}}^{\prime }\left[p\right]=\mathit{H}\left[p\right]+{\sigma }_h\left({\mathit{H}}_i\left[p\right],{\mathit{h}}_c,{\mathit{r}}_p\right),$$ (12)

$${\mathit{X}}_i^{\prime }\left[p\right]={\mathit{X}}_i\left[p\right]+\mathrm{\Delta }{\mathit{x}}_p{\sigma }_x\left({\mathit{H}}_i\left[p\right],{\mathit{h}}_c,{\mathit{r}}_p\right).$$ (13)

To stabilize and accelerate training, layer normalization⁷⁴ is appended at each layer of the equivariant bilevel graph transformer to normalize $\mathit{H}$. The equivariant feed-forward network satisfies E(3)-equivariance. Thanks to the E(3)-equivariance of each module, the whole proposed bilevel graph transformer has the desirable property of E(3)-equivariance (detailed proof in the supplementary).

Sequence refinement with pretrained protein language models and adapters

Protein language models (pLMs), such as the ESM family of models^39,40, have learned extensive evolutionary knowledge from the vast array of natural protein sequences, demonstrating a strong ability to design protein sequences. In PocketGen, we propose to leverage pLMs to help refine the designed protein pocket sequences. To infuse the pLMs with structural information, we implant lightweight structural adapters inspired by previous works^75,76. In our default setting, only one structural adapter was placed after the last layer of pLM. Only the adapter layers are fine-tuned during training, and the other layers of PLMs are frozen to save computation costs. The structural adapter mainly has the following two parts.

Structure-sequence Cross Attention.

The structural representation of the $i$-th residue ${\mathit{h}}_i^{\text{struct}}$ is obtained by mean pooling of ${\mathit{H}}_i$ from the bilevel graph transformer. In the input to the pLMs, the pocket residue types to be designed are assigned with the mask, and we denote the $i$-th residue representation from pLMs as ${\mathit{h}}_i^{\text{seq}}$. In the structural adapter, we perform cross-attention between the structural representations ${\mathit{H}}^{\text{struct}}=\left\{{\mathit{h}}_1^{\text{struct}},{\mathit{h}}_2^{\text{struct}},\cdots ,{\mathit{h}}_{N_s}^{\text{struct}}\right\}$ and sequence representations ${\mathit{H}}^{\text{seq}}=\{{\mathit{h}}_1^{\text{seq}},{\mathit{h}}_2^{\text{seq}},\cdots ,{\mathit{h}}_{N_s}^{\text{seq}}\}$. The query, key, and value matrices are obtained as follows:

$$\mathit{Q}={\mathit{H}}^{\text{seq}}{\mathit{W}}_Q,\mathit{K}={\mathit{H}}^{\text{struct}}{\mathit{W}}_K,\mathit{V}={\mathit{H}}^{\text{struct}}{\mathit{W}}_V,$$ (14)

Where ${\mathit{W}}_Q,{\mathit{W}}_K,{\mathit{W}}_V\in {\mathbb{R}}^{d_h\times d_r}$ are trainable weight matrices. Rotary positional encoding⁷⁷ is applied to the representations, and we omit it in the equations for simplicity. The output of the cross attention is obtained as:

$$\mathrm{C}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(\mathit{Q},\mathit{K},\mathit{V})=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{\mathit{Q}{\mathit{K}}^{\mathrm{\top }}}{\sqrt{d_r}}\right)\mathit{V}.$$ (15)

Bottleneck FFN.

A bottleneck feedforward network (FFN) is appended after the cross-attention to impose non-linearity and abstract representations, inspired by previous works such as Houlsby et al.⁷⁵. The intermediate dimension of the bottleneck FFN is set to be half of the default representation dimension. Finally, the predicted pocket residue type ${\mathit{p}}_{\mathit{i}}$ is obtained using an MLP on the output residue representation.

Training protocol

Inspired by AlphaFold2³⁸, we use a recycling strategy for model training. Recycling facilitates the training of deeper networks without incurring extra memory costs by executing multiple forward passes and computing gradients solely for the final pass. The training loss of PocketGen is the weighted sum of the following three losses:

$${ℒ}_{\mathrm{seq}}=\frac{1}{T}\sum _t\sum _i{l}_{\mathrm{ce}}\left({\hat{\mathit{p}}}_i,{\mathit{p}}_i^t\right);$$ (16)

$${ℒ}_{\mathrm{coord}}=\frac{1}{T}\sum _t\left[\sum _i{l}_{\mathrm{huber}}({\hat{\mathit{X}}}_i,{\mathit{X}}_i^t)+\sum _j{l}_{\mathrm{huber}}\left(\hat{\mathit{x}}\left({\mathit{v}}_j\right),{\mathit{x}}^t\left({\mathit{v}}_j\right)\right)\right];$$ (17)

$${ℒ}_{\mathrm{struct}}=\frac{1}{T}\sum _t\left[\sum _{b\in ℬ}{l}_{\mathrm{huber}}(\hat{b},b^t)+\sum _{\theta \in \Theta }{l}_{\mathrm{huber}}\left(\cos \hat{\theta },\cos {\theta }^t\right)\right];$$ (18)

$$ℒ={ℒ}_{\mathrm{seq}}+{\lambda }_{\mathrm{coord}}{ℒ}_{\mathrm{coord}}+{\lambda }_{\mathrm{struct}}{ℒ}_{\mathrm{struct}},$$ (19)

where $T$ is the total refinement rounds. ${\stackrel{\mathit{ˆ}}{\mathit{p}}}_i,{\stackrel{\mathit{ˆ}}{\mathit{X}}}_i,\stackrel{\mathit{ˆ}}{\mathit{x}}\left({\mathit{v}}_j\right),\hat{b}$, and $\mathrm{c}\mathrm{o}\mathrm{s}\hat{\theta }$ are the ground-truth residue types, residue coordinates, and ligand coordinates, bond lengths, and bond/dihedral angles; ${\mathit{p}}_i^t,{\mathit{X}}_i^t,{\mathit{x}}^t\left({\mathit{v}}_j\right),b^t$, and $\mathrm{c}\mathrm{o}\mathrm{s}{\theta }^t$ are the predicted ones at the $t$-th round by PocketGen. The sequence loss ${ℒ}_{\text{seq}}$ is the cross-entropy loss for pocket residue type prediction; the coordinate loss ${ℒ}_{\text{coord}}$ uses huber loss⁷⁸ for the training stability; the structure loss ${ℒ}_{\text{struct}}$ is further added to supervised bond lengths and bond/dihedral angles for realistic local geometry. $ℬ$ and $\mathrm{\Theta }$ denote all the bonds and angles in the protein pocket (including side chains). ${\lambda }_{\text{coord}}$, and ${\lambda }_{\text{struct}}$ are hyperparameters balancing the three losses. We perform a grid search over {0.5,1.0,2.0,3.0} and choose these hyperparameters based on the validation performance to select the specific parameter values. In the default setting, we set ${\lambda }_{\text{coord}}$ to 1.0 and ${\lambda }_{\text{struct}}$ to 2.0.

Generation protocol

In the generation procedure, PocketGen initializes the sequence with uniform distributions over 20 amino acid types and the coordinates based on linear interpolations and extrapolations. Specifically, we initialize the residue coordinates with linear interpolations and extrapolations based on the nearest residues with known structures in the protein. Denote the sequence of residues as $𝒜={\mathit{a}}_1\cdots {\mathit{a}}_{N_s}$, where $N_s$ is the length of the sequence. Let $\mathit{x}\left({\mathit{a}}_{i,1}\right)\in {\mathbb{R}}^3$ denote the $C_{\alpha }$ coordinate of the $i$-th residue. We take the following strategies to determine the $C_{\alpha }$ coordinate of the $i$-th residue: (1) We use linear interpolation if there are residues with known coordinates at both sides of the $i$-th residue. Specifically, assume $p$ and $q$ are the indexes of the nearest residues with known coordinates at each side of the $i$-th residue $(p<i<q),$ we have: $\mathit{x}\left({\mathit{a}}_{i,1}\right)=\frac{1}{q-p}\left[(i-p)\mathit{x}\left({\mathit{a}}_{q,1}\right)+(q-i)\mathit{x}\left({\mathit{a}}_{p,1}\right)\right]$. (2) We conduct linear extrapolation if the $i$-th residue is at the ends of the chain, i.e., no residues with known structures at one side of the $i$-th residue. Specifically, let $p$ and $q$ denote the index of the nearest and the second nearest residue with known coordinates. The position of the $i$-th residue can be initialized as $\mathit{x}\left({\mathit{a}}_{i,1}\right)=\mathit{x}\left({\mathit{a}}_{p,1}\right)+\frac{i-p}{p-q}\left(\mathit{x}\left({\mathit{a}}_{p,1}\right)-\mathit{x}\left({\mathit{a}}_{q,1}\right)\right)$. Inspired by previous works^31,32, we initialize the other backbone atom coordinates according to their ideal local coordinates relative to the $C_{\alpha }$ coordinates. We initialize the side-chain atoms’ coordinates with the coordinate of their corresponding $C_{\alpha }$ added with Gaussian noise. As for the ligand molecular structure, we initialize it with the reference ligand structure from the dataset.

Since the number of pocket residue types and the number of side chain atoms are unknown at the beginning of generation, each pocket residue is assigned 14 atoms, the maximum number of atoms for residues. After rounds of refinement by PocketGen, the pocket residue types are predicted, and the full atom coordinates are determined by mapping the coordinates to the predicted residue types. For generation efficiency, we set the number of refinement rounds to 3.

Experimental Setting

Dataset.

We consider two widely used datasets for benchmark evaluation: CrossDocked dataset⁴² contains 22.5 million protein-molecule pairs generated through cross-docking. Following previous works^23,55,79, we filter out data points with binding pose RMSD greater than 1 °A, leading to a refined subset with around 180k data points. For data splitting, we use mmseqs2⁸⁰ to cluster data at 30% sequence identity, and randomly draw 100k protein-ligand structure pairs for training and 100 pairs from the remaining clusters for testing and validation, respectively; Binding MOAD dataset⁴³ contains around 41k experimentally determined protein-ligand complexes. Following previous work⁴¹, we keep pockets with valid and moderately ‘drug-like’ ligands with QED score ≥ 0.3. We further filter the dataset to discard molecules containing atom types $\notin \{C,N,O,S,B,Br,Cl,P,I,F\}$ as well as binding pockets with non-standard amino acids. Then, we randomly sample and split the filtered dataset based on the Enzyme Commission Number (EC Number)⁴⁴ to ensure different sets do not contain proteins from the same EC Number main class. Finally, we have 40k protein-ligand pairs for training, 100 pairs for validation, and 100 pairs for testing. For all the benchmark tasks in this paper, PocketGen and all the other baseline methods are trained with the same data split for a fair comparison. In real-world pocket generation and optimization case studies, the protein structures were downloaded from PDB⁸¹.

Implementation.

Our PocketGen model is trained with Adam⁸² optimizer for 5k iterations, where the learning rate is 0.0001, and the batch size is 64. We report the results corresponding to the checkpoint with the best validation loss. It takes around 48 hours to finish training on 1 Tesla A100 GPU from scratch. In PocketGen, the number of attention heads is set as 4; the hidden dimension d is set as 128; $k$ is set to 8 to connect the $k$-nearest neighboring residues to build $ℰ;k^{\prime }$ is set as 3 to encourage sparsity in the attention matrix. For all the benchmark tasks of pocket generation and optimization, PocketGen and all the other baseline methods are trained with the same data split for a fair comparison. We follow the implementation codes provided by the authors to obtain the results of baseline methods. Algorithm 1 and 2 in the supplementary show the pseudo-codes of the training and generation process of PocketGen.

Baselines.

PocketGen is compared with five state-of-the-art representative baseline methods. PocketOptimizer¹⁸ is a physics-based method that optimizes energies such as packing and binding-related energies for ligand-binding protein design. Following the suggestion of the paper, we set the backbone structures fixed. DEPACT¹⁷ is a template-matching method that follows a two-step strategy⁸³ for pocket design. It first searches the protein-ligand complexes in the database with similar ligand fragments. It then grafts the associated residues into the protein scaffold to output the complete protein structure with PACMatch¹⁷. Both the backbone and the sidechain structures are changed in DEPACT. RFDiffusion²⁵, FAIR²³ and dyMEAN²⁴ are deep-learning-based models that for protein generation. To model the influence of ligand molecules, we use a heuristic attractive-repulsive potential to encourage the formation of pockets with shape complementarity to a target molecule following the suggestions of RFDiffusion²⁵ and RFDiffusionAA¹⁶. The residue sequence is decided with ProteinMPNN, and the side-chain conformation is decided with Rosetta⁸⁴ side-chain packing. Our paper restricts our comparisons to RFDiffusion as the source code for RFDiffusionAA is not publicly available. FAIR²³ was specially designed for full-atom protein pocket design via iterative refinement. dyMEAN²⁴ was originally proposed for full atom antibody design, and we adapted it to our pocket design task with proper modifications. More introduction of the baselines are included in Supplementary. The setting of the key hyperparameters of the baseline methods are summarized in Table. S4.

Data availability

The training and test data of this study are available at Zenodo (https://zenodo.org/records/10125312). The project website for PocketGen is at https://zitniklab.hms.harvard.edu/projects/PocketGen.