HighFold-MeD: a Rosetta distillation model to accelerate structure prediction of cyclic peptides with backbone N-methylation and d-amino acids

Abstract

Abstract

Cyclic peptides with backbone N-methylated amino acids (BNMeAAs) and D-amino acids (D-AAs) have gained increasing attention for their stability, membrane permeability, and other therapeutic potentials. Currently, Rosetta simple_cycpep_predict (SCP) can predict their structures through energy-based calculations, but this approach is computationally intensive and time-consuming. Moreover, the available crystal structures of such cyclic peptides remain highly limited, hindering the development of data-driven structure prediction models. To address these challenges, we propose HighFold-MeD, a deep learning-based framework that distills knowledge from Rosetta SCP by fine-tuning the AlphaFold model. First, a cyclic peptide structure dataset is constructed using Rosetta SCP by sampling massive conformations for cyclic peptides with BNMeAAs and D-AAs and evaluating their energy scores. The AlphaFold model is then fine-tuned to incorporate the extended 56 BNMeAAs and D-AAs. Besides, a relative position cyclic matrix is introduced to explicitly model head-to-tail cyclization. Finally, a force field is employed to minimize steric clashes in the predicted structures. Empirical experiments demonstrate that HighFold-MeD achieves accuracy comparable to that of Rosetta based on the sampled datasets by the SCP module of Rosetta, with the key parameters that nstruct = 500 and cyclic_peptide: genkic_closure_attempts = 1000, while accelerating structure prediction by 50-fold, thereby significantly expediting the development of cyclic peptide-based therapeutics.

Scientific contribution

We propose HighFold-MeD, which provides a rapid and relatively accurate approach for predicting the structures of cyclic peptides containing backbone N-methylated amino acids and D-amino acids—key building blocks in peptide drug design. By distilling the knowledge of Rosetta SCP under specific parameters into a fine-tuned AlphaFold framework, our method achieves a 50-fold acceleration while maintaining relatively high accuracy, thereby enabling large-scale cyclic peptide drug design.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13321-025-01111-3.

Introduction

Peptides have increasingly garnered attention in drug discovery owing to their unique properties, including high binding affinity and specificity, low toxicity, and various therapeutic potentials [1–5]. However, research and development of peptide drugs are usually limited to the poor metabolic stability and membrane permeability due to their large molecular weight, high polar surface area, and some other chemical properties [6–9]. Compared to linear peptides, cyclic peptides exhibit greater resistance to hydrolysis and demonstrate enhanced membrane permeability owing to their unique cyclic structure. It is estimated that there are over 40 cyclic peptides or their derivatives that have been used as therapeutics [10–13]. To further improve the pharmaceutical properties of cyclic peptides, the most prevalent approach is to introduce various non-canonical amino acid modifications [14], such as backbone N-methylation and D-AAs. Studies show that backbone N-methylation and D-AAs can significantly improve metabolic stability, membrane permeability, and some other conformational features or properties of amide bonds [15–19].

According to the Quantitative Structure–Activity Relationship (QSAR) theory, accurate structure prediction of cyclic peptides plays a crucial role during the practical cyclic peptide-based drug discovery [20, 21]. With the rapid development of artificial intelligence, especially deep learning technologies [22], many significant progresses have been made in the field of protein structure prediction [23]. Models such as AlphaFold [24, 25], RosettaFold [26], and ESMFold [27] have greatly improved the accuracy and efficiency of protein and peptide structure prediction. However, all these models are limited to canonical amino acids. Recently, models like RoseTTAFold All-Atom [28] have further expanded the scope of deep learning-based structural predictions to complexes of proteins, nucleic acids, small molecules, ions, and modified residues. Building on these milestone achievements, several models for cyclic peptide structure prediction have emerged, such as AfCycDesign [29] and HighFold [30], which successfully predict cyclic peptides and complexes formed by canonical amino acids with head-to-tail and disulfide bridge constraints. Subsequently, the emergence of models such as RINGER [31], NCPepFold [32], HighFold2 [33], AlphaFold3 [34], Cyclic Boltz-1 [35], Boltz-2 [36], and other related frameworks has significantly advanced these predictive tasks into the realm of cyclic peptides that incorporate non-canonical amino acids.

However, HighFold2 and NCPepFold are still limited to cyclic peptides with non-canonical amid acid modifications on side chains, which are incapable of handling cyclic peptides with BNMeAAs and D-AAs. AlphaFold3 and Boltz-2 exhibit limited accuracy in predicting cyclic peptides containing BNMeAAs and D-AAs, and AlphaFold3’s predictions regarding the configurations of D-AAs are frequently inaccurate [37]. On the other side, the SCP module of Rosetta [38, 39] can predict cyclic peptides with BNMeAAs and D-AAs by sampling massive conformations. Although Rosetta SCP can yield relatively accurate predictions, the process is notably time-consuming. Achieving reliable prediction precision generally necessitates thousands of structural sampling operations on average, which significantly constrains the practicality of conducting large-scale predictions.

To address these challenges, we propose the HighFold-MeD model (Fig. 1a), which is a Rosetta distillation model [40] by fine-tuning the AlphaFold framework to dramatically accelerate the structure prediction of cyclic peptides with backbone N-methylation and D-AAs with high accuracy. As shown in Fig. 1b, a structural dataset of cyclic peptides with backbone N-methylation and D-AAs is constructed using the SCP module of Rosetta according to the additional dictionary of 56 BNMeAAs and D-AAs (Fig. 1c, d). During fine-tuning the AlphaFold framework, a relative position cyclic matrix is constructed and fed into the feature embedding module to capture the cyclization information (Fig. 1e). Additionally, to eliminate potential spatial conflicts in the predicted structures and minimize energy, a force field is introduced to adapt to BNMeAAs and D-AAs (Fig. 1f). HighFold-MeD demonstrates superior accuracy in capturing the spatial structural features of cyclic peptides containing BNMeAAs and D-AAs, when compared to existing deep learning-based structure prediction methods. Under resource-constrained conditions, utilizing specific Rosetta SCP parameters, it achieves predictive accuracy comparable to that of Rosetta SCP, while significantly reducing the overall time required for structure prediction.

Fig. 1.

Overview of HighFold-MeD. a Training process of HighFold-MeD. b Workflow of dataset generation and post-processing using Rosetta SCP. xc Definition of rigid groups in BNMeAAs and D-AAs. d Classification of rigid groups in BNMeAAs and D-AAs. e Modification of relative position encoding in the prediction model to enable the prediction of cyclic peptide monomers containing BNMeAAs and D-AAs. f Relaxation of 3D structures containing BNMeAAs and D-AAs to eliminate spatial clashes

Methods

Experiment settings

The HighFold-MeD, AlphaFold3, Boltz-2, and Rosetta with version 3.31 are running on Ubuntu 20.04.4 LTS. The hardware configuration comprises an Intel i9-12900KF CPU featuring 24 cores, an RTX 4090 GPU with 24 GB of memory, a substantial 192 GB of RAM, and a robust 12 TB hard drive.

Datasets

In this study, we utilize the Rosetta SCP to generate the model training and validation datasets, as shown in Fig. 1b. First, we construct a random sequence generation script: generating a list of 39 amino acids without methylation and a list of 37 amino acids with methylation. Amino acids are then randomly selected from these lists to obtain random sequences. We conduct experiments under different sampling size settings, and the results are provided in Additional File 1: Table S1. As the sampling size increases, the prediction accuracy of Rosetta SCP gradually improves. To balance computational resource constraints while ensuring the rationality and diversity of the generated cyclic peptide structures, we set the key parameters in the Rosetta SCP simulation script to nstruct = 500 and cyclic_peptide: genkic_closure_attempts = 1000. It should be noted that, when computational resources allow for higher sampling rates, Rosetta SCP may still achieve higher accuracy for individual targets. Following the structure prediction, we select the lowest-energy conformations for model distillation. Since Rosetta SCP assigns the same names to BNMeAAs and their unmethylated precursors, we modify the obtained PDB files to ensure consistency with the BNMeAAs and D-AAs dictionary in our model. Ultimately, from the 3,300 generated cyclic peptide structures, a total of 2,500 were randomly selected to comprise the training set, while 250 were designated as the validation set. Among the 35 cyclic peptide X-ray crystal structures provided by Baker [41], two structures containing AIB residues were excluded from our study, as it does not involve AIB amino acids. Consequently, the final test set comprises 33 cyclic peptide X-ray crystal structures. To guarantee the model’s generalization capability, we utilize the position-based cyclic Tanimoto coefficient to evaluate the similarity between the 33 cyclic peptides in the test set and the previously mentioned 2,750 sequences. For any two peptide sequences, S1 and S2, the two sequences are rotated through all possible circular alignments to correspond with various starting points on the ring, and the Tanimoto coefficient can be calculated as follows:

$$T=\underset{1\le i\le \left|S_1\right|,1\le j\le \left|S_2\right|}{max}\frac{P_{\mathit{ij}}}{\left|S_1\left|+\right|S_2\right|-P_{\mathit{ij}}}$$

where |∙| denotes the length of the peptide sequence, and P_ij represents the number of identical amino acids at the respective positions i and j. The overall average similarity across all sequence pairs is calculated to be 0.288, indicating that the test set exhibits a generally low level of overlap at the sequence level with the dataset.

Model architecture

To predict cyclic peptides containing BNMeAAs and D-AAs, we extended AlphaFold by incorporating a dictionary of BNMeAAs and D-AAs, adding a cyclic peptide matrix, and modifying the loss function. Thus, HighFold-MeD is proposed by fine-tuning AlphaFold to distill the Rosetta SCP model for structure prediction of cyclic peptides with BNMeAAs and D-AAs.

Addition of a dictionary of BNMeAAs and D-AAs

The structural module of AlphaFold uses single and pairwise representations generated by the Evoformer module to predict the torsion angles of amino acid residues. These predicted torsion angles, along with predefined rigid groups and atomic coordinates, help the model accurately construct atomic positions in three-dimensional space, simplifying structural prediction complexity. To extend the model for BNMeAAs and D-AAs, we have expanded AlphaFold’s amino acid dictionary with the detailed definitions of the rigid groups of these additional residues and initialized their atomic coordinates.

Specifically, 19 BNMeAAs are obtained by methylating the nitrogen atom of canonical amino acids (excluding PRO);19 D-AAs are constructed by generating enantiomers of canonical amino acids (excluding GLY); And 18 D-BNMeAAs are obtained by methylating the main-chain nitrogen of D-AAs (excluding D-GLY). Ultimately, we add 56 BNMeAAs and D-AAs, enabling the model to handle these complex molecules. The names and corresponding codes for the 56 amino acids are presented in Additional File 1: Table S2.

Atoms in amino acids are classified into five rigid groups based on their dependence on specific torsion angles (Additional File 1: Table S3). The backbone rigid group (bb framework), which consists of α-C, β-C, C, and N atoms, serves as the structural core. The φ group, which is initially used to compute the coordinates of the hydrogen atom on the amine group, has been repurposed to determine the position of the CN methyl group due to the substitution of hydrogen in HighFold-MeD. The ω group, which is associated with the hydrogen atom on α-C, is excluded, as hydrogen atoms cannot be predicted by the model. The ψ rigid group, which includes the oxygen atom of the carboxyl group, is distinct from the χ rigid group, which comprises all side-chain atoms (Fig. 1c). The initial coordinates of these rigid groups, which are derived from the structure predicted by Rosetta SCP, provide a foundation for further structural refinement.

As illustrated in Fig. 1d, the orientation of the bb framework, which is centered on α-C, positions C along the positive x-axis while constraining N to the xy-plane, thereby allowing the coordinates of β-C to be determined. The φ group, which is referenced to N as the origin, is defined by placing α-C along the negative x-axis and constraining the adjacent C atom to the xy-plane, thereby determining the position of CN. The ψ rigid group, which is initialized with respect to C, is defined by positioning α-C along the negative x-axis and constraining the nitrogen atom to the xy-plane, thereby determining the oxygen atom’s position. The side-chain χ rigid group, which is subdivided into four subgroups based on distinct torsion angles, accommodates amino acids with more than four torsion angles by neglecting those with negligible influence. Within each subgroup, the third atom is set as the origin, the second atom is aligned along the negative x-axis, and the first atom is positioned within the xy-plane. The relative position of the fourth atom is rotated into the xy-plane using the rotation matrix R_x, as described below:

$$R_x=\left[\begin{array}{ccc}1& 0& 0\\ 0& cos\theta & -sin\theta \\ 0& sin\theta & cos\theta \end{array}\right],Coordinat{e}_{\mathit{init}}=R_x·Coordinat{e}_{\mathit{relative}},$$

where θ represents the rotation angle around the x-axis to align the fourth atom with the xy-plane.

Introduction of relative position cyclic matrix

To accommodate the unique structural characteristics of cyclic peptides, we incorporated a cyclic matrix into the AlphaFold model, inspired by AfCycDesign [29]. This modification enables the model to explicitly capture the cyclic constraints of cyclic peptides, enhancing its ability to predict such structures. In the original AlphaFold model, relative positional encoding defines sequence distances between amino acids. For a linear peptide, the sequence distance between two neighboring residues is 1, while the distance between N-terminal and C-terminal residues is the sequence length—1. For instance, in the linear peptide sequence ACDEFGHIJK, the sequence distances between residue A and CDEFGHIJK are 1, 2, 3, 4, 5, 6, 7, 8, and 9. However, this encoding fails to capture the cyclic nature of peptides, leading to suboptimal performance. To address this issue, we design a custom N × N cyclic offset matrix that introduces cyclic information into the relative positional encoding. In this matrix, the sequence distance between the first and last residues of a cyclic peptide is redefined as 1, simulating the peptide’s closure. For instance, after applying this encoding, the sequence distances between residue A and CDEFGHIJK in a cyclic peptide are redefined as 1, 2, 3, 4, -5, -4, -3, -2, and -1, with negative signs indicating opposite directionality (Fig. 1e). This encoding correctly represents the relationship between terminal residues and comprehensively captures the cyclic nature of peptides. The cyclic offset matrix undergoes one-hot encoding and linear projection before being embedded into AlphaFold’s Evoformer module as part of the pairwise features. This adjustment significantly improves the model’s ability to capture global cyclic peptide characteristics.

Loss function

AlphaFold is trained in an end-to-end manner, with gradients derived from the Frame Aligned Point Error (FAPE) [24] loss and several auxiliary losses from the Structure Module, including the distogram prediction loss, the masked MSA prediction loss, and the model confidence loss. Through ablation studies on these different losses, we opt to use only the FAPE loss in our final model.

Relaxation

BNMeAAs and D-AAs force field

Force field parameters for BNMeAAs and D-AAs are developed following the Amber protocol (https://carlosramosg.com/amber-custom-residue-parameterization). Initially, each BNMeAA and D-AA is built in GaussView6 using the Ace-XXX-NMe capping approach (Step 1 of Fig. 2) [42]. Geometry optimization is performed in the gas phase at the B3LYP/6-31G(d) level, followed by electrostatic potential (ESP) calculations via single-point Hartree–Fock (HF) computations at the 6-31G(d) basis set (Step 2 of Fig. 2) [43], aligning with the standard Amber methodology for deriving partial atomic charges. The ESP is extracted from Gaussian output files using the espgen module in AmberTools [44], and atomic partial charges are fitted using the restrained electrostatic potential (RESP) procedure, with charges of the ACE and NME capping groups constrained to match standard Amber force field values (Step 3 of Fig. 2). Finally, capping groups are removed using the prepgen module in AmberTools, and topology files (.prepin and.frcmod) are generated based on the generation of General Amber Force Field (GAFF), ensuring atom naming consistency with corresponding PDB files (Step 4 of Fig. 2).

Fig. 2.

Preparation of charges and parameters for BNMeAAs and D-AAs

Energy minimization

The predicted peptide structures are loaded into the tleap module of AmberTools, where the Amber ff14SB force field, GAFF, and the newly developed parameters specific to BNMeAAs and D-AAs are integrated to generate the topology and coordinate files required for energy minimization [45].

Energy minimization is performed using OpenMM [46] (version 7.7.0) to relax the predicted structures while preserving their initial conformations as closely as possible. The simulation system is constructed using the AmberPrmtopFile and AmberInpcrdFile classes in OpenMM, with nonbonded interactions set to NoCutoff and all bonds involving hydrogen atoms constrained via HBonds. To prevent significant structural deviations during minimization, harmonic positional restraints are applied to backbone atoms. The minimization process continues until convergence, with an energy tolerance of 2.39 kcal/mol.

Metrics

RMSD

In this study, we use α-C RMSD, backbone RMSD, and all-atom RMSD as the evaluation metrics to quantify the deviation of the model-predicted structures from the reference structures. The RMSD calculation is conducted in three approaches and formulated as follows:

$$RMSD=\sqrt{\frac{1}{N}\sum _{i=1}^N{({\mathbf{r}}_i^{\text{pred}}-{\mathbf{r}}_i^{\text{ref}})}^2}$$

where N denotes the number of all atoms, backbone atoms (C, CA, N, O, and CN), and α-C atoms for the three approaches, respectively.

Structure assessments

Independent structural assessments are conducted using the MolProbity online server (https://molprobity.biochem.duke.edu/) [45]. It evaluates the stereochemical quality and accuracy of macromolecular structures by analyzing key parameters, including atomic contacts, molecular geometry, and backbone torsion angles. MolProbity is widely used in structural biology and provides quantitative and reliable evaluations of biomacromolecule structures.

Results and discussion

A comprehensive performance comparison is conducted among the HighFold-MeD, Rosetta SCP, AlphaFold3, and Boltz-2 in predicting cyclic peptides with BNMeAAs and D-AAs. Subsequently, a systematic analysis is performed to examine the effects of various factors on model optimization. The impact of fine-tuning is evaluated by comparing the model’s performance before and after fine-tuning to confirm its effectiveness. The influence of incorporating a cyclic peptide coding matrix is assessed to verify its positive contribution. The effects of dataset size and the choice of loss function on model performance are analyzed to determine the optimal experimental design. Finally, the effect of force field optimization is examined. Before these comparative analyses, the results of the test set are directly evaluated with the average α-C RMSD of 1.368 Å, indicating a strong consistency between the predictions and the actual structures. Furthermore, in comparison to the Rosetta SCP, the average prediction time has been significantly reduced from 2424.5 s to just 48.6 s. This represents an impressive, nearly 50-fold acceleration, underscoring the efficiency of the HighFold-MeD model.

Performance comparison among highfold-MeD, Rosetta SCP, AlphaFold3, and Boltz-2 in predicting cyclic peptides with BNMeAAs and D-AAs

To ensure fairness in the comparison, Rosetta SCP was executed using the same parameters that were employed during dataset generation. Specifically, nstruct was set to 500, and cyclic_peptide:genkic_closure_attempts was configured to 1000 when included in the analysis. It is important to note that the parameters chosen were established based on a balance between computational time and available resources. Consequently, the results presented herein do not reflect the optimal accuracy of Rosetta SCP.

Figure 3a presents a comparative analysis of the performance among HighFold-MeD, Rosetta SCP, AlphaFold3, and Boltz-2 in predicting cyclic peptides that incorporate BNMeAAs and D-AAs. In terms of structural accuracy, HighFold-MeD demonstrates significantly lower α-C RMSD (1.368 Å), backbone RMSD (1.612 Å), and all-atom RMSD (2.681 Å) compared to AlphaFold3 (2.009 Å, 1.934 Å, and 3.246 Å) as well as Boltz-2 (2.068 Å, 2.097 Å, and 3.138 Å). While Rosetta SCP achieves the lowest RMSDs across all three metrics (1.195 Å, 1.262 Å, and 2.345 Å), HighFold-MeD provides comparable accuracy while being markedly faster; this makes it more suitable for large-scale predictions without compromising competitive accuracy. Figures 3b–d depict the distributions of root mean square deviation (RMSD). Notably, for 12 structures, including D9.16, D8.26, D10.1, D8.17, D8.19, D9.24, D11.25, D8.25, D8.6, D8.5, D8.21, and D8.15, more accurate α-C predictions are achieved compared to Rosetta SCP. It is important to highlight that, among the 33 test cases, HighFold-MeD successfully predicted structures with an α-C RMSD below 1 Å when compared to the native structure in 14 instances. Besides, several structures are visualized to further illustrate performance differences. Figures 3e–g present examples where HighFold-MeD consistently outperforms Rosetta SCP, achieving lower RMSDs in all evaluated metrics, including α-C, backbone, and all-atom predictions (left: HighFold-MeD; right: Rosetta SCP; same applies to Fig. 3f–h). Notably, for D8.26, the α-C RMSD improves from 2.073 to 0.990 Å. Additionally, Figs. 3h–j highlight cases where HighFold-MeD demonstrates superior accuracy in predicting α-C atoms.

Fig. 3.

Comparison and visualization of model performance. a Comparison of RMSD, prediction accuracy of D-AAs configuration, and computational time among HighFold-MeD, Rosetta SCP, AlphaFold3, and Boltz-2 in Predicting Cyclic Peptides with BNMeAAs and D-AAs. b–d RMSD distribution of α-C, backbone, and all-atom measurements for 4 methods. e–j Structural alignment of selected cyclic peptides, where the native structures are shown in gray, HighFold-MeD predictions on the left, and Rosetta SCP predictions on the right. The displayed examples include e D8.26, f D9.24, g D11.25, h D8.17, i D8.5, and j D8.6, with their corresponding RMSD provided. k–p The displayed examples include k D8.26, l D9.24, m D11.25, n D8.17, o D8.5, and p D8.6, showing their respective stick models. The green backbone represents the true structure, while the blue backbone represents the predicted structure

In the prediction of the configurations of D-AAs, both HighFold-MeD and Rosetta SCP achieve a remarkable 100% accuracy, whereas AlphaFold3 (50.3%) and Boltz-2 (75.9%) demonstrate significantly lower performance. Furthermore, in terms of computational efficiency, HighFold-MeD demonstrates an average execution time of merely 48.6 s. This performance renders it nearly 50 times faster than Rosetta SCP (2424.5 s) and surpasses both AlphaFold3 (80.1 s) and Boltz-2 (60.3 s) in speed as well.

Overall, these findings indicate that HighFold-MeD strikes an optimal balance between prediction accuracy, chirality recognition, and computational time when predicting cyclic peptides containing BNMeAAs and D-AAs. So it can be considered as a more practical alternative compared to traditional methods such as Rosetta SCP, as well as contemporary deep learning models like AlphaFold3 and Boltz-2. Additionally, it is observed that the HighFold-MeD model classified all peptide bonds in the test set as trans and failed to correctly predict any cis peptide bonds. Further details are provided in Additional File 1: Table S4. Figure 3K–P illustrate the stick representations of D8.26, D9.24, D11.25, D8.17, D8.5, and D8.6 for direct observation of their predicted trans/cis states.

Comparison before and after fine-tuning

Figure 4a, b shows the results of the RMSD comparison before and after fine-tuning. The fine-tuned model performs best with an average α-C RMSD of 1.368 Å, an average backbone RMSD of 1.612 Å, and an average all-atom RMSD of 2.681 Å. In contrast, the model without fine-tuning has an average α-C RMSD of 2.291 Å, an average backbone RMSD of 2.299 Å, and an average all-atom RMSD of 3.747 Å. Thus, the RMSDs are improved by 0.923, 0.687, and 1.066 Å, respectively, through the model fine-tuning. 30 structures are predicted more accurately after fine-tuning compared to those before fine-tuning in the 33 test sets, demonstrating a huge improvement in the performance of the model fine-tuning. As shown in Fig. 4c–h, several fine-tuned predicted structures achieve an α-C RMSD of less than 1 Å, including D7.6 (0.438 Å) and D8.1 (0.400 Å), demonstrating improved structural accuracy compared to the model without fine-tuning. Figure 4i–n present the stick representations of i D7.6, j D7.8, k D8.1, l D8.9, m D8.14, and n D9.8, allowing direct observation of their predicted trans/cis states.

Fig. 4.

Comparative analysis and visualization of performance before and after fine-tuning. a Comparison of RMSD between the model with fine-tuning and the model without fine-tuning. b RMSD distributions for α-C, backbone, and all-atom measurements, highlighting the enhancement in structural accuracy achieved through fine-tuning with the cyclic peptide matrix. c–h Structural comparisons of selected cyclic peptides, where native structures are shown in gray, predictions generated by the model with fine-tuning are displayed on the left, and predictions from the model without fine-tuning are shown on the right. The selected examples include c D7.6, d D7.8, e D8.1, f D8.9, g D8.14, and h D9.8, with their respective RMSD values presented below each structure. i–n The displayed examples include i D8.26, j D9.24, k D11.25, l D8.17, m D8.5, and n D8.6, showing their respective stick models. The green backbone represents the true structure, while the blue backbone represents the predicted structure

Analysis of the role of the relative position cyclic matrix

When testing cyclic peptides, the initial AlphaFold model generates linear peptide structures, making it unsuitable for generalizing to cyclic peptides with position offset with non-canonical modifications. This limitation is reflected in the high average α-C RMSD of 5.736 Å across the entire test set. However, the incorporation of the cyclic peptide coding matrix reduces the α-C RMSD to 2.291 Å, demonstrating its significant impact. Figure 5a, b presents the complete RMSD distribution.

Fig. 5.

Analysis and visualization of the role of the relative position cyclic matrix. a Comparison of RMSD results with and without matrix testing. b Distribution of RMSD with and without matrix testing. c Comparison of RMSD results with and without matrix fine-tuning. d Distribution of RMSD with and without matrix fine-tuning

During the fine-tuning phase, models with and without the cyclic peptide coding matrix are trained using the same dataset, and their performance is evaluated on the test set. As shown in Fig. 5c and d, the α-C RMSD of the model fine-tuned with the cyclic peptide coding matrix is 1.368 Å, whereas the model fine-tuned without it reaches 1.386 Å. The RMSD for the model with cyclic peptide matrix fine-tuning is slightly lower than that of the model without the matrix fine-tuning: the RMSD for the backbone is 1.612 Å compared to 1.619 Å, and the RMSD for all atoms is 2.681 Å compared to 2.683 Å.

Impact of dataset size on model performance

To investigate the impact of dataset size on model performance, the prediction accuracy of the model is evaluated across various training dataset scales, using α-C RMSD as the performance metric. Figure 6a illustrates the specific RMSD. Figure 6b illustrates the relationship between training dataset size and α-C RMSD, with the horizontal axis representing dataset sizes (100, 500, 1000, 2500, 3000) and the vertical axis representing α-C RMSD. The specific data points are as follows: at a dataset size of 100, the α-C RMSD is 1.937 Å; At 500, it is 1.801 Å; At 1000, it is 1.524 Å; At 2500, it reaches 1.368 Å; And at 3000, it is 1.376 Å.

Fig. 6.

Presents the model performance under different dataset sizes. a Displays the detailed α-C RMSD under different dataset sizes. b Changes in α-C RMSD in different dataset sizes

As shown in Fig. 6b, α-C RMSD exhibits a nonlinear decreasing trend with increasing dataset size, indicating a progressive improvement in model performance. Specifically, when the dataset size increases from 100 to 500, α-C RMSD decreases from 1.937 to 1.801 Å, a reduction of 0.136 Å. This suggests that the increase in data volume contributes to some improvement in model performance. However, due to the limited data size, the model may not yet fully capture the complex features of cyclic peptide structures, leading to relatively high prediction errors. As the dataset size further increases to 1000, α-C RMSD drops to 1.524 Å, a more significant reduction of 0.277 Å, demonstrating that the additional data enables the model to better capture structural features. When the dataset size increases from 1000 to 2500, α-C RMSD further decreases to 1.368 Å, a reduction of 0.156 Å, indicating a continued improvement in model performance and reaching its optimal performance at this point. Finally, when the dataset size reaches 3000, the α-C RMSD exhibits a slight increase. This observation suggests that incorporating an excessive amount of data may introduce noise or lead to overfitting, which could marginally compromise the model’s performance. This suggests that at a dataset size of 2500 samples, the model can adequately learn the diversity of cyclic peptide structures while avoiding the risk of overfitting. Overall, α-C RMSD decreases from 1.937 Å (Dataset size = 100) to 1.368 Å (Dataset size = 2500), a total reduction of 0.569 Å, demonstrating that the model’s prediction accuracy significantly improves as the dataset size increases, whereas further increasing the dataset size may lead to overfitting.

Performance comparison of loss functions in fine-tuning for cyclic structures containing BNMeAAs and D-AAs

A series of ablation experiments is conducted for the selection of different loss functions, as summarized in Table 1. This table demonstrates the contribution of various loss components to structural accuracy by reporting average α-C RMSD, backbone RMSD, and all-atom RMSD based on evaluations performed on a set of 33 cyclic peptides under diverse training conditions, and ultimately, the FAPE loss can be the optimal choice when building the HighFold-MeD model.

Table 1.

The performance effects of different loss functions on HighFold-MeD

The “FAPE” setting employs only the Frame Aligned Point Error loss, whereas the “Except FAPE” condition retains all other losses except FAPE. The term “Combined Loss” denotes the integration of FAPE and all auxiliary losses, which corresponds to the comprehensive loss function utilized in AlphaFold. Throughout all experiments, we maintain consistent weights for each loss term in accordance with those reported in the AlphaFold paper. The results indicate that FAPE alone achieves the lowest RMSD values across all three metrics (α-C RMSD: 1.368 Å; backbone RMSD: 1.612 Å; all-atom RMSD: 2.681 Å), thereby confirming its pivotal role in enhancing structural accuracy. In contrast, when FAPE is excluded (as seen in Except FAPE), there is a significant increase in RMSD values (e.g., α-C RMSD rises from 1.368 to 1.897 Å), underscoring the essential contribution of FAPE to maintaining precise backbone geometry.

When combined with auxiliary losses—including Sidechain FAPE, Chi loss, Confidence loss, Distogram, and Masked MSA—the resulting RMSDs are slightly higher than those achieved using only FAPE but remain considerably lower than those observed under the Except FAPE condition. This suggests that while these auxiliary losses provide complementary constraints, it is still primarily FAPE that drives accurate placement of both backbone and side chains.

In summary, based on our ablation experiments, we conclude that FAPE represents the most suitable loss function for predicting cyclic peptides containing BNMeAAs and D-AAs.

Relaxation of structures containing BNMeAAs and D-AAs

AlphaFold is a groundbreaking deep learning model capable of accurately predicting three-dimensional structures of biomacromolecules. For the initially generated structures, AlphaFold typically employs an Amber force field-based relaxation process (Amber-relax) to perform energy minimization, enhancing the stereochemical quality of the predicted structures. This relaxation step eliminates atomic clashes, optimizes bond lengths and angles, and refines side-chain conformations, ensuring the physical and chemical plausibility of the structures. However, the standard Amber-relax workflow is limited to the 20 canonical amino acids in the available predefined force field parameters. When structures contain BNMeAAs and D-AAs, the absence of corresponding parameters prevents the standard Amber-relax process from proceeding, thereby restricting further refinement of protein structures with chemical modifications or BNMeAAs and D-AAs included. This limitation poses a challenge to the accurate prediction and design of proteins containing BNMeAAs and D-AAs.

To address this limitation, an enhanced Amber-relax strategy is developed by integrating quantum chemical calculations with molecular mechanics. Specifically, Gaussian software and AmberTools are employed to derive Amber-compatible parameter files. Subsequently, the tleap module in AmberTools is used to generate topology and coordinate files for the target structures. Finally, energy minimization of the full structure is performed using the efficient molecular simulation engine OpenMM. Through these steps, force field parameters for 56 BNMeAAs and D-AAs are successfully integrated into the Amber-relax workflow, enabling effective processing and optimization of predicted structures containing noncanonical residues.

As shown in Fig. 7a, the relaxed structures showed significantly improved quality relative to the initial unrelaxed ones. Specifically, the enhanced Amber-relax protocol effectively reduces atomic clashes within the structures (Fig. 7b) and optimizes side-chain conformations through energy minimization. Detailed results are provided in Additional File 1: Table S5.

Fig. 7.

Comparative analysis of the effects pre- and post-relaxation. a Bar chart of MolProbity scores before and after relaxation on the test set. b Distribution of clash_scores before and after relaxation on the test set. c Distribution of α-C RMSD before and after relaxation on the test set. d The distance between the C atom of PRO at position 8 and the CD atom of the residue at position 3 in D8.12 is relaxed from 1.8 to 2.6 Å after the relaxation process

To assess the effect of Amber-relax on prediction accuracy, we compare the predicted structures before and after relaxation with the corresponding native structures. As illustrated in Fig. 7c, Amber-relax basically does not change the overall accuracy of the predicted models, as indicated by the average α-C RMSD value of 1.374 Å for the unrelaxed and relaxed structures, and more detailed results are provided in Additional File 1: Table S6. This result indicates that the primary effect of Amber-relax is the improvement of local geometric features, such as side-chain positioning and stereochemical plausibility, without significantly altering the overall structural prediction accuracy, consistent with previous findings.

Furthermore, detailed local interactions of typical side-chain atoms are examined (Fig. 7d). In the unrelaxed models, clearly unreasonable nonbonded interactions between side-chain atoms are observed. These local structural issues are effectively resolved following the Amber-relax optimization procedure. Collectively, our enhanced Amber-relax workflow has significantly improved both the reliability and local geometric accuracy of the predicted structures.

By combining QM calculations with deep learning, our work substantially extends AlphaFold’s applicability to structures containing BNMeAAs and D-AAs, offering new possibilities for protein design and structural optimization. However, it is observed that some relaxed structures retained minor steric clashes or geometric irregularities, suggesting that future studies could incorporate more refined optimization techniques, such as extended molecular dynamics simulations, to further improve the structural reliability and physical realism of the predicted structures.

Conclusion

Cyclic peptides containing BNMeAAs and D-AAs exhibit outstanding membrane permeability and stability, making them promising candidates for drug development. Structural elucidation of these peptides not only aids in understanding their interactions with protein targets but also provides a critical foundation for novel drug design. However, the scarcity of experimentally determined structures for such peptides has significantly hindered the application of deep learning models in this domain.

In this study, the Rosetta SCP program is utilized to generate a dataset of cyclic peptides containing BNMeAAs and D-AAs. The AlphaFold model was subsequently fine-tuned on this dataset, which enhanced its ability to predict cyclic peptide structures with greater accuracy. This study also underscores the potential of this modeling framework for broader applications involving other noncanonical peptide structures. Additionally, the study provides a novel strategy for generating computational datasets to train specialized deep learning models.

Despite the significant advancements in predictive efficiency and the demonstrated generalizability on the test set, HighFold-MeD still exhibits several inherent limitations that merit attention. Current constraints on computational resources may have partially impacted model optimization. Furthermore, as both the training and validation datasets were entirely generated by Rosetta SCP under specific sampling parameters, the predictive accuracy of HighFold-MeD is predominantly limited by the quality and diversity of these Rosetta SCP-generated data. Although the model’s generalizability has been validated on a test set consisting of 33 experimentally solved cyclic peptide X-ray crystal structures, its core performance remains closely linked to Rosetta SCP data. Leveraging higher-quality and more diverse datasets—either through refining Rosetta SCP parameters to produce improved data or by incorporating experimentally determined cyclic peptide structures—could further enhance predictive performance. Additionally, advancements in computational infrastructure will facilitate the generation of larger-scale datasets and enable more effective model fine-tuning, thereby continuously improving the robustness, generalizability, and predictive accuracy of HighFold-MeD.

Importantly, the model establishes a scalable framework that integrates computational tools with deep learning, thereby extending its applicability to cyclic peptides containing BNMeAAs and D-AAs. This approach mitigates reliance on experimental structural data and alleviates the resource-intensive nature of experimental structure determination, thus facilitating research and development in peptide-based therapeutics. Simultaneously, it is important to acknowledge that with sufficient computational resources, increasing the sampling size would enable Rosetta SCP to achieve higher accuracy for individual targets. However, due to practical computational constraints, HighFold-MeD provides a balanced trade-off between accuracy and speed, rendering it particularly well-suited for large-scale drug screening.

In conclusion, this study demonstrates the substantial potential of combining computational tools with deep learning to address complex cyclic peptide structure prediction challenges, providing valuable insights and guidance for future research on non-canonical amid acid peptides.

Competing interests

The authors declare no competing interests.

Author contributions

Z.C.and S.C.: These authors contributed equally to this work. Z.C. and H.D. conceptualized ideas, proposed methods, and wrote the manuscript. Z.C.and S.C. investigated and implemented the deep learning programs. L.W., Z.W., Q.M., and J.G. completed the data collection. All authors have read and approved the final manuscript.

Funding

This work was supported in part by the Macao Science and Technology Development Fund (Grant No. 0151/2024/RIA2) and the internal grant from Macao Polytechnic University (RP/FCA-07/2024).

Data availability

The HighFold-MeD code can be found at https://github.com/hongliangduan/HighFold-MeD.