Skip to main content
NCBI home page
ACS AuthorChoice logoLink to ACS AuthorChoice
. 2024 Apr 22;64(9):3756–3766. doi: 10.1021/acs.jcim.4c00458

DEBFold: Computational Identification of RNA Secondary Structures for Sequences across Structural Families Using Deep Learning

Tzu-Hsien Yang †,‡,*
PMCID: PMC11094721  PMID: 38648189

Abstract

graphic file with name ci4c00458_0002.jpg

It is now known that RNAs play more active roles in cellular pathways beyond simply serving as transcription templates. These biological mechanisms might be mediated by higher RNA stereo conformations, triggering the need to understand RNA secondary structures first. However, experimental protocols for solving RNA structures are unavailable for large-scale investigation due to their high costs and time-consuming nature. Various computational tools were thus developed to predict the RNA secondary structures from sequences. Recently, deep networks have been investigated to help predict RNA structures directly from their sequences. However, existing deep-learning-based tools are more or less suffering from model overfitting due to their complicated problem formulation and defective model training processes, limiting their applications across sequences from different structural families. In this research, we designed a two-stage RNA structure prediction strategy called DEBFold (deep ensemble boosting and folding) based on convolution encoding/decoding and self-attention mechanisms to enhance the existing thermodynamic structure models. Moreover, the model training process followed rigorous steps to achieve an acceptable prediction generalization. On the family-wise reserved test sets and the PDB-derived test set, DEBFold achieves better structure prediction performance over traditional tools and existing deep-learning methods. In summary, we obtained a cutting-edge deep-learning-based structure prediction tool with supreme across-family generalization performance. The DEBFold tool can be accessed at https://cobis.bme.ncku.edu.tw/DEBFold/.

Introduction

RNA (ribonucleic acid) molecules in cells can serve as not only transcription templates but also noncoding RNAs.1 Many of them carry out specific cellular functions by using their sequence patterns or folding structures.2,3 For example, internal ribosome entry site elements can regulate the translation initiation of transcripts under stress conditions via their nonconservative sequence patterns and partial secondary structures.4,5 Since RNA secondary structures can form the subdomains in tertiary structures6 and can carry out critical cellular functions by themselves,7 figuring out the potential secondary structures of newly found RNA sequences becomes routine in RNA molecule studies.8

Experimental approaches for solving RNA structures include X-ray crystallography, nuclear magnetic resonance (NMR), and cryo-electron microscopy.9 These experiments can recognize the intramolecular RNA base–pair interactions within several angstroms.10 However, a long period of technician training time and high costs are required to carry out these experimental approaches. Recently, several chemical probing methods were developed, including the dimethyl sulfate (DMS) and the selective 2′-hydroxyl analyzed by primer extension (SHAPE) reagents treatment, to help identify the existence of structure pairings on sequence locations by detecting reverse transcriptase-stopping sites.1 When coupled with next-generation sequencing techniques, genome-wide structural information can be obtained.11 Nevertheless, these chemical probing approaches indicate only the existence of some structural forms. They also require additional folding prediction to provide RNA structure–function insights. Because of these reasons, in silico algorithms for identifying RNA structures are essential in the large-scale study of RNA sequences.

Traditional knowledge-based RNA secondary structure prediction algorithms utilize two approaches. The first type of algorithm assumes that an RNA sequence folds into the structure with the minimum free energy. The structures occupying the least free energy states are found through dynamic programming12 or Boltzmann partition functions.13 Researchers also developed a second type of algorithm that further incorporates multiple sequence alignment results when finding the minimum free energy structures.14 This type of tool sifts out the consensus structure either by simultaneous folding and aligning or aligning before folding.15 Recently, with the development of chemical structure probing reagents, many tools can also consider probing read scores as soft constraints to enhance structure prediction performance.16 Although a bundle of tools based on these two approaches has been proposed, these tools bear drawbacks that deteriorate prediction accuracy. First, free energy minimization algorithms may suffer from incomplete parametrization of the thermal models, triggering lower sensitivity.7,17,18 Second, a well-chosen homologous sequence family is indispensable for homology alignment-based tools.19 This requirement is usually not feasible for large-scale RNA profiling. Because of the aforementioned issues, researchers have tried to adopt data-oriented methods based on machine learning and deep-learning techniques as another way to solve the RNA structure prediction problem.

Early machine learning-based methods for RNA secondary structure prediction assisted the prediction process by refitting some predefined parameters in the thermal models.20 Later, some of these data-driven methods further tried to deal with the prediction problem directly from the sequence level.21,22 As a further step from feature-based algorithms, deep-learning tools have become promising for solving RNA structures in an end-to-end manner. However, existing deep-learning-based tools formulated the structure representation using 2D matrix encoding. Since there are far more unpaired position tuples, the 2D matrix structure encoding can easily result in unbalanced positive and negative targets, leading to the need for a large diversity in the training data sets. In a recent study, Szikszai et al.23 found that structure diversities within the data set are fundamental for training structure prediction models. They pinpointed that existing tools were trained only on data with sequence base variations, which suggested the occurrence of model overfitting. Because of the high parametrization and defect training processes, no robust result for diverse RNA structures was ready in deep-learning-based tools.23

In this research, we devised a two-stage strategy called DEBFold (deep ensemble boosting and folding) to overcome the prevalent overfitting issue in deep-learning-based structure prediction tools. Instead of assuming a sparse 2D structure matrix formulation, DEBFold is based on a more compact 1D linear representation and was trained with a family-wise prepared ground-truth data set. We arranged deep convolution encoding/decoding and self-attention mechanisms to integrate and boost existing thermal structure models in DEBFold for better RNA structure prediction. Compared with traditional knowledge-based or deep-learning-based structure prediction methods, DEBFold achieved better F1 score performance on the reserved structure-independent test sets and the PDB-derived test set, indicating good generalization across RNA structural families. Moreover, we demonstrated that DEBFold is robust in its pipeline design. These results suggest that DEBFold is the state-of-the-art deep-learning-based tool that handles cross-family RNA structure prediction. Finally, we implemented a web interface (https://cobis.bme.ncku.edu.tw/DEBFold/) that is freely available for researchers to facilitate the usage of DEBFold.

Methods and Data Sets

DEBFold Workflow

In this research, we designed a deep-learning workflow that integrates knowledge of different thermodynamic structure models with constrained optimization to tackle the RNA secondary structure folding problem. The designed integration algorithm is called DEBFold (deep ensemble boosting and folding). The DEBFold workflow comprises two stages to convert the given RNA sequences into their potential secondary structures (Figure 1): (1) structure location folding probability estimation. DEBFold was designed to take an RNA sequence into a deep-learning network that helps to integrate knowledge from various thermal models. Then, it generates the folding constraints similar to the SHAPE experiment results. (2) SHAPE-like constrained optimization. In the second stage, free energy minimization constrained by the predicted structure location folding probability will help provide the final structure prediction. These two stages are elaborated on in the following subsections.

Figure 1.

Figure 1

Open in a new tab

Overview of DEBFold. The example RNA structure is visualized by forna.24 Notations: Conv—convolution layers; FC—fully connected layers; TransConv—transpose convolution layers; MFE: minimum free energy.

Stage 1: Structure Location Folding Probability Estimation

In Stage 1, DEBFold tries to identify the potential locations that are involved in the final structure pairings based on a structure location identification deep network. DEBFold takes an RNA sequence as the single input. Before the deep network, the pipeline prepares the network input tensor based on the given sequence. The sequence of the given RNA is one-hot encoded into an l x 4 tensor, where l is the length of the sequence. Then, to integrate existing thermodynamics structure models, DEBFold collects the prediction results suggested by 6 different single-sequence prediction tools. The prediction results from the following tools were selected to be part of the DEBFold-prepared input tensors: RNAfold,25 IPknot,6 MaxExpect,26 ProbKnot,27 RNAprob,28 and Fold.29 These single-sequence input structure prediction tools were selected based on their prediction accuracy, calculation efficiency, and ease of incorporation. In our previous research,3 it was evaluated that RNAfold,25 IPknot,6 MaxExpect,26 and ProbKnot27 achieved acceptable structure prediction performance within acceptable time requirements. We further included RNAprob28 and Fold29 to diversify the thermal models while still limiting the time required to generate these inputs. Although ShapeKnots and HotKnots can provide additional thermal model diversity, these two tools have a much longer processing time for long sequences and thus are not integrated. The prediction results of the 6 tools under consideration are separately encoded into individual tensor Pi, where Pij = 1 if the jth location is said to be paired in the predicted structure of tool i. At the end of this step, the sequence one-hot encoding and the Pi’s are stacked into one l × 10 tensor. Since GPU parallel computation requires the lengths of the training batch tensors to be the same, we zero-padded the length of the network input tensor to 512 during the cross-validation training process. For prediction, this model does not need this padding. Therefore, variable-length RNAs can be fed into the structure location identification network when used in prediction. However, because the ground-truth data set collected in this research mainly consists of sequences shorter than 512 bps, the most suitable value of l is suggested to be less than 512 nt long.

The structure location identification deep network can be divided into 3 parts: feature extraction, self-attention, and structure location folding probability formation. The first part of the deep network mixes the input information to extract valuable features for the structure location identification. In previous research, convolution operations have been successfully used to extract sequence pattern features.30,31 Therefore, consecutive convolution operations (eqs 1 and 2) were adopted in the deep network of DEBFold for extracting sequence and structure features

graphic file with name ci4c00458_m001.jpg 1
graphic file with name ci4c00458_m002.jpg 2

where I is the l × 10 input tensor and BN_ReLU refers to the operation of ReLU activation, followed by batch normalization. In the above equation, the convolution operation conv(n,T) with n 1D kernels of size 3 on the l × c tensor T (using stride 2) is defined as previously suggested32,33 (eq 3)

graphic file with name ci4c00458_m003.jpg 3

where T(−1) is the zero-padded element, Inline graphic, j is in the range of 0 and (n – 1), and Kj is the jth kernel.

After the features centered at each location are extracted (as X2 in eq 2), the mutual interactions between each location are considered by the self-attention operation (eqs 4 and 6). A self-attention block helps compute the mutual influence between two nucleotide locations within the given sequence. In order to avoid the gradient vanishing problem in deep neural networks,34 a residual architecture was also implemented in the self-attention block (eqs 5 and 7)

graphic file with name ci4c00458_m005.jpg 4
graphic file with name ci4c00458_m006.jpg 5
graphic file with name ci4c00458_m007.jpg 6
graphic file with name ci4c00458_m008.jpg 7

where (W1, W2) and (W3, W4) are the trainable weights with internal positional feed-forward dimension 1024, and multi_atten(.) is the multihead attention operation. In DEBFold, the self-attention operation was repeated twice in the network architecture. The multihead attention module, denoted as multi_atten(Q, K, V, n), is defined as the following35

graphic file with name ci4c00458_m009.jpg 8
graphic file with name ci4c00458_m010.jpg 9

where WQ,i/WK,i/WV,i are the query/key/value weight matrices that map Q, K, and V into tensors of dimension 512, l is the length of the Q/K/V tensors, n is the number of heads used in this operation, ⊕ represents the tensor concatenation operation, and H is the computed multihead attention tensor. The self-attention operation is obtained by setting the K, Q, and V tensors to be identical. The number of heads in this network was picked to be 16. After the self-attention block, the sequence features for each location are weighted by the potential influences among different locations.

In the last step of this stage, the locations involved in the final structure pairings are identified in the form of potential pairing probabilities based on the final attended hidden feature tensor A (from eq 7). Since the final structure locations are marked segments along the sequence position, DEBFold resorts to consecutive 1D transpose convolutions (eqs 10 and 11. See eqs 12–14 for its definition) to help decode A (from eq 7) to produce segment marking along the positions

graphic file with name ci4c00458_m011.jpg 10
graphic file with name ci4c00458_m012.jpg 11

In the above equations, the 1D transpose convolution transConv(n, F) using n kernels of size 3 on the l × c tensor F (with stride 2) is calculated based on the following formula

graphic file with name ci4c00458_m013.jpg 12
graphic file with name ci4c00458_m014.jpg 13
graphic file with name ci4c00458_m015.jpg 14

where E is the extension tensor for F, 0 ≤ i ≤ 2l – 1, j is in the range of 0 and (n – 1), and Pj is the j kernel of the transpose convolution operation. Finally, the structure location folding probability of each location i in the output tensor Y is individually computed based on O (from eq 11) as

graphic file with name ci4c00458_m016.jpg

where W5 is the 256 × 1 trainable weight matrix shared among all base locations and σ is the sigmoid function. The choice of a weight matrix of the same size as the internal hidden dimension fulfills the need to accept RNAs with variant lengths, making DEBFold unlimited by RNA sequence lengths.

Stage 2: Score-Constrained Optimization Folding

It has been shown that structure probing values as soft folding constraints can benefit the overall RNA secondary structure prediction.36 DEBFold utilizes this concept and incorporates a two-stage pipeline to predict the RNA secondary structures. In Stage 1 of DEBFold, we obtain a structure location folding probability tensor, indicating the possibility of each location being involved in the final structure. The probabilities are inversely related to the read scores measured in probing experiments such as SHAPE-seq. Hence, we defined the SHAPE-like score tensor S by S = 1 – Y. The computed score tensor S is then provided as the SHAPE score file option for the minimum free energy structure prediction optimization tools to be used in soft constraint calculation. In DEBFold, we adopted the linear equation proposed by Deigan et al.37 and used the default slope and intercept values in each tool for this equation to convert the SHAPE-like scores into pseudoenergy terms in the optimization process. With the guidance of score tensor S, more accurate RNA structure predictions can be obtained in the constrained optimization. In the final design of DEBFold, the Fold algorithm29 is chosen as the default constrained optimization algorithm to be coupled with the score tensor S for generating the final structure prediction.

Model Training Hyperparameters

DEBFold Stage 1 is formed by a deep neural network, and we utilized cross-validation (CV) and learning-curve techniques to select the proper hyperparameters. The sum of the binary cross-entropy for each base location was selected as the overall loss function to optimize the deep network. Since the available structural families are not abundant, 25-CV was adopted to fully utilize the data set. In order to avoid overfitting, the average training and validation learning curves were monitored based on the partial loss and partial F1 scores, which exclude the loss and F1 calculation of the padded locations after the original sequence length in each RNA. The final hyperparameters chosen in DEBFold are summarized as follows: (1) training epochs: 250; (2) batch training size: 512; (3) the optimization update algorithm: Adam; (4) learning rate and decay schedule: cosine decay (minimum learning rate = 1 × 10–13) in the first 227 epochs and then exponential decay (multiplicative factor 1 × 10–10) for the rest of the 23 epochs; (5) dropout layers with dropout rate = 0.1 and 0.4 were added to the residual layers and convolution/transpose convolution layers, respectively, to regularize the deep network; and (6) positive weights = 1.8 in the binary cross-entropy loss. The training process was carried out using a NVIDIA RTX 4080 GPU.

Family-Wise Processed RNA Structure Ground-Truth Data Set

Model training in data-based learning approaches requires the data diversity in the training set to mimic the degree of variations in real problems.38 Researchers have experimented that a large number of sequences from a small number of structural families can easily lead to model overfitting.23 The possible reason for the situation is that structurally similar sequences provide limited variations for deep models to fetch the inherent patterns that map sequences to the corresponding structures. In order to prepare a ground-truth data set that incorporates a high structure diversity, we downloaded the data deposited in bpRNA-1m39 and retrieved the sequences with known Rfam families. In bpRNA-1m, RNAs curated from Rfam 12.240 were selected and processed. In total, 43,269 valid RNA sequences from 2125 families were found.

In order to tune and evaluate DEBFold in a structural family independent manner, the ground-truth data set is divided and sampled into two parts: the training-validation and test sets. Since RNAs in the same family are structurally similar,23 the data processing was carried out in a family-wise manner. We resorted to 25-fold cross-validation in DEBFold Stage 1 model training due to the scarcity of known distinct RNA structural families in the collected ground-truth data set. However, the average of the cross-validation results provides only an optimistic estimation of the model performance. An extra test set should be set aside as totally clean data for a fair performance evaluation. Therefore, among the gathered 2125 families, 44 (2%, or 1/50 due to 25-fold cross-validation, of the total 2125 families) of them were reserved in the test set. These 44 families were selected by stratified sampling that preserves the length distribution of all 2125 RNA families. In the calculation of the length distribution of RNA families, the longest sequence within a structural family was chosen as the representative. These 44 families were also ensured to be in distinct Rfam clans from the rest of the remaining families. The remaining 2081 families were used as the training-validation folds in cross-validation. We sampled at most 5 sequences from the 2081 families to be included in the training-validation set (8782 sequences in total). Finally, we took one sequence from each of the 44 test families for testing, resulting in a test set consisting of 44 RNAs. This test set is termed TestSetα in this research. We further confirmed the sequence similarity between the training-validation and test RNA sequences using CD-HIT.41 Under the threshold of 0.85, TestSetα contains no RNA with sequence similarity to any member of the training-validation set. In order to have more extensive testing, we further prepared two additional independent test sets (TestSetβ and TestSetγ) as described in the following two subsections.

Contamination-Free Family-Wise Independent Test Set for Evaluating Existing Deep-Learning-Based Structure Prediction Tools

In this research, we also tried to compare the performance of DEBFold with existing deep-learning-based structure prediction tools. While SPOT-RNA and SPOT-RNA2 were originally trained on the bpRNA-1m data set and there is no retraining code for these tools, it is necessary to gather some other independent test set than TestSetα, which was derived from bpRNA-1m, for ensuring a fair generalization evaluation.38 For this purpose, we downloaded the bpRNA-new data set from the MXfold2 work22 and prepared an independent test set for fairly evaluating the performance of DEBFold with existing deep-learning-based prediction tools. The so-called bpRNA-new data set from the MXfold2 work includes RNA sequences from Rfam 14.4 but excluded those already in Rfam 12.2. We further reorganized the bpRNA-new data set to better help evaluate the deep-learning model performance. In the downloaded bpRNA-new data set, there are a total of 489 families for the collected sequences. Among them, 8 families also appear in the bpRNA-1m data set. Hence, only 481 families with 5265 sequences were suitable and thus retained. These 481 families were also ensured to be in different Rfam clans from sequences in bpRNA-1m. Since most deep-learning-based prediction tools can only handle sequences composed of the four basic nucleotide alphabets, sequences with alphabets other than AU(T)GC were abandoned. For each family, one sequence from each family was selected as the representative. As previously cautioned, benchmark sets with too many short sequences tend to provide biased performance evaluation results.42 In order to eliminate the length biases for the remaining bpRNA-new sequences in a family-wise manner, the remaining 481 families are binned with a length of 50 nts, resulting in 10 bins in total (max-length = 489 nts). In these 10 bins, the minimum number of families within a bin is 4. Therefore, to balance the number of sequences in each length bin, we sampled 4 families from each of these 10 bins to prepare the final test set, resulting in 40 test families (i.e., 40 sequences). The test set consisting of these 40 sequences from different families is called TestSetβ in this research. We also confirmed the sequence similarity between the training-validation and test RNA sequences using CD-HIT.41 Under the threshold of 0.85, TestSetβ contains no RNA sequence similar to any member of the training-validation set.

PDB-Derived Source-Independent Test Set

Since RNA structures deposited in Rfam are actually only a mixture of experimental results, predictions, and homologue modeling inference,43 we sought another test set that gathers RNA structures identified with pure experimental evidence. In the bpRNA database,39 Danaee et al. also collected some PDB-derived RNA secondary structures that were parsed only from the 3D structures obtained using X-ray crystallography and NMR techniques. We downloaded these PDB-derived results and prepared a source-independent test set named TestSetγ. As in TestSetβ, only sequences represented using AU(T)GC were retained. In addition, to have a fair test set that does not include sequences already used in the training process of any tool, we excluded sequences that have more than 85% CD-HIT-calculated similarity to any sequence from the training-validation set used in this research and the TR1 data set used in the training process of SPOT-RNA. In total, 147 PDB-derived RNA structures were available. We also sought to avoid length biases in TestSetγ by equally length-partitioning these sequences into 10 bins (max-length = 338 nts). We take only 3 sequences within each length bin in order to balance the number of sequences in each bin, resulting in 15 independent sequences (several bins have zero sequence in them) in TestSetγ. Overall, TestSetγ serves as a source-independent test set that helps evaluate the tools in predicting structures directly inferred from experiments.

Structure Prediction Evaluation Metrics

To evaluate the RNA secondary structure prediction performance, we resorted to the structure F1 score calculation. The precision, recall, and F1 score for a predicted structure are computed as follows3,8

graphic file with name ci4c00458_m017.jpg 15
graphic file with name ci4c00458_m018.jpg 16

where TP, FP, and FN stand for the numbers of true-positive, false-positive, and false–negative pairs, respectively. For a predicted structure and its corresponding real structure, TP counts the number of correctly predicted pairings, FP is the number of predicted pairings that do not appear in the real structure, and FN considers the number of real base pairings missed by the prediction. The final F1 score can balance the trade-off between the FP and FN values and is thus used as the overall evaluation metric.

Results and Discussion

DEBFold Outperforms Previous Thermodynamics-Based RNA Structure Prediction Tools

We first evaluated the generalization prediction performance of DEBFold and compared its performance with classical thermodynamics-based prediction tools on prepared TestSetα, TestSetβ, and TestSetγ. Nineteen prediction tools that were still publicly available were compared in this section: (1) tools from single-sequence thermodynamics models: RNAfold,25 RNALfold,44 Fold,29 MaxExpect,26 RNAprob,28 RME (both the PARS-model and the DMS-model),45 PKNOTS,46 IPknot,6 HotKnots,47 IterativeHFold,48 ShapeKnots,49 and ProbKnot;27 (2) tools based on homologous sequence alignment and folding: RNAalifold,14 TurboFold,29 SPARSE,50 aliFreeFold,15 LocARNA,51 comRNA,52 and MXSCARNA.53 Algorithms that fail to provide executable codes were not included in this comparison. Default parameters suggested by the authors were adopted in these tools for a fair comparison. For IterativeHFold, we set RNAfold as the restriction structure generator for it. For tools that require multiple sequence alignment results, sequences in the same Rfam family were taken. In some Rfam families, the sequences are nearly identical, even though they were collected from different species. These nearly identical sequences within the same family were used in tools that required homologous alignments if no other homologous sequence was available for that family. If no sequence is available in the same RNA structural family of the given sequence, the required homologues were searched against the RNAcentral database54 using BLAST as suggested in our previous work.3

The results of different tools on TestSetα, TestSetβ, and TestSetγ are summarized in Table 1. In Table 1, the performance of each tool was summarized by its median rank and overall median F1 score on the three test sets. Among single-sequence input tools [RNAfold, RNALfold, Fold, MaxExpect, RNAprob, RME (both the PARS-model and the DMS-model), PKNOTS, IPknot, HotKnots, IterativeHFold, ShapeKnots, and ProbKnot], DEBFold achieves the overall top-one median F1 score rank on the three test sets. The improvement is supposed to be from the integration of various existing thermal knowledge followed by deep learning boosting. Compared with tools that utilize homologous sequences (RNAalifold, TurboFold, SPARSE, aliFreeFold, LocARNA, comRNA, and MXSCARNA), DEBFold still has a better overall median F1 score rank. Combing these comparisons, DEBFold provides cutting-edge performance over existing tools on the test sets. Notice that in TestSetα, tools that rely on sequence alignment results generally perform better than single-sequence input tools since the RNA sequences from Rfam 12.2 mostly come with good family-wise homologous sequences. This advantage may diminish in many real-world use cases, such as in TestSetγ. DEBFold overcomes this restriction by using a deep learning model that integrates the results of many single-sequence structure prediction tools. It is worthwhile noting that DEBFold also boosts the prediction F1 score over the 6 predictions (RNAfold, IPknot, MaxExpect, ProbKnot, RNAprob, and Fold) integrated in the input encoding tensor of the deep network. In summary, DEBFold is concluded to outperform existing classical structure prediction tools with less input requirements.

Table 1. Test Set Median F1 Score Performance Comparison between DEBFold and Other Available Thermodynamics-Based RNA Secondary Structure Prediction Tools on the Three Prepared Test Setsa.

structure prediction tool TestSetα
 TestSetβ
 TestSetγ
 overall median F1 (%) median rank
  F1 (%) P (%) R (%) rank F1 (%) P (%) R (%) rank F1 (%) P (%) R (%) rank    
DEBFold 64.9 62.1 67.9 4 55.7 56.4 56.8 1 77.9 83.3 73.2 1 64.9 1
RNAalifold 64.7 62.2 67.5 5 55.6 50.7 62.2 2 59.6 56.4 63.3 14 59.6 5
ProbKnot 60.6 60.8 60.7 13 53.8 50.4 58.5 6 72.7 77.8 68.3 6 60.6 6
MaxExpect 64.2 54.9 82.9 7 55.0 43.9 81.7 4 69.2 73.0 65.9 8 64.2 7
ShapeKnots 61.9 54.9 71.8 12 55.2 45.7 69.9 3 70.9 73.7 68.3 7 61.9 7
IterativeHFold 63.5 53.9 82.9 8 52.3 39.9 81.2 8 77.9 83.3 73.2 1 63.5 8
RNAfold 63.5 53.9 82.9 8 52.3 39.9 81.2 8 77.9 83.3 73.2 1 63.5 8
RME (PARS model) 63.5 56.2 75.2 10 54.2 43.1 81.7 5 69.2 73.0 65.9 8 63.5 8
RME (DMS model) 63.1 53.7 82.9 11 53.1 38.8 87.5 7 69.2 73.0 65.9 8 63.1 8
SPARSE 65.7 66.7 64.7 2 51.4 38.9 80.5 10 60.5 72.0 52.2 12 60.5 10
TurboFold 64.6 54.8 83.1 6 47.8 57.1 45.5 13 63.6 85.4 50.7 11 63.6 11
LocARNA 65.7 66.7 64.7 2 49.9 47.8 54.1 11 48.1 53.4 43.7 19 49.9 11
HotKnots 59.1 54.7 75.0 14 48.5 40.7 62.1 12 76.3 82.9 70.7 5 59.1 12
IPknot 57.8 47.0 81.5 15 47.7 44.1 52.0 14 76.9 83.3 66.7 4 57.8 14
aliFreeFold 65.7 59.4 67.1 1 44.9 41.8 49.5 16 48.1 53.4 43.7 19 48.1 16
MXSCARNA 47.2 63.2 37.7 16 34.8 29.8 43.3 19 54.5 100.0 37.5 15 47.2 16
Fold 45.8 36.5 73.5 17 43.2 43.6 42.8 17 54.5 72.7 42.0 15 45.8 17
RNAprob 45.8 36.5 73.5 17 43.2 43.6 42.8 17 54.5 72.7 42.0 15 45.8 17
PKNOTs 42.8 37.7 51.0 20 47.5 42.0 55.6 15 54.2 65.3 46.4 18 47.5 18
RNALfold 45.3 51.9 66.9 19 25.4 31.8 21.1 20 60.5 81.2 48.1 13 45.3 19
comRNA 29.1 24.3 37.8 21                 29.1 21
Open in a new tab
a

The median rank column summarizes the rank median of each tool in the three test sets, and the overall median F1 column computes the median F1 score of the three test-set median F1 scores. For comRNA, no reasonable result was provided in TestSetβ and TestSet γ. Notations: P—precision, R—recall.

DEBFold Has Better Generalization Performance than Existing Deep-Learning-Based Attempts

Recently, more and more deep-learning-based structure prediction tools were developed to help predict RNA secondary structures. We also sought to compare DEBFold to these tools. To the best of our knowledge, existing deep-learning-based RNA secondary structure prediction tools include the following: SPOT-RNA,21 SPOT-RNA2,55 MXfold2,22 GCNFold,56 UFold,57 REDFold,58 and e2eFold.59 Since SPOT-RNA and SPOT-RNA2 released by the authors do not encompass any retraining codes and the tools themselves incorporate the whole bpRNA-1m data set in its model training process, independent test sets other than TestSetα, which was prepared from the bpRNA-1m Rfam 12.2 part, should be used to ensure fair comparison between DEBFold and SPOT-RNA/SPOT-RNA2. For this purpose, we downloaded the bpRNA-new data set from the MXfold2 work and prepared TestSetβ based on Rfam 14.4 (excluding those already in Rfam 12.2) for fairly evaluating the performance of DEBFold with existing deep-learning-based prediction tools. In TestSetβ, both the length bias and data contamination issues were eliminated. Details of the TestSetβ preparation steps can be found in the “Contamination-Free Family-Wise Independent Test Set for Evaluating Existing Deep-Learning-Based Structure Prediction Tools” section. Another PDB-derived independent test set, TestSetγ, was also prepared for this purpose. The preparation steps of TestSetγ can be found in the “PDB-Derived Source-Independent Test Set” section.

The final test results of DEBFold and the existing deep-learning-based prediction tools on TestSetα, TestSetβ, and TestSetγ are listed in Table 2. For SPOT-RNA, SPOT-RNA2, and UFold, TestSetα results were not reported due to complete data contamination caused by using the whole bpRNA-1m data set in the training process. REDFold has potential slightly optimistic evaluation results on TestSetα and TestSetβ since a small portion (122 families, whose details were not clearly reported by the original paper) of the Rfam 14.4 families were included in the training process. MXfold2 also included a small portion of the Rfam data (22 Rfam families from Rfam 10.0 and 151 sequences from the S-151Rfam data set, whose details were not clearly reported by the original paper), which might result in potential slightly optimistic results for TestSetα and TestSetβ. We calculated the median performance ranking and median F1 score of each tool on suitable test sets in the comparison. For tools that are suitable to be evaluated on all three test sets (DEBFold, MXfold2, REDFold, GCNfold, and e2efold), DEBFold has superior median F1 score results. Compared with SPOT-RNA, UFold, and SPOT-RNA2, DEBFold shows the top overall median F1 score rank. Among these tools, SPOT-RNA was first pretrained on the bpRNA-1m data set. Then, these pretrained models were fine-tuned and aggregated on the PDB data set. In other words, both the RNA structural families from bpRNA-1m and the PDB-identified RNA sequences were considered in the tool. Because of the final fine-tuning process, the final published SPOT-RNA tool somewhat favors 3D-structure-derived RNA base pairings. However, in PDB-derived TestSetγ, DEBFold still outperforms SPOT-RNA. Therefore, we conclude that DEBFold can provide better results than SPOT-RNA even on the 3D-structure-derived base pairs. It is worth noting that SPOT-RNA2 can achieve good results by considering the homologous sequence information. Nonetheless, SPOT-RNA2 only gets the same median performance rank as DEBFold while demanding more than 7500 times longer execution time than DEBFold (see Table 5). In view of the overall consideration of prediction efficiency and accuracy, DEBFold still has better results than SPOT-RNA2. Overall, the performance improvements in DEBFold are supposed to be attributed to a novel two-stage pipeline that utilizes a 1D structure representation and deep network integration of various thermal models. These analyses indicate that DEBFold can achieve state-of-the-art generalization structure prediction performance over that of currently available deep-learning-based prediction tools.

Table 2. Test Set Median F1 Score Performance Comparison between DEBFold and Other Deep-Learning-Based RNA Structure Prediction Tools on the Three Prepared Test Setsa.

structure prediction tool TestSetα
 TestSetβ
 TestSetγ
 overall median F1 (%) median rank
  F1 (%) P (%) R (%) rank F1 (%) P (%) R (%) rank F1 (%) P (%) R (%) rank    
DEBFold 64.9 62.1 67.9 1 55.7 56.4 56.8 1 77.9 83.3 73.2 2 64.9 1
MXfold2 58.9 62.3 55.9 2 54.3 43.7 73.5 2 78.4 80.6 76.3 1 58.9 2
REDFold 40.4 46.7 37.0 3 41.9 50.6 35.9 3 53.8 56.8 51.2 3 41.9 3
GCNfold 31.9 83.3 20.0 4 27.0 61.0 18.8 4 40.0 81.2 25.0 4 31.9 4
e2eFold 0 0 0 5 1.2 2.3 0.8 5 9.5 26.7 5.8 5 1.2 5
                             
DEBFold         55.7 56.4 56.8 1 77.9 83.3 73.2 2 66.8 1.5
SPOT-RNA2         51.8 44.0 63.1 2 84.2 84.2 84.2 1 68.0 1.5
SPOT-RNA         50.3 57.1 52.5 3 73.7 93.3 60.9 3 62.0 3
UFold         45.8 33.2 77.1 4 48.3 59.6 40.6 4 47.0 4
Open in a new tab
a

The median rank column summarizes the rank median of each tool in the three test sets, and the overall median F1 column computes the median F1 score of the three test-set median F1 scores. For e2eFold, no reasonable result was provided in TestSetα. Notations: P—precision, R—recall.

Table 5. Execution Time Comparison between DEBFold and Other Available RNA Secondary Structure Prediction Tools on the Collected 147 PDB-Derived Sequences.

structure prediction tool execution time (s)
DEBFold 67.3
RNAfold 7.7
RNALfold 9.2
PKNOTS 12.2
Fold 13.4
IterativeHFold 15.5
IPknot 17.3
ProbKnot 20.5
MaxExpect 20.6
RNAProb 22.1
ShapeKnots 382.8
HotKnots 2824.0
RME (PARS model + DMS model) 74906.6
(Homology finding using BLAST) 591.1
RNAalifold 6.1
SPARSE 10.1
MXSCARNA 14.9
LocARNA 22.9
aliFreeFold 31.6
TurboFold 190.0
comRNA 904.3
MXfold2 33.6
GCNfold 63.8
REDFold 72.9
UFold 102.8
e2eFold 123.5
SPOT-RNA 192.8
SPOT-RNA2 511970.3a
Open in a new tab
a

Note that the execution time accumulated for SPOT-RNA2 is only from the prediction of the 15 TestSetγ sequences selected by stratified sampling from the total available 147 PDB-derived sequences.

DEBFold Is Robust against Different Thermodynamics-Constrained Optimization Algorithms

The second step of DEBFold feeds the computed structure location folding probabilities into thermodynamically constrained optimization algorithms to obtain the final RNA secondary structure prediction. Previous sections show that DEBFold can have a better structure prediction over the integrated constitutional prediction results by using more accurate folding scores as the optimization soft constraints. We next evaluated the robustness of DEBFold against different constrained optimization algorithms. In DEBFold, Fold was the adopted algorithm for the thermodynamically constrained optimization of the final predicted structures. Besides Fold, RNAfold and ShapeKnots can also help perform thermodynamics-constrained optimization based on soft constraints. These three constrained optimization algorithms were separately combined with DEBFold Stage I (DEBFold-Fold, DEBFold-RNAfold, and DEBFold-ShapeKnots) and then evaluated on TestSetα, TestSetβ, and TestSetγ. The comparison results are summarized in Table 3. As shown in Table 3, DEBFold achieves nearly identical median F1 score results (within one percent) on the three test sets when different thermodynamics-constrained optimization tools are used. From this comparison, it is suggested that DEBFold can provide accurate folding scores that help boost the final structure prediction and is robust against different thermodynamics-constrained optimization tools.

Table 3. Performance Evaluation for DEBFold Algorithm Robustness Using the Median F1 Scores on TestSetα, TestSetβ, and TestSetγa.

folding algorithm used TestSetα
 TestSetβ
 TestSetγ
  F1 (%) P (%) R (%) F1 (%) P (%) R (%) F1 (%) P (%) R (%)
DEBFold-Fold 64.9 62.1 67.9 55.7 56.4 56.8 77.9 83.3 73.2
DEBFold-RNAfold 64.9 62.1 67.9 56.6 54.8 63.7 77.9 83.3 73.2
DEBFold-ShapeKnots 64.9 62.1 67.9 55.5 55.7 56.8 77.9 83.3 73.2
Open in a new tab
a

DEBFold Stage I deep network combined with different thermodynamics-constrained optimization algorithms were compared. Notations: P—precision, R—recall.

DEBFold Provides Superior Structure Location Labeling Results to the Baseline and the Existing Model

In DEBFold Stage 1, the base locations where the structure pairings occur are labeled. The higher performance of the labeling process usually leads to better constrained optimization results. We compared the structure location labeling performance of DEBFold Stage 1 with the simple consensus model and another existing technique called GRASP.60 In DEBFold Stage 1, the probability of a base location being involved in the final structure pairings is predicted by a deep convolutional network. In previous research, Ke et al. proposed using the XGBoost model on the flattened one-hot encoding tensor of a sequence window to get the pairing probability of an RNA base location and implemented this method as a tool called GRASP.60 In both DEBFold Stage 1 and GRASP, a fair threshold of 0.5 was adopted when the existence of base pairings was labeled for different RNA base locations. On the other hand, the simple consensus model labels the existence of the base pairing for an RNA base location if all 6 tools (RNAfold,25 IPknot,6 MaxExpect,26 ProbKnot,27 RNAProb,28 and Fold29) agree on the pairing existence. In order to evaluate the location labeling results, we resort to the confusion matrix generated from the labeling results of a specific tool for each RNA. For structure location labeling in each RNA, TP is the number of correctly labeled pairing locations, FN represents the number of missed pairing locations, FP shows the number of nonpairing locations mistakenly marked to be paired, and TN counts the number of correctly labeled nonpairing locations. Based on the defined confusion matrix, the recall, precision, and F1 score values for the structure location labeling problem are defined similarly to those in eqs 15 and 16. We calculated the evaluation metrics for each RNA on the three prepared test sets (TestSetα, TestSetβ, and TestSetγ) and obtained the median F1 score of all of the RNAs in each test set. As summarized in Table 4, DEBFold Stage 1 achieves F1 scores superior to those of both the consensus model and GRASP on all three test sets. By this comparison, we conclude that DEBFold Stage 1 is better than the simple consensus model and the existing tool GRASP in labeling the existence of the base pairings for RNA base locations.

Table 4. Labeling Performance Comparison among DEBFold, GRASP, and the Simple Consensus Model on the Three Test Sets (TestSetα, TestSetβ, and TestSetγ)a.

folding algorithm used TestSetα
 TestSetβ
 TestSetγ
  F1 (%) P (%) R (%) F1 (%) P (%) R (%) F1 (%) P (%) R (%)
DEBFold Stage 1 77.0 72.1 82.5 71.8 61.8 85.5 79.5 87.0 73.2
the simple consensus model 70.7 76.3 65.6 55.9 63.3 50.0 77.2 88.9 68.3
GRASP 52.5 45.7 61.5 51.0 48.1 54.2 63.3 77.6 53.5
Open in a new tab
a

The median value is recorded for each test set. Notations: P – precision, R – recall.

Speed Comparison of DEBFold with the Existing Tools

Due to the advancement of sequencing technology, lots of RNA sequences are found, leading to the need for large-scale structure investigation directly on the sequences. To fulfill this need, the execution time required for each prediction tool should also be inspected in addition to structure prediction accuracy. We evaluated and compared the execution time of DEBFold and different structure prediction tools on the collected 147 PDB-derived sequences using a workstation with Intel i7 CPU cores and 128GB RAM. The results are summarized in Table 5. On these 147 RNA sequences, most single-input folding algorithms used less than a minute to process the predictions. Since DEBFold integrates the outcomes of six different single-input folding prediction results, it took around 1 min to finish all predictions. For algorithms that consider homologous sequences (RNAalifold,14 TurboFold,29 SPARSE,50 aliFreeFold,15 LocARNA,51 comRNA,52 and MXSCARNA53), around an additional 10 min was demanded to get the homologous sequences using BLAST. Therefore, DEBFold has better time efficiency than these homology-based tools and achieves better performance. In deep-learning-based predictions, SPOT-RNA required about 3 times more execution time than did DEBFold. Among all tools, SPOT-RNA2 had very low execution efficiency and needed more than 142 h, which is more than 7500 times as long as the execution time of DEBFold, to predict merely the structures of the 15 TestSetγ sequences selected by stratified sampling from all 147 RNAs. More execution time is required to finish all 147 RNAs. The high execution time of SPOT-RNA2 suggests that SPOT-RNA2 is probably not ready for a large-scale RNA structure investigation. Moreover, SPOT-RNA2 requires more than 2TB of disk space to store the information needed for its algorithm. In summary, DEBFold not only achieves state-of-the-art prediction accuracy but also retains an acceptable execution time for large-scale RNA sequence investigation.

Limitations of DEBFold

In this work, we developed a deep-learning model that can integrate the prediction results from different single-sequence folding tools to more accurately predict RNA secondary structures. DEBFold is a two-stage pipeline for RNA secondary structure prediction. In the first stage, DEBFold identifies the locations where structural pairings occur. Then, the pairing probabilities for the base locations of a given sequence are transformed into the SHAPE-like score tensor. Based on the SHAPE-like score tensor, the structure prediction bearing the minimum free energy under these soft constraints is given by Fold. In RNA structure prediction, pseudoknots (intertwined helices on a plane) and noncanonical pairings (the base pairs formed by hydrogen bonding differing from the patterns of standard Watson–Crick rules61) present significant challenges for prediction algorithms. Since DEBFold Stage 1 does not distinguish pseudoknot, canonical, and noncanonical RNA pairings, these pairings are all potentially considered in this stage. It is possible to adopt ShapeKnots or other minimum free energy structure prediction optimizers in DEBFold Stage 2 if these challenging types of base pairings are to be considered in the final optimization process. However, since there are insufficient noncanonical or pseudoknot pairings in the ground-truth data set, DEBFold currently mainly focuses on canonical base pairings.

Previously, Szikszai et al.23 pointed out that deep-learning models for RNA secondary structure prediction can easily overfit a data set if a sequence-based cross-validation and a test process are adopted. This type of overfitting occurs even when a massive number of sequences from only a few families are provided. Szikszai et al. argued that the chief cause for the phenomenon lies within the structural similarities among RNA sequences in the same family. Based on their research, we adopted a family-wise approach in developing DEBFold and further prepared test sets that are more suitable for evaluating this problem in existing deep-learning-based prediction tools. Although DEBFold overcomes the overfitting problem and provides state-of-the-art performance, inherent limitations remain. Currently, the known RNA families are far smaller than the known sequences in the community. Despite the large number of RNAs with known sequences available, most of them belong to the same structural families. Because of the limited RNA structural families available (only 2606 different families in Rfam 14.4), the performance of single-sequence structure prediction using deep learning is largely restricted. The limited RNA structural families also constrain the prediction accuracy of long noncoding RNAs (lncRNAs) since the overfitting problem boils down to the lack of structural family diversity. Currently, popular data sets contain scarce numbers of structural families for long RNAs, inhibiting the structural prediction of lncRNAs. The current version of DEBFold is suitable only for sequences up to 512 bps. Although some of the existing deep-learning-based tools claim to be able to deal with lncRNAs, the moderate performance observed in Table 2 indicates that the originally claimed prediction accuracy may still need more attention. Notice that DEBFold performs better than classical folding algorithms that incorporate multiple-sequence alignments. Therefore, whether multiple-sequence alignments can better help suggest structure modules in the DEBFold pipeline requires further study.

Conclusions

In this research, we designed and implemented a deep learning tool called DEBFold to provide accurate RNA secondary structure prediction for sequences across RNA structural families. DEBFold is verified to be free of the overfitting pitfall occurring in many deep-learning-based structure prediction tools and outperforms the currently available tools. We believe that the development of DEBFold can significantly accelerate the use of artificial intelligence to help us understand structure–function relations for RNAs.

Acknowledgments

The author would like to thank Li-Chang Teng, Zhan-Yi Liao, and Min Hsia for their help in the early stage exploration of the works related to this research. This study was supported by National Cheng Kung University and the National Science and Technology Council of Taiwan (MOST 110-2222-E-006-017, MOST 111-2221-E-006-231, and NSTC 112-2221-E-006-129-MY2). The study was also supported by the Headquarters of University Advancement at National Cheng Kung University and the Ministry of Education, Taiwan.

Data Availability Statement

The implemented DEBFold pipeline and the processed RNA structure ground-truth data sets (including the training-validation set, TestSetα, TestSetβ, and TestSetγ) are available at https://cobis.bme.ncku.edu.tw/DEBFold and https://github.com/cobisLab/DEBFold.

The author declares no competing financial interest.

References

  1. Kwok C. K.; Tang Y.; Assmann S. M.; Bevilacqua P. C. The RNA structurome: transcriptome-wide structure probing with next-generation sequencing. Trends Biochem. Sci. 2015, 40, 221–232. 10.1016/j.tibs.2015.02.005. [DOI] [PubMed] [Google Scholar]
  2. Wan Y.; Kertesz M.; Spitale R. C.; Segal E.; Chang H. Y. Understanding the transcriptome through RNA structure. Nat. Rev. Genet. 2011, 12, 641–655. 10.1038/nrg3049. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Yang T.-H. An aggregation method to identify the RNA meta-stable secondary structure and its functionally interpretable structure ensemble. IEEE/ACM Trans. Comput. Biol. Bioinf. 2022, 19, 75–86. 10.1109/TCBB.2021.3082396. [DOI] [PubMed] [Google Scholar]
  4. Baird S. D.; Lewis S. M.; Turcotte M.; Holcik M. A search for structurally similar cellular internal ribosome entry sites. Nucleic Acids Res. 2007, 35, 4664–4677. 10.1093/nar/gkm483. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Yang T.-H.; Wang C.-Y.; Tsai H.-C.; Liu C.-T. Human IRES Atlas: an integrative platform for studying IRES-driven translational regulation in humans. Database 2021, 2021, baab025. 10.1093/database/baab025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Sato K.; Kato Y.; Hamada M.; Akutsu T.; Asai K. IPknot: fast and accurate prediction of RNA secondary structures with pseudoknots using integer programming. Bioinformatics 2011, 27, i85–i93. 10.1093/bioinformatics/btr215. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Vandivier L. E.; Anderson S. J.; Foley S. W.; Gregory B. D. The conservation and function of RNA secondary structure in plants. Annu. Rev. Plant Biol. 2016, 67, 463–488. 10.1146/annurev-arplant-043015-111754. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Yang T.-H.; Lin Y.-C.; Hsia M.; Liao Z.-Y. SSRTool: a web tool for evaluating RNA secondary structure predictions based on species-specific functional interpretability. Comput. Struct. Biotechnol. J. 2022, 20, 2473–2483. 10.1016/j.csbj.2022.05.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Dagenais P.; Girard N.; Bonneau E.; Legault P. Insights into RNA structure and dynamics from recent NMR and X-ray studies of the Neurospora Varkud satellite ribozyme. Wiley Interdiscip. Rev.: RNA 2017, 8, e1421 10.1002/wrna.1421. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Ma H.; Jia X.; Zhang K.; Su Z. Cryo-EM advances in RNA structure determination. Signal Transduction Targeted Ther. 2022, 7, 58. 10.1038/s41392-022-00916-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Rouskin S.; Zubradt M.; Washietl S.; Kellis M.; Weissman J. S. Genome-wide probing of RNA structure reveals active unfolding of mRNA structures in vivo. Nature 2014, 505, 701–705. 10.1038/nature12894. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Zuker M.; Stiegler P. Optimal computer folding of large RNA sequences using thermodynamics and auxiliary information. Nucleic Acids Res. 1981, 9, 133–148. 10.1093/nar/9.1.133. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. McCaskill J. S. The equilibrium partition function and base pair binding probabilities for RNA secondary structure. Biopolymers 1990, 29, 1105–1119. 10.1002/bip.360290621. [DOI] [PubMed] [Google Scholar]
  14. Bernhart S. H.; Hofacker I. L.; Will S.; Gruber A. R.; Stadler P. F. RNAalifold: improved consensus structure prediction for RNA alignments. BMC Bioinf. 2008, 9 (1), 474. 10.1186/1471-2105-9-474. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Glouzon J.-P. S.; Ouangraoua A. aliFreeFold: an alignment-free approach to predict secondary structure from homologous RNA sequences. Bioinformatics 2018, 34, i70–i78. 10.1093/bioinformatics/bty234. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Lorenz R.; Luntzer D.; Hofacker I. L.; Stadler P. F.; Wolfinger M. T. SHAPE directed RNA folding. Bioinformatics 2016, 32, 145–147. 10.1093/bioinformatics/btv523. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Hollams E. M.; Giles K. M.; Thomson A. M.; Leedman P. J. MRNA stability and the control of gene expression: implications for human disease. Neurochem. Res. 2002, 27, 957–980. 10.1023/A:1020992418511. [DOI] [PubMed] [Google Scholar]
  18. Draper D. E. A guide to ions and RNA structure. RNA 2004, 10, 335–343. 10.1261/rna.5205404. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Jabbari H.; Wark I.; Montemagno C. RNA secondary structure prediction with pseudoknots: contribution of algorithm versus energy model. PLoS One 2018, 13, e0194583 10.1371/journal.pone.0194583. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Zhao Q.; Zhao Z.; Fan X.; Yuan Z.; Mao Q.; Yao Y. Review of machine learning methods for RNA secondary structure prediction. PLoS Comput. Biol. 2021, 17, e1009291 10.1371/journal.pcbi.1009291. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Singh J.; Hanson J.; Paliwal K.; Zhou Y. RNA secondary structure prediction using an ensemble of two-dimensional deep neural networks and transfer learning. Nat. Commun. 2019, 10, 5407. 10.1038/s41467-019-13395-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Sato K.; Akiyama M.; Sakakibara Y. RNA secondary structure prediction using deep learning with thermodynamic integration. Nat. Commun. 2021, 12 (1), 941. 10.1038/s41467-021-21194-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Szikszai M.; Wise M.; Datta A.; Ward M.; Mathews D. H. Deep learning models for RNA secondary structure prediction (probably) do not generalize across families. Bioinformatics 2022, 38, 3892–3899. 10.1093/bioinformatics/btac415. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kerpedjiev P.; Hammer S.; Hofacker I. L. Forna (force-directed RNA): simple and effective online RNA secondary structure diagrams. Bioinformatics 2015, 31, 3377–3379. 10.1093/bioinformatics/btv372. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Lorenz R.; Bernhart S. H.; Höner zu Siederdissen C.; Tafer H.; Flamm C.; Stadler P. F.; Hofacker I. L. ViennaRNA Package 2.0. Algorithms Mol. Biol. 2011, 6, 1–14. 10.1186/1748-7188-6-26. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Lu Z. J.; Gloor J. W.; Mathews D. H. Improved RNA secondary structure prediction by maximizing expected pair accuracy. RNA 2009, 15, 1805–1813. 10.1261/rna.1643609. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Bellaousov S.; Mathews D. H. ProbKnot: fast prediction of RNA secondary structure including pseudoknots. RNA 2010, 16, 1870–1880. 10.1261/rna.2125310. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Deng F.; Ledda M.; Vaziri S.; Aviran S. Data-directed RNA secondary structure prediction using probabilistic modeling. RNA 2016, 22, 1109–1119. 10.1261/rna.055756.115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Reuter J. S.; Mathews D. H. RNAstructure: software for RNA secondary structure prediction and analysis. BMC Bioinf. 2010, 11 (1), 129. 10.1186/1471-2105-11-129. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Yang T.-H.; Shiue S.-C.; Chen K.-Y.; Tseng Y.-Y.; Wu W.-S. Identifying piRNA targets on mRNAs in C. elegans using a deep multi-head attention network. BMC Bioinf. 2021, 22, 503. 10.1186/s12859-021-04428-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Yang T.-H.; Chen J.-C.; Lee Y.-H.; Lu S.-Y.; Wu S.-H.; Chang F.-Y.; Huang Y.-C.; Lee M.-H.; Tseng Y.-Y.; Wu W.-S. Identifying human miRNA target sites via learning the interaction patterns between miRNA and mRNA segments. J. Chem. Inf. Model. 2024, 64, 2445–2453. 10.1021/acs.jcim.3c01150. [DOI] [PubMed] [Google Scholar]
  32. Yang T.-H.; Yang Y.-C.; Tu K.-C. regCNN: identifying Drosophila genome-wide cis-regulatory modules via integrating the local patterns in epigenetic marks and transcription factor binding motifs. Comput. Struct. Biotechnol. J. 2022, 20, 296–308. 10.1016/j.csbj.2021.12.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Yang T.-H.; Yu Y.-H.; Wu S.-H.; Zhang F.-Y. CFA: An explainable deep learning model for annotating the transcriptional roles of cis-regulatory modules based on epigenetic codes. Comput. Biol. Med. 2023, 152, 106375. 10.1016/j.compbiomed.2022.106375. [DOI] [PubMed] [Google Scholar]
  34. He K.; Zhang X.; Ren S.; Sun J.. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016; pp 770–778, 10.1109/cvpr.2016.90. [DOI]
  35. Vaswani A.; Shazeer N.; Parmar N.; Uszkoreit J.; Jones L.; Gomez A. N.; Kaiser Ł.; Polosukhin I. Attention is all you need. Adv. Neur. 2017, 30, 30. [Google Scholar]
  36. Lorenz R.; Hofacker I. L.; Stadler P. F. RNA folding with hard and soft constraints. Algorithms Mol. Biol. 2016, 11, 8. 10.1186/s13015-016-0070-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Deigan K. E.; Li T. W.; Mathews D. H.; Weeks K. M. Accurate SHAPE-directed RNA structure determination. Proc. Natl. Acad. Sci. U.S.A. 2009, 106, 97–102. 10.1073/pnas.0806929106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Abu-Mostafa Y. S.; Magdon-Ismail M.; Lin H.-T.. Learning from Data; AMLBook: New York, 2012; Vol. 4. [Google Scholar]
  39. Danaee P.; Rouches M.; Wiley M.; Deng D.; Huang L.; Hendrix D. bpRNA: large-scale automated annotation and analysis of RNA secondary structure. Nucleic Acids Res. 2018, 46, 5381–5394. 10.1093/nar/gky285. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Nawrocki E. P.; Burge S. W.; Bateman A.; Daub J.; Eberhardt R. Y.; Eddy S. R.; Floden E. W.; Gardner P. P.; Jones T. A.; Tate J.; Finn R. D. Rfam 12.0: updates to the RNA families database. Nucleic Acids Res. 2015, 43, D130–D137. 10.1093/nar/gku1063. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Fu L.; Niu B.; Zhu Z.; Wu S.; Li W. CD-HIT: accelerated for clustering the next-generation sequencing data. Bioinformatics 2012, 28, 3150–3152. 10.1093/bioinformatics/bts565. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Löwes B.; Chauve C.; Ponty Y.; Giegerich R. The BRaliBase dent—a tale of benchmark design and interpretation. Brief. Bioinform. 2017, 18 (2), 306–311. 10.1093/bib/bbw022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Kalvari I.; Nawrocki E. P.; Ontiveros-Palacios N.; Argasinska J.; Lamkiewicz K.; Marz M.; Griffiths-Jones S.; Toffano-Nioche C.; Gautheret D.; Weinberg Z.; Rivas E.; Eddy S. R.; Finn R.; Bateman A.; Petrov A. I. Rfam 14: expanded coverage of metagenomic, viral and microRNA families. Nucleic Acids Res. 2021, 49, D192–D200. 10.1093/nar/gkaa1047. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Hofacker I. L.; Priwitzer B.; Stadler P. F. Prediction of locally stable RNA secondary structures for genome-wide surveys. Bioinformatics 2004, 20, 186–190. 10.1093/bioinformatics/btg388. [DOI] [PubMed] [Google Scholar]
  45. Wu Y.; Shi B.; Ding X.; Liu T.; Hu X.; Yip K. Y.; Yang Z. R.; Mathews D. H.; Lu Z. J. Improved prediction of RNA secondary structure by integrating the free energy model with restraints derived from experimental probing data. Nucleic Acids Res. 2015, 43, 7247–7259. 10.1093/nar/gkv706. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Rivas E.; Eddy S. R. A dynamic programming algorithm for RNA structure prediction including pseudoknots 1 1Edited by I. Tinoco. J. Mol. Biol. 1999, 285, 2053–2068. 10.1006/jmbi.1998.2436. [DOI] [PubMed] [Google Scholar]
  47. Ren J.; Rastegari B.; Condon A.; Hoos H. H. HotKnots: heuristic prediction of RNA secondary structures including pseudoknots. RNA 2005, 11, 1494–1504. 10.1261/rna.7284905. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Jabbari H.; Condon A. A fast and robust iterative algorithm for prediction of RNA pseudoknotted secondary structures. BMC Bioinf. 2014, 15, 147. 10.1186/1471-2105-15-147. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Hajdin C. E.; Bellaousov S.; Huggins W.; Leonard C. W.; Mathews D. H.; Weeks K. M. Accurate SHAPE-directed RNA secondary structure modeling, including pseudoknots. Proc. Natl. Acad. Sci. U.S.A. 2013, 110, 5498–5503. 10.1073/pnas.1219988110. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Will S.; Otto C.; Miladi M.; Möhl M.; Backofen R. SPARSE: quadratic time simultaneous alignment and folding of RNAs without sequence-based heuristics. Bioinformatics 2015, 31, 2489–2496. 10.1093/bioinformatics/btv185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Will S.; Joshi T.; Hofacker I. L.; Stadler P. F.; Backofen R. LocARNA-P: accurate boundary prediction and improved detection of structural RNAs. RNA 2012, 18, 900–914. 10.1261/rna.029041.111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Ji Y.; Xu X.; Stormo G. D. A graph theoretical approach for predicting common RNA secondary structure motifs including pseudoknots in unaligned sequences. Bioinformatics 2004, 20, 1591–1602. 10.1093/bioinformatics/bth131. [DOI] [PubMed] [Google Scholar]
  53. Tabei Y.; Kiryu H.; Kin T.; Asai K. A fast structural multiple alignment method for long RNA sequences. BMC Bioinf. 2008, 9 (1), 33. 10.1186/1471-2105-9-33. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Sweeney B. A.; Petrov A. I.; Burkov B.; Finn R. D.; Bateman A.; Szymanski M.; Karlowski W. M.; Gorodkin J.; Seemann S. E.; Cannone J. J.; et al. RNAcentral: a hub of information for non-coding RNA sequences. Nucleic Acids Res. 2019, 47, D221–D229. 10.1093/nar/gky1034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Singh J.; Paliwal K.; Zhang T.; Singh J.; Litfin T.; Zhou Y. Improved RNA secondary structure and tertiary base-pairing prediction using evolutionary profile, mutational coupling and two-dimensional transfer learning. Bioinformatics 2021, 37, 2589–2600. 10.1093/bioinformatics/btab165. [DOI] [PubMed] [Google Scholar]
  56. Yang E.; Zhang H.; Zang Z.; Zhou Z.; Wang S.; Liu Z.; Liu Y. GCNfold: A novel lightweight model with valid extractors for RNA secondary structure prediction. Comput. Biol. Med. 2023, 164, 107246. 10.1016/j.compbiomed.2023.107246. [DOI] [PubMed] [Google Scholar]
  57. Fu L.; Cao Y.; Wu J.; Peng Q.; Nie Q.; Xie X. UFold: fast and accurate RNA secondary structure prediction with deep learning. Nucleic Acids Res. 2022, 50, e14 10.1093/nar/gkab1074. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Chen C.-C.; Chan Y.-M. REDfold: accurate RNA secondary structure prediction using residual encoder-decoder network. BMC Bioinf. 2023, 24 (1), 122. 10.1186/s12859-023-05238-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Chen X.; Li Y.; Umarov R.; Gao X.; Song L. RNA secondary structure prediction by learning unrolled algorithms. arXiv 2020, 2002.05810. 10.48550/arXiv.2002.05810. [DOI] [Google Scholar]
  60. Ke Y.; Rao J.; Zhao H.; Lu Y.; Xiao N.; Yang Y. Accurate prediction of genome-wide RNA secondary structure profile based on extreme gradient boosting. Bioinformatics 2020, 36, 4576–4582. 10.1093/bioinformatics/btaa534. [DOI] [PubMed] [Google Scholar]
  61. Lemieux S.; Major F. RNA canonical and non-canonical base pairing types: a recognition method and complete repertoire. Nucleic Acids Res. 2002, 30, 4250–4263. 10.1093/nar/gkf540. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

The implemented DEBFold pipeline and the processed RNA structure ground-truth data sets (including the training-validation set, TestSetα, TestSetβ, and TestSetγ) are available at https://cobis.bme.ncku.edu.tw/DEBFold and https://github.com/cobisLab/DEBFold.

Articles from Journal of Chemical Information and Modeling are provided here courtesy of American Chemical Society

RESOURCES