XGBoost framework with feature selection for the prediction of RNA N5-methylcytosine sites

Abstract

5-methylcytosine (m5C) is indeed a critical post-transcriptional alteration that is widely present in various kinds of RNAs and is crucial to the fundamental biological processes. By correctly identifying the m5C-methylation sites on RNA, clinicians can more clearly comprehend the precise function of these m5C-sites in different biological processes. Due to their effectiveness and affordability, computational methods have received greater attention over the last few years for the identification of methylation sites in various species. To precisely identify RNA m5C locations in five different species including Homo sapiens, Arabidopsis thaliana, Mus musculus, Drosophila melanogaster, and Danio rerio, we proposed a more effective and accurate model named m5C-pred. To create m5C-pred, five distinct feature encoding techniques were combined to extract features from the RNA sequence, and then we used SHapley Additive exPlanations to choose the best features among them, followed by XGBoost as a classifier. We applied the novel optimization method called Optuna to quickly and efficiently determine the best hyperparameters. Finally, the proposed model was evaluated using independent test datasets, and we compared the results with the previous methods. Our approach, m5C- pred, is anticipated to be useful for accurately identifying m5C sites, outperforming the currently available state-of-the-art techniques.

Graphical abstract

Chong and colleagues studied 5-methylcytosine (m5C), which is a crucial post-transcriptional modification that plays a vital role in various biological processes. Accurate identification of m5C methylation sites can provide insight into their functional significance. The proposed machine learning based model, m5C-pred, can accurately identify m5C sites in a range of species.

Introduction

Transfer RNAs (tRNAs), ribosomal RNAs (rRNAs), messenger RNAs (mR-NAs), and non-coding RNAs have been revealed to include more than 170 different types of RNA chemical modifications (RCMs).^1,2,3,4,5 Methyltransferase, RNA binding protein, and demethylase are three regulating components that affect RCMs.^1,6,7 One of the most significant mRNA modifications is 5-methylcytosine (m5C), yet it can be difficult to precisely identify these sites. RNA methyltransferase is responsible for catalyzing the addition of a methyl group to the cytosine ring’s fifth position, forming m5C, as shown in Figure 1. Researchers can more clearly comprehend the precise function of 5-cytosine methylation in biological processes by correctly identifying the m5C sites. Numerous experimental techniques, including bisulfite sequencing,^3,8Aza-IP,⁹ m5C-RIP,¹⁰ RBS-seq,¹¹ mi-CLIP,¹² and Oxford Nanopore Technologies,^13,14 have been designed to predict m5C sites in RNA. These techniques have shown some success in locating m5C sites in various species. However, these techniques are overpriced and laborious and typically fall short of correctly identifying m5C sites due to the unstable nature of mRNA molecules.^7,9,15,16 Alternatively, computational may be able to offer a quicker and more affordable method for identifying m5C modification sites.^17,18,19,20

Figure 1.

Illustration of 5-methylcytosine (m5C) modification

Numerous computational techniques have been created until now to identify m5C locations in many species, including Homo sapiens (H. sapiens), Arabidopsis thaliana (A. thaliana), Mus musculus (M. musculus), and Saccharomyces cerevisiae (S. cerevisiae). Feng et al.²¹ developed the first computational model based on support vector machine (SVM) to locate m5C sites, utilizing pseudo k-tuple nucleotide composition (PseKNC) encoding, relying on the experimentally proven m5C sites in H. sapiens. Subsequently, Qiu et al.²² created the iRNAm5C-PseDNC predictor using random forest (RF) to identify m5C sites. After that, Zhang et al.²³ created another model, named m5CHPCR, by employing ensemble learning techniques. Using the same datasets as those of Feng et al.²¹ and Zhang et al.,²³ Sabooh et al.²⁴ created a new model using SVM once more by combining composite encoding features such as di-nucleotide composition, trinucleotide composition (TNC), and tetra-nucleotide composition. A new model named PEA-m5C was proposed by Song et al.²⁵ using RF, primarily concerned with predicting m5C sites in A. thaliana only. After prediction, they tested their model using independent test datasets. For the prediction of m5C locations in eight different cells of H. sapiens and M. musculus, Li et al.²⁶ compiled datasets from the Gene Expression Omnibus (GEO) database and created a web server using an RF classifier and named it RNAm5Cfinder. Lv et al.²⁷ proposed another model named iRNA-m5C using RF and multiple feature extraction techniques including mono-nucleotide binary encoding, natural vector, K-tuple nucleotide frequency component, and PseKNC for the prediction of m5C sites in H. sapiens, A. thaliana, M. musculus, and S. cerevisiae. Recently, Chai et al.²⁸ proposed a technique using stacked ensemble approach for m5C-site prediction in A. thaliana and M. musculus species, outperforming the previous techniques. They used five different machine learning classifiers including XGBoost, SVM, LightGBM, ExtraTree, and GBDT along with logistic regression as the meta-classifier. They utilized mlxtend-package for model stacking. Although m5C sites in the RNA sequences may be recognized well using the stated approaches, it is still feasible that the performance could be enhanced. With advances in different feature extraction and network optimization techniques, we can achieve better performances as we still have a lot of room for improvement available in these species. In this work, we proposed a new model, m5C-pred, to predict m5C sites in the RNA sequences of five distinct species: H. sapiens, A. thaliana, M. musculus, D. melanogaster, and D. rerio. As the foundation for the development of m5C-pred, five different feature extraction techniques were used, including composition of k-spaced nucleic acid pairs (CKSNAP), nucleotide chemical property (NCP), label encoding (LE), electron-ion interaction pseudopotentials of trinucleotide (PseEIIP), and enhanced nucleic acid composition (ENAC). After combining the features of the mentioned techniques, we used SHAP (SHapley Additive exPlanations) for selecting the best features. Our approach was developed using XGBoost classifier based on the chosen feature subset by SHAP. Finally, the effectiveness of our methodology was evaluated against the previous approaches. The outcomes demonstrated that our strategy provides significantly superior performance compared with these available methods.

Results

Predictive performance using different classifiers

We employed three different machine learning classifiers, including XGBoost, light gradient boosting machine (LGBM), and SVM, and based on the chosen encoding strategies and after picking the best features using SHAP, we fine-tuned the parameters using Optuna as stated above. Using the optimized parameters, we calculated the predictive performance of all three classifiers on H. sapiens, M. musculus, A. thaliana, D. melanogaster, and D. rerio using 10-fold cross-validation and on independent test datasets, respectively. Table S7 shows the comparison of the achieved results, which clearly illustrates that the XGBoost classifier is performing the best among the three.

Furthermore, to show the importance of feature selection and hyperparameter optimization, we compared the results using original data, hyperparameter optimized data without SHAP, and hyperparameter optimized data with SHAP on the best performing classifier, XGBoost, as presented in Figure 4. We can clearly see the performance improvement both in the case of 10-fold cross-validation and independent testing after doing the feature selection using SHAP and hyperparameter optimization using Optuna. Compared with the original data (without SHAP and hyperparameter optimization), the accuracies and Matthew’s correlation coefficients (MCC) of H. sapiens improved by 4.75% and 9.2%, respectively, on 10-fold cross-validation and 3.65% and 7.6%, respectively, on independent test data. Similarly, M. musculus improved by 2.56% and 5.13%, respectively, on 10-fold cross-validation and 2.35% and 4.8%, respectively, on independent test data, and A. thaliana improved by 3.35% and 6.94%, respectively, on 10-fold cross-validation and 4.05% and 8.1%, respectively, on independent test data. While the improvement for D. melanogaster, and D. rerio species is also noticeable, as mentioned in Table S7.

Figure 4.

Predictive performance comparison for H. sapiens, M. musculus, and A. thaliana on 10-fold cross-validation and independent testing using XGBoost

Comparison with previous best models

We assessed and compared the achieved results with the previous state-of- the-art techniques including iRNA-m5C,²⁷ m5CPred-SVM,²⁹ m5Cpred- XS,³⁰ RNAm5Cfinder,²⁶ PEA-m5C,²⁵ and Staem5.²⁸ The results of iRNA-m5C for H. sapiens and M. musculus were calculated using the available webserver, and all the remaining results are directly quoted from the papers. Since Staem5 and PEA-m5C have not used the H. sapiens dataset, we have not included them for H. sapiens comparison. The M. musculus dataset was also not included by PEA-m5C, so we were unable to include its comparison. Further, in the case of A. thaliana species, RNAm5Cfinder was ignored for the same reason. The independent test data were kept blind during the training process; hence it has been utilized for the comparison of the models to make sure that the comparison remains unbiased.

Compared with the m5CPred-SVM²⁹ for H. sapiens, the accuracy and MCC have been improved from 77.5% to 55.1%–81.15% and 62.7%, respectively. Similarly, compared with the previous model Staem5,²⁸ the accuracy has been improved from 71.95% to 73.55% for M. musculus and 73.7% to 76.15% for A. thaliana, while the improvement for MCC is noted to be enhanced from 44.2% to 47.3% for M. musculus and from 47.4% to 52.3% for A. thaliana. Further, while comparing with m5Cpred-XS,³⁰ the accuracies of H. sapiens and M. musculus have been improved from 80.4% to 81.16% and 62.0% to 62.7%, while the MCCs improved from 72.3% to 73.55% and 46.0% to 47.3%, respectively. Whereas the performance on A. thaliana was slightly lower; i.e., the accuracy and MCC achieved are 76.15% and 52.3% compared with 77.2% and 54.5% being achieved by m5Cpred-XS. Figures 5, 6, and 7 show a comparison of the proposed model with all the previous best models for H. sapiens, M. musculus, and A. thaliana, in terms of accuracy (Acc), MCC, sensitivity (Sn), specificity (Sp), and area under the curve (AUC). Missing values in these graphs are either not calculated by previous tools or the values are too small to be shown in the figure. As D. melanogaster and D. rerio species datasets are newly collected in this study, in the literature, there is no computational tool available for the purpose of comparison. However, Table S7 shows the achieved results by the proposed model on these species. All findings reveal that the proposed method is more accurate in predicting m5C sites than previous state-of-the-art methods that are already in use.

Figure 5.

Comparison of H. sapiens with existing state-of-the-art methods on independent test datasets

Figure 6.

Comparison of M. musculus with existing state-of-the-art methods on independent test datasets

Figure 7.

Comparison of A. thaliana with existing state-of-the-art methods on independent test datasets

Cross-species validation

Since we have datasets from various species for training, it is essential to show that the model trained on a specific species could predict the m5C locations in another species. Therefore, we tested all other species using the independent test datasets while training on a specific species, and we present the results (accuracies) using the heatmap diagram in Figure 8.

Figure 8.

Cross-species accuracies shown in percentages

The model was trained on the specific dataset mentioned on the y axis and tested on the specific independent dataset mentioned on the x axis.

Here, we can see that while training on the M. musculus dataset, the testing using H. sapiens is good enough because mouse and human share maximum gene similarity, and the number of sequences in the M. musculus dataset is also much higher compared to the H. sapiens dataset. For the remaining, the model tested for the species itself is always able to achieve the highest accuracy. Such results illustrate that the model holds the ability to learn the insight genomic features that can be used to classify methylated and non-methylated sequences.

Discussion

In the proposed architecture, we tested nine different feature encoding techniques, including CKSNAP, NCP, Kmer (k = 2), electron-ion interaction pseudopotentials value (EIIP), LE, electron-ion interaction pseudopotentials of trinucleotide (PseEIIP), binary, TNC, and ENAC. After trying different combinations, we found that the selected feature set acquired from the integrated encoding schemes (including CKSNAP, NCP, LE, PseEIIP, and ENAC) performs the best. These five encoding schemes belong to three major groups, in which CKSNAP, NCP, and ENAC belong to nucleic acid composition, PseEIIP belongs to EIIP, and the last, LE, belongs to assigning the nucleotides with labels. The best features after the integration of these five encoding techniques were extracted using SHAP and three different classifiers were applied, including XGBoost, LGBM, and SVM, and their hyperparameters were tuned using Optuna.³¹ Figure 2 shows the complete architecture of the proposed framework.

Figure 2.

Overview of m5C-pred, a prediction framework consisting of six major steps

(1) Dataset collection using GEO database. (2) Feature encoding using five different techniques. (3) Feature selection using SHAP. (4) Classification using XGBoost, using the Optuna-based optimized parameters. (5) Evaluation of the model. (6) Webserver creation.

Feature selection

We employed SHAP³² as a feature selection technique to eliminate unnecessary features brought on by high-dimensional input data features and to identify the ideal feature subset. SHAP introduced a metric to calculate feature importance, which assigns every feature a value based on how important it is and how it would affect the learning of the network. Assuming we have F number of total features, the Shapley value of a particular feature f can be calculated as

$$\phi f\left(p\right)=\sum _{S\subseteq \frac{N}{f}}\frac{S|!\left(F-\left|S\right|-1\right)!}{F!}\left(p\left(S\cup F\right)-p\left(S\right)\right)\text{,}$$ (Equation 1)

where p is the prediction of proposed model, and S is the feature subset without the feature f.

This technique calculates the output prediction of the proposed model with and without that particular feature f, and afterward, it evaluates the variance among them.

$$importanceoff=p\left(withf\right)-p\left(withoutf\right)$$ (Equation 2)

The total number of features concatenating CKSNAP, NCP, LE, PseEIIP, and ENAC becomes 472. Using SHAP, we calculated the importance of each feature and removed all the features having zero importance in the prediction, which gives a distinct amount of features for different species. The number of features used for training the model of H. sapiens, M. musculus, and A. thaliana were 256, 454, and 468, respectively. Reducing the number of features not only speed up the prediction process but also helped us to optimize the network and achieve higher performance. Consequently, the SHAP technique has the appealing ability to deliver predictions that are easy to understand. Figure 3 depicts the top 20 features based on the importance values calculated using SHAP for H. sapiens, M. musculus, and A. thaliana, respectively.

Figure 3.

20 features with the highest importance value calculated by SHAP algorithm for H. sapiens, M. musculus, and A. thaliana

Feature 1 to feature 96 represents CKSNAP, feature 97 to feature 219 represents NCP, feature 220 to feature 260 represents LE, feature 261 to feature 324 represents PseEIIP, and feature 325 to feature 472 represents ENAC.

Hyperparameter optimization using Optuna

Hyperparameters are the factors that regulate how a machine or deep learning model learns, and their optimization is one of the most fundamental techniques for enhancing the performance of computational models. Traditional methods for hyperparameter optimization, such as grid search, have been extremely time-consuming and difficult to locate minima upon expanding the hyperparameter space and the volume of data. In order to quickly and effectively perform hyperparameter optimization, Akiba et al.³¹ developed an approach called Optuna. Optuna optimizes hyperparameters by combining sampling and pruning algorithms. Optuna enables the dynamic creation and manipulation of hyperparameter search spaces. The discontinuation of uninspiring trials while hyperparameter tuning is referred to as a pruning process. It checks the learning curves of each trial on a regular basis. It then finds the set of parameters that are unlikely to produce a satisfactory outcome and that should be avoided. In such a manner, the best set of parameters is identified to be used by the prediction model. After feature selection, we used this approach to further optimize the network. The parameters and their ranges of each classifier being used to optimize the network are shown in Table S1, while Tables S2–S6 show the best-selected parameters by Optuna for H. sapiens, M. musculus, A. thaliana, D. melanogaster, and D. rerio, respectively.

Materials and methods

Benchmark datasets

For developing and testing any machine or deep learning model, high-quality benchmark datasets are very crucial. In this work, high-quality m5C datasets of five different species including H. sapiens, M. musculus, and A. thaliana were compiled from the literature. The datasets of H. sapiens were constructed by Khoddami et al.¹¹ using the GEO database with the accession number GSE:93751, M. musculus by Yang et al.⁷ using GSE:93751, and A. thaliana by Lv et al.²⁷ using GSE:94065, respectively. Further, for D. melanogaster and D. rerio species, we collected data from m5C-Atlas.³³ Identical sequences having more than 70% similarity were eliminated from the datasets using the CD-HIT tool³⁴ to prevent bias between sequences.

Typically, the benchmark dataset is split into two sections, training data and independent test data. Table S9 summarizes the number of positive and negative samples existing in each training and testing dataset.

Encoding techniques

Feature encoding is incredibly important in the process of building any machine or deep learning model. Five different encoding schemes were selected in this study to make the data readable for the machine, as shown below.

Composition of k-spaced nucleic acid pairs

The CKSNAP feature encoding determines how frequently nucleic acid pairs are separated by any k nucleic acids, where k-value ranges from 0 to 5. For instance, if we take k = 0, we acquire 16 nucleic acid pairs with zero spacing (i.e., AA, AG, AC, CT, CC, CA, TG, AT, CG, GC, GG, GT, TA, TT, GA, TC). A feature vector can thus be expressed as follows:

$${\left(\frac{N_{AA}}{N_{total}},\frac{N_{AU}}{N_{total}},\frac{N_{AC}}{N_{total}},\dots \dots .\frac{N_{TT}}{N_{total}}\right)}_{16}$$ (Equation 3)

The value of each descriptor indicates the makeup of the associated nucleic acid pair in the nucleotide sequence. Here, $N_{total}$ stands for the overall number of evenly spaced nucleic acid pairs in the provided sequence, and NAA stands for the frequency of the nucleic acid pair AA. For a sequence of length L, the values of $N_{total}$ are, L − 1, L − 2, L − 3, L − 4, L – 5, and L −6 for k = 0, 1, 2, 3, 4, and 5, respectively.

Nucleotide chemical property

Adenine (A), cytosine (C), guanine (G), and uracil (U) are the four types of nucleotides that make up the RNA. Every nucleotide has a unique chemical binding and a particular chemical structure. According to these chemical characteristics, the four main types of nucleotides can be divided into three categories, as indicated in Table S10.

According to their chemical characteristics, A can be expressed as (1, 1, 1), G as (1, 0, 0), C as (0, 1, 0), and U as (0, 0, 1).

Label encoding

LE is the process of transforming the nucleotides in numerical format to make them understandable for the network. We simply assigned 1, 2, 3, and 4 to A, U, C, and G respectively.

Electron-ion interaction pseudopotentials of trinucleotide

In EIIP the values of each nucleotide in RNA sequences are given as A: 0.1260, C: 0.1340, G: 0.0806, and U: 0.1335. Assuming EIIP_A, EIIP_C, EIIP_G, and EIIP_U represent EIIP values for A, C, G, and U, the PseEIIP value of trinucleotides can be calculated as

$$V=\left[{EIIP}_{AAA.fAAA},{EIIP}_{AAG.fAAG},\dots .,{EIIP}_{TTT.fTTT}\right]$$ (Equation 4)

Here, fxyz stands for the nth trinucleotide’s normalized frequency, and EIIPxyz = EIIPx + EIIPy + EIIPz, where x, y, and z belong to the RNA nucleotides [A, C, G, or U].

Enhanced nucleic acid composition

The nucleic acid composition (NAC) encoding computes the occurrence of each nucleotide in a nucleotide sequence using the equation:

$$f\left(t\right)=\frac{N\left(x\right)}{N},xϵ\left\{A,C,G,orU\right\}$$ (Equation 5)

Here, N is the length of the sequence, and N(x) is the number of x nucleotides.

The ENAC uses a sequencing window with a specified length to determine the NAC, constantly sliding from 5′ to 3′ terminal. For a sequence of length L, with a window size W, the number of sliding windows will be L − W + 1, and the encoded vector dimension will be (L − W + 1) ∗ 4. To select viable size of window, we have computed results on different values of W. The achieved results are depicted in Table S8. The results show that W = 5 is a suitable value, as for different species, different values of W perform well, but W = 5 is constantly giving better results.

Performance evaluation metrics

Five metrics, including ACC, MCC, Sn, Sp, and AUC, are typically used for binary classification to assess how well the proposed model performs. These metrics can be mathematically defined as follows:

$$Acc=1-\frac{S_{-}^{+}+S_{+}^{-}}{S^{+}+S^{-}}$$ (Equation 6)

$$MCC=\frac{1-\left(\frac{S_{-}^{+}}{S^{+}}+\frac{S_{+}^{-}}{S^{-}}\right)}{\sqrt{\left(1+\frac{S_{+}^{-}-S_{-}^{+}}{S^{+}}\right)\left(1+\frac{{S_{-}^{+}-S}_{+}^{-}}{S^{-}}\right)}}$$ (Equation 7)

$$Sn=1-\frac{S_{-}^{+}}{S^{+}}$$ (Equation 8)

$$Sp=1-\frac{S_{+}^{-}}{S^{-}}$$ (Equation 9)

Here, S⁺ and S⁻ show the total number of positive and negative sequences, while S⁺ shows the incorrectly predicted positive class, and S⁻ is the incorrectly predicted negative class.

Data availability

A friendly web server is made available for researchers at http://nsclbio.jbnu.ac.kr/tools/m5C-pred/, and the data used in this study along with the codes are made available through GitHub: https://github.com/Z-Abbas/m5C-pred.