Imbalanced multi-label learning for identifying antimicrobial peptides and their functional types

Abstract

Motivation: With the rapid increase of infection resistance to antibiotics, it is urgent to find novel infection therapeutics. In recent years, antimicrobial peptides (AMPs) have been utilized as potential alternatives for infection therapeutics. AMPs are key components of the innate immune system and can protect the host from various pathogenic bacteria. Identifying AMPs and their functional types has led to many studies, and various predictors using machine learning have been developed. However, there is room for improvement; in particular, no predictor takes into account the lack of balance among different functional AMPs.

Results: In this paper, a new synthetic minority over-sampling technique on imbalanced and multi-label datasets, referred to as ML-SMOTE, was designed for processing and identifying AMPs’ functional families. A novel multi-label classifier, MLAMP, was also developed using ML-SMOTE and grey pseudo amino acid composition. The classifier obtained 0.4846 subset accuracy and 0.16 hamming loss.

Availability and Implementation: A user-friendly web-server for MLAMP was established at http://www.jci-bioinfo.cn/MLAMP.

Contacts: linweizhong@jci.edu.cn or xudong@missouri.edu

1 Introduction

With rapid increase in the infection resistance of antibiotics, it is urgent to find novel infection therapeutics. Over the past decade antimicrobial peptides (AMPs) have been utilized as potential alternatives for fighting infectious diseases. AMPs are key components of the innate immune system and can protect the host from various pathogenic bacteria. In invertebrates and vertebrates, AMPs have dual roles: rapid microbial killing and subsequent immune modulation (Wang, 2014). These effects result from AMP inducing multiple damages in bacteria by disrupting bacteria membranes (Malmsten, 2014), by inhibiting proteins, DNA and RNA synthesis, or by interacting with certain intracellular targets (Bahar and Ren, 2013). Therefore, AMPs were developed increasingly for new drugs. Some examples of using AMPs in therapeutics have been reported. Popovic et al. (2012) found that peptides with antimicrobial and anti-inflammatory activities had therapeutic potential for treatment of acne vulgaris. Yancheva et al. (2012) synthesized a novel didepsipeptide with antimicrobial activity against four of five tested bacterial strains of Escherichia coli. Conlon et al. demonstrated that peptides with antimicrobial activity from frog skin could stimulate insulin release, and hence had potential as an incretin-based therapy for Type 2 diabetes mellitus (Conlon et al., 2014). In addition, AMPs have been used as anticancer peptides in cancer therapy (Gaspar et al., 2013).

A surge in research on AMPs has promoted the development of various databases and prediction tools. APD2 (Wang et al., 2009) is a system dedicated to establishing a glossary, nomenclature, classification, information search, prediction, design, and statistics of AMPs. It gathered 2544 AMPs from the literature. CAMP (Thomas et al., 2010; Waghu et al., 2013) holds 6756 antimicrobial sequences and 682 3D structures of AMPs, together with prediction and sequence analysis tools. Niarchou et al. (2013) tested all subsequences ranging from 5 to 100 amino acids of the plant proteins in UniProKB/Swiss-prot and constructed an AMP database for plant species, named C-PAmP. Zhao et al. (2013) developed LAMP, a database used to aid the discovery and design of AMPs as new antimicrobial agents. The database contains 3904 natural AMPs and 1643 synthetic peptides. DBAASP was a manually curated database built by Gogoladze et al. (2014), and it collected those peptides for which antimicrobial activities against particular targets have been evaluated experimentally.

Generally, AMPs are short peptides with 10–50 amino acids (Malmsten, 2014) and have very low sequence homology to one another. So it is challenging to identify AMPs and its activities by automatic tools. Researchers made considerable efforts in this regard. In these studies, the support vector machine (SVM) was usually used as prediction engine (Joseph et al., 2012; Khosravian et al., 2013; Lata et al., 2010; Niarchou et al., 2013). Besides, nearest neighbor (Wang et al., 2011) or k-nearest neighbor algorithm (Xiao et al., 2013), random forests (RFs) (Joseph et al., 2012), decision tree model (Lira et al., 2013) and hidden Markov models (HMMs) (Fjell et al., 2007) were also applied as classifiers. Some predictors were only used to identify whether novel peptides are AMPs (Khosravian et al., 2013; Lata et al., 2007; Vishnepolsky and Pirtskhalava, 2014; Wang et al., 2011). In addition to these simple binary classifiers, there were some multi-class classifiers. Lira et al. (2013) created a decision tree model to classify the antimicrobial activities of synthetic peptides into four classes: none, low, medium and high. Joseph developed ClassAMP to predict the propensity of a peptide sequence to have antibacterial, antifungal, or antiviral activity (Joseph et al., 2012). Khamis et al. studied 14 AMP families and sub-families. They selected a specific description of AMP amino acid sequence, and identified compositional and physicochemical properties of amino acids to distinguish each AMP family (Khamis et al., 2015). Furthermore, Xiao et al. (2013) proposed a two-level multi-label classifier, iAMP-2L, which identifies not only whether a peptide is an AMP, but also its functional activities.

Although these methods have their own advantages and did play an important role in the research, they have following problems. First, most models only identified whether a new sequence is AMP, but not its type. Second, it is hard to search short peptides in the database because AMPs usually have only 5–50 amino acids. Methods based on Blast search and gene ontology (Lin et al., 2013) are often ineffective. Last but not least, classifying AMPs’ functions is a multi-label classification (MLC), especially when the number of AMPs with different activities does not distribute evenly. From APD2 (Wang et al., 2009), it is seen that antibacterial peptides occupy more than 90% of all AMPs, which is a highly unbalanced MLC. None of aforementioned automatic models considered the unbalanced amounts among various activities.

In the past two decades, the topic of learning from multi-label datasets (MLDs) has drawn significant attention from researchers. Moreover, MLC methods are increasingly applied in various fields, such as semantic annotation of images (Zhang and Zhou, 2007), categorization (Liu and Chen, 2015) and bioinformatics (Chou, 2015; Chou and Shen, 2007, 2008; Chou et al., 2011, 2012, 2014; Sadasivam and Duraiswamy, 2015; Shen and Chou, 2007, 2009, 2010a,b; Wu et al., 2012; Yu et al., 2013). The existing MLC methods can be grouped into two categories: (i) problem transformation methods, which transform MLC either into one or more single-label classification or regression problem, and (ii) algorithm adaptation methods, which extend specific learning algorithms in order to handle MLDs directly (Tsoumakas et al., 2010). Numerous MLC algorithms were proposed, such as Adaboost.MH and Adaboost.MR (Schapire and Singer, 2000), ML-KNN (Zhang and Zhou, 2007), Classifier chains (Read et al., 2009, 2011) and Multi-label Naïve Bayes (Zhang et al., 2009).

MLC often has serious issues of unbalanced datasets, in which the numbers of samples from minority classes are substantially fewer than from majority classes. For example, in subcellular localization prediction (Lin et al., 2013), the number of the cytoplasm proteins is 44 times the number of the melanosome proteins. A similar situation occurs in many studies (Liu and Chen, 2015; Wan et al., 2012; Wu et al., 2011; Xiao et al., 2011). Standard machine learning algorithms often cannot achieve ideal performance when trained on unbalanced dataset. One approach to address this issue is to adapt existing classifier learning algorithms to strengthen learning with regard to the minority class (Xu et al., 2013). Another approach is to artificially sample the class distribution (Dong and Wang, 2011; Luengo et al., 2011; Tahir et al., 2012). Combining both approaches can also achieve strong classifiers (Zhang et al., 2012). Unquestionably, the sampling approach continues to be popular (Chawla, 2010). Various over- and under-sampling methods have been proposed (Bunkhumpornpat et al., 2009; Chawla, 2010; Chawla et al., 2002; Dong and Wang, 2011; Gao et al., 2011a,b; Luengo et al., 2011; Seiffert et al., 2008; Zhang et al., 2012). Among them, SMOTE (Synthetic Minority Oversampling TEchinque) (Chawla, 2010; Chawla et al., 2002) is a state-of-art over-sampling methods. Chawla (2010) argued that SMOTE creates effective regions for learning the minority class rather than being subsumed by the majority class samples around them. In bioinformatics, some studies have applied SMOTE to balance the skewing benchmark datasets (Jia et al., 2016a,b; Liu et al., 2015; Xiao et al., 2015). In addition, similar approaches have been recently introduced to handle the unbalanced datasets, such as Monte Carlo sampling (Jia et al., 2016c) and fusion ensemble approach (Liu et al., 2016a; Qiu et al., 2016).

Although the aforementioned methods have some success in addressing unbalanced datasets, they have not achieved a satisfactory result in processing multi-labeled and imbalanced datasets simultaneously. Few works address the imbalance problem in MLC. He et al. (2012) took into account the imbalance in predicting subcellular localization of human proteins. Charte et al. (2015) built an under-sampling and oversampling algorithm on MLDs. Those studies improved the multi-label classification performance; however, they have some drawbacks in how to address the multi-label character of the new synthetic instance. In this paper, we tackle the imbalanced problem by a novel oversampling model referred to as ML-SMOTE, which is a synthetic minority oversampling on MLDs. We developed a new tool as a two-level AMP predictor based on ML-SMOTE. For a peptides sequence, we first identify whether it is an AMP. If yes, we then predict what potential activities it has. The first-level is a binary predictor, and the second-level predictor is an unbalanced and multi-labeled multi-classes predictor. The result shows ML-SMOTE can adjust the label set distribution to improve the performance of the predictor.

2 Methods

2.1 Benchmark dataset

The benchmark dataset ${\mathbb{S}}^{Bench}$ used in this study was taken from Xiao et al. (2013). The dataset can be formulated as

$${\mathbb{S}}^{Bench}={\mathbb{S}}^{+}\cup {\mathbb{S}}^{-}$$ (1)

where ${\mathbb{S}}^{+}$ contains 879 AMPs, and ${\mathbb{S}}^{-}$ contains 2405 non-AMPs. The 879 AMPs are formulated as

$${\mathbb{S}}^{+}={\cup }_{i=1}^5{\mathbb{S}}_i$$ (2)

where ${\mathbb{S}}_1$ contains 770 antibacterial peptides, ${\mathbb{S}}_2$ 140 anti-cancer/tumor peptides, ${\mathbb{S}}_3$ 366 antifungal peptides, ${\mathbb{S}}_4$ 84 anti-HIV peptides and ${\mathbb{S}}_5$ 124 antiviral peptides.

2.2 Sequence encoding scheme

To develop a powerful method for classifying AMPs and their functional families according to the sequence information, one of the keys is to formulate the peptides with an effective mathematical expression that can truly reflect the intrinsic correlation with the target to be identified. However, when comparing with other protein functional predictions, the challenge is identifying how AMPs deal with shorter peptides. For a peptides sample P of L amino acids

$$P=R_1{R}_2{R}_3\dots R_L$$ (3)

where $R_i (1\le i\le L)$ represents the ith residue, L is usually between 5 and 50.

In this study, we formulated an amino acids sequence by using Chou’s PseAAC(Chou, 2001, 2005) with the grey model (GM) (Deng, 1989). According to Chou’s general PseAAC formula (Chou, 2009, 2011), the peptides P in Eq. 3 can be represented as

$$\mathbf{P}={\left[{\mathrm{p}}_1\mathrm{ }{\mathrm{p}}_2\mathrm{ }\cdots \mathrm{ }{\mathrm{p}}_{\mathrm{k}}\mathrm{ }\cdots \mathrm{ }{\mathrm{p}}_{\mathrm{\Omega }}\right]}^T$$ (4)

where T is a transpose operator, while the subscript Ω is an integer and its value as well as the components p₁, p₂, … depend on how to extract the desired information from the amino acid sequence of P.

In our study, we use the GM(1,1) model, which is an important and generally used model in GM. GM(1,1) firstly converts a series without any obvious regularity into a strict monotonic increasing series by using the accumulative generation operation (AGO). This process can reduce the randomness and enhance the smoothness of the series and minimize any interference from the random information. Let us assume that

$${\mathrm{X}}^{\left(0\right)}=\left(x^{\left(0\right)}\left(1\right),\mathrm{ }{x}^{\left(0\right)}\left(2\right),\mathrm{ }\dots ,\mathrm{ }{x}^{\left(0\right)}\left(n\right)\mathrm{ }\right)$$ (5)

is a non-negative original series of real numbers with an irregular distribution. Then

$${\mathrm{X}}^{\left(1\right)}=\left(x^{\left(1\right)}\left(1\right),\mathrm{ }{x}^{\left(1\right)}\left(2\right),\mathrm{ }\dots ,\mathrm{ }{x}^{\left(1\right)}\left(n\right)\mathrm{ }\right)$$ (6)

is viewed as the first-order accumulative generation operation (1-AGO) series for X⁽⁰⁾, and the components in X⁽¹⁾ are given by

$$x^{\left(1\right)}\left(k\right)=\mathrm{ }\sum _{i=1}^k{x}^{\left(0\right)}\left(i\right),\mathrm{ }k=\mathrm{1,2},\dots ,\mathrm{ }n$$ (7)

The GM(1,1) model can be expressed by the following grey differential equation with one variable:

$$\frac{d{\mathrm{X}}^{\left(1\right)}}{dt}+a{\mathrm{X}}^{\left(1\right)}=b$$ (8)

where $a$ and b are elements of parameters vector $\widehat{a}$, that is

$$\widehat{a}={[a,\mathrm{ }b]}^T$$ (9)

In Eq. 8, $-a$ is the developing coefficient and b the influence coefficient. They can be solved using a least square estimator.

$$\widehat{a}={\left[a,\mathrm{ }b\right]}^T={\left[B^{T}B\right]}^{-1}{B}^{T}Y$$ (10)

where

$$B=\left[\begin{array}{cc}\begin{array}{c}-0.5\left(x^{\left(1\right)}\left(1\right)+x^{\left(1\right)}\left(2\right)\right)\\ -0.5\left(x^{\left(1\right)}\left(2\right)+x^{\left(1\right)}\left(3\right)\right)\\ ⋮\end{array}& \begin{array}{c}1\\ 1\\ ⋮\end{array}\\ -0.5\left(x^{\left(1\right)}\left(n-1\right)+x^{\left(1\right)}\left(n\right)\right)& 1\end{array}\right]$$ (11)

$$Y=\left[\begin{array}{c}\begin{array}{c}{x}^{\left(0\right)}\left(2\right)\\ x^{\left(0\right)}\left(3\right)\end{array}\\ ⋮\\ x^{\left(0\right)}\left(n\right)\end{array}\right]$$ (12)

The coefficients $-a$ and b should carry some intrinsic information contained in the discrete data sequence X⁽⁰⁾ sampled from the system investigated. In view of this, we incorporate these coefficients into the general form of PseAAC (Eq. 4) to reflect the correlation between the peptide sequence and prediction labels. In order to translate an amino acid sequence expressed with alphabets in Eq. 3 into a non-negative real series in Eq. 5, we need the amino acid numerical codes. In the same manner as that shown in (Xiao et al., 2013), we also use the numerical value of the following five physical-chemical properties for each of the 20 amino acids: (1) hydrophobicity; (2) pk1 ($C^{\alpha }-\mathrm{C}\mathrm{O}\mathrm{O}\mathrm{H}$); (3) pk2 (NH3); (4) PI (25 ^°C); and (5) molecular weight. Finally, we used a 30-D features vector to represent a peptide; i.e. instead of Eq. 4, we now have

$$\mathbf{P}={\left[{\mathrm{p}}_1,\mathrm{ }{\mathrm{p}}_2,\mathrm{ }\dots ,\mathrm{ }{\mathrm{p}}_{20},\mathrm{ }{\mathrm{p}}_{21},\mathrm{ }\dots {\mathrm{p}}_{30}\right]}^T$$ (13)

where ${\mathrm{p}}_i (1\le i\le 20)$ are the frequencies of 20 amino acids; and p₂₁ and p₂₂ are the coefficients of Eq. 10 when amino acids are coded by hydrophobicity numerical values; p₂₃ and p₂₄ are the coefficients of Eq. 10 when amino acids are coded by pk1 numerical values, and so on.

2.3 ML-SMOTE algorithm

In Eq. 2, the AMP function family dataset is an unbalanced MLD, in which the antibacterial peptides have nine times the amount of the anti-HIV peptides. How to handle the MLC in unbalanced MLD is essential for improving prediction performance.

Let $X\subset R^m$ denote an m-dimensions real vector of instance and let

$$Y=\left\{l_1,l_2,\dots ,l_q\right\}$$ (14)

be a class label set. MLD can be represented as

$$D=\left\{\left(x,y\right)|\mathrm{ }x\in X,\mathrm{ }y\subseteq Y\right\}$$ (15)

We define the sample set with the j-th $\left(1\le j\le q\right)$ label as

$$D^{\left(j\right)}=\left\{\left(x^{\left(j\right)},y^{\left(j\right)}\right)\left|\mathrm{ }\right(x^{\left(j\right)},y^{\left(j\right)})\in D\mathrm{ }and\mathrm{ }{l}_j\in y^{\left(j\right)}\right\}$$ (16)

If$\Vert D_{j_1}\Vert \gg \Vert D_{j_2}\Vert$, the class $l_{j_1}$ is a majority class and the class $l_{j_2}$ is a minority class.

Different from SMOTE (Chawla et al., 2002) in a single label dataset, the new synthetic instance maybe have one or more labels. Hence, in (Charte et al., 2015), Charte et al. compared random undersampling (RUS) and random oversampling (ROS) based on Label Power-set (LP) and Multi-Label (ML), respectively. However, their LP-RUS and LP-ROS methods can only work well when the label density is low. Moreover, because their ML-ROS just clones the minority class samples, it is ineffective when these samples simultaneously have the majority class label, which happens often in MLD. In this study, we propose a novel oversampling model named ML-SMOTE. In the following algorithm description, we express a multi-label dataset (see Eq. 15) with N samples as

$$D=\left\{t_i=\left(x_i,y_i\right)|x_i=\left(x_{i,1},\mathrm{ }\dots ,x_{i,m}),\mathrm{ }{y}_i=\mathrm{ }{(y}_{i,1},\dots y_{i,q}\right),1\le i\le N\mathrm{ }\right\}$$ (17)

where $y_{i,j}=\left\{\begin{array}{c}1,\mathrm{ }if\mathrm{ }{x}_i\mathrm{ }ha\mathrm{s}\mathrm{ }{l}_j\mathrm{ }label\\ 0,\mathrm{ }otherwise\mathrm{ }\end{array}\right.(1\le j\le q)$

and the subset D^(j) in which each sample is labeled l_j class:

$$D^{\left(j\right)}=\{t_i^{\left(j\right)}=(x_i^{\left(j\right)},y_i^{\left(j\right)})|\mathrm{ }{x}_i^{\left(j\right)}=(x_{i,1}^{\left(j\right)},\dots x_{i,m}^{\left(j\right)}),\mathrm{ }{y}_i^{\left(j\right)}=(y_{i,1}^{\left(j\right)},\dots y_{i,q}^{\left(j\right)}),\mathrm{ }$$ (18)

$$\mathrm{ }and\mathrm{ }{y}_{i,j}^{\left(j\right)}=1\mathrm{ }\left\}\mathrm{ }\right(1\le j\le q)$$

Algorithm ML-SMOTE algorithm’s pseudo-code

Inputs: Dataset: D with m features and q labels (see Eq. 17); k (the number of nearest neighbors)

Outputs: Preprocessed dataset S

(1) S = D

(2) MeanSize = $\frac{1}{q}{\sum }_{j=1}^q\Vert D^{\left(j\right)}\Vert$ (Ti is defined as Eq. 18)

(3) For j = 1→q

(4)  If $\Vert D^{\left(j\right)}\Vert$ < meanSize

(5)   For each sample$\mathrm{ }{t}_i^{\left(j\right)}$ in $\Vert D^{\left(j\right)}\Vert$, do

(6)    Find k-nearest neighbors set knn of sample $t_i^{\left(j\right)}$in $D^{\left(j\right)}$

(7)    Randomly select a sample z from knn

$$z=\left(z_x,z_y\right),\mathrm{ }{\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{ }z}_x=\left(z_{x,1},\dots z_{x,m}\right),\mathrm{ }{z}_y=\left(z_{y,1},\dots ,z_{y,q}\right)$$

(8) Get a random vector $=\left(\underset{r_1}{\underbrace{r_{\mathrm{1,1}},\dots ,r_{1,m},}}\mathrm{ }\underset{r_2}{\underbrace{r_{\mathrm{2,1}},\dots r_{2,q}}}\right)$, where each element of r is a random number between 0 and 1.

(9) Calculate features of new sample: $v=\left(1-r_1\right).\mathrm{*}{x}_i^{\left(j\right)}+r_1.\mathrm{*}{z}_x.$

     Calculate labels of new sample: $u=\mathrm{I}\mathrm{N}\mathrm{T}\left[\left(1-r_2\right).\mathrm{*}{y}_i^{\left(j\right)}+r_2.\mathrm{*}{z}_y\right]$

     where .* means array multiplying with element by element, and $\mathrm{I}\mathrm{N}\mathrm{T}\left[·\right]$ means round number.

(10)    Add new sample (v, u) to S

(11)   End for

(12)  End if

(13) End for

For a new sample (v, u) synthesized from $t_i^{\left(j\right)}$ and its near neighbor z in${ D}^{\left(j\right)}$,

$$u_w=\left\{\begin{array}{c}1 {\mathrm{ }if\mathrm{ }y}_{\mathrm{i},w}^{\left(j\right)}=1 and z_{y,w}=1 \\ 0 {\mathrm{ }if\mathrm{ }y}_{i,w}^{\left(j\right)}=0 and z_{y,w}=0 \\ 0 or 1, randomly {\mathrm{ }if\mathrm{ }y}_{i,w}^{\left(j\right)}\ne \hspace{0.5em}{z}_{y,w} \end{array}\right.\mathrm{ }(1\le w\le q)$$ (19)

3 Results

After the sequence feature retrieval and ML-SMOTE preprocessing as described above, a two-level AMP predictor named MLAMP was constructed, in which the Ensemble of Classifier Chains (ECC) algorithm (Waghu et al., 2014) was adopted as the prediction method (Fig. 1). We used the canonical implementation of ECC provided by the MULAN (Tsoumakas et al., 2010; Tsoumakas et al., 2011) multi-label learning in the Weka (Hall et al., 2009) library And for ECC, the binary and multi-class learners are implemented on the Weka platform using the Random Forest (RF) algorithm (Breiman, 2001).

Fig.1.

This flowchart shows the training process of MLAMP. T1 represents the data taken from the dataset ${\mathbb{S}}^{Bench}$ for training the 1st-level predictor; T2 represents those from the dataset ${\mathbb{S}}^{+}$ for training the 2nd-level predictor

MLAMP is a two-level prediction engine (See Fig. 1). The first level of MLAMP predicts a query peptide as AMP or non-AMP by using the RF algorithm. It belongs to the case of single-label classification. The following four measures were used for examining the performance of a single-label predictor, they are: (i) overall accuracy or Acc; (ii) Mathew’s correlation coefficient or MCC; (iii) sensitivity or Sn; and (iv) specificity or Sp.

$$\left\{\begin{array}{c}\begin{array}{l}Sn=\frac{TP}{TP+TN}\mathrm{ }\\ Sp=\frac{TN}{TN+FP}\mathrm{ }\\ Acc=\frac{TP+TN}{TP+TN+FP+FN}\mathrm{ }\end{array}\\ MCC=\frac{\left(TP\times TN\right)-\left(FP\times FN\right)}{\sqrt{\left(TP+FP\right)\left(TP+FN\right)\left(TN+FP\right)\left(TN+FN\right)}}\end{array}\right.$$ (20)

where TP represents the true positive; TN, the true negative; FP, the false positive; FN, the false negative.

Although Eq. 20 was often used in the literature to measure the prediction quality of a method, they often lack intuitiveness, especially to biologists, particularly the MCC. According to Chou’s formulation, these four measures can be expressed as (Chen et al., 2016; Lin et al., 2014)

$$\left\{\begin{array}{l}\begin{array}{c}Sn=1-\frac{N_{-}^{+}}{N^{+}},\mathrm{ }0\le Sn\le 1\mathrm{ }\\ Sp=1-\frac{N_{+}^{-}}{N^{-}},\mathrm{ }0\le Sp\le 1\end{array}\\ Acc=1-\frac{N_{-}^{+}+N_{+}^{-}}{N^{+}+N^{-}}, 0\le Acc\le 1\\ MCC=\frac{1-\left(\frac{N_{-}^{+}}{N^{+}}+\frac{N_{+}^{-}}{N^{-}}\right)}{\sqrt{\left(1+\frac{N_{+}^{-}-N_{-}^{+}}{N^{+}}\right)\left(1+\frac{N_{-}^{+}-N_{+}^{-}}{N^{-}}\right)}}, 1\le MCC\le 1\end{array}\right.$$ (21)

where ${\mathrm{N}}^{+}\mathrm{ }$stands for the total number of AMP samples investigated, whereas ${\mathrm{N}}_{-}^{+}$ for the number of AMP samples incorrectly predicted to be of non-AMP; ${\mathrm{N}}^{-}$ for the total number of non-AMP samples investigated, whereas ${\mathrm{N}}_{+}^{-}$ for the number of non-AMP samples incorrectly predicted to be of AMP. With such a formulation as given in Eq. 21, the meanings of sensitivity, specificity, overall accuracy and Mathew’s correlation coefficient and their scopes would become more intuitive and easier-to-understand, particularly for the Mathew’s correlation coefficient, as concurred by a series of studies published very recently (Jia et al., 2016 b,c,d; Lin et al., 2014 , Liu et al., 2016a,b,c; Qiu et al., 2016; Xiao et al., 2016)

If a query peptide is predicted as AMP, the second level of MLAMP will start to classify its functional families. This process belongs to the case of multi-label classification. Hamming loss, Subset Accuracy, Accuracy, Precision and Recall are the mostly used evaluation metrics for the performance of a multi-label classifier (Lin et al., 2013; Tsoumakas and Katakis, 2007; Tsoumakas et al., 2010; Xiao et al., 2013). Suppose ${\mathbb{L}}_{\mathrm{k}}$ is the subset that contains all the labels for the kth sample ${\mathrm{P}}_{\mathrm{k}}$; ${\mathbb{L}}_{\mathrm{k}}^{\mathrm{*}}$ is the subset that contains all the predicted labels for the kth sample ${\mathrm{P}}_{\mathrm{k}}$; N is the total number of samples; and M is the total number of labels. In this study, N = 879 and M = 5. The five metrics have been clearly defined as follows (Chou, 2013):

$$\left\{\begin{array}{l}\mathrm{Precision}=\frac{1}{\mathrm{N}}{\sum }_{\mathrm{k}=1}^{\mathrm{N}}\left(\frac{\Vert {\mathbb{L}}_{\mathrm{k}}\cap {\mathbb{L}}_{\mathrm{k}}^{\mathrm{*}}\Vert }{\Vert {\mathbb{L}}_{\mathrm{K}}^{\mathrm{*}}\Vert }\right) \\ \begin{array}{c}\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{1}{\mathrm{N}}{\sum }_{\mathrm{k}=1}^{\mathrm{N}}\left(\frac{\Vert {\mathbb{L}}_{\mathrm{k}}\cap {\mathbb{L}}_{\mathrm{k}}^{\mathrm{*}}\Vert }{{\mathbb{L}}_{\mathrm{k}}}\right)\mathrm{ }\\ \mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{1}{\mathrm{N}}{\sum }_{\mathrm{k}=1}^{\mathrm{N}}\left(\frac{\Vert {\mathbb{L}}_{\mathrm{k}}\cap {\mathbb{L}}_{\mathrm{k}}^{\mathrm{*}}\Vert }{\Vert {\mathbb{L}}_{\mathrm{k}}\cup {\mathbb{L}}_{\mathrm{k}}^{\mathrm{*}}\Vert }\right)\end{array} \\ \begin{array}{c}\mathrm{S}\mathrm{u}\mathrm{b}\mathrm{s}\mathrm{e}\mathrm{t}\mathrm{ }\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{1}{\mathrm{N}}{\sum }_{\mathrm{k}=1}^{\mathrm{N}}\Delta \left({\mathbb{L}}_{\mathrm{k}},{\mathbb{L}}_{\mathrm{k}}^{\mathrm{*}}\right)\mathrm{ }\\ \mathrm{H}\mathrm{a}\mathrm{m}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{ }\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}=\frac{1}{\mathrm{N}}{\sum }_{\mathrm{k}=1}^{\mathrm{N}}\left(\frac{\Vert {\mathbb{L}}_{\mathrm{k}}\cup {\mathbb{L}}_{\mathrm{k}}^{\mathrm{*}}\Vert -\Vert {\mathbb{L}}_{\mathrm{k}}\cap {\mathbb{L}}_{\mathrm{k}}^{\mathrm{*}}\Vert }{\mathrm{M}}\right)\end{array}\end{array}\mathrm{ }\right.$$ (22)

where $\Vert \mathrm{ }\Vert$ is the operator acting on the set therein to count the number of its elements, and

$$\Delta \left({\mathbb{L}}_k,{\mathbb{L}}_k^{\mathrm{*}}\right)=\left\{\begin{array}{c}\begin{array}{ll}1,& \begin{array}{c}if\mathrm{ }all\mathrm{ }the\mathrm{ }labels\mathrm{ }in\mathrm{ }{\mathbb{L}}_k\mathrm{ }are\\ indentical\mathrm{ }to\mathrm{ }those\mathrm{ }in\mathrm{ }{\mathbb{L}}_k^{\mathrm{*}}\end{array}\mathrm{ }\end{array}\\ \begin{array}{cc}0,& otherwise\mathrm{ }\end{array}\end{array}\right.$$ (23)

When assessing a predictor, the following three cross-validation methods are often used in the literature: independent dataset test, subsampling (K-fold cross-validation) test and jackknife test. However, as elaborated in (Chou and Zhang, 1995), among the three cross-validation methods, the jackknife test is deemed the least arbitrary and most objective because it can always yield a unique result for a given benchmark dataset. Hence, the jackknife test was adopted in this study to examine the anticipated success rates of the current predictor. The process of jackknife test can be explained as follows:

Input: multi-label dataset T = {Pi | 1≤i≤N}.

Output: predicted label set.

For i: 0→N

 T is divided into testing dataset Ts = {Pi},

      and training dataset Tr = T-Ts.

Generate new training dataset Tr’ by using ML-SOMTE on Tr.

 Train model on Tr’ by using ECC algorithm.

 Predict the label set of Pi by the model trained above.

End For

Table 1 compares the performance of MLAMP with an existing method iAMP-2L in the first-level result on the benchmark ${\mathbb{S}}^{Bench}$ (Eq. 1), where overall accuracy Acc and MCC achieved by MLAMP are higher than those achieved by iAMP-2L.

Table 1.

Result obtained by MLAMP in identifying AMP in identifying AMP and non-AMP on benchmark ${\mathbb{S}}^{Bench}$

Predictor	Sn	Sp	Acc	MCC
MLAMPa	77.0%	94.6%	89.9%	0.737
iAMP-2L	87.1%	86.0%	86.2%	0.726

^aThe two parameters, i.e. the number of trees and features used in Random forest were 500 and 6, respectively.

The bold font indicates the better performance in the category.

To further demonstrate the power of the MLAMP predictor, we compared it with other classical predictors on an independent dataset${\mathbb{ }\mathbb{S}}^{Ind}$ containing 920 AMPs and 920 non-AMPs. This comparison was used for independent testing in (Thomas et al., 2010; Xiao et al., 2013). The results listed in Table 2 were obtained by MLAMP, iAMP-2L (Xiao et al., 2013) and CAMP (Thomas et al., 2010) on ${\mathbb{ }\mathbb{S}}^{Ind}$. As shown in Table 2, the performances achieved by MLAMP is remarkably higher than the performances reported by iAMP-2L (Xiao et al., 2013) and CAMP (Thomas et al., 2010) in all metrics (Sn, Sp, Acc and MCC).

Table 2.

Comparison of MLAMP with iAMP-2L and CAMP on the independent dataset ${\mathbb{ }\mathbb{S}}^{Ind}$

Predictor	Algorithm	Sn	Sp	Acc	MCC
MLAMP	Random forest	97.3%	92.1%	94.7%	0.895
iAMP-2L	Fuzzy k-nearest neighbor	97.2%	86.3%	92.2%	0.845
CAMP	Support vector machine	88.4%	66.6%	77.5%	0.55
	Random forest	89.7%	26.0%	57.8%	0.157
	Discriminant analysis	86.6%	64.1%	75.4%	0.508

Furthermore, in the second level prediction, MLAMP also obtained better performance than iAMP-2L. Some different metrics were used from single-label classification, in particular-Hamming loss, Accuracy, Precision, Recall and Subset Accuracy (Tsoumakas et al., 2010) were commonly applied in MLC. Table 3 gives the detailed jackknife test results on the AMP dataset ${\mathbb{S}}^{+}$ (Eq. 2). Especially MLAMP gained a 0.4846 success rate in the strict assessment of subset accuracy and this performance was 5% higher than that by iAMP-2L.

Table 3.

Performance metrics achieved at the 2^nd-level by MLAMP on the AMP dataset .${\mathbb{S}}^{+}$

Predictor	Hamming loss	Accuracy	Precision	Recall	Subset Accuracy
MLAMP	0.1595	0.6864	0.8338	0.7631	0.4846
iAMP-2L	0.1640	0.6687	0.8331	0.7570	0.4305

Why can these metrics be improved so remarkably by using MLAMP? There are two key reasons. The first reason is probably the new peptide feature coding model (see Eq. 13). Table 4 sorts the 30 features in decreasing order after analyzing the benchmark dataset ${\mathbb{S}}^{Bench}$ by the feature selection tool minimal-redundancy-maximal-relevance (mRMR) (Kolde et al., 2016). As shown in Table 4, those features generated by the grey model include more information than amino acids frequency, especially their biochemical properties. And one can draw a conclusion that some physicochemical properties of amino acids may play an important role in AMP, such as molecular weight, PI and Pk2. The second reason points to the new ML-SMOTE model. The AMP dataset ${\mathbb{S}}^{+}$ is an imbalance MLD, and previous studies did not take it into account. After processing the training dataset ${\mathbb{S}}^{+}$ by the ML-SMOTE model, the balance property of the new synthetic training dataset was improved, which can help the machine learning obtained a better performance.

Table 4.

Features in the descending order of importance

Features in Eq. 13	Description of features
P11	Frequency of Methionine
P29	Molecular weight ①
P30	Molecular weight ①
P27	PI ②
P28	PI ②
P26	pk2 ③
P25	pk2 ③
P23	pk1 ④
P21	Hydrophobicity ⑤
P17	Frequency of Threonine
P20	Frequency of Tyrosine
P15	Frequency of Arginine
P18	Frequency of Valine
P19	Frequency of Tryptophan
P14	Frequency of Glutamine
P12	Frequency of Asparagine
P16	Frequency of Serine
P13	Frequency of Proline
P4	Frequency of Glutamic Acid
P24	pk1 ④
P7	Frequency of Histidine
P9	Frequency of Lysine
P8	Frequency of Isoleucine
P6	Frequency of Glycine
P5	Frequency of Phenylalanine
P3	Frequency of Aspartic Acid
P22	Hydrophobicity ⑤
P10	Frequency of Leucine
P2	Frequency of Cysteine
P1	Frequency of Alanine

①: parameter of grey model built by molecular weight; ②: parameter of grey model built by PI; ③: parameter of grey model built by pk2 code; ④: parameter of grey model built by pk1 code; ⑤: parameter of grey model built by hydrophobicity code.

4 Conclusion

Due to increasing antibiotic resistance, AMPs, which are key components of innate immune system, are becoming more and more important in drug development. Efficiently and effectively identifying AMPs and their functional types has become an urgent research topic. The results reported in this study indicate that the novel predictor, MLAMP, provides an accurate and useful tool for researchers to find new infection therapeutics.

MLAMP obtained a better prediction performance than that of a previous method. The primary reason for our good performance is our formulation model’s peptide extraction features. Since AMPs usually have 5–50 amino acids, our model (Eq.13) is good for formulating short peptides. It includes the internal relationship of amino acids sequence in various physical-chemical properties. The second reason is the ML-SMOTE model, which does a good job of handling the lack of balance problem in multi-label datasets. Compared with other methods, the sample synthetized by using ML-SMOTE retains the multi label distributions. It not only accumulates minority samples but also keeps the label density of MLD. In the future, the MLSMOTE model can be extended to assist with imbalance and multi-label datasets for other problems.

For practical applications, a user-friendly web-server for MLAMP has been established at http://www.jci-bioinfo.cn/MLAMP, which allows users to easily obtain their desired results without the need to follow the complicated mathematical equations involved in developing the predictor. Users can submit a peptide sequence to the webserver and subsequently the webserver will return the predicted result in real time. Alternatively, users can choose the batch prediction by entering their e-mail address and their batch input file of many peptide sequences. They will quickly receive an email showing the predicted results from seconds to hours depending on the number of sequences.

Funding

This work has been supported by the National Natural Science Foundation of China (No. 61462047, 31560316) and by the US National Institutes of Health (GM100701)

Conflict of Interest: none declared.