Maximum margin classifier working in a set of strings

Abstract

Numbers and numerical vectors account for a large portion of data. However, recently, the amount of string data generated has increased dramatically. Consequently, classifying string data is a common problem in many fields. The most widely used approach to this problem is to convert strings into numerical vectors using string kernels and subsequently apply a support vector machine that works in a numerical vector space. However, this non-one-to-one conversion involves a loss of information and makes it impossible to evaluate, using probability theory, the generalization error of a learning machine, considering that the given data to train and test the machine are strings generated according to probability laws. In this study, we approach this classification problem by constructing a classifier that works in a set of strings. To evaluate the generalization error of such a classifier theoretically, probability theory for strings is required. Therefore, we first extend a limit theorem for a consensus sequence of strings demonstrated by one of the authors and co-workers in a previous study. Using the obtained result, we then demonstrate that our learning machine classifies strings in an asymptotically optimal manner. Furthermore, we demonstrate the usefulness of our machine in practical data analysis by applying it to predicting protein–protein interactions using amino acid sequences and classifying RNAs by the secondary structure using nucleotide sequences.

1. Introduction

Mathematicians have conducted detailed examinations of a large number of objects, such as numbers, manifolds, equations, functions and operators, throughout the long history of mathematics, but they have not studied strings in detail. A string is an object that computer scientists have addressed in depth. Stringology, a field of computer science, has thoroughly investigated algorithms and data structures for string processing [1,2]. However, computer scientists have not studied strings using a mathematical approach; for example, functions, operators and probabilities on a set of strings provided with topological and algebraic structures have not been investigated.

Numbers and numerical vectors account for a large portion of data. However, in recent years, the amount of string data generated has increased dramatically. For example, large amounts of text data have been produced on the web. In the life sciences, large amounts of data regarding genes, RNAs and proteins have been generated. These data are nucleotide or amino acid sequences and can be represented as strings. Consequently, a random string that randomly generates strings based on a probability law is necessary for string data analysis, much as random variables that randomly generate numbers and stochastic processes that randomly generate functions are essential in various fields. Statistical methods for numerical data were rigorously constructed based on probability theory. Similarly, the development and systematization of methods on the basis of probability theory on a set of strings will be required for text mining techniques and methods for analysing biological sequences.

Let A* be a set of strings on an alphabet A={a₁,…,a_z}. From the viewpoint of mathematics, A* is a monoid with respect to concatenation (the operation of appending one string to the end of another) and a metric space with the Levenshtein distance (the minimal number of deletions, insertions or substitutions required to transform one string into another), and it forms a non-commutative topological monoid. Koyano and co-workers [3,4] were the first to construct a theory for treating strings that are randomly generated according to a probability law by developing a probability theory on the space A* with the above-mentioned mathematical structures and to construct statistical methods for analysing string data using this probability theory. They used the developed statistical methods to address problems that involved biological sequence data. However, those researchers developed specific methods for estimating the α and β diversities of biological communities using gene sequences rather than general-purpose methods for statistical analysis of string data. In this study, we first extend a limit theorem demonstrated in [3] on the asymptotic behaviour of a consensus sequence of strings. This theorem is an analogue of the strong law of large numbers in a p-dimensional real vector space ${\mathbb{R}}^p$, because a consensus sequence of strings is the counterpart in A* of a mean in ${\mathbb{R}}^p$. Using this result, we then develop the theory of statistical machine learning for string classification.

Classifying string data is a common problem in many fields, including computer science and the life sciences because of recent significant increases in the prevalence of string data. The most widely used approach to this problem is to convert strings into numerical vectors using a string kernel and subsequently apply a support vector machine (SVM) [5–9] to the vectors. The earliest string kernels were developed by Haussler [10], Watkins [11] and Lodhi et al. [12]. These papers proposed that the similarity between strings should be defined based on the number of subsequences common to them. Leslie and Paaß and co-workers [13,14] used the spectrum kernel, a string kernel that quantifies the similarity between strings based on the number of common substrings, without considering common subsequences for which gaps are allowed. The spectrum kernel was subsequently extended by Leslie and Vishwanathan and co-workers [15–17]. In addition to these kernels, a number of novel string kernels were developed and applied to problems in bioinformatics by recent studies [18–21]. The spectrum kernel has become the most widely used of these various string kernels, although this kernel discards considerable amounts of the information concerning the order of the letters that compose the strings.

Converting strings into numerical vectors using a string kernel is not bijective and involves information loss. A more serious problem is that this conversion makes it impossible to evaluate, using probability theory, the generalization error of a learning machine, considering that the given data to train and test the machine are strings generated according to probability laws. Consequently, the performance of a learning machine has been evaluated based on whether the machine yields better results compared with other machines in a certain simulation experiment or in the application to a certain real dataset, and the fundamental evaluation of a learning machine by theoretically evaluating its generalization error has been abandoned. In this study, we develop a learning machine that classifies strings without converting them into numerical vectors by constructing a direct sum decomposition of A* under the principle of margin maximization, as an SVM does in ${\mathbb{R}}^p$ (§§2 and 3). We then provide a theoretical evaluation of the generalization error of our learning machine (§5) by applying the theoretical result demonstrated in §4. We also demonstrate the usefulness of our machine for practical data analysis by applying it to predicting protein–protein interactions based on amino acid sequences and classifying RNAs by the secondary structure based on nucleotide sequences (§6).

2. Specification of the problem

In the following, we refer to a classifier that decomposes a space into two disjoint subsets by choosing a hyperplane under the principle of margin maximization as an SVM, although an SVM also has other characteristics, such as (i) learning on the dual of a vector space, (ii) extracting features from input vectors, and (iii) using kernel functions. We consider a plane ${\mathbb{R}}^2$ for the sake of simplicity. A line in ${\mathbb{R}}^2$ is represented as $\{\mathit{\text{z}}\in {\mathbb{R}}^2:\mathit{\text{z}}=\alpha \mathit{\text{x}}+\beta \mathit{\text{y}},\alpha +\beta =1,\alpha ,\beta \in \mathbb{R}\}$ for $\mathit{\text{x}},\mathit{\text{y}}\in {\mathbb{R}}^2$ and $\{(x,y)\in {\mathbb{R}}^2:y=ax+b,x\in \mathbb{R}\}$ for $a,b\in \mathbb{R}$. The first representation uses the vector space structure of ${\mathbb{R}}^2$, because addition and scalar multiplication are used in the representation, and the second representation uses the field structure of $\mathbb{R}$, because addition and multiplication are used. We denote a set of strings on the alphabet A={a₁,…,a_z} by A*. The intrinsic operation and distance on A* are concatenation (hereafter denoted by ⋅) and the Levenshtein distance (hereafter denoted by d_L), respectively. Therefore, we provide A* with algebraic and topological structures using ⋅ and d_L. A* forms a non-commutative topological monoid, but it does not form a vector space or field. Therefore, ‘a line’ cannot be defined in A* using the above two forms. However, this does not mean that a line cannot be defined in A*. Thus, we consider the following two questions: (i) Can ‘a line’ be defined in A* in some way? (ii) If so, can A* be decomposed into two disjoint subsets by using ‘the line’? The answer to the first question is ‘Yes’, whereas the answer to the second question is ‘No’.

By considering a curve in a space to be a subset of the space that is obtained by repeating the operation of connecting a point in the space to one of its contiguous points, we can roughly define ‘a curve’ in A*, for example, in the following manner: if d_L(s_i,s_i+1)=1, i=1,…,n−1 holds for s₁,…,s_n∈A*, we call {s₁,…,s_n} ‘a curve’ in A*. Furthermore, considering a segment between two points in a space to be the shortest curve that connects the two points, we can define ‘a segment’ in A* as follows: we suppose that d_L(s,s′)=n for s,s′∈A*. s can be transformed into s′ by performing one of three types of operation, insertion, deletion and substitution, n times. We denote a string obtained by performing the first i operations of the n operations on s by s_(i) for each i=1,…,n−1. For the uniqueness of ‘a segment’ that connects two given strings, we suppose that the order of priority is given among insertion, deletion and substitution and that a series of operations is performed on s in ascending order with respect to the letter number in s and according to the order of priority among the three operations. We call {s,s₍₁₎,…,s_(n−1),s′} ‘a segment’ in A* that connects s and s′.

Therefore, we consider the decomposition of a sufficiently large subset for applications that are composed of strings whose length is less than or equal to that of s, although not the entire space of A*, by choosing a sufficiently long string s and drawing a segment between s and the empty string (a string composed of zero letters). The alphabet A={a₁,…,a_z} forms a metric space with the Hamming distance d_H(a_i,a_j)=0 (if i=j) or 1 (if i≠j). By comparison, the set of real numbers $\mathbb{R}$ also forms a metric space with the absolute value of the difference d(x,y)=|x−y| as well as a totally ordered set with respect to the usual less-than-or-equal relation ≤. The distance d and the total order ≤ on $\mathbb{R}$ are consistent in the sense that if x≤y and y≤z, then d(x,y)≤d(x,z) holds for any $x,y,z\in \mathbb{R}$. Such an intrinsic total order as the less-than-or-equal relation ≤ on $\mathbb{R}$ does not exist on A. By defining a total order that is consistent with the Hamming distance d_H in the sense mentioned above, can we make A form a totally ordered set without destroying its structure as a metric space? This task is impossible owing to the definition of d_H. Consequently, we have the following problem: ${\mathbb{R}}^2$ can be divided into upper and lower half-spaces $H^{+}=\{(x,y)\in {\mathbb{R}}^2:ax+b\le y\}-\ell$ and $H^{-}=\{(x,y)\in {\mathbb{R}}^2:y\le ax+b\}-\ell$ with a line $\ell =\{(x,y)\in {\mathbb{R}}^2:y=ax+b\}$. H⁺ and H⁻ are defined using the total order ≤ on $\mathbb{R}$. In other words, for the concepts of upper and lower areas of a line in the direct product space ${\mathbb{R}}^2$ to make sense, the total order on the direct product factor $\mathbb{R}$ is required. The analogies of a curve and segment can be defined in A* in the above manner using the Levenshtein distance. However, the concepts of upper and lower areas of a segment cannot make sense without destroying the structure of A as a metric space, because a total order that is consistent with the Hamming distance cannot be defined on A. Consequently, A* cannot be divided by determining such a non-closed subset as a line, in contrast to ${\mathbb{R}}^2$.

However, the above discussion does not indicate that A* cannot be divided into two disjoint subsets in any manner. As the Jordan curve theorem [22] and the Jordan–Brouwer separation theorem [23] of topology state, ${\mathbb{R}}^2$ and ${\mathbb{R}}^p$ (p≥3) can be divided into two disjoint subsets by choosing a closed curve and hypersphere without determining a line or hyperplane, respectively. Can we decompose A* into two disjoint subsets by using a method other than by drawing a line? We set U(s,r)={t∈A*:d_L(t,s)≤r} for s∈A* and $r\in {\mathbb{Z}}^{+}$ (${\mathbb{Z}}^{+}$ represents the set of positive integers) and consider the decomposition of A* into U(s,r) and U(s,r)^c=A*−U(s,r). In other words, we examine a method of drawing a sphere in A* and subsequently decomposing A* into its interior and exterior. In this manner, the decomposition does not require the concepts of upper and lower areas. In the following, we refer to ∂U(s,r)={t∈A*:d_L(t,s)=r} as a discriminant sphere and the number of strings in U(s,r) as the size of ∂U(s,r). We set the convention that strings that lie on ∂U(s,r) are classified into a class of positive examples.

3. Learning machine working in A*

To decompose A* in the manner described in §2, it is necessary to specify the centre s∈A* and the radius $r\in {\mathbb{Z}}^{+}$ of a discriminant sphere ∂U(s,r) given positive and negative examples. We say that the positive examples X_m={s₁,…,s_m} and negative examples Y _n={t₁,…,t_n} are spherically separable if there exists s₀∈A* such that $\underset{1\le i\le m}{max}\{d_{\mathrm{L}}(s_i,s_0)\}<\underset{1\le i\le n}{min}\{d_{\mathrm{L}}(t_i,s_0)\}$ holds and that X_m and Y _n are spherically inseparable if they are not spherically separable. We denote a set of m-tuples of strings for which a consensus sequence is uniquely determined by [(A*)^m]. A formal definition of a consensus sequence is provided in §S2 of the electronic supplementary material for this paper. We suppose s₁,…,s_m∈[(A*)^m] in the following and choose the consensus sequence ${\overline{s}}_m$ of positive examples s₁,…,s_m as the centre of a discriminant sphere.

We first consider the problem of choosing the radius of a discriminant sphere for the case in which the positive and negative examples are spherically separable. Similar to a discriminant hyperplane of an SVM in ${\mathbb{R}}^p$, the distance between a string and a discriminant sphere is the distance between the string and a string in the sphere that is nearest to the string, and given samples of positive examples and negative examples, the margin of a discriminant sphere is the distance between the sphere and an example in the samples that is nearest to the sphere. Under the principle of margin maximization, the following result can be immediately obtained: if the positive examples X_m={s₁,…,s_m} and negative examples Y _n={t₁,…,t_n} are spherically separable with respect to ${\overline{s}}_m$, the radius of a discriminant sphere that maximizes the margin is given by

$$r^{\ast }=\frac{1}{2}\{\underset{1\le i\le m}{max}\{d_{\mathrm{L}}(s_i,{\overline{s}}_m)\}+\underset{1\le i\le n}{min}\{d_{\mathrm{L}}(t_i,{\overline{s}}_m)\}\}.$$ 3.1

If r* is not an integer, then we arbitrarily choose one of the integers closest to r*.

Next, we consider the case in which the positive examples X_m and negative examples Y _n are spherically inseparable. We denote subsamples of the positive and negative examples that a discriminant sphere $\mathrm{\partial }U({\overline{s}}_m,r)$ with a centre ${\overline{s}}_m$ and a radius r correctly classifies by $X_m({\overline{s}}_m,r)$ and $Y_n({\overline{s}}_m,r)$, respectively. We denote the number of elements of a finite set S by ♯S. The numbers of strings in X_m and in Y _n that $\mathrm{\partial }U({\overline{s}}_m,r)$ misclassifies are represented by $m-\mathrm{♯}{X}_m({\overline{s}}_m,r)$ and $n-\mathrm{♯}{Y}_n({\overline{s}}_m,r)$, respectively. If the two samples X_m and Y _n are spherically inseparable, we choose the radius of a discriminant sphere based on the principle of minimizing the number of misclassified inputs and maximizing the margin, which is a modification of the principle used by an ordinary SVM in ${\mathbb{R}}^p$ in soft margin optimization. If the positive and negative examples are spherically separable, the following procedure is reduced to choosing the radius according to equation (3.1).

Step 1 (minimizing the number of misclassified inputs). Search for a set of radii that minimize the number of misclassified inputs, i.e. a set of positive integers $\tilde{r}$ that satisfy

$$\tilde{r}=\arg \underset{r\in {\mathbb{Z}}^{+}}{min}\{m-\mathrm{♯}{X}_m({\overline{s}}_m,r)+n-\mathrm{♯}{Y}_n({\overline{s}}_m,r)\}$$

or, equivalently,

$$\tilde{r}=\arg \underset{r\in {\mathbb{Z}}^{+}}{max}\{\mathrm{♯}{X}_m({\overline{s}}_m,r)+\mathrm{♯}{Y}_n({\overline{s}}_m,r)\}.$$

We denote this set by $\tilde{R}$. $\tilde{R}$ is a non-empty finite set.

Step 2 (maximizing the margin). Choose $r^{\ast }\in \tilde{R}$ that maximizes the distance to the closest string that is correctly classified (if such r* is not uniquely determined, then we arbitrarily choose one of them). This step is formally written as follows: the distances between $s\in X_m({\overline{s}}_m,r)$ and $\mathrm{\partial }U({\overline{s}}_m,r)$ and between $t\in Y_n({\overline{s}}_m,r)$ and $\mathrm{\partial }U({\overline{s}}_m,r)$ are equal to $r-d_{\mathrm{L}}(s,{\overline{s}}_m)$ and $d_{\mathrm{L}}(t,{\overline{s}}_m)-r$, respectively. These distances are not necessarily equal when s and t are support strings, in contrast to support vectors for an ordinary SVM in ${\mathbb{R}}^p$, because their sum $r-d_{\mathrm{L}}(s,{\overline{s}}_m)+d_{\mathrm{L}}(t,{\overline{s}}_m)-r=d_{\mathrm{L}}(t,{\overline{s}}_m)+d_{\mathrm{L}}(s,{\overline{s}}_m)$ may be odd. The optimal radius r* is represented as $r^{\ast }=\arg \underset{\tilde{r}\in \tilde{R}}{max}\rho (\tilde{r})$ for

$$\rho (\tilde{r})=\underset{(s,t)\in X_m({\overline{s}}_m,\tilde{r})\times Y_n({\overline{s}}_m,\tilde{r})}{min}min\{\tilde{r}-d_{\mathrm{L}}(s,{\overline{s}}_m),d_{\mathrm{L}}(t,{\overline{s}}_m)-\tilde{r}\},\tilde{r}\in \tilde{R},$$

and the support strings are given by

$$(s^{\ast },t^{\ast })=\arg \underset{(s,t)\in X_m({\overline{s}}_m,r^{\ast })\times Y_n({\overline{s}}_m,r^{\ast })}{min}min\{r^{\ast }-d_{\mathrm{L}}(s,{\overline{s}}_m),d_{\mathrm{L}}(t,{\overline{s}}_m)-r^{\ast }\}.$$

To search for the set $\tilde{R}$, it is sufficient to examine only those radii between $\underset{1\le i\le m}{max}\{d_{\mathrm{L}}(s_i,{\overline{s}}_m)\}$ and $\underset{1\le i\le n}{min}\{d_{\mathrm{L}}(t_i,{\overline{s}}_m)\}-1$.

Let N and ℓ be the number and the maximum length of input strings, respectively. The time complexity is O(Nℓ) for constructing a centre string of a discriminant sphere, O(Nℓ²) for computing Levenshtein distances from all input strings to the centre string, O(Nℓ) for computing the number $\mathrm{♯}{X}_m({\overline{s}}_m,r)+\mathrm{♯}{Y}_n({\overline{s}}_m,r)$ for all candidates for r, because there exist O(ℓ) candidates for r, O(Nℓ) for searching $\tilde{r}$ and O(Nℓ) for searching r*. Therefore, the total time complexity for constructing the classifier is equal to O(Nℓ²).

4. Limit theorems in A*

In §3, we constructed a statistical learning machine that classifies string data by forming a direct sum decomposition of A*. In this section, we demonstrate a generalization of theorem 2 described in the electronic supplemental material of [3], which will be applied to an asymptotic analysis of the generalization error of our learning machine in §5. We also derive several corollaries from the generalization. The notation and definitions of terms used in this section are provided in §S2 of the electronic supplementary material for this paper. In the following sections, the alphabet is A={a₁,…,a_z−1}. We set a_z=e for the empty letter e and refer to $\overline{A}=A\cup \{e\}=\{a_1,\dots ,a_z\}$ as the extended alphabet. We denote the set of all strings on $\overline{A}$ by A*. Roughly describing, $(\mathrm{\Omega },\mathcal{F},P)$ is an underlying probability space, and $\mathcal{M}(\mathrm{\Omega },\overline{A})$ and $\mathcal{M}(\mathrm{\Omega },A^{\ast })$ are sets of random letters and of random strings, respectively. μ(α₁,…,α_n) denotes the consensus letter of random letters α₁,…,α_n, and μ(σ₁,…,σ_n) and κ(σ₁,…,σ_n) represent the consensus sequence and variance of random strings σ₁,…,σ_n, respectively. $[\mathcal{M}{(\mathrm{\Omega },\overline{A})}^n]$ and $[\mathcal{M}{(\mathrm{\Omega },A^{\ast })}^n]$ are sets of n-tuples of random letters the consensus letter of which is uniquely determined and of n-tuples of random strings the consensus sequence of which is uniquely determined, respectively.

Let $\{{\alpha }_i:i\in {\mathbb{Z}}^{+}\}\subset \mathcal{M}(\mathrm{\Omega },\overline{A})$. We set

$$p(i,h)=P(\{\omega \in \mathrm{\Omega }:{\alpha }_i(\omega )=a_h\}),\overline{p}(h,n)=\frac{1}{n}\sum _{i=1}^{n}p(i,h)$$

for each h=1,…,z. p(i,h) represents the probability that the ith random letter realizes the hth letter in the extended alphabet $\overline{A}$, and $\overline{p}(h,n)$ represents the average probability that the hth letter in $\overline{A}$ is observed when n observations are made. For a statement S, S a.s. represents that S holds with probability one. First, theorem 1 from the electronic supplemental material of [3] can be generalized as follows.

Theorem 4.1 —

We suppose that $\{{\alpha }_i:i\in {\mathbb{Z}}^{+}\}\subset \mathcal{M}(\mathrm{\Omega },\overline{A})$ (i) satisfies $({\alpha }_1,\dots ,{\alpha }_n)\in [\mathcal{M}{(\mathrm{\Omega },\overline{A})}^n]$ for each $n\in {\mathbb{Z}}^{+}$, and (ii) is independent. If (iii) $\iota =\arg \underset{1\le h\le z}{max}\overline{p}(h,n)$ is uniquely determined independent of n, there exists $n_0\in {\mathbb{Z}}^{+}$ such that if n≥n₀, then

$$\mu ({\alpha }_1,\dots ,{\alpha }_n)=a_{\iota }a.s.$$

holds.

Proofs of all results described in this paper are provided in §S1 of the electronic supplementary material. In theorem 4.1, the independence of random letters α₁,…,α_n is assumed, but the identical distribution is not. Therefore, the consensus letters can vary among the distributions of α₁,…,α_n. Theorem 4.1 states that even if the distributions of α₁,…,α_n do not have an identical consensus letter, the consensus letter μ(α₁,…,α_n)(ω) of the observed letters converges to a letter a_ι under the above conditions. Even if α₁,…,α_n do not have an identical distribution, by the definition of a_ι, we have

$$m^{\mathrm{\prime }}({\alpha }_1)=\cdots =m^{\mathrm{\prime }}({\alpha }_n)=a_{\iota }$$

if α₁,…,α_n have an identical consensus letter. Thus, theorem 1 from the electronic supplemental material of [3] is a special case of theorem 4.1.

Let $\{{\sigma }_i=\{{\alpha }_{ij}:j\in {\mathbb{Z}}^{+}\}:i\in {\mathbb{Z}}^{+}\}\subset \mathcal{M}(\mathrm{\Omega },A^{\ast })$. We set

$$p(i,j,h)=P(\{\omega \in \mathrm{\Omega }:{\alpha }_{ij}(\omega )=a_h\}),\overline{p}(j,h,n)=\frac{1}{n}\sum _{i=1}^{n}p(i,j,h)$$

for each h=1,…,z.

Corollary 4.2 —

If (i) $\{{\sigma }_i=\{{\alpha }_{ij}:j\in {\mathbb{Z}}^{+}\}:i\in {\mathbb{Z}}^{+}\}\subset \mathcal{M}(\mathrm{\Omega },A^{\ast })$ satisfies $({\sigma }_1,\dots ,{\sigma }_n)\in [\mathcal{M}{(\mathrm{\Omega },A^{\ast })}^n]$ for each $n\in {\mathbb{Z}}^{+},$ (ii) $\{{\alpha }_{ij}:i\in {\mathbb{Z}}^{+}\}$ is independent for each $j\in {\mathbb{Z}}^{+}$ and (iii) $\iota (j)=\arg \underset{1\le h\le z}{max}\overline{p}(j,h,n)$ is uniquely determined independent of n, there exists $n_0\in {\mathbb{Z}}^{+}$ such that if n≥n₀, then

$$\mathit{\mu }({\sigma }_1,\dots ,{\sigma }_n)=\{a_{\iota (j)}:j\in {\mathbb{Z}}^{+}\}a.s.$$

holds.

Theorem 2 from the electronic supplementary material of [3] can be obtained as a special case of corollary 4.2. We denote almost sure convergence by $\stackrel{a.s.}{⟶}$ and the expectation of a random variable X by E[X]. Using corollary 4.2, theorem 3 from the electronic supplementary material of [3] is extended as follows.

Corollary 4.3 —

We suppose that (i) $\{{\sigma }_i=\{{\alpha }_{ij}:j\in {\mathbb{Z}}^{+}\}:i\in {\mathbb{Z}}^{+}\}\subset \mathcal{M}(\mathrm{\Omega },A^{\ast })$ satisfies $({\sigma }_1,\dots ,{\sigma }_n)\in [\mathcal{M}{(\mathrm{\Omega },A^{\ast })}^n]$ for each $n\in {\mathbb{Z}}^{+}$ and that (ii) $\{{\sigma }_i:i\in {\mathbb{Z}}^{+}\}$ is independent. If (iii) $\iota (j)=\arg \underset{1\le h\le z}{max}\overline{p}(j,h,n)$ is uniquely determined independent of n, then we have

$$\mathit{\kappa }({\sigma }_1,\dots ,{\sigma }_n)\stackrel{a.s.}{⟶}\frac{1}{n}\sum _{i=1}^n\mathrm{E}[d_{\mathrm{L}}({\sigma }_i,\{a_{\iota (j)}:j\in {\mathbb{Z}}^{+}\})]$$

as $n\to \mathrm{\infty }$. More strictly than under condition (iii), if (iv) $\{{\sigma }_i\}\subset [\mathcal{M}(\mathrm{\Omega },A^{\ast })]$ holds and (v) {σ_i} have an identical family of finite dimensional distributions, we have

$$\mathit{\kappa }({\sigma }_1,\dots ,{\sigma }_n)\stackrel{a.s.}{⟶}{\mathit{\text{v}}}^{\prime }({\sigma }_1)$$

as $n\to \mathrm{\infty }$.

Because a consensus sequence of strings is a majority vote, unlike a mean of real vectors, the distributions of random letters that compose a consensus sequence converge to the Dirac measures for letters of the extended alphabet, rather than to distributions such as the normal distribution, under the conditions of corollary 4.2.

Corollary 4.4 —

We suppose that $\{{\sigma }_i=\{{\alpha }_{ij}:j\in {\mathbb{Z}}^{+}\}:i\in {\mathbb{Z}}^{+}\}\subset \mathcal{M}(\mathrm{\Omega },A^{\ast })$ satisfies the conditions of corollary 4.2. We denote the Dirac measure for a letter $a_{\iota (j)}\in \overline{A}$ such that $\iota (j)=\arg \underset{1\le h\le z}{max}\overline{p}(h,n,j)$ by δ_ι(j) (ι(j) is independent of n and unique). Then, there exists $n_0\in {\mathbb{Z}}^{+}$ such that if n≥n₀, then a sequence of distributions of random letters that compose μ(σ₁,…,σ_n) is equal to $\{{\delta }_{\iota (j)}:j\in {\mathbb{Z}}^{+}\}$.

Let $\{{\sigma }_i=\{{\alpha }_{ij}:j\in {\mathbb{Z}}^{+}\}:i\in {\mathbb{Z}}^{+}\}\subset \mathcal{M}(\mathrm{\Omega },A^{\ast })$. We assume that $\{{\sigma }_i:i\in {\mathbb{Z}}^{+}\}$ is independent and $\{{\alpha }_{ij}:i\in {\mathbb{Z}}^{+}\}$ is identically distributed (therefore, p(i,j,h) is independent of i) for each $j\in {\mathbb{Z}}^{+}$. Then, noting lemmas 2 and 3 from the supplemental material of [3] and the proof of theorem 4.1, we observe that if there exists $j_0\in {\mathbb{Z}}^{+}$ such that $\arg \underset{1\le h\le z}{max}p(i,j_0,h)$ has more than two elements, then a consensus letter of α_1j0,…,α_nj0 and consequently, a consensus sequence of σ₁,…,σ_n are not determined with probability one as $n\to \mathrm{\infty }$. Thus, if there exists $j_0\in {\mathbb{Z}}^{+}$ such that $\{{\alpha }_{i{j}_0}:i\in {\mathbb{Z}}^{+}\}$ has the uniform distribution on $\overline{A},$ then a consensus sequence of σ₁,…,σ_n is not determined with probability one as $n\to \mathrm{\infty },$ even if $\{{\sigma }_i:i\in {\mathbb{Z}}^{+}\}$ is independent.

5. Asymptotic optimality of the proposed learning machine

As described in §1, in the conventional framework of converting strings into numerical vectors using a string kernel and subsequently applying a classifier working in a numerical vector space to the vectors, it is impossible to theoretically evaluate the generalization error of the classifier for string classification, because conversion using a string kernel is not bijective. Consequently, to evaluate the performance of a classifier for string data, we have no option but to apply the classifier to certain datasets and repeat the cross-validation. In this section, by applying corollary 4.2 demonstrated in §4 on the asymptotic behaviour of a consensus sequence of random strings, we theoretically consider whether our statistical learning machine working in A* constructed in §3 is optimal in terms of its generalization error. Generally, in classical statistics, the problems of estimation and hypothesis testing are considered under the assumption that the data are generated according to an unknown but unique population distribution. However, machine learning is typically applied to mining datasets that are considerably larger than the datasets analysed using traditional statistical methods. Therefore, it is absolutely impossible to suppose that data are generated according to an unknown but unique distribution. In this section, we theoretically analyse the generalization error of our learning machine in a setting in which both positive and negative examples are generated according to an unknown number of unknown distributions.

For $f_1,f_2\in {\mathbb{Z}}^{+}$, we suppose that the ith positive example s_i is generated according to one of f₁ unknown distributions with probability functions ${\mathit{\text{p}}}_1^{(1)},\dots ,{\mathit{\text{p}}}_1^{(f_1)}$ on A* for each i=1,…,m and the ith negative example t_i is generated according to one of f₂ unknown distributions with probability functions ${\mathit{\text{p}}}_2^{(1)},\dots ,{\mathit{\text{p}}}_2^{(f_2)}$ on A* for each i=1,…,n. f₁ and f₂ are also unknown. Thus, the models that generate positive and negative examples are finite mixture models

$${\mathit{\text{p}}}_1(s)=\sum _{k=1}^{f_1}{\pi }_1^{(k)}{\mathit{\text{p}}}_1^{(k)}(s)\mathrm{and}{\mathit{\text{p}}}_2(t)=\sum _{k=1}^{f_2}{\pi }_2^{(k)}{\mathit{\text{p}}}_2^{(k)}(t),$$

respectively, where $\sum _{k=1}^{f_i}{\pi }_i^{(k)}=1$ and ${\pi }_i^{(1)},\dots ,{\pi }_i^{(f_i)}\in (0,1)$ for i=1,2. (For i=1,2 and k=1,…,f_i, ${\mathit{\text{p}}}_i^{(k)}$ is introduced as p_σ in §S2 of the electronic supplementary material for this paper.) D₁ and D₂ represent the supports of p₁ and p₂, respectively, i.e. D_i={s∈A*:p_i(s)>0} for i=1,2. We assume that D₁−D₂≠∅ and D₂−D₁≠∅. If D₁∩D₂=∅, then the probability that the generalization error becomes zero after finite times of learning is equal to one. Therefore, we consider the case of D₁∩D₂≠∅ in the following. The generalization error E₀(s,r) of a discriminant sphere ∂U(s,r) is written as E₀(s,r)=E₁(s,r)+E₂(s,r) for

$$E_1(s,r)=\sum _{t\in D_1\cap U{(s,r)}^c}{\mathit{\text{p}}}_1(t)\mathrm{and}{E}_2(s,r)=\sum _{t\in D_2\cap U(s,r)}{\mathit{\text{p}}}_2(t).$$ 5.1

We formally set

$$(s^{†},r^{†})=\arg \underset{(s,r)\in A^{\ast }\times {\mathbb{Z}}^{+}}{min}{E}_0(s,r),r^{†}(s_0)=\arg \underset{r\in {\mathbb{Z}}^{+}}{min}{E}_0(s_0,r)$$

for each s₀∈A*. r^†(s₀) is the radius of a discriminant sphere that is optimal in terms of the generalization error given a centre. We denote the relative frequencies of t in X_m and in Y _n by ${\widehat{\mathit{\text{p}}}}_1(t)$ and ${\widehat{\mathit{\text{p}}}}_2(t)$, respectively, for any t∈A*. We set

$${\hat{E}}_1(s,r)=\sum _{t\in X_m\cap U{(s,r)}^c}{\widehat{\mathit{\text{p}}}}_1(t)\mathrm{and}{\hat{E}}_2(s,r)=\sum _{t\in Y_n\cap U(s,r)}{\widehat{\mathit{\text{p}}}}_2(t)$$ 5.2

for s∈A* and $r\in {\mathbb{Z}}^{+}$.

Assuming that ${\overline{s}}_m$ is used as the centre of a discriminant sphere, we first consider whether r* converges to an optimal radius in terms of the generalization error as our learning machine updates r* through a learning process. Note that if positive examples are generated in a manner in which the conditions of corollary 4.2 described in the previous section are satisfied, the optimal radius is r^†({a_ι(j)}), because ${\overline{s}}_m$ is equal to {a_ι(j)} with probability one, given a sufficient number of positive examples.

Theorem 5.1 Asymptotic optimality of r* —

If (i) the positive examples s₁,…,s_m are realizations of random strings σ₁,…,σ_m that are independent and distributed according to the mixture model p₁ and the negative examples t₁,…,t_n are realizations of random strings τ₁,…,τ_n that are independent and distributed according to p₂, (ii) σ₁,…,σ_m satisfy the conditions of corollary 4.2 for each $m\in {\mathbb{Z}}^{+}$ and (iii) there uniquely exists r^†({a_ι(j)}), then we have

$$r^{\ast }\stackrel{a.s.}{⟶}{r}^{†}(\{a_{\iota (j)}\})$$

as $m,n\to \mathrm{\infty }$. In other words, r* converges to a radius that is asymptotically optimal given ${\overline{s}}_m$ as the centre of a discriminant sphere with probability one.

In the proof of theorem 5.1, only the principle of minimizing the number of misclassified inputs was used, and the principle of maximizing the margin was not required (see Section S1 of the electronic supplementary material), because the samples of the positive and negative examples accurately reflected their population distributions in the asymptotic setting. This suggests that the reason an ordinary SVM working in ${\mathbb{R}}^p$ has a high predictive performance in a number of applications is that margin maximization plays a role in reducing the probability of misclassifying examples in a test sample when the positive and negative examples in a training sample do not necessarily accurately reflect their population distributions because, for example, a training sample is not sufficiently large; in other words, margin maximization is a reasonable principle for classifying data in a sample in cases where the sample does not include sufficient information on the population distribution.

We next consider the optimality of ${\overline{s}}_m$. We address this problem in the following setting, which models the situation in which the positive and negative examples are spherically inseparable: D₁∩D₂≠∅ holds and p₁ and p₂ satisfy the conditions that (i) p₁(s) is monotonically non-increasing with respect to d_L(s,{a_ι(j)}) on D₁ and (ii) there exists $d_0\in {\mathbb{Z}}^{+}$ such that p₂(s) is monotonically non-decreasing with respect to d_L(s,{a_ι(j)}) on $D_2^{\mathrm{\prime }}=\{s\in D_2:d_{\mathrm{L}}(s,\{a_{\iota (j)}\})\le d_0\}$ and $D_1\not\supset D_2^{\mathrm{\prime }}$. We assume that d₀ is sufficiently large and consider only discriminant spheres that are disjoint with ${(D_1\cup D_2^{\mathrm{\prime }})}^c$. We do not assume that r* is chosen as the radius of a discriminant sphere. Note that ♯U(s,r) increases monotonically with respect to the length of s and r. We denote sets of pairs $(s^{\mathrm{\prime }},r^{\mathrm{\prime }})\in A^{\ast }\times {\mathbb{Z}}^{+}$ such that $\mathrm{♯}U(s^{\mathrm{\prime }},r^{\mathrm{\prime }})=\mathrm{♯}U({\overline{s}}_m,r)$, $\mathrm{♯}U(s^{\mathrm{\prime }},r^{\mathrm{\prime }})\le \mathrm{♯}U({\overline{s}}_m,r)$, and $\mathrm{♯}U(s^{\mathrm{\prime }},r^{\mathrm{\prime }})\ge \mathrm{♯}U({\overline{s}}_m,r)$ by B₀(r),B₁(r), and B₂(r), respectively, for any $r\in {\mathbb{Z}}^{+}$.

Theorem 5.2 (Asymptotic optimality of ${\overline{s}}_m$) —

In the setting described above, if random strings σ₁,…,σ_m that generate positive examples satisfy the conditions of corollary 4.2 for each $m\in {\mathbb{Z}}^{+},$ we have

$$E_j({\overline{s}}_m,r)\le E_j(s^{\mathrm{\prime }},r^{\mathrm{\prime }})\text{a.s.}$$

as $m,n\to \mathrm{\infty }$ for any $r\in {\mathbb{Z}}^{+},$ (s′,r′)∈B_j(r), and j∈{0,1,2}. In other words, for any radius r, a discriminant sphere with a centre ${\overline{s}}_m$ is asymptotically optimal in a class of discriminant spheres that are equal to it in size and has asymptotically minimum probabilities of false-negatives of discriminant spheres of equal or smaller size and of false-positives of discriminant spheres of equal or larger size.

Let ${\sigma }_i=\{{\alpha }_{ij}:j\in {\mathbb{Z}}^{+}\}$ be a random string that generates the ith positive example for each i=1,…,m. If σ₁,…,σ_m are independent, α_1j,…,α_mj are also independent for each $j\in {\mathbb{Z}}^{+}$, but the converse is not true. In theorem 5.2, the independence of α_1j,…,α_mj is assumed for each $j\in {\mathbb{Z}}^{+}$, but the independence of σ₁,…,σ_m is not. Furthermore, the independence of random strings that generate negative examples is not also assumed. If we choose two different points x,x′ in ${\mathbb{R}}^p$ and then make two hyperspheres with centres x and x′ and equal radii, the measures (volumes for p=3) of the hyperspheres are equal. However, choosing two different strings s,s′ in A* and then making two spheres with centres s and s′ and equal radii with respect to the Levenshtein distance, the numbers of strings that the spheres contain are not necessarily equal, which is an essential reason why the statement of theorem 5.2 is divided into cases j=0,1,2.

6. Applications to biological sequence analysis

Our statistical learning machine classifies strings in an almost optimal manner under the conditions of theorems 5.1 and 5.2 in §5 when the training samples are sufficiently large. However, large training samples are not necessarily obtainable in all problems of classifying strings. How accurately does our machine classify strings in such cases? In this section, we examine the usefulness of our learning machine in practical data analysis by applying it to analysing amino acid sequences of proteins and nucleotide sequences of RNAs, including cases in which sufficiently large samples cannot be obtained.

(a). Application to predicting protein–protein interactions

A protein is a polymer of 20 types of amino acids and can be represented as a string on an alphabet A={a,…,z}−{b,j,o,u,x,z} composed of 20 letters. Predicting protein–protein interactions is one of the most important problems in bioinformatics, because most proteins fulfill their functions after forming a complex with other proteins. A domain of a protein generally interacts with multiple domains of other proteins. However, only a few proteins have domains that interact with a number of domains of other proteins and function as a hub in a protein–protein interaction network [24,25]. Thus, large numbers of positive examples cannot necessarily be obtained in the problem of predicting protein–protein interactions. We formulated this prediction problem as the problem of classifying domains of proteins into two classes of domains that interact and do not interact with a given domain and applied our machine working in A* to this problem. We examined the classification accuracy of our machine through comparisons with the SVM with the two-spectrum kernel (the dot product of spectral representations of two strings [13]), the SVM with the spatial kernel with t=2,k=1, and d=2 (see [26] for these parameters), and the one-nearest-neighbour method that employs outputs from the spatial kernel as dissimilarities.

We first prepared positive examples by using the three-dimensional interacting domains (3did) database [27], which contains high-resolution, three-dimensional structural data for domain–domain interactions obtained from the Protein Data Bank (PDB) [28]. The PDB includes three-dimensional structures of proteins and protein complexes obtained from experiments such as X-ray crystal structural analysis and nuclear magnetic resonance spectroscopy. We randomly selected amino acid sequences of 20 protein domains, ‘1il1 A:134–215’ (which denotes the amino acid subsequence from residue 134 to 215 in chain A of PDB ID 1il1), ‘2bnq E:127–220’, ‘1inq B:1011–1092’, ‘1it9 H:133–216’, ‘1it9 L:123–210’, ‘1ikv A:317–419’, ‘1cff A:84–145’, ‘1iza A:1–20’, ‘1ifh L:119–206’, ‘1p2c F:1501–1627’, ‘1j5o B:238–307’, ‘1mcz A:190–325’, ‘1at1 B:101–152’, ‘1ikv B:1238–1307’, ‘1lw6 E:27–275’, ‘3tu4 F:24–93’, ‘1jgw H:11–137’, ‘1z8u B:6–106’, ‘4e5x A:1–179’ and ‘4e5x B:11–92’. For each of the amino acid sequences of these 20 domains, we collected sequences of interacting domains from 3did without any redundancy in sequences. Sequences having extremely different lengths relative to the sequences collected from the database were not included in the samples of positive examples. It should be noted that the same amino acid sequence can be included in different entries of the PDB.

We next consider the procedure for preparing negative examples. Real sequences that would not be positive examples and artificial and randomly generated sequences have been used as negative examples in the development and validation of classifiers for string and sequence data. For example, in promoter prediction in Escherichia coli, Horton & Kanehisa [29] used sequences randomly chosen from coding regions as the negative examples. However, as Larsen et al. [30] indicated, the use of such negative examples is very different from a real biological discrimination problem. For the problem of predicting the interaction between miRNA and mRNA, recent studies [31–33] used randomly generated artificial sequences as the negative examples. However, as recent studies [34–36] demonstrated experimentally, such sequences often interact with miRNA, and therefore, it is uncertain if they are negative examples. Even if the randomly generated sequences are real negative examples, they may be unrealistically different from positive examples. In this case, as Bandyopadhyay & Mitra [37] indicated, the positive and negative examples are easily distinguishable, and a classifier that yields poor performance on other independent datasets may be produced.

Thus, in this study, we considered a procedure for preparing negative examples that were somewhat similar to the positive examples but were not likely to interact based on biophysical chemistry data. Bogan & Thorn [38] compiled a database of 2325 alanine mutants of heterodimeric protein–protein complexes and examined which amino acids are located in the interfaces of protein–protein complexes with a relatively high frequency. Ma et al. [39] identified amino acids that are located in the interfaces with a high frequency on the basis of 1629 two-chain interface entries in the PDB. Based on the results of these studies, Arg, Asp, Trp and Tyr are located at the interface of protein–protein interactions with high frequency, whereas Lys and Glu are located at interfaces with low frequency. Arg and Lys have a positive charge, and Arg tends to be located at interfaces; conversely, Lys does not tend to be located at interfaces. By contrast, Asp and Glu are negatively charged, whereas Asp tends to be located at interfaces, Glu does not. These observations imply that amino acids in which the side chain has relatively low entropy tend to be located at interfaces. π electrons in the side chains of aromatic amino acids interact strongly with positively charged amino acids [40], and, for this reason, aromatic amino acids such as Trp and Tyr are located at interfaces with high frequency. Therefore, we generated a negative example from each positive example using the following procedure.

Step (i). We first substituted Glu for Arg and Lys for Asp, Trp and Tyr in a positive example with a probability of 0.5 (because if all the Arg, Asp, Trp and Tyr were replaced, the resulting negative examples would not contain the four types of letters that represent these amino acids). According to van Holde et al. [41], the frequencies of Arg, Asp, Trp and Tyr in common proteins are 5.1%, 5.3%, 1.4% and 3.2%, respectively. Hence, in this step, approximately 7.5% of the letters in the positive example that represented amino acids that tend to be located in interfaces were replaced with letters representing amino acids that do not tend to be located in interfaces.

Step (ii). Next, we trisected a positive example with the substitutions in step (i), chose a letter in the intermediate substring at random, and transposed the order of the first half substring, from the first letter in the substituted positive example to the chosen letter, and the second half substring, composed of the other letters in the substituted positive example.

The sum of the numbers of positive and negative examples is given as N in table 1 (the number of positive examples = the number of negative examples =N/2 from the procedure of preparing the negative examples described in the above paragraph). Our examination included cases with an insufficient sample size N=20 up to a case with a large sample size N=126. We constructed the above-mentioned four machines for each of the 20 domains. We evaluated the performance of the machines using the four indices of accuracy =(TP+TN)/N, precision =TP/(TP+FP), recall =TP/(TP+FN) and F-measure =2×precision×recall/(precision+recall), where TP,FP,TN and FN represent the numbers of true-positives, false-positives, true-negatives and false-negatives, respectively. After randomly dividing each of the samples of positive and negative examples into three subsamples of equal size, we used the union of two subsamples as a training sample and the other subsample as a test sample to compute the above indices.

Table 1.

Results of the simulation experiments on the prediction of protein–protein interactions. The first row in each of the 10 panels presents ‘the PDB ID of the protein the chain: the initial residue number in the interaction site–the last residue number’. N and $\overline{l}$ denote the sample size and the mean length of the positive and negative examples, respectively. MMC A*, SVM spec, SVM spat and NN spat represent the developed learning machine working in A*, the SVM with the spectrum kernel, the SVM with the spatial kernel and the nearest-neighbour method that uses outputs from the spatial kernel as dissimilarities, respectively.

	MMC A*	SVM spec	SVM spat	NN spat	MMC A*	SVM spec	SVM spat	NN spat
	1il1 A:134–215 N=20 $\overline{l}=87.20$				2bnq E:127–220 N=22 $\overline{l}=79.45$
accuracy	0.9433	0.8267	0.8570	0.7163	1.0000	0.9475	1.0000	0.9864
precision	0.9500	0.8833	0.8538	0.6897	1.0000	1.0000	1.0000	0.9758
recall	0.9533	0.7933	0.8620	0.7900	1.0000	0.8950	1.0000	1.0000
F-measure	0.9516	0.8359	0.8554	0.7316	1.0000	0.9446	1.0000	0.9872
	1inq B:1011–1092 N=32 $\overline{l}=83.63$				1it9 H:133–216 N=36 $\overline{l}=87.28$
accuracy	0.9140	0.8380	0.8475	0.5318	0.9817	0.8267	0.9194	0.7796
precision	0.9105	0.8860	0.8180	0.5275	0.9914	0.9971	0.8937	0.7600
recall	0.9520	0.8080	0.8963	0.6723	0.9733	0.6567	0.9533	0.8186
F-measure	0.9308	0.8452	0.8547	0.5907	0.9823	0.7919	0.9224	0.7875
	1it9 L:123–210 N=40 $\overline{l}=84.05$				1ikv A:317–419 N=46 $\overline{l}=171.83$
accuracy	0.8900	0.8029	0.7840	0.5506	1.0000	1.0000	1.0000	0.8853
precision	0.8951	0.9527	0.7611	0.5443	1.0000	1.0000	1.0000	1.0000
recall	0.9086	0.6543	0.8300	0.7413	1.0000	1.0000	1.0000	0.7641
F-measure	0.9018	0.7758	0.7927	0.6264	1.0000	1.0000	1.0000	0.8653
	1cff A:84–145 N=56 $\overline{l}=20.57$				1iza A:1–20 N=88 $\overline{l}=25.02$
accuracy	0.8211	0.7944	0.6225	0.3816	0.9727	0.8160	0.8945	0.7364
precision	0.8846	0.8698	0.6128	0.4129	0.9782	0.9669	0.8575	0.6857
recall	0.7511	0.7311	0.6621	0.5107	0.9693	0.6560	0.9477	0.9379
F-measure	0.8124	0.7944	0.6353	0.4555	0.9737	0.7817	0.9001	0.7920
	1ifh L:119–206 N=96 $\overline{l}=82.08$				1p2c F:1501–1627 N=126 $\overline{l}=111.83$
accuracy	0.9738	0.9125	0.9567	0.7611	0.9562	0.9481	0.9649	0.8184
precision	0.9688	0.9663	0.9450	0.7218	0.9546	0.9841	0.9615	0.7952
recall	0.9825	0.8575	0.9700	0.8691	0.9610	0.9114	0.9689	0.8585
F-measure	0.9756	0.9087	0.9572	0.7882	0.9578	0.9464	0.9651	0.8254
	1j5o B:238–307 N=20 $\overline{l}=171.60$				1mcz A:190–325 N=20 $\overline{l}=169.00$
accuracy	1.0000	1.0000	0.9990	0.9680	1.0000	0.9567	1.0000	1.0000
precision	1.0000	1.0000	0.9982	1.0000	1.0000	1.0000	1.0000	1.0000
recall	1.0000	1.0000	1.0000	0.9360	1.0000	0.9133	1.0000	1.0000
F-measure	1.0000	1.0000	0.9990	0.9663	1.0000	0.9547	1.0000	1.0000
	1at1 B:101–152 N=22 $\overline{l}=93.18$				1ikv B:1238–1307 N=30 $\overline{l}=114.33$
accuracy	1.0000	0.9625	0.9982	0.9864	1.0000	0.9060	0.9193	0.9660
precision	1.0000	1.0000	0.9967	0.9750	1.0000	1.0000	0.8846	0.9376
recall	1.0000	0.9250	1.0000	1.0000	1.0000	0.8120	0.9667	0.9987
F-measure	1.0000	0.9610	0.9983	0.9870	1.0000	0.8962	0.9232	0.9671
	1lw6 E:27–275 N=38 $\overline{l}=63.0$				3tu4 F:24–93 N=40 $\overline{l}=74.20$
accuracy	0.9800	0.9317	0.9347	0.9251	0.9986	0.9771	0.9900	0.9372
precision	1.0000	0.9810	0.9117	0.8905	1.0000	1.0000	0.9810	0.9177
recall	0.9600	0.8867	0.9653	0.9716	0.9971	0.9543	1.0000	0.9650
F-measure	0.9796	0.9315	0.9366	0.9284	0.9985	0.9766	0.9902	0.9398
	1jgw H:11–137 N=52 $\overline{l}=252.77$				1z8u B:6–106 N=66 $\overline{l}=105.00$
accuracy	1.0000	0.9878	1.0000	0.9722	1.0000	0.9318	0.9906	0.9424
precision	1.0000	1.0000	1.0000	0.9479	1.0000	1.0000	0.9982	0.9503
recall	1.0000	0.9756	1.0000	1.0000	1.0000	0.8636	0.9830	0.9339
F-measure	1.0000	0.9876	1.0000	0.9732	1.0000	0.9268	0.9904	0.9419
	4e5x A:1–179 N=80 $\overline{l}=110.25$				4e5x B:11–92 N=126 $\overline{l}=178.84$
accuracy	0.9754	0.9669	0.9235	0.6077	1.0000	1.0000	1.0000	0.9710
precision	0.9777	0.9895	0.9207	0.5942	1.0000	1.0000	1.0000	0.9867
recall	0.9754	0.9446	0.9275	0.7011	1.0000	1.0000	1.0000	0.9549
F-measure	0.9765	0.9665	0.9238	0.6420	1.0000	1.0000	1.0000	0.9703

We calculated the mean accuracy, precision, recall and F-measure by repeating this process 50 times. The mean values obtained in this procedure are shown in table 1. Furthermore, values obtained by averaging these mean accuracies, precisions, recalls and F-measures over the 20 protein domains and their standard deviations are provided in table S1 in §S3 of the electronic supplementary material. Moreover, we tested the significance of the differences between the mean accuracies, precisions, recalls and F-measures for each pair of the four methods using the Wilcoxon signed-rank test. The results are shown in table S2. The differences are statistically significant, except for those between the mean precisions of the developed learning machine and the SVM with spectrum kernel and between the mean recalls of the SVM with the spectrum kernel and the nearest-neighbour method. The mean precision from the SVM with the spectrum kernel is higher than that from the SVM with the spatial kernel, whereas the SVM with the spatial kernel is higher than the SVM with the spectrum kernel in the other three indices. Therefore, the developed learning machine has the highest predictive power, and the nearest-neighbour method has the lowest predictive power. The predictive powers of the SVMs with the spectrum kernel and with the spatial kernel are almost equal.

(b). Application to classifying RNAs by the secondary structure

Next, we applied the learning machine developed in this study, the SVM with the spectrum kernel, the SVM with the spatial kernel and the nearest-neighbour method that uses outputs from the spatial kernel as dissimilarities to predicting whether RNAs belong to a family with similar secondary structures based on nucleotide sequences. We chose 10 RNA families registered in the Rfam database [42] at random. Each of the families is composed of RNAs that have similar secondary structures. We used sequence data of the RNAs in each family as positive examples after deleting gaps of the alignment in the sequences. Sequences that are not likely to form secondary structures similar to those of the RNAs in each family are needed for negative examples. Therefore, we prepared a negative sequence from each positive sequence according to step (ii) of the procedure described in §6a to change the positions of complementary pairs of nucleotides in the RNA sequences. For each family, a sample of negative sequences was prepared in the above-mentioned manner and sequences in other families were not used as negative examples.

As in §6a, we calculated the mean accuracy, precision, recall and F-measure by repeating the threefold cross-validation 50 times. The obtained mean values are shown in table 2. Furthermore, the means and standard deviations of the mean accuracies, precisions, recalls and F-measures from the four methods over the 10 RNA families are provided in table S1. Moreover, table S2 shows the results of testing the significance of the differences of the mean accuracies, precisions, recalls and F-measures between each pair of the four methods using the Wilcoxon signed-rank test. The differences of all of the four indices between the SVMs with the spectrum kernel and with the spatial kernel are not statistically significant. The differences between the other pairs of the four methods are statistically significant, except for those of the mean recalls between the SVM with the spectrum kernel and the nearest-neighbour method and between the SVM with the spatial kernel and the nearest-neighbour method. Therefore, from these tables, it is concluded that the developed learning machine has the highest predictive power, followed by the SVMs with the spectrum kernel and with the spatial kernel, and the nearest-neighbour method has the lowest predictive power, which is a result similar to that obtained in §6a.

Table 2.

Results of the simulation experiments on the classification of RNAs. The first row in each of the five panels presents the ID of the RNA family. N, $\overline{l}$, MMC A*, SVM spec, SVM spat, and NN spat are the same as in table 1.

	MMC A*	SVM spec	SVM spat	NN spat	MMC A*	SVM spec	SVM spat	NN spat
	RF01987 N=24 $\overline{l}=78.33$				RF02116 N=36 $\overline{l}=166.22$
accuracy	1.0000	0.7625	0.5983	0.4630	0.9833	0.6283	0.5044	0.4413
precision	1.0000	0.7763	0.5998	0.4774	0.9943	0.6496	0.5043	0.4618
recall	1.0000	0.8150	0.6083	0.7683	0.9733	0.6833	0.4833	0.6364
F-measure	1.0000	0.7952	0.6016	0.5859	0.9820	0.6660	0.4905	0.5323
	RF02142 N=42 $\overline{l}=181.52$				RF00216 N=46 $\overline{l}=302.26$
accuracy	0.9986	0.6486	0.7271	0.5726	1.0000	0.8175	0.8248	0.5996
precision	1.0000	0.8376	0.7394	0.5599	1.0000	0.9400	0.8073	0.5663
recall	0.9971	0.4514	0.7067	0.6353	1.0000	0.7000	0.8557	0.5105
F-measure	0.9985	0.5866	0.7203	0.5908	1.0000	0.8024	0.8302	0.5182
	RF00228 N=46 $\overline{l}=571.39$				RF02495 N=68 $\overline{l}=76.74$
accuracy	1.0000	0.6500	0.7948	0.4696	0.9991	0.7445	0.8715	0.5825
precision	1.0000	0.6355	0.8062	0.4794	1.0000	0.8521	0.8785	0.5581
recall	1.0000	0.8175	0.7817	0.7696	0.9982	0.6164	0.8653	0.7153
F-measure	1.0000	0.7151	0.7917	0.5882	0.9990	0.7153	0.8706	0.6257
	RF00042 N=74 $\overline{l}=90.05$				RF00061 N=158 $\overline{l}=246.95$
accuracy	1.0000	0.8333	0.8581	0.4922	0.6227	0.5131	0.4189	0.4842
precision	1.0000	0.8998	0.8600	0.4722	0.8550	0.5134	0.4212	0.4883
recall	1.0000	0.7633	0.8595	0.1493	0.3100	0.5277	0.4461	0.7100
F-measure	1.0000	0.8259	0.8583	0.2054	0.4438	0.5205	0.4326	0.5758
	RF01794 N=182 $\overline{l}=162.14$				RF00229 N=184 $\overline{l}=251.73$
accuracy	0.9463	0.6103	0.8625	0.5443	1.0000	0.6348	0.8433	0.4682
precision	0.9551	0.6168	0.8259	0.5319	1.0000	0.6613	0.8434	0.4707
recall	0.9393	0.6013	0.9196	0.6229	1.0000	0.5587	0.8443	0.5471
F-measure	0.9453	0.6090	0.8699	0.5722	1.0000	0.6057	0.8435	0.5051

7. Conclusion

String data analysis has a wide range of tasks, such as string comparison, quantifying string similarity, string clustering and identifying string features. In this study, we addressed the problem of classifying strings of these various problems. As described in §1, it is impossible to theoretically evaluate the performance of learning machines that convert strings into numerical vectors using string kernels, which are not bijective, and subsequently analyse the vectors, using probability theory. Therefore, to evaluate the performance of such learning machines, we have no option but to depend on the numerical method in which the cross-validation is repeated. However, as often pointed out, results of the performance evaluation in this manner vary greatly, depending on the datasets used. Dealing with these problems by applying the probability theory on A* developed in [3] is the main purpose of this study, and we conducted this study, placing special emphasis on the theoretical aspect of string data analysis.

Many methods for the above-mentioned tasks of string data analysis are based on string representations in the kernel-based methods. Many previous studies proved the kernel-based methods for string representations to be useful in practical string data analysis. See, for example, [26,43–52] for important recent studies on the kernel-based methods for string data. However, it is impossible to evaluate, applying traditional probability theory that has been constructed on spaces such as the Euclidean space ${\mathbb{R}}^p$ and Hilbert space L², the performance of the methods based on the string kernels, considering that a sample of observed strings is a part of a population generated according to a probability law. In this study, we addressed the problem of supervised classification of strings and obtained the theoretical results on the performance of the learning machine that works in A* by applying the limit theorem in probability theory on A* demonstrated in this paper. In [53], we approached the problem of unsupervised clustering of strings by introducing a parametric probability distribution on A* and developing a theory of EM algorithm for the mixture model of the distributions and demonstrated the asymptotic optimality of the constructed clustering procedure by combining the limit theorems obtained in this paper. It is a future challenge to develop a method for evaluating the performance of the kernel-based methods of string data analysis in a theoretical manner by extending probability theory on A* and inspecting the correspondence between observed strings and their representations by string kernels.

Ethics

There are no ethical concerns with regard to this manuscript.

Data accessibility

All data used in this study are openly available from the 3did and Rfam databases. The source code for classifying strings using the learning machine that works in A* constructed in this study is available if requested.

Author' contributions

H.K. conceived of the study, constructed the theory, performed the simulation experiments and drafted the manuscript. M.H. helped to perform the simulation experiments. T.A. helped to constructed the theory.

Competing interests

The authors declare that they have no competing interests.

Funding

This work was supported in part by grant-in-aid for Challenging Exploratory Research from the Japan Society for the Promotion of Science (26610037).