Bayesian Non-linear Support Vector Machine for High-Dimensional Data with Incorporation of Graph Information on Features

Abstract

Support vector machine (SVM) is a popular classification method for analysis of high dimensional data such as genomics data. Recently a number of linear SVM methods have been developed to achieve feature selection through either frequentist regularization or Bayesian shrinkage, but the linear assumption may not be plausible for many real applications. In addition, recent work has demonstrated that incorporating known biological knowledge, such as those from functional genomics, into the statistical analysis of genomic data offers great promise of improved predictive accuracy and feature selection. Such biological knowledge can often be represented by graphs. In this article, we propose a novel knowledge-guided nonlinear Bayesian SVM approach for analysis of high-dimensional data. Our model uses graph information that represents the relationship among the features to guide feature selection. To achieve knowledge-guided feature selection, we assign an Ising prior to the indicators representing inclusion/exclusion of the features in the model. An efficient MCMC algorithm is developed for posterior inference. The performance of our method is evaluated and compared with several penalized linear SVM and the standard kernel SVM method in terms of prediction and feature selection in extensive simulation studies. Also, analyses of genomic data from a cancer study show that our method yields a more accurate prediction model for patient survival and reveals biologically more meaningful results than the existing methods.

I. Introduction

Recently, rapid advances in high-throughput technologies have generated a large amount of omics data such as gene expressions data. As a result, new challenges have emerged related to the analysis and interpretation of such omics data. For instance, in genomics studies, the number of gene expression features is often much larger than the sample size. Because of this high dimensionality, one of the challenges is to avoid over-fitting the data. Another challenge is feature selection, i.e., selection of a subset of informative features, leading to more interpretable results.

The linear support vector machine (SVM) is a popular technique to handle such high dimensionality and has been extended to select informative features by applying penalties on the coefficients. Bradley et al. [1], Song et al. [2] and Zhu et al. [3] applied the LASSO technique [4] into SVM (L₁-SVM); Wang et al. [5] proposed a double regularization SVM (Dr-SVM), which combines the L₁ and ridge penalties to encourage the selection of correlated features. Zhang et al. [6] and Becker et al. [7] adopted the smoothly clipped absolute deviation (SCADSVM) penalty [8] to eliminate biases in estimating nonzero coefficients; Zou et al. [9] suggested the L_∞ penalized SVM to encourage all the features in the same group to be selected simultaneously. In addition to frequentist approaches, Bayesian SVM with variable selection methods have received much attention recently with many successful applications [10], [11]. Bayesian approaches can naturally incorporate the prior knowledge and make posterior inference explaining uncertainty of model parameters. Sun et al. [12] proposed a knowledge-guided Bayesian linear SVM which enables incorporation of the prior network information among predictors, known as KBSVM, and shows that it outperforms a number of preexisting linear SVM approaches that do not take advantage of such knowledge.

However, if the data are not linearly separable, the existing linear SVM methods may not be adequate. To address this problem, the non-linear separation for SVM can be realized by mapping the original data into some high dimensional feature space where the data is linearly separable and constructing an optimal hyperplane in this space. This mapping is performed by a kernel function, which is referred to as ”the kernel trick” (Vapnik et al. [13]). Although it is more challenging to perform feature selection for non-linear SVM, some techniques based on the frequentist framework have been developed. Weston et al. [14] reduced the feature dimensions by minimizing bounds on the leave-one-out error via a gradient approach; Zhang et al. [15] implemented a smoothing spline ANOVA framework to conduct simultaneous classification and feature selection; Mangasarian and Wild [16] proposed an approach that inserts a diagonal indicator matrix into the non-linear kernel and minimizes the objective function as well as the number of features selected. There is little work on feature selection in the Bayesian non-linear SVM framework, to the best of our knowledge.

In this work, we propose to incorporate the prior graph information such as pathways from functional genomics to further guide feature selection. One of the primary motivations for incorporating such pathways information is that weak signals are often grouped into pathways and accounting for the structure information among them has the potential to increase power of detecting key signatures and yield biologically more meaningful results. Such informative priors for related features often lie on undirected acyclic graphs where nodes represent genes and edges represent functional interactions between genes. Some recent works use the known graph or network information describing the relationships between features to guide feature selection, which leads to improvement in prediction and feature selection, especially for high dimensional data. For example, Li and Li [17] proposed a network-contrained penalty to encourage smoothness of the connected features with respect to a graph; Pan et al. [18] developed a group penalty using the weighted L_γ norm to realize ”grouped” feature selection; Kim et al. [19] proposed a nonconvex penalty without assuming the coefficients for the connected features being similar. In the Bayesian framework, Li and Zhang [20] and Stingo and Vannucci [21] proposed spike and slab priors combined with the Markov Random Field (MRF) prior to encourage the joint selection of features. Zhou and Zheng [22] developed a Bayesian random graph-constrained model to allow uncertainty over the graph. More recently, Chang et al. [23] developed a Bayesian shrinkage approach by assigning independent Laplace priors on the regression coecients, while incorporating the graph information via the hyperprior imposed on the shrinkage parameters of the Laplace distributions.

However, the above-mentioned approaches for incorporating graph information are only applicable to linear models and no existing work has been developed for non-linear SVM. To fill this important gap, we propose a Bayesian non-linear SVM feature selection method with incorporation of the graph information (BNSVM). The non-linear classifier in our model is assumed to be drawn from a zero mean Gaussian process (Henao et al. [24]) with a special covariance matrix. This covariance matrix is constructed by the usual non-linear kernel function embedded with latent binary variables representing the selection status of features. Furthermore, Ising priors are assigned to the latent binary variables to incorporate the graphical structure of the features. This Ising prior allows our model to encourage both group-wise inclusion and exclusion of neighboring features, and therefore, further improve the prediction performance.

By using the data augmentation techniques developed by Polson et al. [25], we re-express the likelihood, incorporate the graph-guided priors, and employ the Metropolis Hastings (MH) sampling within Gibbs sampling algorithm to perform Bayesian inference and prediction. The performance of our method is investigated by extensive simulation studies in comparison with L₁-SVM, standard linear SVM (L₂-SVM) and the knowledge-guided Bayesian linear SVM method (KBSVM [12]) as well as the non-linear SVM (Kernel-SVM) methods in terms of prediction and feature selection. Of note, among these existing methods, only L₁-SVM and KBSVM can perform feature selection. We also apply our methods to a glioblastoma cancer study with a large number of genes, and construct a classification model to predict the survival status and identify the subset of genes that are predictive of patient survival.

The remainder of the paper is organzed as follows. Section II describes the Bayesian model and the MCMC algorithm. Section III presents the algorithm for prediction. Section IV and Section V present the simulation study and the real data application, respectively. Section VI concludes the paper with brief discussion remarks.

II. Model

A. Likelihood

Let $D={\left\{x_i,y_i\right\}}_{i=1}^n$ be n samples in the training set, where $x_i\in {ℝ}^p$ are the feature inputs and y_i ∈ {−1, 1} are the corresponding binary labels. Our task is to learn a classification rule from the training set so that we can assign a class label to any new subject observed in the future. The linear SVM is a large margin classifier which seperates two classes by maximizing the margin between them. The classical SVM seeks to find the optimal function f by solving the following regularized hinge loss objective:

$$\widehat{f}=\underset{f}{\text{arg}\text{min}}\left(\sum _{i=1}^N{\left(1-y_{i}f\left(x_i\right)\right)}_{+}+\lambda R(f)\right),$$ (1)

where (x)₊ ≡ max(x, 0) and R(f) is a regularization term reflecting the complexity of f. λ is a tuning parameter controlling the tradeoff between error minimization and the complexity of f. The classification function is then given by sign $(\widehat{f}(x))$. Note that the optimization problem (1) is equivalent to finding the mode of the following pseudo-posterior:

$$\pi (f|X,y,\nu )\propto \pi (f|X)L(y|X,f,\nu )$$ (2)

$$\propto \pi (f|X)\prod _{i=1}^n\nu e^{-2\nu \text{max}\left(1-y_{i}f\left(x_i\right),0\right)}$$ (3)

$$\propto \pi (f|X)\prod _{i=1}^n{\int }_0^{\infty }\frac{\sqrt{\nu }}{\sqrt{2\pi {\rho }_i}}{e}^{-\frac{\nu {\left({\rho }_i+1-y_{i}f\left(x_i\right)\right)}^2}{2{\rho }_i}}d{\rho }_i$$ (4)

The proposed pseudo-likelihood enables ν in (3) to learn the overall scale of the errors and a Gamma prior $\mathcal{G}\left(a_{\nu },b_{\nu }\right)$ is assigned for ν. a_ν represents the shape parameter, b_ν represents the rate parameters of the Gamma distribution, and the values can be tuned in an uninformative or data-driven manner. (4) re-expresses (3) as a location-scale mixture of Gaussians by introducing a latent variable ρ_i (Polson et al. [25]), which facilitates Gibbs sampling as described in Section II-D. Following Henao et al. [24], we assume the non-linear classifiers f(x_i) to be drawn from a zero-mean Gaussian process $\mathcal{G}\mathcal{P}(0,K)$. The details are given in the following session.

B. Prior for the non-linear classifier f(x)

A Gaussian process is a collection of random variables where the joint distribution of any combination of the variables is Gaussian. Such a process f(x) is completely determined by its mean function μ(x) = E(f(x)) and the covariance kernel function k(x, x′) = E[(f(x) − μ(x))(f(x′) − μ(x′))]. We assume the prior mean functions μ(x) to be zero, which implies that there is no preference of positive or negative values for the mean given no data.

In our case, the random variables f₁, … , f_n replace f(x₁), … , f(x_n) and their prior covariance matrix K ≡ K(γ) ≡ K(γ, X) is given by an n × n matrix with non-linear kernels $k\left(x_i,x_j\right)={\varphi }^{\sum _{l=1}^p{\gamma }_l{\left(x_{il}-x_{jl}\right)}^2}$, where ϕ ∈ (0, 1) and γ_l is the latent binary variable indicating the inclusion or exclusion of the lth feature of the model. This covariance structure encourages f_i and f_j to be highly correlated when x_i and x_j are close, or uncorrelated if x_i and x_j are far enough. The parameter ϕ determines the sensitivity of correlation to the distance. We assign the uniform distribution as the prior for ϕ.

The contribution of our work is to insert a binary variable γ_l for each feature l into the kernel function controlling the number of features used in the model.

C. Ising prior for γ

γ_l plays an important role of performing selection of each feature l. Usually, the iid Bernoulli prior is assigned for γ, allowing the predictors to be independently selected, while ignoring the underlying structure information among predictors. The prior structrual information of predictors is represented by a graph $\mathcal{G}=\langle V,E\rangle$, where V = {1, … , p} represents the set of and the edge set E ⊂ {(j, k) : j, k ∈ V, j ≠ k} represents associations between the predictors. To take into account the fact that adjacent features are likely to influene the response jointly, we take the Ising prior for γ given as follows.

$$\pi (\gamma )=C_{\mu ,\eta }{e}^{-\mu {\sum }_j{\gamma }_j+\eta {\sum }_{j\ne k}{G}_{jk}\mathbb{I}\left({\gamma }_j={\gamma }_k\right)},$$ (5)

where C_μ,η is the normalizing constant and $\mathbb{I}(\cdot )$ is the indicator function. Here, G is the adjacency matrix of $\mathcal{G};G_{jk}$ indicates the presence of an edge between the predictors j and k. The tuning parameters μ controls the sparsity of γ and η controls the smoothness of γ over E. When γ_j = 1, its neighbors are more likely to stay at ”1”. Similarly, when γ_j = 0, its neighbors are likely to stay at ”0”. Thus, the Ising prior will encourage the group-wise feature selection of the j-th and the k-th features if there is an edge between them.

D. Posterior Inference and MCMC

The full data pseudo-posterior distribution is given by

$$\begin{gathered} \pi (\nu ,\rho ,\gamma ,\varphi ,f|y,X)\propto \prod _{i=1}^n\frac{\sqrt{\nu }}{\sqrt{2\pi {\rho }_i}}{e}^{-\frac{\nu {\left({\rho }_i+1-y_i{f}_i\right)}^2}{2{\rho }_i}} \\ \times e^{-\mu {\sum }_j{\gamma }_j+\eta {\sum }_{j\ne k}{G}_{jk}\mathbb{I}\left({\gamma }_j={\gamma }_k\right)} \\ \times |K{|}^{-\frac{1}{2}}{e}^{-\frac{1}{2}{f}^T{K}^{-1}f}\times {\nu }^{a_{\nu }-1}{e}^{-b_{\nu }\nu }. \end{gathered}$$

MCMC is implemented by Metropolis Hastings within Gibbs algorithm. Most of the conditional distributions can be easily sampled. We have

$$\nu |f,y,\rho ~\mathcal{G}\left(a_{\nu }+\frac{3n}{2},b_{\nu }+\sum _{i=1}^n\frac{{\left({\rho }_i+1-y_i{f}_i\right)}^2}{2{\rho }_i}\right),$$ (6)

and we have

$${\rho }_i|f,y,\nu ~\mathcal{G}\mathcal{I}\mathcal{N}\left(1/2,\nu ,\nu {\left(1-y_i{f}_i\right)}^2\right),$$

where $\mathcal{G}\mathcal{I}\mathcal{N}\left(p,a,b\right)$ stands for the Generalized Inverse Gaussian distribution. Alternatively, the conditional distribution of ${\rho }_i^{-1}$ is an inverse Gaussian distribution, denoted by $\mathcal{I}\mathcal{N}$.

$${\rho }_i^{-1}|f,y,\nu ~\mathcal{I}\mathcal{N}\left({|1-y_i{f}_i|}^{-1},\nu \right),$$ (7)

where the density function of $\mathcal{I}\mathcal{N}\left(\mu ,\lambda \right)$ is defined as below:

$$f(x;\mu ,\lambda )=\frac{\sqrt{\lambda }}{\sqrt{2\pi x^3}}{e}^{\frac{-\lambda {(x-\mu )}^2}{2{\mu }^{2}x}}.$$

The conditional distribution of γ_j follows the Bernoulli distribution which is given by

$${\gamma }_j|{\gamma }_{-j},f,X~Ber\left(\frac{\prod \left({\gamma }_j=1,{\gamma }_{-j}\right)}{\prod \left({\gamma }_j=1,{\gamma }_{-j}\right)+\prod \left({\gamma }_j=0,{\gamma }_{-j}\right)}\right),$$ (8)

where γ_−j = (γ₁, … , γ_j−1, γ_j+1, … , γ_p) and $\prod \left(\gamma \right)=|K(\gamma ){|}^{-\frac{1}{2}}{e}^{-\frac{1}{2}{f}^{T}K{(\gamma )}^{-1}f}{e}^{-\mu {\gamma }_j+\eta \sum _{k\ne j}{G}_{jk}I\left({\gamma }_j={\gamma }_k\right)}$.

The conditional distribution of ϕ is given by

$$\pi (\varphi |f,\gamma ,X)\propto |K{|}^{-\frac{1}{2}}{e}^{-\frac{1}{2}{f}^T{K}^{-1}f}$$ (9)

We use the Metropolis-Hasting algorithm to draw samples from π(ϕ|f, γ, X). Since ϕ ∈ (0, 1), the logit normal distribution $q\left(x;\mu ,{\tau }^2\right)=\frac{1}{\tau \sqrt{2\pi }}\frac{1}{x(1-x)}{e}^{-\frac{{(\text{logit}(x)-\mu )}^2}{2{\tau }^2}}$ is chosen as the proposed distribution. Assume the last state is ϕ^t−1, we draw a sample ϕ* from q(x; ϕ^t−1, τ²), where τ² is set to a value that keeps the acceptance rate around 40%. We accept or reject the current proposed ϕ* with probability α_t.

$${\alpha }_t=\text{min}\left(1,\frac{\pi \left({\varphi }^{*}|f,\gamma ,X\right){\varphi }^{*}\left(1-{\varphi }^{*}\right)}{\pi \left({\varphi }^{t-1}|f,\gamma ,X\right){\varphi }^{t-1}\left(1-{\varphi }^{t-1}\right)}\right)$$

Let D_ρ = diag(ρ₁, … , ρ_n) and z be a vector with entries $z_i=\left(1+{\rho }_i^{-1}\right)y_i$. The conditional distribution for f is a multivariate Gaussian:

$$f|y,\nu ,\rho ,K~\mathcal{N}\left(\nu {\left(\nu D_{\rho }^{-1}+K^{-1}\right)}^{-1}z,{\left(\nu D_{\rho }^{-1}+K^{-1}\right)}^{-1}\right)$$ (10)

III. Prediction

To predict the label for a new testing case, the inference is divided into two steps: first computing the distribution of the latent non-linear function f* corresponding to the test case; second cpmputing the predictive label probabilities of the test case.

Let θ = (γ, ν, ρ, ϕ)^T denote the vector of model parameters. The predictive distribution of f* for a new testing data vector $x_{p\times 1}^{*}$ given the training dataset X, y and θ can be written as

$$f^{*}|x^{*},X,y,\theta ~\mathcal{N}\left(\mu ,{\sigma }^2\right),$$ (11)

where

$$k^{*}={\left(k\left(x^{*},x_1\right),\dots ,k\left(x^{*},x_n\right)\right)}^T,$$

$$k^{*}=k\left(x^{*},x^{*}\right),$$

$$\Sigma ={\left(K+{\nu }^{-1}{D}_{\rho }\right)}^{-1},$$

$$\mu =k^{*T}\Sigma D_{\rho }z,$$

$${\sigma }^2=k^{*}-k^{*T}\Sigma k^{*}.$$

Then, we can use the probit link to compute the conditional predictive class probabilities.

$$\begin{gathered} \pi \left(y^{*}=1|x^{*},X,y,\theta \right)=\int \Phi \left(f^{*}\right)\pi \left(f^{*}|x^{*},X,y,\mathit{\theta }\right)d{f}^{*} \\ =\Phi \left(\frac{k^{*T}\Sigma D_{\rho }z}{\sqrt{1+k^{*}-k^{*T}\Sigma k^{*}}}\right). \end{gathered}$$ (12)

The derivations of (11) and (12) can be found in Appendix. To estimate the marginal predictive probability π(y* = 1|x*, X, y), the MCMC samples of θ are used.

$$\widehat{p}=\frac{1}{M}\sum _{m=1}^{M}p\left(y^{*}=1|x^{*},X,y,{\theta }_m\right),$$ (13)

where θ₁, … , θ_M are the MCMC samples of θ.

The prediction error can be measured by the cross-entropy between the predictive probabilities and the actual class.

$$\text{CE}=-\sum _{i=1}^N\mathbb{I}\left(y_i^{*}=1\right)\text{log}{\widehat{p}}_i-\sum _{i=1}^N\mathbb{I}\left(y_i^{*}=-1\right)\text{log}\left(1-{\widehat{p}}_i\right),$$

where N is the sample size of testing data. The decision can be made by ${\widehat{y}}_i=\text{sign}\left({\widehat{p}}_i-0.5\right)$ and the associated prediction error can be reported as follows.

$$\text{PE}=\frac{1}{N}\sum _{i=1}^N\mathbb{I}\left({\widehat{y}}_i\ne y_i^{*}\right).$$ (14)

IV. Simulations

A. Design of Experiment

In this section, we study the performance of our BNSVM methods through the simulated additive models. We simulate the examples for both the graph-related covariance structure and independent covariance structure for the input features. In each experimental setting, we generate 100 datasets, each with a training sample for fitting, a validation sample for tuning and an independent test sample for computing the following performance metrics: the prediction error (PE), prediction sensitivity (PSEN), prediction specificity (PSPEC), Matthews Correlation Coefficients (MCC), feature selection true positive (FSTP) and feature selection false positive (FSFP).

For comparision, prediction errors are reported by (14) as not all methods provide probabilistic prediciton. The prediction sensitivity is calculated as the proportion of positives (y = 1) that are correctly identified and the prediction specificity is calculated as the proportion of negatives (y = −1) that are correctly identified. MCC is defined as

$$\frac{TP\times TN-FP\times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$$ (15)

where TP is the number of true positives, TN is the number of true negatives, FP is the number of false positives and FN is the number of false negatives. FSTP is the percentage of important features selected by the model among the total important features and FSFP is the percentage of unimportant features selected by the model among the total unimportant features.

The sample size for training, validation and testing data is 200, and the feature dimension p is set at 20, 100 and 500, representing both n > p and n < p cases. In addition to L₁-SVM, L₂-SVM (the linear SVM with L₂ penalty) and Kernel-SVM (the standard non-linear SVM), our method is compared with KBSVM with the correctly specified graph (KBSVM, G* = G) and a simplified version of KBSVM without using graph information (KBSVM, η = 0). Of note, Sun et al. [12] showed that KBSVM outperforms several penalized linear SVM methods such as Dr-SVM and SCADSVM, and provided additional details of KBSVM. Tables 1 and 2 present the average performance metrics over the 100 simulated datasets for each simulation setting.

Table I.

Simulation results for correlated structure among features. p is the total number of features, and q is the number of relevant features. η = 0 represents the working graph G* is not incorporated in KBSVM and BNSVM methods. – indicates no feature selection for the corresponding method.

Method	PE (%)	PSen (%)	PSpec(%)	MCC (%)	FSTP (%)	FSFP (%)
p = 20, q =10
L1-SVM	48.30 (1.15)	43.41 (3.61)	59.80 (4.67)	4.15 (2.80)	53.00 (11.35)	61.00 (11.81)
L2-SVM	49.20 (0.86)	46.55 (2.30)	54.93 (2.75)	1.56 (1.72)	–	–
KBSVM, η = 0	46.10 (0.60)	39.87 (3.21)	62.61 (2.74)	9.38 (0.80)	22.00 (8.24)	14.00 (9.70)
KBSVM, G* = G	45.15 (0.58)	40.09 (4.92)	62.83 (5.68)	10.82 (1.38)	24.00 (8.41)	12.00 (9.23)
Kernel-SVM	17.35 (0.62)	79.90 (0.98)	85.31 (1.41)	65.49 (1.27)	–	–
BNSVM, η = 0	16.00 (0.73)	67.04 (2.49)	91.73 (1.17)	60.85 (2.92)	95.00 (2.52)	11.00 (1.68)
BNSVM, G* = noisyG	17.80 (0.83)	70.15 (1.10)	93.11 (1.33)	66.01 (1.74)	95.00 (3.20)	10.00 (1.98)
BNSVM, G* = partialG	16.35 (0.89)	72.73 (1.43)	94.29 (0.83)	68.83 (1.70)	98.00 (1.25)	11.00 (2.19)
BNSVM, G* = G	15.50 (0.52)	73.54 (1.06)	94.18 (1.01)	69.42 (1.50)	100.00 (0.00)	7.00 (2.00)
p = 100, q = 20
L1-SVM	46.60 (0.81)	53.75 (1.60)	53.01 (2.14)	6.79 (1.61)	59.50 (9.01)	62.00 (7.98)
L2-SVM	48.75 (0.83)	46.43 (0.94)	51.00 (1.54)	1.15 (1.82)	–	–
KBSVM, η = 0	46.50 (0.68)	40.63 (6.62)	56.77 (1.82)	7.57 (1.31)	42.00 (8.30)	36.13 (9.85)
KBSVM, G* = G	45.50 (0.61)	46.77 (5.01)	61.64 (4.16)	9.08 (1.19)	25.00 (6.70)	21.12 (7.11)
Kernel-SVM	34.75 (1.01)	63.42 (2.62)	66.81 (1.99)	30.55 (2.07)	–	–
BNSVM, η = 0	30.95 (1.36)	55.31 (3.47)	78.67 (6.96)	42.13 (3.16)	62.00 (11.84)	8.25 (6.96)
BNSVM, G* = noisyG	35.10 (1.90)	43.00 (4.37)	75.16 (2.91)	34.94 (3.75)	56.00 (8.43)	12.25 (3.86)
BNSVM, G* = partialG	32.50 (1.12)	52.83 (2.30)	85.15 (3.40)	37.19 (3.09)	64.50 (8.99)	5.75 (1.33)
BNSVM, G* = G	27.50 (0.94)	74.05 (3.78)	71.01 (3.79)	46.32 (1.66)	66.50 (6.78)	5.13 (1.32)
p = 500, q = 20
L1-SVM	46.00 (1.19)	52.21 (1.77)	55.86 (1.31)	8.03 (2.39)	20.50 (4.33)	21.34 (3.31)
L2-SVM	48.30 (1.13)	47.56 (2.04)	49.25 (1.49)	3.44 (2.27)	–	–
KBSVM, η = 0	45.90 (1.01)	42.47 (7.16)	51.75 (2.10)	7.70 (2.04)	44.00 (12.20)	37.10 (13.07)
KBSVM, G* = G	44.90 (0.71)	44.68 (6.00)	55.92 (1.37)	12.31 (0.83)	48.00 (13.33)	46.73 (13.69)
Kernel-SVM	45.05 (0.76)	58.02 (4.47)	51.78 (5.33)	10.20 (1.52)	–	–
BNSVM, η = 0	36.40 (0.82)	55.13 (1.98)	74.00 (2.87)	34.19 (2.64)	62.00 (8.67)	1.27 (0.31)
BNSVM, G* = noisyG	42.50 (0.38)	48.80 (2.45)	69.07 (1.69)	31.82 (2.70)	53.00 (7.69)	3.33 (0.41)
BNSVM, G* = partialG	37.56 (1.32)	52.53 (3.11)	74.95 (2.53)	35.47 (2.38)	63.00 (8.62)	1.56 (0.31)
BNSVM, G* = G	33.35 (1.01)	57.48 (2.50)	78.09 (2.50)	36.73 (2.84)	65.50 (8.85)	1.40 (0.29)

Table II.

Simulation results for independent structure among features. – indicates no feature selection for the corresponding method.

Method	PE (%)	PSen (%)	PSpec(%)	MCC (%)	FSTP (%)	FSFP (%)
p = 20, q =10
L1-SVM	47.89 (0.71)	41.12 (3.24)	63.98 (3.64)	4.22 (1.00)	42.22 (9.97)	41.11 (8.65)
L2-SVM	48.55 (1.04)	41.06 (1.94)	56.21 (2.87)	3.12 (2.12)	–	–
KBSVM, η = 0	45.55 (0.73)	33.79 (4.20)	74.83 (3.43)	9.58 (1.35)	28.00 (7.23)	19.00 (9.63)
Kernel-SVM	20.20 (0.68)	79.30 (1.02)	80.29 (1.48)	50.74 (1.35)	–	–
BNSVM, η = 0	20.95 (1.06)	86.00 (2.41)	72.11 (3.00)	59.43 (2.02)	97.00 (2.00)	20 (5.76)
BNSVM, G* = noisyG	21.60 (2.49)	84.30 (1.32)	72.58 (5.05)	57.50 (4.71)	90.00 (8.39)	25.00 (6.13)
p = 100, q = 20
L1-SVM	47.70 (0.92)	51.38 (1.70)	54.96 (1.06)	4.55 (1.84)	69.00 (7.38)	70.62 (6.56)
L2-SVM	50.00 (1.22)	47.18 (1.50)	52.39 (1.61)	3.18 (2.22)	–	–
KBSVM, η = 0	46.20 (0.74)	48.42 (4.19)	57.02 (3.63)	5.47 (2.25)	43.00 (11.57)	44.12 (11.67)
Kernel-SVM	38.95 (1.33)	65.91 (4.68)	55.77 (3.01)	22.86 (3.14)	–	–
BNSVM, η = 0	32.87 (0.95)	55.53 (4.15)	59.38 (2.07)	34.32 (1.58)	63.50 (10.76)	1.85 (0.34)
BNSVM, G* = noisyG	35.88 (0.88)	43.83 (4.13)	71.99 (4.44)	29.28 (3.79)	54.50 (12.74)	22.63 (12.18)
p = 500, q = 20
L1-SVM	45.50 (0.91)	53.24 (1.02)	54.07 (1.80)	9.04 (1.80)	21.00 (4.00)	20.90 (2.43)
L2-SVM	49.85 (1.13)	48.98 (1.73)	51.49 (1.01)	2.32 (2.16)	–	–
KBSVM, η = 0	44.85 (0.67)	42.38 (3.87)	56.05 (1.62)	8.83 (1.53)	60.00 (12.60)	55.62 (13.98)
Kernel-SVM	45.35 (0.88)	47.66 (4.70)	62.54 (4.12)	10.69 (1.51)	–	–
BNSVM, η = 0	36.97 (0.76)	49.11 (1.52)	66.06 (2.58)	30.20 (1.22)	59.50 (9.41)	1.85 (0.34)
BNSVM, G* = noisyG	41.67 (0.89)	44.80 (1.93)	65.49 (3.08)	27.98 (1.04)	56.00 (9.68)	2.04 (0.31)

B. Parameter Tuning

For L₁-SVM, L₂-SVM and Kernel-SVM, we use the penalizedSVM R-package [26] to tune the parameters in the validation datasets. For KBSVM, three parameters (μ, η and ${\sigma }_0^2$) need to be tuned. For our method BNSVM, when η = 0, representing no graph incorporated, only μ needs to be tuned; when η ≠ 0, representing the graph information is incorporated, two parameters (μ, η) need to be tuned to achieve the best performance in terms of PE.

C. Generating data from an underlying true graph

The additive model is used to demonstrate the benefits of incorporating prior network information into our BNSVM method. The general idea is that the covariance structure of the simulated data has the graph information embedded, which mimics the genetic data with underlying interactions between genes. If we utilize the known graph to guide our algorithm, we should be able to improve the prediction performance and identify the relevant features. As mentioned, we use the undirected graph $\mathcal{G}$ to represent the network among predictors. The model can be written as : $x~\mathcal{N}\left(0,{\Omega }_{p\times p}^{-1}\right)$, $f(x)=x_1^2+x_2^2+\dots +x_q^2$, where x₁, …, x_q are the first q dimensions of x and f is only relevant with the first q features. The binary response y is determined by a cut-off value of f(x), which divides the two classes almost equally. The precision matrix Ω = (ω_ij) is the inverse of the covariance matrix with each entry

$${\omega }_{ij}=\{\begin{array}{ll}0,\hfill & (i,j)\notin E\hfill \\ {\rho }_{ij}\in [-1,1],\hfill & \text{ otherwise }\hfill \end{array}$$

Here we adopt the Gaussian graphical model and allow the precision matrix to represent the connection strength between predictors. Thus, the precision matrix Ω of x should have a similar pattern to G which is the adjacency matrix G.

Fig. 1 shows the procedure of how to generate the covariance matrix from the graph. First, we pre-define a undirected acyclic graph $\mathcal{G}$, which has p = 20 predictors and the first q = 10 are the important features, then we generate the corresponding adjacency matrix G which is a symmetric p×p matrix, with element ”1” representing the edge between connected predictors and ”0” representing no edges. Note the diagonal elements of G are 0 because each predictor itself is not connected. Second, we generate the same size p×p matrix with random numbers over an interval of [−1, 1] for each edge. Third, the smallest eigenvalue of the resulting matrix is calculated, if it is positive, the precision matrix can be acquired by resaling the diagonal elements to be 1; if negative, some small number is added on the diagonal of the resulting matrix to make it positive definite, then rescale and obtain the precision matrix. The covariance matrix is acquired by inverting the precision matrix.

Fig. 1.

The true graph $\mathcal{G}$ (a) with the subsets in red are the relevant q features, the corresponding adjacency matrix G (b), precision matrix Ω (c) and covariance matrix Σ (d)

D. Generating working graph

Once the simulated data with graph-related covariance structure is acquired from IV-C, we use the graph to guide our algorithm to make classification and feature selection. However, the graph might not be correct in practice. For example, in genetic study, the pathways in the database might be incomplete or noisy. To mimic these situations, we define the working graph adjacency matrix G* under three conditions: 1. G indicating that the truth is known, 2. a partial graph (partialG) indicating that the truth is partially known or 3. a noisy graph (noisyG) indicating that the graph is completely random. Fig. 2 shows the three conditions.

Fig. 2.

Three settings for working graph G*.

The partial graph can be generated by removing some weak signals of the original precision matrix Ω (Fig. 2(b)). If the absolute value of the elements of Ω less than a pre-set value, then they are set to zero; The resulting matrix is converted to the binary matrix with zeros and ones, and the diagonal elements are set to zero as shown in Fig. 2(d).

The noisy graph can be generated by randomly assigning ”1” on the lower triangle part of the corresponding adjacency matrix and making the upper part the same as the lower part. A noisy graph example is shown in Figure 2(e).

E. Simulation results

Four settings are specified for our method BNSVM: no working graph incorporated (η = 0), the working graph G* is assigned by a noisy graph (noisyG), a partial graph (partialG) and the true underlying graph (G). Two settings are specified for KBSVM: no working graph incorporated (η = 0), the working graph G* is assigned by the true graph. Table I summarizes the simulation results for both n > p and n < p cases.

When p = 20 and q = 10, our method BNSVM(G* = G) gives the smallest PE and the highest MCC. The prediction performance for the four linear methods L₁-SVM, L₂-SVM, KBSVM(η = 0) and KBSVM(G* = G) are similar. The non-linear Kernel-SVM gives a relatively low PE; while even no graph guided (η = 0), our method has a lower PE and high MCC, comparing to the settings of G* = noisyG and G* = partialG. When the working graph is assigned by noisyG, PE increase about 15% comparing to the setting of BNSVM(G* = G), while still gives satisfactory MCC, FSTP and FSFP. When the working graph is assigned by the true graph G, the lowest PE is achieved, moreover, it discovers all the relevant features while with the lowest FSFP at 7%. Note that when the graph is incorporated, the PE of the KBSVM drops 2% and the FSTP increases 9%, which confirms the advantage of incorporating prior knowledge even the wrong model is applied.

When p = 100 and q = 20, PE for BNSVM(G* = G) still gives the smallest PE comparing to the other settings and other existing methods. When the working graph is assigned by noisyG, the performance is very close to Kernel-SVM. If no working graph is incorporated, PE doesn’t inclease a lot comparing to the case when true graph is assigned. This observation may indicate that even without the graph guidance, our algorithm still work well. If the prior knowledge is not certain, we prefer not use it. We also note that when the dimension p increases, both FSTP and FSFP decrease because the ratio of the relevant features becomes small comparing to the total number of dimensions.

When p = 500, since our training sample size is n = 200, this is the n < p case. We see that all the methods generate a relatively high PE and low MCC. The performance of L₁-SVM, L₂-SVM, KBSVM and Kernel-SVM are similar, give the PE around 45%, while PE of our methods (η = 0, G* = partialG and G* = G) is below 40%. In general, our method gives the smallest PE, the greatest MCC, and high FSTP. Even when G* is assigned by noisyG, the performance of our method doesn’t deteriorate too much.

In addition, we simulate a new dataset which has the independent covariance structure $\left(\mathbf{\text{x}}~\mathcal{N}\left(0,{\mathit{I}}_{p\times p}\right)\right)$. In the graphical view, all the predictors are isolated without connections. Therefore, we only need specify two settings to test for our model: η = 0 and $\mathcal{G}*=\mathit{\text{ noisy}}\mathcal{G}$ and one setting η = 0 for KBSVM. The results are summarized in Table II. When p = 20, Kernel-SVM performs the best in terms of PE, our method (η = 0) achieves a similar PE but has the highest MCC. When p = 100 and p = 500, we observe that our method (η = 0) performs the best among the other settings as well as other existing methods.

In the simulation section, we consider two datasets with different covariance structures (i.e correlated or independent). We have found that the existing linear SVM methods (i.e. L₁-SVM, L₂-SVM, KBSVM) don’t perform well for complex data, as well as the non-linear kernel-SVM in high dimension settings. If the working graph is partially or fully correctly specified, our method outperforms all the other methods in terms of both prediction and selection accuracy. If the working graph is not available or noisy, the performance is comparable to the kernel-SVM in low dimension settings (i.e p = 20, 100), while perform better in the high dimension setting (p = 500).

V. Data Analysis

Glioblastoma is known as one of the most aggressive brain cancers, only 12% of the samples were censored and it is also related to other cancer development. We obtained a glioblastoma data set from the Cancer Genome Atlas Network [27]. This data set includes survival times (T) and the gene expression levels of p = 12, 999 genes (X) for 303 glioblastoma patients. To perform the classification, a new indicator variable Z is defined to denote the one year survival outcome:

$$Y=\{\begin{array}{ll}1,\hfill & T<365,\Delta =0,\hfill \\ 0,\hfill & T>365,\hfill \end{array}$$

where Δ represents censoring. Those subjects with T < 365, Δ = 1 are removed so the total number of subjects is 286 with P(Y = 1) = 45%, P(Y = 0) = 55%. In this section, we apply our methods and other existing methods to classify the survival status of the glioblastoma patients.

First, we fit an univariate logistic regression model for each gene x:

$$\mathit{\text{log}}\frac{p(x)}{1-p(x)}={\beta }_0+{\beta }_{1}x$$ (16)

We calculate the p value from the logistic regression and create a list of ascend ordered p-value as well as the corresponding genes. Second, we use the gene-ranking methods to select important genes, for example, we select the top 500 genes in the list and generate the new feature inputs from the gene expression levels.

Second, we create a graph G for all the 12, 999 genes from the Kyoto Encyclopedia of Genes and Genomes (KEGG) database, and we use an algorithm to retrieve the connections within the top 500 genes, and then map them to the working graph G*.

We specify two settings (η = 0 and G* = G) for both our model and KBSVM (Sun et al. [12]) to compare with other methods. The optimal tuning parameters for each methods are chosen by the minimum 5-fold cross-validation error. The average cross-validation error and the number of selected genes are summarized in Table III. Note that L₂-SVM doesn’t perform feature selection, so all the 500 genes are selected. L₁-SVM selects most of the 500 genes and has a similar performance to L₂-SVM. Kernel-SVM achieves a lower cross-validation error, which may suggest that the non-linear classifier is more plausible. KBSVM with graph incorporated (G* = G) produces a lower cross-validation error comparing to the no graph guided one, which may indicate that the prior graph improves the prediction accuracy. For our method BNSVM, if the graph is not incorporated (η = 0), fewer genes are detected, but still provides the satisfactory cross-validation error; if the graph is incorporated (G* = G), more genes are detected and achieves the smallest cross-validation error.

Table III.

Results of the analysis of TCGA data. n = 286, p = 500.

	CV error (%)	# selected genes
L1-SVM	29.55	468
L2-SVM	31.27	500
KBSVM, η = 0	26.94	233
KBSVM, G* = G	25.87	439
Kernel-SVM	25.86	500
BNSVM, η = 0	25.00	67
BNSVM, G* = G	21.43	232

To validate the selected genes by our method, a pathway enrichment analysis is conducted via ToppGene Suite (Chen et al. [28]). If no graph information is provided (η = 0), only 67 genes including several key genes such as PICK1 and IL22 that are members of glioma-related pathways are selected and no significant pathways are identified. If the graph information is provided (G* = G), our BNSVM method encourages the selection of the connected genes. As a result, a number of pathways are enriched in the set of selected genes, such as extracellular matrix organization (4.18×10⁻²), protein processing in endoplasmic reticulum (3.29×10⁻⁴), and unfolded protein response (9.34 × 10⁻²), where the numbers in the parentheses are the Bonferroni-adjusted p values. The pathways detected have been found to be related to cancer cell development and survival (Pointer et al. [29], Koh et al. [30], Clarke et al. [31], Hiramatsu et al. [32]. Moreover, the most highly enriched diseases are malignant neoplasm of ovary, glioblastoma, and malignant tumor of colon. These results further confirm that the integration of existing biological knowledge yields biologically more meaningful results.

VI. Discussion

This article introduces a knowledge-guided Bayesian non-linear SVM approach that uses the structural information between features to guide feature selection and is more robust than the existing linear SVM methods. Our numerical studies demonstrate that the proposed method outperforms several existing methods including the knowledge-guided Bayesian linear SVM and the standard nonlinear SVM in terms of prediction and feature selection. In the analysis of the real gene expression data, our results suggest that the integration of prior biological knowledge into our model leads to an increased ability to identify important genes, and yields biologically meaningful results and improved prediction accuracy.

APPENDIX

A. The derivation of (11)

Consider a new testing data vector x*, the corresponding f* is the value of non-linear function f(x) evaluated at x*. The joint joint prior distribution of the training outputs f, and the test outputs f* according to the prior is

$$\left[\begin{array}{c}f\\ f^{*}\end{array}\right]~\mathcal{N}\left(0,\left[\begin{array}{cc}K& k^{*}\\ k^{*T}& k^{*}\end{array}\right]\right)$$ (17)

The the conditional distribution of f* given f follows

$$f^{*}|x^{*},X,\theta ,f~\mathcal{N}\left(k^{*T}{K}^{-1}f,k^{*}-k^{*T}{K}^{-1}{k}^{*}\right).$$ (18)

Since the conditional distribution for f is given by (10), we have

$$f^{*}|x^{*},X,y,\theta ~\mathcal{N}\left(k^{*T}\Sigma D_{\rho }z,k^{*}-k^{*T}\Sigma k^{*}\right).$$ (19)

Hence, (11) holds.

B. The derivation of (12)

Note that the first equality in (12) implies that π(y* = 1|x*, X, y, θ) = P(u < f*) where $u~\mathcal{N}(0,1)$, f* follows (19), and u and f* are independent. Since

$$f^{*}-u~\mathcal{N}\left(k^{*T}\Sigma D_{\rho }z,1+k^{*}-k^{*T}\Sigma k^{*}\right),$$

the second equality in (12) follows.