Functional support vector machine

Abstract

Linear and generalized linear scalar-on-function modeling have been commonly used to understand the relationship between a scalar response variable (e.g. continuous, binary outcomes) and functional predictors. Such techniques are sensitive to model misspecification when the relationship between the response variable and the functional predictors is complex. On the other hand, support vector machines (SVMs) are among the most robust prediction models but do not take account of the high correlations between repeated measurements and cannot be used for irregular data. In this work, we propose a novel method to integrate functional principal component analysis with SVM techniques for classification and regression to account for the continuous nature of functional data and the nonlinear relationship between the scalar response variable and the functional predictors. We demonstrate the performance of our method through extensive simulation experiments and two real data applications: the classification of alcoholics using electroencephalography signals and the prediction of glucobrassicin concentration using near-infrared reflectance spectroscopy. Our methods especially have more advantages when the measurement errors in functional predictors are relatively large.

1 Introduction

Functional data analysis (FDA) has been widely used to analyze data that are measured over time or some other continuum, such as electroencephalography (EEG) signals that reflect the brain’s spontaneous electrical activity over a period of time. FDA exploits the smoothness of the processes by considering the data as being continuously defined functions. One instance of FDA is scalar-on-function modeling (Reiss et al. 2017) which is commonly used to understand the relationship between a continuous scalar response variable and one or more functional predictors. When the response variable is binary, functional logistic regression may be used. Methods for fitting such models typically assume a linear relationship between the response variable and the functional predictors. Naturally, though, such techniques are sensitive to model misspecification when the relationship between the response variable and the functional predictors is nonlinear.

Support vector machines (SVMs; e.g. Hastie et al. 2009, Chapter 12) are among the most robust prediction models, and they work by constructing hyperplanes for classification and regression analysis. When the original input feature space is not linearly separable, an SVM will map the input features into a higher-dimensional space using a kernel trick. In the higher-dimensional space, the transformed features become linearly separable and that results in nonlinear boundaries in the original feature space. A straightforward approach to applying SVMs to functional data is simply to regard each discrete measurement as an input feature. However, such an approach does not take into account either the high correlation between repeated measurements or the continuous nature of the functional data. In addition, the standard SVM cannot be used when the observed measurements are irregular, ie in any situation in which functional observations are not observed completely on a common grid.

In this article, our goal is to exploit SVM techniques for classification and regression while accounting for the particular characteristics of functional data. Functional principal component analysis (FPCA; Besse and Ramsay 1986; Ramsay and Dalzell 1991) extracts the features that characterize the functional data and that “explain” most of the variance. These features are termed functional principal components (FPCs), and functional data can typically be well represented by the first few FPCs (Rice and Silverman 1991). FPCA generally employs regularization to account both for the smoothness of the functional data and for the measurement error (Rice and Silverman 1991; Ramsay and Silverman 2005; Besse and Ramsay 1986; Xiao et al. 2016). Wu and Liu (2013) proposed an approach to the classification of functional data that is based on applying SVM to FPC scores. This is similar in spirit to what we are proposing here, but their approach applies only to classification problems and is based on a model that requires each class to have a unique functional mean, which is similar to functional linear discriminant analysis (James and Hastie 2001). Our method also differs from kernel PCA (Schölkopf et al. 1997) which first applies a kernel function to project the original data into a higher-dimensional feature space, and then perform PCA on the kernel-induced feature space to extract the principal components. In contrast, our methods leverage the continuous nature of the functional data by first applying FPCA to extract functional PCs that capture the underlying variability in the functional data and then exploiting a kernel-based approach (specifically, SVM) to predict the outcome using these FPC scores as features. Thus, our approach combines the benefits of FPCA for functional data analysis and the power of kernel methods for classification or prediction tasks.

Here, we propose and validate a novel approach that extends the earlier work by Wu and Liu (2013) in two directions. Firstly, we allow both classification and regression problems, and secondly, we make no assumptions about the nature of the functional data. (The setting of Wu and Liu (2013) is one of five scenarios considered in Section 3.) We refer to our method as functional support vector machine (FSVM). We conduct simulation studies to explore the performance of FSVM and compare it with more traditional functional regression approaches and with classical SVM. We apply the methods to the EEG data to evaluate the classification performance and to near-infrared reflectance spectroscopy (NIRS) data to evaluate the regression performance. We conclude the paper with some discussion and remarks on potential future directions.

2 Methodology

In this section, we describe the approach, we take to address the problem of using functional data observations to predict scalar outcomes. We will do this by taking a two-step procedure. In the first step, we employ FPCA to extract FPC scores. In the second step, we utilize these FPC scores as features within a regular SVM algorithm.

Let $X\in L^2(\mathcal{I})$ be a random function, where $L^2(\mathcal{I})$ is a bounded subspace of the separable Hilbert space, $\mathcal{I}$ is a compact interval and ${\int }_{\mathcal{I}}E(X{(t)}^2)<\infty$. In practice, we typically have observations of such functions at a finite number of points, $t_1<\dots <t_J\in \mathcal{I}$ that are contaminated with error. Let ${\mathit{x}}_i={(x_{i1},\dots ,x_{iJ})}^T$ denote the observed data of the ith sample, where $x_{ij}=X_i(t_j)+{\eta }_{ij}$, and the ${\eta }_{ij}$’s, $j=1,\dots ,J$ are independent and identically distributed (iid) and independent of the $X_i(t_j)$’s. In addition, we observe univariate “response” variables, either class labels 1 and –1 (in a binary classification problem) or a continuous scalar response (in a regression problem). Our training set is $T=\left\{({\mathit{x}}_1,y_1),\dots ,({\mathit{x}}_n,y_n)\right\}$.

Let $\mu (t)=E(X(t))$ be the mean function and $\Sigma (s,t)=\text{Cov}(X(s),X(t))$ be the covariance function. By Mercer’s theorem, $\Sigma (s,t)=\sum _{k=1}^{\infty }{\lambda }_k{\varphi }_k(s){\varphi }_k(t)$, where ${\lambda }_1\ge {\lambda }_2\ge \dots$ are the eigenvalues and ${\varphi }_k$’s are the corresponding orthonormal eigenfunctions. Then ${\left\{{\varphi }_k\right\}}_{k=1}^{\infty }$ form a complete and orthonormal system of $L^2(\mathcal{I})$.

$X_i(t)$ can be represented by $\mu (t)$ and the FPCA expansion (Karhunen 1950)

$$\begin{array}{c}{X}_i(t)=\mu (t)+\sum _{k=1}^{\infty }{A}_{ik}{\varphi }_k(t),\hfill \end{array}$$ (2.1)

where $A_{ik}={\int }_{\mathcal{I}}(X_i(t)-\mu (t)){\varphi }_k(t)dt$ are the FPC scores of $X_i(t)$. The A  ${}_{ik}$ ’s are random variables with $E(A_{ik})=0,\hspace{0.17em}\mathit{var}(A_{ik})={\lambda }_k$ and are uncorrelated across k.

We can estimate smooth FPCs (or eigenfunctions) ${\varphi }_k$’s and the corresponding FPC scores A  ${}_{ik}$ ’s using any FPCA approach (e.g. Ramsay and Silverman 2005; Yao et al. 2005; Xiao et al. 2016).

If we can choose K such that the first K FPCs account for most of the variation in $\{X_1,\dots ,X_n\}$ and thus a good approximation can be realized from only the first K FPCs $X_i(t)\approx \mu (t)+\sum _{k=1}^K{A}_{ik}{\varphi }_k(t)$, then it is natural to consider using only the first K FPC scores ${\mathit{A}}_i={(A_{i1},A_{i2},\dots ,A_{iK})}^T$ to predict y  ${}_i$  . We will use these scores as input features for classification and regression analysis through SVM techniques.

2.1 Binary outcomes

We consider a binary classification problem using ${\widehat{\mathit{A}}}_i$ as input features in an attempt to minimize

$$\begin{array}{c}\sum _{i=1}^{n}I(y_i\ne \mathit{sign}(f({\widehat{\mathit{A}}}_i))\hfill \end{array}$$ (2.2)

over f, where f is some decision function. Suppose $\widehat{f}$ is the minimizer of (2.2), then ${\widehat{y}}_i=\mathit{sign}(\widehat{f}({\widehat{\mathit{A}}}_i))$. Stated differently, the two classes can be separated by a decision boundary $\widehat{f}(\widehat{\mathit{A}})=0$. Equation (2.2) is equivalent to the misclassification error $\sum _{i=1}^{n}I(y_i\ne {\widehat{y}}_i)$. Note that $y_i\widehat{f}({\widehat{\mathit{A}}}_i)>0$ when $y_i={\widehat{y}}_i$ and $y_i\widehat{f}({\widehat{\mathit{A}}}_i)<0$ when $y_i\ne {\widehat{y}}_i$. To solve the classification problem (2.2), we exploit SVM techniques, which we outline here.

An alternative to using the non-convex $0-1$ loss function in (2.2), would be to consider instead the hinge loss function and solve an optimization problem that also includes a penalty to control the smoothness of f  

$$\begin{array}{c}\underset{f}{\min }\left\{\sum _{i=1}^n{\left[1-y_{i}f({\widehat{\mathit{A}}}_i)\right]}_{+}+\frac{\lambda }{2}\left|\right|f|{|}^2\right\},\hfill \end{array}$$ (2.3)

where ${\left[x\right]}_{+}=\max (x,0)$ and $\lambda$ is a tuning parameter. $0\le {\left[1-y_i\widehat{f}({\widehat{\mathit{A}}}_i)\right]}_{+}<1$ when $y_i\widehat{f}({\widehat{\mathit{A}}}_i)>0$, and ${\left[1-y_i\widehat{f}({\widehat{\mathit{A}}}_i)\right]}_{+}>1$ when $y_i\widehat{f}({\widehat{\mathit{A}}}_i)<0$. Therefore, the hinge loss is large if y  ${}_i$ is misclassified and a penalty is imposed to control the smoothness of f. If the input feature space (ie space of $\widehat{\mathit{A}}$: ${\mathcal{R}}^K$) is linearly separable, then we can find a linear decision function $f(\widehat{\mathit{A}})={\beta }_0+{\widehat{\mathit{A}}}^T\mathit{\beta }$ and solve the following optimization problem:

$$\begin{array}{c}\underset{{\beta }_0,\mathit{\beta }}{\min }\left\{\sum _{i=1}^n{\left[1-y_i({\beta }_0+{\widehat{\mathit{A}}}_i^T\mathit{\beta })\right]}_{+}+\frac{\lambda }{2}\left|\right|\mathit{\beta }|{|}^2\right\},\hfill \end{array}$$ (2.4)

To increase model flexibility, we consider $f(\widehat{\mathit{A}})$ in the form $f(\widehat{\mathit{A}})={\omega }_0+\sum _{i=1}^n{\omega }_{i}k(\widehat{\mathit{A}},{\widehat{\mathit{A}}}_i)$, where $k:{\mathcal{R}}^K\times {\mathcal{R}}^K\to \mathcal{R}$ is a symmetric positive-definite kernel function, the corresponding space of functions $\mathcal{H}$ is a reproducing kernel Hilbert space (RKHS), and $\left|\right|f|{|}^2=\langle f,f\rangle ={\mathit{\omega }}^T\mathit{K\omega }$ (Berlinet and Thomas-Agnan 2011). Thus, solving the optimization problem (2.3) is equivalent to solving

$$\begin{array}{c}\underset{{\omega }_0,\mathit{\omega }}{\min }\left\{\sum _{i=1}^n{\left[1-y_{i}f({\widehat{\mathit{A}}}_i)\right]}_{+}+\frac{\lambda }{2}{\mathit{\omega }}^T\mathit{K\omega }\right\},\hfill \end{array}$$ (2.5)

where $\mathit{K}$ is a $n\times n$ matrix of $k({\widehat{\mathit{A}}}_i,{\widehat{\mathit{A}}}_l)$ for all pairs (i, l) and $\mathit{\omega }={({\omega }_1,\dots ,{\omega }_n)}^T$. Two popular choices for $k(·,·)$ are the linear kernel $k({\widehat{\mathit{A}}}_i,{\widehat{\mathit{A}}}_l)=1+{\widehat{\mathit{A}}}_i^T{\widehat{\mathit{A}}}_l$ which is particularly appropriate when the feature space is linearly separable; and the Gaussian kernel $k({\widehat{\mathit{A}}}_i,{\widehat{\mathit{A}}}_l)=\text{exp}(-\gamma ||{\widehat{\mathit{A}}}_i-{\widehat{\mathit{A}}}_l|{|}^2)$, where $\gamma$ is a bandwidth parameter, which has demonstrated superior performance in practice when the feature space is not linearly separable. When $k(·,·)$ is a linear kernel, (2.5) is equivalent to (2.4) and has the relationship $\mathit{\beta }=\sum _{i=1}^n{\omega }_i{\widehat{\mathit{A}}}_i,\hspace{0.17em}{\beta }_0={\omega }_0+\sum _{i=1}^n{\omega }_i$. If $k(·,·)$ is a Gaussian kernel, then $f(\widehat{\mathit{A}})$ is a nonlinear decision function. The “kernel trick” involves the first mapping $\widehat{\mathit{A}}$ to a higher-dimensional RKHS by means of a transforming function h with $k({\widehat{\mathit{A}}}_i,{\widehat{\mathit{A}}}_l)=\langle h({\widehat{\mathit{A}}}_i),h({\widehat{\mathit{A}}}_l)\rangle$, where $\langle ·,·\rangle$ is an inner product. Next, the trick involves finding a linear decision function in the higher-dimensional space. The resulting decision function is typically nonlinear in the original space. Since the optimization problem (2.5) involves h only through the inner products, we only require knowledge of the kernel function $k(·,·)$ but not $h(·)$.

An equivalent version (e.g. Hastie et al. 2009, Chapter 12) of the optimization problem (2.5) is

$$\begin{array}{c}\underset{\mathit{\alpha }}{\max }\left\{\sum _{i=1}^n{\alpha }_i-\frac{1}{2}\sum _{i=1}^n\sum _{l=1}^n{\alpha }_i{\alpha }_l{y}_i{y}_{l}k({\widehat{\mathit{A}}}_i,{\widehat{\mathit{A}}}_l)\right\}\hfill \end{array}$$ (2.6)

subject to the constraints

$$\begin{array}{c}0\le {\alpha }_i\le C,i=1,\dots ,n,\hfill \\ \sum _{i=1}^n{\alpha }_i{y}_i=0,\hfill \end{array}$$

where ${\alpha }_i$ are Lagrange multipliers and C is a regularization parameter that controls the tradeoff between the misclassification error and decision boundary complexity. Smaller C leads to a smoother decision boundary while larger C will allow a wiggly boundary. The solution of (2.6) is $\widehat{f}(\widehat{\mathit{A}})={\widehat{\omega }}_0+\sum _{i=1}^n{\widehat{\alpha }}_i{y}_{i}k(\widehat{\mathit{A}},{\widehat{\mathit{A}}}_i)$, where ${\widehat{\omega }}_0$ is computed as ${\widehat{\omega }}_0=y_i-\sum _{j=1}^n{\widehat{\alpha }}_j{y}_{j}k({\widehat{\mathit{A}}}_i,{\widehat{\mathit{A}}}_j)$ for any ${\widehat{\mathit{A}}}_i$ for which $0<{\widehat{\alpha }}_i<C$ and ${\widehat{y}}_i=\mathit{sign}(\widehat{f}({\widehat{\mathit{A}}}_i))$. We call the resulting method the functional support vector classifier (FSVC).

Note that in contrast to our method that has a unique decision function $f(\mathit{A})$, the proposal of Wu and Liu (2013) involved a separate decision function f  ${}_s$ for each class with parameter functions ${\beta }_s$, where s is the class index. They first estimated FPC scores to reconstruct $X_i(t)$, ie ${\widehat{X}}_i(t)=\widehat{\mu }(t)+\sum _{k=1}^K{\widehat{A}}_{ik}{\widehat{\varphi }}_k(t)$, then estimated the class label ${\widehat{y}}_i=\mathit{argma}{x}_s{f}_s({\widehat{X}}_i(·))$. Their optimization problem for estimating the parameters ${\beta }_s$ in f  ${}_s$ is similar to Equations (2.4) and (2.5), but it involves a more complex truncated-hinge loss function in contrast to the hinge loss used in our methods. This difference results in a computationally expensive nonconvex optimization problem that requires the use of the difference convex algorithm (An and Tao 1997).

2.2 Continuous outcomes

In a scalar-on-function regression with a continuous scalar response, the response y  ${}_i$ is modeled as

$$\begin{array}{c}{y}_i={\beta }_0+m(X_i)+{\epsilon }_i,\hfill \end{array}$$ (2.7)

where $m:L^2(\mathcal{I})\to \mathcal{R}$, ${\beta }_0$ is a scalar intercept, ${\epsilon }_i$ denotes error and ${\epsilon }_i$’s are iid and independent of the X  ${}_i$ ’s.

If the relationship between the X  ${}_i$ ’s and the y  ${}_i$ ’s were known to be linear, then we would consider predicting y  ${}_i$ by $f({\widehat{\mathit{A}}}_i)={\beta }_0+{\widehat{\mathit{A}}}_i^T\mathit{\beta }$. More generally, we predict y  ${}_i$ by $f({\widehat{\mathit{A}}}_i)={\omega }_0+\sum _{j=1}^n{\omega }_{j}k({\widehat{\mathit{A}}}_i,{\widehat{\mathit{A}}}_j)$ to allow for a nonlinear relationship and so we must solve an optimization problem

$$\begin{array}{c}\underset{{\omega }_0,\mathit{\omega }}{\min }\left\{\sum _{i=1}^{n}L\left(y_i-f({\widehat{\mathit{A}}}_i)\right)+\frac{\lambda }{2}{\mathit{\omega }}^T\mathit{K\omega }\right\}\hfill \end{array}$$ (2.8)

for some general loss function $L(·)$. Here, we use the $ϵ$-insensitive loss function which is similar to hinge loss for classification and generally used in support vector regression (SVR):

$$\begin{array}{c}{L}_{ϵ}(r)=\{\begin{array}{cc}0\hfill & \text{if} \left|r\right|<ϵ,\hfill \\ \left|r\right|-ϵ,\hfill & \text{otherwise}.\hfill \end{array}\hfill \end{array}$$ (2.9)

and solve (2.8) by $\nu$-SVR algorithm (Schölkopf et al. 2000). Equivalently, we could solve the quadratic programming problem

$$\begin{array}{c}\underset{{\alpha }_i,{\alpha }_i^{*}}{\min }\left\{\frac{1}{2}\sum _{i=1}^n\sum _{l=1}^n({\alpha }_i^{*}-{\alpha }_i)({\alpha }_l^{*}-{\alpha }_l)k({\widehat{\mathit{A}}}_i,{\widehat{\mathit{A}}}_l)-\sum _{i=1}^n{y}_i({\alpha }_i^{*}-{\alpha }_i)\right\}\hfill \end{array}$$ (2.10)

subject to

$$\begin{array}{c}0\le {\alpha }_i,{\alpha }_i^{*}\le \frac{C}{n},i=1,\dots ,n,\hfill \end{array}$$

 

$$\begin{array}{c}\sum _{i=1}^n({\alpha }_i^{*}-{\alpha }_i)=0,\sum _{i=1}^n({\alpha }_i^{*}+{\alpha }_i)\le C\nu ,\hfill \end{array}$$

where ${\alpha }_i$ and ${\alpha }_i^{*}$ are Lagrange multipliers, C is a regularization parameter that controls the complexity of decision function, and $\nu$ is a tuning parameter with $0\le \nu \le 1$ that controls the fraction of errors (Schölkopf et al. 2000). The regression estimate then takes the form $\widehat{f}(\widehat{\mathit{A}})={\widehat{\omega }}_0+\sum _{i=1}^n({\widehat{\alpha }}_i^{*}-{\widehat{\alpha }}_i)k(\widehat{\mathit{A}},{\widehat{\mathit{A}}}_i)$. We call the resulting method Functional Support Vector Regression (FSVR). We refer to both techniques as instances of Functional Support Vector Machine (FSVM).

As in many classification and regression problems, with FSVM it is necessary to select several tuning parameters. We have two tuning parameters that are required to get the input for SVM: a smoothing parameter that controls the smoothness of the covariance matrix and the number of FPCs K. If a linear kernel is used in SVM, a regularization parameter C is inherent in SVM for binary outcomes while a regularization parameter C and a tuning parameter $\nu$ are inherent in SVM for continuous outcomes. If a Gaussian kernel is used in SVM, then the bandwidth parameter $\gamma$ is an additional tuning parameter. We simultaneously select the tuning parameters in the FPCA and SVM steps to control the performance. We select the optimal combinations of tuning parameters involved in the two steps that minimize the misclassification error $\sum I(y_i\ne \widehat{f}({\widehat{\mathit{A}}}_i))$ for binary outcomes and mean-squared error for continuous outcomes. We will discuss the details of choosing tuning parameters in Sections 3 and 4.

The above procedures are summarized into Algorithm 1 for binary outcome and Algorithm 2 for continuous outcome.

Algorithm 1:

Functional support vector classification (FSVC)

1. Perform FPCA on observed data x  ${}_{ij}$ to estimate FPC scores A  ${}_{ik}$ ’s and choose the first K FPC scores ${\widehat{\mathit{A}}}_i={({\widehat{A}}_{i1},{\widehat{A}}_{i2},\dots ,{\widehat{A}}_{iK})}^T$.

2. Solve a classification problem using ${\widehat{\mathit{A}}}_i={({\widehat{A}}_{i1},{\widehat{A}}_{i2},\dots ,{\widehat{A}}_{iK})}^T$ as input features by SVM techniques:

   1. Solve the convex quadratic programming problem (2.5) to obtain ${\widehat{\alpha }}_i$.

   2. Given ${\widehat{\alpha }}_i$’s, $\widehat{f}(\widehat{\mathit{A}})={\widehat{\omega }}_0+\sum _{i=1}^n{\widehat{\alpha }}_i{y}_{i}k(\widehat{\mathit{A}},{\widehat{\mathit{A}}}_i)$, where ${\widehat{\omega }}_0$ is computed as ${\widehat{\omega }}_0=y_i-\sum _{j=1}^n{\widehat{\alpha }}_j{y}_{j}k({\widehat{\mathit{A}}}_i,{\widehat{\mathit{A}}}_j)$ for any ${\widehat{\mathit{A}}}_i$ for which $0<{\widehat{\alpha }}_i<C$. Thus, obtain ${\widehat{y}}_i=\text{sign}(\widehat{f}({\widehat{\mathit{A}}}_i))$.

3. Simultaneously select optimal tuning parameters in steps 1 and 2 by cross-validated misclassification errors $\sum I(y_i\ne \text{sign}(\widehat{f}({\widehat{\mathit{A}}}_i))$: the smoothing parameter that controls the smoothness of covariance matrix, the number of FPCs K, the regularization parameter C that controls the complexity of decision boundary, and the bandwidth parameter $\gamma$ if Gaussian kernel is used.

Algorithm 2:

Functional support vector regression (FSVR)

1. Perform FPCA on observed data x  ${}_{ij}$ to estimate FPC scores A  ${}_{ik}$ ’s and choose the first K FPC scores ${\widehat{\mathit{A}}}_i={({\widehat{A}}_{i1},{\widehat{A}}_{i2},\dots ,{\widehat{A}}_{iK})}^T$.

2. Solve a regression problem using ${\widehat{\mathit{A}}}_i={({\widehat{A}}_{i1},{\widehat{A}}_{i2},\dots ,{\widehat{A}}_{iK})}^T$ as input features by SVM techniques:

   1. Solve the convex quadratic programming problem (2.10) to obtain ${\widehat{\alpha }}_i$ and ${\widehat{\alpha }}_i^{*}$.

   2. Given ${\widehat{\alpha }}_i,{\widehat{\alpha }}_i^{*},\hspace{0.17em}\widehat{f}(\widehat{\mathit{A}})={\widehat{\omega }}_0+\sum _{i=1}^n({\widehat{\alpha }}_i^{*}-{\widehat{\alpha }}_i)k(\widehat{\mathit{A}},{\widehat{\mathit{A}}}_i)$, where ${\widehat{\omega }}_0$ is computed as ${\widehat{\omega }}_0=y_i-\sum _{j=1}^n({\widehat{\alpha }}_j^{*}-{\widehat{\alpha }}_j)k({\widehat{\mathit{A}}}_i,{\widehat{\mathit{A}}}_j)$ for any ${\widehat{\mathit{A}}}_i$ for which $0<{\widehat{\alpha }}_i,{\widehat{\alpha }}_i^{*}<C$. Thus, obtain ${\widehat{y}}_i=\widehat{f}({\widehat{\mathit{A}}}_i)$.

3. Simultaneously select optimal tuning parameters in steps 1 and 2 by cross-validated mean squared error $\sum (y_i-\widehat{f}({\widehat{\mathit{A}}}_i){)}^2$: the smoothing parameter that controls the smoothness of covariance matrix, the number of FPCs K, the regularization parameter C that controls the complexity of decision function, the tuning parameter $\nu$ that controls the fraction of errors, and the bandwidth parameter $\gamma$ if Gaussian kernel is used.

2.3 Connection with linear scalar-on-function regression

When a linear kernel is chosen, the FSVM method is similar to scalar-on-function regression that is based on a principal component basis (Reiss et al. 2017). Note that FSVM using a linear kernel predicts y  ${}_i$ by

$$\begin{array}{cc}f(\mathit{A})={\beta }_0+{\mathit{A}}^T\mathit{\beta }\hfill & \hfill \\ ={\beta }_0+\sum _{k=1}^K{\beta }_k{\int }_{\mathcal{I}}(X(t)-\mu (t)){\varphi }_k(t)dt\hfill & \hfill \\ ={\beta }_0+{\int }_{\mathcal{I}}(X(t)-\mu (t))\left(\sum _{k=1}^K{\beta }_k{\varphi }_k(t)\right)dt\hfill & \hfill \\ ={\beta }_0+{\int }_{\mathcal{I}}(X(t)-\mu (t))\beta (t)dt\hfill & \hfill \\ ={\beta }_0+{\mathit{A}}^T\left({\int }_{\mathcal{I}}\varphi (t){\varphi }^T(t)dt\right)\mathit{\beta },\hfill & \hfill \end{array}$$

where $\beta (t)=\sum _{k=1}^K{\beta }_k{\varphi }_k(t)$. The functional linear model (Ramsay and Silverman 2005) with a principal component basis minimizes

$$\begin{array}{c}\sum _{i=1}^n{\left\{y_i-{\beta }_0-{\mathit{A}}_i^T\left({\int }_{\mathcal{I}}\varphi (t){\varphi }^T(t)dt\right)\mathit{\beta }\right\}}^2,\hfill \end{array}$$

over ${\beta }_0$ and $\mathit{\beta }$, whereas we use the $ϵ$-insensitive loss as in (2.9).

For the functional classification problem, our method directly estimates the classifier $\mathit{sign}(f(\mathit{A}))$, while a functional logistic regression with principal component basis models

$$\begin{array}{c}\hspace{0.17em}\text{log}\hspace{0.17em}\left(\frac{\pi }{1-\pi }\right)={\beta }_0+{\mathit{A}}^T\left({\int }_{\mathcal{I}}\varphi (t){\varphi }^T(t)dt\right)\mathit{\beta },\hfill \end{array}$$

where $\pi =P(y=1|X(t))$.

3 Simulation Studies

We evaluated our methods through several simulation scenarios for both binary and continuous outcomes with varying sample sizes and numbers of observation time points for the training data set. For each simulated dataset, we compared our methods with the following alternative methods on a validation set of sample size 10,000: (1) scalar-on-function regression (functional logistic regression for binary outcomes and functional linear regression for continuous outcomes); (2) classical SVM using each measurement as an input feature, using the linear kernel; (3) classical SVM using each measurement as an input feature, using the Gaussian kernel; and (4) functional linear discriminant analysis for binary outcomes.

3.1 Simulation design

We have two main situations—binary y and continuous y. For each of them, the following scenarios were considered: (1) Scenario 1 (Settings 1–3), y was generated from a function of FPC scores with error ${\epsilon }_i$, including both linear and nonlinear cases; (2) Scenario 2 (Setting 4), y was generated from scalar-on-function regression models (ie functional logistic regression for binary y and functional linear regression for continuous y); (3) Scenario 3 (Setting 5), we generated two groups with their mean curve functions ${\mu }_1(·)$ and ${\mu }_2(·)$, respectively. When y was binary, $y_i=1$ for the sample in group 1 and $y_i=-1$ for the sample in group 2. When y was continuous, $y_i=\pm 1+{\epsilon }_i$; (4) Scenario 4 (Setting 6) we generated two groups with the same mean curve functions but different covariance functions. The generation of y is the same as Scenario 3. We also considered Scenario 5 (Setting 7) in which data were generated from a functional linear discriminant model for a binary outcome.

In Scenario 1 (Settings 1–3), we generated

$$\begin{array}{c}{X}_i(t)=\sum _{k=1}^5\sqrt{{\lambda }_k}{Z}_{ik}{\varphi }_k(t),\hfill \end{array}$$

where $Z_{i1},\dots ,Z_{i5}$ were iid N(0, 1), ${\varphi }_k(t)$’s were eigenfunctions generated from Legendre polynomials and then transformed onto [0, 1] and scaled by requiring $\int {\varphi }_k^2(t)dt=1$ (Figure S1 of the Supplementary Materials), and $({\lambda }_1,\dots ,{\lambda }_5)=(1,0.6,0.3,0.1,0.05)$. The FPC scores were $A_{ik}=\sqrt{{\lambda }_k}{Z}_{ik},\hspace{0.17em}k=1,\dots ,5,$  $i=1,\dots ,n$.

The observed data were $x_{ij}=X_i(t_j)+{\eta }_{ij},$  $j=1,\dots ,J$, where t  ${}_j$ ’s were evenly spaced on [0, 1]. We set $y_i=1$ if ${\pi }_i>0.5,\hspace{0.17em}{y}_i=-1$ if ${\pi }_i\le 0.5$, where ${\pi }_i=g^{-1}(l_i({\mathit{A}}_i)+{\epsilon }_i),\hspace{0.17em}g(·)$ was a logit link for binary y  ${}_i$ , and set $y_i=l_i({\mathit{A}}_i)+{\epsilon }_i$ for continuous y  ${}_i$ , $l_i({\mathit{A}}_i)$ was generated according to the specifications in Figures 1 and r3, where only the first two FPC scores are useful for predicting y  ${}_i$ in the linear case of Setting 1, as well as for the nonlinear cases in Setting 2 (involving interactions of the two FPC scores), and Setting 3 (representing the quadratic case).

Figure 1:

Simulation performance for binary outcomes of Scenarios 1–2. Left column: ${\eta }_{ij}\sim N(0,1)$; right column: ${\eta }_{ij}\sim N(0,10)$.

In Scenario 2 (Setting 4), we set $X_i(t)=\sum _{k=1}^5{a}_{ik}{\varphi }_k(t)$, where ${\varphi }_k(t),k=1,\dots ,5$ were cubic B-spline basis functions on the interval $\mathcal{I}=[0,1]$. The basis coefficients a  ${}_{ik}$ ’s were generated independently from N(0, 1). The observed data $x_{ij}=X_i(t_j)+{\eta }_{ij},j=1,\dots ,J$, where t  ${}_j$ ’s were evenly spaced on [0, 1]. The parameter function was in terms of the same basis functions $\beta (t)=\sum _{k=1}^5{\beta }_k{\varphi }_k(t)$. We set $y_i=1$ if ${\pi }_i>0.5,\hspace{0.17em}{y}_i=-1$ if ${\pi }_i\le 0.5$, where ${\pi }_i=g^{-1}(l_i+{\epsilon }_i)$, g was logit link for binary y  ${}_i$ , and set $y_i=l_i+{\epsilon }_i$ for continuous y  ${}_i$ , $l_i={\beta }_0+\int X_i(t)\beta (t)dt={\beta }_0+{\mathit{a}}_i^T\psi \mathit{\beta },$ where $\psi$ was the matrix of inner products between the cubic B-splines, ${\mathit{a}}_i={(a_{i1},a_{i2},\dots ,a_{i5})}^T$ and $\mathit{\beta }={({\beta }_1,{\beta }_2,\dots ,{\beta }_5)}^T$. We set $\mathit{\beta }={(0.8,-1,0.5,1.5,0.5)}^T$ for each corresponding basis function and ${\beta }_0=0$ without loss of generality.

In Scenario 3 (Setting 5), we generated ${\mu }_1(t)=\text{cos}(10t-\pi /4)+0.5\hspace{0.17em}\text{sin}(8t-\pi /4)$ and ${\mu }_2(t)=\text{cos}(8t-\pi /4)+0.6\hspace{0.17em}\text{sin}(10t-\pi /4)$ (Figure S2 of the Supplementary Materials), where $t\in [0,1]$. We generated $x_{ij}={\mu }_1(t_j)+{\eta }_{ij}$ for sample in group 1 and $x_{ij}={\mu }_2(t_j)+{\eta }_{ij}$ for sample in group 2.

In Scenario 4 (Setting 6), we generated ${\mu }_1(t)={\mu }_2(t)=\mu (t)=\text{cos}(10t-\pi /4)+0.5\hspace{0.17em}\text{sin}(8t-\pi /4)$, where $t\in [0,1]$. We set $X_i(t)=\mu (t)+\sum _{k=1}^5\sqrt{{\lambda }_{1k}}{Z}_{ik}{\varphi }_k(t)$ for sample in group 1 and $X_i(t)=\mu (t)+\sum _{k=1}^5\sqrt{{\lambda }_{2k}}{Z}_{ik}{\varphi }_k(t)$ for sample in group 2 where $Z_{i1},\dots ,Z_{i5}$ were i.i.d. N(0, 1), ${\varphi }_k(t)$’s were eigenfunctions generated from Legendre polynomials and then transformed onto [0, 1] and scaled by requiring $\int {\varphi }_k^2(t)dt=1$, and $({\lambda }_{11},\dots ,{\lambda }_{15})=(1,0.6,0.3,0.1,0.05)$ and $({\lambda }_{21},\dots ,{\lambda }_{25})=2({\lambda }_{11},\dots ,{\lambda }_{15})$. The observed data were $x_{ij}=X_i(t_j)+{\eta }_{ij},j=1,\dots ,J$.

In Scenario 5 (Setting 7), we generated data from a functional linear discriminant model. We set $X_i(t)=\sum _{k=1}^2{a}_{ik}{\varphi }_k(t)$, where ${\varphi }_k(t),k=1,2$ were B-spline basis functions of order 2 on the interval $\mathcal{I}=[0,1]$. The basis coefficients a  ${}_{ik}$ ’s were generated independently from N(0, 1). We set $y_i=1$ if $\int X_i(t){\beta }_1(t)dt>\int X_i(t){\beta }_2(t)dt$ and $y_i=-1$ otherwise, where ${\beta }_1(t)=\text{cos}(2\pi /3){\varphi }_1(t)+\text{sin}(2\pi /3){\varphi }_2(t)$ for group 1 and ${\beta }_2(t)=\text{cos}(4\pi /3){\varphi }_1(t)+\text{sin}(4\pi /3){\varphi }_2(t)$ for group 2.

In all simulation settings, we generated ${\eta }_{ij}$ as iid from N(0, 1) or N(0, 10) to evaluate the performance under different levels of measurement errors in x  ${}_{ij}$  . For error in y  ${}_i$ , we generated ${\epsilon }_i$ as being iid from N(0, 1).

For our implementation of FSVM, we first used FACE (Xiao et al. 2016) to estimate FPCs. We fixed the number of knots to be 10 when J = 40 and to be 35 when J = 50 and J = 100 as suggested in Xiao et al. (2016). We searched smoothing parameter $\tau$ in the grid of $(0.5,1,5,10)$ and the number of FPCs K in the grid of $(1,2,\dots ,5)$. For the intrinsic tuning parameters in SVM, we searched C in the grid of $(0.01,0.2575,0.5050,0.7525,1)$ for both binary and continuous outcomes, and $\nu$ in the grid of $(0.1,0.3,0.5,0.8)$ for continuous outcomes. The optimal tuning parameters were determined by five-fold cross-validations. We minimized the cross-validated misclassification errors $\sum I(y_i\ne \mathit{sign}(\widehat{f}({\widehat{\mathit{A}}}_i))$ in FSVC and cross-validated mean squared errors $\sum (y_i-\widehat{f}({\widehat{\mathit{A}}}_i){)}^2$ in FSVR. When the Gaussian kernel was used, we determined the bandwidth parameter $\gamma$ in Gaussian kernel by the median heuristic (ie the median of $\left|\right|{\widehat{\mathit{A}}}_i-{\widehat{\mathit{A}}}_l|{|}^2$) to reduce computational time (Fukumizu et al. 2009; Caputo et al. 2002). To implement classical SVM, we set the searching grids of C and $\nu$ to be the same as for FSVM. To implement linear scalar-on-function regressions, we used penalized scalar-on-function regression with the thin plate regression spline basis as the default setting in R package refund (Goldsmith et al. 2011). For functional linear discriminant analysis, we used FACE to estimate FPCs and then considered the FPC scores as input features to perform standard linear discriminant analysis.

The three situations for the number of observed functions in the training data and number of observations per function were n = 50, J = 40; n = 50, J = 100; and n = 100, J = 50. We repeated the procedures 100 times for each instance.

3.2 Simulation results

3.2.1 Binary outcomes

Figure 1 shows the results of the simulation experiments for binary y. We provide boxplots of the classification accuracy for functional linear discriminant analysis, functional logistic regression, SVC using the linear kernel, SVC using the Gaussian kernel, FSVC using the linear kernel, and FSVC using the Gaussian kernel.

FSVC performed well relative to the other methods in all settings with varying sample size, number of time points, and measurement error levels of ${\eta }_{ij}$. For Setting 1 in which the decision function $l_i({\mathit{A}}_i)$ is a linear function (Fig. 1a), FSVC using the linear kernel performed the best among those considered and showed to be more advantageous when the measurement error in ${\mathit{x}}_i$ was larger. For Settings 2 and 3 in which l  ${}_i$ is a nonlinear function (Figures 1b and 1c), FSVC using the Gaussian kernel came out ahead of the others in terms of higher accuracy, while functional linear discriminant analysis performed the worst.

For Scenario 2 (Setting 4) in which y  ${}_i$ was generated from functional logistic regression models (Fig. 1d), as would be expected, functional logistic regression performed the best among others. Among the rest, functional discriminant analysis and FSVC using the linear kernel outperformed the others. For Scenario 3 (Setting 5) in which different groups had different mean functions (Fig. 2a), FSVC using the linear kernel came ahead of the others. For Scenario 4 (Setting 6) in which groups with the same mean functions but different covariance functions, FSVC using the Gaussian kernel came head of the others especially when noise was large (Fig. 2b). For Scenario 5 (Setting 7) in which data were generated from a functional linear discriminant model (Fig. 2c), as would be expected, functional linear discriminant analysis performed the best among those considered and the performance of FSVC using the linear kernel and functional logistic regression were similar when the measurement error was larger.

Figure 2:

Simulation performance for binary outcomes of Scenarios 3–5. Left column: ${\eta }_{ij}\sim N(0,1)$; right column: ${\eta }_{ij}\sim N(0,10)$.

3.2.2 Continuous outcomes

Figure 3 shows the results of the simulation experiments of continuous y. We provide boxplots of the root-mean-square error (RMSE) for functional linear regression, SVR using the linear kernel, SVR using the Gaussian kernel, FSVR using the linear kernel, and FSVR using the Gaussian kernel.

Figure 3:

Simulation performance for continuous outcomes of Scenarios 1–4. Left column: ${\eta }_{ij}\sim N(0,1)$; Right column: ${\eta }_{ij}\sim N(0,10)$. $†$: the RMSEs of SVR linear were large and exceeded the figure when $N=50,J=40$.

For Setting 1 in which $l_i({\mathit{A}}_i)$ was a linear function of ${\mathit{A}}_i$ (Fig. 3a), FSVR using the linear kernel performed the best among those considered. For Settings 2 and 3 in which $l_i({\mathit{A}}_i)$ was a nonlinear function of ${\mathit{A}}_i$ (Figures 3b and 3c), the performance of FSVR using the Gaussian kernel and SVR using the Gaussian kernel was similar and outperformed the rest when ${\eta }_{ij}\sim N(0,1)$. When ${\eta }_{ij}\sim N(0,10)$, FSVR using the Gaussian kernel came out ahead of the others. For Scenario 2 (Setting 4) in which y  ${}_i$ was generated from functional linear regressions (Fig. 3d), the performance of FSVR using the linear kernel was similar to functional linear regression. For Scenario 3 (Setting 5) in which different groups had different mean functions, FSVR using the linear kernel outperformed the others (Fig. 3e). For Scenario 4 (Setting 6) in which groups with the same mean functions but different covariance functions, the performance of FSVR using the Gaussian kernel and SVR using the Gaussian kernel outperformed the rest and FSVR using the Gaussian kernel was better than SVR using the Gaussian kernel when N = 50 and J = 40 (Fig. 3f).

4 Real Data Applications

In this section, we explore the performance of our proposed methods when they are applied to two different data sets, one in which the outcome is binary and one with a continuous outcome.

4.1 Binary outcome: EEG data

EEG is a neuroimaging technique that measures the electrical activity of the brain via electrodes placed on the scalp surface. We apply our classification procedure to a large study (Zhang et al. 1995) that collected EEG data of 77 alcoholics and 45 controls, measuring the amplitude at 64 electrodes each sampled at frequencies 1 to 256 Hz. Each subject completed multiple trials where two picture stimuli appeared in succession. In each trial, the two pictures were either matching or non-matching and the subjects were asked to decide whether the two pictures were the same. Previous studies have found different patterns in the spectra between matching and non-matching trials (Zhang et al. 1995). Thus, our goal is to use the difference of these patterns to classify alcoholics and controls.

In our analysis, we considered only the P1 electrode located in the parietal region and the P5 electrode located in the occipital region using one matching trial and one non-matching trial for each subject. Figures 4a and 4b show the difference between spectra for all subjects at P1 and P5 electrodes. From each group, we randomly selected 80% of subjects to form the training data set leaving the remaining 20% to form the validation data set.

Figure 4:

Waveform, classification accuracy, and AUC of EEG study. a) and b) Amplitude of the difference between matching condition and nonmatching condition of all subjects at P1 and P5 electrodes at 1 to 256 Hz, respectively. Solid lines: alcoholics; Dashed lines: controls. c) and d) Accuracy of classifying alcoholics and controls by P1 and P5 waveforms across 100 bootstrap samples. e) and f) AUC of classifying alcoholics and controls by P1 and P5 waveforms across 100 bootstrap samples.

For both electrodes, we fixed the number of knots to be 15. We searched smoothing parameter $\tau$ in the grid of $(0.5,1,5,10)$, the number of FPCs K in the grid of $(1,2,\dots ,10)$, and C in the grid of $(0.01,0.2575,0.5050,0.7525,1)$. The optimal tuning parameters were determined by five-fold cross-validation which minimized the cross-validated misclassification errors. When the Gaussian kernel was used, the bandwidth parameter $\gamma$ was determined by the median heuristic. We compared the FSVC approaches with the same alternative approaches as in the simulation section. The search grids of K and C for SVC approaches were the same as for FSVC approaches. We repeated the analysis 100 times by randomly assigning subjects to the training and validation data sets.

The median number of FPCs selected by FSVC approaches for P1 was eight and that for P5 was seven. FPCA results on the EEG dataset for all subjects are shown in Figure S3 of the Supplementary Materials. For both P1 and P5 electrodes, although the first three FPCs accounted for more than 80% of the total variance, the fourth to eighth FPCs also contributed to classification.

The classification accuracies of each method across the 100 bootstrap samples are shown in Figures 4c and 4d. In terms of the mean and median accuracy estimates, FSVC using the Gaussian kernel outperformed all the other methods for both P1 and P5 waveforms. FSVC using the Gaussian kernel was the only method that had a median value of classification accuracy above 0.7. We also evaluated the AUC by varying the threshold values from 0.1 to 0.9 by 0.2. For both P1 and P5 waveforms, FSVC using the Gaussian kernel outperformed all other methods (Figures 4e and 4f).

4.2 Continuous outcome: near-infrared reflectance spectroscopy data

Glucobrassicin (GBS), a type of glucosinolate in most Brassica vegetables, such as Brussels sprouts and cabbage, produces indole-3-carbinol, a compound that suppresses tumors and increases carcinogen metabolism and excretion (Kassie et al. 2007). It is widely thought that increasing GBS consumption could reduce cancer risk (van Poppel et al. 1999). The standard method to quantify GBS requires high-performance liquid-chromatography, which is expensive and time consuming (Norm et al. 1992). Alternatively, a NIRS technique that utilizes the near-infrared region of the electromagnetic spectrum would be preferred to measure GBS concentrations because it is inexpensive, fast, and nondestructive (Font et al. 2005; Toledo-Martín et al. 2017).

We analyzed the cabbage and Brussels sprout data set that consists of 92 samples (Renner and Fritz 2020). The NIR spectra were between 950 and 1650 nm recorded as the $\text{log}\hspace{0.17em}$ of 1/reflectance, resulting in 141 wavelengths. Our goal is to use the NIR spectra (Fig. 5a) to predict the fresh weight and dry weight GBS concentrations. We randomly selected 50 samples as training data and the remaining 42 samples as validation data. We implemented FSVR using both the linear and Gaussian kernels. We fixed the number of knots to be 35 and searched smoothing parameter $\tau$ in the grid of (0.5, 1, 5, 10), the number of FPCs K in the grid of (1, 2, …, 10), C in the grid of (0.01, 0.2575, 0.5050, 0.7525, 1), and $\nu$ in the grid of $(0.1,0.3,0.5,0.8)$. The optimal tuning parameters were determined by five-fold cross-validations. When the Gaussian kernel was used, the bandwidth parameter $\gamma$ in the Gaussian kernel was determined by the median heuristic. We repeated the analysis 100 times by randomly assigning samples to the training and validation data sets. We compared FSVR approaches with functional linear regression and SVR using the linear and Gaussian kernels. For SVR, the searching grids of K, C, and $\nu$ were the same as in FSVR.

Figure 5:

NIR spectra and root mean square error (RMSE) of NIR study. a) NIR spectra of all samples. Solid line: NIR spectra for each sample; Dashed line: average NIR spectra over samples. b) and c) RMSE of predicting dry and fresh weight GBS concentrations across 100 bootstrap samples.

The median of the selected number of FPCs was eight for FSVR. We present the FPCA results on the entire data set in Figure S4 of the Supplementary Materials. Although the first FPC accounted for 92.6% of the total variance, the other seven FPCs also contributed to help predict GBS concentrations.

We present the RMSE of each method for predicting dry and fresh weight GBS concentrations across the 100 bootstrap samples in Figures 5b and 5c. FSVR using the Gaussian kernel outperformed all the other methods in terms of the mean, median, and standard deviation of RMSE for predicting both dry weight and fresh weight GBS concentrations.

The average running time per analysis on the EEG dataset was about 8.6 s and on the NIRS dataset was about 6.4 s on an Intel Core i9 3.6 GHz processor.

5 Discussion

In this work, we propose a novel nonparametric approach to use functional data for classification and regression analysis. Our methods exploit the continuous nature of the functional data by smoothing FPCA and using SVM techniques to construct linear or nonlinear hyperplanes. We also discuss the connections between our methods using linear kernel and linear scalar-on-function regression using principal component basis. The superior performance of our methods is demonstrated in the extensive simulation experiments and real data applications. Our methods especially have more advantages when the measurement errors in functional predictors are relatively large because the smoothed FPCA accounts for noise. When the measurement errors in functional predictors are smaller, SVM performed similarly to FSVM in some scenarios, which suggests that there is a tradeoff between smoothness and dimension reduction. In our real data applications, we also see that some FPC scores can be useful in prediction even though they accounted for a relatively small amount of the variance. In the implementations of classical SVM, dimension reduction is not typically used because the kernel trick maps the features into a higher-dimensional space and dimension reduction goes the opposite way. When a linear kernel is used in FSVMs, the coefficients of $\mathit{\beta }$ indicates the importance of the input features. However, FSVMs with Gaussian kernels do not provide explicit feature importance. To identify the relevance of features, one can employ feature selection algorithms such as recursive feature elimination (Guyon et al. 2002).

Several extensions can be considered. We only considered binary outcomes and continuous outcomes in this work, but our method can be easily generalized to multi-class classifications by either one-vs-rest or one-vs-one strategies (Bishop and Nasrabadi 2006). In our model, all the predictors are functional predictors. When additional scalar variables are also helpful in classification and prediction, we can include them as additional input features in the SVM step.

Our technique can also be extended to estimate individualized treatment rules along the lines of outcome-weighted learning (Zhao et al. 2012) which used the SVM technique to solve a weighted classification problem with outcomes as weights. When functional features (e.g. neuroimaging measurements) are available, we can implement FSVM.

6 SOFTWARE

The R code for implementing our method is available on GitHub at https://github.com/shanghongxie/FSVM.

Supplementary Material

Supplementary material is available at Biostatistics Journal online.

Funding

This work was partially supported by the National Institutes of Health grant 5 R01 MH099003. Xie was also supported by the Center of Statistical Research, and the Joint Lab of Data Science and Business Intelligence at the Southwestern University of Finance and Economics.

Conflict of interest statement

None declared.