Sparse Machine Learning in Banach Spaces

Abstract

The aim of this expository paper is to explain to graduate students and beginning researchers in the field of mathematics, statistics and engineering the fundamental concept of sparse machine learning in Banach spaces. In particular, we use binary classification as an example to explain the essence of learning in a reproducing kernel Hilbert space and sparse learning in a reproducing kernel Banach space (RKBS). We then utilize the Banach space ${\ell }_1\left(\mathbb{N}\right)$ to illustrate the basic concepts of the RKBS in an elementary yet rigorous fashion. This paper reviews existing results in the author’s perspectives to reflect the state of the art of the field of sparse learning, and includes new theoretical observations on the RKBS. Several open problems critical to the theory of the RKBS are also discussed at the end of this paper.

1. Introduction

Most of machine learning methods are to learn a function from available data. Mathematically, we need a hypothesis space to hold functions to be learned [13]. Choices of the hypothesis space are critical for learning outcomes. Traditionally, a reproducing kernel Hilbert space (RKHS), a Hilbert space where the point-evaluation functionals are continuous, is chosen, due to the existence of an inner product that can be used to measure similarity in data and continuous point-evaluation functionals that often are used to sample data.

While learning in an RKHS offers attractive features, it suffers from a major drawback. To explain it, we define the notion of sparsity. We say a vector or a sequence is sparse, we mean most of its components are zero. When it is not sparse, we call it dense. We stress here that this definition of density is peculiar to the data science community, and it does not mean (topologically) dense in the usual sense. Sparsity also refers to a representation of a function under a basis when the coefficient vector is sparse. A major drawback of learning in a Hilbert space is that a learned solution tends to be dense. This feature is a consequence of the smoothness of Hilbert spaces used as hypothesis spaces. Such a dense learned solution leads to high computational costs when the learned solution is used in prediction or other decision making procedures since once a function is learned it will be used repeatedly many times. The suffering is even more severe in the context of big data analytics. For example, Google would want its search engine to be able to make instant recommendations. Due to increasingly greater amounts of data and related model sizes, demands for more competent data processing models have emerged. As pointed in [21], the future of machine learning is sparsity. It is immensely desirable to learn sparse solutions in the sense that they have substantially fewer nonzero components or fewer terms in their representation.

Most data sets have intrinsic sparsity. As a matter of fact, data that we encounter often have certain embedded sparsity structures in the sense that if they are represented in certain ways, their intrinsic characteristics can concentrate on a few scattering spots. To construct sparse solutions, we require that the norm of the hypothesis space for the learning model have certain sparsity promoting property.

With a sparsity promoting norm for the hypothesis space, when a solution of a learning model is represented in an appropriate basis, the solution can have a sparse representation. It is well-known that the norm of a usual Hilbert space does not promote sparsity due to its smoothness. Norms of certain nonsmooth Banach spaces have the ability to induce sparsity for solutions of learning methods having the spaces as hypothesis spaces, such as the 1-norm. For this reason, some Banach spaces have been used as hypothesis spaces for sparse machine learning methods. With sparse solutions, machine learning models can considerably reduce storage, computing time and communication time.

We are interested in a class of Banach spaces which are spaces of functions because function values are useful in machine learning. In particular, we would like the point-evaluation functionals in such spaces to be continuous. This gives rise to a special class of Banach spaces: the reproducing kernel Banach spaces (RKBSs) which was introduced in [58]. During the last decade, great interest has been paid to understanding of this class of function spaces and their uses in applications. We will discuss these function spaces and learning in the spaces.

The goal of this expository paper is to review recent advances in sparse machine learning in Banach spaces. We will use binary classification, a basic problem in data science, as an example to motivate the notion of sparse learning in Banach spaces. In Section 2, we present a brief review of classification. Section 3 is devoted to a presentation of learning in an RKHS. In Section 4 we illustrate the theory of the RKBS by the example of the ${\ell }_1\left(\mathbb{N}\right)$ space. We discuss in Section 5 learning in a Banach space. We present a numerical example in Section 6 to demonstrate a sparse classification method in a Banach space. In Section 7, we elaborate several unsolved mathematical problems related to the theory of RKBS and sparse learning.

2. A Review of Classification

In this section, we review the classical support vector machine for binary classification which was discussed in [52].

A typical binary classification problem maybe described as follows: Given two sets of points in the Euclidean space ${ℝ}^d$, we wish to find a surface to separate them. A simplest way to separate two sets of points is to use a hyperplane, since among all types of surfaces a hyperplane is the easiest to construct and to work with. For this purpose, we wish to construct the hyperplane

$$w^{\top }x-b=0,\hspace{1em}x\in {ℝ}^d,$$ (2.1)

where $\mathbf{w}\in {ℝ}^d$ and $b\in ℝ$ are to be determined so that the hyperplane (2.1) has the largest distance to both of the two sets. Specifically, we let the training data $D:=\left\{\right(x_k,y_k):k=1,2,\dots ,N\}$ be composed of input data points

$$X≔\left\{x_k:k=1,2,\dots ,N\right\}\subset {ℝ}^d$$

and output data values

$$Y≔\left\{y_k:k=1,2,\dots ,N\right\}\subset \{-1,1\}.$$

We intend to find a hyperplane determined by the linear function

$$s(x)≔w_s^{\top }x-b_s,\hspace{1em}x\in {ℝ}^d,$$ (2.2)

which separates the training data $D$ into two groups, one with label $y_k=1$ which corresponds to $s\left({\mathbf{x}}_k\right)>0$ and another with label $y_k=-1$ which corresponds to $s\left({\mathbf{x}}_k\right)<0$. The parameters $\left(w_s,b_s\right)\in {ℝ}^d\times ℝ$ in equation (2.2) are chosen such that the hyperplane maximizes its distance to the points in $D$. This gives us a decision rule to predict the labels for new points, that is,

$$r(x)≔\text{sign}(s(x)),\text{ for }x\in {ℝ}^d.$$

This method is called the support vector machine (SVM) [12].

Specifically, as illustrated in Figure 2.1 we select two parallel hyperplanes

$$w^{\top }x-b=1\text{ and }{w}^{\top }x-b=-1$$

that separate the two classes of data so that the distance between the two hyperplanes is as large as possible. The region bounded by these two hyperplanes is called the margin. We compute the distance $\mathbf{d}$ between these two planes. Let ${\mathbf{x}}_0\in {ℝ}^d$ be a point on the first plane. That is, ${\mathbf{x}}_0$ satisfies the equation

$$w^{\top }{x}_0-b=1.$$ (2.3)

The distance $\mathbf{d}$ between these two planes is given by

$$d=\frac{\left|w^{\top }{x}_0-b+1\right|}{{\Vert w\Vert }_2}=\frac{2}{{\Vert w\Vert }_2}.$$

Maximizing the distance $\mathbf{d}$ is equivalent to minimizing $\frac{1}{d}=\frac{1}{2}{\Vert w\Vert }_2$. To prevent data points from falling into the margin, we require

$$w^{\top }{x}_k-b\ge 1\text{, if }{y}_k=1\text{ or }{w}^{\top }{x}_k-b\le -1\text{, if }{y}_k=-1.$$

These constraints state that each data point must lie on the correct side of the margin, and they may be rewritten as

$$y_k\left(w^{\top }{x}_k-b\right)\ge 1,\hspace{1em}k=1,2,\dots ,N.$$ (2.4)

The pair $({\mathbf{w}}_s,b_s)$ is solved by a constrained minimization problem

$$\text{min}\left\{\frac{1}{2}{\Vert w\Vert }_2:(w,b)\in {ℝ}^d\times ℝ\right\}$$ (2.5)

subject to (2.4). Let

$$C≔\left\{(w,b)\in {ℝ}^d\times ℝ:y_k\left(w^{\top }{x}_k-b\right)\ge 1,\hspace{1em}k=1,2,\dots ,N\right\}.$$ (2.6)

It can be verified that $C$ is a convex set. The minimization problem (2.5) with constraints (2.4) may be rewritten as

$$\text{min}\left\{\frac{1}{2}{\Vert w\Vert }_2:w\in C\right\},$$ (2.7)

where the constraint set $C$ is defined by (2.6). Model (2.7) is called the hard margin support vector machine (SVM), which is a constrained minimization problem.

Figure 2.1:

Classification (Courtesy: Wikipedia)

It is computationally advantageous to reformulate the constrained minimization problem (2.7) as an unconstrained one. This leads to the soft margin SVM model. Specifically, we rewrite the constraints (2.4) as

$$1-y_k\left(w^{\top }{x}_k-b\right)\le 0,\hspace{1em}k=1,2,\dots ,N,$$ (2.8)

and make use of the ReLU (Rectified Linear Unit) function

$$\sigma (s)≔\text{max}\left\{0,s\right\},\text{ for }s\in ℝ$$

to define the hinge loss function

$$L(y,t)≔\sigma (1-yt),\hspace{1em}\text{for }y\in \left\{-1,1\right\}\text{ and }t\in ℝ.$$

When (2.8) is satisfied, we have that $L\left(y_k,w^{\top }{x}_k-b\right)=0$ and when (2.8) is not satisfied, we have that

$$L\left(y_k,w^{\top }{x}_k-b\right)=1-y_k\left(w^{\top }{x}_k-b\right)=\left|1-y_k\left(w^{\top }{x}_k-b\right)\right|>0,$$

which is the case that we would like to prevent from occurring. For this reason, we would like to minimize the nonnegative fidelity term

$$\frac{1}{N}\sum _{k=1}^{N}L\left(y_k,w^{\top }{x}_k-b\right).$$

Combining this with the minimization (2.5) leads to the regularization problem

$$\text{min}\left\{\frac{1}{N}\sum _{k=1}^{N}L\left(y_k,w^{\top }{x}_k-b\right)+\lambda {\Vert w\Vert }_2^2:(w,b)\in {ℝ}^d\times ℝ\right\},$$ (2.9)

where $\lambda >0$ is a regularization parameter. Regularization problem (2.9) is called the soft margin SVM. Upon solving minimization problem (2.9), we obtain the pair $({\mathbf{w}}_s,b_s)$ which defines the function $s$ for classification.

3. Learning in Reproducing Kernel Hilbert Spaces

We discuss in this section the notion of feature maps in machine learning. In particular, we show the necessity of introducing feature maps in machine learning, which leads to learning in an RKHS.

We first motivate the introduction of feature maps in machine learning by returning to the classification problem reviewed in the last section. When given data sets can be separated by a hyperplane, the hard/soft margin SVM discussed in the last section will do a reasonably good job for classification. However, in most cases of practical applications, it is not possible to separate two sets of points entirely by a hyperplane in the same space. Certain misclassification may occur. In general, we might allow a low degree of misclassification but do not tolerate a high degree of misclassification. We are required to separate two sets with a low degree of misclassification.

One way to alleviate misclassification is to map data sets to a higher (or even an infinite) dimensional space and perform classification in the new space. Note that sparse data sets are easier to be separated by a hyperplane, with a low degree of misclassification. Non-sparse data sets in a lower dimensional space may be made sparse if they are projected to a higher or even an infinite dimensional space by an appropriate map. A nice idea is to find a feature map $\Phi$ to map lower dimensional data sets into a higher dimensional space to gain sparsity for the resulting data sets. The sparsity of the mapped data sets in the higher dimensional space can significantly reduce the degree of misclassifications. In other words, the feature map $\Phi$ transfers the classification problem in a lower dimensional space to one in a higher dimensional space, where it is possible to separate point sets mapped from ${ℝ}^d$ by a “hyperplane”, with a lower degree of misclassification. This is illustrated in Figure 3.2. A crucial issue is the choice of the feature map $\Phi$. From a pure theoretical standpoint, “any” function $\Phi$ is a feature map. The choice of the feature map $\Phi$ pretty much depends on specific applications. Feature maps lead to the notion of kernel based learning or learning in RKHSs.

Figure 3.2:

An illustration of data mapping

We next demonstrate how a feature map defines a kernel. A feature map has a well-known remarkable property which we show below. We recall that ${\mathbf{x}}_j\in {ℝ}^d$, $j=1,2,\dots ,n$, are the $n$ data points.

Proposition 3.1 Let $\mathscr{H}$ be a real Hilbert space with inner product ${\langle ·,·\rangle }_{\mathscr{H}}$. If $\Phi :{ℝ}^d\to \mathscr{H}$, then for all $n\in \mathbb{N}$, $X_n≔\left\{x_j\in {ℝ}^d:j=1,2,\dots ,n\right\}$, the matrices

$$\Phi \left(X_n\right)≔\left[{\langle \Phi \left(x_i\right),\Phi \left(x_j\right)\rangle }_{\mathscr{H}}:i,j=1,2,\dots ,n\right]$$

are positive semi-definite.

Proof: For all $n\in \mathbb{N}$, suppose that $c_j\in ℝ$, $j=1,2,\dots ,n$, are arbitrarily given. Let $\mathbf{c}≔{[c_1,c_2,\dots ,c_n]}^{\top }$. For all ${\mathbf{x}}_j\in {ℝ}^d$, $j=1,2,\dots ,n$, there holds

$$\begin{array}{c}{c}^{\top }\Phi \left(X_n\right)c=\sum _{i=1}^n\sum _{j=1}^n{c}_i{c}_j{\langle \Phi \left(x_i\right),\Phi \left(x_j\right)\rangle }_{\mathscr{H}}\\ ={\langle \sum _{i=1}^n{c}_i\Phi \left(x_i\right),\sum _{j=1}^n{c}_j\Phi \left(x_j\right)\rangle }_{\mathscr{H}}\\ ={\Vert \sum _{i=1}^n{c}_i\Phi \left(x_i\right)\Vert }^2\ge 0.\end{array}$$

This confirms that the matrices $\Phi \left(X_n\right)$ are positive semi-definite. □

The feature map $\Phi :{ℝ}^d\to \mathscr{H}$ naturally leads to a function

$$K(x,y)≔{\langle \Phi (x),\Phi (y)\rangle }_{\mathscr{H}},\hspace{1em}x,y\in {ℝ}^d,$$ (3.1)

which is a kernel according to Proposition 3.1 and the following definition.

Definition 3.2 Let $\mathbf{X}\subset {ℝ}^d$. A function $K:\mathbf{X}\times \mathbf{X}\to ℝ$ is called a kernel if it is symmetric and positive semi-definite, that is,

$$\sum _{i=1}^n\sum _{j=1}^n{c}_i{c}_{j}K\left(x_i,x_j\right)\ge 0,\hspace{1em}foralln\in \mathbb{N},x_j\in X,c_j\in ℝ,j=1,2,\dots ,n.$$

Kernels were studied over 100 years ago in the context of integral equations [29]. See, also [4]. Kernels can define distances of data sets to measure their similarity.

Function values are often used in machine learning. Hence, it would be desirable to require that the Hilbert space $\mathscr{H}$ of functions defined on $\mathbf{X}$ that we work with has the property that the point-evaluation functionals are continuous in the space. Precisely, we have the following definition.

Definition 3.3 A Hilbert space $\mathscr{H}$ of functions defined on $\mathbf{X}$ is called an RKHS if for every point-evaluation functional ${\delta }_{\mathbf{x}}$, $\mathbf{x}\in \mathbf{X}$, there exists a constant $M_{\mathbf{x}}$ such that for all $f\in \mathscr{H}$

$$\left|{\delta }_{x}f\right|=\left|f(x)\right|\le M_x{\Vert f\Vert }_{\mathscr{H}}.$$

We now illustrate Definition 3.3 by a simplest example. To this end, we consider the space ${\ell }_2\left(\mathbb{N}\right)$ of real sequences $\mathbf{f}≔[f_1,f_2,\dots ]$ such that ${\Vert f\Vert }_2:={\sum }_{k\in \mathbb{N}}{\left|f_k\right|}^2<\infty$. Clearly, elements of ${\ell }_2\left(\mathbb{N}\right)$ are functions defined on the set $\mathbf{X}≔\mathbb{N}$ and ${\ell }_2\left(\mathbb{N}\right)$ is a Hilbert space, and thus, it is a Hilbert space of functions on $\mathbb{N}$. Moreover, ${\ell }_2\left(\mathbb{N}\right)$ is isometrically isomorphic to ${\left({\ell }_2\right(\mathbb{N}\left)\right)}^{*}$. The point-evaluation functional has the form ${\delta }_{\mathbf{x}}\left(\mathbf{f}\right)={\delta }_j\left(\mathbf{f}\right)≔f_j$, for $\mathbf{x}≔j\in \mathbb{N}$. It follows that

$$\left|{\delta }_j(f)\right|=\left|f_j\right|\le {\Vert f\Vert }_2,\hspace{1em}\text{for all}f\in {\ell }_2(\mathbb{N}),$$

with $M_{\mathbf{x}}$ in Definition 3.3 being $M_j≔1$, for all $j\in \mathbb{N}$. Therefore, according to Definition 3.3, ${\ell }_2\left(\mathbb{N}\right)$ is an RKHS on $\mathbb{N}$ with the kernel $K:=\left[{\delta }_{i,j}:i,j\in \mathbb{N}\right]$, where ${\delta }_{i,j}≔1$ if $i=j$ and 0 otherwise, for $i,j\in \mathbb{N}$.

Immediately from Definition 3.3, we observe that the closeness of two functions in an RKHS in the norm of the space implies their pointwise closeness.

An RKHS is associated with a kernel. The following well-known result can be found in [1].

Theorem 3.4 If a Hilbert space $\mathscr{H}$ of functions on $\mathbf{X}$ is an RKHS, then there exists a unique kernel $K:\mathbf{X}\times \mathbf{X}\to ℝ$ such that $K(\mathbf{x},·)\in \mathscr{H}$, $\mathbf{x}\in \mathbf{X}$ and for all $f\in \mathscr{H}$,

$$f(x)={\langle f(\cdot ),K(x,\cdot )\rangle }_{\mathscr{H}},\hspace{1em}x\in X.$$ (3.2)

If $K:\mathbf{X}\times \mathbf{X}\to ℝ$ is a kernel, then there exists a unique RKHS $\mathscr{H}$ on $\mathbf{X}$ such that $K(\mathbf{x},·)\in \mathscr{H}$, $\mathbf{x}\in \mathbf{X}$ and for all $f\in \mathscr{H}$, the reproducing property (3.2) holds true.

Proof: Suppose that $\mathscr{H}$ is an RKHS of functions on $\mathbf{X}$. Since the point-evaluation functional ${\delta }_{\mathbf{x}}f≔f\left(\mathbf{x}\right)$, for each $\mathbf{x}\in \mathbf{X}$, is continuous, by the Riesz representation theorem, there exists $k_{\mathbf{x}}\in \mathscr{H}$ such that $f\left(\mathbf{x}\right)={\delta }_{\mathbf{x}}f={\langle f,k_{\mathbf{x}}\rangle }_{\mathscr{H}}$ for all $f\in \mathscr{H}$. For each $\mathbf{x},\mathbf{y}\in \mathbf{X}$, we define a function $K:\mathbf{X}\times \mathbf{X}\to ℝ$ by $K(\mathbf{x},\mathbf{y})≔k_{\mathbf{x}}\left(\mathbf{y}\right)$ for $\mathbf{x},\mathbf{y}\in \mathbf{X}$. Clearly, we have that for all $\mathbf{x}\in \mathbf{X}$, $K(\mathbf{x},·)\in \mathscr{H}$ and (3.2) holds true. The uniqueness of such a function $K$ can be verified. It remains to show that $K$ is a kernel. The symmetry of $K$ can be seen from the computation

$$K(x,y)=k_x(y)={\langle k_x(\cdot ),k_y(\cdot )\rangle }_{\mathscr{H}}={\langle K(x,\cdot ),K(y,\cdot )\rangle }_{\mathscr{H}}={\langle K(y,\cdot ),K(x,\cdot )\rangle }_{\mathscr{H}}=K(y,x).$$

It remains to prove that it is positive semi-definite. For any $n\in \mathbb{N}$, ${\mathbf{x}}_j\in \mathbf{X}$, $c_j\in ℝ$, $j=1,2,\dots ,\mathrm{n}$, we let

$$c≔{\left[c_1,c_2,\dots ,c_n\right]}^{\top }\text{ and }{K}_n≔\left[K\left(x_i,x_j\right):i,j=1,2,\dots ,n\right]\text{. }$$

By the reproducing property and symmetry of the kernel, we have that

$$\begin{gathered} c^{\top }{K}_{n}c=\sum _{i=1}^n\sum _{j=1}^n{c}_i{c}_{j}K\left(x_i,x_j\right) \\ =\sum _{i=1}^n\sum _{j=1}^n{c}_i{c}_j{\langle K\left(x_j,\cdot \right),K\left(x_i,\cdot \right)\rangle }_{\mathscr{H}} \\ ={\langle \sum _{j=1}^n{c}_{j}K\left(x_j,\cdot \right),\sum _{i=1}^n{c}_{i}K\left(x_i,\cdot \right)\rangle }_{\mathscr{H}} \\ ={\Vert \sum _{i=1}^n{c}_{i}K\left(x_i,\cdot \right)\Vert }^2\ge 0. \end{gathered}$$

Thus, $K$ is a kernel. We omit the proof for the converse. □

Theorem 3.4 clearly states that every function in the RKHS can be reproduced by the underlying kernel. Equation (3.2) is called the reproducing property. For this reason, the kernel is also called the reproducing kernel. Moreover, the kernel (3.1) defined by a feature map $\Phi$ determines an RKHS $\mathscr{H}$. The mapped RKHS $\mathscr{H}$ is more complex than the original space ${ℝ}^d$ and thus by using it as a hypothesis space of a learning method we can better represent our learned function. We expect that learning outcomes in this space will be more accurate than those in the original Euclidean space. In particular, data sets mapped from ${ℝ}^d$ to $\mathscr{H}$ via the feature map are likely more sparse than the original data sets and thus classification of the mapped data sets may result in substantially less misclassification.

Two most popular kernels in machine learning are the polynomial kernel of degree $r$

$$K(x,y)≔{\left(x^{\top }y+c\right)}^r,\hspace{1em}\text{for }\hspace{1em}x,y\in {ℝ}^d,$$ (3.3)

where $c\in ℝ$ is a constant and the Gaussian kernel

$$K(x,y)≔e^{-{\theta }^2{\Vert x-y\Vert }_2^2},\hspace{1em}\text{ for }\hspace{1em}x,y\in {ℝ}^d.$$ (3.4)

Google Scholar shows that the term “polynomial kernel” is used in titles of 53,900 articles and the term “Gaussian kernel” is used in titles of 649,000 articles. Gaussian kernels are used widely in application due to its remarkable flexibility in fitting data. In Figures 3.3, we illustrate that by altering the parameter $\theta$ we can control the shape of the graphs of the Gaussian kernel.

Figure 3.3:

$K(\mathbf{x},0)$ with $\theta =1.6$ (left); with $\theta =3$ (right)

Following the above discussion, we map the data points ${\mathbf{x}}_j\in {ℝ}^d$ to the RKHS $\mathscr{H}$ by the feature map $\Phi$ and seek a function $f\in \mathscr{H}$ that defines a decision rule for classification. This discussion leads us to extend the regularization problem (2.9) for classification to

$$\text{min}\left\{\frac{1}{n}\sum _{k=1}^{n}L\left(y_k,{\langle f,\Phi \left(x_k\right)\rangle }_{\mathscr{H}}\right)+\lambda {\Vert f\Vert }_{\mathscr{H}}^2:f\in \mathscr{H}\right\}$$ (3.5)

to determine a decision function $f$ ∈ the RKHS $\mathscr{H}$. It follows from the reproducing property that

$${\langle f,\Phi \left({\text{x}}_k\right)\rangle }_{\mathscr{H}}={\langle f,K\left({\text{x}}_k,\cdot \right)\rangle }_{\mathscr{H}}=f\left({\text{x}}_k\right).$$

Thus, the regularization problem (3.5) for classification is rewritten as

$$\text{min}\left\{\frac{1}{n}\sum _{k=1}^{n}L\left(y_k,f\left(x_k\right)\right)+\lambda {\Vert f\Vert }_{\mathscr{H}}^2:f\in \mathscr{H}\right\}.$$ (3.6)

Clearly, the regularization problem (2.9) with $b=0$ is a special case of (3.6). To see this, we introduce the reproducing kernel

$$K(x,\mathbf{\text{y}})≔\langle \mathbf{\text{x}},\mathbf{\text{y}}\rangle ,\hspace{1em}\text{for all}x,y\in {ℝ}^d\text{. }$$

The corresponding RKHS has the form $\mathscr{H}≔\{\langle ·,\mathbf{x}\rangle :\mathbf{x}\in {ℝ}^d\}$, with ${\langle \langle ·,x\rangle ,\langle ·,y\rangle \rangle }_{\mathscr{H}}:=\langle x,y\rangle$ for all $\mathbf{x},\mathbf{y}\in {ℝ}^d$, as its inner product. According to the reproducing property that for each $f≔\langle ·,\mathbf{w}\rangle$ with $\mathbf{w}\in {ℝ}^d$, we deduce that $f\left(\mathbf{x}\right)=\langle \mathbf{x},\mathbf{w}\rangle ={\mathbf{w}}^{\top }\mathbf{x}$, for all $\mathbf{x}\in {ℝ}^d$, and ${\parallel f\parallel }_{\mathscr{H}}={\parallel \mathbf{w}\parallel }_2$. Thus, problem (3.6) reduces to (2.9) with $b=0$.

The above discussion of classification motivates us to consider learning methods in RKHSs. For example, given a pair $\{X,Y\}$ of data sets $X≔\{{\mathbf{x}}_j\in {ℝ}^d:j=1,2,\dots ,m\}$, and $Y≔\{y_j\in ℝ:j=1,2,\dots ,m\}$, we wish to learn a function $f^{*}$ from an RKHS $\mathscr{H}$. This learning method may be described as the minimization problem

$$f^{*}=\text{argmin}\left\{F(X,Y,f)+\lambda {\Vert f\Vert }_{\mathscr{H}}:f\in \mathscr{H}\right\},$$ (3.7)

where $F$ denotes the empirical fidelity term which measures the closeness of a learned solution to the given data and $\lambda >0$ is a regularization parameter. The learning model (3.7) covers many learning problems including minimal norm interpolation, regularization network, support vector machines, kernel principal component analysis, kernel regression and deep regularized learning. Moreover, the recent work [33] established a framework where distributions are mapped into an RKHS in which the kernel methods can be extended to probability measures.

A great success of learning in Hilbert spaces has been achieved. At the same time theory of the RKHS gains coordinated development [19, 56, 57, 59]. In particular, motivated from needs of developing multiscale bases for an RKHS, refinable kernels were investigated in [56, 57, 59]. A mathematical foundation of learning in Hilbert spaces may be found in [13].

Two fundamental mathematical problems were raised regarding learning in an RKHS. The first one regards the form of a solution of the learning problem (3.7) which seeks a solution in an infinite dimensional space $\mathscr{H}$. The second is about the approximation property of the learning methods. Does the learned solution converge to the “true solution” of the practical problem as data points become dense and their number tends to infinity?

Answer to the first fundamental problem is the celebrated representer theorem [40]. A solution of the learning problem (3.7) in an RKHS $\mathscr{H}$ can be represented as a linear combination of a finite number of the kernel sessions, kernel $K$ evaluated at the input training data points, namely,

$$K\left({\mathbf{\text{x}}}_k,\mathbf{\text{x}}\right),\hspace{1em}k=1,2,\dots ,m.$$ (3.8)

That is, a solution of the learning problem (3.7) has the form

$$f^{*}(\mathbf{\text{x}})=\sum _{k=1}^m{c}_{k}K\left(x_k,x\right),$$ (3.9)

for some suitable parameters $c_k\in ℝ$, where $K$ is the kernel associated with the space $\mathscr{H}$. Here the integer $m$ is the number of the data points involved in the fidelity term of (3.7). This result is called the representer theorem for the learning method (3.7). The most remarkable feature of the representer theorem lies in a finite number of kernel sessions that represent a learned solution in the infinite dimensional space $\mathscr{H}$. In other words, even though we seek a learned solution in the infinite dimensional space $\mathscr{H}$ hoping it will provide much more richness in representing a learned solution, the learned solution is indeed located in the finite dimensional subspace of $\mathscr{H}$, spanned by the kernel sessions (3.8). Although the modern form of the representer theorem was found in [40], its origin dates back to [5] and [23] that appeared fifty years ago.

An answer to the second fundamental problem for the learning method (3.7) is the universality of a kernel. According to the representer theorem (3.9), a solution $f^{*}$ of the learning method (3.7) is expressed as linear combination of a finite number of the kernel sessions. This motivates us to ask the following question: Can a continuous function on a compact set be approximated arbitrarily well by the kernel sessions as the data points getting dense in the set? It is well-known from the Weiersstrass Theorem in Real Analysis [38] that a continuous function on a compact domain can be arbitrarily approximated by polynomials. However, this is not always true when polynomials are replaced by kernel sessions of an arbitrary kernel. To explain this, we assume that the input space $\mathbf{X}$ is a Hausdorff topological space and that all kernels to be considered are continuous on $\mathbf{X}\times \mathbf{X}$. Moreover, we let $\mathbf{Z}$ be an arbitrarily fixed compact subset of $\mathbf{X}$ and let $C\left(\mathbf{Z}\right)$ denote the space of all continuous real-valued functions from $\mathbf{Z}$ to $ℝ$ or $ℂ$ equipped with maximum norm ${\parallel ·\parallel }_{\mathbf{Z}}$. For simplicity, we restrict our discussion to real-valued function.

We first examine the polynomial kernel (3.3). We have the following negative result.

Proposition 3.5 Let $K:{ℝ}^d\times {ℝ}^d\to ℝ$ be a polynomial kernel. There exists a continuous function defined on a compact set $\mathbf{Z}\subset {ℝ}^d$ which cannot be approximated in the uniform norm arbitrarily close by any linear combination of the kernel sessions $K({\mathbf{x}}_j,·)$, $j=1,2,\dots ,m$, for any $m$ and any $\{{\mathbf{x}}_j:j=1,2,\dots ,m\}\subset \mathbf{Z}$.

Proof: Since a polynomial kernel has the form (3.3), we will construct a function $f\in C\left(\mathbf{Z}\right)$ so that $f$ cannot be approximated in the uniform norm arbitrarily close by

$$f^{*}(x)≔\sum _{j=1}^m{c}_j{\left(x_j^{\top }x+c\right)}^r,\hspace{1em}x\in Z,$$ (3.10)

for any positive integer $m$ and any $\{{\mathbf{x}}_j:j=1,2,\dots ,m\}\subset \mathbf{Z}$.

In fact, for simplicity, we choose $f$ as a polynomial of degree $r+1$ with leading coefficient 1. No matter how large the integer $m$ is and no matter what ${\mathbf{x}}_j$, $j=1,2,\dots ,\mathrm{m}$, are chosen, $f^{*}$ defined by equation (3.10) is a polynomial of degree $r$. The assertion is proved by the fact that a nondegenerate polynomial of degree $r+1$ cannot be approximated in the uniform norm arbitrarily close by a polynomial of degree $r$. For example, we can verify this statement for the special case when $d=1$, $r=1$, $\mathbf{Z}=[-1,1]$, and $f\left(x\right)≔x^2$. In this case, we have that $f^{*}\left(x\right)=ax+b$, $a,b\in ℝ$, $x\in [-1,1]$. Let $e\left(x\right)≔f\left(x\right)-f^{*}\left(x\right)$, for $x\in [-1,1]$. Clearly, $e\left(x\right)=x^2-ax-b$. By the characterization of the best uniform approximation by the linear polynomial [36], we see that

$$\text{inf}\left\{{\Vert e\Vert }_{\left[-1,1\right]}:a,b\in ℝ\right\}=\text{max}\left\{\left|x^2-\frac{1}{2}\right|:x\in [-1,1]\right\}=\frac{1}{2}>0.$$

Hence, no matter how large the positive integer $m$ is chosen, no matter what points $x_j$, $j=1,2,\dots ,m$, are chosen, the corresponding error ${\Vert e\Vert }_{[-1,1]}\ge \frac{1}{2}$. □

This example motivates us to introduce in [32] the notion of universality of a kernel. We now describe the definition of the universal kernel. Given a kernel $K:\mathbf{X}\times \mathbf{X}\to ℝ$, we introduce the function $K_{\mathbf{y}}:X\to ℝ$ defined at every $\mathbf{x}\in \mathbf{X}$ by the equation $K_{\mathbf{y}}\left(\mathbf{x}\right)≔K(\mathbf{x},\mathbf{y})$ and form the space of kernel sections $K(Z)≔\overline{\text{span}}\left\{K_y:y\in Z\right\}$. The set $K\left(\mathbf{Z}\right)$ consists of all functions in $C\left(\mathbf{Z}\right)$ which are uniform limits of functions of the form (3.9) where $\{{\mathbf{x}}_j:j=1,2,\dots ,m\}\subset \mathbf{Z}$. We say that a kernel $K:\mathbf{X}\times \mathbf{X}\to ℝ$ is universal, if for any given prescribed compact subset $\mathbf{Z}$ of $\mathbf{X}$, any positive number $∊$ and any function $f\in C\left(\mathbf{Z}\right)$ there is a function $g\in K\left(\mathbf{Z}\right)$ such that

$$\Vert f-g{\Vert }_z\le ϵ.$$

A characterization of a universal kernel was originally established in [32]. Moreover, convenient sufficient conditions of universal kernels were given there and several classes of commonly used kernels including the Gaussian kernel defined by equation (3.4) were shown to be universal. However, it follows from the definition of the universal kernel and Proposition 3.5 that the polynomial kernels are not universal.

Although a solution of a learning method in an RKHS has nice features, it suffers from the denseness in its representation. To illustrate this point, we consider a simple minimum norm interpolation problem in ${\ell }_2\left(\mathbb{N}\right)$. We seek ${\mathbf{x}}^{*}\in {\ell }_2\left(\mathbb{N}\right)$ for such that

$${\Vert x^{*}\Vert }_2=\text{inf}\left\{{\Vert x\Vert }_2:x\in {\ell }_2(\mathbb{N}),{\langle a_i,x\rangle }_2=y_i,i=1,2\right\},$$ (3.11)

where

$$y_1≔3,y_2≔4,a_1≔\left(\frac{1}{n}:n\in \mathbb{N}\right),a_2≔\left(\frac{1}{{(-2)}^{n-1}}:n\in \mathbb{N}\right).$$ (3.12)

According to [10], the solution of problem (3.11)-(3.12) is given by

$${\mathbf{\text{x}}}^{*}\approx \left(\frac{0.4924584}{n}+\frac{2.7004714}{{(-2)}^{n-1}}:n\in \mathbb{N}\right).$$ (3.13)

Clearly, every component of the solution ${\mathbf{x}}^{*}$ is nonzero and thus, the solution is dense.

In the context of big data analytics, the dimension of data is large. It is desirable to obtain a sparse solution, a solution with substantial number of zero components. We next illustrate the intrinsic characteristic of a space that may or may not lead to a sparse solution. To this end, we consider a learning problem in a Hilbert space

$$\text{min}\left\{{\Vert x\Vert }_2:x\in {ℝ}^d\right\},\text{ subject to }Ax=b$$

and a Banach space

$$\text{min}\left\{{\Vert x\Vert }_1:x\in {ℝ}^d\right\},\text{ subject to }Ax=b.$$

Figures 3.4 and 3.5 illustrate that the minimum norm interpolations in the ${\ell }_1$ space are sparse but in the ${\ell }_2$ they are dense. This is because the geometry of the unit balls of these two different norms are different: The unit balls of the ${\ell }_2$ are smooth, which do not promote sparsity, while those of the ${\ell }_1$ have corners at coordinate axes, which promote sparsity.

Figure 3.4:

The ${\ell }_2$-Norm vs the ${\ell }_1$-Norm: A two dimension illustration.

Figure 3.5:

The ${\ell }_2$-Norm vs the ${\ell }_1$-Norm: A three dimension illustration.

We now return to the classification problem discussed in Section 2. To obtain a sparse solution, we replace the ${\ell }_2$ norm in (2.9) by the ${\ell }_1$ norm. This gives rise to the ${\ell }_1$ soft margin SVM model

$$\text{min}\left\{\frac{1}{N}\sum _{k=1}^{N}L\left(y_k,w^{\top }{x}_k-b\right)+\lambda {\Vert w\Vert }_1:(w,b)\in {ℝ}^d\times ℝ\right\}.$$ (3.14)

To close this section, we summarize the pros and cons of using RKHSs for machine learning. The pros include:

• The canonical inner product of an RKHS provides a convenient approach to defining a measure for comparison which is an inevitable operation in machine learning.

• A representer theorem of a machine learning method in an RKHS reduces an infinite dimensional problem to a finite dimensional problem, with a canonical finite dimensional space determined by the training data.

• Universal kernels guarantee the approximation property.

The cons are mainly on the following aspect:

• Learning methods in an RKHS result in dense solutions. It is computationally expensive to use dense solutions in prediction and other practical applications.

4. Reproducing Kernel Banach Spaces for Machine Learning

Discussion in the previous section regarding the denseness of a learned solution in a Hilbert space leads us to explore Banach spaces as hypothesis spaces for machine learning hoping to gain sparsity for a learning solution in the spaces. This is because the class of Banach spaces which includes Hilbert spaces as special cases offers more choices for a hypothesis space for a machine learning method. In particular, certain Banach spaces with special geometric features may lead to sparse learning solutions. Aiming at developing sparse learning methods, the notion of the RKBS was first introduced in [58] in 2009. In this section, we review briefly the development of the RKBS during the last decade.

As pointed out earlier, the aim of most of machine learning methods is to construct functions whose values can be used for prediction or other decision making purposes. For this reason, it would be desirable to consider a Banach space of functions defined over a prescribed set as our hypothesis space for learning. A Banach space $ℬ$ is called a space of functions on a prescribed set $\mathbf{X}$ if $ℬ$ is composed of functions defined on $\mathbf{X}$ and for each $f\in ℬ$, ${\parallel f\parallel }_{ℬ}=0$ implies that $f\left(\mathbf{x}\right)=0$ for all $\mathbf{x}\in \mathbf{X}$. This implies that in a Banach space $ℬ$ of functions, the pointwise function evaluation $f\left(\mathbf{x}\right)$ must be well-defined for all $\mathbf{x}\in \mathbf{X}$ and all $f\in ℬ$. The standard $L_p\left(\mathbf{X}\right)$ spaces are Banach function spaces, but not Banach spaces of functions, since their elements are equivalent classes rather than functions, and the pointwise function evaluation is not defined in these spaces. Because function values are crucial in decision making, we would expect that they are stable with respect to functions chosen in the space. For this purpose, we would prefer the point-evaluation functionals ${\delta }_{\mathbf{x}}:ℬ\to ℝ$ defined by

$${\delta }_x(f)≔f(x),\hspace{1em}\text{for all}x\in X$$

being continuous with respect to functions in the space $ℬ$. This is exactly the way in which the RKHS was defined [1]. This consideration gives rise to the following definition of the RKBS.

Definition 4.1 A Banach space $ℬ$ of functions defined on a prescribed set $\mathbf{X}$ is called an RKBS if point-evaluation functionals ${\delta }_{\mathbf{x}}$, for all $\mathbf{x}\in \mathbf{X}$, are continuous on $ℬ$, that is, for each $\mathbf{x}\in \mathbf{X}$ there exists a constant $c_{\mathbf{x}}>0$ such that

$$\left|{\delta }_x(f)\right|\le c_x{\Vert f\Vert }_{ℬ},\hspace{1em}forallf\in ℬ.$$

The original definition of an RKBS was given in Definition 1 of [59] in a somewhat restricted version, where $ℬ$ was assumed to reflexive, see [27] for further generalization. In Definition 4.1, we do not have the restriction. It follows from Definition 4.1 that if $ℬ$ is an RKBS on $\mathbf{X}$, and $f$, $f_n\in ℬ$, for $n\in \mathbb{N}$, then ${\parallel f_n-f\parallel }_{ℬ}\to 0$ implies $f_n\left(\mathbf{x}\right)\to f\left(\mathbf{x}\right)$, as $n\to \infty$, for each $\mathbf{x}\in \mathbf{X}$.

In the RKHS setting, due to the well-known Riesz representation theorem, one can identify a function in the same space to represent each point-evaluation functional. That is, a Hilbert space is isometrically isomorphic to its dual space, by which we refer to the space of all continuous linear functionals on the Hilbert space. This naturally leads to the unique reproducing kernel associated with the RKHS. The kernel in conjunction with the inner product of the Hilbert space yields the reproducing property. However, in general, a Banach space is not isometrically isomorphic to its dual space. Moreover, continuity of the point-evaluation functionals does not guarantee the existence of a kernel. For this reason, we need to pay special attention to the notion of the reproducing kernel for an RKBS.

For a Banach space $ℬ$, we denote by ${ℬ}^{*}$ its dual space, the space of all continuous linear functionals on $ℬ$. It is well-known that the dual space of a Banach space is again a Banach space. Definition 4.1 of the RKBS ensures that when $ℬ$ is an RKBS on $\mathbf{X}$, the point-evaluation functionals

$${\delta }_{\text{x}}\in {ℬ}^{*},\hspace{1em}\text{for all}\mathbf{\text{x}}\in X\text{. }$$ (4.1)

When $ℬ$ is an RKBS on $\mathbf{X}$, we let ${ℬ}^{\prime }$ denote the completion of the linear span $\widetilde{ℬ}$ of all the point-evaluation functionals ${\delta }_{\mathbf{x}}$ on $ℬ$ for $\mathbf{x}\in \mathbf{X}$, under the norm of ${ℬ}^{*}$. It follows from (4.1) that $\widetilde{ℬ}\subseteq {ℬ}^{*}$. Thus, ${ℬ}^{\prime }\subseteq {ℬ}^{*}$ since ${ℬ}^{*}$ is complete. Moreover, we have that ${ℬ}^{\prime }$ is the smallest Banach space that contains all point-evaluation functionals on $ℬ$. We will call ${ℬ}^{\prime }$ the $\delta$-dual space of $ℬ$. Because in machine learning, we are interested in Banach spaces of functions, we further suppose that the $\delta$-dual space ${ℬ}^{\prime }$ is isometrically isomorphic to a Banach space ${ℬ}^{\#}$ of functions on a set ${\mathbf{X}}^{\prime }$. This hypothesis is satisfied for most examples that we encounter in application. For instance, the space that we will discuss later in this section has this property. In the rest of this paper, we will not distinguish ${ℬ}^{\prime }$ and ${ℬ}^{\#}$.

For a Banach space $ℬ$ of functions defined on the set $\mathbf{X}$, we let ${\langle ·,·\rangle }_{ℬ\times {ℬ}^{\prime }}$ denote the dual bilinear form on $ℬ\times {ℬ}^{\prime }$ induced by restriction of the dual bilinear form on $ℬ\times {ℬ}^{*}$ to $ℬ\times {ℬ}^{\prime }$. When there is no ambiguity, we will write it as ${\langle ·,·\rangle }_{ℬ}$. We now define a reproducing kernel, which provides a closed-form function representation for the point-evaluation functionals, for an RKBS.

Definition 4.2 Let $ℬ$ be an RKBS on a set $\mathbf{X}$ with the δ-dual space ${ℬ}^{\prime }$. Suppose that ${ℬ}^{\prime }$ is isometrically isomorphic to a Banach space of functions on a set ${\mathbf{X}}^{\prime }$. A function $K:\mathbf{X}\times {\mathbf{X}}^{\prime }\to ℝ$ is called a reproducing kernel for $ℬ$ if $K(\mathbf{x},·)\in {ℬ}^{\prime }$ for all $\mathbf{x}\in \mathbf{X}$, and

$$f(\mathbf{\text{x}})={\langle f(\cdot ),K(\mathbf{\text{x}},\cdot )\rangle }_{ℬ},\hspace{1em}\mathit{\text{for all}}f\in ℬ\text{.}$$ (4.2)

If in addition ${ℬ}^{\prime }$ is an RKBS on ${\mathbf{X}}^{\prime }$, $K(·,y)\in ℬ$ for all $\mathbf{y}\in {\mathbf{X}}^{\prime }$, and

$$g(y)={\langle K(\cdot ,y),g(\cdot )\rangle }_{ℬ},\hspace{1em}\text{for all}g\in {ℬ}^{\prime },$$ (4.3)

we call ${ℬ}^{\prime }$ an adjoint RKBS of $ℬ$. In this case, the function

$$K^{\prime }(x,y)≔K(y,x),\text{ for }x\in X^{\prime },y\in X,$$

is a reproducing kernel for ${ℬ}^{\prime }$, and we call $ℬ$, ${ℬ}^{\prime }$ a pair of RKBSs.

The original definition of a reproducing kernel appeared in [58] with a choice ${ℬ}^{\prime }={ℬ}^{*}$, which was extended in [55]. Also, see [46, 49] for further development.

Equation (4.2) furnishes a representation of the point-evaluation functional in terms of the kernel and the dual bilinear form. The present form in Definition 4.2 of a reproducing kernel for an RKBS is slightly different from the various forms known in the literature. We feel that the form described in Definition 4.2 better captures the essence of reproducing kernels. Unlike a reproducing kernel for an RKHS, a reproducing kernel for an RKBS is not necessarily symmetric or positive semi-definite. In a special case when an RKBS satisfies the following hypothesis, we can establish the positive semi-definiteness of its kernel.

Hypothesis (H1): Spaces $ℬ$, ${ℬ}^{\prime }$ are a pair of RKBSs on a set $\mathbf{X}$, the $\delta$-dual space ${ℬ}^{\prime }$ is isometrically isomorphic to a Banach space of functions on $\mathbf{X}$, and a reproducing kernel $K:X\times X\to ℝ$ for $ℬ$ is symmetric.

Suppose that Hypothesis (H1) is satisfied. By symmetry of the kernel $K$, we have that $K(·,\mathbf{x})\in {ℬ}^{\prime }$ for all $\mathbf{x}\in \mathbf{X}$ and $K(\mathbf{y},·)\in ℬ$ for all $\mathbf{y}\in \mathbf{X}$. In this case, ${\langle K(·,\mathbf{x}),K(·,\mathbf{x})\rangle }_{ℬ\times {ℬ}^{\prime }}$ is well-defined for $\mathbf{x}\in \mathbf{X}$. We further require that the following hypothesis be satisfied.

Hypothesis (H2): There exists a positive constant $A$ such that

$$\frac{{\langle f_n,f_n\rangle }_{ℬ\times {ℬ}^{\prime }}}{{\Vert f_n\Vert }_{ℬ}^2}\ge A,\hspace{1em}\text{for all}{f}_n≔\sum _{i=1}^n{c}_{i}K\left(\cdot ,x_i\right)\ne 0,n\in \mathbb{N},x_i\in X,c_i\in ℝ,i=1,2,\dots ,n.$$ (4.4)

In a loose sense, inequality (4.4) has a geometric interpretation: “cosine” of the “angle” formed by $\frac{{\langle f_n,f_n\rangle }_{ℬ\times {ℬ}^{\prime }}}{{\Vert f_n\Vert }_{ℬ}^2}$ is uniformly above zero, for all $f_n\ne 0$, $n\in \mathbb{N}$, ${\mathbf{x}}_i\in \mathbf{X}$, $c_i\in ℝ$, $i=1,2,\dots ,n$. In other words, the spaces $ℬ$ and ${ℬ}^{\prime }$ cannot be “perpendicular” to each other, with respect to the dual bilinear form ${\langle ·,·\rangle }_{ℬ\times {ℬ}^{\prime }}$. When $ℬ$ is a Hilbert space, inequality (4.4) reduces to an equality with $A=1$, since in such a case $ℬ={ℬ}^{\prime }$. Under Hypotheses (H1) and (H2), we can establish the positive semi-definiteness of a reproducing kernel.

Theorem 4.3 Suppose that a pair $ℬ$, ${ℬ}^{\prime }$ of RKBSs satisfy Hypotheses (H1) with a reproducing kernel $K:X\times X\to ℝ$ and (H2). For all $n\in \mathbb{N}$, all ${\mathbf{x}}_j\in \mathbf{X}$, $j=1,2,\dots ,n$, let

$$K_n≔\left[K\left(x_i,x_j\right):i,j=1,2,\dots ,n\right].$$ (4.5)

Then, for all $n\in \mathbb{N}$ and all ${\mathbf{x}}_j\in \mathbf{X}$, $j=1,2,\dots ,n$, the matrices $K_n$ are positive semi-definite.

Proof: It suffices to show for all $c≔{\left[c_1,c_2,\dots ,c_n\right]}^{\top }\in {ℝ}^n$ that ${\mathbf{c}}^{\top }{K}_n\mathbf{c}\ge 0$. For $\mathbf{y}\in \mathbf{X}$, we introduce the notation $k_{\mathbf{y}}≔K(·,y)$. By the reproducing property (4.2) and symmetry of the kernel $K$, we observe that

$$\begin{gathered} K\left(x_i,x_j\right)=k_{x_j}\left(x_i\right) \\ ={\langle k_{x_j}(\cdot ),K\left(x_i,\cdot \right)\rangle }_{ℬ} \\ ={\langle K\left(\cdot ,x_j\right),K\left(x_i,\cdot \right)\rangle }_{ℬ} \\ ={\langle K\left(\cdot ,x_j\right),K\left(\cdot ,x_i\right)\rangle }_{ℬ}. \end{gathered}$$

Therefore, according to Hypotheses (H1) and (H2), for all $c≔{\left[c_1,c_2,\dots ,c_n\right]}^{\top }\in {ℝ}^n$, we obtain that

$$\begin{gathered} c^{\top }{K}_{n}c=\sum _{i=1}^n\sum _{j=1}^n{c}_i{c}_{j}K\left(x_i,x_j\right) \\ =\sum _{i=1}^n\sum _{j=1}^n{c}_i{c}_j{\langle K\left(\cdot ,x_j\right),K\left(\cdot ,x_i\right)\rangle }_{ℬ} \\ ={\langle \sum _{i=1}^n{c}_{i}K\left(\cdot ,x_i\right),\sum _{i=1}^n{c}_{i}K\left(\cdot ,x_i\right)\rangle }_{ℬ} \\ \ge A{\Vert \sum _{i=1}^n{c}_{i}K\left(\cdot ,x_i\right)\Vert }_{ℬ}^2\ge 0, \end{gathered}$$

proving the positive semi-definiteness of $K_n$. □

In general, the kernel matrix (4.5) is not well-defined unless $\mathbf{X}={\mathbf{X}}^{\prime }$.

We next illustrate the notion of the RKBS and its kernel with an example. For this purpose, we consider the space ${\ell }_1(\mathbb{N})$ of real sequences $\mathbf{f}≔[f_1,f_2,\dots ]$ such that ${\Vert f\Vert }_1≔{\sum }_{k\in \mathbb{N}}\left|f_k\right|<\infty$. Elements of ${\ell }_1(\mathbb{N})$ are functions defined on the set $X≔\mathbb{N}$. It is well-known that ${\ell }_1(\mathbb{N})$ is a Banach space, and thus, it is a Banach space of functions on $\mathbb{N}$. For each $j\in \mathbb{N}$, the point-evaluation functional ${\delta }_j$ has the form

$${\delta }_j(f)=f_j.$$ (4.6)

We first establish that ${\ell }_1(\mathbb{N})$ is an RKBS, which is stated in the next theorem.

Theorem 4.4 The space ${\ell }_1(\mathbb{N})$ is an RKBS on $\mathbb{N}$.

Proof: It suffices to verify that the point-evaluation functionals on the space ${\ell }_1(\mathbb{N})$ are continuous. All point-evaluation functionals on space ${\ell }_1(\mathbb{N})$ are ${\delta }_j$, for all $j\in \mathbb{N}$. Hence, for all $j\in \mathbb{N}$, it follows from (4.6) that

$$\left|{\delta }_j(f)\right|=\left|f_j\right|\le {\Vert f\Vert }_1,\hspace{1em}\text{for all}f\in {\ell }_1(\mathbb{N}).$$

This ensures that for all $j\in \mathbb{N}$, the point-evaluation functionals ${\delta }_j$ are all continuous on the space ${\ell }_1(\mathbb{N})$. By Definition 4.1, the space ${\ell }_1(\mathbb{N})$ is an RKBS over $\mathbb{N}$. □

We next identify a reproducing kernel for the RKBS ${\ell }_1(\mathbb{N})$. By ${\ell }_{\infty }(\mathbb{N})$ we denote the Banach space of real bounded sequences on $\mathbb{N}$ under the supremum norm. Namely, for any $a≔\left[a_1,a_2,\dots \right]\in {\ell }_{\infty }(\mathbb{N})$, we have that ${\Vert a\Vert }_{\infty }≔\text{sup}\left\{\left|a_k\right|:k\in \mathbb{N}\right\}<\infty$. We further denote by $c_0(\mathbb{N})$ the set of real sequences that are convergent to zero in the sense that for all $a≔\left[a_1,a_2,\dots \right]\in c_0(\mathbb{N})$, there holds ${\text{lim}}_{k\to \infty }{𝑎}_k=0$. The set $c_0(\mathbb{N})$ is a Banach space of real convergent sequences under the supremum norm defined on $\mathbb{N}$. Thus, for all $a\in c_0(\mathbb{N})$, there holds that ${\Vert a\Vert }_{\infty }<+\infty$. This ensures that $c_0(\mathbb{N})\subset {\ell }_{\infty }(\mathbb{N})$. Moreover, since a nonzero constant sequence is in ${\ell }_{\infty }\left(\mathbb{N}\right)$ but not in $c_0\left(\mathbb{N}\right)$, $c_0\left(\mathbb{N}\right)$ is a true subspace of ${\ell }_{\infty }\left(\mathbb{N}\right)$. It is well-known [37] that $c_0^{*}\left(\mathbb{N}\right)={\ell }_1\left(\mathbb{N}\right)$ and ${\ell }_1^{*}\left(\mathbb{N}\right)={\ell }_{\infty }\left(\mathbb{N}\right)$ It follows that ${\left({\ell }_1^{*}\left(\mathbb{N}\right)\right)}^{*}={\left({\ell }_{\infty }\left(\mathbb{N}\right)\right)}^{*}\subset c_0^{*}\left(\mathbb{N}\right)={\ell }_1\left(\mathbb{N}\right)$. Thus, ${\left({\ell }_1^{*}\left(\mathbb{N}\right)\right)}^{*}\ne {\ell }_1\left(\mathbb{N}\right)$. That is, the Banach space ${\ell }_1\left(\mathbb{N}\right)$ is not reflexive and its predual is $c_0\left(\mathbb{N}\right)$.

The dual bilinear form on ${\ell }_1\left(\mathbb{N}\right)\times {\ell }_{\infty }\left(\mathbb{N}\right)$ has a concrete form. Specifically, for any $f\in {\ell }_1\left(\mathbb{N}\right)$ and $a\in {\ell }_{\infty }\left(\mathbb{N}\right)$, we define

$${\langle f,a\rangle }_{{\ell }_1(\mathbb{N})}:=\sum _{k\in \mathbb{N}}{a}_k{f}_k.$$ (4.7)

Restricting the definition (4.7) to the subspace $c_0\left(\mathbb{N}\right)$ of ${\ell }_{\infty }\left(\mathbb{N}\right)$ yields the dual bilinear form on ${\ell }_1\left(\mathbb{N}\right)\times c_0\left(\mathbb{N}\right)$. It follows from (4.7) for any $f\in {\ell }_1\left(\mathbb{N}\right)$ and $a\in c_0\left(\mathbb{N}\right)$ that ${\langle f,a\rangle }_{{\ell }_1(\mathbb{N})}={\langle a,f\rangle }_{c_0(\mathbb{N})}$. For this reason, we will drop the subscript and simply write ${\langle f,a\rangle }_{{\ell }_1(\mathbb{N})}=\langle f,a\rangle$ and ${\langle a,f\rangle }_{c_0(\mathbb{N})}=\langle a,f\rangle$. Moreover, since $\langle \mathbf{f},\mathbf{a}\rangle =\langle \mathbf{a},\mathbf{f}\rangle$ according to (4.7), we will use $\langle \mathbf{f},\mathbf{a}\rangle$ and $\langle \mathbf{a},\mathbf{f}\rangle$ interchangeably when no ambiguity may raise.

Proposition 4.5 If ${\ell }_1{\left(\mathbb{N}\right)}^{\prime }$ denotes the δ-dual space of ${\ell }_1\left(\mathbb{N}\right)$, then ${\ell }_1{\left(\mathbb{N}\right)}^{\prime }=c_0\left(\mathbb{N}\right)$.

Proof: We first show that $c_0\left(\mathbb{N}\right)\subseteq {\ell }_1{\left(\mathbb{N}\right)}^{\prime }$. Let $a≔\left[a_1,a_2,\dots \right]\in c_0\left(\mathbb{N}\right)$ be arbitrary. For all $f≔\left[f_1,f_2,\dots \right]\in {\ell }_1\left(\mathbb{N}\right)$, we have that

$$\langle a,f\rangle =\sum _{j\in \mathbb{N}}{a}_j{f}_j=\sum _{j\in \mathbb{N}}{a}_j{\delta }_j(f).$$

This implies that $a={\sum }_{j\in \mathbb{N}}{a}_j{\delta }_j$. Namely, $\mathbf{a}$ is a linear combination of the point-evaluation functionals ${\delta }_j$, for $j\in \mathbb{N}$. Hence, $c_0\left(\mathbb{N}\right)\subseteq {\ell }_1{\left(\mathbb{N}\right)}^{\prime }$.

Conversely, we establish that ${\ell }_1{\left(\mathbb{N}\right)}^{\prime }\subseteq c_0\left(\mathbb{N}\right)$. Let $a\in {\ell }_1{\left(\mathbb{N}\right)}^{\prime }$ be arbitrary. Since ${\ell }_1{\left(\mathbb{N}\right)}^{\prime }$ is the completion of the linear span of all the point-evaluation functionals ${\delta }_j$ on ${\ell }_1\left(\mathbb{N}\right)$ for $j\in \mathbb{N}$, under the norm of ${\ell }_{\infty }\left(\mathbb{N}\right)$, there exist $k_n^j\in \mathbb{N}$, for $j=1,2,\dots ,K_n$, $n\in \mathbb{N}$ such that sequences $a_n≔{\sum }_{j=1}^{K_n}{\gamma }_{k_n^j}{\delta }_{k_n^j}$ satisfies

$$\underset{n\to \infty }{\text{lim}}{\Vert a-a_n\Vert }_{\infty }=0.$$ (4.8)

Clearly, $a_n\in c_0\left(\mathbb{N}\right)$ for all $n\in \mathbb{N}$. Since $c_0\left(\mathbb{N}\right)$ is a Banach space, equation (4.8) ensures that $a\in c_0\left(\mathbb{N}\right)$, which implies that ${\ell }_1{\left(\mathbb{N}\right)}^{\prime }\subseteq c_0\left(\mathbb{N}\right)$. □

Proposition 4.5 guarantees that ${\ell }_1{\left(\mathbb{N}\right)}^{\prime }\subseteq c_0\left(\mathbb{N}\right)$, which allows us to identify a reproducing kernel for the RKBS ${\ell }_1\left(\mathbb{N}\right)$. To this end, we define a function $K:\mathbb{N}\times \mathbb{N}\to ℝ$, which is a semi-infinite matrix. Specifically, for $i,j\in \mathbb{N}$, we let ${\delta }_{i,j}≔1$ if $i=j$ and 0 otherwise. We then define

$$K≔\left[K_{i,j}:i,j\in \mathbb{N}\right]\text{ where }{K}_{i,j}≔{\delta }_{i,j},\text{ for }i,j\in \mathbb{N}\text{. }$$ (4.9)

Theorem 4.6 The function $K≔\left[K_{i,j}:i,j\in \mathbb{N}\right]$ defined by (4.9) is a reproducing kernel for RKBS ${\ell }_1\left(\mathbb{N}\right)$.

Proof: According to Proposition 4.5, we have that ${\ell }_1{\left(\mathbb{N}\right)}^{\prime }=c_0\left(\mathbb{N}\right)$, which is a Banach space of functions on $X^{\prime }≔\mathbb{N}$. By the definition (4.9) of the function $K$, we find that $K\left(i,\cdot \right)=K_{i,\cdot }={\delta }_{i,\cdot }\in c_0\left(\mathbb{N}\right)$, for all $i\in \mathbb{N}$. Moreover, for all $f≔\left[f_1,f_2,\dots \right]\in {\ell }_1\left(\mathbb{N}\right)$, we have for all $i\in \mathbb{N}$ that

$$f_i=\sum _{j\in \mathbb{N}}{\delta }_{i,j}{f}_j=\sum _{j\in \mathbb{N}}K\left(i,j\right)f_j=\langle K\left(i,\cdot \right),f\rangle .$$

This proves the reproducing property. By Definition 4.2 with $ℬ≔{\ell }_1\left(\mathbb{N}\right)$ and ${ℬ}^{\prime }≔c_0\left(\mathbb{N}\right)$, we conclude that the function $K$ is a reproducing kernel for ${\ell }_1\left(\mathbb{N}\right)$. □

Note that Theorem 4.6 gives the simplest example of the RKKS, since any principal (and finite) sub-matrix of the (infinite operator) $K$ is the identity matrix. This result is not covered by the setting described in [58]. In the same manner, we consider the predual space $c_0\left(\mathbb{N}\right)$.

Theorem 4.7 The space $c_0\left(\mathbb{N}\right)$ is an RKBS on $\mathbb{N}$.

Proof: We establish that the point-evaluation functionals on $c_0\left(\mathbb{N}\right)$ are continuous. All point-evaluation functionals on space $c_0\left(\mathbb{N}\right)$ are ${\delta }_j$, for all $j\in \mathbb{N}$. Hence, we observe for all $a\in c_0\left(\mathbb{N}\right)$ that

$$\left|{\delta }_j(a)\right|=\left|a_j\right|\le {\Vert a\Vert }_{\infty },\hspace{1em}\text{for all}j\in \mathbb{N}.$$

This ensures that the point-evaluation functionals on $c_0\left(\mathbb{N}\right)$ are continuous. Again, by Definition 4.1, $c_0\left(\mathbb{N}\right)$ is an RKBS over $\mathbb{N}$. □

We next identify a reproducing kernel for the RKBS $c_0\left(\mathbb{N}\right)$ of functions defined on $X^{\prime }≔\mathbb{N}$.

Theorem 4.8 The function $K≔\left[K_{i,j}:i,j\in \mathbb{N}\right]$ defined by (4.9) is a reproducing kernel for $c_0\left(\mathbb{N}\right)$. The space $c_0\left(\mathbb{N}\right)$ is an adjoint RKBS of ${\ell }_1\left(\mathbb{N}\right)$, the spaces ${\ell }_1\left(\mathbb{N}\right)$, $c_0\left(\mathbb{N}\right)$ form a pair of RKBSs, and the kernel $K$ is symmetric and positive semi-definite.

Proof: By Theorem 4.7, $c_0\left(\mathbb{N}\right)$ is an RKBS on $X^{\prime }≔\mathbb{N}$. Furthermore, we have that $K\left(\cdot ,j\right)≔K_{\cdot ,j}={\delta }_{\cdot ,j}\in {\ell }_1\left(\mathbb{N}\right)$, for all $j\in \mathbb{N}$. Moreover, we have the dual reproducing property that for all $a\in c_0\left(\mathbb{N}\right)$,

$$a_j=\sum _{i\in \mathbb{N}}{\delta }_{i,j}{a}_i=\sum _{i\in \mathbb{N}}K(i,j)a_i=\langle K(\cdot ,j),a\rangle .$$

It follows from Definition 4.2 that the function $K$ defined by (4.9) is a reproducing kernel for the RKBS $c_0\left(\mathbb{N}\right)$. Therefore, $c_0\left(\mathbb{N}\right)$ is an adjoint RKBS of ${\ell }_1\left(\mathbb{N}\right)$.

Clearly, spaces ${\ell }_1\left(\mathbb{N}\right)$, $c_0\left(\mathbb{N}\right)$ with the kernel $K$ satisfy Hypotheses (H1)-(H2). The positive definiteness of $K$ can be seen immediately since for any $n\in \mathbb{N}$, $x_i\in \mathbb{N}$, $i=1,2,\dots ,n$, the kernel matrix $K_n≔\left[K\right({\mathbf{x}}_i,{\mathbf{x}}_j):i,j=1,2,\dots ,n]$ is the identity matrix. □

Several nontrivial RKBSs may be found in [46, 53, 55, 58]. In particular, an RKBS isometrically isomorphic to the space ${\ell }_1\left(\mathbb{N}\right)$ was constructed in [46]. We now briefly describe a RKBS on a continuous function space. We choose $\mathbf{X}$ as a locally compact Hausdorff space. By $C_0\left(\mathbf{X}\right)$ we denote the space of all continuous functions $f:X\to ℝ$ such that for each $ϵ>0$, the set $\{\mathbf{x}\in \mathbf{X}:|f\left(\mathbf{x}\right)|\ge ϵ\}$ is compact. We equip the maximum norm on $C_0\left(\mathbf{X}\right)$, that is,

$${\Vert f\Vert }_{\infty }≔\underset{x\in X}{\text{sup}}|f(x)|,\hspace{1em}\text{for all}f\in C_0(X).$$

The Riesz-Markov representation theorem states that the dual space of $C_0\left(\mathbf{X}\right)$ is isometrically isomorphic to the space $𝔐\left(\mathbf{X}\right)$ of the real-valued regular Borel measures on $\mathbf{X}$ endowed with the total variation norm ${\Vert ·\Vert }_{\text{TV}}$. We suppose that $K:X\times X\to ℝ$ satisfies $K(\mathbf{x},·)\in C_0\left(\mathbf{X}\right)$ for all $\mathbf{x}\in \mathbf{X}$ and the density condition $\overline{\text{span}}\left\{K\left(x,\cdot \right):x\in X\right\}=C_0\left(X\right)$. We then introduce the space of functions on $\mathbf{X}$ by

$$ℬ≔\left\{f_{\mu }≔{\int }_{X}K(\cdot ,x)d\mu (x):\mu \in \mathfrak{M}(X)\right\}$$ (4.10)

equipped with ${\Vert f_{\mu }\Vert }_{ℬ}≔{\Vert \mu \Vert }_{\text{TV}}$. Clearly, $ℬ$ defined by (4.10) is a Banach space of functions defined on $\mathbf{X}$. It was established in [53] that $ℬ$ is an RKBS, the $\delta$-dual ${ℬ}^{\prime }$ of $ℬ$ is isometrically isomorphic to the space $C_0\left(\mathbf{X}\right)$. Moreover, the function $K$ is a reproducing kernel for the RKBS $ℬ$ in the sense of Definition 4.2. More information about the space $ℬ$ may be found in [3, 27, 47, 53].

Last decade has witnessed the rapid development of the theory of the RKBS and its applications since the publication of the original paper [58]. Continued understanding proper definition and construction of the RKBS and related theoretical issues remains research topics of great interest [18, 20, 22, 45, 46, 50, 51, 55, 60, 61, 62]. In particular, construction of RKBSs was proposed in [55] by using Mercer’s kernels. A class of vector-valued reproducing kernel Banach spaces with the ${\ell }_1$ norm was constructed in [26] and it was used in multi-task learning. Separability of RKBSs was studied in [34]. Statistical margin error bounds were given in [8] for $L_1$-norm support vector machines. Lipschitz implicit function theorems in RKBSs were established in [43]. A converse sampling theorem in RKBSs was given in [7]. The notion of the RKBS has been used in statistics [39, 44], game theory [48], and machine learning [3, 11, 17, 25, 35, 52].

5. Learning in a Banach Space

Most of learning methods may be formulated as a minimum norm interpolation problem or a regularization problem. We consider both of the cases in this section. We will focus on solution representation of these learning problems.

Recently, representer theorems of learning methods in Banach spaces received considerable attention. In the framework of a semi-inner-product RKBS [58], the representer theorem was derived from the dual elements and the semi-inner-product [58, 61]. In [55], for a reflexive and smooth RKBS, a representer theorem of a solution of the regularized learning problem was established using the Gâteaux derivative of the norm function and the reproducing kernel. In addition, the representer theorem was generalized to a non-reflexive and non-smooth Banach space which has a predual space [22, 50, 51, 52]. Due to lack of the Gâteaux derivative, other tools such as the subdifferential of the norm function and the duality mapping need to be used to describe the representer theorem.

In this section, we assume that $ℬ$ is a general Banach space with the dual space ${ℬ}^{*}$ or predual space ${ℬ}_{*}$ unless it is stated otherwise. We first study the minimum norm interpolation problem. Given $y≔\left[y_1,y_2,\dots ,y_m\right]\in {ℝ}^m$ and ${\nu }_j\in {ℬ}^{*}$, $j=1,2,\dots ,m$, we seek $f^{*}\in ℬ$ such that

$${\Vert f^{*}\Vert }_{ℬ}=\text{inf}\left\{{\Vert f\Vert }_{ℬ}:f\in ℬ,{\langle {\nu }_j,f\rangle }_{ℬ}=y_j,j=1,2,\dots ,m\right\}.$$ (5.1)

While minimum norm interpolation in a Hilbert space is a classical topic [14, 15], its counterpart in a Banach space has drawn much attention in the literature [9, 10, 30, 50, 52] due to its connection with compressed sensing [6, 16]. An existence theorem of a solution of the minimum norm interpolation problem was established in Proposition 1 of [52].

Minimum norm interpolation problem (5.1) with $ℬ≔{\ell }_1\left(\mathbb{N}\right)$ was investigated in [10], which may be stated as follows: Given $y_j\in ℝ$ and $a_j\in c_0\left(\mathbb{N}\right)$, $j=1,2,\dots ,m$, we seek $x_1^{*}\in {\ell }_1\left(\mathbb{N}\right)$ such that

$${\Vert x_1^{*}\Vert }_1=\text{inf}\left\{{\Vert x\Vert }_1:\mathbf{\text{x}}\in {\ell }_1(\mathbb{N}),\langle a_j,x\rangle =y_j,j=1,2,\dots ,m\right\}.$$ (5.2)

Problem (5.2) is an infinite dimensional version of compressed sensing considered in [6, 16]. According to [10], it may be reduced to a finite dimensional linear programming problem by a duality approach, leading to a sparse solution. Sparse learning methods in ${\ell }_1\left(\mathbb{N}\right)$ were studied in [2, 10, 50, 52].

The solution of problem (5.2) with interpolation conditions (3.12) has a sparse solution, which is given by $x_1^{*}=\left(\frac{7}{2},-1,0,0,\dots \right)$ (see, [10] for details about this example). This solution has only two nonzero components while the solution of the minimum ${\ell }_2$ norm interpolation problem (3.11) with the same interpolation conditions has infinite numbers of nonzero components. The solution $x_1^{*}$ is sparse because the Banach space ${\ell }_1\left(\mathbb{N}\right)$ promotes sparsity while the solution ${\mathbf{x}}^{*}$, given by (3.13), of problem (3.11) is dense because the Hilbert space ${\ell }_2\left(\mathbb{N}\right)$ does not promote sparsity. Several forms of the representer theorem for a solution of problem (5.1) were established in [52].

We now turn to investigating the regularized learning problem in a Banach space. Such a problem may be formulated from the ill-posed problem

$$ℒ(f)=y,$$ (5.3)

where $\mathbf{y}$ is given data and $ℒ$ represents either a physical system or a learning system. We define a data fidelity term ${𝒬}_{\mathbf{y}}\left(ℒ\right(f\left)\right)$ from (5.3) by using a loss function ${𝒬}_{\text{y}}:{ℝ}^m\to {ℝ}_{+}$ and choose a regularization term ${\lambda }_{\phi }\left({\parallel f\parallel }_{ℬ}\right)$ with a regularizer $\phi :{ℝ}_{+}\to {ℝ}_{+}$. We then solve the regularization problem

$$\text{inf}\left\{{𝒬}_y(ℒ(f))+\lambda \phi \left({\Vert f\Vert }_{ℬ}\right):f\in ℬ\right\},$$ (5.4)

where $\lambda$ is a positive regularization parameter. Here, ${𝒬}_{\mathbf{y}}\left(u\right)$, for $u\in {ℝ}^m$, measures the “lost” of $\mathbf{u}$ from given data $\mathbf{y}$. For example, one may choose ${𝒬}_{\mathbf{y}}\left(u\right)≔{\parallel \mathbf{u}-\mathbf{y}\parallel }_2$. Regularized learning problems in Banach spaces were originated in [30] and since then, desired representer theorems for such learning problems have received considerable attention in the literature. For existence results of a solution of problem (5.4), the readers are referred to [52].

When the operator $ℒ:ℬ\to {ℝ}^m$ in (5.3) is defined by

$$ℒ\left(f\right)≔{\langle {\nu }_j,f\rangle }_{ℬ}\text{, for }j=1,2,\dots ,m\text{, }$$ (5.5)

the regularization problem (5.4) is intimately related to the minimum norm interpolation problem (5.1). Their relation is described in the next proposition.

Proposition 5.1 Suppose that $ℬ$ is a Banach space with the dual space ${ℬ}^{*}$, ${\nu }_j\in {ℬ}^{*}$, $j\in {\mathbb{N}}_m$ and $ℒ$ is defined by (5.5). For a given $y_0\in {ℝ}^m$, let ${𝒬}_{y_0}:{ℝ}^m\to {ℝ}_{+}$ be a loss function, $\phi :{ℝ}_{+}\to {ℝ}_{+}$ be an increasing regularizer and $\lambda >0$. Let $\widehat{f}\in ℬ$ be a solution of the regularization problem (5.4) with $\mathbf{y}≔{\mathbf{y}}_0$. Then the following statements hold true:

1. A solution $\widehat{g}\in ℬ$ of problem (5.1) with $y≔ℒ\left(\widehat{f}\right)$ is a solution of the regularization problem (5.4) with $\mathbf{y}≔{\mathbf{y}}_0$.

2. If $\phi$ is strictly increasing, then $\widehat{f}$ is a solution of problem (5.1) with $y≔ℒ\left(\widehat{f}\right)$.

Statement (ii) of Proposition 5.1 was claimed in [30] without details of proof. A complete proof for Proposition 5.1 may be found in [52].

We are interested in characterizing a solution of the regularization problem (5.4) with an operator $ℒ$ defined by (5.5) having an adjoint operator ${ℒ}^{*}:{ℝ}^m\to {ℬ}_{*}$. The following two representer theorems are due to [52]. For a given $y_0\in {ℝ}^m$, let ${𝒬}_{y_0}:{ℝ}^m\to {ℝ}_{+}$ be a loss function. We first consider the case when the Banach space $ℬ$ has a smooth predual.

Theorem 5.2 Suppose that $ℬ$ is a Banach space having the smooth predual space ${ℬ}_{*}$, and ${\nu }_j\in {ℬ}_{*}$, $j\in {\mathbb{N}}_m$. Let ${𝒢}_{*}\left(\nu \right)$ denote the Gâteaux derivative of the norm $\parallel ·\parallel {ℬ}^{*}$ at $\nu \in {ℬ}^{*}$. Let $\phi :{ℝ}_{+}\to {ℝ}_{+}$ be a strictly increasing regularizer and $\lambda >0$. Then

$$f_0≔{\Vert {ℒ}^{*}(\widehat{c})\Vert }_{{ℬ}_{*}}{𝒢}_{*}\left({ℒ}^{*}(\widehat{c})\right),\hspace{1em}\widehat{c}\in {ℝ}^m,$$ (5.6)

is a solution of the regularization problem (5.4) with $\mathbf{y}≔{\mathbf{y}}_0$ if and only if $\widehat{c}\in {ℝ}^m$ is a solution of the finite dimensional minimization problem

$$\text{inf}\{{𝒬}_{y_0}\text{(}{\Vert {ℒ}^{*}\left(c\right)\Vert }_{{ℬ}_{*}}ℒ\left({𝒢}_{*}\left({ℒ}^{*}\left(c\right)\right)\right)+\lambda \phi \left({\Vert {ℒ}^{*}\left(c\right)\Vert }_{{ℬ}_{*}}\right):c\in {ℝ}^m\}.$$ (5.7)

We now consider the case when the predual of the Banach space $ℬ$ is not necessarily smooth.

Theorem 5.3 Suppose that $ℬ$ is a Banach space having the predual space ${ℬ}_{*}$, ${\nu }_j\in {ℬ}^{*}$, $j\in {\mathbb{N}}_m$. Let $\phi :{ℝ}_{+}\to {ℝ}_{+}$ be a regularizer and $\lambda >0$.

1. If $\phi$ is increasing, then there exists a solution $f_0$ of the regularization problem (5.4) with $\mathbf{y}≔{\mathbf{y}}_0$ such that

   $$f_0\in \rho \partial {\Vert \cdot \Vert }_{{ℬ}_{*}}\left(\sum _{j\in {\mathbb{N}}_m}{c}_j{\nu }_j\right),$$ (5.8)

   for some $c_j\in ℝ$, $j\in {\mathbb{N}}_m$, with $\rho ≔{\Vert {\sum }_{j\in {\mathbb{N}}_m}{c}_j{\nu }_j\Vert }_{{ℬ}_{*}}$, where $\partial {\Vert \cdot \Vert }_{{ℬ}_{*}}$ denotes the subdifferential of the norm.

2. If $\phi$ is strictly increasing, then every solution $f_0$ of the regularization problem (5.4) with $\mathbf{y}≔{\mathbf{y}}_0$ satisfies (5.8) for some $c_j\in ℝ$, $j\in {\mathbb{N}}_m$.

Item (i) of Theorem 5.3 indicates that there exists a solution $f_0$ of the regularization problem (5.4) with $\mathbf{y}≔{\mathbf{y}}_0$ that satisfies (5.8), which is a generalization of the stationary point condition to minimization problems with non-differentiable objective function. The essence of Theorems 5.2 and 5.3 is that a solution $f_0$, defined by (5.6) or (5.8), of the infinite dimensional regularization problem (5.4) is determined completely by a finite number of parameters. Values of these parameters can be obtained by solving either a finite dimensional optimization problem (5.7) or a nonlinear system. When the space $ℬ$ is a Hilbert space, the nonlinear system reduces to a linear one. In particular, when $ℬ$ is an RKBS with a kernel $K$, the functionals ${\nu }_j$ in Theorems 5.2 and 5.3 have convenient representations in terms of the kernel $K$. We will demonstrate this point later in this section. These representer theorems serve as a base for further development of efficient numerical solvers of the regularization problem (5.4). One may find more discussions about representer theorems in Banach spaces in [41].

Not all Banach spaces will produce sparse learning solutions. Only Banach spaces with certain geometric features can lead to a sparse learning solution. It has been shown in [10, 50] that minimum norm interpolation and regularization problems in ${\ell }_1(\mathbb{N})$ have sparse solutions. As it is shown in Theorems 4.4 and 4.6, the space ${\ell }_1(\mathbb{N})$ is an RKBS on $\mathbb{N}$ with the kernel $K$ defined by (4.9). Hence, a learning solution in the RKBS ${\ell }_1(\mathbb{N})$ can be expressed in terms of the kernel $K$. Integrating Theorems 4.4 and 4.6 with those known in the literature [50], we have the following sparse representer theorem for a learning solution in ${\ell }_1(\mathbb{N})$. For any learning solution $f\in {\ell }_1\left(\mathbb{N}\right)$, there exists a vector of positive integers $\left[n_1,\dots ,n_q\right]\in {\mathbb{N}}^q$ with $q\le m$ such that

$$f=\sum _{i\in {\mathbb{N}}_q}{c}_{i}K\left(\cdot ,n_i\right).$$ (5.9)

Clearly, the solution $\mathbf{f}$ has only $q$ nonzero components. Here, the vector $\left[n_1,\dots ,n_q\right]\in {\mathbb{N}}^q$ represents the support of the learning solution $\mathbf{f}$ defined by (5.9) and the coefficients $c_i$ are determined by the fidelity data. The paper [10] suggests that we can first convert the minimum norm interpolation problem to its dual problem which allows us to identify the positive integer $q$ and the support $\left[n_1,\dots ,n_q\right]\in {\mathbb{N}}^q$, and then solve a linear system to obtain the coefficients $c_i$. The idea of [10] has been extended in [9] to solve a class of regularization problems.

We now return to the binary classification problem and describe a model based on an RKBS for it. We choose the RKBS $ℬ$ defined by (4.10) with $X={ℝ}^d$ and

$$K(x,y)≔\text{exp}\left(-\frac{{\Vert x-y\Vert }_2^2}{2{\mu }^2}\right),\hspace{1em}x,y\in {ℝ}^n,$$

as the hypothesis space. The classification problem in the RKBS $ℬ$ may be described by

$$\text{min}\left\{\frac{1}{n}\sum _{k=1}^{n}L\left(y_k,f\left(x_k\right)\right)+\lambda {\Vert f\Vert }_{ℬ}:f\in ℬ\right\},$$ (5.10)

where the fidelity term is the same as in problem (3.6). A representer theorem for a solution of the learning method (5.10) was obtained in [53]. To state it, for a function $f:{ℝ}^d\to ℝ$ with ${\parallel f\parallel }_{\infty }<\infty$, we define the set $𝒩(f)≔\left\{x\in {ℝ}^d:|f(x)|={\Vert f\Vert }_{\infty }\right\}$. We denote by $h^{*}$ a solution of minimization problem (5.10) and let $y≔\left[h^{*}\left(x_k\right):k\in {\mathbb{N}}_n\right]\in {ℝ}^n$. It was proved in [53] that there exists a solution of (5.10) having the form

$$f^{*}≔\sum _{k=1}^n{\alpha }_{k}K\left(\cdot ,z_k\right),\text{ for some }{z}_k\in 𝒩\left(\sum _{j=1}^n{c}_j^{*}K\left(x_j,\cdot \right)\right)\text{ and }{\alpha }_k\in ℝ,k\in {\mathbb{N}}_n,$$ (5.11)

where $c^{*}\in {ℝ}^n$ is a solution of optimization problem

$$\text{max}\left\{y^{\top }c:c=\left[c_j:j\in {\mathbb{N}}_n\right]\in {ℝ}^n,{\Vert \sum _{j=1}^n{c}_{j}K\left(x_j,\cdot \right)\Vert }_{\infty }=1\right\}.$$ (5.12)

Although the points ${\mathbf{z}}_k$, $k\in {\mathbb{N}}_n$ may not be the same as the points ${\mathbf{x}}_k$, $k\in {\mathbb{N}}_n$, motivated from (5.11), we seek a solution of the regularization problem (5.10) in the form ${\tilde{f}}^{*}≔{\sum }_{k=1}^n{\alpha }_{k}K\left(\cdot ,x_k\right)$. By plugging ${\tilde{f}}^{*}$ into (5.10) and adding a bias term $b\in ℝ$, we obtain the ${\ell }_1$-SVM classification model has the form

$$\text{min}\left\{L_D(\alpha ,b)+\lambda {\Vert \alpha \Vert }_1:\alpha \in {ℝ}^n,b\in ℝ\right\},$$ (5.13)

where

$$L_D(\alpha ,b)≔\sum _{j\in {\mathbb{N}}_n}\text{max}\left\{1-y_j\left(\sum _{k\in {\mathbb{N}}_n}{\alpha }_{k}K\left(x_k,x_j\right)+b\right),0\right\}.$$

The discrete minimization problem (5.13) is the same as (3.14). We comment that the space ${ℝ}^n$ with the ${\ell }_1$-norm is a finite dimensional RKBS according to Theorem 4.4.

Next, we elaborate the numerical solution of the regularized learning problem. Due to the use of a Banach space as the hypothesis space in learning, we are inevitably facing solving the regularized learning problem (5.4) with the Banach norm. Major challenges for solving this problem include infinite dimensionality, nonlinearity and nondifferentiability. The regularization problem (5.4) by nature is infinite dimensional, since we seek a solution in the hypothesis space which is an infinite dimensional Banach space, even though the fidelity term is determined by a finite number of data points. The use of a Banach space as a regularization term leads to nonlinearity. Moreover, employing a sparsity promotion norm results in a non-differentiable optimization problem. Representer theorems presented in this section for a solution of this problem indicate that the regularization problem (5.4) is essentially of finite dimension. However, how the representer theorems can be used in developing practical algorithms requires further investigation. A recent paper [9] is making an attempt toward this direction. Specifically, we reformulated the regularization problem (5.4) in the following way: When the fidelity term of the regularization problem (5.4) has a form in a Banach norm and a regularization term in another Banach norm, we construct a direct sum space based on the two Banach spaces for the data fidelity term and the regularization term, and then recast the objective function as the norm of a suitable quotient space of the direct sum space. In this way, we express the original regularized problem as a best approximation problem in the direct sum space, which is in turn reformulated as a dual optimization problem in the dual space of the direct sum space. The dual problem is to find the maximum of a linear function on a convex polytope, which is of finite dimension and may be solved by numerical methods such as linear programming. The solution of the dual optimization problem provides related extremal properties of norming functionals, by which the original problem is reduced to a finite dimensional optimization problem, which is then reformulated as a finite dimensional fixed-point equation [24, 31], solved by iterative algorithms.

6. Numerical Results

This section is devoted to a numerical example about classification of handwriting digits by using the ${\ell }_1$ SVM model (5.13) with a comparison to the ${\ell }_2$ SVM model. The handwriting digit classification by using the ${\ell }_1$-SVM was studied in [28].

In our experiment, we use the database MNIST of handwriting digits, which is originally composed of 60,000 training samples and 10,000 testing samples of the digits “0” through “9”. We consider classifying two handwriting digits “7” and “9” taken from MNIST. Handwriting digits “7” and “9” are often easy to cause obfuscation in comparison to other pairs of digits. We take 8,141 training samples and 2,037 testing samples of these two digits from the database. Specifically, we consider training data set $D≔\left\{\left(x_j,y_j\right):j\in {\mathbb{N}}_n\right\}\subset {ℝ}^d\times \{\pm 1\}$, where ${\mathbf{x}}_j$ are images of digits 7 or 9. We wish to find a function that defines a hyperplane separating the data $D$ into two groups with labels $y_j=1$ (digit 7) and $y_j=-1$ (digit 9).

We employ the model (5.10) and its related discrete form (5.13) described in section 5 for our experiment. For a comparison purpose, we also test the learning model (5.13) with the squared loss function

$$L_D(\alpha ,b)≔\frac{1}{2}\sum _{j\in {\mathbb{N}}_n}{\left(\sum _{k\in {\mathbb{N}}_n}{\alpha }_{k}K\left(x_k,x_j\right)+b-y_j\right)}^2$$

as a fidelity term in problem (5.13). We compare the classification performance and solution sparsity of problem (5.13) with those of its Hilbert space counterpart (3.6), which leads to problem (5.13) with the ${\ell }_1$-norm being replaced by the ${\ell }_2$-norm.

The objective functions of both the minimization problem (5.13) and its ${\ell }_2$-norm counterpart are not differentiable and thus they can not be solved by a gradient based iteration algorithm. Instead, we employ the Fixed Point Proximity Algorithm (FPPA) developed in [24, 31] to solve these non-smooth minimization problems. Specifically, we first use the Fermat rule to reformulate a solution of the minimization (5.13) as a solution of an inclusion relation defined by the subdifferential of the objective function of (5.13). We then rewrite the inclusion relation as an equivalent fixed-point equation involving the proximity operator of the functions $L_D$ and ${\parallel ·\parallel }_1$ which appear in the objective function of (5.13). The fixed-point equation leads to the FPPA, for solving (5.13), which is guaranteed to converge under a mild condition. We refer the interested readers to [24, 31] for more information on the FPPA.

Numerical results are reported in Tables I–IV. In these tables, $\lambda$ represents values of the regularization parameter, #NZ the sparsity level (number of nonzero entries), and TRA and TEA the learning accuracy (correction rate) for training datasets and testing datasets, respectively. Specifically,

$$\text{TRA }≔\frac{N{P}_{\text{train }}}{N_{\text{train }}}\text{ and TEA}≔\frac{N{P}_{\text{test }}}{N_{\text{test }}},$$

where ${NP}_{\text{train}}$ and ${NP}_{\text{test}}$ denote the number of predicted labels equal to the given labels of the training and testing data points, respectively, and $N_{\text{train}}$ and $N_{\text{test}}$ denote the total number of the training and testing data points, respectively.

Table I:

Classification using the Gaussian Kernel with $\mu =4.8$: Results for the ${\ell }_1$-SVM model with the square loss function.

$\lambda$	0.5	0.7	1.4	1.8	4.0	6.0	487.7
#NZ	1762	880	481	340	187	142	0
TRA	99.46%	99.19%	98.61%	98.34%	97.57%	97.16%	50.67%
TEA	98.77%	98.82%	98.48%	98.38%	97.79%	97.40%	50.47%

Table IV:

Classification using the Gaussian Kernel with $\mu =4$: Results for the ${\ell }_2$-SVM model with hinge loss.

$\lambda$	0.001	0.2	2	12	22	47	128
#NZ	8, 141	8, 141	8, 141	8, 141	8, 141	8, 141	8, 141
TRA	99.90%	99.84%	99.18%	98.13%	97.68%	97.07%	96.31%
TEA	99.02%	98.97%	98.33%	97.89%	97.59%	97.05%	96.41%

Explanations on the choice of the regularization parameter $\lambda$ are in order. In Table I, we randomly selected seven different values of $\lambda$ from the interval $(0,{\lambda }_{\text{max}}]$, where ${\lambda }_{\text{max}}$ is determined by Remark 4.12 in [28]. In each of the remaining tables, we selected seven different values of $\lambda$ in an increasing order such that the ${\ell }_1$-SVM classification model (5.13) and the related ${\ell }_2$ model produce the same accuracy. In this way, we can compare the sparsity of the corresponding solutions of these two models. Moreover, the largest $\lambda$ in Table I is given by the smallest value such that $\#NZ=0$, since for the ${\ell }_1$-SVM model with the square loss function, we have Remark 4.12 of [28] to determine such a $\lambda$ value. However, in Table III, which is for the ${\ell }_1$-SVM model with the hinge loss function, we do not have a theoretical result similar to Remark 4.12 of [28] to determine the smallest value that ensures $\#NZ=0$ and hence, we empirically choose the $\lambda$ value such that $\#NZ=0$.

Table III:

Classification using the Gaussian Kernel with $\mu =4$: Results for the ${\ell }_1$-SVM model with the hinge loss function.

$\lambda$	0.1	0.2	1	2	4	10	435
#NZ	552	481	167	92	56	34	0
TRA	99.99%	99.99%	99.08%	98.17%	97.53%	96.30%	50.67%
TEA	98.72%	98.77%	98.38%	98.09%	97.45%	96.27%	50.47%

The numerical results presented in Tables I–IV confirm that the two models generate learning results with comparable accuracy, while the solution of the ${\ell }_1$-SVM model can have different levels of sparsity according to values of $\lambda$, but that of the ${\ell }_2$-SVM model is always dense disregarding the choice of $\lambda$. In passing, we point it out that the hinge loss function outperforms the square loss function as a fidelity term for the classification problem.

7. Remarks on Future Research Problems

In this section, we elaborate several open mathematical problems critical to the theory of RKBSs and machine learning methods on a Banach space.

It has been demonstrated that learning in certain Banach spaces leads to a sparse solution and learning in a Hilbert space yields a dense solution. Sparse learning solutions can save tremendous computing effort and storage when they are used in decision making processes. However, learning in Banach spaces introduces new challenges. First of all, the theory of the RKBS is far away from completion. Construction of reproducing kernels for RKBSs suitable for different applications is needed. For example, related to the definition of the reproducing kernel, we have the following specific problem: Suppose that $ℬ$ is a Banach space of functions on $\mathbf{X}$ and point-evaluation functionals on $ℬ$ are all continuous. Let ${ℬ}^{\prime }$ denote the the completion of the linear span of all the point-evaluation functionals on $ℬ$ under the norm of ${ℬ}^{*}$. In general, a Banach space $ℬ$ of functions is not isometrically isomorphic to its dual ${ℬ}^{*}$ or its $\delta$-dual ${ℬ}^{\prime }$. We take $ℬ≔{\ell }_1\left(\mathbb{N}\right)$ as an example. We have known from section 4 that ${ℬ}^{*}={\ell }_{\infty }\left(\mathbb{N}\right)$ and ${ℬ}^{\prime }=c_0\left(\mathbb{N}\right)$. Clearly, the space ${\ell }_1\left(\mathbb{N}\right)$ is neither isometrically isomorphic to ${\ell }_{\infty }\left(\mathbb{N}\right)$ nor $c_0\left(\mathbb{N}\right)$. In Definition 4.2 of a kernel, we need the assumption that ${ℬ}^{\prime }$ is isometrically isomorphic to a Banach space of functions on some set ${\mathbf{X}}^{\prime }$. This assumption holds true when $ℬ≔{\ell }_1\left(\mathbb{N}\right)$ since ${ℬ}^{\prime }≔c_0\left(\mathbb{N}\right)$ is a Banach space of functions (sequences) defined on $\mathbb{N}$. We are interested in knowing to what extent is this true in general?

Theorem 4.3 reveals that under Hypotheses (H1) and (H2), a reproducing kernel as defined in Definition 4.2 is positive semi-definite. Can the two hypotheses be weakened? We are curious to know how a given function of two arguments will lead to a pair of RKBSs.

A problem related to sparse learning in a Banach space may be stated as follows: It has been understood that the Banach space ${\ell }_1\left(\mathbb{N}\right)$ promotes sparsity for a learning solution in the space. What is the essential geometric feature for a general Banach space of functions that can lead to a sparse learning solution in the space?

Finally, practical numerical algorithms for learning a function from an RKBS require systematical investigation.

Answers to these challenging issues would contribute to completing the theory of the RKBS and making it a useful hypothesis space for machine learning, hoping leading to practical numerical methods for learning in an RKBS.

Table II:

Classification using the Gaussian Kernel with $\mu =4.8$: Results for the ${\ell }_2$-SVM model with the square loss function.

$\lambda$	0.005	1.0	2.3	6.0	43.0	96.0	490
#NZ	8, 141	8, 141	8, 141	8, 141	8, 141	8, 141	8, 141
TRA	100%	99.50%	99.18%	98.69%	97.56%	97.16%	95.93%
TEA	99.21%	98.87%	98.72%	98.43%	97.69%	97.25%	96.41%