Universal Approximation with Quadratic Deep Networks

Abstract

Recently, deep learning has achieved huge successes in many important applications. In our previous studies, we proposed quadratic/second-order neurons and deep quadratic neural networks. In a quadratic neuron, the inner product of a vector of data and the corresponding weights in a conventional neuron is replaced with a quadratic function. The resultant quadratic neuron enjoys an enhanced expressive capability over the conventional neuron. However, how quadratic neurons improve the expressing capability of a deep quadratic network has not been studied up to now, preferably in relation to that of a conventional neural network. Regarding this, we ask four basic questions in this paper: (1) for the one-hidden-layer network structure, is there any function that a quadratic network can approximate much more efficiently than a conventional network? (2) for the same multi-layer network structure, is there any function that can be expressed by a quadratic network but cannot be expressed with conventional neurons in the same structure? (3) Does a quadratic network give a new insight into universal approximation? (4) To approximate the same class of functions with the same error bound, is a quantized quadratic network able to enjoy a lower number of weights than a quantized conventional network? Our main contributions are the four interconnected theorems shedding light upon these four questions and demonstrating the merits of a quadratic network in terms of expressive efficiency, unique capability, compact architecture and computational capacity respectively.

1. Introduction

Over recent years, deep learning has become the mainstream approach for machine learning. Since AlexNet [1], increasingly more advanced neural networks [2–6] are being proposed, such as GoogleNet, ResNet, DenseNet, GAN and their variants, to enable practical performance comparable to or beyond what human delivers in computer vision [7], speech recognition [8], language processing [9] game playing [10], medical imaging [11–13], and so on. A heuristic understanding of why these deep learning models are so successful is that these models represent knowledge in hierarchy and facilitate high-dimensional non-linear functional fitting. It seems that deeper structures are correlated with greater capacities to approximate more complicated functions.

The representation power of neural networks has been rigorously studied since the eighties. The first result is that a network with a single hidden layer can approximate a continuous function at any accuracy given an infinite number of neurons [14]. This means that the network can be extremely wide. Then, a number of papers were published investigating the complexity of approximation. For example, Barron et al. [49] studied one-hidden-layer feedforward networks using sigmoidal activations, which can approximate a function family, wherein the first moment of the magnitude distribution of the Fourier transform of the function is bounded by a constant, with an accuracy in the order of $\frac{\mathbb{1}}{\mathbb{n}}$, where n is the number of nodes and Kainen et al. [50] demonstrated that for worst-case error, a generic function with dimensionality d in the Hilbert space can be approximated by a one-hidden-layer neural network with an error of ϵ(d)κ(n), where ϵ(d) is bounded above by a polynomial in d and κ(n) is n^−1/m, m > 0. Kon et al. shows that given that the number of nodes n is no smaller than the number of examples k, the optimal network to approximate a function in reproducing kernel Hilbert space is given by ${\sum }_i^k{c}_i\mathit{G}\left(t_i,x\right)$, where t_i is the sample input and G is the RBF neuron [51]. Please note that the optimality means the minimal error achieved in the worst case over all the possible f. Schmitt borrowed the results based on Vapnik-Chervonenkis dimension to prove that to approximate any polynomials, the network size need to grow $\mathcal{O}\left({(\mathit{\text{logk}})}^{1/4}\right)$, where k is the degree of the polynomial [52]. With the emergence of deep neural networks, studies have been performed on theoretical benefits of deep models over shallow ones [15–24]. One way [15, 19] is to construct a special kind of functions that are easy to be approximated by deep networks but hard by shallow networks. It has been reported in [16] that a fully-connected network with ReLU activation can approximate any Lebesgue integrable function in the L₁-norm sense, provided a sufficient depth and at most d+4 neurons in each layer, where d is the number of inputs. Through a similar analysis, it was reported in [22] that ResNet with one single neuron per layer is a universal approximator. Moreover, it was demonstrated in [15] that a special class of functions is hard to be approximated by a conventional network with a single hidden layer unless an exponential number of neurons are used. Bianchini et al. [23] utilized a topological measure to characterize the complexity of functions that are realized by neural networks, and proved that deep networks can represent functions of much higher complexity than the shallow counterparts. Kurkova et al. [24] showed that unless with sufficiently many network units (more than any polynomial of the logarithm of the size of the domain), a good approximation cannot be achieved with a shallow perceptron network for almost any uniformly chosen function on a specially constructed domain.

In our previous studies [25–28], we proposed quadratic neurons and deep quadratic networks. In a quadratic neuron, the inner product of an input vector and the corresponding weights in a conventional neuron is replaced with a quadratic function. The resultant quadratic neuron enjoys an enhanced expressive capability over the conventional neuron (Here conventional neurons refer to neurons that perform activation of an inner-product, and conventional networks refer to networks completely comprising conventional neurons). Actually, a quadratic neuron can be viewed as a fuzzy logic gate, and a deep quadratic network is nothing but a deep fuzzy logic system [27]. Furthermore, we successfully designed a quadratic autoencoder for a real-world low-dose CT problem [28]. Note that high-order neurons [29–30] were taken into account in the early stage of artificial intelligence, but they are not connected to deep networks and suffer from a combinatorial explosion with the number of parameters due to high order terms. In contrast, our quadratic neuron uses a limited number of parameters (tripled that of a conventional neuron) and performs a cost-effective high-order operation in the context of deep learning. For quadratic deep networks, we already developed a general backpropagation algorithm [26], enabling the network training process.

However, how quadratic neurons improve the expressing capability of a deep quadratic network has not been theoretically studied up to now, preferably in relation to that of a conventional neural network. In this paper, we ask four basic questions regarding the expressive capability of a quadratic network: (1) for the one-hidden-layer network structure, is there any function that a quadratic network can approximate much more efficiently than a conventional network? (2) for the same multi-layer network structure, is there any function that can be expressed by a quadratic network but cannot be expressed with conventional neurons in the same structure? (3) Does a quadratic network give a new insight and a new method for universal approximation? (4) To approximate the same class of functions with the same error bound, is a quadratic network able to enjoy a lower number of weights than a conventional network? If the answers to these questions are favorable, quadratic networks should be significantly more powerful in many machine learning tasks.

In this paper, our contributions are to present four theorems addressing the above questions respectively and positively, thereby establishing the intrinsic advantages of quadratic networks over conventional networks. More specifically, these theorems characterize the merits of a quadratic network in terms of expressive efficiency, unique representation, compact architecture and computational capacity. We answer the first question with the first theorem, given the network with only one hidden layer and admissible activation function, there exists a function that a quadratic network can approximate it with a polynomial number of neurons but a conventional network can only do the same level approximation with an exponential number of neurons. Regarding the second question and the second theorem below, with the ReLU activation function, any continuous radial function can be approximated by a quadratic network in a structure of no more than four neurons in each layer but the function cannot be approximated by a conventional network of the same structure [16], while with the third theorem below we provide a new insight and a new method for a universal presentation from the perspective of the Algebraic Fundamental Theorem. Without introducing complex numbers, a univariate polynomial of degree n can be uniquely factorized into a product of quadratic terms instead of first-order terms. Since a ReLU quadratic network can represent any univariate polynomial in a unique and global manner, and by the Weierstrass theorem and the Kolmogorov theorem that multivariate functions can be represented through summation and composition of univariate functions, we can approximate any multivariate function with a well-structured ReLU quadratic neural network, justifying the universal approximation power of the quadratic network. To our best knowledge, our quadratic network is the first-of-the-kind global universal approximator. Finally, with the fourth theorem below, we attempt to address the theoretical foundation for quantization of a neural network. To approximate the same class of functions with the same error bound, a quantized quadratic network demands a much lower number of weights than a quantized conventional network. The saving is almost in the order of $\mathcal{O}\left(\lambda \left(lo{g}^{\frac{1}{\lambda -1}+1}\left(\frac{1}{ϵ}\right)\right){\left(\frac{1}{ϵ}\right)}^{\frac{d}{n}}\right)$, λ ≥ 2, d, n ≥ 1. Compared to the previous studies that theoretically explored the properties of quantized networks, our result adds unique insights into this aspect.

There are prior papers related to but different from our contributions [40–45]. Motivated by a need for more powerful activation, Livni et al. [40] and Krotov et al. [41] proposed to use the quadratic activation: σ(z) = z² and rectified polynomials in the neuron respectively. Despite somewhat misleading in their name, networks with quadratic activation or rectified polynomials and our proposed networks that consist of quadratic neurons have fundamental differences. At the cellular level, a neuron with quadratic activation is still characterized with a linear decision boundary, while our quadratic neuron allows a quadratic decision boundary. In [40], the authors demonstrated that networks with quadratic activation are as expressive as networks with threshold activation, and constant-depth networks with quadratic activation can learn in polynomial time. In contrast, our work goes further showing that the expressibility of the quadratic network is superior to that of the conventional networks; for example, a single quadratic neuron can implement the XOR gate, and a quadratic network of finite width and depth can represent a finite-order polynomial up to any desirable accuracy. In [42], Du et al. showed that over-parameterization and weight decay are instrumental to the optimization aided by quadratic activation. [43] reported how a neural network can provably learn a low-degree polynomial with gradient descent search from scratch, with an emphasis on the effectiveness of the gradient descent method. [44] presents that $O\left(log\frac{p}{ϵ}\right)$ layers of binary units and $O\left(\mathit{\text{plog}}\frac{p}{ϵ}\right)$ ReLU units can approximate $f(x)={\sum }_{i=0}^p{a}_i{x}^i$ with closeness of ϵ. In contrast, our Theorem 3 is based on the Algebraic Fundamental Theorem to provide an exact representation of any finite-order polynomial. [45] is on factorization machine (FM) dedicated to combine high order features, clearly different from the polynomial factorization we propose to perform using a quadratic network.

2. Preliminaries

Quadratic/Second-order Neuron:

The n-input function of a quadratic/second-order neuron before being nonlinearly processed is expressed as:

$$h(x)=(\sum _{i=1}^n{w}_{ir}{x}_i+b_r)(\sum _{i=1}^n{w}_{ig}{x}_i+b_g)+\sum _{i=1}^n{w}_{ib}{x}_i^2+c=(w_r{x}^T+b_r)(w_g{x}^T+b_g)+w_b{(x^2)}^T+c,$$ (1)

where x denotes the input vector, w_r,w_g,w_b are vectors of the same dimension with x and b_r, bg, c are adjustable biases. Our quadratic function definition only utilizes 3n parameters, which is more compact than the general second-order representation requiring $\frac{n(n+1)}{2}$ parameters. While our quadratic neuron design is unique, other papers on quadratic neurons are also in the later literature; for example, [31] proposed a type of neurons with paraboloid decision boundaries. It is underlined that the emphasis of our work is not only on quadratic neurons individually but also deep quadratic networks in general.

One-hidden-layer Networks:

The generic function represented by a one-hidden-layer conventional network is as follows:

$$f_1(x)=\sum _{l=1}^k{t}^l{\sigma }_l\left(w^{l}x+b^l\right),$$ (2)

where l refers to the l^th layer. In contrast, the generic function represented by a one-hidden-layer quadratic network is:

$$f_2(x)=\sum _{l=1}^k{t}^l{\sigma }_l\left[\left(w_r^l{x}^T+b_r^l\right)\left(w_g^l{x}^T+b_g^l\right)+w_b^l{\left(x^2\right)}^T+c^l\right].$$ (3)

In our Theorem 1 below, we will compare the representation capability of a quadratic network and that of a conventional network assuming that both networks have the same one-hidden-layer structure.

L-Lipschitz Function:

A L-Lipschitz function f from ${ℝ}^n$ to $ℝ$ is defined by the following property:

$$|f(x)-f(y)|\le L\Vert x-y\Vert$$

Radial Function:

A radial function only depends on the norm of its input vector, generically denoted as f(∥x∥). The functions mentioned in Theorems 1 and 2 are all radial functions. By its nature, the quadratic neuron is well suited for modeling of a radial function.

Euclidean Unit-volume Ball:

In a d-dimensional space, let R_d be the radius of a Euclidean ball B_d such that R_dB_d has the unit volume. Euclidean ball is used to define density function μ later. Here, the main reason why the μ is defined on the Euclidean ball is because the function f being constructed is a radial function, and the Fourier transform of f is also radial. Thus, it is appropriate to investigate f in the Euclidean ball, which simplifies the analysis and helps focusing on the essence of the problem. The extension of considering other domains (such as cube) will be considered for future work.

Bernstein Polynomial:

$l_{n,m}=C_n^m{x}^m{(1-x)}^{n-m}$, 0 ≤ m ≤ n. The n-th Bernstein polynomial of f(x) in (0, 1) is defined as

$${\mathit{B}}_n(x)=\sum _{m=0}^{n}f\left(\frac{m}{n}\right)l_{n,m}(x)$$

Previously, the Bernstein Polynomials are used to prove the Weierstrass theorem.

Admissible function:

An activation function $\sigma :ℝ\to ℝ$ is called admissible if:

1. σ is measurable

2. there are K, κ > 0, |σ(x)| ≤ K(1 + |x|^κ) for all $x\in ℝ$

3. given any L-Lipschitz function $f:ℝ\to ℝ$ which is constant outside a bounded interval [−R, R] and any δ > 0, there are scalars $a\in ℝ$ and (α_i, β_i, γ_i)_i=1,2,…,w such that

$$|f(x)-[a+\sum _{i=1}^w{\alpha }_i\sigma \left({\beta }_{i}x-{\gamma }_i\right)]|\le \delta ,$$

where $w\le c_{\sigma }\frac{RL}{\delta }$.

The standard activation functions such as ReLU and sigmoid satisfy the above three properties. Detailed proofs are out of the scope of this paper, please refer to [15,45].

Function space ${\mathcal{F}}_n^d$:

The Sobolev space ${\mathcal{W}}^{n,\infty }\left({[0,1]}^d\right)$ with n= 1, 2, … is defined on [0, 1]^d, lying in L^∞ together with their weak derivatives of up to order n. The function space ${\mathcal{F}}_n^d$ is made of any function $f\in {\mathcal{W}}^{n,\infty }\left({[0,1]}^d\right)$ and

$$\underset{{}_{n:|n|\le n}}{\text{max}}ess{\text{ sup}}_{x\in {[0,1]}^d}|{\mathit{D}}^{n}f(x)|\le 1,$$

where $|n|={\sum }_{k=1}^d{n}_k$.

3. Four Theorems

First, we present four theorems, and then give their proofs respectively.

Theorem 1: For an admissible activation function σ(·) and for some universal constants c > 0, C > 0, C′ > 0, c_σ > 0, there exist a probability measure μ and a radial function $\tilde{g}:{ℝ}^d\to ℝ$, where d > C, that is bounded on [−2,2] and supported on $||\mathit{x}||\le C^{\prime }\sqrt{d}$ satisfying:

1. $\tilde{g}$ can be approximated by a single-hidden-layer quadratic network with C′ c_σd^3.75 neurons, which is denoted as f₂.

2. For every function f₁expressed by a single-hidden-layer conventional network with at most ce^cd neurons, we have:

$$E_{x~\mu }{\left(f_1(x)-f_2(x)\right)}^2\ge \delta$$

for some positive constant δ.

Theorem 2: For any compactly supported, radial, continuous function $f:{ℝ}^n\to ℝ$ and any ϵ > 0, there exists a function that can be implemented by a ReLU-activated quadratic network with at most four neurons in each layer, such that:

$$\underset{x\in {ℝ}^d}{\text{sup}}|f(x)-g(x)|dx\le ϵ.$$

Theorem 3: With ReLU as activation function, the quadratic network is a global universal approximator.

Theorem 4: For any $f\in {\mathcal{F}}_d^n$ and any ϵ ∈ (0, 1), there is a ReLU quadratic network with λ distinct weights that can approximate f with ϵ, satisfying (i) the depth is $\mathcal{O}\left(log\left(log\left(\frac{1}{ϵ}\right)\right)+log\left(\frac{1}{ϵ}\right)\right)$ (ii) the number of weights is $\mathcal{O}\left(log\left(log\left(\frac{1}{ϵ}\right)\right){\left(\frac{1}{ϵ}\right)}^{\frac{d}{n}}+log\left(\frac{1}{ϵ}\right){\left(\frac{1}{ϵ}\right)}^{\frac{d}{n}}\right)$.

3.1. Theorem 1

Key Idea for Proving Theorem 1:

The form of functions represented by a single-hidden-layer conventional network is $f_1(x)=\sum _{l=1}^k{t}^l{\sigma }_l\left(w^{l}x+b^l\right)$. It is observed that the distribution of the Fourier transform of f₁(x) is supported on a finite collections of lines. The support covered by the finite lines are sparse in the Fourier space, especially for a high dimension and high frequency regions, unless an exponential number of lines are involved. Thus, a possible target function to be constructed should have major components at high-frequencies. A suitable candidate has been constructed in [15]:

$$\tilde{g}\left(x\right)=\sum _{i=1}^N{ϵ}_i{g}_i\left(\Vert x\Vert \right),$$

where ϵ_i ∈ {−1, 1}, N is a polynomial function of d, g_i(∥x∥) = 1 {∥x|| ∈ Δ_i,} are radial indicator functions over disconnected intervals. According to the definition, $\tilde{g}$ is well bounded. Although the constructed $\tilde{g}$ is hard to approximate by a conventional network, it is easy to approximate by a quadratic network, because $\tilde{g}$ is a radial function. Consequently, it is feasible for a single-hidden-layer quadratic network to approximate the radial function with a polynomial number of neurons. Note that $\tilde{g}(x)$ is discontinuous, hence it cannot be perfectly expressed by a neural network with continuous activation functions. Therefore, let us use a probability measure μ for network quality assessment. With μ, the distance between $\tilde{g}$ and f represented by a network is characterized as $E_{x~\mu }{(f(x)-\tilde{g}(x))}^2$. In particular, μ = ϕ², where ϕ is the inverse Fourier transform of the indicator function defined on the Euclidean ball: 1{x ∈ B}. Therefore, $E_{x~\mu }{(f(x)-\tilde{g}(x))}^2=\Vert \widehat{f\varphi }-\widehat{\tilde{g}\varphi }{\Vert }_{L_2}$, where $\widehat{f\varphi }$, $\widehat{\tilde{g}\varphi }$ are the frequency forms of fϕ and $\tilde{g}\varphi$. The physical meaning of $E_{x~\mu }{(f(x)-\tilde{g}(x))}^2$ is to measure the closeness of $\tilde{g}$ and f in frequency domain within Euclidean ball B.

Please note that c and c_σ are different constants. In Lemma 1, as long as the number of neurons is no more than ce^cd, the distance between f₁ and $\tilde{g}$ in the sense of L₂(μ) will be greater than a constant. In Lemma 2, c_σ is a fixed constant after the activation function σ is pre-specified.

The contribution of our work:

In our Theorem 1, we demonstrate the utility of a quadratic network in approximating the constructed radial function $\tilde{g}$, which is straight-forward but interesting and non-trivial. Furthermore, our result holds for all the admissible activation functions, while the proof in [15] is only for the ReLU activation function. Proposition 1 in [15] demonstrates that $\tilde{g}$ cannot be well approximated by a single-hidden-layer conventional network with a polynomial number of neurons. We put this proposition here as Lemma 1 for completeness. We would like to emphasize that although we put Lemma 1 and other Lemmas on a equal footing, Lemma 1 is more important than other lemmas. Actually, our Theorem 1 is merely a corollary of Proposition 1 in [15] when ReLU activation is used. Again, our contribution is the proof that the one-hidden-layer quadratic network using an admissible activation function can approximate $\tilde{g}$.

Lemma 1:

There are universal constants c,C > 0 such that for every d > C, there is a probability measure μ on ${ℝ}^d$ such that for any α > C and N ≥ Cα^1.5d², there exists a function $\tilde{g}(x)={\sum }_{i=1}^N{ϵ}_i{g}_i(x)$, where ϵ_i ∈ {−1, 1}, i = 1, 2, …, N, such that for any function of the form: $f_1(x)={\sum }_{l=1}^k{t}^l{\sigma }_l\left(w^{l}x+b^l\right)$ with σ_l as admissible function and k ≤ ce^cd, we have:

$${\Vert f_1-\tilde{g}\Vert }_{L_2(\mu )}\ge \delta /\alpha ,$$

where δ > 0 is a universal constant.

Universal constants means they are independent from the dimension d. To illustrate $\tilde{g}$ is expressible with a quadratic network, we know from Lemma 12 of [15] that a continuous Lipschitz function g can approximate $\tilde{g}$, what is remained for us to do is to use a quadratic network with a polynomial number of neurons to approximate g.

Lemma 2:

Given an admissible activation function σ, there is a constant c_σ ≥ 1 (depending on σ and other parameters) such that for any L-Lipschitz function $g:ℝ\to ℝ$, which is constant outside a bounded interval [r, R] (r ≥ 1) and any δ, there exist scalars a, {α_i, β_i, γ_i }, i = 1,…, w, $w\le c_{\sigma }\frac{\left(R^2-r^2\right)L}{4\sqrt{r}\delta }$ with which we have:

$$h\left(\Vert x{\Vert }^2\right)=a+\sum _{i=1}^w{\alpha }_i\sigma \left({\beta }_i\Vert x{\Vert }^2-{\gamma }_i\right)$$

satisfies:

$$\underset{\Vert x\Vert \in [r,R]}{\text{sup}}|g(\Vert x\Vert )-h\left(\Vert x{\Vert }^2\right)|<\delta .$$

Proof: $g:ℝ\to ℝ$ is an L-Lipschitz function and supported on [r, R], r > 0. Let us define $s(x)=g(\sqrt{x})$, which is of $\frac{L}{2\sqrt{r}}$-Lipschitz and supported on [r², R²]. By the property of an admissible function, we construct a function $h(t)=a+{\sum }_{i=1}^w{\alpha }_i\sigma \left({\beta }_{i}t-{\gamma }_i\right)$, which satisfies the following condition:

$$\underset{l\in \left[r^2,R^2\right]}{\text{sup}}|s(l)-h(l)|<\delta ,$$

Where $w=c_{\sigma }\frac{\left(R^2-r^2\right)L}{4\sqrt{r\delta }}$. Next, we have:

$$\underset{\Vert x\Vert \in [r,R]}{\text{sup}}|g(\Vert x\Vert )-h\left(\Vert x{\Vert }^2\right)|=\underset{\Vert x\Vert \in [r,R]}{\text{sup}}|s(\Vert x{\Vert }^2)-h\left(\Vert x{\Vert }^2\right)|=\underset{l\in \left[r^2,R^2\right]}{\text{sup}}|s(l)-h(l)|<\delta$$

Lemma 3:

There are a universal constant C > 0 and δ ∈ (0, 1), for d ≥ C and any choice of ϵ_i ∈ {−1, 1}, i = 1, 2, …, N, there exists a function f₂ expressed by a single-hidden-layer quadratic network of a width of at most $c_{\sigma }\frac{3N{\alpha }^{1.5}{d}^{1.75}}{4\delta }$ and with the range [−2,+2] such that

$${\Vert f_2-\tilde{g}\Vert }_{L_2(\mu )}\le \frac{\sqrt{3}}{\alpha d^{\frac{1}{4}}}+\delta .$$

Proof: In Lemma 2, we make the following substitutions: $R=2\alpha \sqrt{d}$, $r=\alpha \sqrt{d}$, L = N, h = g. Notably, α can be greater than 1, and $r=\alpha \sqrt{d}>1$ satisfying the condition of Lemma 2. Thus, g(∥x∥) is expressible by a single-hidden-layer quadratic network with at most $c_{\sigma }\frac{3N{\alpha }^{1.5}{d}^{1.75}}{4\delta }$ neurons. Coupled with Lemma 12 of [15], Lemma 3 is immediately obtained.

Proof of Theorem 1:

By the combination of Lemmas 1 and 3, the proof for Theorem 1 is straightforward. In Lemma 1, by choosing α = C, N = Cα^1.5d², we have

$${\Vert f_1-\tilde{g}\Vert }_{L_2(\mu )}\ge {\delta }_1$$

Let $\delta \le \frac{{\delta }_1}{2}-\frac{\sqrt{3}}{C{d}^{\frac{1}{4}}}$ which demands d is very large so that δ > 0, to approximate $\tilde{g}$ we need the number of quadratic neurons being at most

$$c_{\sigma }\frac{3N{\alpha }^{1.5}{d}^{1.75}}{4\delta }=c_{\sigma }\frac{3C^{2.5}{d}^{3.75}}{4\delta }\le C^{\prime }{c}_{\sigma }{d}^{3.75}$$

(where C′ is a universal constant depending on the universal constants C,δ₁) such that

$${\Vert f_2-\tilde{g}\Vert }_{L_2(\mu )}\le \frac{\sqrt{3}}{C{d}^{\frac{1}{4}}}+\delta \le \frac{{\delta }_1}{2}.$$

Therefore, we have ${\Vert f_1-f_2\Vert }_{L_2(\mu )}\ge \frac{{\delta }_1}{2}$ The proof is completed.

Classification Example:

The above exponential difference between the conventional and quadratic networks is due to the dimension d. To demonstrate this, we constructed an example for separation of two concentric rings. In this example, there are 60 instances in each of the two rings representing two classes. With only one quadratic neuron in a single hidden layer, the rings were totally separated, while at least six conventional neurons are required to complete the same task, as shown in Figure 1.

Figure 1:

Classification of two concentric rings with conventional and quadratic networks (d=2). To succeed in the classification, a conventional network requires at least 6 neurons (Left), and a quadratic network only takes one neuron.

3.2. Theorem 2

Key Idea for Proving Theorem 2:

It was proved in [16] that an n-input function that is not constant along any direction cannot be well approximated by a conventional network with no more than n – 1 neurons in each layer. However, for such a radially defined function, it is feasible for a quadratic network to approximate the function with width=4, which breaks the lower width bound n+4 given in [16]. The trick for the approximation by a deep conventional network is adapted for analysis of a deep quadratic network so that the function can be approximated in a “quadratic” way via composition layer by layer. To compute a radial function, we need to find the norm and then evaluate the function approximation by the norm. With a quadratic neuron, the norm can be easily found. Heuristically speaking, a quadratic network with no more than n – 1 neurons in each layer could approximate a radial function very well, even if the function is not constant along any direction. Therefore our theorem positively shows that it is more natural and more effective to represent functional non-linearity with a quadratic network. Finally, since a quadratic network with width at most 4 can approximate any radial function and radial function family can approximate any continuous function [47], a quadratic network can serve as a width-bounded universal approximator.

Proof of Theorem 2:

As per structure outlined in Figure 2, a conventional ReLU network with four neurons in each layer can implement any function of the form:

$$[0,+\infty ]\to ℝ:x\to \theta +\sum _{l=1}^k{\gamma }_l\text{ReLU}\left({\alpha }_{l}x+{\beta }_l\right)$$

, which is actually any continuous piecewise linear function defined on non-negative real number domain. Because function $f:{ℝ}^d\to ℝ$ is radial and continuous, without loss of generality, Let h(∥x∥²) = f(x), while h is function:$[0,+\infty ]\to ℝ$. It implies that for any ϵ > 0, there will be a continuous piecewise linear function $g:[0,+\infty ]\to ℝ$, and such that

$$\underset{[0,+\infty ]}{\text{sup}}|h(s)-g(s)|<ϵ.$$

Figure 2:

Illustration of a conventional network structure that allows to implement any continuous univariate piecewise linear function

g can be implemented by a conventional ReLU network as shown in Figure 2. Therefore, the function $x\to g\left(\Vert x{\Vert }_2^2\right):{ℝ}^d\to ℝ$ can be implemented with a quadratic ReLU network of the same structure by calculating norm of input. Then we will obtain:

$$\underset{x\to {ℝ}^d}{\text{sup}}|f(x)-g\left(\Vert x{\Vert }_2^2\right)|=\underset{x\to {ℝ}^d}{\text{sup}}|h\left(\Vert x{\Vert }_2^2\right)-g\left(\Vert x{\Vert }_2^2\right)|=\underset{s\in [0,+\infty ]}{\text{sup}}|h(s)-g(s)|<ϵ$$

Lemma 4:

[47] Let $K:{ℝ}^r\to ℝ$ be an integrable bounded function such that K is continuous almost everywhere and ${\int }_{{ℝ}^r}K(x)dx\ne 0$. Then the functional family $S_K:{\sum }_{i=1}^M{w}_{i}K\left(\frac{\Vert x-z_i\Vert }{\sigma }\right)$,(M is a positive integer, σ > 0 and $w_i\in ℝ$) is dense in $L^p\left({ℝ}^r\right)$.

Proof: Please see [Theorem 1, 47].

Corollary 2:

With ReLU activation function, residual quadratic networks with at most four neurons in one layer is universal approximator.

Proof: As Figure 3 shown, because lemma 4 makes sure that the quadratic ReLU network can approximate function $K\left(\frac{\Vert x-z_i\Vert }{\sigma }\right),i=1,2\dots$. We use residual connections (red lines) to make the network modules that represent different $K\left(\frac{\Vert x-z_i\Vert }{\sigma }\right),i=1,2\dots$ and we use another groups of residual connections (green lines) to aggregate the outputs of these modules, such that quadratic ReLU network can represent any function from the family S_K at any accuracy. By triangle relationship, we conclude that residual quadratic ReLU network with no more than four neurons in each layer is universal approximator.

Figure 3:

Quadratic ReLU network with no more than four neurons using shortcuts in each layer is universal approximator. The red shortcuts are used for passing the input and the green shortcuts for aggregating the outputs of different modules.

Compared with the results in [16] and [22], our Corollary 2 presents another special width-bounded universal approximator, which is slimmer than that of [16] and has sparser shortcuts than that of [22], The upper limit of the width required by our width-bounded universal approximator is 4, while the counterpart in [16] is n + 4 given an n–input. Although only one neuron is demanded in each layer in [22], much denser residual shortcuts are employed as the trade-off. In our work, only sparser residual connections are needed to integrate different blocks.

3.3. Theorem 3

Key Idea for Proving Theorem 3:

For universal approximation by neural networks, the current mainstream strategies are all based on piecewise approximation in terms of L₁ L_∞, or other distance measures. For the purpose of piecewise approximation, the functional space is divided into numerous hypercubes, which are intervals in the one-dimensional case, according to a specified accuracy to approximate a target function in every hypercube. With quadratic neurons and the trick that two ReLU neurons can organically execute a linear operation, we can instead use a global approximating method that is much more efficient. At the same time, the quadratic network structure is neither too wide nor too deep, which can be regarded as a size-bounded universal approximator, in contrast to what we have introduced before. What’s more, aided by Algebraic Fundamental theorem, our novel proof reveals the uniqueness and facility of our proposed quadratic networks that cannot be elegantly made by networks with quadratic activation. More importantly, Theorem 3 gives a first-of-the-kind global universal approximator, facilitating feature representations in deep learning.

First, we show any univariate polynomial of degree N can be exactly expressed by a quadratic network with a complexity of $\mathcal{O}\left(lo{g}_2(N)\right)$ in depth. Next we refer the result [34] regarding Hilbert’s thirteen problem that multivariate functions can be represented with a group of separable functions, and then finalize the proof.

Lemma 5:

With ReLU as activation function, any univariate polynomial of degree N can be perfectly computed by a quadratic network with depth of $\mathcal{O}\left(lo{g}_2(N)\right)$ and width of no more than N.

Proof: According to the Algebraic Fundamental Theorem [35], a general univariate polynomial of degree N can be expressed as $P_N(x)=C{\prod }_i^l{}^1\left(x-x_i\right){\prod }_j^l{}^2\left(x^2+a_{j}x+b_j\right)$, where l₁ + 2l₂ = N. The ReLU activation has an important property: f(x) = ReLU(f(x)) − ReLU(−f(x)), which is often resorted in the literature, such as [48]. The network we construct is shown in Fig. 4 (Each neuron actually represents two parallel neurons to implement a linear operation). Every two neurons are grouped in the first layer to compute (x–x_i), (x–x_i)(x–x_i+1) or x²+a_ix+b_i, then the second layer merely uses half the number of neurons in the first layer to combine the outputs of the first layer. By repeating such a process, the quadratic network with the depth of O(log₂(N)) can exactly represent P_N(x).

Figure 4:

Quadratic network approximating a univariate polynomial according to the Algebraic Fundamental Theorem.

The following lemma shows that any univariate function f(x) continuous in [0, 1] can be approximated with a Bernstein polynomial up to any accuracy, which is the Bernstein version of the proof for the Weierstrass theorem.

Lemma 6:

Let f(x) is a continuous function over [0, 1], we have

$$\underset{n\to +\infty }{\text{lim}}{\mathit{B}}_n(x)=f(x)$$

Proof: It is well known; please refer to [33].

Corollary 2:

Any continuous univariate function supported on [0, 1] can be approximated by a quadratic network with ReLUs up to any accuracy.

Lemma 7:

Every continuous n-variable function on [0, 1]ⁿ can be represented in the form:

$$f\left(x_1,\dots ,x_n\right)=\sum _{i=1}^{2n+1}{g}_i(\sum _{j=1}^n{h}_{ij}(x_j))$$

Proof: This is a classical theorem made by Kolmogorov and his student Arnold. Please refer to [34].

Proof of Theorem 3:

Combining Lemmas 5–7, it can be shown that the ReLU-embedded quadratic network is a universal approximator, and importantly it realizes a universal approximation in a global manner.

Let us look at the following numerical example to illustrate our novel quadratic network based factorization method. First, 100 instances were sampled of a function, g(x) = (x² + l)(x–l)(x² + 1.7x+1.2)in [−1,0]. Instead of taking opposite weights and biases to create linearity as used in proof for clarity, here we have incorporated shortcuts to make our factorization method trainable in terms of adaptive offsets. Using the ReLU activation function, we trained a four-layer network 4-3-1-1 with shortcuts to factorize this polynomial, as shown in Fig. 5. The parameters in connections marked by green symbols are fixed to perform multiplication. The shortcut connections and vanilla connection are denoted by green and red lines respectively. The neurons in the first layer will learn the shifted factors T₁, T₂, T₃, with unknown constant offsets C₁,C₂,C₃. The whole network will be combined in pairs to form in the next layer. Then, the neurons in the third layer will multiply T₁ and T₂T₃ to obtain T₁T₂T₃. Finally, the neuron in the output layer is a linear one that will be trained to undo the effect of the constant offsets aided by the shortcuts. We trained the network to learn the factorization by initializing the parameters multiple times, with the number of iterations 600 and the learning rate 2.0e-3. The final average error is less than 0.0051. In this way, the function was learned to be $g^{\prime }(x)=\left(1.0117592x^2+0.02033176x+1.0040002\right)(0.00819827x^2+0.9977611x-1.0023365)\left(1.0113174x^2+1.7042247x+1.1885077\right)$, which agrees well with the target function g(x) = (x² + 2)(x − l)(x² + 1.7x + 1.2).

Figure 5:

Quadratic network with shortcuts to learn the soft factorization of an exemplary univariate polynomial.

Mathematically, we can handle the general factorization representation problem as follows. Let us denote A_i = m_i, A_ij = m_im_j, A_ijk = m_im_jm_k, …, $A_{123\dots n}={\prod }_i^n{m}_i$, which are generic factors for i = 1, 2, …, n, the question becomes if ${\Pi }_i^n\left(m_i+C_i\right)-{\Pi }_i^n{m}_i$ can be represented as the linear combination of A₀, A_i, A_ij, A_ijk, …, A₁₂₃…(n−1) and a constant bias? If the answer is positive, then our above-illustrated factorization method can be extended for factorization of any univariate polynomial. Here we offer a proof by mathematical induction.

For n = 1, we have m₁ + C₁ – m₁ = C_1. Assume that we can represent ${\prod }_i^p\left(m_i+C_i\right){\prod }_i^p{m}_i$ with a linear combination of A_i,A_ij,A_ijk, …, A_{123…(p–1)} and a constant, denoted as f(A_i,A_ij,A_ijk, …, A_{123…(p–1)}). Then, ${\prod }_i^{p+1}(m_i+C_i-{\prod }_i^{p+1}{m}_i=\left(m_{p+1}+C_{p+1}\right)\left({\prod }_i^p\left(m_i+C_i\right)-{\prod }_i^p{m}_i\right)+C_{p+1}{\prod }_i^p{m}_p=\left(m_{p+1}+C_{p+1}\right)f\left(A_i,A_{ij},A_{ijk},\dots ,A_{123,(p-1)}\right)+C_{p+1}{\prod }_i^p{m}_p$. Clearly, the above terms m_p+1f(A_i,A_ij,A_ijk, …, A_{123…(p–1)}, $C_{p+1}{\prod }_i^p{m}_p$ are both linear terms of A_i,A_ij,A_ijk, …, A_123…p and C_p+1f(A_i,A_ij,A_ijk, …, A_{123…(p–1)}) is a linear representation of A_i,A_ij,A_ijk, …, A_{123…(p–1)} plus constant bias as well. Combining these together, we immediately have a desirable linear representation of ${\prod }_i^{p+1}\left(m_i+C_i\right)-{\prod }_i^{p+1}{m}_i$ that concludes our proof.

3.4. Theorem 4

Key Idea for Proving Theorem 4:

It is notable that a quadratic neuron internally realizes addition and multiplication operations in a unique way, which enables quantized quadratic neurons as building blocks for construction of an arbitrary function in a top-down way. That is the key mechanism by which the saving of $\mathcal{O}\left(\lambda \left({log}^{\frac{1}{\lambda -1}+1}\left(\frac{1}{ϵ}\right)\right){\left(\frac{1}{ϵ}\right)}^{\frac{d}{n}}\right)$, λ ≥ 2 is achieved in terms of the number of weights. Figure 6 clearly demonstrates our heuristics.

Figure 6:

Quantized quadratic ReLU networks with weight 2⁻¹ and −2⁻¹ to approximate any number with an error bound of 2^−t

Lemma 8:

A connection with a weight w ∈ [−1,1] can be approximated by a ReLU network with λ ≥ 2 distinct weights, satisfying that (i) the ReLU network is equivalent to weight w′ with error no more than 2^−t specifically, |w – w′| < 2^−t; (ii) the depth is $\mathcal{O}[log(t)];$ (iii) the width is $\mathcal{O}(t);$ and (iv) the number of weights is at most $\mathcal{O}(t)$.

Proof: We first consider λ = 2 (we select two different weights as 2⁻¹, −2⁻¹), w ≥ 0 and x ≥ 0, and will relax the constraints later. Setting w = 0 directly leads to an empty operation. There are three other necessary operations perfectly represented by a ReLU quadratic network: E_u (unity operation) E_p (power operation) and E_m (multiplication), which take 4, 8, 8 neurons respectively, as shown in Figure 6. Our goal is to approximate w in accuracy of 2^−t. Let us define

$${\mathit{K}}_c\equiv \left\{2^{-1},2^{-2},\dots ,2^{-(t-1)},2^{-t}\right\}.$$

With a homogeneous weight 2⁻¹, all the members in K_c can be represented by the composition of E_u, E_p and E_m operations. Because a binary expansion of W_c can denote any integral multiple of 2^−t like a numeral system with the radix of 2^−t, which is equivalent to represent any number with an error bound of 2^−t. To implement a binary expansion, one more layer with unity and empty operations is needed. Therefore, the overall quantized quadratic network has log(t) + 1 layers, with no more than 24t weights.

Regarding the situation of w < 0, we are able to construct −1 with −2⁻¹ as the final layer. To relax the assumption x ≥ 0, we can use two parallel sub-networks that take x and −x respectively. Due to the gate property of ReLU, the output sign of a sub-network can be flipped in the last layer. In the case of λ > 2, because our construction is top-down, it is interesting that $\frac{1}{2}$ and $-\frac{1}{2}$ are enough for the construction. We can assign two weight are $\frac{1}{2}$ and $-\frac{1}{2}$ respectively among λ distinct weights. Therefore λ does not affect the final complexity.

Proof of Theorem 4:

The proof consists of the following two steps. First, we utilize f′ to approximate $f\in {\mathcal{F}}_d^n$, and then we construct f′ precisely with our quantized ReLU quadratic networks. For m = (m₁, …, m_d) ∈ {0, 1, …, N}^d, ψ_m(x) is defined as:

$${\psi }_m(x)={\Pi }_{k=1}^{d}h\left(3N{x}_k-3m_k\right),$$

where N is a constant and

$$h(x)=\{\begin{array}{l}1|x|\le 1\hfill \\ 2-|x|1\le |x|\le 2\hfill \\ 0|x|\ge 2\hfill \end{array}$$

ψ_m(x) forms a partition of the unity on [0, 1]^d:$\sum _m{\psi }_m(x)\equiv 1$, where x is supported on [0, 1]^d. Note that the support of ψ_m(x) is $\left\{x:|x_k-\frac{m_k}{N}|<\frac{1}{N},\forall k\right\}$. For any m, there is a corresponding order n – 1 Taylor expansion of the function f at $x=\frac{m}{N}:$

$$Q_m(x)={\sum _{n:|n|<n}\frac{D^{n}f}{n!}|}_{x=\frac{m}{N}}{(x-\frac{m}{N})}^n,$$

where $n!={\Pi }_{k=1}^d{n}_k!$, and ${\left(x-\frac{m}{N}\right)}^n={\Pi }_{k=0}^d{\left(x_k-\frac{m_k}{N}\right)}^{n_k}$. Furthermore, we define $Q_m^{\prime }(x)={\sum }_{n:|n|<n}{\gamma }_{m,n}{\left(x-\frac{m}{N}\right)}^n$, where γ_m,n denotes some integral multiple of $\frac{1}{n}{\left(\frac{d}{N}\right)}^{n-|n|}$ that is the closest to ${\frac{D^{n}f}{n!}|}_{x=\frac{m}{N}}$. Thus, the approximation of f is constructed as:

$$f^{\prime }(x)=\sum _m{\psi }_m{\mathit{Q}}_m^{\prime }(x)$$

Hence, we have

$$\underset{x\in {[0,1]}^d}{\text{sup}}|f(x)-f^{\prime }(x)|=\underset{x\in {[0,1]}^d}{\text{sup}}|\sum _m{\psi }_m\left(f(x)-{\mathit{Q}}_m^{\prime }(x)\right)|\le \sum _m\underset{x\in \left\{|x_k-\frac{m_k}{N}|<\frac{1}{N},\forall k\right\}}{\text{sup}}|f(x)-{\mathit{Q}}_m^{\prime }(x)|\le 2^d\underset{x\in \left\{|x_k-\frac{m_k}{N}|<\frac{1}{N},\forall k\right\}}{\text{max}}|f(x)-{\mathit{Q}}_m(x)|+2^d\underset{x\in \left\{|x_k-\frac{m_k}{N}|<\frac{1}{N},\forall k\right\}}{\text{max}}|{\mathit{Q}}_m^{\prime }(x)-{\mathit{Q}}_m(x)|$$

$$\le \frac{2^d{d}^n}{n!}{\left(\frac{1}{N}\right)}^n\underset{n:|n|=n}{\text{max}}ess{\text{sup}}_{x\in {[0,1]}^d}|{\mathit{D}}^{n}f(x)|+2^d\sum _{n:|n|<n}\text{max}\left(|\frac{{\mathit{D}}^{n}f}{n!}-{\gamma }_{m,n}|\right){\left(x-\frac{m}{N}\right)}^n\le \frac{2^d{d}^n}{n!}{\left(\frac{1}{N}\right)}^n{\left(\frac{1}{N}\right)}^n+\frac{2^d}{n}\left({\left(\frac{d}{N}\right)}^n+\dots +{\left(\frac{d}{N}\right)}^1{\left(\frac{d}{N}\right)}^{n-1}\right)\le 2^d{\left(\frac{d}{N}\right)}^n\left(1+\frac{1}{n!}\right)\le 2^{d+1}{\left(\frac{d}{N}\right)}^n$$

Therefore, if we choose $N\ge {\left(\frac{2^{d+1}{d}^n}{ϵ}\right)}^{\frac{1}{n}}$, for ϵ ∈ (0, 1), any $f\in {\mathcal{F}}_{d,n}$ can be approximated $f^{\prime }$ with an error bound ϵ. We rewrite

$$f^{\prime }(x)=\sum _{m\in {\{0,1,\dots ,N\}}^d}\sum _{n:|n|<n}{\gamma }_{m,n}{\psi }_m{\left(x-\frac{m}{N}\right)}^n=\sum _{m\in {\{0,1,\dots ,N\}}^d}\sum _{n:|n|<n}{\gamma }_{m,n}{f}_{m,n}^{\prime }(x),$$

where $f_{m,n}^{\prime }(x)={\psi }_m{\left(x-\frac{m}{N}\right)}^n$ is the linear combination of no more than dⁿ(N + 1)^d terms of $f_{m,n}^{\prime }(x)$. By Lemma 8, with $t=log\left(\frac{n{N}^n}{d^n}\right)$ we can use a sub-network of $t=log\left(\frac{n{N}^n}{d^n}\right)$ to represent the constant weights γ_m,n.Then, the remaining work is to construct $f_{m,n}^{\prime }(x)$. N Is chosen as 2^l, where l is some large integer, such that all $\frac{m_i}{N}$ can be obtained precisely in the way of Lemma 8.

The strategy of using a quadratic network to construct $f^{\prime }$ is illustrated in Figure 7. Table 1 presents the complexities of individual blocks of interest, which helps us to analyze the complexity of the whole network. Although the whole complexity can be expressed in an explicit way, here we use the $\mathcal{O}$ notation for clarity and simplicity. Let N_d and N_w be the network depth and the number of weights. According to Table 1, we have $N_d=\mathcal{O}(log(t)+log(N))$ and $N_w=\mathcal{O}\left(N^{d}log(t)+N^{d}log(N)\right)$. Because $t=log\left(\frac{n{N}^n}{d^n}\right)$ and $N={\left(\frac{2^{d+1}{d}^n}{ϵ}\right)}^{\frac{1}{n}}=\mathcal{O}\left({\left(\frac{1}{ϵ}\right)}^{\frac{1}{n}}\right)$ substituting them into the estimates results in

$$N_d=\mathcal{O}\left(log(log(\frac{1}{ϵ}))+log(\frac{1}{ϵ})\right)$$

$$N_w=\mathcal{O}\left(log\left(log\left(\frac{1}{ϵ}\right)\right){\left(\frac{1}{ϵ}\right)}^{\frac{d}{n}}+log\left(\frac{1}{ϵ}\right){\left(\frac{1}{ϵ}\right)}^{\frac{d}{n}}\right)$$

Figure 7:

Strategy of using a quantized quadratic network to implement $f^{\prime }(x)$

Table I:

Descriptions of different building blocks.

Block	Operation	Width	Depth
BN	$\text{Construct }\frac{1}{N},\dots ,\frac{N-1}{N}$	N	log(N)
${\mathit{B}}_N^n$	$\text{Construct }{\left(x_k-m_k\right)}^{n_k}$	n	log(n)
Bh	Implement h(x)	12	1
Bm	Multiply d terms	2d	log(2d)
Bγ	Construct γ	t	log(t)

As we know, the quantization of the network [38] is an effective and robust way for model compression, even a binarized network performs well in some meaningful tasks. However, the reason why quantization is robust and useful was rarely studied. Ding et al. [39] reported that to obtain an approximation error ϵ, the upper bound for a quantized ReLU network is delineated by

$$\mathcal{O}\left(\lambda \left(lo{g}^{\frac{1}{\lambda -1}+1}\left(\frac{1}{ϵ}\right)\right){\left(\frac{1}{ϵ}\right)}^{\frac{d}{n}}+log\left(\frac{1}{ϵ}\right){\left(\frac{1}{ϵ}\right)}^{\frac{d}{n}}\right),$$ (4)

where λ is the number of distinct weights. Because it is convenient for quadratic neurons to implement multiplication, our quantized ReLU quadratic network considerably reduces the upper bound to $\mathcal{O}\left(log\left(log(\frac{1}{ϵ})\right){(\frac{1}{ϵ})}^{\frac{d}{n}}+log(\frac{1}{ϵ}){(\frac{1}{ϵ})}^{\frac{d}{n}}\right)$. It is observed that $\mathcal{O}\left(log\left(log(\frac{1}{ϵ})\right){(\frac{1}{ϵ})}^{\frac{d}{n}}\right)$ is significantly lower than $\mathcal{O}\left(\lambda \left(lo{g}^{\frac{1}{\lambda -1}+1}\left(\frac{1}{ϵ}\right)\right){\left(\frac{1}{ϵ}\right)}^{\frac{d}{n}}\right)$ therefore the saving made by quantized quadratic network is almost in an order of $\mathcal{O}\left(\lambda \left(lo{g}^{\frac{1}{\lambda -1}+1}(\frac{1}{ϵ})\right){(\frac{1}{ϵ})}^{\frac{d}{n}}\right)$. More interesting, the complexity bound achieved by quantized quadratic networks is independent of the value of λ, as explained in our proof.

4. Discussions and Conclusion

The rational for the first theorem is easily understandable. Since it is commonly known that a conventional network with one hidden layer is a universal approximator, the best thing we can do to justify the quadratic network is to find a class of functions that can be more efficiently approximated by a quadratic network than with a conventional network. By the first theorem, the quadratic network is indeed more efficient than the conventional counterpart.

The previous studies demonstrate that any function of n–variables that is not constant along any direction cannot be well represented by a fully-connected ReLU network with no more than n – 1 neurons in each layer [16]. However, breaking this network width bound, our second theorem states that when a radial function is not constant in any direction, the network width 4 is sufficient for a ReLU quadratic network to approximate the function accurately. Since it is more convenient to represent a nonlinear function with more nonlinear neurons, the incapability of conventional networks can be compensated by quadratic networks. Theorem 2 implies a slimmer and/or sparser width-bounded universal approximator.

Our third theorem is inspiring that a general polynomial function can be exactly expressed by a quadratic network via ReLU activation in a novel way of data-driven network-based algebraic factorization. Although third theorem assumes the ReLU activation function, considering ReLU is arguably the most important activation function, it makes a great sense in illustrating the capacity of quadratic networks. Most importantly, Theorem 3 shows that quadratic networks are advantageous in terms of expressibility and efficiency. By calculating a quadratic function of input variables, quadratic neurons create second-order terms. The second-order terms serve as nonlinear building blocks that are different from non-linearity generated only through nonlinear activation, and clearly preferred from the perspective of the Algebraic Fundamental Theorem. Accurately forming a polynomial allows global computation and high efficiency as compared to the popular piece-wise approximation with conventional neurons in a deep network. In our quadratic network construction, both the depth and width are limited, while all previous universal approximators based on ReLU are either too wide or too deep to achieve a high approximation accuracy. This fact suggests that quadratic networks could potentially minimize the number of neurons and free parameters for superior machine learning performance. It is surprising that although every quadratic neuron possesses a number of parameters three times as many as that of the corresponding conventional neuron, quadratic networks can use less parameters than the conventional counterparts in important cases, such as the presentation of radial functions and the approximation to multivariate polynomials.

Interestingly, our results (Corollary 1 and Theorem 3) imply that quadratic networks are highly adaptive. For example, while a quadratic network can do piece-wise approximation by degenerating into conventional networks, a generic quadratic network is capable of approximating any function in a global way by implementing basic algebraic factors. Also, quadratic networks can mimick RBF networks on the “wavelet” scale, as indicated by Corollary 1.

As we know, deep learning generalizes well in many applications. Some researchers questioned that quadratic networks may tend to over-fitting because the added neural complexity may not be necessary. With our above proved theorems, we refute that since quadratic networks de facto are more compact and more effective than the conventional counterpart in major circumstances, quadratic networks could improve the generalization ability, according to Occam’s razor principle. Since general features are either quadratic or can be expressed as combinations or compositions of quadratic features, quadratic deep networks will be universal, instead of being restricted to the circumstances where only quadratic features are relevant.

In conclusion, we have analyzed the approximation ability of the quadratic networks and proved four theorems, suggesting a great potential of quadratic networks in terms of expressive efficiency, unique representation, compact structure and computational capacity. Future research will be focused on evaluating the performance of deep quadratic networks in typical applications.

Biography

Fenglei Fan received the Bachelor’s degree from the Harbin Institute of Technology before joining Dr. G. Wang’s Laboratory. He is currently pursuing the Ph.D. degree with the Department of Biomedical Engineering, Rensselaer Polytechnic Institute. His research interests include applied mathematics, machine learning, and medical imaging.