Predicting the Critical Number of Layers for Hierarchical Support Vector Regression

Abstract

Hierarchical support vector regression (HSVR) models a function from data as a linear combination of SVR models at a range of scales, starting at a coarse scale and moving to finer scales as the hierarchy continues. In the original formulation of HSVR, there were no rules for choosing the depth of the model. In this paper, we observe in a number of models a phase transition in the training error—the error remains relatively constant as layers are added, until a critical scale is passed, at which point the training error drops close to zero and remains nearly constant for added layers. We introduce a method to predict this critical scale a priori with the prediction based on the support of either a Fourier transform of the data or the Dynamic Mode Decomposition (DMD) spectrum. This allows us to determine the required number of layers prior to training any models.

1. Introduction

Many of the machine learning algorithms require the correct choice of hyperparameters to give the best description of the given data. One of the most popular methods for choosing hyperparameters is doing grid-search. In grid search, the model is trained for some points in hyperparameter space which are called a grid and then the model with the lowest error on the validation set is chosen. Other methods use alternative optimization algorithms, such as genetic algorithms. All these methods include training model and calculating the error multiple times, which can be very expensive if the hyperparameter space is large.

The purpose of this paper is to develop a method for determining hyperparameters for Support Vector Regression (SVR) with Gaussian kernels from time-series data only. Unlike other approaches for choosing hyperparameters, our method identifies a set of hyperparameters without needing to train models and perform a grid search (or executing some other hyperparameter optimization algorithm), thereby bypassing a potentially costly step. The proposed method identifies the inherent scale and complexity of the data and adapts the SVR accordingly. In particular, we give a method for determining the scale of Gaussian kernel by connecting it to the most important frequencies of the signal, as determined by either a Fourier transform or Dynamic Mode Decomposition. Thus we leverage classical and generalized harmonic analysis to inform the choice of hyperparameters in modern machine learning algorithms. A pertinent question is why not just use FFT? The answer is that HSVR are better suited to model strongly locally varying data whereas, for FFT to be efficient, the data need to possess some symmetry such as space or time translation.

Previous Work

There are many different approaches in tuning SVR hyperparameters for reducing the generalization error. Some popular methods of estimating generalization error are Leave-one-out (LOO) score and k cross-validation score. They are easy to implement, but for calculating those measures more models have to be trained for each combination of hyperparameters that need to be tested. This can be prohibitively expensive, so other error estimates, which are easier to calculate, were developed. These methods include the Xi-Alpha bound [1], the generalized approximate cross-validation [2], the approximate span bound [3], the VC bound [3], the radius-margin bound [3] or the quality functional of the kernel [4].

For choosing hyperparameters, the simplest method is to perform grid search over the space of hyperparameters and then choose the ones with the lowest error estimation. However, grid search suffers from the curse of dimensionality, scaling exponentially with the dimension of the hyperparameter configuration space. Other work in hyperparameter optimization (HPO) seek to mitigate this problem. Random search samples the configuration space and can serve as a good baseline [5,6]. Bayesian optimization techniques to find optimal hyperparameters, often using Gaussian processes as surrogate functions, offers a more more computational efficient algorithm than grid search or random search requiring fewer attempts to find the optimal parameters [6]. However, Bayesian optimization in this form requires more computational resources [6].

There are also gradient-based approaches [7,8,9]. There are also several derivative-free optimization methods. For example, in [10], a pattern search methodology for hyperparameters, the parameter optimization method based on simulated annealing [11], Bayesian method based on MCMC for estimating kernel parameters [12], and also methods based on Genetic algorithms [13,14,15].

What all of these methods have in common is that they require training a model and evaluating the error for each parameter set that needs to be tried. This can be quite expensive, both in time and computational resources, if there are a lot of data or we want to train multiple models at the same time. They are iterative, which means that we usually do not know how many models will be trained before getting an estimation of the best hyperparameters. There are methods, such as early cutoff, that try to reduce these burdens, but fundamentally a model needs to be trained for each set of candidate hyperparameters.

In contrast, our approach gives a set of hypeparameters without ever computing a single model. Specifically, in this paper, we are interested in modeling multiscale signals with a linear combination of SVRs. Two fundamental questions are (1) how many SVR models are going to be used? and (2) what should the scale be for each SVR model. Clearly the dimension of the configuration space for the scale parameters is conditional on the number of layers. Our methods are built off fast spectral methods like fast Fourier transform and Dynamic Mode Decomposition and return both the number of layers (number of SVR models) and the scales for each SVR without ever having to train a model. Bypassing the expensive step of computing a model for each candidate set of hyperparameters can lead to dramatic savings in computational time and resources.

2. Methods

2.1. Support Vector Regression

Support Vector Machines (SVM) have been introduced in [16] as a method for classification. The goal was to set hyperplane between classes, where hyperplane is defined by linear combination of a subset of a training set, called Support Vectors. The problem is formulated as a quadratic optimization problem that is convex so it has a unique solution. The problem with SVM is that it can only find a linear boundary between classes which is often not possible. The trick is to map the training data to a higher dimensional space and then use kernel functions to represent the inner product of two data vectors projected onto this space. Then only inner products are needed for finding the best parameters. The advantage of this approach is that we can implicitly map data onto infinite dimensional space. One of the most used kernels is Gaussian function defined by

$$k(x,x^{\prime })=exp\left(-\frac{\parallel x-x^{\prime }{\parallel }_2^2}{{\sigma }^2}\right)=exp(-\gamma \parallel x-x^{\prime }{\parallel }_2^2),$$ (1)

where x and $x^{\prime }$ are n-dimensional feature vectors, ${\sigma }^2$ is the variance of the Gaussian function, and $\gamma =1/{\sigma }^2$ is the scale parameter that is usually specified as an input parameter to SVR toolboxes.

The SVM approach has been extended in [17] to regression problems and it is called Support Vector Regression (SVR). Let $S=\{(x_1,y_1),\dots ,(x_n,y_n)\}$ be the training set, where $x_i$ is vector in input space $X\subset {\mathbf{R}}^D$ and $y_i\in \mathbf{R}$ desired output. The aim of SVR is to find a regression function $f:X\to \mathbf{R}$:

$$f\left(x\right)={\omega }^T\Phi \left(x\right)+b,$$ (2)

where $\omega$ is the weight vector and $\Phi$ is a mapping of the data points to a higher-dimensional space and b is threshold constant. $\omega$ and b can be found by solving following optimization problem:

$$\begin{array}{cc}\hfill & \underset{\omega ,b}{min}\frac{1}{2}{\omega }^T\omega +C{\Sigma }_{i=1}^n{E}_i^{+}+C{\Sigma }_{i=1}^n{E}_i^{-},\hfill \end{array}$$ (3)

$$\begin{array}{cc}\hfill & s.t{y}_i-{\omega }^T\Phi \left(x\right)-b\le ϵ+E_i^{+}\hfill \end{array}$$ (4)

$$\begin{array}{cc}\hfill & {\omega }^T\Phi \left(x\right)+b-y_i\le ϵ+E_i^{-}\hfill \end{array}$$ (5)

$$\begin{array}{cc}\hfill & E_i^{+},E_i^{-}\ge 0,(i=1,\dots ,n).\hfill \end{array}$$ (6)

The parameter $ϵ$ determines width of tube around the regression curve and points inside it do not contribute to the loss function. Parameter C adjusts the trade off between the regression error and regularization, $E^{+}$ and $E^{-}$ are slack variables for relaxing approximation constraints and measure the distance of each data point from the $ϵ$ tube. In practice, the dual problem is solved, which can be written as:

$$\begin{array}{cc}\hfill & \underset{{\alpha }^{+},{\alpha }^{-}}{max}-\frac{1}{2}({\alpha }^{+}-{\alpha }^{-})K({\alpha }^{+}-{\alpha }^{-})-ϵ\sum _{i=1}^n({\alpha }_i^{+}+{\alpha }_i^{-})+\sum _{i=1}^n{y}_i({\alpha }_i^{+}-{\alpha }_i^{-}),\hfill \end{array}$$ (7)

$$\begin{array}{cc}\hfill & s.t\sum _{i=1}^n({\alpha }_i^{+}-{\alpha }_i^{-})=0\hfill \end{array}$$ (8)

$$\begin{array}{cc}\hfill & {\alpha }_i^{+},{\alpha }_i^{-}\in [0,C],(i=1,\cdots ,n).\hfill \end{array}$$ (9)

where ${\alpha }^{+},{\alpha }^{-}\in {\mathbf{R}}^n$ are the dual variables and $K\in {\mathbf{R}}^{n\times n}$ is the kernel matrix evaluated from a kernel function, $K_{ij}=k(x_i,x_j)$ where $k(x,x^{\prime })$ is the kernel function. Solving that problem, the regression function becomes:

$$f\left(x\right)=\sum _{i=1}^n({\alpha }_i^{+}-{\alpha }_i^{-})k(x,x_i)+b.$$ (10)

Coefficients satisfy following conditions:

$$|{\alpha }_i^{+}-{\alpha }_i^{-}|=\left\{\begin{array}{cc}0\hfill & \parallel y_i-f\left(x_i\right)\parallel <ϵ\hfill \\ \in (0,C)\hfill & \parallel y_i-f\left(x_i\right)\parallel =ϵ\hfill \\ C\hfill & \parallel y_i-f\left(x_i\right)\parallel >ϵ\hfill \end{array}\right.$$

Data points for which $|{\alpha }_i^{+}-{\alpha }_i^{-}|$ is non-zero are called Support Vectors.

2.2. Dynamic Mode Decomposition (Dmd)

Dynamic Mode Decomposition (DMD) was introduced in [18] as a method for extracting dynamic information from flow fields that are either generated by numerical simulation or measured in physical experiment. Rowley et al. connected DMD with Koopman operator theory [19].

Let the data be expressed in a series of snapshots, given by matrix ${\mathbf{V}}_{\mathbf{1}}^{\mathbf{N}}$:

$${\mathbf{V}}_{\mathbf{0}}^{\mathbf{N}}=\{v_0,v_1,\cdots ,v_N\},$$ (11)

where $v_i\in {\mathbb{R}}^m$ stands for the i-th snapshot of the flow field. We assume there exists a linear mapping $\mathbf{A}$ which relates each snapshot $v_i$ to next one $v_{i+1}$,

$$v_{i+1}=\mathbf{A}{v}_i,$$ (12)

and that this mapping is approximately same during each sampling interval, so we approximately have

$${\mathbf{V}}_{\mathbf{0}}^{\mathbf{N}}=\{v_0,\mathbf{A}{v}_0,\cdots ,{\mathbf{A}}^N{v}_0\}.$$ (13)

We assume that characteristics of the system can be described by the spectral information in $\mathbf{A}$. This information is extracted in a data-driven manner using ${\mathbf{V}}_{\mathbf{0}}^{\mathbf{N}}$. The idea is to use ${\mathbf{V}}_{\mathbf{0}}^{\mathbf{N}}$ to construct an approximation of $\mathbf{A}$. In [18], this was done as follows. Define $\mathbf{X}={\mathbf{V}}_{\mathbf{0}}^{\mathbf{N}-\mathbf{1}}$ and $\mathbf{Y}={\mathbf{V}}_{\mathbf{1}}^{\mathbf{N}}$. Let the singular value decomposition of $\mathbf{X}$ be $\mathbf{X}=\mathbf{U}\Sigma {\mathbf{V}}^{*}$. Then the representation of a compression of $\mathbf{A}$ is defined as

$$\tilde{\mathbf{A}}={\mathbf{U}}^{*}\mathbf{Y}\mathbf{V}{\Sigma }^{+}$$ (14)

where ${\Sigma }^{+}$ is the Moore–Penrose pseudo-inverse of $\Sigma$. Note that (14) is analytically equivalent to ${\mathbf{U}}^{*}\mathbf{AU}$, the compression of $\mathbf{A}$ to the subspace spanned by the columns of $\mathbf{X}$, if $\mathbf{Y}=\mathbf{AX}$. Eigenvectors and eigenvalues of $\mathbf{A}$ are approximations of eigenvectors and eigenvalues of the Koopman operator.

The Koopman operator is an infinite dimensional, linear operator K that acts on all scalar functions g on M as

$$Kg\left(x\right)=g\left(f\right(x\left)\right),$$ (15)

where f is a dynamical system such that $x_{k+1}=f\left(x_k\right)$. Let ${\lambda }_j$ and ${\varphi }_j$ be eigenvalues and eigenfunctions, i.e.,

$$K{\varphi }_j\left(x\right)={\lambda }_j{\varphi }_j\left(x\right).$$ (16)

Let $g\left(x\right)$: $M\to {\mathcal{R}}^p$ be vector of any quantities of interest. If g lies in span of ${\varphi }_j$, then it can be written as

$$g\left(x\right)=\sum _{j=1}^{\infty }{\varphi }_j\left(x\right)v_j.$$ (17)

Then we can express $g\left(x_k\right)$ as

$$g\left(x_k\right)=K^{k}g\left(x_0\right)=K^k\sum _{j=1}^{\infty }{\varphi }_j\left(x_0\right)v_j=\sum _{j=1}^{\infty }{\lambda }_j^k{\varphi }_j\left(x_0\right)v_j.$$ (18)

The Koopman eigenvalues, ${\lambda }_j$ characterize the temporal behaviour of the corresponding Koopman mode $v_j$, the phase of ${\lambda }_j$ determines its frequency, and the magnitude determines the growth rate.

2.3. Hierarchical Support Vector Regression

Classical SVR models which use kernels of a single scale have difficulties approximating multiscale signals. For example, consider the function $f\left(x\right)=x+sin\left(2\pi x^4\right)$, for $x\in [0,2]$, whose graph is given in Figure 1. This function has a continuum of scales. Figure 2 highlights the difficulties that classical single-scale SVR has in modeling such signals. Using too large a scale $\sigma$, the detailed behavior of the data set is not captured [20]; such a model may, however, be useful for a coarse-scale extrapolation outside the training set. Models employing a very small scale $\sigma$ can capture the training set in detail. However, this makes them very sensitive to noise in the training and severely limits the model’s ability to generalize outside the training set [20].

Figure 1.

Multiscale example function: $f\left(x\right)=x+\sin \left(2\pi x^4\right)$.

Figure 2.

SVR results with different scale of Gaussian kernel $\sigma$ = 0.5, 0.05, 0.005, 0.0005. A large kernel provides smooth regression, but cannot reconstruct the details. A small kernel overfits, is unable to generalize, and can be sensitive to noise.

In [20], the authors introduced a multiscale variant of Support Vector Regression which they termed Hierarchical Support Vector Regression (HSVR). The idea behind HSVR is to train multiple SVR models, organized as a hierarchy of layers, each with different scale $\sigma$. The HSVR model is then a sum of those individual models, which we write as

$$S\left(x\right)=\sum _{\ell =0}^L{a}_{\ell }(x;{\sigma }_{\ell },ϵ),$$ (19)

where L is number of layers and $a_{\ell }(x;{\sigma }_{\ell },ϵ)$ is SVR model on layer ℓ with Gaussian kernel with parameter ${\sigma }_{\ell }$. Each SVR layer realizes a reconstruction of the target function at a certain scale. Training the HSVR model precedes from coarser scales to finer scales as follows. Let ${\sigma }_0>{\sigma }_1>{\sigma }_L>0$ be specified. For ${\sigma }_0$, an SVR model $a_0(x;{\sigma }_0,ϵ)$ is trained on the signal $f\left(x\right)$ (the 0-th residual) and the residual $r_1\left(x\right)=f\left(x\right)-a_0(x;{\sigma }_0,ϵ)$ is computed. We then proceed inductively for $\ell \ge 1$. Given the residual $r_{\ell }\left(x\right)$, train a model $a_{\ell }(x;{\sigma }_{\ell },ϵ)$ to approximate it and compute new residual $r_{\ell +1}\left(x\right)=r_{\ell }\left(x\right)-a_{\ell }(x;{\sigma }_{\ell },ϵ)$. The $(L+1)$-th residual is then

$$r_{L+1}\left(x\right)=f\left(x\right)-a_0(x;{\sigma }_0,ϵ)-\cdots -a_L(x;{\sigma }_L,ϵ)=f\left(x\right)-S\left(x\right).$$ (20)

Graphically, the process looks like Figure 3.

Figure 3.

Flowchart of the HSVR modeling process. The input data is first used to compute the scales used for the HSVR model (see Algorithms 1, 2 and 4). At layer 0, an SVR model is trained at the coarsest scale ${\gamma }_0$. The residual is computed by taking the difference between the signal and the model. This residual is then modeled with an SVR model at the next coarsest scale ${\gamma }_1$. A new residual is computed by taking the difference of the old residual and the ${\gamma }_1$ SVR model. This process is repeated until the pre-computed scales are exhausted.

The HSVR model contains a number of hyperparmeters that need to be specified, namely, $ϵ$, $C_{\ell }$, the number of layers L to take, and the specific scales ${\sigma }_{\ell }$ for those layers. In [20], the authors chose to use exponential decay relationship between the scales, such as ${\sigma }_{\ell +1}={\sigma }_{\ell }/\sqrt{decay}$. This, however, still leaves the critical choices of ${\sigma }_0$ and L unspecified. In all of our experiments that follow, we choose $ϵ$ to be 1 percent of the variation of the signal

$$ϵ=0.01(maxf(x)-minf(x\left)\right).$$ (21)

For each layer, $C_{\ell }$ was specified as

$$C_{\ell }=5(\underset{i}{max}{r}_{\ell -1}\left(x_i\right)-\underset{i}{min}{r}_{\ell -1}\left(x_i\right)).$$ (22)

which was the choice given in [20]. Additionally, the $decay$ variable was chosen to be 2 so that ${\sigma }_{\ell +1}={\sigma }_{\ell }/\sqrt{decay}$. Python’s scikit-learn library [21] was used throughout this paper. Its implementation of SVR requires the input parameter $\gamma$, rather than $\sigma$ from (1). These two parameters are related as $\gamma =1/{\sigma }^2$. Thus, the equivalent decay rate of the input parameter is

$${\gamma }_{\ell +1}={\gamma }_{\ell }*decay={\gamma }_{\ell }*2.$$ (23)

In the next sections, we take up the task of efficiently determining the hyperparameters L and ${\sigma }_0$ without the expensive step of performing grid search or training any models.

3. Predicting the Depth of Models

In this section, we will describe how error changes while training HSVR and we will provide methods of estimating the number of layers of such hierarchical model.

3.1. Phase Transition of the Training Error

For the rest of the paper, we have training data set $\left\{\right(x_{train},y_{train}\left)\right\}$ and testing data set $\left\{\right(x_{test},y_{test}\left)\right\}$ and the model $S\left(x\right)={\sum }_{\ell =1}^L{a}_{\ell }(x;{\sigma }_l)$, as in (19). Let $S_i\left(x\right)={\sum }_{\ell =1}^i{a}_{\ell }(x;{\sigma }_l)$ for $i=1,\cdots ,L$. For each layer, the HSVR (prediction) error is calculated as

$$r_i=\underset{\left\{\right(x_{test},y_{test}\left)\right\}}{max}|y_{test}-S_i\left(x_{pred}\right)|=\underset{\left\{\right(x_{test},y_{test}\left)\right\}}{max}\left|y_{test}-y_{i,pred}\right|,$$ (24)

where we have denoted $y_{i,pred}=S_i\left(x_{pred}\right)$. Therefore, $r_i$ denotes the maximum error between the true signal at the test points $x_{pred}$ and an HSVR model with i layers. By examining the change of the values $r_i$, it can be seen that there is a sudden drop in error, almost until tolerance $ϵ$, so that adding additional layers cannot reduce error any more. We will call that value the critical-sigma and denote it with ${\sigma }_c$. The drop in error can be seen in Figure 4.

Figure 4.

Residuals while training HSVR model with decreasing $\sigma$ as shown on x-axis. Both HSVR models exhibit a phase transition in their approximation error.

3.2. Critical Scales: Intuition and the Fourier Transform

Since the SVR models are fitting the data with Gaussians, a heuristic for choosing the scale will be shown on basic examples of periodic function and then expanded on more general cases in next sections.

Let $f\left(x\right)=sin\left(2\pi fx\right)$, where x is in meters and f is the frequency in cycles per meter. The frequency will be related to the scale, $\sigma$, of the Gaussian:

$$G_{\sigma }(x,c)=exp\left(-\frac{{\parallel x-c\parallel }_2^2}{{\sigma }^2}\right).$$ (25)

To relate the maximum frequency, f, to the scale, $\sigma$, we use the heuristic that we want 3 standard deviations of the gaussian to be half of the period. That is we want

$$3\sigma =\frac{T}{2},$$ (26)

Since $T=\frac{1}{f}$, then

$$3\sigma =\frac{1}{2f}.$$ (27)

Figure 5 justifies this heuristic.

Figure 5.

Fitting Gaussians to half-periods of sinusoids using the heuristic (27).

3.3. Determining Scales with FFT

We assume that we can learn HSVR model with scales $\sigma$ corresponding to important frequencies in the FFT of the signal. The assumption is that these will give an information how to train the model. If in the FFT of the signal, there are a lot of frequencies, we assume that we need more HSVR layers and each will learn the most dominant scale at this level.

Our assumption is that required number of scales in the HSVR model can be determined by the data. If there are a lot of frequencies in FFT of the signal related to relatively big coefficients, we can assume that signal is more challenging for single SVR to model it. As in [20], we refine the scales. Here we use exponential decay and use only frequencies for which corresponding coefficients in FFT are large enough in order to avoid numerical problems and adding insignificant frequencies. The procedure is summarized in Algorithm 1. The Filtering scales algorithm is summarized in Algorithm 2. The model building is outlined in Algorithm 3.

Algorithm 1 Determining scales of HSVR model

Input:$(x_i,y_i),i=0,\cdots n-1$, where $x_i$ are equidistant points in domain and $y_i$ values of function we want to model 1: $dx=x\left[1\right]-x\left[0\right]$ 2: $freq=$ FFT frequencies of the signal 3: $C=FFT\left(y\right)$ 4: $C=C/max\left(\right|C\left|\right)$ # normalize coefficients respect to $L1$-norm 5: $fre{q}_{support}=freq\left[\right|C|>0.01]$ 6: $scales=dx/(6*fre{q}_{support})$ 7: $scales$ = sort scales in descending order 8: $scales=filter\left(scales\right)$ 9: return $scales$

Algorithm 2 Filtering scales

Input: $scales$ = vector of scales determined from FFT, $decay$ 1: $scale{s}_{filtered}=\left[scales\left[0\right]\right]$ 2: $n=len\left(scales\right)$ 3: for i in range(1,n): 4:    if $scale{s}_{filtered}[-1]/scales\left[i\right]>=decay$: 5:       $scale{s}_{filtered}.append\left(scales\left[i\right]\right)$ 6: return: $scale{s}_{filtered}$

Algorithm 3 Train HSVR

Input: $(x_i,y_i),i=0,\cdots n-1$ scales (output of Algorithm 1) 1: $ϵ=0.01({max}_i\left(y_i\right)-{min}_i\left(y_i\right))$ 2: $r_0=y=[y_0,\cdots ,y_{n-1}]$ 3: model = $\left[\right]$ # comment: empty list to hold the SVR model at each layer 4: $m=len\left(scales\right)$ # comment: number of HSVR layers 5: for i in range(0, m): 6:    ${\sigma }_i$ = scales[i] 7:    $C_i=5(max\left(r_i\right)-min\left(r_i\right))$ 8:    $sv{r}_i$ = fitted SVR on $(x,r_i)$ with parameters ${\sigma }_i$, $C_i$ and tolerance $ϵ$ 9:    predictions = $sv{r}_i$.predict(x) 10:    $r_{i+1}=r_i-predictions$ 11:    model.append($sv{r}_i$) 12: return: model

3.4. Determining Scales with Dynamic Mode Decomposition

Let $(x_i,y_i)$, $i=0,\cdots ,n$ be training set. Output data $y_i$ is organized into Hankel matrix Y with M rows and N columns $(M>N)$. We will extract relevant frequencies with Hankel DMD, described in [22], using the DMD_RRR described in [23]. The DMD_RRR algorithm returns the residuals (rez) which determine how accurately the eigenvalues are computed, the eigenvalues ($\lambda$), and the eigenvectors (Vtn). As before, suppose we have a signal

$$f\left(x_n\right)=f\left(n\delta x\right),(n=0,\dots ,N-1).$$ (28)

We map this scalar-valued functions into a higher-dimensional space by delay-embedding. We choose $M<N$. The delay-embedding of the signal is the matrix

$$H=\left[\begin{array}{cccc}f\left(x_0\right)& f\left(x_1\right)& \cdots & f\left(x_{N-M}\right)\\ f\left(x_1\right)& f\left(x_2\right)& \cdots & f\left(x_{N-M-1}\right)\\ ⋮& ⋮& \ddots & ⋮\\ f\left(x_{M-1}\right)& f\left(x_M\right)& \cdots & f\left(x_{N-1}\right)\end{array}\right],$$ (29)

so that for $j=0,\dots ,N-M$

$$H[:,j]=\left[\begin{array}{c}f\left(x_j\right)\\ f\left(x_{j+1}\right)\\ ⋮\\ f\left(x_{M+j-1}\right).\end{array}\right]$$ (30)

We define a generalized Hankel matrix as an $m\times n$ rectangular matrix whose entries $H_{i,j}$ satisfy

$$H_{i,j}=H_{i+k,j-k}$$ (31)

for all indices such that $0\le i,i+k\le m-1$ and $0\le j,j-k\le n-1$, where $k\in \mathbb{Z}$. In simpler terms, a generalized Hankel matrix is a rectangular matrix that is constant on anti-diagonals. A generalized Hankel matrix can also be thought of a submatrix of a larger, regular Hankel matrix. Clearly, the delay-embedding of the scalar signal, Equaion (29), is an example of a generalized Hankel matrix which is why it was denoted as H.

The input matrices for DMD algorithms will be $\mathbf{X}$ and $\mathbf{Y}$, where $\mathbf{X}$ is the first $N-M$ columns of $\mathbf{H}$ and $\mathbf{Y}$ is the last $N-M$ columns. Frequencies are calculated as follows:

$${\omega }_i=\frac{1}{2\pi i}ln\left(\frac{{\lambda }_i}{|{\lambda }_i|}\right);$$ (32)

i.e., we just scale the DMD eigenvalues so that they are on the unit circle and then extract frequency of the resulting complex exponential, $exp\left(i2\pi (f_{\lambda }\Delta x)\right)=\lambda /\left|\lambda \right|$. Let $\Omega$ contain the values ${\omega }_{\lambda }=f_{\lambda }\Delta x$. These are directly analogous to the frequencies computed using FFT.

We replace the support of the FFT with the support of the DMD frequency as follows. We will take all the values of $\Omega$ whose corresponding residual is less than some specified tolerance, tol, and whose corresponding mode’s norm is greater than some percentage, $\eta$, of the total power of the modes. In other words, we only consider the frequencies which were calculated accurately enough and whose modes give a significant contribution to the signal; $\parallel mode\left[j\right]\parallel$ is analogous to the modulus of a FFT coefficient.

For each mode $Vtn[:,i]$ the energy is $c_i$ for which ${\parallel Y[:,0]-c*Vtn[:,i]\parallel }_2$ is minimal. Total power is defined as

$$T={\left(\sum _{v_i}{c}_i^2\right)}^{\frac{1}{2}}.$$ (33)

Like in determining ${\sigma }_c$ using FFT, the frequency support of $S_{DMD}$ is defined as

$$S_{DMD}=\{{\omega }_i:rez\left[i\right]<tol,|c_i|>\eta T\},$$ (34)

where $rez\left[i\right]$ is residual corresponding to the i-th mode. These considerations can be summarized in Algorithm 4.

Algorithm 4 Estimating scales from data using Hankel DMD

Input: time step: $\Delta x$, time series f: $f\left[n\right]=f(\Delta xn)$, length of time series vector: N, tolerance for support: $tol$, $\eta$, M: number of rows of Hankel matrix 1: H = Hankel matrix made from f with M rows and N-M columns 2: $rez,\lambda ,Vtn=DMD\_RRR\left(H\right)$ 3: $\omega =\frac{1}{2\pi i}ln\left(\frac{\lambda }{\left|\lambda \right|}\right)$ 4: $T=0$ 5: for i = 0 to N − 1: 6:    $E\left[i\right]=|\langle Y[:,0],Vtn[:,i]\rangle |$ 7:    $T=T+E{\left[i\right]}^2$ 8: $T=\sqrt{T}$ 9: $S_{DMD}=\left[\right]$ 10: for i = 0 to N − 1: 11:    if $rez\left[i\right]<tol$ and $energy\left[i\right]>\eta T$ 12:       $S_{DMD}.append\left(\omega \left[i\right]\right)$ 13: return ${\sigma }_{DMD}=\frac{\Delta x}{6S_{DMD}}$

4. Results

In this section, results of our methods are provided. We demonstrate our methods on explicitly defined function, system of ODEs, and finally on vorticity data from a fluid mechanics simulations. In all cases,

$$ϵ=0.01\left(\underset{\left\{y_{train}\right\}}{max}{y}_{train}-\underset{\left\{y_{train}\right\}}{min}{y}_{train}\right)$$ (35)

and the error is calculated as in (24).

4.1. Explicitly Defined Functions

We model explicitly defined functions $f\left(x\right)$ on $[0,2]$. The dataset consists of 2001 equidistant points in that interval, where every other point is used for the training set. The error is calculated as the maximum absolute value of difference between prediction and actual value (Equation (24)). Results are summarized in Table 1. The parameter $ϵ$ is specified as above. Each method predicts a certain number of layers required to push the model error close to $ϵ$. As seen in the table, both the FFT and DMD approaches produce models that give similar errors (near $ϵ$). Although the DMD approach results in models with less layers in general, it fails for polynomials and the exponential function. This is because we normalize eigenvalues to the unit circle. An extension of this could use the modulus $\left|\lambda \right|$ as well as $\omega$ in $e^{\lambda +i\omega }$.

Table 1.

Results for explicitly defined functions, when using scales determined from FFT with decay 2 and $ϵ$ given by (35), for $e^x$ DMD did not output frequencies different from 0 (entries denoted by * in the corresponding row).

Function	$\mathit{ϵ}$	Predicted # of Layers (FFT)	Error (FFT)	Predicted # of Layers (DMD)	Error (DMD)
$sin\left(2\pi x\right)$	0.02	1	0.02	1	0.02
$sin\left(20\pi x\right)$	0.0199	1	0.021	1	0.02
$sin\left(200\pi x\right)$	0.019	1	0.093	1	0.097
$100sin\left(20\pi x\right)$	1.99	1	2	1	2.01
40 cos(2$\pi x)$	0.8	1	0.8	1	0.8
100 cos(20$\pi x)$	2	1	2.03	1	2
sin(2$\pi x^2)$	0.0199	5	0.02	1	0.02
$x+x^2+x^3$	0.14	2	0.14	1	8
$e^x$	0.063	1	0.064	*	*
$x+sin\left(2\pi x^4\right)$	0.03	7	0.037	1	0.034
$cos\left(2\pi x\right)+sin\left(20\pi x\right)$	0.0397	2	0.0404	2	0.042
$cos\left(20\pi x\right)sin\left(15\pi x\right)$	0.02	2	0.021	2	0.022
$cos{\left(32\pi x\right)}^3$	0.0199	1	0.022	2	0.022
$sin\left(13\pi x\right)+sin\left(17\pi x\right)+$ $sin\left(19\pi x\right)+sin\left(23\pi x\right)$	0.076	1	0.077	1	0.077
$sin\left(50\pi x\right)sin\left(20\pi x\right)cos\left(15\pi x\right)$	0.0187	3	0.02	2	0.02
$sin\left(40\pi x\right)cos\left(10\pi x\right)+$ $3sin\left(20x\right)sin\left(40x\right)$	0.064	5	0.065	3	0.066
$sin\left(2x\right)cos\left(32x\right)$	0.0198	5	0.02	1	0.02

4.2. ODE’s

We will demonstrate the methods on modeling solutions of the Lorenz system of ODE

$$\begin{array}{cc}\hfill x\left(t\right)& =-10x+10y,\hfill \end{array}$$ (36)

$$\begin{array}{cc}\hfill y\left(t\right)& =28x-y-xz,\hfill \end{array}$$ (37)

$$\begin{array}{cc}\hfill z\left(t\right)& =xy-\frac{8}{3}z,\hfill \end{array}$$ (38)

with initial conditions $x_1\left(0\right)=1.0$, $x_2\left(t\right)=1.0$, $x_3\left(t\right)=1.0$, on [0, 10]. The system evolved solved for 500 equidistant time steps which was used for training set. A solution consisting of 2000 equidistant time steps is then used for test set. Each $x\left(t\right)$, $y\left(t\right)$, $z\left(t\right)$ were regarded as separate signals to model. All hyperparameters are determined as before. Results are summarized in Table 2. For both systems, we can see that error drops nearly to $ϵ$, but the DMD approach results in models with less layers.

Table 2.

Results of training HSVR with scales used from FFT and DMD on Lorenz system.

Function	$\mathit{ϵ}$	Predicted # of Layers (FFT)	Error (FFT)	Predicted # of Layers (DMD)	Error (DMD)
x(t)	0.314	6	0.325	2	0.324
y(t)	0.408	6	0.469	2	0.469
z(t)	0.468	5	0.485	2	0.494

4.3. Vorticity Data

This data was provided by Georgia Institute of Technology [24,25]. It contains information about vorticity of a fluid field is some computational box. For each point in space, we want to model vorticity at that point as time evolves. Vorticity in every other time step is used for training and we test our models on rest of the time steps. We analyzed doubly periodic and non periodic data.

4.3.1. Doubly Periodic Data

This data was generated assuming doubly periodic boundary conditions. The dataset contains information about the vorticity on $128\times 128$ equidistant space-grid on [0, 0.1016]². There are in total 1201 snapshots with time step $dt=0.03125$. This results in a tensor of dimensions $128\times 128\times 1201$. For each fixed point in space, there is a signal with 1201 time steps. A few examples of such signals are shown in Figure 6.

Figure 6.

Examples of vorticity at 3 different space-points for fluid simulations with doubly periodic boundary conditions.

For each of these signals, every other point in time is used for training the HSVR model, which results in total of $128\times 128$ models. Results are summarized in Table 3. For each model, the error is calculated as in (24) with $i=L$, where L is number of layers. Since $ϵ$ is given by (35), i.e., 1% of the range of the training data, the ratio error/$ϵ$ gives a measure of error in percentages of range of signal. A ratio of 2 would imply that the maximum error of the model over the test set was only 2% of the range of the test data; i.e.,

$$2=\frac{error}{ϵ}\iff error=2ϵ=0.02\left(\underset{\left\{y_{train}\right\}}{max}{y}_{train}-\underset{\left\{y_{train}\right\}}{min}{y}_{train}\right).$$ (39)

Table 3.

Results after training HSVR on doubly periodic vorticity data.

	$\mathit{ϵ}$	Predicted # of Layers (FFT)	Error (FFT)	Predicted # of Layers (DMD)	Error (DMD)
min	0.0199	6	0.038	1	0.0399
mean	0.0354	8	0.082	3	0.148
max	0.0488	9	0.284	5	2.463

Histograms of these ratios for both FFT and DMD are shown in Figure 7. Most models produced on error close the $ϵ$ threshold (a perfect match would give a ratio of 1).

Figure 7.

Histograms of $error/ϵ$ for models trained on vorticity data with doubly periodic boundary conditions. $error$ is the model error given by (24) (with $i=L$) and $ϵ$ is given by (35). There were ${128}^2=\mathrm{16,384}$ total models trained. The count on the vertical axis is the number of models that fell into the corresponding bin.

4.3.2. Non Periodic Data

The data contains information about vorticity on $359\times 279$ equidistant space-grid on with step $dx=dy=0.05$. No assumption of periodicity of boundary condition was made. There are in total 1000 snapshots with time step $dt=1$ ms. This results in tensor $359\times 279\times 1000$. Similar to the dataset with periodic boundary conditions, the HSVR model is trained for each fixed point in space, which results in $359\times 279$ models. The error’s and $ϵ$’s are calculated as before. A few examples of such signals are in Figure 8. Results are summarized in Table 4 and a histogram of ratios error/$ϵ$ for FFT and DMD are shown in Figure 9. For both doubly periodic and non periodic data, we can see that a large majority of the models have a ratio between 1 and 2 which means the maximum error of a majority models is $error\le 2ϵ=0.02({max}_{\left\{y_{train}\right\}}{y}_{train}-{min}_{\left\{y_{train}\right\}}{y}_{train})$ which is less than 2% of the range of the training data. For DMD approach, the majority of models have ratios less 10 which corresponds to a maximum error of $error\le 10ϵ=0.01({max}_{\left\{y_{train}\right\}}{y}_{train}-{min}_{\left\{y_{train}\right\}}{y}_{train})$, which is less than 10% of the range of the training data. From Table 3 and Table 4 we can see that DMD estimates less layers, but with bigger error after training.

Figure 8.

Examples of vorticity in 3 space-points for non periodic data.

Table 4.

Results after training HSVR on non periodic vorticity data.

	$\mathit{ϵ}$	Predicted # of Layers (FFT)	Error (FFT)	Predicted # of Layers (DMD)	Error (DMD)
min	0.019	5	0.0006	2	0.0006
mean	0.035	7	0.0975	3	0.197
max	0.048	9	0.667	7	6.137

Figure 9.

Histograms of $error/ϵ$ for models trained on vorticity data with non-periodic boundary conditions. $error$ is the model error given by (24) (with $i=L$) and $ϵ$ is given by (35). There were $359\times 279=\mathrm{100,161}$ total models trained. The count on the vertical axis is the number of models that fell into the corresponding bin.

5. Discussion

Both approaches (FFT and DMD) to estimating the number of layers required by HSVR to push the modeling error close to the $ϵ$ threshold show promise and allow computation of the HSVR depth a priori. While the DMD approach often predicted a smaller number of layers, usually with comparable error, our choice of unit circle normalization prevented it from performing well on functions without oscillation. Furthermore, the DMD algorithm itself comes from the dynamical systems community and requires that the domain of the signal (the x variables) be strictly ordered. For the explicitly defined functions we considered, there was a strict spatial ordering since the domains of the functions were intervals on the real line. For the vorticity data, the time signals at each spatial point were to be modeled and therefore the data points could be strictly ordered in time. For functions whose domain is multi-dimensional, say ${\mathbb{R}}^2$, there is no strict ordering and the DMD approach, as formulated here, would break down. The method based on the Fourier transform seems to have more promise in analyzing multivariable, multiscale signals, as it can compute multidimensional wave vectors.

A recent paper [26] also deals with the number of layers of a model and “scales”. However, in the mentioned paper, the authors are concerned with how far a signal or gradients will propagate through a network before dying. The scales they compute control how many layers the gradient or signal can propagate before they die. If the network is too deep the gradients go to zero before fully backpropagating through the network, resulting in an untrainable network. The result they compute is a fundamental characteristic of the network and is independent of the dataset. It does not matter what the input data is: constant, single scale, multiscale, etc, the scale parameters they compute are not affected.

Conversely, we are most focused on multiscale signals and tailoring the architecture to best represent such signals. The scales we compute are inherent properties of the dataset, not inherent properties of the network model. The length of the network adapts to the scales contained in the dataset.

It would be interesting in the future to combine the two methodologies, with the methods in [26] giving bounds on the number of layers and then adapting our techniques to analyze the inherent scales in the dataset. This could tell us whether a simple, fully, connected, feedforward network could adequately represent the signal or if something more complex was needed.

6. Conclusions

In this work we presented method for choosing hyperparameters for HSVR from time-series data only. We described two approaches, using FFT or DMD. We saw that estimating hyperaparameters with FFT results in a model with more layers and smaller error, whereas the DMD approach gave models that had less layers (and thus more efficient models).

Author Contributions

Conceptualization, R.M., M.F., Z.D. and I.M. (Igor Mezić); Data curation, I.M. (Iva Manojlović); Validation, I.M. (Iva Manojlović); Writing—original draft, I.M. (Iva Manojlović); Writing—review & editing, R.M., M.F., Z.D. and I.M. (Igor Mezić). All authors have read and agreed to the published version of the manuscript.

Funding

This research was partially supported by the DARPA contract HR0011-18-9-0033, the Air Force Office of Scientific Research contract FA9550-17-C-0012, and the Croatian Science Foundation grant IP-2019-04-6268.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not available.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.