LSSVR Model of G-L Mixed Noise-Characteristic with Its Applications

Abstract

Due to the complexity of wind speed, it has been reported that mixed-noise models, constituted by multiple noise distributions, perform better than single-noise models. However, most existing regression models suppose that the noise distribution is single. Therefore, we study the Least square $SVR$ of the Gaussian–Laplacian mixed homoscedastic ($GLM-LSSVR$) and heteroscedastic noise ($GLMH-LSSVR$) for complicated or unknown noise distributions. The ALM technique is used to solve model $GLM-LSSVR$. $GLM-LSSVR$ is used to predict short-term wind speed with historical data. The prediction results indicate that the presented model is superior to the single-noise model, and has fine performance.

1. Introduction

In practical applications, if the data are collected in a multi-source environment, the noise distribution is complex and unknown. Therefore, it is almost impossible for a single-noise distribution to clearly describe the real-noise [1]. $LSSVR$ is a method of $LR$ that implements a sum-of-squares error function together with regularization, thus controlling the bias–variance trade-off [2,3]. It is intended to find the concealed linear structures in the original data [4,5]. For the sake of transition from linear to nonlinear function, the following generalization can be made [6]: by mapping input vectors into a high-dimensional feature space H (H is Hilbert space) through some nonlinear-mapping, seek the solution of the optimization problem in space H. Using a suitable kernel function $K(•,•)$, nonlinear-mappings can be estimated by kernel $LSSVR$, which is an extended $LR$ with kernel techniques. In recent years, $LSSVR$ as a data-rich nonlinear forecasting tool has been increasingly welcomed [7], which is applicable in many different contexts [8,9,10], such as machine learning, optical character recognition, and especially wind speed/power forecasting.

Generally, the existing techniques used for wind-speed forecasting include: (i) physical; (ii) statistical (also called data-driven); and (iii) artificial intelligence (AI)-based methods. The physical models attempt to estimate wind flow around and inside the wind farm using physical laws governing the atmospheric behavior [11,12]. The statistical models seek the relationships between a set of explanatory variables and the on-line measured generation data, and the historical wind speed data recorded at the site are only used to establish the statistical model. We can model it in a variety of ways, including persistence method and auto-regressive model [13,14]. AI methods include artificial neural networks (ANNs) [15], deep learning [16], SVR machines [17,18], and the hybrid methods [19,20].

Suykens et al. [21,22,23] proposed least square support vector regression model with Gaussian noise ($LSSVR$, also known as kernel ridge regression ($KRR$)). Mixed-model based on multi-objective optimization [24,25], mixed-method based on singular spectrum analysis, firefly algorithm, and BP neural network predict wind speed with complicated noise [26], indicating that the mixed prediction method has the ability of powerful prediction. Mixed $LSSVR$ machine [27] is applied to forecast the wind speed noise, which improves performance of wind-speed prediction. $GLM-SVR$ [28] models fitted by Gaussian–Laplacian (G-L) mixed noise are developed, and good performance is obtained compared with the existing regression algorithm.

To solve the above problems, we study model $LSSVR$ of G-L mixed noise-characteristic for complex or unknown noise distribution. In this case, we construct a technique to search the optimal solution of the corresponding regression task. Although many $LSSVR$ algorithms have been implemented in past years, we exploit ALM method, as shown in Section 4. If the task is not differentiable or discontinuous, the sub gradient descent method can be employed, or the SMO [29] can also be used if there is a very large sample size.

The structure of this paper is as follows. Section 2 derives the optimal empirical risk loss by Bayesian principle. Section 3 constructs the $LSSVR$ model of G-L mixed noise. Section 4 gives the solution and algorithm design of $GLM-LSSVR$. In Section 5, the numerical experiment of short-term wind-speed prediction is presented. Finally, we conclude the work.

2. Bayesian Principle to Mixed Noise Empirical Risk Loss

Given the Dataset

$$D_N=\{(A_1,y_1),(A_2,y_2),\cdots ,(A_N,y_N)\},$$ (1)

where $A_i={(x_{i1},x_{i2},\cdots ,x_{in})}^T\in R^n$, $y_i\in R(i=1,2,\cdots ,N)$ is the training data. R represents real number set, $R^n$ is the n-dimensional Euclidean-Space, and N is the sample size. Superscript T is the transpose of matrix. Assuming that the sample of dataset $D_N$ is generated by the additive noise function $\xi$, the relationship between the measured value $y_i$ and predicted value $f\left(A_i\right)$ is:

$$y_i=f\left(A_i\right)+{\xi }_i,i=1,2,\cdots ,N$$ (2)

where ${\xi }_i$ is random, i.i.d. (independent, identical probability distribution) with $p\left({\xi }_i\right)$ of mean $\mu$ and standard deviation $\sigma$. Generally, the noise $PDF$ (probability density function) $p\left(\xi \right)=p(y-f(A\left)\right)$ is unknown. It is necessary to predict unknown target $f\left(A\right)$ from training set $D_f\subseteq D_N$.

Following the authors of [30,31], the optimal empirical risk loss in the sense of Maximum Likelihood (MLE) is

$$l\left(\xi \right)=l(A,y,f(A\left)\right)=-Logp(y-f(A\left)\right),$$ (3)

i.e., the empirical risk loss $l\left(\xi \right)$ is the log-likelihood of noise characteristic.

It is assumed that noise in Equation (2) is Laplacian, with $PDF$$p\left(\xi \right)=\frac{1}{2}{e}^{-\left|\xi \right|}$. By Equation (3), in MLE the optimal empirical risk loss should be $l\left(\xi \right)=\left|\xi \right|$.

Suppose noise in Equation (2) is Gaussian of zero mean and homoscedastic standard deviation $\sigma$. By Equation (3), the empirical risk loss of Gaussian noise with homoscedasticity is $l\left(\xi \right)=\frac{1}{2{\sigma }^2}{\xi }^2$. The noise in Equation (2) is Gaussian of zero mean and heteroscedastic standard deviation ${\sigma }_i$. By Equation (3), the empirical risk loss for Gaussian-noise with heteroscedasticity is $l\left({\xi }_i\right)=\frac{1}{2{\sigma }_i^2}{\xi }_i^2$ ($i=1,\cdots ,N$).

Assume noise $\xi$ in Equation (2) is the mixed noise of two kinds of noise with the $PDF$s $p_1\left(\xi \right)$ and $p_2\left(\xi \right)$, respectively. Suppose that $p\left(\xi \right)=[p_1{\left(\xi \right)}^{{\lambda }_1}]·[p_2{\left(\xi \right)}^{{\lambda }_2}]$. By Equation (3), the corresponding empirical risk loss of mixed-noise is

$$l\left(\xi \right)={\lambda }_1·l_1\left(\xi \right)+{\lambda }_2·l_2\left(\xi \right).$$ (4)

where $l_1\left(\xi \right)>0,l_2\left(\xi \right)>0$ are the convex empirical risk losses of the above two kinds of noise characteristic, respectively. The weight factors are ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

Figure 1 displays the Gaussian–Laplacian (G-L) empirical risk loss of different parameters (the parameter lambda is $\lambda$) [29].

Figure 1.

G-L empirical risk loss of different parameters.

3. $\mathit{LSSVR}$ Model of G-L Mixed Noise-Characteristic

Given the training samples $D_f\subseteq D_N$, construct the linear regressor $f\left(A\right)={\varpi }^T·A+b$. To deal with nonlinear problems, it can be summarized as follows: mapping input vectors $A_i\in R^n$ into high-dimension feature space H through the nonlinear mapping $\Phi$ (take a prior distribution), induced by nonlinear kernel function $K(A_i,A_j)$, kernel mapping $\Phi$ is any positive definite Mercer kernel.

Definition 1

([6,28]). Positive definite Mercer kernel: Assume that X is a subset of $R^n$. Assume that the kernel function $K(A_i,A_j)$ defined on $X\times X$ is a positive definite Mercer kernel functionl the kernel mapping Φ is called a positive definite Mercer kernel if there is mapping $\Phi :X\to H$ (H is Hilbert Space), such that

$$K(A_i,A_j)=(\Phi \left(A_i\right)·\Phi \left(A_j\right)),(i,j=1,2,\cdots ,N).$$ (5)

where $(·)$ represents the inner-product in Space H.

Therefore, the optimization problem of Space H is solved. At present, the input vectors $(A_i·A_j)$ are replaced by inner product $(\Phi \left(A_i\right)·\Phi \left(A_j\right))$ in feature space H. Through the use of kernel $K(A_i,A_j)=(\Phi \left(A_i\right)·\Phi \left(A_j\right))$, the linear model be extended to a nonlinear $LSSVR$.

In general, the mixed distribution has fine approximation ability to any continuous distribution. When there is no prior knowledge of real-noise, it can well adapt to unknown or complicated noise. Thus, it is presented that a uniform model $LSSVR$ with mixed noise characteristics ($M-LSSVR$). The primal problem of model $M-LSSVR$ is formalized as

$$\begin{array}{c}{\displaystyle Min\{g_{P_{M-LSSVR}}=\frac{1}{2}{\varpi }^T·\varpi +\frac{C}{N}·[{\lambda }_1·\sum _{i=1}^N\left(l_1\left({\xi }_i\right)\right)+}\hfill \\ {\lambda }_2·\sum _{i=1}^N\left(l_2\left({\xi }_i\right)\right)\left]\right\}\hfill \\ s.t.:{\xi }_i=y_i-{\varpi }^T·\Phi \left(A_i\right)-b\hfill \end{array}$$ (6)

where parameter $\varpi \in R^n$ represents weight-vector, b is the bias-term, $C>0$ is the penalty parameter, and the weight factors are ${\lambda }_1,{\lambda }_2\ge 0$, ${\lambda }_1+{\lambda }_2=1$. $(A_i,y_i)\in D_N$, $\Phi \left(A\right)$ is a nonlinear mapping which transfers the input dataset to a higher-dimensional feature space H. ${\xi }_i=y_i-{\varpi }^T·\Phi \left(A_i\right)-b$ is the random noise variable at time $i(i=1,2,\cdots ,N)$. $l_1\left({\xi }_i\right)>0,l_2\left({\xi }_i\right)>0(i=1,2,\cdots ,l)$ is the convex loss-functions for noise characteristic in sample-point $(A_i,y_i)\in D_N$ ($(i,j=1,2,\cdots ,N)$).

In the application domain, most distributions do not obey Gaussian distribution, and they also do not satisfy Laplacian distribution. the noise distribution is complicated, and it is almost impossible to describe real noise with a single distribution. It has been reported that mixed noise models, constituted by multiple noise distributions, perform better than single-noise model [1]. As the function fitting -machine, the goal is to estimate an unknown function $f\left(A\right)$ from dataset $D_f\subseteq D_N$. In this section, G-L mixed homoscedastic and heteroscedastic noise distributions are used to fit complicated noise characteristic.

3.1. $LSSVR$ Model of G-L Mixed Homoscedastic Noise-Characteristic

Suppose noise in Equation (2) is Gaussian of zero mean and homoscedastic standard deviation $\sigma$. By Equation (3), we have that the empirical risk loss of homoscedastic-Gaussian-noise characteristic is $l_1\left(\xi \right)=\frac{1}{2{\sigma }^2}·{\xi }^2$. The Laplacian-noise is $l_2\left(\xi \right)=\left|\xi \right|$. Adopting G-L mixed homoscedastic noise distribution to fit complicated noise-characteristic, by Equation (4), the empirical risk loss about G-L mixed homoscedastic noise is $l\left(\xi \right)=\frac{{\lambda }_1}{2{\sigma }^2}·{\xi }^2+{\lambda }_2·\left|\xi \right|$. Putting forward the $LSSVR$ model of G-L mixed homoscedastic noise-characteristic ($GLM-LSSVR$), the primal problem of $GLM-LSSVR$ is depicted as

$$\begin{array}{c}{\displaystyle Min\{g_{P_{GLM-LSSVR}}=\frac{1}{2}{\varpi }^T·\varpi +\frac{C}{N}·(\frac{{\lambda }_1}{2{\sigma }^2}·\sum _{i=1}^N{\xi }_i^2}\hfill \\ +{\lambda }_2·\sum _{i=1}^N{\xi }_i\left)\right\}\hfill \\ s.t.:{\xi }_i=y_i-{\varpi }^T·\Phi \left(A_i\right)-b\hfill \end{array}$$ (7)

where parameter vector $\varpi \in R^n$, ${\sigma }^2$ is homoscedastic, $C>0$ is a penalty parameter, and the weight factors are ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

Proposition 1.

The solution of the primal problem in Equation (7) of $GLM-LSSVR$ is existent and unique about ϖ.

Theorem 1.

The dual problem of the primal problem in Equation (7) is

$$\begin{array}{c}{\displaystyle Max\{g_{D_{GLM-LSSVR}}=-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i·{\alpha }_j·K(A_i,A_j)+}\hfill \\ \sum _{i=1}^N{\alpha }_i·y_i-\frac{N}{2C·{\lambda }_1}\sum _{i=1}^N{({\sigma }^2·{\alpha }_i-C·{\lambda }_2)}^2\}\hfill \\ s.t.:\sum _{i=1}^N{\alpha }_i=0\hfill \end{array}$$ (8)

where ${\sigma }^2$ is homoscedastic, $C>0$ is a penalty parameter, and the weight factors are ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

Proof. 

We introduce Lagrange functional $L(\varpi ,b,\alpha ,\xi )$ as $L(\varpi ,b,\alpha ,\xi )=\frac{1}{2}{\varpi }^T·\varpi +\frac{C}{N}·(\frac{{\lambda }_1}{2{\sigma }^2}·{\sum }_{i=1}^N{\xi }_i^2+{\lambda }_2·{\sum }_{i=1}^N{\xi }_i)+{\sum }_{i=1}^N{\alpha }_i(y_i-{\varpi }^T·\Phi \left(A_i\right)-b-{\xi }_i).$

Minimizing $L(\varpi ,b,\alpha ,\xi )$ and deriving the partial-derivative $\varpi ,b,\xi$, respectively, on the basis of KKT-conditions, we get

$${\nabla }_{\varpi }\left(L\right)=0,{\nabla }_b\left(L\right)=0,{\nabla }_{\xi }\left(L\right)=0.$$

We obtain

$$\varpi =\sum _{i=1}^N{\alpha }_i·\Phi \left(A_i\right),$$

$$\sum _{i=1}^N{\alpha }_i=0,$$

$$\frac{C}{N}·(\frac{{\lambda }_1}{{\sigma }^2}·{\xi }_i+{\lambda }_2)-{\alpha }_i=0(i=1,2,\cdots ,N).$$

The extreme condition is replaced by $L(\varpi ,b,\alpha ,\xi )$, and the maximum value of $\alpha$ is obtained. The dual problem in Equation (8) of the primal problem in Equation (7) is derived. □

Therefore,

$${\varpi }_i=\sum _{i=1}^N{\alpha }_i·\Phi \left(A_i\right),$$

$$b=\frac{1}{N}\sum _{i=1}^N[y_i-\sum _{j=1}^N{\alpha }_i·K(A_i,A_j)-\frac{1}{{\lambda }_1}·(\frac{N·{\sigma }^2·{\alpha }_i}{C}-{\lambda }_2)].$$

The decision-maker for $GLM-LSSVR$ may be represented as

$$f\left(A\right)={\varpi }^T·\Phi \left(A\right)+b=\sum _{i=1}^N{\alpha }_{i}K(A_i,A)+b.$$

where the parameter vector $\varpi \in R^n$, $\Phi :R^n\to H$, $(\Phi \left(A_i\right)·\Phi \left(A_j\right))$ is the inner-product of H and $K(A_i,A_j)=(\Phi \left(A_i\right)·\Phi \left(A_j\right))$ is the kernel-function.

Suppose the noise in Equation (2) is Gaussian homoscedastic noise, which is Gaussian noise of zero mean and the homoscedastic variance ${\sigma }^2$. Thus, the dual problem of $LSSVR$ can be derived by Theorem 2:

$$\begin{array}{c}{\displaystyle Max\{g_{D_{LSSVR}}=-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N({\alpha }_i·{\alpha }_j·K(A_i,A_j))}\hfill \\ +\sum _{i=1}^N({\alpha }_i·y_i)-\frac{N}{2C}·\sum _{i=1}^N({\sigma }^2·{\alpha }_i^2)\}\hfill \\ D_{GN-KRR}:s.t.\sum _{i=1}^N{\alpha }_i=0.\hfill \end{array}$$ (9)

3.2. $LSSVR$ Model of G-L Mixed Heteroscedastic Noise-Characteristic

It is assumed that the noise in Equation (2) is Gaussian of zero mean and heteroscedastic standard deviation ${\sigma }_i$, that is ${\sigma }_i\ne {\sigma }_j$, $i\ne j(i,j=1,\cdots ,N)$. From Equation (3), the empirical risk loss of heteroscedastic Gaussian-noise characteristic is $l_1\left({\xi }_i\right)=\frac{1}{2{\sigma }_i^2}·{\xi }_i^2$ and the loss-function of Laplacian-noise is $l_2\left({\xi }_i\right)=\left|{\xi }_i\right|$, $(i=1,\cdots ,N)$. Utilizing G-L mixed heteroscedastic noise distribution to predict complicated noise-characteristic, from Equation (4), the loss function corresponding to G-L mixed heteroscedastic noise is $l\left({\xi }_i\right)=\frac{{\lambda }_1}{2{\sigma }_i^2}·{\xi }_i^2+{\lambda }_2·\left|{\xi }_i\right|(i=1,\cdots ,N)$. The new model $LSSVR$ with G-L mixed heteroscedastic noise-characteristic ($GLMH-LSSVR$) is proposed. The primal problem of $GLMH-LSSVR$ is depicted as

$$\begin{array}{c}{\displaystyle Min\{g_{P_{GLMH-LSSVR}}=\frac{1}{2}{\varpi }^T·\varpi +}\hfill \\ \frac{C}{N}·(\frac{{\lambda }_1}{2}·\sum _{i=1}^N\frac{1}{{\sigma }_i^2}{\xi }_i^2+{\lambda }_2·\sum _{i=1}^N{\xi }_i)\}\hfill \\ s.t.:{\xi }_i=y_i-{\varpi }^T·\Phi \left(A_i\right)-b\hfill \end{array}$$ (10)

where the parameter vector is $\varpi \in R^n$, ${\sigma }_i^2(i=1,2,\cdots ,N)$ are heteroscedastic, and $C>0$ is the penalty parameter. The weight-factors are ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

Proposition 2.

The solution of the primal problem in Equation (10) of $GLMH-LSSVR$ is existent and unique about ϖ.

Theorem 2.

The dual problem of model $GLMH-LSSVR$ in Equation (10) is

$$\begin{array}{c}{\displaystyle Max\{g_{D_{GLMH-LSSVR}}=-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i·{\alpha }_j·K(A_i,A_j)+}\hfill \\ \sum _{i=1}^N{\alpha }_i·y_i-\frac{N}{2C·{\lambda }_1}\sum _{i=1}^N{({\sigma }_i^2·{\alpha }_i-C·{\lambda }_2)}^2\}\hfill \\ s.t.:\sum _{i=1}^N{\alpha }_i=0\hfill \end{array}$$ (11)

where ${\sigma }_i^2(i=1,2,\cdots ,N)$ are heteroscedastic and $C>0$ is the penalty parameter. The weight factors are ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

Proof. 

It is easier to derive the proof of Theorem 2 by analogy with Theorem 1. □

We have

$${\varpi }_i=\sum _{i=1}^N{\alpha }_i·\Phi \left(A_i\right),$$

$$b=\frac{1}{N}\sum _{i=1}^N[y_i-\sum _{j=1}^N{\alpha }_i·K(A_i,A_j)-\frac{1}{{\lambda }_1}·(\frac{N·{\sigma }_i^2·{\alpha }_i}{C}-{\lambda }_2)].$$

The decision-maker for $GLMH-LSSVR$ may be expressed as

$$f\left(A\right)={\varpi }^T·\Phi \left(A\right)+b=\sum _{i=1}^N{\alpha }_{i}K(A_i,A)+b,$$

where the parameter vector is $\varpi \in R^n$, $\Phi :R^n\to H$, and $K(A_i,A_j)$ is the kernel function.

Suppose noise in Equation (2) is G-L mixed-homoscedastic-noise, in which Gaussian-noise of zero mean and homoscedastic-variance ${\sigma }^2$, Theorem 1 can be deduced from Theorem 2.

4. Solution from ALM

In this section, we use Augmented Lagrange-multiplier method (ALM) [32] to solve the dual problem in Equation (8) by applying Gradient descent or Newton’s method to a sequence of equality-constrained problems. By eliminating equality constraints, arbitrary equality constraints can be reduced to equivalent unconstrained problems [33,34]. If there are large-scale training samples, some rapid optimization techniques can be combined with the proposed model, for example the sequential minimal optimization (SMO) algorithm [29] and the stochastic gradient decent (SDG) algorithm [35].

Theorems 1 and 2 provide effective recognition techniques for $GLM-LSSVR$ and $GLMH-LSSVR$, respectively. In this section, we derive the solution from ALM and the algorithm for model $LSSVR$ of G-L mixed homoscedastic noise characteristic ($GLM-LSSVR$). Analogously, the solution of model $GLMH-LSSVR$ can be obtained by ALM method.

(1) Let dataset be $D_N=\{(A_1,y_1),(A_2,y_2),\dots ,(A_N,y_N)\}$, where $A_i\in R^n$, $y_i\in R$, $i=1,\dots ,N$.

(2) The optimal parameters $C,{\lambda }_1,{\lambda }_2$ were searched by using the 10-fold cross-validation strategy, and the appropriate kernel function $K(•,•)$ was selected.

(3) Solve model $GLM-LSSVR$ of the problem in Equation (8), and get the optimal solution $\alpha =({\alpha }_1,\cdots ,{\alpha }_N)$.

(4) Build the decision-function as follows

$$f\left(A\right)={\varpi }^T·\Phi \left(A\right)+b=\sum _{i=1}^N{\alpha }_{i}K(A_i,A)+b.$$

The parameter vector is $\varpi \in R^n$, $b=\frac{1}{N}{\sum }_{i=1}^N[y_i-{\sum }_{j=1}^N{\alpha }_i·K(A_i,A_j)-\frac{1}{{\lambda }_1}·(\frac{N·{\sigma }^2·{\alpha }_i}{C}-{\lambda }_2)]$, $\Phi :R^n\to H$, $(\Phi \left(A_i\right)·\Phi \left(A_j\right))$ ($(i,j=1,2,\cdots ,N)$) is the inner product in H, $K(A_i,A_j)=(\Phi \left(A_i\right)·\Phi \left(A_j\right))$ is a kernel function.

5. Case Study

This section tests and verifies the validity of constructed model $GLM-LSSVR$ by comparing it with other techniques in the Heilongjiang, China dataset $D_N$. This case study consists of the following subsections: G-L mixed-noise characteristic of wind speed, prediction performance evaluation criteria, and short-term wind-speed forecasting based on an actual dataset.

5.1. G-L Mixed-Noise-Characteristic of Wind-Speed

To demonstrate the effectiveness of the proposed model, we collected wind speed data from Heilongjiang. The dataset consists of more than one year of wind speed data, recording wind speed values every 10 min. We first discovered the G-L mixed noise and conducted experiments on it. We found that turbulence is the main reason for the high uncertainty of wind speed random fluctuations. From the perspective of wind energy, the most significant feature of wind energy resources is their variability. Now, it shows the distribution of wind speed. Take a wind speed value every 5 s and calculate the histogram of wind speed within 1–2 h. Two typical distributions are given: one is calculated when the wind speed is high and the other is calculated when the wind speed is low (see Figure 2 and Figure 3, respectively).

Figure 2.

High wind speed distribution.

Figure 3.

Low wind speed distribution.

We analyzed the one-month time-series dataset, and used the persistence method to investigate the error distribution [32]. The results show that the wind speed error $\xi$ obtained from the persistence prediction is not subject to single distribution, while approximately to G-L mixed distribution, and $PDF$ of $\xi$ is $p\left(\xi \right)=\frac{1}{2}{e}^{-\left|\xi \right|}·\frac{1}{2{\sigma }^2}{\xi }^2$, as shown in Figure 4.

Figure 4.

G-L mixed distribution of wind-speed forecasting-error with the persistence method.

As can be seen from the above charts and figures, wind speed error approximately satisfies G-L mixed distribution. This is a mixed kind of task.

5.2. Prediction Performance Evaluation Criteria

It is generally known that no prediction model forecasts perfectly. The predictable performance of $\nu -SVR$, $GN-SVR$, $LSSVR$, and $GLM-LSSVR$ also has certain evaluation criteria, for example MAE (mean absolute error), RMSE (root mean square error), MAPE (mean absolute percentage error), and SEP (the standard error of prediction). The four criteria be defined as follows:

$$MAE=\frac{1}{N}\sum _{i=1}^N|y_i^{\prime }-y_i|,$$ (12)

$$MAPE=\frac{1}{N}\sum _{i=1}^N\frac{|y_i^{\prime }-y_i|}{y_i}\times 100\%,$$ (13)

$$RMSE=\sqrt{\frac{1}{N}{\sum }_{i=1}^N{(y_i^{\prime }-y_i)}^2},$$ (14)

$$SEP=\frac{RMSE}{\overline{y}}\times 100\%,$$ (15)

where N is the size of the dataset $D_N$, $y_i$ is the ith actual observed data, and $y_i^{\prime }$ is the ith forecasted-result. $\overline{y}$ is the mean value of observations $y_i\in D_N$[36,37,38,39,40]. $MAE$ shows how similar the predicted value is to the observed value, while $RMSE$ measures overall deviation between predicted value and observed value. $MAPE$ is the ratio between error and observed value. $SEP$ is the ratio of $RMSE$ to average observation. They are dimensionless measurements of accuracy of wind speed system, and are sensitive to small changes.

5.3. Short-Term Wind-Speed Forecasting with Real dataset

In this section, 2160 consecutive data (1–2160, time span of 15-days) are extracted as the training set and 720 consecutive data (2161–2880, time span of 5-days) are extracted as the testing set. The input vector is $\overline{A_i}=(x_{i-11},x_{i-10},\cdots ,x_{i-1},x_i)$, $x_j$ is the actual observed data of wind speed at moment $j(j=i-11,i-10,\cdots ,i)$, and the forecasting value is $x_{i+step}$, where $step=1,3,6$. That is, the above models are used to forecast wind speed of each point $x_i$ after 10, 30 and 60 min, respectively. Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 describe the forecasting results given by models $\nu -SVR$, $GN-SVR$, $LSSVR$, and $GLM-LSSVR$.

Figure 5.

Result of four wind-speed forecasting models after 10 min.

Figure 6.

Error of four wind-speed forecasting models after 10 min.

Figure 7.

Residual box plot of four wind-speed forecasting models after 10 min.

Figure 8.

Result of four wind-speed forecasting models after 30 min.

Figure 9.

Error of four wind-speed forecasting models after 30 min.

Figure 10.

Residual box plot of four wind-speed forecasting models after 30 min.

Figure 11.

Result of four wind-speed forecasting models after 60 min.

Figure 12.

Error of four wind-speed forecasting models after 60 min.

Figure 13.

Residual box plot of four wind-speed forecasting models after 60 min.

The models $\nu -SVR$, $GN-SVR$, $LSSVR$, and $GLM-LSSVR$ were implemented in Matlab 7.8. Initial parameters of $GLM-LSSVR$ were $C\in [1,200]$, $\nu \in (0,1)$, and ${\lambda }_1,{\lambda }_2\in [0,1]$. The optimal parameters $C,\nu ,{\lambda }_1,{\lambda }_2$ were searched by using 10-fold cross-validation technique. The technology of parameter selection is studied in detail in [41,42]. In this simulation, parameters were set to $C=181,\nu =0.5,{\lambda }_1=0.5,{\lambda }_2=0.5$. The practical application demonstrates that both polynomial kernel and Gaussian kernel perform well under the assumption of smoothness. Under these circumstances, models $\nu -SVR$, $GN-SVR$, $LSSVR$, and $GLM-LSSVR$ employ polynomial and Gaussian kernel functions [43]:

$$K(A_i,A_j)={((A_i,A_j)+1)}^d,$$

$$K(A_i,A_j)=e^{-\frac{\parallel A_i-A_j{\parallel }^2}{{\sigma }^2}},$$

where d is a positive integer and $\sigma$ is a positive number.

The dual problem of $\nu -SVR$ and $SVR$ of the Gaussian-noise model ($GN-SVR$) and $LSSVR$ are as follows.

$\nu -SVR$: The authors of [41,44] define the dual problem of $\nu -SVR$ as

$$\begin{array}{c}{\displaystyle Max\{g_{D_{\nu -SVR}}=-\frac{1}{2}\sum _{i\in RSV}\sum _{j\in RSV}({\alpha }_i^{*}-{\alpha }_i)({\alpha }_j^{*}-{\alpha }_j)}\hfill \\ ·K(A_i,A_j)+\sum _{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)·y_i\}\hfill \\ s.t.:\sum _{i=1}^N({\alpha }_i^{*}-{\alpha }_i)=0\hfill \\ 0\le {\alpha }_i^{(*)}\le \frac{C}{N},i=1,\cdots ,N\hfill \\ \sum _{i=1}^N({\alpha }_i+{\alpha }_i^{*})\le C·\nu ,i=1,\cdots ,N.\hfill \end{array}$$ (16)

$GN-SVR$: The authors of [45,46] studied $SVR$ with equality constraints and inequality constraints. The loss-function of Gaussian-noise is $c\left({\xi }_i\right)={\xi }_i^2/2$, ($i=1,\cdots ,N$). Thus, thus dual problem of $GN-SVR$ is

$$\begin{array}{c}{\displaystyle max\{g_{D_{GN-SVR}}=-\frac{1}{2}\sum _{i\in RSV}\sum _{j\in RSV}({\alpha }_i^{*}-{\alpha }_i)({\alpha }_j^{*}-{\alpha }_j)K(A_i,A_j)+}\hfill \\ \sum _{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)y_i-\frac{N}{2C}\sum _{i=1}^N({\alpha }_i^2+{\left({\alpha }_i^{*}\right)}^2)\}\hfill \\ s.t.:\sum _{i=1}^N({\alpha }_i^{*}-{\alpha }_i)=0\hfill \\ 0\le {\alpha }_i^{(*)}\le \frac{C}{N},i=1,\cdots ,N\hfill \\ \sum _{i=1}^N({\alpha }_i+{\alpha }_i^{*})\le C·\nu ,i=1,\cdots ,N.\hfill \end{array}$$ (17)

$LSSVR$: [22] studied $LS-SVR$ for Gaussian-noise model. The dual problem of $LS-SVR$ is

$$\begin{array}{c}{\displaystyle max\{g_{D_{LS-SVR}}=-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N({\alpha }_i·{\alpha }_j·K(A_i,A_j))}\hfill \\ +\sum _{i=1}^N({\alpha }_i·y_i)-\frac{N}{2C}·\sum _{i=1}^N\left({\alpha }_i^2\right)\}\hfill \\ s.t.:\sum _{i=1}^N{\alpha }_i=0.\hfill \end{array}$$ (18)

where ${\xi }_i,{\xi }_i^{*}$ are slack-variables. $C>0$, $\nu \in (0,1]$ are constants. For $\nu -SVR$ and $GN-SVR$, the size of $ϵ$ is not gained, but is a variable whose value is compromised by a constant with the model complexity and relaxation variables through $\nu$ [35].

In Figure 5, Figure 8 and Figure 11, wind-speed forecasting-results at $A_i$-point of $\nu -SVR$, $GN-SVR$, $LSSVR$, and $GLM-LSSVR$ are presented after 10, 30, and 60 min, respectively. Figure 6, Figure 9, and Figure 12 show the error statistic of wind-speed prediction using the above four models. The box plots (Figure 7, Figure 10, and Figure 13) of several noise levels further intuitively demonstrate the comparative effect of error statistics using the above four wind-speed forecasting models. The statistical criteria of $MAE$, $MAPE$, $RMSE$ and $SEP$ are displayed in Table 1, Table 2 and Table 3.

Table 1.

Error statistic of four wind-speed forecasting models after 10 min.

Model	MAE (m/s)	RMSE (m/s)	MAPE (%)	SEP (%)
$\nu -SVR$	0.4280	0.5833	8.02	7.12
$GN-SVR$	0.4256	0.5789	7.92	7.07
$LSSVR$	0.4219	0.5768	7.94	7.06
$GLM-LSSVR$	0.4190	0.5711	7.91	7.05

Table 2.

Error statistic of four wind-speed forecasting models after 30 min.

Model	MAE (m/s)	RMSE (m/s)	MAPE (%)	SEP (%)
$\nu -SVR$	0.7979	1.0116	23.36	12.53
$GN-SVR$	0.7368	0.9886	19.93	11.89
$LSSVR$	0.7109	0.9226	17.17	11.43
$GLM-LSSVR$	0.6185	0.8241	10.71	10.19

Table 3.

Error statistic of four wind-speed forecasting models after 60 min.

Model	MAE (m/s)	RMSE (m/s)	MAPE (%)	SEP (%)
$\nu -SVR$	0.9994	1.2580	33.93	15.66
$GN-SVR$	0.9728	1.2355	31.78	15.37
$LSSVR$	0.9646	1.2177	29.01	15.16
$GLM-LSSVR$	0.8835	1.1180	25.72	13.97

From box-whisker plots in Figure 7, Figure 10, and Figure 13, as well as Table 1, Table 2 and Table 3, it can be concluded that, in most cases, the forecasting-error of $GLM-LSSVR$ calculation is superior to $\nu -SVR$, $GN-SVR$ and $LSSVR$. With the increase of prediction horizon to 30 and 60 min, the forecasting error of different models increases and the relative error decreases. Thus, in these cases, it is not that important. However, Table 1, Table 2 and Table 3 show that, under all the criteria of $MAE$, $MAPE$, $RMSE$, and $SEP$, the Gaussian–Laplacian mixed-noise model is slightly better than the classical model.

6. Conclusions

Most existing regression-techniques suppose that the noise model is single. Wind-speed forecasting is complicated due to its volatility and uncertainty, thus it is difficult to model with a single-noise distribution. This section summarizes our main work: (1) optimal empirical risk loss of G-L mixed noise is deduced by Bayesian principle; (2) the $LSSVR$ of G-L mixed homoscedastic noise ($GLM-LSSVR$) and G-L mixed heteroscedastic noise ($GLMH-LSSVR$) for complicate noise is developed; (3) the dual problem of $GLM-LSSVR$ and $GLMH-LSSVR$ is obtained using Lagrange-functional and according to KKT conditions; (4) the stability and effectiveness of the algorithm are guaranteed by solving $GLM-LSSVR$ with the ALM method; and (5) the proposed technology is used to predict short-term wind speed by historical data, and then forecast the wind speed at some time after 10, 30, and 60 min, respectively. The comparison results display that the proposed model is better than classical technologies in statistical criteria.

In the same way, we can also study Gaussian–Laplacian, or Gaussian–Weibull mixed noise classification models. The new hybrid noise models would effectively solve complicated noise classification problems.

Abbreviations

The following abbreviations are used in this manuscript:

LR	Linear regression model
$\nu$-SVR	$\nu$-Support vector regression
GN-SVR	$\nu$-SVR model of Gaussian homoscedastic-noise
LSSVR	Least squares support vector regression model
GLM-LSSVR	LSSVR model of Gaussian–Laplacian mixed homoscedastic-noise
ALM	Augmented Lagrange multiplier method

Author Contributions

Conceptualization, S.Z.; Formal analysis, S.Z. and T.Z.; Methodology, S.Z. and L.S.; Writing–original draft, S.Z. and T.Z.; Writing–review & editing, W.W. and B.C. All authors have read and approved the final published version. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the National natural science foundation of China (NSFC) (Nos. 11702087 and 61772176) and Natural Science Foundation Project of Henan (No. 182300410130).

Conflicts of Interest

The authors declare no conflict of interest.