Distributed Support Vector Ordinal Regression over Networks

Abstract

Ordinal regression methods are widely used to predict the ordered labels of data, among which support vector ordinal regression (SVOR) methods are popular because of their good generalization. In many realistic circumstances, data are collected by a distributed network. In order to protect privacy or due to some practical constraints, data cannot be transmitted to a center for processing. However, as far as we know, existing SVOR methods are all centralized. In the above situations, centralized methods are inapplicable, and distributed methods are more suitable choices. In this paper, we propose a distributed SVOR (dSVOR) algorithm. First, we formulate a constrained optimization problem for SVOR in distributed circumstances. Since there are some difficulties in solving the problem with classical methods, we used the random approximation method and the hinge loss function to transform the problem into a convex optimization problem with constraints. Then, we propose subgradient-based algorithm dSVOR to solve it. To illustrate the effectiveness, we theoretically analyze the consensus and convergence of the proposed method, and conduct experiments on both synthetic data and a real-world example. The experimental results show that the proposed dSVOR could achieve close performance to that of the corresponding centralized method, which needs all the data to be collected together.

1. Introduction

Many real-world data labels have natural orders that are usually called ordinal labels. For example, fault severity in industrial processes is usually divided into {harmless, slight, medium, severe}. Ordinal regression, which aims at predicting ordinal labels for given patterns, has attracted a great deal of research in many fields, such as disease severity assessment [1], satisfaction evaluation [2], wind-speed prediction [3], age estimation [4], credit-rating prediction [5], and fault severity diagnosis [6]. Although classical classification and regression methods can be applied to the ordinal regression problem [7,8], they require additional prior information about the distances between labels. Otherwise, they often perform unsatisfactorily since they cannot fully use ordering information [9,10].

To tackle the aforementioned problems of classical classification and regression methods, many ordinal regression methods were proposed [10]. Among them, the most popular type of approaches are threshold models, which assume that a continuous latent variable underlies the ordinal response [10]. In threshold models, the order of the labels is represented by a set of ordered thresholds. These ordered thresholds define a series of intervals, and the data label depends on the interval the corresponding latent variable falls into. Among the threshold models, support vector ordinal regression (SVOR) [11,12] is widely used because of good generalization performance. A representative work is the support vector ordinal regression with implicit constraints (SVORIM) proposed in [11,12]. This determines each threshold by taking all the samples into consideration, where the threshold inequality constraints can be satisfied without explicit constraints.

Most of the existing ordinal regression methods have been developed in a centralized framework. However, in practice, data used for ordinal regression may be distributed in a network [13]. Each node of the network collects and stores part of the data, and it is not enough for a single node to train a model with good performance. For instance, in industrial processes, sensors are often used in factories to monitor the operating status of equipment and diagnose fault severity. Due to the rarity of faults, a single sensor can only collect very few data, and the faults encountered by each factory may also be different. To train a proper model, we need to use as many data as possible. However, in some realistic scenarios, it is difficult for data to be transmitted to a central node for various reasons [13]. For example, factories may not want to leak data regarding their equipment in order to protect privacy. Moreover, if the data are collected by image sensors or video sensors, it may be difficult for a single machine to store and process such a large amount of data. In such situations, centralized methods are inapplicable, and distributed methods are more suitable choices.

In this paper, we propose a distributed support vector ordinal regression algorithm based on the SVORIM method to deal with more complex nonlinear problems in distributed ordinal regression. First, we formulate a constrained optimization problem for SVORIM in the distributed scenarios. Classical methods usually solve the problem by transforming it into the dual problem. In distributed circumstances where the original data cannot be transmitted to others, it is difficult for classical methods to calculate the kernel function values and optimize the dual variables because they require data from different nodes. Thus, we adopted a random approximation method and the hinge loss function to transform the optimization problem to overcome the above difficulties. Increasing the number of random approximation dimensions can improve the approximation accuracy, but brings redundancy. In order to find an appropriate number of approximation dimensions, we further added a sparse regularization term of the approximation dimension number to the objective function. Through the above steps, we transformed the original problem into a convex optimization problem with consensus constraints. Then, to solve the problem, we propose a subgradient-based algorithm called distributed SVOR (dSVOR) where each node only uses its own data and the parameter estimates exchanged from its neighbors. To verify the effectiveness of dSVOR, we theoretically analyze its consensus and convergence, and conducted some experiments on synthetic data and a real-world example. The experimental results show that the proposed distributed algorithm under additional constraints could achieve close performance to that of the corresponding centralized method, which needs all the data to be collected to a central node.

The main contributions of this paper are summarized as follows.

• 1.Existing work on distributed ordinal regression [14] uses a linear model; therefore, it cannot deal with the problems of linearly inseparable data. We extended the SVOR method to distributed scenarios to solve distributed ordinal regression problems with linearly inseparable data.
• 2.We developed a decentralized implementation of SVOR, and propose a dSVOR algorithm. In the proposed algorithm, the kernel feature map is approximated by random feature maps to avoid transmitting the original data, and sparse regularization is added to avoid excessively high approximation dimensions.
• 3.The consensus and convergence of the proposed algorithm are theoretically analyzed.

The rest of this paper is organized as follows. In Section 2, we introduce related works. The ordinal regression problem and the SVORIM method are introduced in Section 3 as preliminary knowledge. In Section 4, we formulate the distributed support vector ordinal regression problem, propose the dSVOR algorithm, and perform theoretical analysis of the proposed algorithm. Experiments were conducted to evaluate the effectiveness of the proposed algorithm and they are presented in Section 5. Lastly, in Section 6, we draw some conclusions.

2. Related Works

Ordinal Regression Methods. Many ordinal regression methods have been proposed to solve ordinal regression problems. The ordered logit model [15,16] makes assumptions about the distribution of the prediction error of the latent variable, and uses the cumulative distribution function to build the label cumulative probability function. The support vector ordinal regression (SVOR) [11,12] maximizes margins between two adjacent labels. Variants of SVOR with nonparallel hyperplanes were discussed in [17,18]. There are also ordinal regression methods that solve ordinal regression problems by solving a series of binary classification subproblems. In [4,19], extended labels were extracted from the original ordinal labels to learn a binary classifier (such as support vector machine [19] or logistic regression [4]); then, a ranking rule was constructed from the binary classifier to predict ordinal labels. In [20], the authors used the stick-breaking process to construct a series of binary classification subproblems to guarantee that the cumulative probabilities were monotonically decreasing. However, the above ordinal regression methods are all centralized and are infeasible in distributed scenarios.

Distributed methods. Distributed methods were extensively studied in many fields, such as distributed estimation [21,22], distributed optimization [23,24], distributed clustering [25], distributed Kalman filter [26], and distributed anomaly detection [27]. However, as far as we know, there are few works investigating distributed ordinal regression [14]. In [14], the authors proposed a distributed generalized ordered logit model, which is a linear model and therefore cannot handle complex problems.

3. Preliminaries

3.1. Ordinal Regression Problem

The classification problem aims at classifying the K-dimensional input vector $\mathit{x}\in \mathcal{X}\subseteq {\mathbb{R}}^K$ into one of Q discrete categories $y\in \mathcal{Y}=\{{\mathcal{C}}_1,{\mathcal{C}}_2,\dots ,{\mathcal{C}}_Q\}$. The ordinal regression problem is a type of classification problem in which the data labels have a natural order ${\mathcal{C}}_1\prec {\mathcal{C}}_2\prec \cdots \prec {\mathcal{C}}_Q$, where ≺ is an order relation [10]. The purpose of ordinal regression is to find a mapping function $f:\mathcal{X}\to \mathcal{Y}$ to predict the ordinal labels for new patterns given a training set of N samples $D=\{({\mathit{x}}_i,y_i),i=1,\dots ,N\}$.

3.2. Support Vector Ordinal Regression with Implicit Constraints

Let $\varphi \left(\mathit{x}\right)$ denote the feature vector in a high-dimensional reproducing kernel Hilbert space (RKHS) of input vector $\mathit{x}$. The inner product in the RKHS is defined by the reproducing kernel function: $K(\mathit{x},{\mathit{x}}^{\prime })=\varphi \left(\mathit{x}\right)·\varphi \left({\mathit{x}}^{\prime }\right)$.

Support vector machines construct a discriminant hyperplane in the RKHS by maximizing the distance between support vectors and the discriminant hyperplane. The discriminant hyperplane is defined by an optimal direction $\mathit{w}$ and a single optimal threshold b. It divides the feature space into two regions for two classes.

The support vector ordinal regression constructs $Q-1$ parallel discriminant hyperplanes for Q ordinal labels where these hyperplanes are defined by optimal direction $\mathit{w}$ and $Q-1$ thresholds ${\left\{b_q\right\}}_{q=1,\dots ,Q-1}$. The ordinal information in the labels is represented by threshold inequalities $b_1\le b_2\le \cdots \le b_{Q-1}$. For convenience, vector $\mathit{b}={[b_1{b}_1\cdots b_{Q-1}]}^T$ was used to denote these thresholds.

In [11,12], the SVORIM method determined a threshold $b_q$ by utilizing the samples of all the labels. For threshold $b_q$, each sample belonging to ${\mathcal{C}}_p,\forall p\le q$ should have a function value less than $b_q-1$; otherwise, ${\xi }_{pi}^q=\mathit{w}·\varphi \left({\mathit{x}}_i^p\right)-(b_q-1)$ is the empirical error of ${\mathit{x}}_i^p$ for $b_q$. Similarly, each sample belonging to ${\mathcal{C}}_p,\forall p>q$ should have a function value greater than $b_q+1$; otherwise, ${\xi }_{pi}^{*q}=(b_q+1)-\mathit{w}·\varphi \left({\mathit{x}}_i^p\right)$ is the empirical error of ${\mathit{x}}_i^p$ for $b_q$.

As proved in [11,12], this approach has the property that the threshold inequalities can be automatically satisfied after convergence without explicitly including the corresponding constraints. This method is called support vector ordinal regression with implicit constraints and is formulated as follows:

$$\begin{array}{cc}\hfill \underset{\mathit{w},\mathit{b},\mathbf{\xi },{\mathbf{\xi }}^{*}}{min}& \frac{1}{2}{\parallel \mathit{w}\parallel }^2+C\sum _{q=1}^{Q-1}\sum _{p=1}^q\sum _{i=1}^{N^p}{\xi }_{pi}^q+C\sum _{q=1}^{Q-1}\sum _{p=q+1}^Q\sum _{i=1}^{N^p}{\xi }_{pi}^{*q}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathit{w}·\varphi \left({\mathit{x}}_i^p\right)-b_q\le -1+{\xi }_{pi}^q,{\xi }_{pi}^q\ge 0,\forall i,q\mathrm{and}p=1,\dots ,q\hfill \\ \hfill & \mathit{w}·\varphi \left({\mathit{x}}_i^p\right)-b_q\ge +1-{\xi }_{pi}^{*q},{\xi }_{pi}^{*q}\ge 0,\forall i,q\mathrm{and}p=q+1,\dots ,Q,\hfill \end{array}$$ (1)

where C is a predefined positive constant. The above problem can be solved by solving the dual problem, which can be derived with standard Lagrangian techniques. Let ${\beta }_{pi}^q\ge 0,{\gamma }_{pi}^q\ge 0,{\beta }_{pi}^{*q}\ge 0$, and ${\gamma }_{pi}^{*q}\ge 0$ be the Lagrangian multipliers for the constraints in the above equation. The dual problem is the following maximization problem [11,12].

$$\begin{array}{cc}\hfill \underset{\mathbf{\beta },{\mathbf{\beta }}^{*}}{max}& -\frac{1}{2}\sum _{p,i}\sum _{p^{\prime },i^{\prime }}(\sum _{q=1}^{p-1}{\beta }_{pi}^{*q}-\sum _{q=p}^{Q-1}{\beta }_{pi}^q)(\sum _{q=1}^{p^{\prime }-1}{\beta }_{p^{\prime }{i}^{\prime }}^{*q}-\sum _{q=p^{\prime }}^{Q-1}{\beta }_{p^{\prime }{i}^{\prime }}^q)K({\mathit{x}}_i^p,{\mathit{x}}_{i^{\prime }}^{p^{\prime }})\hfill \\ \hfill & +\sum _{p,i}(\sum _{q=1}^{p-1}{\beta }_{pi}^{*q}+\sum _{q=p}^{Q-1}{\beta }_{pi}^q)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \sum _{p=1}^q\sum _{i=1}^{N^p}{\beta }_{pi}^q=\sum _{p=q+1}^Q\sum _{i=1}^{N^p}{\beta }_{pi}^{*q},\forall q\hfill \\ \hfill & 0\le {\beta }_{pi}^q\le C,\forall i,q\mathrm{and}p\le q\hfill \\ \hfill & 0\le {\beta }_{pi}^{*q}\le C,\forall i,q\mathrm{and}p>q.\hfill \end{array}$$ (2)

For a new pattern $\mathit{x}$, SVORIM calculates the function value $\mathit{w}·\varphi \left(\mathit{x}\right)$ and then decides its category according to the interval the function value falls into, where the intervals are defined by thresholds ${\left\{b_q\right\}}_{q=1,\dots ,Q-1}$.

4. Distributed Support Vector Ordinal Regression Algorithm

4.1. Network and Data Model

In this paper, we consider a network consisting of M nodes. We could use a graph $\mathcal{G}=(\mathcal{M},\mathcal{E})$ to represent this network. It consisted of a set of nodes $\mathcal{M}=\{1,2,\dots ,M\}$ and a set of edges $\mathcal{E}$. Each edge $(m,n)\in \mathcal{E}$ connected a pair of distinct nodes. We used ${\mathcal{N}}_m=\left\{n\right|(m,n)\in \mathcal{E}\}$ to represent the set of neighbors of node $m\in \mathcal{M}$.

Data used for ordinal regression are distributedly collected and stored by the M nodes of this network. The i-th sample of node m is represented as $({\mathit{x}}_{m,i},y_{m,i})$, where ${\mathit{x}}_{m,i}\in \mathcal{X}$ and $y_{m,i}\in \mathcal{Y}$. More specifically, at node m, the total number of samples is $N_m$, the number of samples that belong to ${\mathcal{C}}_q$ is $N_m^q$, and the i-th sample of ${\mathcal{C}}_q$ is denoted as $({\mathit{x}}_{m,i}^q,y_{m,i}^q)$.

Figure 1 shows a schematic of a distributed network. In distributed networks, due to limited storage, computation and communication resources and the need for privacy protection, node m can only transmit some parameters ${\theta }_m$ instead of the original data to its neighbor nodes in ${\mathcal{N}}_m$, and perform local computation using only its own data ${\left\{({\mathit{x}}_{m,i},y_{m,i})\right\}}_{1\le i\le N_m}$ and the parameters exchanged from its neighbors. Each node should eventually obtain a model consensus with that obtained by other nodes, and the performance of the model should be close to that of the model trained using all the data.

Figure 1.

Schematic of a distributed network. Node m only transmits its parameters ${\theta }_m$ with nodes in ${\mathcal{N}}_m$.

4.2. Problem Formulation

In centralized SVOR, the objective is to find an optimal direction $\mathit{w}$ and a vector $\mathit{b}$. If the data from all the nodes of the distributed network can be collected together, then parameters $\theta =\{\mathit{w},\mathit{b}\}$ can be obtained by solving Problem (1).

In distributed situations, data are not allowed to be transmitted to a central node. Each node can only use its own data and some parameters from its neighbors. In this case, each node m has a local estimate ${\theta }_m$ of $\theta$. With a connected network, we imposed constraints ${\theta }_m={\theta }_n,\forall (m,n)\in \mathcal{E}$ to ensure the consensus of ${\left\{{\theta }_m\right\}}_{m=1,\dots ,M}$. Then, the corresponding optimization problem in distributed scenarios can be written as follows:

$$\begin{array}{cc}\hfill min& \frac{1}{2}\sum _{m=1}^M{\parallel {\mathit{w}}_m\parallel }^2+C\sum _{m=1}^M\sum _{q=1}^{Q-1}\sum _{p=1}^q\sum _{i=1}^{N_m^p}{\xi }_{m,pi}^q+C\sum _{m=1}^M\sum _{q=1}^{Q-1}\sum _{p=q+1}^Q\sum _{i=1}^{N_m^p}{\xi }_{m,pi}^{*q}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\mathit{w}}_m·\varphi \left({\mathit{x}}_{m,i}^p\right)-b_{m,q}\le -1+{\xi }_{m,pi}^q,{\xi }_{m,pi}^q\ge 0,\hfill \\ \hfill & \forall m,i,q\mathrm{and}p=1,\dots ,q\hfill \\ \hfill & {\mathit{w}}_m·\varphi \left({\mathit{x}}_{m,i}^p\right)-b_{m,q}\ge +1-{\xi }_{m,pi}^{*q},{\xi }_{m,pi}^{*q}\ge 0,\hfill \\ \hfill & \forall m,i,q\mathrm{and}p=q+1,\dots ,Q\hfill \\ \hfill & {\mathit{w}}_m={\mathit{w}}_n,{\mathit{b}}_m={\mathit{b}}_n,\forall (m,n)\in \mathcal{E},\hfill \end{array}$$ (3)

where ${\xi }_{m,pi}^q$ is the empirical error of ${\mathit{x}}_{m,i}^p$ for $b_{m,q}$ when $p=1,\dots ,q$ and ${\xi }_{m,pi}^{*q}$ is the empirical error of ${\mathit{x}}_{m,i}^p$ for $b_{m,q}$ when $p=q+1,\dots ,Q$. With the help of the consensus constraints, this problem is equivalent to Problem (1).

4.3. Problem Transformation

In classical solutions, a primal problem is solved by solving the corresponding dual problem. Applying such methods to Distributed Problem (3) is confronted with two major difficulties:

• 1.For nonlinear kernel functions, the dimension of the RKHS is unknown, and we can only calculate the inner product of $\varphi \left({\mathit{x}}_{m,i}\right)$ and $\varphi \left({\mathit{x}}_{n,j}\right)$ rather than them. Because the data are distributed in various nodes of the network, the kernel function $K({\mathit{x}}_{m,i},{\mathit{x}}_{n,j})$ requiring data from different nodes is difficult to calculate without transmitting the original data.
• 2.The dual variables of samples should satisfy constraints in (2). In the distributed scenarios, the dual variables of the first constraint in (2) are usually from different nodes. Since each node is only allowed to exchange information with its neighbors, it is difficult to optimize these dual variables.

To overcome the first difficulty, we use a random approximate function [28] $z:{\mathbb{R}}^K\to {\mathbb{R}}^D$, where $D>K$, to map the data to a D-dimensional space instead of RKHS. In this study, for Gaussian kernel function

$$K(\mathit{x},{\mathit{x}}^{\prime })=exp\left(-\frac{\parallel \mathit{x}-{\mathit{x}}^{\prime }{\parallel }^2}{{\sigma }^2}\right),$$ (4)

we adopted $z\left(\mathit{x}\right)={[z_{{\mathbf{\omega }}_1}\left(\mathit{x}\right),\dots ,z_{{\mathbf{\omega }}_D}\left(\mathit{x}\right)]}^T$, where each dimension $z_{{\mathbf{\omega }}_i}\left(\mathit{x}\right)$ was

$$z_{{\mathbf{\omega }}_i}\left(\mathit{x}\right)=\sqrt{\frac{2}{D}}cos({\mathbf{\omega }}_i^T\mathit{x}+{\psi }_i),$$ (5)

where ${\psi }_i$ is drawn uniformly from $[0,2\pi ]$, and ${\mathbf{\omega }}_i$ is drawn from the Fourier transform of Gaussian kernel function

$$p\left(\mathbf{\omega }\right)={\left(2\pi \right)}^{-\frac{K}{2}}exp\left(-\frac{{\sigma }^2{\parallel \mathbf{\omega }\parallel }^2}{2}\right).$$ (6)

As proved in [28], if dimensional number D is large enough, $z{\left(\mathit{x}\right)}^{T}z\left({\mathit{x}}^{\prime }\right)$ can approximate $K(\mathit{x},{\mathit{x}}^{\prime })$ well, and $z\left(\mathit{x}\right)$ can approximate $\varphi \left(\mathit{x}\right)$ well. According to Cover’s theorem [29], a complex pattern-classification problem nonlinearly cast in a high-dimensional space is more likely to be linearly separable than it is in a low-dimensional space. Therefore, to ensure good performance, we should set a relatively large D. For other shift-invariant kernels such as Laplacian and Cauchy, the authors in [28] provided corresponding finite-dimensional random approximate functions. For additive homogeneous kernels, such as Hellinger’s, ${\chi }^2$, intersection and Jensen-Shannon, the authors in [30] also provided efficient finite-dimensional approximate mapping functions. For a linear kernel function, random approximation is not necessary, so we defined $z\left(\mathit{x}\right)=\varphi \left(\mathit{x}\right)=\mathit{x}$.

With the random approximation, mapping function $\varphi \left(\mathit{x}\right)$ in (3) is replaced by $z\left(\mathit{x}\right)$. The calculation of $z\left(\mathit{x}\right)$ only requires one data point from a single node instead of a pair of data from different nodes like the kernel function, so the first difficulty is solved.

After the random approximation is performed, the data are mapped into a D-dimensional feature space instead of the RKHS with unknown dimension. Thus, we could directly solve the primal problem instead of the dual problem, which automatically tackles the second difficulty.

With the use of hinge loss function $L\left(x\right)=max(1-x,0)$ [31], the problem can be rewritten as follows:

$$\begin{array}{cc}\hfill min& \frac{1}{2}\sum _{m=1}^M{\parallel {\mathit{w}}_m\parallel }^2+C\sum _{m=1}^M\sum _{q=1}^{Q-1}\sum _{p=1}^q\sum _{i=1}^{N_m^p}L(b_{m,q}-{\mathit{w}}_m·z\left({\mathit{x}}_{m,i}^p\right))\hfill \\ \hfill & +C\sum _{m=1}^M\sum _{q=1}^{Q-1}\sum _{p=q+1}^Q\sum _{i=1}^{N_m^p}L({\mathit{w}}_m·z\left({\mathit{x}}_{m,i}^p\right)-b_{m,q})\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\mathit{w}}_m={\mathit{w}}_n,{\mathit{b}}_m={\mathit{b}}_n,\forall (m,n)\in \mathcal{E}.\hfill \end{array}$$ (7)

4.4. Sparse Regularization

In the above steps, a D-dimensional random approximate function $z\left(\mathit{x}\right)$ is used to approximate the unknown mapping function $\varphi \left(\mathit{x}\right)$. In general, a large D can lead to small approximation error and good classification performance. However, an overlarge D may cause redundancy, which wastes storage space, and brings high computational complexity and high communication costs. There is a trade-off between the above two aspects, so we added a sparse regularization term. The regularization term pushes some dimensions of ${\mathit{w}}_m$ to 0, which means that these dimensions are redundant and can be discarded. When some dimensions of ${\mathit{w}}_m$ converge to 0, these dimensions do not need to be calculated, stored and transmitted.

The $l_0$-norm is typically used to measure sparsity. However, it is nonconvex, and $l_0$-norm-based problems are NP-hard. In practice, we can use the $l_1$-norm as a convex approximation of the $l_0$-norm. Introducing the $l_1$-norm into the objective function in (7), we obtain

$$\begin{array}{cc}\hfill min& \sum _{m=1}^M\left[(1-\alpha )\frac{1}{2}\parallel {\mathit{w}}_m{\parallel }^2+\alpha {\parallel {\mathit{w}}_m\parallel }_1\right]\hfill \\ \hfill & +C\sum _{m=1}^M\sum _{q=1}^{Q-1}\sum _{p=1}^q\sum _{i=1}^{N_m^p}L(b_{m,q}-{\mathit{w}}_m·z\left({\mathit{x}}_{m,i}^p\right))\hfill \\ \hfill & +C\sum _{m=1}^M\sum _{q=1}^{Q-1}\sum _{p=q+1}^Q\sum _{i=1}^{N_m^p}L({\mathit{w}}_m·z\left({\mathit{x}}_{m,i}^p\right)-b_{m,q})\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\mathit{w}}_m={\mathit{w}}_n,{\mathit{b}}_m={\mathit{b}}_n,\forall (m,n)\in \mathcal{E},\hfill \end{array}$$ (8)

where $\alpha \in [0,1]$ controls the proportion of the $l_1$-norm sparsity regularization term in the entire regularization term. A larger $\alpha$ can lead to a sparser solution of ${\mathit{w}}_m$. Therefore, since we set a relatively large D to ensure good performance, we could set a relatively large $\alpha$ to reduce redundancy.

We could view this problem from another perspective. If the last two terms in (8) are regarded to be the objective function, the first two terms combined together can be seen as a similar penalty to the elastic net penalty in [32], where $\alpha$ measures the weight of the $l_1$-norm penalty term.

After the above steps, we transformed Problem (3) into a convex optimization problem with consensus constraints (8).

4.5. Distributed SVOR Algorithm

In this subsection, we propose the dSVOR algorithm to solve Problem (8). First, we used the following notation for convenience

$$\begin{array}{cc}\hfill J_m\left({\theta }_m\right)& =(1-\alpha )\frac{1}{2}\parallel {\mathit{w}}_m{\parallel }^2+\alpha {\parallel {\mathit{w}}_m\parallel }_1+C\sum _{q=1}^{Q-1}\sum _{p=1}^q\sum _{i=1}^{N_m^p}L(b_{m,q}-{\mathit{w}}_m·z\left({\mathit{x}}_{m,i}^p\right))\hfill \\ \hfill & +C\sum _{q=1}^{Q-1}\sum _{p=q+1}^Q\sum _{i=1}^{N_m^p}L({\mathit{w}}_m·z\left({\mathit{x}}_{m,i}^p\right)-b_{m,q}),\hfill \end{array}$$ (9)

which is a convex function. The calculation of $J_m\left({\theta }_m\right)$ does not need the data and estimated parameters from other nodes. Then, Problem (8) can be rewritten as follows:

$$\begin{array}{cc}\hfill min& J=\sum _{m=1}^M{J}_m\left({\theta }_m\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\theta }_m={\theta }_n,\forall (m,n)\in \mathcal{E}.\hfill \end{array}$$ (10)

To deal with consensus constraints ${\theta }_m={\theta }_n,\forall (m,n)\in \mathcal{E}$, we adopted the penalty function method. The penalty function used in this paper is ${\parallel {\theta }_m-{\theta }_n\parallel }^2$, and the corresponding positive penalty coefficient is ${\lambda }_{mn}$. Then, the optimization problem becomes

$$\begin{array}{c}\hfill min\sum _{m=1}^M{J}_m\left({\theta }_m\right)+\sum _{(m,n)\in \mathcal{E}}{\lambda }_{mn}{\parallel {\theta }_m-{\theta }_n\parallel }^2\end{array}$$ (11)

The larger the ${\lambda }_{mn}$ is, the closer the solutions of Problems (11) and (10) are.

We then applied the subgradient method to optimize Problem (11). For the hinge loss function $L\left(x\right)=max(1-x,0)$, we adopted the following subgradient:

$$L^{\prime }\left(x\right)=\left\{\begin{array}{cccc}\hfill & -1,\hfill & \hfill x<1& \\ \hfill & 0,\hfill & \hfill x\ge 1\end{array}\right.,$$ (12)

and for the $l_1$-norm, we adopted

$$sgn\left(x\right)=\left\{\begin{array}{cccc}\hfill & 1,\hfill & \hfill x>0& \\ \hfill & -1,\hfill & \hfill x<0& \\ \hfill & 0,\hfill & \hfill x=0\end{array}\right..$$ (13)

At step $k+1$, the iterative equation is

$$\begin{array}{cc}\hfill {\theta }_m^{k+1}& ={\theta }_m^k-{\eta }^k{\nabla }_{{\theta }_m}{J}_m\left({\theta }_m^k\right)-2{\eta }^k\sum _{n\in {\mathcal{N}}_m}{\lambda }_{mn}({\theta }_m^k-{\theta }_n^k),\hfill \end{array}$$ (14)

where ${\eta }^k$ is the step size in step $k+1$, which is positive. The specific subgradients are

$$\begin{array}{cc}\hfill {\nabla }_{{\mathit{w}}_m}{J}_m\left({\theta }_m^k\right)& =(1-\alpha ){\mathit{w}}_m^k+\alpha sgn\left({\mathit{w}}_m^k\right)\hfill \\ \hfill & -C\sum _{q=1}^{Q-1}\sum _{p=1}^q\sum _{i=1}^{N_m^p}{L}^{\prime }(b_{m,q}^k-{\mathit{w}}_m^k·z\left({\mathit{x}}_{m,i}^p\right))z\left({\mathit{x}}_{m,i}^p\right)\hfill \\ \hfill & +C\sum _{q=1}^{Q-1}\sum _{p=q+1}^Q\sum _{i=1}^{N_m^p}{L}^{\prime }({\mathit{w}}_m^k·z\left({\mathit{x}}_{m,i}^p\right)-b_{m,q}^k)z\left({\mathit{x}}_{m,i}^p\right),\hfill \end{array}$$ (15)

$$\begin{array}{cc}\hfill {\nabla }_{b_{m,q}}{J}_m\left({\theta }_m^k\right)& =C\sum _{p=1}^q\sum _{i=1}^{N_m^p}{L}^{\prime }(b_{m,q}^k-{\mathit{w}}_m^k·z\left({\mathit{x}}_{m,i}^p\right))\hfill \\ \hfill & -C\sum _{p=q+1}^Q\sum _{i=1}^{N_m^p}{L}^{\prime }({\mathit{w}}_m^k·z\left({\mathit{x}}_{m,i}^p\right)-b_{m,q}^k).\hfill \end{array}$$ (16)

In the subgradient method, in order to converge to the optimal solution, step size ${\eta }^k$ should satisfy [33]

$$\sum _{k=0}^{+\infty }{\eta }^k=+\infty ,\mathrm{and}\sum _{k=0}^{+\infty }{\left({\eta }^k\right)}^2<+\infty .$$ (17)

We can rearrange Iterative Equation (14) as follows.

$$\begin{array}{cc}\hfill {\theta }_m^{k+1}& =(1-2{\eta }^k\sum _{n\in {\mathcal{N}}_m}{\lambda }_{mn}){\theta }_m^k+\sum _{n\in {\mathcal{N}}_m}2{\eta }^k{\lambda }_{mn}{\theta }_n^k-{\eta }^k{\nabla }_{{\theta }_m}{J}_m\left({\theta }_m^k\right).\hfill \end{array}$$ (18)

If we use the following notations for convenience

$$\begin{array}{cc}\hfill & c_{mn}=2{\eta }^k{\lambda }_{mn},c_{mm}=1-\sum _{n\in {\mathcal{N}}_m}2{\eta }^k{\lambda }_{mn},\hfill \end{array}$$ (19)

the iterative equation can be rewritten as

$$\begin{array}{c}\hfill {\theta }_m^{k+1}=\sum _{n\in {\mathcal{N}}_m\cup \left\{m\right\}}{c}_{mn}{\theta }_n^k-{\eta }^k{\nabla }_{{\theta }_m}{J}_m\left({\theta }_m^k\right).\end{array}$$ (20)

It can be divided into two steps, i.e., a combination step and an adaption step:

$${\varphi }_m^k=\sum _{n\in {\mathcal{N}}_m\cup \left\{m\right\}}{c}_{mn}{\theta }_n^k,$$ (21)

$${\theta }_m^{k+1}={\varphi }_m^k-{\eta }^k{\nabla }_{{\theta }_m}{J}_m\left({\theta }_m^k\right).$$ (22)

In Combination Step (21), node m combines the parameters estimated by its neighbors and itself to obtain an intermediate estimate ${\varphi }_m^k$, where the combination coefficient of node m and its neighbor n is denoted as $c_{mn}$. In Adaption Step (22), node m uses the subgradient calculated by using only its own data to update ${\theta }_m$.

Combination coefficients ${\left\{c_{mn}\right\}}_{\forall (m,n)\in \mathcal{E}}$ represent a cooperation rule among nodes. Equation (19) was not used to define $\left\{c_{mn}\right\}$ because ${\lambda }_{mn}$ was not defined in advance. In distributed algorithms, combination coefficients are generally determined by a certain cooperative protocol. In this study, we used the Metropolis rule [34]:

$$c_{mn}=\left\{\begin{array}{cccc}\hfill & \frac{1}{max\left(\right|{\mathcal{N}}_m|,|{\mathcal{N}}_n\left|\right)},\hfill & \hfill n\in {\mathcal{N}}_m& \\ \hfill & 1-\sum _{n\in {\mathcal{N}}_m}{c}_{mn},\hfill & \hfill m=n& \\ \hfill & 0,\hfill & \hfill \mathrm{otherwise}\end{array}\right.,$$ (23)

where $|{\mathcal{N}}_m|$ denotes the degree of node m, and

$$\mathit{C}\mathbf{1}=\mathbf{1},{\mathbf{1}}^T\mathit{C}={\mathbf{1}}^T,$$ (24)

where $\mathit{C}$ is an $M\times M$ matrix whose entries are defined by (23).

Equation (19) shows that ${\lambda }_{mn}=\frac{c_{mn}}{2{\eta }^k}$. Step size ${\eta }^k$ satisfies (17), where the latter implies that ${lim}_{k\to \infty }{\eta }^k=0$. As $k\to \infty$, step size ${\eta }^k\to 0$ and penalty coefficient ${\lambda }_{mn}\to \infty$, which renders the solutions of Problems (11) and (10) nearly equal.

The whole processes of dSVOR are summarized in Algorithm 1.

Algorithm 1 Distributed SVOR algorithm

Initialization: initialize hinge loss function weight C, sparsity regularization weight $\alpha$, random approximate dimension D, and total iteration number T. Each node m initializes ${\theta }_m=\{{\mathit{w}}_m,{\mathit{b}}_m\}$. for$k=1:T$    for $m=1:M$       Communication Step: communicate parameters ${\theta }_m$ with neighbors $n\in {\mathcal{N}}_m$.    end for    for $m=1:M$       Combination Step: compute intermediate estimate ${\varphi }_m^k$ via (21).       Computation Step: Compute the subgradients ${\nabla }_{{\mathit{w}}_m}{J}_m\left({\theta }_m^k\right)$, ${\nabla }_{b_{m,q}}{J}_m\left({\theta }_m^k\right)$ via (15) and (16);       Adaption Step: update ${\theta }_m^{k+1}$ via (22).    end for end for

Remark 1.

In the above problems, $\varphi (·)$ is a nonlinear mapping function that maps input $\mathit{x}$ into a RKHS for classification, and input $\mathit{x}$ is the original data or extracted features. In general, function $\varphi (·)$ can also be regarded to be a generalized feature mapping function that extracts features of $\mathit{x}$, and maps $\mathit{x}$ into a feature space for classification. Thus, it can also use an artificial neural network with learnable parameters. However, that may destroy the convexity of the problem, so that it is no longer guaranteed to converge to the global optimum.

4.6. Theoretical Analysis

In this subsection, we theoretically analyze the consensus and convergence of dSVOR.

We first introduce a reasonable assumption that is needed in analysis. According to [34], when the graph is not bipartite, this assumption can be guaranteed.

Assumption 1.

Spectral radius $\rho (\mathit{C}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T)<1$, where $\mathit{C}$ is the combination coefficient matrix set as in Equation (23).

Then, we give two theorems about consensus and convergence each.

Theorem 1

(Consensus). If Assumption 1 holds, and step size ${\eta }^k$ satisfies Condition (17), then ${lim}_{k\to \infty }\parallel {\theta }_m^k-{\overline{\theta }}^k\parallel =0,\forall m$, where ${\overline{\theta }}^k=\frac{1}{M}{\sum }_{m=1}^M{\theta }_m^k$.

Theorem 2

(Convergence). If Assumption 1 holds, and step size ${\eta }^k$ satisfies Condition (17), then ${lim}_{k\to \infty }{\sum }_{m=1}^M{J}_m\left({\theta }_m^k\right)=J^{*}$, where $J^{*}=minJ$.

For the proof, see Appendix A and Appendix B for details.

5. Experiments

In this section, we carry out experiments on synthetic data and a real-world example to demonstrate the performance of the proposed dSVOR algorithm.

We implemented the following algorithms for comparison:

1. proposed dSVOR algorithm (dSVOR);

2. centralized SVOR (cSVOR), which relies on all the data available in a central node;

3. distributed SVOR with a noncooperative strategy (ncSVOR). In ncSVOR, each node uses only its own data to train a model without any information exchanged with other nodes.

All the algorithms were implemented using the PyTorch framework [35].

There are three points to emphasize:

• 1.The centralized method needs data in a central node. For comparison, we artificially collected all the data distributed in the nodes of the network together to render it applicable, which is impractical in reality.
• 2.In cSVOR [11,12], problems were solved by the SMO algorithm instead of subgradient-based algorithms, so we only display its final results.
• 3.The distributed algorithms were subject to additional constraints, so a distributed algorithm is generally satisfactory if it can achieve comparable performance to the corresponding centralized algorithm.

In this study, we used the prediction accuracy (ACC) and mean absolute error (MAE) on the testing set as the performance evaluation metrics. ACC is a commonly used metric in classification problems, but it does not consider the ordered information of the labels. MAE is the mean absolute deviation of the predicted rank from the true one, which is commonly used in ordinal regression. Using a function $\mathcal{O}(·)$ to denote the position of a certain label in the ordinal scale, i.e., $\mathcal{O}\left({\mathcal{C}}_q\right)=q,q=1,\dots ,Q$, we have

$$MAE=\frac{1}{N}\sum _{i=1}^N|\mathcal{O}\left(y_i\right)-\mathcal{O}\left({\widehat{y}}_i\right)|\in [0,Q-1].$$ (25)

The performance of distributed algorithms (dSVOR and ncSVOR) is defined as the mean performance of models obtained by each node. The distributed algorithms ran on a randomly generated connected network that consisted of 20 nodes. For fair comparison, on a certain dataset, all implemented algorithms used the same parameters. All the results were obtained by averaging the results of 10 independent experiments.

5.1. Synthetic Data

In this subsection, we evaluate the performance of all algorithms on two synthetic datasets. On the first dataset, samples could be separated by a set of parallel straight lines if ignoring noises, and samples of the second dataset could be separated by a set of concentric circles. Figure 2a,b show some samples of these two datasets from one of the 10 independent experiments. Both datasets had 1200 samples: 1000 were used as the training set, and the others were the testing set. The training samples were randomly assigned to 20 nodes to simulate the situation where the data were collected and stored by these nodes in a distributed manner.

Figure 2.

Data visualization of the (a) first and (b) second synthetic datasets.

These two synthetic datasets were generated with the following methods. For the first dataset, we generated 1200 samples with uniform distribution from a rectangular area $x_{1min}\le x_1\le x_{1max},x_{2min}\le x_2\le x_{2max}$. We then used three straight lines ${\{x_1=b_i\}}_{i=1,2,3}$ to divide this area into 4 parts for 4 classes. The data labels were determined by their locations. Then, Gaussian noise with 0 mean and ${\sigma }_1$ standard deviation was added to each dimension of input vector $\mathit{x}={\left[x_1{x}_2\right]}^T$. After that, these samples were rotated around the origin with $\beta$. Without loss of generality, in the experiments, these parameters were set as follows:

$$\begin{array}{c}{x}_{1min}=-2,x_{1max}=2,x_{2min}=-1,x_{2max}=1,\\ b_1=-1,b_2=0,b_3=1,{\sigma }_1=0.5,\beta =\frac{\pi }{8}.\end{array}$$

For the second dataset, we generated 1200 samples with uniform distribution from a circle $x_1^2+x_2^2<R^2$, which could be divided into four parts by three concentric circles ${\{x_1^2+x_2^2=R_i^2\}}_{i=1,2,3}$. The data labels were determined by their locations. Then, Gaussian noise with 0 mean and ${\sigma }_2$ standard deviation was added to each dimension of input vector $\mathit{x}={\left[x_1{x}_2\right]}^T$. Without loss of generality, the parameters were set to be $R=4$, $R_1=1$, $R_2=2$, $R_3=3$, ${\sigma }_2=0.2$.

On the first dataset, we used a linear kernel function. In all methods, positive constant C was set to be $1000/N$, where N is the number of samples of all nodes. Because the feature space was only 2-dimensional, the sparse regularization term in our method was not necessary. Thus, we set the coefficient of sparse regularization term $\alpha =0$. In the distributed algorithm, we used the following diminishing step size:

$${\eta }^k=\frac{{\eta }^0}{1+\tau k},$$ (26)

which satisfied Condition (17). In (26), parameter ${\eta }^0$ determines the initial step size, and $\tau$ determines the decreasing rate of the diminishing step size. We empirically set ${\eta }^0=0.1$ and $\tau =0.01$ in the following experiments.

Figure 3a,b show the ACC and MAE curves of different algorithms on the first synthetic dataset. As time increased, the MAE of our dSVOR algorithm decreased, and the ACC increased significantly. After about 500 iterations, the dSVOR algorithm converged to a value that was almost the same as that of cSVOR, while the result of ncSVOR was still some distance away from them. This means that it was not enough for a single node to train a model with good performance using its own data. The proposed dSVOR algorithm, which uses the local data of each node and the parameter estimates from neighbor nodes, could achieve a similar performance to that of the corresponding centralized method.

Figure 3.

(a) ACC and (b) MAE curves of different algorithms on the first synthetic dataset.

Figure 4 gives the parameters of each node estimated by different algorithms. In the ncSVOR algorithm, the estimated parameters obtained by different nodes were quite different. Thus, the model obtained by each node with its own data was quite different from the model trained using all the data. In contrast, the estimated parameters of different nodes in dSVOR were almost the same as the parameters in cSVOR. This illustrates the consensus of the proposed dSVOR algorithm. Because we used a linear kernel function here, optimal direction $\mathit{w}$ in the centralized method had an explicit expression that allowed for us to compare it with the estimates of the distributed algorithms. In the following experiments using nonlinear kernel functions, we do not give the results about consensus.

Figure 4.

Final estimated parameters of different methods.

On the second dataset, we used a Gaussian kernel function. The kernel size was set to be $\sigma =\frac{1}{K}$ after Z-score normalization, where K is the dimension of input space. In all methods, positive constant C was set to be $1000/N$. As analyzed before, in our method, we set a relatively large D and a relatively large a, $D=200$, $\alpha =0.9$. $\alpha$ was not set to 1 because we wanted to use the strong convexity of the $l_2$-norm regularization term to increase the convexity of the objective function, which is theoretically beneficial to the optimization of the problem. The learning rate parameters were still set to be ${\eta }^0=0.1$ and $\tau =0.01$.

Figure 5a,b show the ACC and MAE curves of different algorithms on the second synthetic dataset. The proposed dSVOR algorithm was able to obtain almost the same result as that of the centralized method, while ncSVOR could not.

Figure 5.

(a) ACC and (b) MAE curves of different algorithms on the second synthetic dataset.

We also conducted experiments under different hyperparameters D and $\alpha$ to show the parameter sensitivity of dSVOR. Figure 6 gives the MAEs of dSVOR for different D when $\alpha$ was fixed as 0.9. As D increased, the performance of dSVOR gradually improved and was eventually almost the same as that of the centralized method. With a relatively large approximation dimension $D\ge 100$, dSVOR could always obtain a similar MAE to that of cSVOR. However, as mentioned before, an overlarge D may cause redundancy. So, when using a large D to ensure good performance, it is better to use the sparse regularization term to reduce the redundancy. Figure 7a,b gives the MAEs of dSVOR and the proportions of dimensions of ${\mathit{w}}_m$ that were equal to 0 for different $\alpha$ when D is fixed as 200. The MAE was stable under different $\alpha$, but the sparsity of ${\mathit{w}}_m$ was greatly affected by $\alpha$. A small $\alpha$ led to a dense ${\mathit{w}}_m$, which caused a lot of redundancy. A large $\alpha$ could bring a sparse ${\mathit{w}}_m$, where the dimensions that converged to 0 could no longer be stored, calculated, and transmitted after converging to 0, thus saving storage, computation, and communication resources.

Figure 6.

MAEs of dSVOR on the second synthetic dataset for different D when $\alpha$ is fixed as 0.9.

Figure 7.

Results of dSVOR on the second synthetic dataset for different $\alpha$ when D is fixed as 200. (a) MAEs; (b) proportions of dimensions of ${\mathit{w}}_m$ that are equal to 0.

5.2. A Real-World Example

We now take the distributed fault severity diagnosis of rolling element bearings as a real-world example to illustrate the effectiveness of dSVOR.

Rolling element bearings are widely used in factory equipments. The fault severity diagnosis of bearings is a crucial task to ensure reliability in industrial processes. In recent years, data-driven methods have been widely used to identify faults and their severity [36]. To achieve good performance, these data-driven methods usually require a lot of data. However, due to the rarity of faults, a single sensor can only collect very few fault data, and the faults encountered by each factory may also be different. Thus, data from many sensors in many factories are needed to train a proper model. Sometimes, factories may not want to leak the data about their equipments, so it is not allowed to transmit the data to others. The centralized methods which need all the data available in a central node become inapplicable. The distributed methods become a better choice. Taking into account the ordinal information in the fault severity, it is suitable to apply the proposed dSVOR algorithm.

In this study, we used the rolling element bearings data provided by the Case Western Reserve University (CWRU) [37] for experiments. CWRU data were the vibration signals of drive end and fan end bearings collected by sensors at 12,000 and 48,000 samples/s under four different loads of 0–3 hp. There are three types of faults: outer race (OR), inner race (IR), and ball (B) faults, and each type has at most four severity levels (fault width: 0.18, 0.36, 0.53, 0.71 mm). In the experiments, we used drive end bearing data collected at 12,000 samples/s, and performed 4-level fault severity diagnosis in a total of 12 situations (3 different fault types and 4 different loads).

We adopted the feature based on permutation entropy (PE) proposed in [38] as the input $\mathit{x}$. For one datum, we intercepted a sequence of length 2400 from vibration signal data. This sequence was decomposed into a series of intrinsic mode functions (IMFs) by ensemble empirical mode decomposition (EEMD) with 100 ensembles and 0.2 noise amplitude to catch information on multiple time scales. Then, the PE values of the first 5 IMFs are calculated as the input feature of this piece of data.

For each fault severity level, we randomly took 300 training samples and 200 testing samples, and the samples in the testing set were different from those in the training set. For 4-level fault level diagnosis, there were a total of 1200 training samples and 800 testing samples. These training samples were randomly assigned to 20 nodes to simulate the situation where the data were collected and stored by these nodes in a distributed manner.

In the experiments, we used a Gaussian kernel function with kernel size $\sigma =\frac{1}{K}$ after Z-score normalization. In all methods, positive constant C was set to be 10,000/N. In our method, we still set a relatively large hyperparameter D and $\alpha$, $D=200$, $\alpha =0.9$. The other parameters used the same settings as before, i.e., ${\eta }^0=0.1$ and $\tau =0.01$.

Table 1 shows the experimental results where the value was the mean ± standard deviation of 10 independent experiments. The performance of ncSVOR was worse than that of cSVOR because each node only had part of the training samples that were not enough to represent the entire training set to train a proper model. Compared to ncSVOR, the proposed dSVOR algorithm could achieve similar results to those of cSVOR. In dSVOR, each node can only use the data of its own and exchange some estimated parameters with neighbor nodes. It was satisfactory to be able to achieve performance close to that of the centralized method that uses all the data from all nodes.

Table 1.

ACCs and MAEs of different algorithms in a real-world example (mean ± std).

Fault Type	Load	cSVOR		ncSVOR		dSVOR
		ACC	MAE	ACC	MAE	ACC	MAE
>OR	0	0.9585 ± 0.0057	0.0415 ± 0.0057	0.7977 ± 0.0211	0.2069 ± 0.0230	0.9553 ± 0.0064	0.0447 ± 0.0064
	1	0.9317 ± 0.0147	0.0683 ± 0.0147	0.7376 ± 0.0228	0.2726 ± 0.0264	0.9278 ± 0.0136	0.0727 ± 0.0138
	2	0.9547 ± 0.0091	0.0457 ± 0.0096	0.7901 ± 0.0136	0.2172 ± 0.0153	0.9517 ± 0.0094	0.0492 ± 0.0099
	3	0.9253 ± 0.0099	0.0747 ± 0.0099	0.7599 ± 0.0158	0.2489 ± 0.0173	0.9243 ± 0.0095	0.0758 ± 0.0096
>IR	0	0.8853 ± 0.0133	0.1149 ± 0.0133	0.7472 ± 0.0087	0.2589 ± 0.0091	0.8844 ± 0.0120	0.1157 ± 0.0120
	1	0.8624 ± 0.0112	0.1376 ± 0.0112	0.7288 ± 0.0103	0.2781 ± 0.0110	0.8556 ± 0.0137	0.1444 ± 0.0137
	2	0.8435 ± 0.0109	0.1565 ± 0.0109	0.7071 ± 0.0116	0.3000 ± 0.0133	0.8391 ± 0.0113	0.1611 ± 0.0113
	3	0.8726 ± 0.0095	0.1291 ± 0.0091	0.7238 ± 0.0110	0.2918 ± 0.0122	0.8632 ± 0.0094	0.1392 ± 0.0091
B	0	0.7768 ± 0.0110	0.2586 ± 0.0129	0.5440 ± 0.0184	0.5975 ± 0.0311	0.7594 ± 0.0221	0.2771 ± 0.0245
	1	0.7836 ± 0.0105	0.2419 ± 0.0099	0.5770 ± 0.0124	0.5284 ± 0.0195	0.7710 ± 0.0067	0.2540 ± 0.0106
	2	0.8256 ± 0.0088	0.1886 ± 0.0088	0.5820 ± 0.0156	0.5341 ± 0.0264	0.8177 ± 0.0147	0.1980 ± 0.0150
	3	0.8627 ± 0.0167	0.1541 ± 0.0193	0.6345 ± 0.0138	0.4648 ± 0.0253	0.8485 ± 0.0169	0.1710 ± 0.0204

Taking the dataset of the IR fault type and 0 hp load as examples, we also show the results of dSVOR under different hyperparameters D and $\alpha$ in Figure 8 and Figure 9. Figure 8 shows that, with a relatively large random approximation dimension $D\ge 100$, dSVOR could obtain a similar MAE to that of cSVOR, which illustrates the effectiveness of the random approximation. Figure 9 shows that a relatively large $\alpha$ can lead to a sparse ${\mathit{w}}_m$ without affecting the MAE performance, thus effectively reducing redundancy.

Figure 8.

MAEs of dSVOR on the CWRU dataset of IR fault type and 0 hp load for different D when $\alpha$ is fixed as 0.9.

Figure 9.

The results of dSVOR on the CWRU dataset of IR fault type and 0 hp load for different $\alpha$ when D is fixed as 200 (a) the MAEs (b) the proportions of dimensions of ${\mathit{w}}_m$ that are equal to 0.

6. Conclusions

When data are distributedly collected and stored by multiple nodes, and are difficult to transmit to a central node, existing centralized ordinal regression methods become inapplicable. To this end, in order to handle the ordinal regression problem in distribution scenarios, we extended the SVORIM to a distributed version, and derived a distributed SVOR (dSVOR) algorithm. In dSVOR, each node combines the parameters estimated by its neighbors and performs local calculations using only its own data. After convergence, each node can obtain a model whose performance is close to that obtained by the centralized method relying on all the data available in a central node. Theoretically, we analyzed the consensus and the convergence of dSVOR. Practically, we carried out experiments on synthetic data and a real-world example to illustrate its effectiveness.

In our future work, we intend to consider how to automatically determine the proper parameters in dSVOR, e.g., introducing multi-kernel learning to automatically find suitable parameters of random approximate. We also aim to design adaptive strategies for adjusting combination coefficients.

Appendix A. Proof of Theorem 1

Proof. 

For convenience, we use the following notation:

$$\begin{array}{c}\hfill {\mathbf{\theta }}^k={[{\theta }_1^k,\dots ,{\theta }_M^k]}^T,{\mathbf{G}}_{\mathbf{\theta }}^k={[{\nabla }_{{\theta }_1}{J}_1\left({\theta }_1^k\right),\dots ,{\nabla }_{{\theta }_M}{J}_M\left({\theta }_M^k\right)]}^T.\end{array}$$ (A1)

Then, Iterative Equation (20) can be written as follows.

$${\mathbf{\theta }}^{k+1}=\mathit{C}{\mathbf{\theta }}^k-{\eta }^k{\mathbf{G}}_{\mathbf{\theta }}^k.$$ (A2)

Considering ${\overline{\theta }}^k=\frac{1}{M}{\mathbf{1}}^T{\mathbf{\theta }}^k$, the proof of ${lim}_{k\to \infty }\parallel {\theta }_m^k-{\overline{\theta }}^k\parallel =0,\forall m$ can be done by proving ${lim}_{k\to \infty }\parallel {\mathbf{\theta }}^k-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T{\mathbf{\theta }}^k\parallel =0$. We first construct

$$\begin{array}{cc}\hfill {\mathbf{\theta }}^{k+1}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T{\mathbf{\theta }}^{k+1}& =(\mathbf{I}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T){\mathbf{\theta }}^{k+1}=(\mathbf{I}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T)(\mathit{C}{\mathbf{\theta }}^k-{\eta }^k{\mathbf{G}}_{\mathbf{\theta }}^k)\hfill \\ \hfill & =(\mathit{C}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T){\mathbf{\theta }}^k-(\mathbf{I}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T){\eta }^k{\mathbf{G}}_{\mathbf{\theta }}^k.\hfill \end{array}$$ (A3)

Notice that

$$\mathit{C}\frac{1}{M}\mathbf{1}{\mathbf{1}}^T=\frac{1}{M}\mathbf{1}{\mathbf{1}}^T=\frac{1}{M}\mathbf{1}{\mathbf{1}}^T\frac{1}{M}\mathbf{1}{\mathbf{1}}^T.$$ (A4)

We have

$$\begin{array}{cc}\hfill {\mathbf{\theta }}^{k+1}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T{\mathbf{\theta }}^{k+1}& =(\mathit{C}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T){\mathbf{\theta }}^k-(\mathbf{I}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T){\eta }^k{\mathbf{G}}_{\mathbf{\theta }}^k\hfill \\ \hfill & -\mathit{C}\frac{1}{M}\mathbf{1}{\mathbf{1}}^T{\mathbf{\theta }}^k+\frac{1}{M}\mathbf{1}{\mathbf{1}}^T\frac{1}{M}\mathbf{1}{\mathbf{1}}^T{\mathbf{\theta }}^k\hfill \\ \hfill & =(\mathit{C}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T)({\mathbf{\theta }}^k-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T{\mathbf{\theta }}^k)-(\mathbf{I}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T){\eta }^k{\mathbf{G}}_{\mathbf{\theta }}^k.\hfill \end{array}$$ (A5)

For convenience, we use notation $\Delta {\mathbf{\theta }}^k={\mathbf{\theta }}^k-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T{\mathbf{\theta }}^k$, then

$$\begin{array}{c}\hfill \Delta {\mathbf{\theta }}^{k+1}=(\mathit{C}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T)\Delta {\mathbf{\theta }}^k-(\mathbf{I}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T){\eta }^k{\mathbf{G}}_{\mathbf{\theta }}^k.\end{array}$$ (A6)

We then prove ${lim}_{k\to \infty }\parallel \Delta {\mathbf{\theta }}^k\parallel =0$.

Taking the $l_2$-norm on both sides of the above equation, we have

$$\begin{array}{cc}\hfill \parallel \Delta {\mathbf{\theta }}^{k+1}\parallel & \le \parallel \mathit{C}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T\parallel \parallel \Delta {\mathbf{\theta }}^k\parallel +{\eta }^k\parallel \mathbf{I}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T\parallel \parallel {\mathbf{G}}_{\mathbf{\theta }}^k\parallel =\rho \parallel \Delta {\mathbf{\theta }}^k\parallel +{\eta }^{k}c\parallel {\mathbf{G}}_{\mathbf{\theta }}^k\parallel ,\hfill \end{array}$$ (A7)

where $c=\parallel \mathbf{I}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T\parallel$ is a positive constant, and $\rho$ denotes the spectral norm of $\mathit{C}-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T$, which $<1$ according to Assumption A1.

Since $J_m\left({\theta }_m\right)$ is Lipschitz continuous, ${\mathbf{G}}_{\mathbf{\theta }}^k$ is bounded, so there exists a positive constant L satisfying $\parallel {\mathbf{G}}_{\mathbf{\theta }}^k\parallel \le L$. Thus,

$$\parallel \Delta {\mathbf{\theta }}^{k+1}\parallel \le \rho \parallel \Delta {\mathbf{\theta }}^k\parallel +{\eta }^{k}cL.$$ (A8)

Now we prove that ${lim}_{k\to \infty }\parallel \Delta {\mathbf{\theta }}^k\parallel =0$. To achieve this, we constructed an auxiliary variable $u^k$ that satisfied

$$u^{k+1}=\rho u^k+{\eta }^{k}cL,$$ (A9)

and $u^0=\parallel \Delta {\mathbf{\theta }}^0\parallel \ge 0$. If $u^k\ge \parallel \Delta {\mathbf{\theta }}^k\parallel \ge 0$, then

$$u^{k+1}=\rho u^k+{\eta }^{k}cL\ge \rho \parallel \Delta {\mathbf{\theta }}^k\parallel +{\eta }^{k}cL\ge \parallel \Delta {\mathbf{\theta }}^{k+1}\parallel .$$ (A10)

So $u^k\ge \parallel \Delta {\mathbf{\theta }}^k\parallel \ge 0$ for all $k\ge 0$. With $\rho <1$ and ${lim}_{k\to \infty }{\eta }^k=0$, we have ${lim}_{k\to \infty }{u}^k=0$. Then

$$0\le \underset{k\to \infty }{lim}\parallel \Delta {\mathbf{\theta }}^k\parallel \le \underset{k\to \infty }{lim}{u}^k=0.$$ (A11)

So we have

$$\underset{k\to \infty }{lim}\parallel {\mathbf{\theta }}^k-\frac{1}{M}\mathbf{1}{\mathbf{1}}^T{\mathbf{\theta }}^k\parallel =\underset{k\to \infty }{lim}\parallel \Delta {\mathbf{\theta }}^k\parallel =0.$$ (A12)

The proof of Theorem 1 is completed. □

Appendix B. Proof of Theorem 2

Proof. 

From Equation (20), we can obtain

$$\begin{array}{cc}\hfill \sum _{m=1}^M{\theta }_m^{k+1}& =\sum _{m=1}^M\sum _{n\in {\mathcal{N}}_m\cup \left\{m\right\}}{c}_{mn}{\theta }_n^k-\sum _{m=1}^M{\eta }^k{\nabla }_{{\theta }_m}{J}_m\left({\theta }_m^k\right)\hfill \\ \hfill & =\sum _{m=1}^M{\theta }_m^k-\sum _{m=1}^M{\eta }^k{\nabla }_{{\theta }_m}{J}_m\left({\theta }_m^k\right).\hfill \end{array}$$ (A13)

From Theorem 1, we have ${lim}_{k\to \infty }\parallel {\theta }_m^k-{\overline{\theta }}^k\parallel =0$. So, for a sufficiently large $k=k_1$, the above equation can be written as follows:

$${\overline{\theta }}^{k_1+1}={\overline{\theta }}^{k_1}-\frac{{\eta }^{k_1}}{M}\sum _{m=1}^M{\nabla }_{{\theta }_m}{J}_m\left({\overline{\theta }}^{k_1}\right),$$ (A14)

where ${\nabla }_{{\theta }_m}{J}_m\left({\overline{\theta }}^{k_1}\right)$ denotes the subgradient of $J_m\left({\theta }_m\right)$ with respect to ${\theta }_m$ when ${\theta }_m={\overline{\theta }}^{k_1}$.

Supposing ${\theta }^{*}=argminJ$, we have

$$\begin{array}{cc}\hfill \parallel {\overline{\theta }}^{k_1+1}-{\theta }^{*}{\parallel }^2& =\parallel {\overline{\theta }}^{k_1}-\frac{{\eta }^{k_1}}{M}\sum _{m=1}^M{\nabla }_{{\theta }_m}{J}_m\left({\overline{\theta }}^{k_1}\right)-{\theta }^{*}{\parallel }^2\hfill \\ \hfill & =\parallel {\overline{\theta }}^{k_1}-{\theta }^{*}{\parallel }^2+{\parallel \frac{{\eta }^{k_1}}{M}\sum _{m=1}^M{\nabla }_{{\theta }_m}{J}_m\left({\overline{\theta }}^{k_1}\right)\parallel }^2\hfill \\ \hfill & -2\frac{{\eta }^{k_1}}{M}\sum _{m=1}^M{\nabla }_{{\theta }_m}{J}_m{\left({\overline{\theta }}^{k_1}\right)}^T({\overline{\theta }}^{k_1}-{\theta }^{*}).\hfill \end{array}$$ (A15)

Since $J_m\left({\theta }_m\right)$ is Lipschitz continuous for all m, there exists a positive constant L satisfying $\parallel \frac{1}{M}{\sum }_{m=1}^M{\nabla }_{{\theta }_m}{J}_m\left({\theta }_m\right){\parallel }^2\le L^2$. Thus,

$$\begin{array}{cc}\hfill \parallel {\overline{\theta }}^{k_1+1}-{\theta }^{*}{\parallel }^2& \le \parallel {\overline{\theta }}^{k_1}-{\theta }^{*}{\parallel }^2+{\left({\eta }^{k_1}\right)}^2{L}^2-2\frac{{\eta }^{k_1}}{M}\sum _{m=1}^M{\nabla }_{{\theta }_m}{J}_m{\left({\overline{\theta }}^{k_1}\right)}^T({\overline{\theta }}^{k_1}-{\theta }^{*}).\hfill \end{array}$$ (A16)

Because $J_m\left({\theta }_m\right)$ is convex, we have

$$\begin{array}{c}\hfill {\nabla }_{{\theta }_m}{J}_m{\left({\overline{\theta }}^{k_1}\right)}^T({\overline{\theta }}^{k_1}-{\theta }^{*})\ge J_m\left({\overline{\theta }}^{k_1}\right)-J_m\left({\theta }^{*}\right).\end{array}$$ (A17)

Then

$$\begin{array}{cc}\hfill \parallel {\overline{\theta }}^{k_1+1}-{\theta }^{*}{\parallel }^2& \le \parallel {\overline{\theta }}^{k_1}-{\theta }^{*}{\parallel }^2+{\left({\eta }^{k_1}\right)}^2{L}^2-2\frac{{\eta }^{k_1}}{M}\sum _{m=1}^M(J_m\left({\overline{\theta }}^{k_1}\right)-J_m\left({\theta }^{*}\right))\hfill \\ \hfill & =\parallel {\overline{\theta }}^{k_1}-{\theta }^{*}{\parallel }^2+{\left({\eta }^{k_1}\right)}^2{L}^2-2\frac{{\eta }^{k_1}}{M}(J\left({\overline{\theta }}^{k_1}\right)-J^{*}).\hfill \end{array}$$ (A18)

If ${lim}_{k\to \infty }{\sum }_{m=1}^M{J}_m\left({\theta }_m^k\right)\ne J^{*}$, $\exists \epsilon >0,k_2>0$, $\forall k\ge k_2,J\left({\overline{\theta }}^{k_2}\right)-J^{*}>\epsilon$. Let $k_3=max\{k_1,k_2\}$. Then

$$\begin{array}{cc}\hfill \parallel {\overline{\theta }}^{k_3+1}-{\theta }^{*}{\parallel }^2& <\parallel {\overline{\theta }}^{k_3}-{\theta }^{*}{\parallel }^2+{\left({\eta }^{k_3}\right)}^2{L}^2-2\frac{{\eta }^{k_3}}{M}\epsilon .\hfill \end{array}$$ (A19)

Taking the summation of both sides of the above equation over $k=k_3,\dots ,k_3+k^{*}$, we obtain

$$\begin{array}{cc}\hfill \parallel {\overline{\theta }}^{k_3+k^{*}}-{\theta }^{*}{\parallel }^2& <\parallel {\overline{\theta }}^{k_3}-{\theta }^{*}{\parallel }^2+\sum _{k=k_3}^{k_3+k^{*}}{\left({\eta }^k\right)}^2{L}^2-\sum _{k=k_3}^{k_3+k^{*}}2\frac{{\eta }^k}{M}\epsilon .\hfill \end{array}$$ (A20)

Since $\parallel {\overline{\theta }}^{k_3+k^{*}}-{\theta }^{*}{\parallel }^2\ge 0$, we have

$$\begin{array}{c}\hfill \parallel {\overline{\theta }}^{k_3}-{\theta }^{*}{\parallel }^2+\sum _{k=k_3}^{k_3+k^{*}}{\left({\eta }^k\right)}^2{L}^2>\sum _{k=k_3}^{k_3+k^{*}}2\frac{{\eta }^k}{M}\epsilon .\end{array}$$ (A21)

Thus,

$$\begin{array}{c}\hfill \frac{\parallel {\overline{\theta }}^{k_3}-{\theta }^{*}{\parallel }^2+L^2{\sum }_{k=k_3}^{k_3+k^{*}}{\left({\eta }^k\right)}^2}{\frac{2}{M}{\sum }_{k=k_3}^{k_3+k^{*}}{\eta }^k}>\epsilon .\end{array}$$ (A22)

Since ${\sum }_{k=0}^{+\infty }{\eta }^k=+\infty$ and ${\sum }_{k=0}^{+\infty }{\left({\eta }^k\right)}^2<+\infty$,

$$\begin{array}{c}\hfill \underset{k^{*}\to \infty }{lim}\frac{\parallel {\overline{\theta }}^{k_3}-{\theta }^{*}{\parallel }^2+L^2{\sum }_{k=k_3}^{k_3+k^{*}}{\left({\eta }^k\right)}^2}{\frac{2}{M}{\sum }_{k=k_3}^{k_3+k^{*}}{\eta }^k}=0,\end{array}$$ (A23)

which conflicts with Equation (A22). Thus,

$$\underset{k\to \infty }{lim}\sum _{m=1}^M{J}_m\left({\theta }_m^k\right)=J^{*}.$$ (A24)

The proof of Theorem 2 is completed. □

Author Contributions

Conceptualization, H.L. and C.L.; methodology, H.L. and C.L.; formal analysis, H.L., J.T. and C.L.; writing—original draft preparation, H.L.; writing—review and editing, H.L., J.T. and C.L. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

the authors declare no conflict of interest.

Funding Statement

This work was supported by the National Natural Science Foundation of China (grant No. U20A20158), the Key-Area Research and Development Program of Guangdong Province (grant No. 2021B0101410004), and the National Program for Special Support of Eminent Professionals.