Supervised Nonnegative Matrix Factorization to Predict ICU Mortality Risk

Abstract

ICU mortality risk prediction is a tough yet important task. On one hand, due to the complex temporal data collected, it is difficult to identify the effective features and interpret them easily; on the other hand, good prediction can help clinicians take timely actions to prevent the mortality. These correspond to the interpretability and accuracy problems. Most existing methods lack of the interpretability, but recently Subgraph Augmented Nonnegative Matrix Factorization (SANMF) has been successfully applied to time series data to provide a path to interpret the features well. Therefore, we adopted this approach as the backbone to analyze the patient data. One limitation of the original SANMF method is its poor prediction ability due to its unsupervised nature. To deal with this problem, we proposed a supervised SANMF algorithm by integrating the logistic regression loss function into the NMF framework and solved it with an alternating optimization procedure. We used the simulation data to verify the effectiveness of this method, and then we applied it to ICU mortality risk prediction and demonstrated its superiority over other conventional supervised NMF methods.

I. Introduction

With the fast development of machine learning and data mining, automated predictive modeling becomes possible when combining with the increasingly available medical data in hospitals and clinical institutions. These models can discover latent patterns hidden in the data and help clinicians make timely and accurate decisions. The data generated in the Intensive Care Unit (ICU) are abundant and complicated due to the diverse variable types (especially the continuous time series data). One key event in ICU is patient death. However, the complex nature of the ICU data usually makes the clinicians difficult to make timely and correct decisions. This makes predictive modeling of the mortality risk an important problem.

In this paper, we describe a supervised nonnegative matrix factorization (NMF) algorithm that performs predictive modeling by the exploration of atomic and the higher-order features jointly. We automate the mining of higher-order features by converting data into graph representation. Moreover, supervised latent group identification reduces dimensionality of different feature types for patients, and simultaneously group temporal trends to form effective features for patient outcome prediction. Applications on patient physiological time series [1] show significant performance improvements from multiple baselines.

II. Related Work

Nonnegative Matrix Factorization (NMF) refers to the set of problems on approximating a nonnegative matrix as the product of several nonnegative matrices. The problem has become popular since the work [2], where they form a nonnegative matrix by concatenating the set of pixel intensity vectors stretched from human facial images. After factorizing such matrix into the product of two matrices, they found that one matrix can be interpreted as the set of image basis with part based representation of human faces, and the other matrix has the coefficients if we were to reconstruct the face image from those bases. Because of the nonnegativity constraints, NMF is not a convex problem and they developed a multiplicative updates algorithm to obtain a stationary solution, and they prove the convergence of the algorithm in their paper [3].

Since then people have been working on NMF from various aspects. [4] showed that there is some equivalence between NMF and K-means/spectral clustering and thus claimed NMF can be used for data clustering purpose [5]. [6] further developed a t-NMF approach that can perform co-clustering on both matrix columns and rows. They also discussed the various variants of NMF in [7]. [8] extended NMF to the case when the matrix approximation loss is measured by Bregman divergence, which is a much more general loss with both Frobenius norm and KL divergence that was introduced in [3] as its special cases. On the solution procedure aspect, multiplicative updates has been recognized for its slow convergence and poor quality. [9] reviewed the general algorithms before 2007, three classes of algorithms are categorized. The first class is multiplicative updates, the second class is gradient based methods such as [10], [11], [12], the third class is the alternating least squares (ALS) algorithm [13], [14], [15]. Also in 2007, [16] proposed an efficient projected gradient approach for NMF, which adopted Taylor expansion strategy to reduce the cost. [17] also proposed an active set type of method called principal block pivoting to solve the NMF problem. [18] adopted the alternating direction method of multipliers (ADMM) to solve the NMF with beta-divergence. Hsieh and Dhillon [19] designed a fast coordinate descent method to accelerate the FastHals [20].

Apart from basic NMF method introduced above, there are also many variants. They can be grouped into three groups. The first group enforced some constraints into basic NMF to obtain certain desirable charateristics, such as sparse NMF [10], [21], [22], orthogonal NMF [23], [24], [25], discrininant NMF [26], [27] and manifold NMF [28], [29]. The second group named structured NMF modified the standard formulation of NMF, including weighted NMF [30], convolutive NMF [31] and nonnegative matrix trifactorization [25]. The third group is the generalized NMF, including semi-NMF [32], nonnegative tensor factorization [33], matrix-set factorization [34] and kernel NMF [35]. For details, refer to the NMF survey paper [36].

Although these general NMFs decrease the dimension of the original data successfully, it cannot guarantee that the prediction ability is retained because they’re unsupervised methods that have not used the label information. To tackle this problem, several supervised NMF methods were proposed. They can be classified into two categories. The first category included [26], [27], [37] is the method incorporating the idea of linear discriminative analysis to improve the prediction ability of the reduced representation, while the second category included [38], [39] is the method introducing a loss function to take the label information into consideration.

Our proposed supervised NMF belongs to the second category. The previous two methods [38], [39] enforced a frebenius loss to constrain the label information, however, it seems more like the regression not classification. Therefore, the frebenius loss constraint is not enough to exploit the label information for classification. Instead of the frebenius loss constraint, we enforce the classification loss constraint explicitly to guarantee the prediction ability of the learned representation. In fact, we proposed a supervised NMF framework that can incorporate any classification loss function. In this paper, we implement one with the logistic regression, and other classification loss functions can be considered to be integrated in furture work.

III. Methods

A. NMF and Logistic Regression

NMF [2] is a popular representation learning technique. Assume that we have data matrix $\mathit{X}\in {\mathcal{R}}^{n\times m}$ and its corresponding label $y\in {\mathcal{R}}^n$. NMF aims to learn two nonnegative matrices $\mathit{U}\in {\mathcal{R}}^{n\times r}$ and $\mathit{V}\in {\mathcal{R}}^{r\times m}$ to approximate a nonnegative matrix $\mathit{X}\in {\mathcal{R}}^{n\times m}$, each row of which contains m features of one of the n samples. Then NMF can be formulated in matrix form X = UV, and at the same time U and V are constrained to be nonnegative. Therefore, NMF is formulated as the following optimization problem.

$$\begin{array}{l}\underset{\mathit{U},\mathit{V}}{\text{min}}\Vert \mathit{X}-\mathit{U}\mathit{V}{\Vert }_F^2\\ s.t.\mathit{U}\succcurlyeq 0,\mathit{V}\succcurlyeq 0,\end{array}$$ (1)

where $\Vert \cdot {\Vert }_F^2$ indicates squared frobenius norm and U ≽ 0, V ≽ 0 mean that U and V are both entry-wise nonnegative. r rows of V are considered as new bases while r columns of U are viewed as the coefficients with which the original samples can be represented with a linear combination of bases. U can be considered as a low-dimensional representation of X since generally r < m.

It can be seen that NMF cannot utilize the label information to learn the representation and thus it cannot guarantee the classification performance. To take the advantage of the supervised information, we will introduce the supervised learning methods, precisely, the loss function for supervised learning methods. Supervised learning is the learning paradigm that uses labels to learn, like classification. Herein, we will introduce a popular supervised learning method Logistic regression (LR).

Different from NMF, LR can make good use of the label information to classify. The optimization problem of LR can be formulated as follows:

$$\underset{\mathit{w},b}{\text{min}}\sum _{i=1}^n\text{ln}\left(1+\text{exp}\left(-y_i\left(\sum _{j=1}^r{w}_j{\mathit{X}}_{ij}+b\right)\right)\right)+\frac{1}{2}\beta \left(\sum _{j=1}^r{w}_j^2+b^2\right)$$ (2)

where X_i,j indicates the jth feature of ith data point of the X, w and b indicate the weight and bias of LR. In Eq. (2) the first item is the loss function while the second one is the regularization item to prevent over-fitting. Parameter β is used to balance the loss and regularization items.

B. Supervised NMF Problem Formulation

Given data matrix X^n×m and label vector y^n×1, where n is the number of patients, m is the number of subgraphs, and the number of the labels is 1. Note that entries of y is 1 or 0. Coefficient matrix U^n×r, basis matrix V^r×m and weight matrix w^r×1, where r is the number of subgraph groups. The supervised nonnegative matrix factorization (SNMF) can be formulated as

$$\begin{array}{l}\underset{{}_{\mathit{U},\mathit{V},\mathit{w},b}}{\text{min}}{L}_f+L_{lr}+L_r\\ s.t.\mathit{U}\succcurlyeq 0,\mathit{V}\succcurlyeq 0,\end{array}$$ (3)

where the first item $L_f=\frac{1}{2}\Vert \mathit{X}-\mathit{U}\mathit{V}{\Vert }_F^2$ is the loss function for NMF, the second item $L_f=\alpha {\sum }_{i=1}^n\text{ln}\left(1+\text{exp}\left(-y_i\left({\sum }_{j=1}^r{w}_j{u}_{ij}+b\right)\right)\right)$ is the loss function for LR and the third item $L_r=\frac{1}{2}\beta \left({\sum }_{j=1}^r{w}_j^2+b^2\right)+\frac{1}{2}\gamma \Vert \mathit{U}{\Vert }_F^2$ is the regularization for NMF and LR.

Obviously, NMF and LR are integrated into this united framework. Since U is the new representation we aim to learn, LR works on it instead of original data matrix X. The last item is used to regularize U. α, β and γ are used to balance the role of those corresponding items.

C. Optimization

For the supervised NMF optimization problem (3), we can find that NMF and LR objectives are integrated together. In introduction part we have mentioned that projected gradient descent method [38] can solve this problem and gradient descent method is also the general algorithm to solve LR. we can also see that some of the variables U, V, w and b are interwined with each other, alternative minimization is an suitable tool to solve this kind of problems.

Our optimization will be split into four subproblem minimizations: minimize U with fixed V, w, and b; minimize V with fixed U, w, and b; minimize w with fixed U, V, and b; minimize b with fixed U, V, w. These four subproblem minimizations will be alternatively executed until the predefined termination condition is satisfied. For each of the four steps, projected gradient descent or gradient descent will be adopted to solve the subproblems.

The gradients of the objective function in Eq. (3) with respect to the four variables are given below, and the detailed derivations are provided in Appendix (can be obtained in the link https://github.com/guoqingchao/Supervised-Nonnegative-Matrix-Factorization-to-Predict-ICU-Mortality-Risk).

$${\nabla }_{\mathit{U}}L=-\mathit{X}{\mathit{V}}^T+\mathit{U}\mathit{V}{\mathit{V}}^T+\gamma \mathit{U}-\alpha \left(\mathit{y}{\mathit{w}}^T\right)\oslash \mathit{D}.$$ (4)

$${\nabla }_{\mathit{V}}L=-{\mathit{U}}^T\mathit{X}+{\mathit{U}}^T\mathit{U}\mathit{V}.$$ (5)

$${\nabla }_{\mathit{w}}L=-\alpha {(\mathit{U}\odot \mathit{Y}\oslash \mathit{D})}^T*{\mathit{e}}_{\mathit{n}}+\beta \mathit{w}.$$ (6)

$${\nabla }_{\mathit{b}}L=-\alpha {\mathit{e}}_{\mathit{n}}^T*(\mathit{y}\oslash (1+\text{exp}(\mathit{U}\mathit{w}+b)\odot \mathit{y}))+\beta b.$$ (7)

where ⊘ indicates element-wise division and ⊙ indicates element-wise multiplication. D is obtained by repeating the column vector (1+exp(Uw + be_n)) ⊙ y r times to form the matrix of size n×r. Y is obtained with the same operator to vector y.

The update formulation of these four variables are given below:

$${\mathit{U}}^{t+1}={\mathcal{P}}_{+}\left[{\mathit{U}}^t-{\eta }_{\mathit{U}}^t{\nabla }_{\mathit{U}}L\left({\mathit{U}}^t,{\mathit{V}}^t,{\mathit{w}}^t,b^t\right)\right]$$ (8)

$${\mathit{V}}^{t+1}={\mathcal{P}}_{+}\left[{\mathit{V}}^t-{\eta }_{\mathit{V}}^t{\nabla }_{\mathit{V}}L\left({\mathit{U}}^{t+1},{\mathit{V}}^t,{\mathit{w}}^t,b^t\right)\right]$$ (9)

$${\mathit{w}}^{t+1}={\mathit{w}}^t-{\eta }_{\mathit{w}}^t{\nabla }_{\mathit{w}}L\left({\mathit{U}}^{t+1},{\mathit{V}}^{t+1},{\mathit{w}}^t,b^t\right)$$ (10)

$$b^{t+1}=b^t-{\eta }_b^t{\nabla }_{b}L\left({\mathit{U}}^{t+1},{\mathit{V}}^{t+1},{\mathit{w}}^{t+1},b^t\right)$$ (11)

The scalar form of the problem Eq.(3) can be written in matrix form as follows.

$$\begin{array}{l}\underset{{}_{\mathit{U},\mathit{V},\mathit{w},b}}{\text{min}}{L}_f+L_{lr}+L_r\\ s.t.\mathit{U}\succcurlyeq 0,\mathit{V}\succcurlyeq 0,\end{array}$$ (12)

where the first item $L_f=\frac{1}{2}\Vert \mathit{X}-\mathit{U}\mathit{V}{\Vert }_F^2$ indicates the frobenius loss function for NMF, the second item L_lr = αe_n^T * ln (1 + exp(−(U_w + be_n) ⊙ y))) indicates the loss function for LR and the third item $L_r=\frac{1}{2}\beta \left({\mathit{w}}^T\mathit{w}+b^2\right)+\frac{1}{2}\gamma \Vert \mathit{U}{\Vert }_F^2$ indicates the regulairzation for NMF and LR.

To understand the algorithm clear, we summarized the algorithm in Algorithm 1.

IV. Experiments

Before applying the proposed method SNMF to the ICU mortality risk prediction problem, we verify the effectiveness of SNMF on the simulation data.

A. Simulation study

To demonstrate the effectiveness of the proposed supervised NMF method, we generate a simulation data to verify it. The data generating process includes three steps: Firstly, the coefficient matrix U is generated with two Gaussian distributions to indicate the true latent representation, the mean vectors and covariance matrices of the two Gaussian distributions are μ₁ = (1,1,1,1,1), Σ1 = diag(1,1,1,1,1) and μ₂ = (3,3,1,1,1), Σ2 = diag(1,1,1,1,1), respectively. 250 points are generated for each group, so that ${\mathit{U}}_1\in {\mathcal{R}}^{250\times 5}$ and ${\mathit{U}}_2\in {\mathcal{R}}^{250\times 5}$ indicate the coefficient matrices corresponding to two groups of data. In order to guarantee U nonnegative, all the negative entries are set to zero. Here, we use vector y to indicate which Gaussian distribution the point is from and form the labels of the generated data. Secondly, the basis matrix $\mathit{V}\in {\mathcal{R}}^{5\times 10}$ is generated with the Uniform distribution in (0,1) and set all the negative entries to zero to make sure V is nonnegative. Thirdly, the noise e is generated using a Gaussian distribution with the same mean and covariance matrix with U * V and then added into the matrix U * V, see Eq. (13).

$$\mathit{X}=\mathit{U}*\mathit{V}+\eta *e$$ (13)

where e is the noise from the Gaussian distribution $\mathcal{N}(\mathit{\mu },\Sigma )$ and μ and Σ indicate the mean and covariance matrix of data matrix U * V, η indicates the noise level. Final data X is ${\mathcal{R}}^{500\times 10}$. It should be noted that all the negative entries of X are set to zero to obtain the nonnegative matrix X. y takes values 0 or 1 to indicate the label. If the data point is generated from the first Gaussian distribution, its label will be 0 otherwise 1.

We split the obtained data matrix X and corresponding labels y into training and test sets with the ratio half to half, and then we compared the representation obtained from unsupervised NMF with our proposed SNMF. In addition, another supervised NMF method FNMF [27] is also added to compare. For the generated coefficient matrix U, we know there are 2 distinguished dimensions, to make sure these methods can capture this distinguished information, for all the four methods, we set the dimension of the new representation as 2. For our proposed SNMF, the range of parameters α, β, γ are from {0,0.001,0.01,0.1}. It should be noted that when all of α, β, γ take value 0, SNMF degenerates to NMF, and in our experimental results we adopt this as unsupervised NMF. To tune parameters efficiently, the parameters α, β, γ are set to α/p, β/r and γ/(n × r) where p is the number of the features of data matrix X, r is the number of the features of the latent representation matrix U and n is the number of the samples of data matrix X. To further explore how these methods perform on the data with different noise level, we conduct the experimental comparison on the simulation data with three noise level 0, 0.2, and 0.5. The metric we adopt is the area under ROC (receiver operating characteristic) curve (AUC). All the experimental results are demonstrated in Table I, the best result is shown in bold font for each case.

TABLE I.

The AUC on the simulation data (%), the value in bold indicates the difference with the rest is significant (p value <0.01).

Noise Level	Data Split	NMF	FNMF	SNMF
η = 0.5	Test	90.35	89.19	91.78
η = 0.2	Test	93.91	94.40	95.68
η = 0	Test	95.23	95.96	97.15

From Table I, we can find that on the data without any noise, FNMF performs better than NMF but worse than our SNMF, because FNMF learned the new representation with the help of the label information while NMF doesn’t. Compared with SNMF, FNMF just guarantees the data points within the same class close and those in different classes far away in the new representation but it still doesn’t use the label information directly to approach the final goal classification, so SNMF is more effective. With increasing the noise level, we can see all the methods’ performance degrade, but SNMF outperforms all the other methods all the time. During all the comparisons, NMF performs worse than SNMF is because when it learned the 2 dimensional new representation, it maybe not exploit the distinguished dimensional information sufficiently because of its unsupervised property, but for SNMF, it makes full use of that distinguished dimensional information due to the classification loss item in the Eq. (3).

B. ICU Mortality Risk Prediction

1). Data Processing:

The data we adopted to predict ICU mortality risk is processed as in Paper [1]. To make the data processing clearly, a schematic of the data processing workflow is illustrated in Figure 1. The time series data from the second half of the first day after patients’ admissions to ICU are the original data we need to process. To address the issues of missing data and irregular sampling interval, we performed linear interpolation and resampling at regularly spaced time intervals on the original time series to discretize in the time axis. Although a more complex imputation algorithm may be more plausible [40], [41], we follow the SANMF approach to enable fair comparison. Note that the time interval 2 hours are determined by 5-fold cross validation over choices of 1, 2, 4, or 6 hour intervals. For detailed processing information, the authors can refer to Paper [1].

Fig. 1.

Schematic of the time series data processing.

2). Experiment Setting and Result:

The processed subgraph count data are split equally, stratified by mortality, into the training set and test set. There are 3932 cases in the training set while 3931 cases in the test set. The statistics of the data are demonstrated in Table II. For our proposed SNMF, we need to learn the new representation U from the training data and then train the logistic regression on top of U to check its performance on the U obtained on test data. Besides parameter r (the number of groups), our SNMF still have parameters α, β, γ, we will use 5-fold cross validation to identify the parameters α, β, γ. The range of parameters α, β, γ are from −2 to 2 with exponential base 10. To tune these parameters efficiently, the parameters α, β, γ are set to α/p, β/r and γ/(n × r), which is the sampe with that in simulation data. For the number of groups, we will follow the way in [1], and extend its range if the performance will still rise up to 120. From the subsquent Figure 2, we can find that the range for number of groups are from 50 to 150 (We did not show the performance in range 10 to 40 because their performance is worse than the current range). The metric we adopt is the area under ROC (receiver operating characteristic) curve (AUC). All the experimental results are demonstrated in Figure 2.

TABLE II.

Statistics of Experimental Data

Mortality	Total Cases	Training Cases	Test Cases
≤ 30 days	788	383(9.7%)	405(10.3 %)
> 30 days or alive	7075	3549(90.3%)	3526(89.7 %)

Fig. 2.

Average AUC comparison under different number of groups on the hold-out test data.

From Figure 2, we can see that almost with each group number, our proposed SNMF outperforms NMF, this should because SNMF take the label information into consideration when learning the low-dimensional representation while NMF doesn’t. In some case NMF can perform a bit better than SNMF on the hold-out test data, because there are additional 3 parameters to tune for SNMF and the distributions between training and test data may be inconsistent and this may cause the parameters identified during cross validation not perform well on the hold-out test data. Figure 2 shows that the best test AUC 0.8562 for SNMF occurs at 120 groups while that best test AUC 0.8508 for NMF happens at 60 groups, and the superiority of SNMF over NMF on ICU data is statistically significantly (p-value of permutation test is 0.0307).

V. Discussion

From the classification results on the ICU data, our SNMF indeed performs better than the unsupervised NMF counterpart from the statistic significancy perspective. However, the benifit is not big, this maybe because the information NMF retained has already possessed discriminative classification prediction ability.

As for the ICU data, its original data form is time series, the processing workflow proposed in paper [1], [42] indeed opened a feasible way to deal with the prediction problem easily, meanwhile, some discriminative information may be lost during the process, such as the interpolation and sampling phases may be not precise. A direct way to deal with time series data and conduct classification is interesting and promising. In addition, feature selection [43], [44] is closely related to the present work, it can be further compared with the proposed method while working on time series data directly in future work.

There are several different data types of ICU data, like vital sign and lab test information, they can be considered as multiview information, and then multi-view learning methods [45], [46], [47], [48] can be considered to be adopted to solve this problem.

VI. Conclusions

In this paper, we have proposed a supervised NMF method to learn a discriminative representation, based on which the classification performance is improved compared with its unsupervised NMF counterpart and other supervised NMF method. We adopted the projected gradient descent algorithm to solve this problem. The results on synthetic data and ICU data verified its superiority. The learned representation with our method has better prediction ability, which can guide the clinician well in reality.

Since we just explored to integrate NMF with the simple classifier logistic regresssion, other advanced classifier (classification loss function) can be used to replace logistic regression. For instrance, as with the fast development of deep learning, it is promising to combine deep neural network with NMF to learn a more discriminative representation to boost the prediction performation of the ICU mortality risk further.

Appendix

For convenience to derive the gradients with respect to each variable, let $L_X=\frac{1}{2}\Vert \mathit{X}-\mathit{U}\mathit{V}{\Vert }_F^2+\frac{1}{2}\gamma \Vert \mathit{U}{\Vert }_F^2$ and $L_{\text{y}}\text{=}\alpha {\sum }_{i=1}^n\text{ln}\left(1+\text{exp}\left(-y_i\left({\sum }_{j=1}^r{w}_j{u}_{ij}+b\right)\right)\right)+\frac{1}{2}\beta ({\sum }_{j=1}^r{w}_j^2+b^2),L=L_X+L_y$ denotes the objective function in Eq (12).

$$\begin{gathered} {\nabla }_{\mathit{U}}{L}_X=\frac{1}{2}{\nabla }_U\left(\text{Tr\hspace{0.17em}}\left((\mathit{X}-\mathit{U}\mathit{V}){(\mathit{X}-\mathit{U}\mathit{V})}^T\right)+\gamma \text{Tr\hspace{0.17em}}\left(\mathit{U}{\mathit{U}}^T\right)\right) \\ =\frac{1}{2}{\nabla }_{\mathit{U}}\left(\text{Tr\hspace{0.17em}}\left(-\mathit{X}{\mathit{V}}^T-\mathit{U}\mathit{V}\mathit{X}+\mathit{U}\mathit{V}{V}^T{\mathit{U}}^T\right)+\gamma \text{Tr\hspace{0.17em}}\left(\mathit{U}{\mathit{U}}^T\right)\right) \\ =\frac{1}{2}\left(-2\mathit{X}{\mathit{V}}^T+2\mathit{U}\mathit{V}{\mathit{V}}^T+2\gamma \mathit{U}\right) \\ =-\mathit{X}{\mathit{V}}^T+\mathit{U}\mathit{V}{\mathit{V}}^T+\gamma \mathit{U} \end{gathered}$$ (14)

$$\begin{gathered} {\nabla }_{\mathit{V}}{L}_X=\frac{1}{2}{\nabla }_V\left(\text{Tr\hspace{0.17em}}\left((\mathit{X}-\mathit{U}\mathit{V}){(\mathit{X}-\mathit{U}\mathit{V})}^T\right)\right)) \\ =\frac{1}{2}{\nabla }_{\mathit{V}}\left(\text{Tr\hspace{0.17em}}\left(-\mathit{X}{\mathit{V}}^T-\mathit{U}\mathit{V}{\mathit{X}}^t+\mathit{U}\mathit{V}{\mathit{V}}^T{\mathit{U}}^T\right)\right) \\ =\frac{1}{2}\left(-2{\mathit{u}}^T\mathit{X}+2{\mathit{U}}^T\mathit{U}\mathit{V}\right) \\ =-{\mathit{U}}^T\mathit{X}+{\mathit{U}}^T\mathit{U}\mathit{V} \end{gathered}$$ (15)

$$\begin{gathered} {\nabla }_{w_j}{L}_y=\alpha \sum _{i=1}^n\frac{\text{exp}\left(-y_i\left({\sum }_{j=1}^r{w}_j{u}_{ij}+b\right)\right)*(-1)*y_i{u}_{ij}}{1+\text{exp}\left(-y_i\left({\sum }_{j=1}^r{w}_j{u}_{ij}+b\right)\right)} \\ +\beta w_j \\ =-\alpha \sum _{i=1}^n\frac{y_i{u}_{ij}}{1+\text{exp}\left(y_i\left({\sum }_{j=1}^r{w}_j{u}_{ij}+b\right)\right)}+\beta w_j \end{gathered}$$ (16)

$${\nabla }_{\mathit{w}}{L}_y=-\alpha {(\mathit{U}\odot \mathit{Y}\oslash \mathit{D})}^T*{\mathit{e}}_{\mathit{n}}+\beta \mathit{w}$$ (17)

The denominator matrix D is obtained by repeating the column vector (1+exp(Uw + be_n)) ⊙ y r times to form the matrix of size n×r. Y is obtained with the same operator to vector y.

$${\nabla }_{u_{ij}}{L}_y=\alpha \frac{\text{exp}\left(-y_i\left({\sum }_{j=1}^r{w}_j{u}_{ij}+b\right)\right)*(-1)*y_i{w}_j}{1+\text{exp}\left(-y_i\left({\sum }_{j=1}^r{w}_j{u}_{ij}+b\right)\right)}$$ (18)

It can be written in matrix form as follows.

$${\nabla }_{\mathit{U}}{L}_y=-\alpha \left(\mathit{y}{\mathit{w}}^T\right)\oslash \mathit{D}$$ (19)

The denominator matrix D is the same with that in Eq. (17).

$$\begin{array}{l}{\nabla }_b{L}_y=\alpha \sum _{i=1}^n\frac{\text{exp}\left(-y_i\left({\sum }_{j=1}^r{w}_j{u}_{ij}+b\right)\right)*(-1)*y_i}{1+\text{exp}\left(-y_i\left({\sum }_{j=1}^r{w}_j{u}_{ij}+b\right)\right)}+\beta b\\ =-\alpha \sum _{i=1}^n\frac{y_i}{1+\text{exp}\left(y_i\left({\sum }_{j=1}^r{w}_j{u}_{ij}+b\right)\right)}+\beta b.\end{array}$$ (20)

It can be written in matrix form as follows.

$${\nabla }_b{L}_y=-\alpha {\mathit{e}}_{\mathit{n}}^T*(y\oslash (1+\text{exp}(\mathit{U}\mathit{w}+b)\odot \mathit{y}))+\beta b.$$ (21)