Algorithm 2. Deep learning model optimization based on Kullback-Leibler risk function and GMM.

1 1em

Data: Observation data set $\mathcal{D}=\{(y_1,x_1),\dots ,(y_N,x_N)\}$, posterior model parameters of Algorithm 1 ${\hat{\mathrm{\Theta }}}_{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{t}}$

Result: Optimized model parameters ${\mathrm{\Theta }}^{\ast }$

2 Initialize deep learning model parameters (including LSTM and GMM parameters);

3 Initialize regularization parameters $\lambda$;

4 Phase 1: Pre-training deep learning model;

5 while Pre-training stopping criterion not reached do

6   Minimize the objective function $\mathcal{L}$ (Formula (29));

7   through stochastic gradient descent (SGD);

8 Phase 2: Model tuning and integration;

9 Set Kullback-Leibler risk function and Gaussian mixture model weights;

10 while Model tuning stopping criterion not reached do

11   Use the Formula (25) to calculate the Kullback-Leibler gradient;

12   MCMC update of GMM parameters using Metropolis-Hastings algorithm;

13   Integrate Algorithm 1 ${\hat{\mathrm{\Theta }}}_{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{t}}$ as prior information;

14   Update $\lambda$ to balance Kullback-Leibler risk and GMM (see Eqs. (28) and (29));

15 Calculate the optimal posterior parameters ${\mathrm{\Theta }}^{\ast }$;

16 return ${\mathrm{\Theta }}^{\ast }$