Algorithm 1: HPO EM algorithm for synthetic data
1. Generate randomly hyperparameters value $\eta$, $\lambda$, $\mu$ and the dataset $X=\left\{x_1,\dots ,x_N\right\}$
2. Choose a suitable linear basis function $\psi (x_n)$ to get an $N\times M$ matrix $\Psi$ by using (12).
3. Generate parameters value $\omega$ randomly sampled by (2).
4. Generate N-dimensional vector $\epsilon$ sampled randomly by the Gaussian distribution $N(0,{\lambda }^{-1})$ and then generate $T=(t_1,\dots ,t_N)$ by (4) and (5), respectively.
5. E step. Compute the mean $m$ and covariance $C$ using the current hyperparameter values.
$m={(\Lambda +\lambda {\Psi }^T\Psi )}^{-1}(\Lambda \mu +\lambda {\Psi }^{T}T)$,	(49)
$C={(\Lambda +\lambda {\Psi }^T\Psi )}^{-1}$.	(50)
6. M step. Estimate again the hyperparameters by employing the mean $m$ and covariance $C$ obtained by step 5 and the following update equations
${\eta }_i^{new}={\displaystyle \frac{1}{m_i^2+C_{ii}}}$,	(51)
${\lambda }^{new}={\displaystyle \frac{N}{{‖T-\Psi m‖}^2+Tr({\Psi }^T\Psi C)}}$,	(52)
${\mu }^{new}=m$.	(53)
7. Compute the likelihood function or the log likelihood function given by the following result:
$p(T|X,\eta ,\lambda ,\mu )=\sqrt{{\left({\displaystyle \frac{\lambda }{2\pi }}\right)}^N}{\left|\Lambda \right|}^{{\displaystyle \frac{1}{2}}}{\left|C\right|}^{{\displaystyle \frac{1}{2}}}\exp (-G(m))$
or
$\ln p(T|X,\eta ,\lambda ,\mu )={\displaystyle \frac{N}{2}}\ln \lambda -G(m)+{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^M\ln }{\eta }_i+{\displaystyle \frac{1}{2}}\ln \left|C\right|-{\displaystyle \frac{N}{2}}\ln (2\pi )$
and then determine the convergence of the hyperparameters or the likelihood. If convergence is not satisfied, back to step 5. If the likelihood or the log likelihood converges, then the algorithm’s computational complexity is $O(M)$.