Algorithm 1.

Optimizing hyperparameters with the decoupled Laplace approximation

Require: inputs x, choices y
Require: initial hyperparameters θ0, subset of hyperparameters to be optimized θOPT
1:	tepeat
2:	Optimize for w given current θ → wMAP, Hessian of log-posterior Hθ, log-evidence E
3:	Determine Gaussian prior $\mathcal{N}\left(0,C_{\theta }\right)$ and Laplace appx. posterior $\mathcal{N}\left({\mathbf{w}}_{\text{MAP}},-H_{\theta }^{-1}\right)$
4:	Calculate Gaussian approximation to likelihood $\mathcal{N}\left({\mathbf{w}}_L,\text{Γ}\right)$ using product identity, where ${\text{Γ}}^{-1}=-\left(H_{\theta }+C_{\theta }^{-1}\right)\text{and}{\mathbf{w}}_L=-\text{Γ}{H}_{\theta }{\mathbf{w}}_{\text{MAP}}$
5:	Optimize E w.r.t. θOPT using closed form update (with sparse operations) ${\mathbf{w}}_{\text{MAP}}=-H_{\theta }^{-1}{\text{Γ}}^{-1}{\mathbf{w}}_L$
6:	Update best θ and corresponding best E
7:	until θ converges
8:	return wMAP and θ with best E