Algorithm 1.

EM algorithm for Multimodal Dictionary Learning

Input: Training multimodal image patches: {x̃i}; i ∈ 1,…,N; Initialize multimodal dictionary D̃ = D̃0, D̃0 is trained on all of the x̃i;

Output: Refined dictionary $\widehat{\stackrel{\sim }{D}}$

1: (E-step) compute δp (p(h=0| x̃i, θ)), where

$${\delta }_p(p)=\{\begin{array}{cc}1,& \text{if}p\ge 0.5,\\ 0,& \text{otherwise}.\end{array}$$ (5)

$$p(h=0\mid {\stackrel{\sim }{x}}_i,\theta )=\frac{p({\stackrel{\sim }{x}}_i\mid h=0,\theta )p(h=0)}{p({\stackrel{\sim }{x}}_i\mid h=1,\theta )p(h=1)+p({\stackrel{\sim }{x}}_i\mid h=0,\theta )p(h=0)}.$$ (6)

update θ1 and θ0 in (3) based on δp (p(h=0|x̃i, θ)).

2: (M-step) update D̃ and α as follows1,

$$\begin{array}{l}{\stackrel{\sim }{D}}^{(t)}=arg\underset{\stackrel{\sim }{D}}{min}\sum _{i=1}^N{\delta }_p(p(h=0\mid {\stackrel{\sim }{x}}_i,\theta ))(\frac{1}{2}{\Vert {\stackrel{\sim }{x}}_i-\stackrel{\sim }{D}{\alpha }_i\Vert }_2^2+\lambda {\Vert {\alpha }_i\Vert }_1),\\ \mathrm{s}.\mathrm{t}.{\Vert {\stackrel{\sim }{D}}_j\Vert }_2^2\le 1,j=1,2,\dots ,k.\\ {\alpha }_i^{(t)}=arg\underset{{\alpha }_i}{min}{\delta }_p(p(h=0\mid {\stackrel{\sim }{x}}_i,\theta ))(\frac{1}{2}{\Vert {\stackrel{\sim }{x}}_i-{\stackrel{\sim }{D}}^{(t)}{\alpha }_i\Vert }_2^2+\lambda {\Vert {\alpha }_i\Vert }_1).\end{array}$$ (7)

3: Iterate E and M steps until convergence reached.