Algorithm 1.

JAMIE

	Input: $\mathit{X},\mathit{Y},\mathit{F}$
	Params: ${\tau }_{\mathrm{m}\mathrm{i}\mathrm{n}},{\tau }_{\mathrm{m}\mathrm{a}\mathrm{x}},b_s$, $(\alpha ,\beta ,\gamma )$, $\rho ,{\tau }_{\text{max\_lapses}},{\ell }_{\text{min\_change}}$
	Output: $f_E^X(\cdot )$, $f_E^Y(\cdot )$, $f_D^X(\cdot )$, $f_D^Y(\cdot )$, ${\mathit{L}}^{\mathit{X}},{\mathit{L}}^{\mathit{Y}}$
1:	$\left[{\mathit{K}}_{\mathit{X}},{\mathit{K}}_{\mathit{Y}}\right]=[dist(\mathit{X}),dist(\mathit{Y})]$
2:	$\tilde{\mathit{F}}=\mathrm{a}\mathrm{r}\mathrm{g}{\mathrm{m}\mathrm{i}\mathrm{n}}_{\alpha ,\overline{\mathit{F}}} {\parallel \alpha {\mathit{K}}_{\mathit{X}}-\overline{\mathit{F}}{\mathit{K}}_{\mathit{Y}}{\overline{\mathit{F}}}^T\parallel }_F^2$
3:	$\mathit{F}\leftarrow \rho \mathit{F}+(1-\rho )\tilde{\mathit{F}}$
4:	${\ell }_{\text{best}}\leftarrow \mathrm{\infty },{\tau }_{\text{lapses}}\leftarrow 0$
5:	${\tau }_{\text{current}}\leftarrow 0$
6:	Initialize $f_E^X,f_E^Y,f_D^X,f_D^Y$ with random weights
7:	while ${\tau }_{\text{current}}<{\tau }_{\text{max}}$ and ${\tau }_{\text{lapses}}<{\tau }_{\text{max\_lapses}}$ do
8:	Randomly sample ${\mathit{X}}_s\in {\mathbb{R}}^{d_{\mathit{X}}\times b_s}$ from $\mathit{X}\in {\mathbb{R}}^{d_{\mathit{X}}\times n_{\mathit{X}}}$ and ${\mathit{Y}}_s\in {\mathbb{R}}^{d_{\mathit{Y}}\times b_s}$ from ${\mathit{Y}}_s\in {\mathbb{R}}^{n_{\mathit{Y}}\times n_{\mathit{Y}}}$
9:	Extract submatrix ${\mathit{F}}_s\in {\mathbb{R}}^{b_s\times b_s}$ from $\mathit{F}\in {\mathbb{R}}^{n_{\mathit{X}}\times n_{\mathit{Y}}}$ and ${\tilde{\mathit{F}}}_{\mathit{s}}\in {\mathbb{R}}^{b_s\times b_s}$ from $\tilde{\mathit{F}}\in {\mathbb{R}}^{n_{\mathit{X}}\times n_{\mathit{Y}}}$
10:	Optionally transform ${\mathit{X}}_s,{\mathit{Y}}_s$ using PCA trained on $\mathit{X},\mathit{Y}$
11:	Normalize columns of ${\mathit{F}}_s$ and ${\tilde{\mathit{F}}}_{\mathit{s}}$ to summation 1
12:	Randomly sample $L_i^{{\mathbf{X}}_{\mathrm{s}}}$ from distributions $f_E^X\left({\mathit{X}}_s\right)$ and $L_j^{{\mathbf{Y}}_{\mathrm{s}}}$ from distributions $f_E^Y\left({\mathit{Y}}_s\right)$ for $i,j\in \left\{1,2,\dots ,b_s\right\}$
13:	${\mathit{M}}^{{\mathit{X}}_s}\leftarrow \left({\mathit{L}}^{{\mathit{X}}_s}+\delta {\mathit{L}}^{{\mathit{Y}}_s}{\mathit{F}}_s^T\right){\left(\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left(1+\delta {\mathit{F}}_s{\mathbf{1}}_{b_s}\right)\right)}^{-1}$
14:	${\mathit{M}}^{{\mathit{Y}}_s}\leftarrow \left({\mathit{L}}^{{\mathit{Y}}_s}+{\delta }^{-1}{\mathit{L}}^{{\mathit{X}}_s}{\mathit{F}}_s\right){\left(\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left(1+{\delta }^{-1}{\mathit{F}}_s^T{\mathbf{1}}_{b_s}\right)\right)}^{-1}$
15:	${\tilde{\mathit{X}}}_{\mathit{s}}\leftarrow f_D^X\left({\mathit{M}}^{{\mathit{X}}_s}\right)$ and ${\tilde{\mathit{Y}}}_s\leftarrow f_D^Y\left({\mathit{M}}^{{\mathit{Y}}_s}\right)$
16:	If ${\mathit{X}}_s,{\mathit{Y}}_s$ were transformed using PCA, perform inverse transform on ${\tilde{\mathit{X}}}_{\mathit{s}},{\tilde{\mathit{Y}}}_{\mathit{s}}$
17:	$\kappa \leftarrow {\left(1+\mathrm{e}\mathrm{x}\mathrm{p}\left(\frac{-5\left({\tau }_{\text{current}}-0.5{\tau }_{\mathrm{m}\mathrm{i}\mathrm{n}}\right)}{0.5{\tau }_{\mathrm{m}\mathrm{i}\mathrm{n}}}\right)\right)}^{-1}$
18:	Calculate ${\ell }_{\mathrm{k}\mathrm{l}}$ via Equation 4
19:	Calculate ${\ell }_{\text{combination}}$ via Equation 6
20:	Calculate ${\ell }_{\text{reconstruction}}$ via Equation 7
21:	Calculate ${\ell }_{\text{alignment}}$ via Equation 9
22:	${\ell }_{\text{total}}\leftarrow {\ell }_{\text{kl}}+\alpha {\ell }_{\text{combination}}+\beta {\ell }_{\text{reconstruction}}+\gamma {\ell }_{\text{alignment}}$
23:	if ${\tau }_{\text{current}}\ge {\tau }_{\mathrm{m}\mathrm{i}\mathrm{n}}$ then
24:	if ${\ell }_{\text{total}}<{\ell }_{\text{best}}-{\ell }_{\text{min\_change}}$ then
25:	${\ell }_{\text{best}}\leftarrow {\ell }_{\text{total}}$
26:	${\tau }_{\text{lapses}}\leftarrow 0$
27:	else
28:	${\tau }_{\text{lapses}}\leftarrow {\tau }_{\text{lapses}}+1$
29:	end if
30:	end if
31:	Update $f_E^X,f_E^Y,f_D^X,f_D^Y$ using gradient descent on ${\ell }_{\text{total}}$
32:	${\tau }_{\text{current}}\leftarrow {\tau }_{\text{current}}+1$
33:	end while
34:	${\mathit{L}}^{\mathit{X}}\leftarrow f_E^X(\mathit{X})$, ${\mathit{L}}^{\mathit{Y}}\leftarrow f_E^Y(\mathit{Y})$