Table A.3.

Optimized quantity, minimized loss function, and constraints for RBM, Infomax ICA, PCA, sparse PCA, and sNMF. The quantities are as follows: W, de-mixing matrix (transposed for SMF) or energy interaction terms (RBM); X, a M × N data matrix; x_n, nth data point or the nth column of the data matrix X; s_.n, the nth column of the source matrix S; s_j., the jth row of the source matrix S; a, b, linear additive terms; M, the number of observable dimensions; C, the number of components; $1_{C\times C}$, an C × C identity matrix; w_.j, the jth column of W; λ_j, the L_1sparsity constant on w_.j. For RBM and ICA, losses are derived from logistic nonlinearities for comparison, and not the exact losses used in the results.

	Opt. Quantity	Minimized Loss Function	Constraints
RBM	Free Energy	$\mathcal{F}\left({\mathit{x}}_n\right)=-{\mathbf{a}}^{\mathsf{T}}{\mathit{x}}_n-{\sum }_{j=1}^H\log (1+{\exp }^{{\mathit{x}}_n^{\mathsf{T}}{\mathit{w}}_{\cdot j}+b_j})$
Infomax ICA	Entropy	$-\mathcal{H}\left({\mathit{s}}_{\cdot n}\right)=-\log \left(\det \mid \mathbf{W}\mid \right)+{\sum }_{j=1}^C(2\log (1-{\exp }^{-{\mathit{x}}_n^{\mathsf{T}}{\mathit{w}}_{\cdot j}-b_j})+({\mathit{x}}_n^{\mathsf{T}}{\mathit{w}}_{\cdot j}+b_j))$
PCA	Variance	–var (sj.) = –∥Xw.j∥2	${\mathbf{W}}^{\mathsf{T}}\mathbf{W}=1_{C\times C}$
sPCA	Lasso criterion	$\beta ={\Vert {\mathit{x}}_n-\mathbf{W}{\mathbf{W}}^{\mathsf{T}}{\mathit{x}}_n\Vert }^2+{\sum }_{j=1}^C\left({\lambda }_j{\Vert {\mathit{w}}_{\cdot j}\Vert }_1\right)$	${\mathbf{W}}^{\mathsf{T}}\mathbf{W}=1_{C\times C},{\lambda }_j>0$
sNMF	Squared error	$\mathcal{E}{\left({\mathit{s}}_{\cdot n}\right)}^2-\frac{1}{2}{\Vert {\mathit{x}}_n-{\mathit{s}}_{\cdot n}{\mathbf{W}}^{\mathsf{T}}\Vert }_F^2$	∥w.j∥1 ≤ λj, W ≥ 0, s.n ≥ 0