Multidataset Independent Subspace Analysis with Application to Multimodal Fusion

Abstract

Unsupervised latent variable models—blind source separation (BSS) especially—enjoy a strong reputation for their interpretability. But they seldom combine the rich diversity of information available in multiple datasets, even though multidatasets yield insightful joint solutions otherwise unavailable in isolation.

We present a direct, principled approach to multidataset combination that takes advantage of multidimensional subspace structures. In turn, we extend BSS models to capture the underlying modes of shared and unique variability across and within datasets. Our approach leverages joint information from heterogeneous datasets in a flexible and synergistic fashion. We call this method multidataset independent subspace analysis (MISA).

Methodological innovations exploiting the Kotz distribution for subspace modeling, in conjunction with a novel combinatorial optimization for evasion of local minima, enable MISA to produce a robust generalization of independent component analysis (ICA), independent vector analysis (IVA), and independent subspace analysis (ISA) in a single unified model.

We highlight the utility of MISA for multimodal information fusion, including sample-poor regimes (N = 600) and low signal-to-noise ratio, promoting novel applications in both unimodal and multimodal brain imaging data.

I. Introduction

BLIND source separation (BSS) [1], [2] is the recovery of unknown latent source signals from their observed mixtures without knowing the mixing process. It is widely adopted in signal, image, and video processing areas, including chemometrics [3], speech [4], multispectral imaging [5], [6], medical imaging [7], [8], and video processing [9], [10]. The “blind” property (unknown source and mixing) is highly effective, especially in applications lacking a precise model of the measured system(s) and with data confounded by noise of unknown or variable characteristics.

In our recent review [1], we introduced a unified multidataset multidiversity multidimensional framework for subspace modeling. It provided a fresh perspective on BSS, identifying both single-dataset multidimensional (SDM) and multidataset unidimensional (MDU) research as subproblems, and outlining a path to reconcile them. In turn, a new class of multidataset multidimensional (MDM) problems became apparent, emphasizing the potential benefits of general latent subspace correspondence across datasets.

Models designed for MDM problems are extremely flexible. A single joint model not only encodes higher complexity through features of flexible dimensionality (the subspaces y_k) but also accommodates arbitrary links among these features over multiple datasets/modalities (x_m). To illustrate (Fig. 1), we consider a multivariate information functional I(y) that operates simultaneously on the joint probability density function (pdf) of all subspaces p(y_k). It captures the association modes underlying multidatasets while adaptively learning multiple linear transformations W_m (dashed lines). When datasets represent modalities, this directly leverages multimodal joint information and lets it guide the decompositions naturally. Combining different multimodal views of the same system, this generalized approach to multimodal fusion offers broader, unique insights into its underlying properties and behavior.

Fig. 1. Subspace identification from multidatasets with MISA.

We consider the general case of M datasets/modalities (x_m) jointly decomposed, without loss of generality, into C sources y_mi each, via linear transformations W_m. Here, each x_m would be either audio or video streams, indicating fusion via the joint analysis of all datasets. Sources are combined into d_k-dimensional subspaces y_k and all-order statistics is utilized to gauge their associations and pursue subspace independence. Only a single correspondence “axis” is required, e.g., time, meaning there is a video frame for each audio sample in audio/video (a/v) data fusion, although the method is not limited to a/v, fusion, nor temporal synchrony specifically. Subspaces establish links among groups of sources across different datasets/modalities. Therefore, multidataset independent subspace analysis (MISA) blindly recovers hidden linked features of flexible dimensionality from multiple datasets and modalities.

Aiming at generality, we pursue statistical independence among subspaces y_k to achieve joint BSS for MDM. Initial investigation of this approach [11]–[13] indicated the presence of critical issues. These included premature convergence to local minima, rigid hard-coded subspace distribution parameters, and a restricted orthogonal regularization for W_m.

Here, we propose a vastly improved expansion to address these issues. We use combinatorial optimization to search over subspace configurations P (Fig. 2) and escape local minima, all-order statistics (i.e., both second- and higher-order statistics—SOS and HOS, respectively) to model p(y_k) via the more general Kotz distribution [14], and a scale-controlled formulation for numerical stability. We also generalize usage to non-orthogonal W_m sans data reduction. We refer to this robust, performant approach simply as multidataset independent subspace analysis (MISA) (Fig. 1). In the formulation below, p(y) represents the joint pdf of all sources, and p(y_k) the pdf of the k-th subspace.

Fig. 2. General architecture of linear MDM problems.

The lower layer corresponds to one V_m × 1 observation of each input data stream x_m. The middle layer represents the C_m sources. The top layer establishes the K subspaces y_k, which are collections of statistically dependent sources (indicated by same-colored connections), following the compositions laid out in the assignment matrix P. This architecture suggests a natural hierarchy among models [1], [15, Ch. 8] in accordance with the number of datasets comprising x and the occurrence of multidimensional sources within any single dataset. MDU and SDM problems include the simpler SDU case, and the most general MDM problem contains all others as special cases.

Let I(y) be the Kullback-Leibler (KL) divergence, an information functional useful for comparing two pdfs p(y) and q(y), where, here, $q(y)={\prod }_{k=1}^{K}p\left(y_k\right)$ is the desired factor pdf of p(y). Then let h(·) be the joint differential entropy, $h(z)=-\mathbb{E}[\text{ln}p(z)]$, for a random vector z with pdf p(z), $\mathbb{E}[\cdot ]$ being the expected value operator, and let P_k be the subset of P assigning specific sources into subspace k. Consequently,

$$\begin{gathered} I(y)=-h(y)+\sum _{k=1}^{K}h\left(y_k\right) \\ =-h(Wx)+\sum _{k=1}^{K}h\left(P_{k}Wx\right). \end{gathered}$$ (1)

We propose to estimate a collection of linear transformations y = Wx simultaneously from all datasets by solving:

$$\underset{W,P}{\text{min}}I(y),$$ (2)

for any W, subspace assignments P, and data streams x. This convenient formulation, which gives mutual information (MI) when the random vector y is two-dimensional, only attains its lower bound of I(y) = 0 when p(y) = q(y), implying that the identified subspaces are indeed statistically independent. A sketch of the convergence proof for this approach is provided as supplemental material.

With MISA, direct study of the interactions and associations among multiple datasets and modalities becomes feasible, in a truly synergistic way. Consequently, joint sources y_k emerge naturally as a direct result of the shared variability estimated from all-order statistical dependences among datasets. Breaking from the limited, rigid paradigm of MDU models dominating current multimodal research [15, Ch. 8], it allows general subspace associations and even absent features in specific datasets. As a unifying toolkit, MISA can execute many general unconventional BSS tasks as well as classical special cases such as independent component analysis (ICA) [16], independent subspace analysis (ISA) [17], and independent vector analysis (IVA) [18]. Also, it outperforms several algorithms in each of these tasks, successfully achieving generalized subspace identification from multidatasets. This uniform implementation yields user accessibility and intuition thanks to the umbrella formulation and methodologies introduced here.

In the current paper, we demonstrate that MISA (our proposed method) outperforms algorithms such as Infomax [19], [20], Laplace IVA (IVA-L) [18], and Gaussian-Laplace IVA (IVA-GL) [21] in challenging experiments and realistic scenarios satisfying the requisites outlined in [22]. MISA’s remarkable performance and stability in certain extremely noisy cases (signal-to-noise ratio (SNR) of 0.0043dB) highlights the benefit of careful multidataset subspace dependence modeling with all-order statistics. Likewise, MISA with greedy permutations (MISA-GP) clearly outperforms joint blind diagonalization with SOS (JBD-SOS) [23] and EST_ISA [24] even at low SNR levels (SNR of 3dB). This shows the benefit of combinatorial optimization to escape local minima in subspace analyses.

Hybrid data results on representative biomedical imaging features and realistic data dimensionality further support the high estimation quality and flexibility of MISA. These include novel applications in high-temporal-resolution functional magnetic resonance imaging (MRI), and multimodal fusion of heterogeneous neurobiological images and signals. The latter also demonstrates feasibility of data fusion even at low SNR and sample-poor regimes (number of observations N = 600), with examples involving functional, structural, and diffusion MRI, as well as electroencephalography (EEG) data. Subspace analysis in its general MDM form has not yet been conducted in a multimodal fusion setting. To the best of our knowledge, MISA is the only approach which can directly investigate this use-case using all-order statistics. Original code and data are available at https://github.com/rsilva8/MISA, with examples to accompany the descriptions in supplemental material (Sections II-B and II-D therein), and detailed derivation of the gradients.

In the following, Section II states the general MDM problem. Section III puts our contributions in context with related works, followed by our methodology description in Section IV. Finally, Sections V and VI present our results and conclusions, respectively. Frequently used acronyms are listed in Table I.

TABLE I.

Frequently used acronyms.

SDU single dataset unidimensional	SOS second-order statistics
SDM single dataset multidimensional	HOS higher-order statistics
MDU multidataset unidimensional	pdf probability density function
MDM multidataset multidimensional	GP greedy permutations
ICA independent component analysis	PCA principal component analysis
ISA independent subspace analysis	IVA independent vector analysis

II. Background

The MDM problem can be formally stated as follows. Given N observations of M ≥ 1 datasets, identify an unobservable latent source random vector $y={\left[y_1^{\top }\cdots y_M^{\top }\right]}^{\top }$, with $y_m={\left[y_{m1}\cdots y_{m{C}_m}\right]}^{\top }$ (C_m sources per dataset), from an observed random vector $x={\left[x_1^{\top }\cdots x_M^{\top }\right]}^{\top }$, with $x_m={\left[x_1\cdots x_{V_m}\right]}^{\top }$ (V_m-dimensional datasets), generated via a mixture vector function f (y, θ) with unknown parameters θ. The m-th V_m × N data matrix containing N observations of x_m along its columns is denoted X_m, and the $\overline{V}\times N$ matrix concatenating all X_m is denoted simply as X (likewise for Y and Y_m). Both y and f (y, θ) have to be learned blindly, i.e., without knowledge of either of them. For tractability, assume:

1. the number of latent sources C_m, which may differ in each dataset, is known to the experimenter;

2. f (y, θ) = Ay is a linear transformation, with θ = A;

3. A is a $\overline{V}\times \overline{C}$ block diagonal matrix with M blocks, describing a separable layout structure [1] representing x_m = A_my_m, m = 1…M, where $\overline{C}={\sum }_{m=1}^M{C}_m$, $\overline{V}={\sum }_{m=1}^M{V}_m$, each block A_m is V_m × C_m, and V_m is the intrinsic dimensionality of each dataset;

4. some latent sources y_mi ∈ y are statistically related to each other, and this dependence is undirected (non-causal), occurring within and/or across datasets;

5. related sources establish d_k-dimensional subspaces¹ y_k, k = 1…K, with K and the subspace compositions laid out by the experimenter in sparse assignment matrices $P_k\in {\{0,1\}}^{d_k\times \overline{C}}$, such that $P={\left[P_1^{\top }\cdots P_k^{\top }\cdots P_K^{\top }\right]}^{\top }$ is a permutation matrix;

6. subspaces do not relate to each other, i.e., either $p\left(y\right)={\prod }_{k=1}^{K}p\left(y_k\right)$ or the cross-correlations ρ_k,k′ = 0, k ≠ k′.

Under these assumptions, recovering sources y amounts to finding a linear transformation W for the unmixing vector function y = Wx. This occurs when W = A⁻, the pseudoinverse of A, implying W is also block diagonal and satisfies y_m = W_mx_m. The experimenter’s priors on the subspace structure within/between one or more datasets, plus the type of statistics describing within/between subspace relation, determines how P is set and, thus, whether and how the model simplifies to the classical special cases [1]. Our focus will be on MDM models driven by statistical independence among subspaces and dependence within subspaces, namely MISA, in the case of an overdetermined system with V_m ≥ C_m, without implying W is square via the typical principal component analysis (PCA). Table II summarizes our key notations.

TABLE II.

Key notations.

M, m	Number of datasets/modalities, counter
N, n	Number of observations, counter
K, k	Number of subspaces, counter
C, c	Number of sources, counter
Vm	intrinsic dimension of m-th dataset
dk	dimension of k-th subspace
dmk	dimension of k-th subspace in m-th dataset
X, x	Data matrix, and vector
Y, y, y	Source matrix, vector, and element
ym	Sources from m-th dataset
ymc	c-th source in m-th dataset
yk	k-th subspace
A, Am	mixing matrix, and its m-th block
W, Wm	unmixing matrix, and its m-th block
P, Pk	full and k-th subset assignment matrices
${\Sigma }_k^{\mathbf{\text{y}}}$, ${\text{R}}_k^{\mathbf{\text{y}}}$	covariance and correlation matrices of yk
Patterns for Special Cases
SDU:	M = 1, K = C, dk = 1 ∀k
SDM:	M = 1, K < C, 1 ≤ dk < C ∀k
MDU:	M > 1, K = Cm ∀m, dmk = 1 ∀k, m

In multimodal brain imaging research, various types of data can be utilized. MRI scans (e.g., structural, diffusion, functional, etc.) typically consist of 3D images, sometimes with an extra dimension. EEGs record the temporal evolution of scalp electric potentials, typically dozens of electrodes at the same time. After collecting two or more such modalities on the same subject, the information is often summarized to a single 3D image and/or time series for each modality. These summary features are obtained from multiple subjects and jointly analyzed with data fusion. Usually, only in-brain signal is considered from 3D images. Those in-brain voxels (volume pixels) are stacked into a single 1D vector prior to fusion. Other modalities, data preparation, and feature generation approaches exist but will not be discussed in this work.

III. Related Work

A. Applications

MDM problems permeate many fields and yet are largely undeveloped. In multimodal fusion of heterogeneous data [25], [26], robust identification of flexible joint features (y_k) originating from all data modalities (x_m) can yield one-of-a-kind views into a system’s properties. This is a prominent direction in mental health research for biomarker identification and early diagnosis, with potential to convey new strategies for disease severity assessment and translation into personalized treatments [1]. In classification, the association/dependence inherent to multimodal features y_k means that good separability in one dataset promotes features with similar property in other datasets, and vice-versa.

The benefits of model flexibility are also notable in various multiset analyses. In the case of multisubject unimodal data (x_m) [27]–[32], it would better preserve subject specificity. In analyses that combine multi-site datasets (x_m) from different scanners/devices, it could naturally mitigate harmonization issues [33], [34] since site/device-variability would seldom explain multidataset associations. In sensor fusion [5], [25], [35], [36], where noise characteristics can be similar if multiple sensors (x_m) share the same environment, it would allow better detection (and potential removal) of noise. For hyperspectral imaging [37]–[39], hyperspectral features (y_k) of higher complexity could be identified in time-lapse studies. For domain-adaptive image recognition [40]–[44], enhanced common and unique representations (y_k) could be identified across image domains (x_m). For multi-view image and video processing [9], [45], [46], objects with complex temporal patterns could be better characterized using (unimodal) higher-dimensional y_k, not to mention potential fusion with audio features [47]–[50] via multimodal y_k.

B. Methods

Our review of BSS in brain imaging [1] studied the underlying strategies of many methods. It offered a general, broad view of how different methods relate to each other by defining a common hierarchical taxonomy to accurately describe them. The unified framework introduced in that work provided a clear path for general MDM model development, which we adopted here to break from current MDU paradigms [15, Ch. 8]. However, it did not consider any of the issues addressed here, including combinatorial optimization, scale control, and non-orthogonal W_m. These were also missing from our early investigations in [11]–[13]. Besides the vastly expanded methodology—which also introduces the general Kotz distribution for MISA—the current work presents a large number of new experiments and realistic applications.

Notably, the Kotz distribution was first introduced for BSS in [51] but applications were limited to MDU problems (IVA specifically). Consequently, that work cannot be applied to cases where d_mk > 1. In addition, its implementation treated the iteratively updated subspace covariances ${\Sigma }_k^{\mathbf{\text{y}}}$ as constant with respect to W (previously, [30], [52] had hard-coded ${\Sigma }_k^{\mathbf{\text{y}}}=I$). The gradients derived for our MISA implementation do not make that assumption and, thus, yield a different search direction than [51] at each step during optimization, even for the IVA case. Also, the optimization approach in [51] was based on simple line search, which is rather different from the interior-point barrier optimization (with bounds and option for non-linear constraints) we utilize here. We also note the use of our novel scale control formulation for numerical stability.

Another work [53] also explores identification of subspace structures in the general MDM setting. However, it is limited to subspaces with Gaussian distribution and, thus, can only leverage SOS to identify subspaces. In contrast to our approach with the Kotz distribution, the approach in [53] cannot leverage HOS for subspace identification. Moreover, our option for the Kotz distribution implies that it suffices to set the parameters in (4) to ψ_G (Section IV-B) and our model simplifies to the same model in [53], highlighting the generality of MISA. The same argument applies to [23], [54].

Finally, premature convergence to local minima due to the mis-assignment of sources to subspaces is a known challenge for SDM model fitting [55]. However, general MDM problems have drastically more intricate within- and cross-dataset subspace-to-subspace interactions. When subspaces span multiple datasets, a combinatorially higher amount of possible local minima (upwards of ${\prod }_{k=1}^K\left(\underset{d_k}{\overline{C}-{\sum }_{l=0}^{k-1}{d}_l}\right)=6\cdot {10}^{19}$ in Section V-B4) undermines the numerical optimization performance (here, d₀ ≜ 0). While combinatorial issues are common in other research areas [56]–[58], they have been largely neglected in BSS literature because of how simple (and often irrelevant) they are for ICA.

In Sections IV-D and IV-E we propose novel combinatorial optimization algorithms for evasion of local minima in the numerical optimization of (1). To the best of our knowledge, this is the first attempt at disentangling these permutation ambiguities in the general MDM case. In contrast to [59], our approach serves only to move a particular solution out of a local minima so that the numerical optimization may resume. Plus, the structural subspace priors contained in P_k guide our combinatorial procedures without relying on ancillary objective functions to determine residual source dependences.

IV. Methodology

A. Scale Control

An inherent property of independence is invariance to arbitrary scaling of each or any source (i.e., multiplication by a non-zero scalar value), which is why ICA sources have scale ambiguity. This has an important implication on the geometry of the resulting objective function we seek to optimize. First, visualize the elements of W into a $\overline{D}$-dimensional vector $(\overline{D}=\overline{V}\overline{C})$ w = vec(W) as would be done in a typical numerical optimization setting. Due to scale invariance, evaluation of the objective function on either w or aw, where a is a non-zero scalar, yields the same value.

Since the objective function evaluates to the same values along the line² spanned by w, only certain changes in the direction of w incur changes in the objective function. Consequently, it suffices to look for a solution on the surface of the hypersphere associated with a given a, since the landscape of objective function values would be identical across concentric (hyper) shells (Fig. 3 (a)). Moreover, scale invariance induces a “star” shape to the contour lines of the objective function in this scenario (Fig. 3 (b)). Since gradients are orthogonal to contour lines, they also ought to be orthogonal to w and lie on the tangent hyperplane of any given hypersphere (Fig. 3 (c)).

Fig. 3. Geometry of the independence-driven objective function in SDU problems.

(a) Due to the scale invariance property of statistical independence, evaluation of the objective function on either w or aw, where a is a non-zero scalar, yields the same value. Only certain changes in the direction of w incur changes in the objective function. Thus, the solution space of independence-driven SDU problems lies on a hypersphere. (b) Scale invariance induces a “star” shape on the contour lines. (c) Consequently, the gradient of a scale invariant function must lie on the tangent hyperplane of the hypersphere associated with a given w.

The main implication is that stepping in the (negative) direction of the gradient towards a local minimum will likely inflate w and lead the search direction in an outward spiral with respect to w. This can be a problem if the norm of w grows indefinitely and eventually becomes numerically unstable. More importantly, as the norm of w increases toward outer shells, the landscape of the objective function starts to stretch (because its values are kept the same while the surface area of the hypersphere grows). Consequently, the gradient grows shorter regardless of its proximity to any local minimum. The smaller gradient will then lead to shorter step lengths, likely yielding very little improvement at latter stages of the numerical optimization and deterring convergence.

This issue is often disregarded in the literature (incidentally, the Infomax algorithm [19] is free of this issue) and should be addressed prior to evaluation of the efficient relative gradient [2, Ch. 4]. One simple approach to address it is to constrain the norm of w. While direct, implementing this approach can be quite inefficient. Rather, since any scale is equally acceptable (at least in theory), we propose to control the estimated source scales by fixing them in the model. Specifically, this is accomplished by assigning the estimated subspace correlation matrix $R_k^y$ as the model dispersion matrix D_k in the Kotz distribution, effectively making the objective function scale selective rather than scale invariant (Section IV-B). Therefore, whenever the source estimates from the data do not support the model variances associated with this choice of $D_k=R_k^{\mathbf{\text{y}}}$, the mismatch induces changes in W that lead their variances towards the prescribed ones. In summary, the proposed scale selective formulation eliminates scaling issues without the need for a formal constraint.

B. Objective Function

Equation (1) admits some simplifications following a few manipulations. First, we note that h(y) = h(Wx) = h(x) + ln|det(W)|, and h(x) can be discarded since it is constant with respect to W. Second, $\text{ln}|\text{det}(W)|={\sum }_{m=1}^M\text{ln}\left|\text{det}\left(W_m\right)\right|$ since W is block diagonal. Finally, when V_m ≠ C_m, for any m, the determinant of W_m is undefined. In order to circumvent this issue, we propose to substitute the determinant by the product of the singular values of W_m, i.e., ${\prod }_{i=1}^{C_m}{\sigma }_{mi}$, where σ_mi are the diagonal elements of ${\mathbf{\Lambda }}_m={\mathbf{\text{U}}}_m^{\top }{\mathbf{\text{W}}}_m{\mathbf{\text{V}}}_m$ originating from the singular value decomposition $W_m=U_m{\Lambda }_m{V}_m^{\top }$. We note that $|\text{det}W|={\prod }_{i=1}^C|{\sigma }_{mi}|$ when W is non-singular and square. Altogether, we can recast (1) as:

$$\check{I}(y)=-\sum _{m=1}^M{J}_{D_m}-\sum _{k=1}^K\mathbb{E}\left[\text{ln}p\left(y_k\right)\right],$$ (3)

where $J_{D_m}={\sum }_{i=1}^{C_m}\text{ln}\left|{\sigma }_{mi}\right|$, and y_k = P_kWx.

This formulation is still incomplete because p(y_k) is undefined. Here we choose to model each subspace pdf as a multivariate Kotz distribution [14], [60]:

$$p\left(y_k\right)=\frac{{\beta }_k{\lambda }_k^{{\nu }_k}\Gamma \left(\frac{d_k}{2}\right){\left(y_k^{\top }{D}_k^{-1}{y}_k\right)}^{{\eta }_k-1}}{{\pi }^{\frac{d_k}{2}}{\left(\text{det}{D}_k\right)}^{\frac{1}{2}}\Gamma \left({\nu }_k\right)}{e}^{-{\lambda }_k{\left(y_k^{\top }{D}_k^{-1}{y}_k\right)}^{{\beta }_k}}$$ (4)

where d_k is the subspace dimensionality, β_k > 0 controls the shape of the pdf, λ_k > 0 the kurtosis (i.e., the degree of peakedness), and ${\eta }_k>\frac{2-d_k}{2}$ the hole size, while ${\nu }_k\triangleq \frac{2{\eta }_k+d_k-2}{2{\beta }_k}>0$ and ${\alpha }_k\triangleq \frac{\Gamma \left({\nu }_k+{\beta }_k^{-1}\right)}{{\lambda }_k^{{\beta }_k^{-1}}{d}_k\Gamma \left({\nu }_k\right)}$ for brevity. Γ(·) denotes the gamma function. The positive definite dispersion matrix D_k is related to the covariance matrix ${\Sigma }_k^{\mathbf{\text{y}}}$ by $D_k={\alpha }_k^{-1}{\Sigma }_k^y$.

This is a good choice of pdf since it includes the multivariate power exponential family, particularly the classical multivariate Gaussian and multivariate Laplace distributions when the parameter set ψ_k = [β_k, λ_k, η_k] is set to ${\mathit{\psi }}_G=\left[1,\frac{1}{2},1\right]$ and ${\mathit{\psi }}_L=\left[\frac{1}{2},1,1\right]$, respectively.

Minimizing (3) is equivalent to maximizing the (log−) likelihood of y_k. In the following, we estimate ${\Sigma }_k^{\mathbf{\text{y}}}$ from the data. This is appealing because the sample average ${\overline{\sum }}^{\text{x}}$ is readily available and can be conveniently combined with W to produce an approximation of ${\Sigma }_k^{\mathbf{\text{y}}}$ for substitution in D_k. This simple choice permits the reparameterization of ${\Sigma }_k^{\mathbf{\text{y}}}$ as a function of W, specifically ${\overline{\sum }}_k^{\mathbf{\text{y}}}=\frac{1}{N-1}{P}_{k}WX{X}^{\top }{W}^{\top }{P}_k^{\top }$.

Two well-conceived dispersion matrix parameter choices are proposed for the Kotz distribution, one emphasizing invariance to source scales and the other not, resulting in two useful objective functions. Firstly, we let Y_k = P_kWX and use n to index each of the N observations used in the sample mean approximation of the expected value $\mathbb{E}[\cdot ]$ in (3). Secondly, based on the log-likelihood ln p(y_k), we define $J_{C_k}=\text{ln}\text{det}{D}_k$, $J_{F_k}=\text{ln}\left(y_k^{\top }{D}_k^{-1}{y}_k\right)$, and $J_{E_k}={\left(y_k^{\top }{D}_k^{-1}{y}_k\right)}^{{\beta }_k}$. Then, we let $D_k={\alpha }_k^{-1}{\overline{\sum }}_k^{\mathbf{\text{y}}}$ for the standard scale invariant case:

$$\begin{gathered} \check{I}(y)=-\sum _{m=1}^M{J}_{D_m}+\frac{1}{2}\sum _{k=1}^K{J}_{C_k}-f\left(K,{\beta }_k,{\lambda }_k,{\eta }_k,d_k,{\nu }_k\right) \\ -\sum _{k=1}^K\frac{{\eta }_k-1}{N}\sum _{n=1}^N{J}_{F_{kn}}+\sum _{k=1}^K\frac{{\lambda }_k}{N}\sum _{n=1}^N{J}_{E_{kn}}, \end{gathered}$$ (5)

where

$$\begin{gathered} f\left(K,{\beta }_k,{\lambda }_k,{\eta }_k,d_k,{\nu }_k\right)=\sum _{k=1}^K[\text{ln}{\beta }_k+{\nu }_k\text{ln}{\lambda }_k+\text{ln}\Gamma \left(\frac{d_k}{2}\right) \\ -\frac{d_k}{2}\text{ln}\pi -\text{ln}\Gamma \left({\nu }_k\right)], \end{gathered}$$

with gradient given by:

$$\begin{gathered} \nabla \check{I}{\left(W\right)}_{m{i}_k}=\left[B_k+\left[I-B_k{Y}_k^{\top }\right]A_k\right]X_m^{\top }-{\left(W_m^{-}\right)}^{\top } \\ \forall k\in \{1,\dots ,K\},\forall m\in \{1,\dots ,M\} \end{gathered}$$ (6)

where i_k represents all source indices (rows of ∇Ĭ(W)_m) assigned to subspace k, ◦ is the Hadamard product, and

$$\begin{gathered} A_k={\left[{\overline{\sum }}_k^{\mathbf{\text{y}}}\right]}^{-1}{\mathbf{\text{Y}}}_k \\ {B}_k=A_k\text{diag}\left(t_k\right) \\ {t}_k=\left(\frac{2{\beta }_k{\lambda }_k}{N}{z}_k^{{\beta }_k}+\frac{2\left(1-{\eta }_k\right)}{N}\right)\circ z_k^{-1} \\ {z}_k=\left[z_{k1},z_{kn},\dots ,z_{kN}\right] \\ {z}_{kn}=y_{kn}^{\top }{\left[{\alpha }_k^{-1}{\Sigma }_k^y\right]}^{-1}{y}_{kn}. \end{gathered}$$

For the scale-controlled approach, we let $D_k=R_k^y$, and the correlation matrix $R_k^y\triangleq {\gamma }_k^{\top }{\Sigma }_k^y{\gamma }_k$, and ${\gamma }_k\triangleq {\left(I_{d_k}\circ {\Sigma }_k^y\right)}^{-\frac{1}{2}}$. In this case, only correlations are estimated from the data, while variances are fixed at α_k. The advantage of this choice is that it controls the scale of the sources rather than letting them be arbitrarily large/small.

In the scale-controlled case, Ĭ(y) is identical to (5), except $J_{C_k}=\text{ln}\text{det}\left({\gamma }_k{\Sigma }_k^{\mathbf{\text{y}}}{\gamma }_k^{\top }\right)$, $J_{F_{kn}}=\text{ln}\left(y_{kn}^{\top }{\left[{\gamma }_k{\Sigma }_k^y{\gamma }_k^{\top }\right]}^{-1}{y}_{kn}\right)$ and $J_{E_{kn}}={\left(y_{kn}^{\top }{\left[{\gamma }_k{\Sigma }_k^y{\gamma }_k^{\top }\right]}^{-1}{y}_{kn}\right)}^{{\beta }_k}$, with gradient:

$$\begin{gathered} \nabla \check{I}{\left(W\right)}_{m{i}_k}=[{\overline{\gamma }}_k^{-1}{B}_k+[{\overline{\gamma }}_k{G}_k-B_k{A}_k^{\top } \\ +\left[Z_{\Sigma }^{-1}-{\overline{\gamma }}_k^2\right]]Y_k]X_m^{\top }-{\left(W_m^{-}\right)}^{\top } \\ \forall k\in \{1,\dots ,K\},\forall m\in \{1,\dots ,M\} \end{gathered}$$ (7)

where

$$\begin{gathered} {\overline{\gamma }}_k={\left(I\circ Z_{\Sigma }\right)}^{-\frac{1}{2}} \\ {\mathbf{\text{Z}}}_{\Sigma }={\mathbf{\text{P}}}_k{\mathbf{\text{WXX}}}^{\top }{\mathbf{\text{W}}}^{\top }{\mathbf{\text{P}}}_k^{\top } \\ {G}_k=I\circ \left(B_k{Y}_k^{\top }\right) \\ {B}_k=A_k\text{diag}\left(t_k\right) \\ {A}_k={{\mathbf{\text{Z}}}_{\Sigma }}^{-1}{\overline{\gamma }}_k^{-1}{\mathbf{\text{Y}}}_k \\ {t}_k=\left(\frac{2{\beta }_k{\lambda }_k}{N}{z}_k^{{\beta }_k}+\frac{2\left(1-{\eta }_k\right)}{N}\right)\circ z_k^{-1} \\ {z}_k=\left[z_{k1},\dots ,z_{kn},\dots ,z_{kN}\right] \\ {z}_{kn}=y_{kn}^{\top }{\left[{\gamma }_k{\overline{\sum }}_k^y{\gamma }_k^{\top }\right]}^{-1}{y}_{kn}. \end{gathered}$$

While the equations presented above are general and support any choice of subspace specific parameters ψ_k, in the examples presented here, we opted to use the same set ψ_k = ψ_L for all subspaces, modeling subspaces as multivariate Laplace distributions with correlation estimation. The derivation of the gradients can be found in supplemental material along with a description of the relative gradient update ∇Ĭ(W)W^⊤W [2, Ch. 4] [61] we used together with the L-BFGS algorithm with bounds (L-BFGS-B) [62], [63] available in the nonlinear constraint optimization function fmincon of MATLAB’s Optimization Toolbox. Nonlinear constraints such as those shown next can be easily incorporated in fmincon’s interior-point barrier method [64, Ch. 19] [65].

C. Pseudoinverse Reconstruction Error

In the overdetermined case, i.e., when V_m > C_m and W is wide, it is necessary to constrain W in order to evade ill-conditioned solutions. The error incurred by W in reconstructing the data samples can indirectly guide and constrain W. The mean squared error (MSE) between x and $\widehat{\text{x}}$ gives the following formulation of the reconstruction error (RE):

$$E=\mathbb{E}\left[\parallel \widehat{x}-\text{x}{\parallel }_2^2\right]\approx \frac{1}{N}\sum _{n=1}^{n=N}{\Vert {\widehat{x}}_n-{\text{x}}_n\Vert }_2^2.$$ (8)

Firstly, the optimal linear estimator of x based on y for a system with estimation error e’, such as y = Wx + e’, is Ây, where  is the minimizer of MSE:

$$\widehat{A}={\Sigma }^x{W}^{\top }{\left(W{\Sigma }^x{W}^{\top }+{\Sigma }^{e’}\right)}^{-1},$$ (9)

and Σ^x is the data covariance. In the high SNR regime, diag (WΣ^xW^⊤) ≫ diag (Σ^e’) element-wise and, as discussed in [66], yields

$$\widehat{A}={\Sigma }^x{W}^{\top }{\left(W{\Sigma }^x{W}^{\top }\right)}^{-1}={\Sigma }^x{W}^{\top }{\Sigma }^{y-1}.$$ (10)

This choice of  always minimizes the error no matter how far W is from the true W_⋆ and serves little as a constraint.

Assuming unit source variances and data whitened such that ${\Sigma }^{\text{x}}=\mathbb{E}\left[{\text{xx}}^{\top }\right]=\text{I}$, in ICA problems W_⋆ must be row orthonormal, i.e., $W_{\star }{W}_{\star }^{\top }=I$. Our previous work [12] utilized  = W^⊤ to reconstruct x as $\widehat{x}=W^{\top }Wx$ instead. Under the whitening assumption, this can be implemented in (8) as a soft regularizer provably equivalent to regularization by either the Frobenius norm ${\Vert W^{\top }W-I\Vert }_{\text{F}}^2$ or ${\Vert W{W}^{\top }-I\Vert }_{\text{F}}^2$, when the regularizer constant approaches infinity [67]. Therefore, this approach effectively penalizes non-orthogonal W.

Here, our investigation of the singular value decomposition (SVD) of W reveals that, if the matrix has orthonormal rows, then its singular values are all 1 and W = USV^⊤ = UV^⊤, where S = I, U are the left singular vectors of W, and V its right singular vectors. Therefore, W^⊤W = VU^⊤UV^⊤ = VV^⊤. Since W is wide, V is tall, which implies VV^⊤ ≠ I, in general. Thus, using ${\widehat{x}}_n=W^{\top }W{x}_n$, the RE simplifies as:

$$E_{\top }\approx \frac{1}{N}\sum _{n=1}^{n=N}{\Vert \left(V{V}^{\top }-I\right)x_n\Vert }_2^2.$$ (11)

This clearly shows that RE with  = W^⊤ implicitly acts as a constraint on the right singular vectors of W, selecting those whose outer product approximates the identity matrix I.

If not orthonormal, W^⊤W = VS²V^⊤ since S ≠ I. Thus, we propose to use the pseudoinverse W⁻ = W^⊤(WW^⊤)⁻¹ in lieu of W^⊤, with ${\widehat{x}}_n=W^{-}W{x}_n$. Then, this pseudoinverse RE (PRE) (E₋) also simplifies as (11). This result follows from the SVD of the pseudoinverse W⁻ = VS⁻¹U^⊤ and W⁻W = VS⁻¹U^⊤USV^⊤ = VV^⊤. Unlike before, this formulation effectively constrains V in the general case. Note that since Σ^x = I in the case of white data, the optimal estimator (10) simplifies to  = W^⊤ (WW^⊤)⁻¹ = W⁻, i.e., the pseudoinverse gives the least error when the data is white (if the SNR is high), regardless of the values contained in W. Thus, for white data, we conclude that the RE formulation (E_⊤) is more appropriate than PRE (E₋). Our experience, however, suggests that W is far more likely non-orthogonal in real noisy, non-white data, justifying our preference for E₋.

Furthermore, we introduce a normalization term, dividing E₋ by x_norm, the average power in the data, and we get the proportion of power missed:

$$E\approx \frac{1}{{\text{x}}_{\text{norm}}}\frac{1}{N}\sum _{n=1}^{n=N}{\Vert W^{\top }{\left(W{W}^{\top }\right)}^{-1}W{x}_n-x_n\Vert }_2^2,$$ (12)

where $x_{\text{norm}}\approx \frac{1}{N}{\sum }_{n=1}^{n=N}{\Vert x_n\Vert }_2^2$. Its gradient has the form:

$$\nabla E(W)=C-C{W}^{-}W$$ (13)

where

$$\begin{gathered} C=\frac{2}{{\text{x}}_{\text{norm}}N}{\left[W^{-}\right]}^{\top }B \\ B=X{Z}^{\top }+Z{X}^{\top } \\ Z=W^{-}WX-X. \end{gathered}$$

Since X and W are block-diagonal, these operations can be computed separately on each dataset by replacing X with X_m and W with W_m. This can be used both as a data reduction approach or a nonlinear constraint for optimization.

Finally, in MDU problems, when there is prior knowledge supporting linear dependence (i.e., correlation) within subspaces, then one useful and popular approach is to use group PCA projection to initialize all blocks of W [68]. It works by performing a single data reduction step on datasets concatenated along the V dimension. We have investigated this approach in a separate work [69], offering efficient algorithms to enable this procedure when the number of datasets is very large (M > 10000). For comparison purposes, we also considered the use of group PCA (gPCA) as an alternate initialization approach for W in our experiments.

D. MISA with Greedy Permutations (SDM Case)

We present a greedy optimization approach to counter local minima resulting from arbitrary source permutations. To illustrate, consider a single dataset and assume P_k is a user-specified prior. Using abbreviated notation throughout, suppose P₁ = [1 1 1 0 0] and P₂ = [0 0 0 1 1] define a partitioning of five sources into two subspaces: p(y) = p(y_k=1)p(y_k=2) = p(y₁, y₂, y₃)p(y₄, y₅), where p(·) is a joint pdf. It would be equally acceptable if the data supported either p(y) = p(y₄, y₅)p(y₁, y₂, y₃) (entire subspace permutation) or p(y) = p(y₁, y₃, y₂)p(y₅, y₄) (within-subspace permutation) or even some combination of these two cases. However, if the data supported p(y) = p(y₁, y₄)p(y₂, y₃, y₅), then that would not be equivalently acceptable, denoting a local minimum.

When these occur, the numerical optimization in Section IV-B stops early, at the newly found local minimum. At that point, we propose to check whether another permutation of sources would attain a lower objective value. This entails two challenges: 1) given the combinatorial nature of the task, even mild numbers of sources lead to huge numbers of candidate permutations, and 2) when the optimization stops early, most sources are still mixed and there is not enough refinement to establish which sources are dependent and belong in the same subspace. The low refinement precludes the combinatorial problem since it hinders the ability to distinguish between dependent and independent sources in the first place.

Firstly, therefore, we propose to transform the single-dataset multidimensional (SDM) ISA task into single-dataset unidimensional (SDU) ICA. We do that by temporarily voiding and replacing subspaces of size d_k ≥ 2 by multiple sources (each with d_k = 1), and then restarting the numerical optimization from the current W estimate (local minimum). This pushes all sources towards being independent from each other. However, dependent sources will only be as independent as possible and will retain some of their dependence. Partly motivated by [59], this approach secures enough refinement to distinguish among subspaces. Thus, given sources that are as independent as possible, we propose a greedy search for any residual dependence among them. The greedy solution is valid because the specific ordering within subspaces is irrelevant. Unlike [59], our approach does not require accessory objective functions to detect dependent sources. Instead, it uses the same scale invariant objective defined in (5).

The procedure is 1) switch to the ICA model (effectively, make P = I), 2) numerically optimize it, 3) reassign sources into subspaces one at a time. In the latter, as indicated in Algorithm 1 (GP), each source is assigned sequentially to each subspace (if two or more are assigned to the same subspace, they are reassigned together thereafter). Thus, the model changes with every assignment, and simple evaluation of the objective cost(·) (without numerical optimization) produces a value for each particular assignment. The scale invariant formulation ensures source variances do not influence the estimation. The assignment minimizing the objective function determines to which subspace a source belongs. Here, assume that k = K + 1 inserts one more row in P for a new subspace; [:, p] are the contents of columns indexed by p (conversely for rows); find(·) recovers the indexes of all non-zero elements; remove_empty_rows(P) removes rows from P containing only zero entries; eps is the machine’s precision.

After repeating this procedure for all sources, in an attempt to solve the original model, we order the identified subspaces so as to match the original prescribed subspace structure P as closely as possible. This final sorting (match(·)) defines a specific permutation of the sources, which we then use to reorder the rows of the local minimum solution W for the original ISA problem, effectively moving that solution out of the local minimum. After that, we resume the numerical optimization of the original ISA problem until another minimum is found. In our experiments, repeating this procedure just twice in a row (T = 2) and taking the best out of three solutions sufficed to drastically improve results. In Algorithm 2 (MISA-GP_SDM), MISA(·) represents the numerical optimization (Section IV-B).

A direct benefit of this approach is that more dependence tends to be retained within subspaces as compared to [59]. That is a desirable property because it leaves room for further post-processing and investigation. Another advantage of our approach is that it can match source assignments to user-prescribed subspace priors (P) when they are available.

E. MISA with Greedy Permutations (MDM Case)

The previous approach addresses cross-subspace interference issues due to incorrect allocation of the sources and, therefore is appropriate for SDM problems. However, it is not sufficient to perform such procedure in MDM problems since ambiguities may also occur at the subspace level, i.e., incorrect allocation of the dataset-specific subspaces.

Consider the following example for a model with three subspaces spanning two datasets, each dataset containing five sources. Assume the correct assignment of sources is as follows: p₁(y₁₁, y₂₁, y₂₂)p₂(y₁₂, y₁₃, y₂₃)p₃(y₁₄, y₁₅, y₂₄, y₂₅), where the notation y_mi refers to source i from dataset m, and p_k(·) is the joint pdf of subspace k. Since MISA-GP_SDM is designed for single datasets, at best, it produces p₁(y₁₁)p₂(y₁₂, y₁₃)p₃(y₁₄, y₁₅) for m = 1 and p₁(y₂₁, y₂₂)p₂(y₂₃)p₃(y₂₄, y₂₅) for m = 2. Then, from a global perspective, these solutions would yield the correct subspace assignment above, thus solving the MDM problem. However, it is equally acceptable for SDM solvers to produce either p₁(y₁₁)p₂(y₁₄, y₁₅)p₃(y₁₂, y₁₃) for m = 1 or p₁(y₂₄, y₂₅)p₂(y₂₃)p₃(y₂₁, y₂₂) for m = 2 if the datasets are evaluated separately (notice the bold subscripts). Together they imply p₁(y₁₁, y₂₄, y₂₅)p₂(y₁₄, y₁₅, y₂₃)p₃(y₁₂, y₁₃, y₂₁, y₂₂), which does not match the correct assignment and, thus, fails to produce a solution for the MDM problem. What we have illustrated here is that within-dataset permutations of equal-sized subspaces may induce mismatches across datasets if the datasets are processed separately. Another complicating factor are subspaces absent from a particular dataset.

Borrowing from the ideas in Section (IV-D), we propose three approaches to address these issues. The first, extends the greedy search to all datasets by sequentially assigning each source (in every dataset) to every subspace and accepting the assignments that reduce the objective function. This would yield a complexity of at least $O(\overline{C}K)$, and $O\left({\overline{C}}^2\right)$ in the (unlikely) worst case of $K=\overline{C}$. The second, processes each dataset separately (as in the previous example) and then applies the same greedy strategy at the level of subspaces instead. Effectively, this approach cycles through each subspace sequentially, trying to determine which of them can be combined to form a larger subspace. This yields a complexity of O(C_mKM) + O(K²M). The final approach is to test all possible permutations of subspaces with the same size, after processing each dataset separately, which yields O((K!)^M). While this can quickly become computationally prohibitive, it can also identify better solutions since it evaluates all subspace permutations of interest. In this work, we elected to use the third approach when the number of sources is small and the second when that number becomes larger (subspace_perm(·)). Full procedures are indicated in Algorithm 3 (MISA-GP).

V. Results

We present results on multiple experiments satisfying the requisites outlined in [22], including a summary of various controlled simulations on carefully crafted synthetic data, as well as hybrid data and comparisons with several algorithms.

A. General Simulation Setup and Evaluation

In the following, we consider the problem of identifying statistically independent subspaces. Thus, in all experiments, each subspace y_k is a random sample with N observations from Laplace distribution. Subspace observations are linearly mixed via a random A as x = Ay + e, where e is additive sensor white noise. A is generated from a standard Gaussian distribution. Its singular values are then adjusted to yield the condition number cond (A) prescribed in Table III. Also, the white Gaussian noise e (zero mean and unit variance) is multiplied by a scalar value in order to attain the SNR prescribed in Table III. The SNR is the power ratio between the SNRdB noisy signal x and the noise e. The equality $\text{SNR}={10}^{\frac{\text{SNRdB}}{10}}$ permits decibel (dB) specifications.

TABLE III.

Summary of simulation results. (a, b) Median (over 10 dataset instances) of best MISI (over 10 initializations per dataset). (c, d) Median MISI (over 10 initializations, 1 dataset instance).

SNRdB		30	10	3	0.46	0.0043
ICA1	PRE+Infomax	0.0222	0.0292	0.0685	0.0928	0.2576
	PRE+MISA	0.0145	0.0165	0.0261	0.1932	0.2743
IVA2	PRE+IVA-L	0.0111	0.0136	0.0197	0.0277	0.5158
	PRE+MISA	0.0059	0.0088	0.0113	0.0151	0.0338
	gPCA+IVA-L	0.0081	0.0090	0.0095	0.0094	0.1271
	gPCA+MISA	0.0044	0.0045	0.0049	0.0065	0.0205
ISA3	PRE+JBD-SOS	0.2700	0.2804	0.2996	0.3255	0.3712
	PRE+MISA	0.1153	0.1275	0.1320	0.1495	0.3404
	PRE+MISA-GP	0.0366	0.0670	0.0794	0.1140	0.3404
(a) Varying SNRdB, Fixed cond(A) = 7

cond(A)		1	3	7	15
ICA1	PRE+Infomax	0.0804	0.0188	0.0216	0.0493
	PRE+MISA	0.1934	0.0148	0.0161	0.0267
IVA2	PRE+IVA-L	0.1923	0.1013	0.0749	0.0505
	PRE+MISA	0.0052	0.0045	0.0049	0.0078
	gPCA+IVA-L	0.0090	0.0086	0.0092	0.0095
	gPCA+MISA	0.0052	0.0045	0.0049	0.0067
ISA3	PRE+JBD-SOS	0.2905	0.2792	0.2815	0.2962
	PRE+MISA	0.1008	0.1065	0.1202	0.1351
	PRE+MISA-GP	0.0395	0.0330	0.0612	0.0743
(b) Fixed SNRdB = 3, Varying cond(A)

ρk,max	0	0.1	0.23	0.39	0.5	0.65
IVA-GL	0.4767	0.0361	0.0114	0.0199	0.0184	0.0186
MISA	0.0273	0.0098	0.0072	0.0062	0.0061	0.0049
(c) IVA1: Increasing max. subspace correlation ρk,max

	ISA1 (ρk = 0)		ISA2 (ρk > 0.2)
	dk = k	dk = 4	dk = k	dk = 4
EST_ISA	–	.0.7557	–	0.7766
JBD-SOS	0.2600	0.3496	0.2826	0.3739
MISA	0.0239	0.0162	0.0369	0.0326
(d) Varying vs Fixed subspace dimensionality dk

The quality of results is evaluated using the normalized multidataset Moreau-Amari intersymbol interference (MISI) (14), which extends the ISI [70], [71] to multiple datasets.

$$\begin{gathered} \text{MISI}(H)=\frac{0.5}{K(K-1)}[\sum _{i=1}^K\left(-1+\sum _{j=1}^K\frac{\left|h_{ij}\right|}{\underset{k}{\text{max}}\left|h_{ik}\right|}\right) \\ +\sum _{j=1}^K\left(-1+\sum _{i=1}^K\frac{\left|h_{ij}\right|}{\underset{k}{\text{max}}\left|h_{kj}\right|}\right)] \end{gathered}$$ (14)

where H is a matrix with elements $h_{ij}=1^{\top }|P_i\widehat{W}A{P}_j^{\top }|1$, with (i, j) = 1…K, i.e., the sum of absolute values from all elements of the interference matrix ŴA corresponding to subspaces i and j, and Ŵ is the solution being evaluated.

For fairness, all algorithms are initialized with the same W₀. See optimization parameters in supplemental material.

B. Summary of Synthetic Data Simulations

The performance of MISA in a series of synthetic data experiments with different properties is summarized below (Table III). Complete details are available as supplemental material online.

1. ICA 1 $(\overline{V}>N)$: effects of additive noise (a) and condition number (b) are assessed in a moderately large ICA problem ($\overline{C}=75$, M = 1) with rectangular mixing matrix A $(\overline{V}=8000)$ at a fairly small sample size regime (N = 3500). Under low SNR (b), MISA outperforms Infomax when cond (A) ≠ 1. At high SNR (a), MISA outperforms Infomax more often than not.

2. IVA 1 (V_m < N, V_m = C_m): MISA performance is assessed in an IVA problem (c), in which subspaces span all of M = 10 datasets. Specifically, we study the case when no data reduction is required (i.e., V_m = C_m = 16), noise is absent, and observations are abundant (N = 32968). The striking feature observed here is that the performance of IVA-GL [21] is much more variable than that from MISA, especially with high correlation within the subspaces. MISA performs well even at low within-subspace correlation levels and is highly stable when these correlations are larger than 0.2.

3. IVA 2 (V_m < N): Effects of additive noise (a) and condition number (b) are assessed in a larger IVA problem (C_m = 75, M = 16) with rectangular mixing matrix A (V_m = 250) and an abundant number of observations N = 32968. Data reduction with either group PCA (gPCA) or pseudoinverse RE (PRE) produced equivalent results in this large N scenario. Under low SNR, increasing the condition number had a fairly small detrimental effect on the performance of both IVA-L [18] and MISA. More importantly, while both IVA-L and MISA performed very well at mild-to-high SNR levels, the performance of MISA on extremely noisy scenarios (SNRdB = 0.0043) is remarkable (0.1 < MISI < 0.01), irrespective of using PRE or gPCA.

4. ISA 1 and 2 ($\overline{V}<N,\overline{V}=\overline{C}$): MISA performance is assessed in ISA problems (d), in which subspaces are multidimensional, with M = 1. Specifically, we study the case when no data reduction is required (i.e., $\overline{V}=\overline{C}=28$), noise is absent, and the number of observations N is abundant. Fixed and varying configurations of K = 7 subspaces are considered, at two subspace correlation ρ_k settings. The striking feature observed here is that the performance of both JBD-SOS [23] and EST_ISA [24] is very poor in all cases, even when within-subspace correlations are present. MISA-GP is the only method with good performance, highlighting the large benefit of our approach for evasion of local minima.

5. ISA 3 $(\overline{V}>N)$: Effects of additive noise (a) and condition number (b) are assessed in a mildly large ISA problem ($\overline{C}=51$, M = 1) with variable subspace dimensionalities d_k, rectangular mixing matrix A $\left(\overline{V}=8000\right)$ at a fairly small sample size regime (N = 5250). Under a challenging SNR, JBD-SOS and MISA fail in virtually all cases (MISI > 0.1). Inclusion of combinatorial optimization enables MISA-GP to perform quite well at mild-to-high SNR levels (SNRdB ≥ 3).

Execution times for Table III (a–b) are reported in Table IV. The timings were recorded on a Linux server (Ubuntu 16.04) with an Intel Xeon E5–2630v4 (10-core, 20-thread, 3.5GHz) CPU, 256GB RAM (DDR4, 2.4GHz). The code was executed in native Matlab without any optimizations.

TABLE IV.

Timing summary. (a, b) Mean (over 10 dataset instances) of median time (over 10 initializations per dataset). Times are reported in seconds.

SNRdB		30	10	3	0.46	0.0043
ICA1	PRE+Infomax	22.6	20.5	4.4	3.3	2.7
	PRE+MISA	743.3	1392.3	2397.5	1543.3	52.0
IVA2	PRE+IVA-L	3128.7	3230.1	3094.7	3794.6	3375.4
	PRE+MISA	11645	11087	9391.6	11323	8134.5
	gPCA+IVA-L	2352.3	2170.1	2080.5	2369.5	3167.1
	gPCA+MISA	4610.2	4432.4	4408.5	5068.8	6354.5
ISA3	PRE+JBD-SOS	2965.3	3007.2	3088.1	3016.6	2934.2
	PRE+MISA	222.8	503.5	729.4	897.6	655.8
	PRE+MISA-GP	2811.4	3479.7	3998.7	4290.0	1429.1
(a) Varying SNRdB, Fixed cond(A) = 7

cond(A)		1	3	7	15
ICA1	PRE+Infomax	4.2	25.1	25.1	7.2
	pre+misa	1582.8	391.9	740.1	1889.0
IVA2	PRE+IVA-L	2120.3	1784.0	1929.4	2376.6
	pre+misa	4112.9	3895.4	4238.5	6205.9
	gPCA+IVA-L	2803.7	2393.1	2318.0	2548.4
	gPCA+MISA	5689.4	4635.0	4544.7	5267.6
ISA3	PRE+JBD-SOS	2999.9	2935.2	2934.6	3021.1
	pre+misa	1145.7	483.6	780.3	1012.5
	pre+misa-gp	3307.3	1354.9	3413.6	3893.3
(b) Fixed SNRdB = 3, Varying cond(A)

The timings are higher in Table IV (a) than in Table IV (b) for PRE-based ICA1 and IVA2 experiments. This is consistent with a corresponding MISI reduction, which was due to a less strict stopping condition for the PRE gradient norm. This suggests that allowing more noise to leak from the PRE step not only yields poorer MISI performance but also significantly slows down convergence (about 3–4 times slower than comparable experiments in Table IV (b)).

In ICA1, Infomax is 1–2 orders of magnitude faster, owing to its inherently different stochastic optimization strategy and gradient implementation, which is optimized for a single dataset. The difference, however, is not due to a difference in algorithmic complexity. Importantly, Infomax is limited and cannot generalize beyond SDU problems like MISA.

In IVA2, MISA takes at least twice as long to converge than IVA-L but attains better results in terms of MISI. Note that the maximum number of iterations in IVA-L was set to four times the total number of iterations until convergence for MISA on the same problem, from the same starting point.

In ISA3, MISA-GP timings are comparable to those of JBD-SOS. However, MISA-GP attains about one order of magnitude better results in terms of MISI.

Overall, the reported timings support that the computational cost of MISA is tractable, especially given it enables universal application to different problems.

C. Hybrid Data Experiments

We present three major results on novel applications of BSS to brain image analysis, open sourcing realistic hybrid data standards (https://github.com/rsilva8/MISA) that test estimation limits at small sample size. The first pushes the conditions of experiment ICA 1 and emulates a single-subject temporal ICA of functional MRI (fMRI). The second investigates the use of IVA with V_m > N for multimodal fusion of brain MRI-derived data. Finally, the last experiment evaluates the value of MDM models without data reduction for fusion of functional MRI (fMRI) and EEG neural signals.

Given the real features from prior publications utilized here, our experiments indeed reflect the usual size of fMRI, sMRI, and EEG datasets in neuroimaging multimodal fusion. Typically, studies combine 2–4 modalities (here, 2–3) with intrinsic dimensionality V_m of 15k-300k voxels, and 600 timepoints. The last example also illustrates how MISA can recover sources even without data reduction of the V_m dimension. Moreover, we illustrate source estimation with 600–1000 subjects, which is 3–10 times bigger than typical multimodal fusion datasets. Furthermore, the typical number sources in multimodal fusion ranges from 4 to 30 (our experiments are 4 to 20). Lastly, to the best of our knowledge, no other work has attempted general subspace estimation (d_mk > 1) in multimodal fusion, which is feasible with MISA, as we demonstrate in our last experiment.

1). Single-Subject Temporal ICA of fMRI:

Here we consider temporal ICA of fast acquisition fMRI. The dimensionality of the data is $\overline{V}=\text{voxels}\approx 60\text{k}$ and N = time points ≈ 1300. In order to better assess the performance of MISA in a realistic scenario, we propose to set the mixing matrix A as the real part of the data. First, we let $\overline{C}=20$ sources. Then, A must be a 60k × 20 matrix. In order to have it correspond to real data, we assign to it the first twenty well-established aggregate spatial maps (3D volumes) published in [72].

For the synthetic part of the data, we propose to simulate a 20 × 1334 matrix of timecourses y by generating realistic autocorrelated samples that mimic observed fMRI timecourses to a good extent. Sampling 20 such timecourses that retain independence with respect to each other is challenging because independently sampled autocorrelated time series tend to be correlated with one another. Building on the simulation principles outlined in [22], we seek to avoid randomly correlated timecourses (sources) in order to prevent mismatches to the underlying ICA model we wish to test. In the same spirit, we also wish to have sources sampled from the same distribution used in the model, here a Laplace distribution. We developed the following steps in order to meet all these requirements:

1. Design a joint autocorrelation matrix R^yy for all sources. For the example above, this means a $\overline{C}N\times \overline{C}N$ block-diagonal correlation matrix $(\overline{C}N=26680)$ with $\overline{C}$ blocks of size N × N. Each block is designed with an exponentially decaying autocorrelation function with an autocorrelation around 0.85 between time point n and n − 1, and around 0.2 between n and n − 10. This structure retains autocorrelation within each N-long section of an observation while retaining uncorrelation/independence among sections.

2. Generate 50k $\overline{C}N$-dimensional observations using a Gaussian copula [73] and the autocorrelation matrix R^yy from step 1. Using copulas enables transformation of the marginal distributions while retaining their correlation/dependence.

3. For each of the 50k copula-sampled observations, transform the sample into a Laplace distribution.

4. For each of the 50k transformed $\overline{C}N$-dimensional observations, reshape them into a $\overline{C}\times N$ matrix and compute the resulting $\overline{C}\times \overline{C}{R}^{\mathbf{y}}$ correlation matrix.

5. Compute the median correlation matrix $R_{med}^{\mathbf{y}}$ over the 50k observed R^y.

6. Retain the transformed observation whose R^y is closest to $R_{med}^{\mathbf{y}}$ and reject the rest.

This type of rejection sampling effectively produces the desired outcome. Finally, Gaussian noise is added to the mixture for a low SNRdB = 3. The condition number of A was 4.59.

In the results, the data was reduced using PRE and then processed with MISA to obtain independent timecourses. The correlation between ground-truth (GT) and PRE+MISA spatial map estimates (RM) is presented in Fig. 4, and the spatial maps (estimating A from Ŵ⁻) in Fig. 5. MISI = 0.0365.

Fig. 4. Correlation with the ground-truth (hybrid temporal ICA).

The correlation between the spatial map estimates from MISA with PRE (RM) and the ground-truth (GT) is very high with little residual similarity across sources, suggesting the analysis was successful.

Fig. 5. Side-by-side comparison with the ground-truth (hybrid temporal ICA).

The clear resemblance to the ground-truth maps suggests a successful recovery of the mixing matrix A. The sample correlation r is shown below each matched pair. Maps are sorted from highest to lowest correlation.

2). Multimodal IVA of sMRI, fMRI, and FA:

In this multimodal fusion of structural MRI (sMRI), fMRI, and Fractional Anisotropy (FA) diffusion MRI data, the dimensionalities are V₁ = voxels ≈ 300k, V₂ = voxels ≈ 67k, V₃ = voxels ≈ 15k, respectively, and N = subjects = 600 (each modality measured on the same subject). We pursue a hybrid setting where only the mixing matrices A_m are taken from real datasets to overcome typically small N in patient population studies. First, we let C_m = 20 sources in each dataset. Then, A₁, A₂, and A₃ must be 300k×20, 67k×20, and 15k×20, respectively. To each, we assign the first twenty aggregate 3D spatial maps published in [74], [72], [75], respectively.

For the simulated part of the data, we generate three 20×600 matrices of subject expression levels y. K = 20 subspaces, each with d_k = 3 and N = 600 observations, were sampled independently from a Gaussian copula, using an inverse exponential autocorrelation function with maximal correlation varying from 0.65 to 0.85 for each subspace. These were transformed to Laplace distribution marginals (not multivariate Laplace) so as to induce a controlled mismatch between the data (only SOS dependence) and the model subspace distributions (multivariate Laplace—all-order dependence). Finally, Gaussian noise was added separately in each dataset for a low SNRdB = 3. The condition numbers of A₁, A₂, and A₃ were 1.52, 4.59, 1.63, respectively.

In the results, the data was reduced using PRE and then processed with MISA to obtain independent subject expression levels. Per-modality correlation between ground-truth and PRE+MISA spatial maps are presented in Fig. 6, and spatial maps (estimating A from Ŵ⁻) in Fig. 7. MISI = 0.0273.

Fig. 6. Correlation with the ground-truth (multimodal IVA).

The correlation between the spatial map estimates from MISA with PRE (RM) and the ground-truth (GT) is very high in all modalities, with little residual similarity across sources, suggesting the analysis was successful.

Fig. 7. Summary of multimodal IVA maps.

In each panel, ground-truth (GT) maps are presented on the left and maps estimated from MISA with PRE (RM) on the right. Each subspace represents the multimodal set of maps (joint features) with highest, median, and minimum correlation with the GT, from top to bottom, respectively. No IVA-L comparison available since it failed to converge, likely due to the small sample size (N = 600) or inability to detect SOS dependence.

3). Multimodal MISA of fMRI, and EEG:

We show the value of MDM models without data reduction for fusion of EEG event-related potentials (ERP) and fMRI datasets, with dimensionality V₁ = time points ≈ 600, V₂ = voxels ≈ 67k, respectively, and N = subjects = 1001. Let C₁ = 4 and C₂ = 6 sources in the ERP and fMRI datasets, respectively, organized into K = 4 subspaces (y_mi represents source i from dataset m):

$$\begin{gathered} k=1:\text{IVA-type, sources}{y}_{11}\text{and}{y}_{21}\left(d_k=2\right); \\ k=2:\text{MISA-type, sources}{y}_{12},y_{22}\text{and}{y}_{23}\left(d_k=3\right); \\ k=3:\text{MISA-type, sources}{y}_{13},y_{14}\text{and}{y}_{24}\left(d_k=3\right); \\ k=4:\text{ISA-type, sources}{y}_{25}\text{and}{y}_{26}\left(d_k=2\right). \end{gathered}$$

Utilizing real spatial maps and timecourses, A₁ and A₂ must be 600 × 4 and 67k × 6, respectively, this time ensuring they form column-orthogonal mixings (with Gram-Schmidt).

For the simulated part of the data, we generate 4×1001 and 6 × 1001 matrices of subject expression levels for ERP and fMRI datasets, respectively. A total of K = 4 d_k-dimensional subspaces with N = 600 observations each were sampled from a multivariate Laplace distribution, using an inverse exponential autocorrelation function with maximal correlation of 0.65 for each subspace. Noise was absent in both datasets. The condition number was 1.00 for both A₁ and A₂.

Fig. 8 shows the results obtained from constrained MISA-GP, i.e., with  = W^⊤ RE constraint using (8). No data reduction was performed on the data. The spatial fMRI maps and ERP timecourses were produced by estimating A from Ŵ^⊤. Since subspace independence is invariant to linear transformations (arbitrary basis) within any subspace [17], the estimation yields timecourses (red) and maps (middle) that do not match the GT exactly. In an attempt to correct for that, we performed additional within-modality ICAs on the columns of A_m corresponding to subspaces. This effectively selected for a particular basis within each subspace (right maps and cyan timecourses). The ability to choose a particular representation demonstrates the kinds of post-processing enabled by MDM models. Overall, this result validates and illustrates the benefit of a constrained optimization approach.

Fig. 8. Multimodal MISA of fMRI and ERP.

GT maps are presented on the left of each panel, MISA-GP estimates in the middle, and corrected MISA-GP estimates on the right. GT ERPs are presented in blue, MISA-GP ERPs in dashed red, and corrected MISA-GP ERPs in dashed cyan.

VI. Conclusion

We have presented MISA, an approach that solves multiple BSS problems (including ICA, IVA, ISA, and more) under the same framework, with remarkable performance and improved robustness even at low SNR. In particular, we have derived a general formulation that controls for source scales, leveraging the flexible Kotz distribution in an interior point non-linear constraint optimization, with PRE as a general and flexible formulation for either direct subspace estimation or dimensionality reduction, in conjunction with combinatorial optimization for evasion of local minima, permitting self-correction to the closest subspace structures supported by the data (MISA-GP). Altogether, the proposed methods permit all-order statistics linkage across multidatasets as well as features of higher complexity to be identified and fully exploited in a direct, principled, and synergistic way, even at sample sizes as low as N = 600.

Flexible approaches like MISA are key to meet the growing complexity of multidataset tasks. These complexities are incorporated in the hybrid dataset standards we open source here, built from relevant results published in the brain imaging BSS literature. Generalizations building on this work could be easily developed exploring other divergence families. Future work will focus on compiling real multimodal datasets to validate MISA’s ability to capture reliable modes of shared and unique variability across and within modalities.

It is also worth noting the natural trade-off that exists between flexibility and complexity. In practice, given some problem specification and prior information, a dedicated algorithm offers the simplest solution. However, the lack of flexibility therein often limits its utility to explore different scenarios. Our work considers a more general case, where one general solution is easily simplified by taking the domain information into account for a given problem. The complexity is unchanged in comparison to a dedicated algorithm. But the general algorithm makes it very easy to switch between models and explore different solutions.

We suggest that further optimization for computational efficiency is certainly possible.

Biographies

Rogers F. Silva (Member, IEEE) received the B.Sc. degree in Electrical Engineering in 2003 from the Catholic University (PUCRS), Porto Alegre, Brazil, the M.S. degree in Computer Engineering (with minors in Statistics and in Mathematics) in 2011, and the Ph.D. degree in Computer Engineering with distinction in 2017, both from The University of New Mexico, Albuquerque, NM, USA. He is currently a Research Scientist at the tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS) with Georgia State University, Georgia Institute of Technology, and Emory University. Previously, he was a Postdoctoral Fellow at The Mind Research Network, a Data Scientist with Datalytic Solutions, and worked as an engineer, lecturer, and consultant. As a multidisciplinary scientist, he develops algorithms for statistical and machine learning, image analysis, numerical optimization, memory efficient large scale data reduction, and distributed analyses, focusing on multimodal, multi-subject neuroimaging data from thousands of subjects. His interests are: multimodal data fusion, statistical and machine learning, image, video and data analysis, multiobjective, combinatorial and constrained optimization, signal processing, and neuroimaging.

Sergey M. Plis received the Ph.D. degree in Computer Science in 2007 from The University of New Mexico, Albuquerque, NM, USA. Currently, he is an associate professor of Computer Science at the Georgia State University, and the Director of Machine Learning core at the tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS). His research interests lie in developing novel and applying existing techniques and approaches to analyzing large scale datasets in multimodal brain imaging and other domains. He develops tools that fall within the fields of machine learning and data science. One of his key goals is to take advantage of the strengths of imaging modalities and infer structure and patterns that are hard to obtain non-invasively and/or that are unavailable for direct observation. In the long term this amounts to developing methods capable of revealing mechanisms used by the brain to form task-specific transient interaction networks and their cognition-inducing interactions via multimodal fusion at features and interaction levels. His ongoing work is focused on inferring multimodal probabilistic and causal descriptions of these function-induced networks based on fusion of fast and slow imaging modalities. This includes feature estimation via deep learning-based pattern recognition and learning causal graphical models.

Tülay Adalı (Fellow, IEEE) received the Ph.D. degree in Electrical Engineering from North Carolina State University, Raleigh, NC, USA, in 1992 and joined the faculty at the University of Maryland Baltimore County (UMBC), Baltimore, MD, the same year. She is currently a Distinguished University Professor in the Department of Computer Science and Electrical Engineering at UMBC.

She has been active in conference and workshop organizations. She was the general or technical co-chair of the IEEE Machine Learning for Signal Processing (MLSP) and Neural Networks for Signal Processing Workshops 2001–2008, and helped organize a number of conferences including the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). She has served or currently serving on numerous editorial boards and technical committees of the IEEE Signal Processing Society. She was the Chair of the MLSP Technical Committee , 2003–2005 and 2011–2013, the Technical Program Co-Chair for ICASSP 2017, Special Sessions Chair for ICASSP 2018 and ICASSP 2024. She is currently serving as the Vice President for Technical Directions of the IEEE Signal Processing Society.

Prof. Adali is a Fellow of the IEEE and the AIMBE, a Fulbright Scholar, and an IEEE Signal Processing Society Distinguished Lecturer. She is the recipient of a 2020 Humboldt Research Award, 2010 IEEE Signal Processing Society Best Paper Award, 2013 University System of Maryland Regents’ Award for Research, and an NSF CAREER Award. Her current research interests are in the areas of statistical signal processing, machine learning, and their applications, with emphasis on medical image analysis and fusion.

Marios S. Pattichis (Senior Member, IEEE) received the B.Sc. degree (Hons.) in computer sciences, the B.A. degree (Hons.) in mathematics, the M.S. degree in electrical engineering, and the Ph.D. degree in computer engineering from The University of Texas at Austin, Austin, in 1991, 1993, and 1998, respectively. He is currently a Professor with the Department of Electrical and Computer Engineering, University of New Mexico (UNM), Albuquerque. His current research interests include digital image and video processing, video communications, dynamically reconfigurable computer architectures, biomedical, space, and educational image processing applications.

At UNM, he holds the Gardner Zemke Professorship in Teaching at ECE and he is a Fellow of the Center for Collaborative Research and Community Engagement with the College of Education and Human Sciences. He was a recipient of the 2016 Lawton-Ellis and the 2004 Distinguished Teaching Awards from the Department of Electrical and Computer Engineering, UNM. For his development of the digital logic design labs at UNM, he was recognized by the Xilinx Corporation, in 2003 and by the UNM School of Engineerings Harrison Faculty Excellent Award, in 2006. He was a Founding Co-PI of the Configurable Space Microsystems Innovations and Applications Center (COSMIAC), UNM, where he is currently the Director of ivPCL. He was the General Chair of the 2008 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI). He was general Co-Chair of the 2020 IEEE SSIAI. He is currently a Senior Associate Editor for the IEEE Transactions on Image Processing. He has also served as Senior Associate Editor for the IEEE Signal Processing Letters. He has been an Associate Editor of the IEEE Transactions on Image Processing, the IEEE Transactions on Industrial Informatics. He has served as a Guest Associate Editor for the IEEE Transactions on Information Technology in Biomedicine.

Vince D. Calhoun (Fellow, IEEE) received the B.S. degree in electrical engineering from the University of Kansas, Lawrence, KS, USA, in 1991, the M.S. degrees in biomedical engineering and information systems from The Johns Hopkins University, Baltimore, MD, USA, in 1993 and 1996, respectively, and the Ph.D. degree in electrical engineering from the University of Maryland Baltimore County, Baltimore, in 2002.

Dr. Calhoun is the founding director of the tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS) and a Georgia Research Alliance eminent scholar in brain health and image analysis where he holds appointments at Georgia State University, Georgia Institute of Technology and Emory University. He was previously the President of the Mind Research Network and Distinguished Professor of Electrical and Computer Engineering at the University of New Mexico. He is the author of more than 800 full journal articles and over 850 technical reports, abstracts and conference proceedings. His work includes the development of flexible methods to analyze functional magnetic resonance imaging data such as independent component analysis (ICA), deep learning for neuroimaging, data fusion of multimodal imaging and genetics data, neuroinformatics tools, and the identification of biomarkers for disease. His research is funded by the NIH and NSF among other funding agencies. Dr. Calhoun is a fellow of the Institute of Electrical and Electronic Engineers, The American Association for the Advancement of Science, The American Institute of Biomedical and Medical Engineers, The American College of Neuropsychopharmacology, and the International Society of Magnetic Resonance in Medicine. He served as the chair for the Organization for Human Brain Mapping from 2018–2019, and is a past chair of the IEEE Machine Learning for Signal Processing Technical Committee. He currently serves on the IEEE BISP Technical Committee and is also a member of the IEEE Data Science Initiative Steering Committee.