Bayesian Simultaneous Factorization and Prediction Using Multi-Omic Data

Abstract

Integrative factorization methods for multi-omic data estimate factors explaining biological variation. Factors can be treated as covariates to predict an outcome and the factorization can be used to impute missing values. However, no available methods provide a comprehensive framework for statistical inference and uncertainty quantification for these tasks. A novel framework, Bayesian Simultaneous Factorization (BSF), is proposed to decompose multi-omics variation into joint and individual structures simultaneously within a probabilistic framework. BSF uses conjugate normal priors and the posterior mode of this model can be estimated by solving a structured nuclear norm-penalized objective that also achieves rank selection and motivates the choice of hyperparameters. BSF is then extended to simultaneously predict a continuous or binary phenotype while estimating latent factors, termed Bayesian Simultaneous Factorization and Prediction (BSFP). BSF and BSFP accommodate concurrent imputation, i.e., imputation during the model-fitting process, and full posterior inference for missing data, including “blockwise” missingness. It is shown via simulation that BSFP is competitive in recovering latent variation structure, and demonstrate the importance of accounting for uncertainty in the estimated factorization within the predictive model. The imputation performance of BSF is examined via simulation under missing-at-random and missing-not-at-random assumptions. Finally, BSFP is used to predict lung function based on the bronchoalveolar lavage metabolome and proteome from a study of HIV-associated obstructive lung disease, revealing multi-omic patterns related to lung function decline and a cluster of patients with obstructive lung disease driven by shared metabolomic and proteomic abundance patterns.

1. Introduction

Despite a growing body of statistical methods for multi-omic data analysis, several studies motivate new methodology that can simultaneously (1) identify latent factors that explain variation within or across multi-omic datasets, (2) use the latent factors to predict an outcome, (3) perform missing data imputation, and (4) fully characterize uncertainty in the latent factors, outcome predictions, and imputed data. Our motivation concerns obstructive lung disease (OLD), which remains a frequent comorbidity among individuals living with HIV despite increasing usage of combination antiretroviral therapies (Hirani et al., 2011). We do not have a complete understanding of the molecular factors associated with risk of developing OLD in this population. We aim to use multi-omic data collected from bronchoalveolar lavage (lung) fluid in patients with HIV to characterize patterns of molecular variation within this unique cohort and to relate these patterns to clinical measurements of lung function.

To accomplish these tasks, we consider integrative factorization methods which decompose and partition latent variation across multiple data sources. There are several examples of exploratory integrative factorization methods, both non-Bayesian (Shen et al., 2012; Lock et al., 2013; Yang and Michailidis, 2016; Gaynanova and Li, 2019), and Bayesian (Klami et al., 2013; Chekouo et al., 2017; Argelaguet et al., 2018), which can be used to identify associations between sources in the form of low-rank structured variation. Factors underlying these low-rank structures may reflect biological patterns, though these methods do not explicitly account for a clinical or biological phenotype. A less-explored area is simultaneous factorization and prediction, which can simultaneously estimate factors while also relating them to an outcome. We can incorporate prediction into existing exploratory methods in two steps: (1) identify a small number of latent factors explaining variation in the data, and (2) use these as covariates in a predictive model as in principal components regression (Massy, 1965). This was described in Kaplan and Lock (2017) for multi-omic data, in Samorodnitsky et al. (2022) for multi-omic and multi-cohort data, and in Hellton and Thoresen (2016) for clustering. Recently, one-step simultaneous factorization and prediction procedures have been proposed (Zhang and Gaynanova, 2021; Palzer et al., 2022; Safo et al., 2022), including some Bayesian approaches (Chekouo and Safo, 2023; White et al., 2021). Two-step approaches do not propagate uncertainty associated with the estimated factorization into the predictive model. Additionally, existing one-step approaches do not provide a framework for statistical inference in the estimated factorization and predictive model. Finally, few methods, if any, can accommodate missing values in both the omics sources and outcome and provide full posterior inference for imputed values.

To fill these gaps, we propose Bayesian Simultaneous Factorization (BSF) for exploratory multi-omics integration that estimates a partitioned factorization consisting of joint structure (variation shared across sources) and individual structure (variation specific to each source). BSF can be viewed as an extension of probabilistic matrix factorization (PMF) (Mnih and Salakhutdinov, 2007; Salakhutdinov and Mnih, 2008) with a Gaussian likelihood and conjugate Gaussian priors on the loadings and scores. The posterior mode is the solution to a structured nuclear-norm penalized objective which matches Park and Lock (2020)’s UNIFAC decomposition, a multi-source factorization of joint and individual structures using a nuclear-norm penalty to estimate the structure ranks. Solving this objective achieves rank selection, motivates our choice of prior hyperparameters, and allows efficient initialization of a Gibbs sampling algorithm to estimate the posterior distributions of the factorization parameters. We also propose Bayesian Simultaneous Factorization and Prediction (BSFP), a one-step procedure which extends BSF by estimating factors underlying joint and individual structures while simultaneously relating them to an outcome. BSF and BSFP both offer full posterior inference for the estimated structures, and BSFP properly accounts for uncertainty in the predictive model due to the estimated factorization, offering posterior inference for the latent factors and the predictive model simultaneously. We focus on a continuous outcome, but this is naturally extended to accommodate any Bayesian predictive model and we describe our implementation for a binary outcome in the Supplementary Materials. Finally, BSF and BSFP can be used for multiple imputation, including in “blockwise” missing scenarios in which an entire sample from a source is unavailable, and offer full posterior inference for the imputed values in both the omics sources and, in the case of BSFP, the outcome-of-interest.

The remainder of our article is organized as follows: in Section 2, we review PMF, UNIFAC, and introduce BSF and BSFP. In Section 3, we compare BSFP to existing one- and two-step approaches to factorization and prediction via simulation and assess the imputation accuracy of BSF against other approaches. In Section 4, we describe applying BSFP to predict lung function using proteomic and metabolomic data to study HIV-associated OLD. We conclude with a discussion of the methodology and potential new directions.

2. Methods

2.1. Notation

We first introduce notation used throughout. Bold, uppercase letters, e.g. $\mathbf{\text{X}}$, denote matrices. Bold, lowercase letters, e.g. $\mathbf{\text{y}}$, denote vectors. Unbolded uppercase and lowercase letters, e.g. $R$ and $q$, denote scalars. For illustration, consider $q$ data sources, e.g. omics datasets, measured on $n$ samples. Let ${\mathbf{\text{X}}}_s:p_s\times n$ represent source $s$, containing $p_s$ biomarkers, oriented such that the columns are the samples. ${\mathbf{\text{X}}}_1,\dots ,{\mathbf{\text{X}}}_q$ are linked by their columns, meaning they contain biomarkers measured on a shared set of $n$ samples. We use ${\mathbf{\text{X}}}_s[j,i],j=1,\dots ,p_s$ and $i=1,\dots ,n$ to denote expression of the $j\text{th}$ feature by the $i\text{th}$ sample in source $s$. Similarly, ${\mathbf{\text{X}}}_s[j,\cdot ]$ represents the $j\text{th}$ feature across all samples (the $j\text{th}$ row of ${\mathbf{\text{X}}}_s$) and ${\mathbf{\text{X}}}_s[\cdot ,i]$ represents the expression values for the $i\text{th}$ sample (the $i\text{th}$ column of ${\mathbf{\text{X}}}_s$). We use subscripts and unbolded letters to index elements within a vector, e.g., $y_i$ is the $i\text{th}$ entry in $\mathbf{\text{y}}$. We let ${\mathbf{\text{X}}}_{\cdot }={\left(\begin{array}{c}{\mathbf{\text{X}}}_1^T\dots {\mathbf{\text{X}}}_q^T\end{array}\right)}^T$ denote the column-concatenated, full data matrix containing biomarkers from all $q$ sources such that $R=\text{rank}\left({\mathbf{\text{X}}}_{\cdot }\right)$. Let $p={\sum }_{s=1}^q{p}_s$ represent the total number of observed biomarkers. We define the squared Frobenius norm, ${\Vert \cdot \Vert }_F^2$, of ${\mathbf{\text{X}}}_s$ as ${\sum }_{j=1}^{p_s}{\sum }_{i=1}^n{\mathbf{\text{X}}}_s{[j,i]}^2$, i.e. the sum of squared entries in ${\mathbf{\text{X}}}_s$, and the nuclear norm, ${\Vert \cdot \Vert }_{\ast }$, of ${\mathbf{\text{X}}}_s$ as ${\sum }_{k=1}^{R_s}{\sigma }_k\left({\mathbf{\text{X}}}_s\right)$ where ${\sigma }_k\left({\mathbf{\text{X}}}_s\right)$ denotes the $k\text{th}$ singular value of ${\mathbf{\text{X}}}_s$ and $R_s=\text{rank}\left({\mathbf{\text{X}}}_s\right)$.

2.2. Review of Probabilistic Matrix Factorization (PMF)

Before describing our proposed method in Section 2.4, we review probabilistic matrix factorization (PMF) in this section and UNIFAC in Section 2.3. Consider a single real-valued matrix, $\mathbf{\text{X}}\in {ℝ}^{p\times n}$. PMF is a Bayesian linear factor model, in which observations in $\mathbf{\text{X}}$ are assumed to be driven by a small number, $r<\text{rank}(\mathbf{\text{X}})$, of latent factors or components. These latent factors are contained in a matrix $\mathbf{\text{V}}\in {ℝ}^{n\times r}$, termed scores, which are mapped to the space spanned by the observed features or biomarkers by matrix $\mathbf{\text{U}}\in {ℝ}^{p\times r}$, termed loadings. Assuming $\mathbf{\text{X}}=\mathbf{\text{U}}{\mathbf{\text{V}}}^T+\mathbf{\text{E}}$, then $\mathbf{\text{U}}{\mathbf{\text{V}}}^T:p\times n$ is a low-rank approximation to the observed $\mathbf{\text{X}}$ where $\text{Var}(\mathbf{\text{E}}[j,i])={\sigma }^2$ for $j=1,\dots ,p,i=1,\dots ,n$. We refer to $\mathbf{\text{U}}{\mathbf{\text{V}}}^T$ as a structure, as it contains structured variation underlying $\mathbf{\text{X}}$. PMF imposes the following conditional likelihood on the observed entries in $\mathbf{\text{X}}$ given $\mathbf{\text{U}}$, $\mathbf{\text{V}}$, and ${\sigma }^2$:

$$\mathbf{\text{X}}\mid \mathbf{\text{U}},\mathbf{\text{V}},{\sigma }^2\sim \mathbf{\text{∏}}_{j=1}^p\mathbf{\text{∏}}_{i=1}^n\text{Normal}\left(\mathbf{\text{X}}[j,i]\mid \mathbf{\text{U}}[j,\cdot ]\mathbf{\text{V}}{[i,\cdot ]}^T,{\sigma }^2\right),$$ (1)

where $\text{Normal}(\cdot \mid \cdot ,\cdot )$ represents the density of the univariate Gaussian distribution with mean $\mathbf{\text{U}}[j,\cdot ]\mathbf{\text{V}}{[i,\cdot ]}^T$ and variance ${\sigma }^2$. Mnih and Salakhutdinov (2007) impose mean-zero, Gaussian priors on the factorization components, $\mathbf{\text{U}}$ and $\mathbf{\text{V}}$:

$$\begin{array}{l}\mathbf{\text{U}}\mid {\sigma }_U^2\sim \mathbf{\text{∏}}_{j=1}^p\text{Multivariate-Normal}\left(\mathbf{\text{U}}[j,\cdot ]\mid 0,{\sigma }_U^2{\mathbf{\text{I}}}_{r\times r}\right)\text{and}\\ \mathbf{\text{V}}\mid {\sigma }_V^2\sim \mathbf{\text{∏}}_{i=1}^n\text{Multivariate-Normal}\left(\mathbf{\text{V}}[i,\cdot ]\mid 0,{\sigma }_V^2{\mathbf{\text{I}}}_{r\times r}\right),\end{array}$$ (2)

where ${\mathbf{\text{I}}}_{r\times r}$ is an $r\times r$ identity matrix and ${\sigma }_U^2,{\sigma }_V^2>0$.

2.3. Review of UNIFAC

Whereas PMF applies to a single matrix, the UNIFAC method was developed as a simultaneous low-rank decomposition of multiple matrices (e.g., multi-omic data) using random matrix theory. Consider $q$ omics sources, ${\mathbf{\text{X}}}_1,\dots ,{\mathbf{\text{X}}}_q$ as defined in Section 2.1, row-centered to have mean 0. The UNIFAC decomposition is as follows:

$$\begin{array}{l}{\mathbf{\text{X}}}_{\cdot }={\mathbf{\text{S}}}_{\cdot }+{\mathbf{\text{E}}}_{\cdot }\hfill \\ ={\mathbf{\text{J}}}_{\cdot }+{\mathbf{\text{A}}}_{\cdot }+{\mathbf{\text{E}}}_{\cdot }\hfill \\ ={\mathbf{\text{U}}}_{\cdot }{\mathbf{\text{V}}}^T+{\mathbf{\text{W}}}_{\cdot }{\mathbf{\text{V}}}_{\cdot }^T+{\mathbf{\text{E}}}_{\cdot }\hfill \\ =\left(\begin{array}{c}{\mathbf{\text{U}}}_{\mathbf{1}}\\ ⋮\\ {\mathbf{\text{U}}}_{\mathbf{\text{q}}}\end{array}\right){\mathbf{\text{V}}}^T+\left(\begin{array}{ccc}{\mathbf{\text{W}}}_{\mathbf{1}}& & \\ & \ddots & \\ & & {\mathbf{\text{W}}}_{\mathbf{\text{q}}}\end{array}\right)\left(\begin{array}{c}{{\mathbf{\text{V}}}_{\mathbf{1}}}^T\\ ⋮\\ {{\mathbf{\text{V}}}_{\mathbf{\text{q}}}}^T\end{array}\right)+\left(\begin{array}{c}{\mathbf{\text{E}}}_{\mathbf{1}}\\ ⋮\\ {\mathbf{\text{E}}}_{\mathbf{\text{q}}}\end{array}\right),\hfill \end{array}$$ (3)

where ${\mathbf{\text{J}}}_1,\dots ,{\mathbf{\text{J}}}_q$ are matrices of rank $r<R$ and ${\mathbf{\text{A}}}_1,\dots ,{\mathbf{\text{A}}}_q$ are matrices of rank $r_s<R_s$. ${\mathbf{\text{J}}}_s$ contains joint structure, latent expression patterns shared by all sources, as it is reflected in source $s$. Individual structure, ${\mathbf{\text{A}}}_s$, contains latent expression patterns unique to source $s$. ${\mathbf{\text{E}}}_s$ reflects Gaussian noise with variance ${\sigma }_s^2$ not captured in the decomposition. ${\mathbf{\text{J}}}_s$ is decomposed into $\mathbf{\text{V}}:n\times r$, the joint scores, which contain latent factors expressed by the samples in all sources, and the joint loadings, ${\mathbf{\text{U}}}_s:p_s\times r$ which maps these factors to the observed biomarkers in source $s$. Similarly, ${\mathbf{\text{A}}}_s$ is decomposed into ${\mathbf{\text{V}}}_s:n\times r_s$, the individual scores, which contain the latent factors unique to each source, and the individual loadings, ${\mathbf{\text{W}}}_s:p_s\times r_s$, which map the factors to the feature space spanned by the biomarkers in source $s$. Park and Lock (2020) (and the extension described in Lock et al. (2022)) propose estimating the UNIFAC decomposition in Equation 3 by minimizing the following structured nuclear-norm penalized objective:

$$\left\{{\widehat{\mathbf{\text{J}}}}_s,{\widehat{\mathbf{\text{A}}}}_s\mid s=1,\dots q\right\}=\underset{{\left\{{\mathbf{\text{J}}}_s,{\mathbf{\text{A}}}_s\right\}}_{s=1}^q}{\text{min}}\frac{1}{2}\sum _{s=1}^q{\Vert {\mathbf{\text{X}}}_s-{\mathbf{\text{J}}}_s-{\mathbf{\text{A}}}_s\Vert }_F^2+\text{λ}{\Vert \mathbf{\text{J}}\cdot \Vert }_{\ast }+\sum _{s=1}^q{\text{λ}}_s{\Vert {\mathbf{\text{A}}}_s\Vert }_{\ast },$$ (4)

where ${\widehat{\mathbf{\text{J}}}}_s={\widehat{\mathbf{\text{U}}}}_s{\widehat{\mathbf{\text{V}}}}^T$ and ${\widehat{\mathbf{\text{A}}}}_s={\widehat{\mathbf{\text{W}}}}_s{\widehat{\mathbf{\text{V}}}}_s^T$. Equation 4 is convex and minimized using an iterative soft singular value thresholding algorithm on the singular values of the structures. The algorithm cycles through fixing the individual structures at their most up-to-date values and estimating the joint structure and vice versa. At each iteration, the value of each structure is estimated by minimizing the squared Frobenius norm difference between the observed data and the structure plus a nuclear norm penalty on the structure being estimated. The solution to this sub-problem is given by a soft-thresholding operator on the singular values of the observed data, which retains the left- and right-singular vectors, i.e., the factors, but shrinks to 0 any singular values smaller than $\text{λ}$ (for joint structure) or ${\text{λ}}_s$ (for individual structure). Thus, rank selection for the joint and individual structures is a function of the tuning parameters, $\text{λ}$ and ${\text{λ}}_s$. Fixing ${\text{λ}}_s=\sqrt{n}+\sqrt{p_s}$ is a reasonable choice because it provides a tight upper bound on the largest singular value of the error, ${\mathbf{\text{E}}}_s$, assuming the sources have unit error variance, ${\sigma }_s^2=1$ (Rudelson and Vershynin, 2010). This effectively retains structure driven by components describing biological or technical variation in the data and which are not attributed to error or random noise. This choice also meets the requirements established in Park and Lock (2020) for a uniquely identifiable and non-zero decomposition, which we discuss further in Section 2.6. The penalty on the joint structure, $\text{λ}$, is fixed to $\text{λ}=\sqrt{p}+\sqrt{n}$ by an analogous argument, as ${\mathbf{\text{E}}}_{\cdot }$ is also a mean-zero Gaussian random matrix. Prior to estimating the decomposition, we scale the sources to have unit error variance by dividing each by its estimated error standard deviation calculated by the median absolute deviation (MAD) estimator given in Equation 47 in Gavish and Donoho (2017).

2.4. Bayesian Simultaneous Factorization (BSF)

The resulting factorization from UNIFAC is also the mode of a Bayesian posterior that naturally extends the PMF model, which we leverage for our Bayesian Simultaneous Factorization (BSF) model. Theorem 1 in Park and Lock (2020) establishes the equivalence between minimizing the nuclear norm objective (4) and minimizing a similar objective with matrix-defined $L_2$ penalties (Frobenius norms) on the scores and loadings:

$$\begin{array}{l}\left\{{\widehat{\mathbf{\text{U}}}}_s,\widehat{\mathbf{\text{V}}},{\widehat{\mathbf{\text{W}}}}_s,{\widehat{\mathbf{\text{V}}}}_s\mid s=1,\dots q\right\}=\\ \underset{{\left\{{\mathbf{\text{U}}}_s,\mathbf{\text{V}},{\mathbf{\text{W}}}_s,{\mathbf{\text{V}}}_s\right\}}_{s=1}^q}{\text{min}}\sum _{s=1}^q{\Vert {\mathbf{\text{X}}}_s-{\mathbf{\text{U}}}_s{\mathbf{\text{V}}}^T-{\mathbf{\text{W}}}_s{{\mathbf{\text{V}}}_s}^T\Vert }_F^2+\text{λ}\left({\Vert {\mathbf{\text{U}}}_{\cdot }\Vert }_F^2+{\Vert \mathbf{\text{V}}\Vert }_F^2\right)+\sum _{s=1}^q{\text{λ}}_s\left({\Vert {\mathbf{\text{W}}}_s\Vert }_F^2+{\Vert {\mathbf{\text{V}}}_s\Vert }_F^2\right)\cdot \end{array}$$ (5)

Further, Equation 5 is proportional to the log-posterior for a Bayesian model with Gaussian errors and Gaussian priors on the scores and loadings:

$$\begin{array}{l}{\mathbf{\text{X}}}_{\cdot }\mid {\mathbf{\text{U}}}_{\cdot },\mathbf{\text{V}},{\mathbf{\text{W}}}_{\cdot },{\text{V}}_{\cdot }\sim \\ \mathbf{\text{∏}}_{s=1}^q\mathbf{\text{∏}}_{i=1}^n\mathbf{\text{∏}}_{j=1}^{p_s}\text{Normal}\left({\mathbf{\text{X}}}_s[j,i]\mid {\mathbf{\text{U}}}_s[j,\cdot ]\mathbf{\text{V}}{[i,\cdot ]}^T+{\mathbf{\text{W}}}_s[j,\cdot ]{\mathbf{\text{V}}}_s{[i,\cdot ]}^T,1\right),\end{array}$$ (6)

and

$$\begin{array}{l}{\mathbf{\text{U}}}_s[j,\cdot ]\sim \text{Normal}\left(\mathbf{0},{\text{λ}}^{-1}{\mathbf{\text{I}}}_{r\times r}\right),\\ \mathbf{\text{V}}[i,\cdot ]\sim \text{Normal}\left(\mathbf{0},{\text{λ}}^{-1}{\mathbf{\text{I}}}_{r\times r}\right),\\ {\mathbf{\text{W}}}_s[j,\cdot ]\sim \text{Normal}\left(\mathbf{0},{\text{λ}}_s^{-1}{\mathbf{\text{I}}}_{r_s\times r_s}\right)\text{and}\\ {\mathbf{\text{V}}}_s[i,\cdot ]\sim \text{Normal}\left(\mathbf{0},{\text{λ}}_s^{-1}{\mathbf{\text{I}}}_{r_s\times r_s}\right)\cdot \end{array}$$ (7)

Prior to model fitting, we row-center the features and scale the sources to have error variance 1 using the MAD estimator. Then, the prior variances for ${\mathbf{\text{U}}}_{\cdot }$, $\mathbf{\text{V}}$, ${\mathbf{\text{V}}}_{\cdot }$, ${\mathbf{\text{W}}}_{\cdot }$ are fixed at the penalties described in Section 2.3 so that the posterior mode of the decomposition of ${\mathbf{\text{X}}}_{\cdot }$ matches the UNIFAC decomposition. We then apply the iterative soft singular value thresholding algorithm of UNIFAC to identify the posterior mode and initialize a Gibbs sampling algorithm to sample from the full posterior distributions of ${\mathbf{\text{U}}}_{\cdot }$, $\mathbf{\text{V}}$, ${\mathbf{\text{W}}}_{\cdot }$, and ${\mathbf{\text{V}}}_{\cdot }$.

The factorization for each low-rank term in the decomposition (e.g, ${\mathbf{\text{U}}}_{\cdot }{\mathbf{\text{V}}}^T$) corresponds to a PMF model with ${\sigma }_U^2={\sigma }_V^2$, but the connection to UNIFAC provides several advantages. First, note that equality of variances does not restrict the model, as their scales are not independently identifiable (see Section 2.6) due to the dependence of ${\mathbf{\text{U}}}_{\cdot }’\text{s}$ scale on ${\mathbf{\text{V}}}_{\cdot }’\text{s}$. Moreover, these tuning parameters and the ranks are conveniently fixed via the random matrix theory as discussed in Section 2.3. The tuning parameters are set to $\text{λ}=\sqrt{p}+\sqrt{n}$ and ${\text{λ}}_s=\sqrt{p_s}+\sqrt{n}$ and the ranks are induced by solving the UNIFAC objective with these penalties. As described in Salakhutdinov and Mnih (2008), the choice of tuning parameters can dramatically impact model results, and using cross validation with multiple sources of data is not straightforward (Owen and Perry, 2009). Lastly, an efficient singular value thresholding algorithm can be used to find the mode, circumventing issues of convergence due to poor initialization in Gibbs sampling.

2.5. Bayesian Simultaneous Factorization and Prediction (BSFP)

Now, suppose we have a continuous phenotype $\mathbf{\text{y}}$ in addition to ${\mathbf{\text{X}}}_1,\dots ,{\mathbf{\text{X}}}_q$ for a shared cohort of $n$ samples. We are interested in predicting $\mathbf{\text{y}}$ using the $q$ sources. We extend the BSF model to include prediction of $\mathbf{\text{y}}$ (referred to as BSFP) by assuming the following relationship between the factors, $\mathbf{\text{V}}$ and ${\mathbf{\text{V}}}_s$ for $s=1,\dots ,q$, and $\mathbf{\text{y}}$:

$$\begin{array}{l}\mathbf{\text{y}}={\mathbf{\text{V}}}^{\ast }{\mathit{\beta }}_{\cdot }+{\mathbf{\text{e}}}_y={\beta }_0+\mathbf{\text{V}}{\mathit{\beta }}_{joint}+\sum _{s=1}^q{\mathbf{\text{V}}}_s{\mathit{\beta }}_{indiv,s}+{\mathbf{\text{e}}}_y\text{and}\\ {\mathbf{\text{e}}}_y\sim \text{Normal}\left(\mathbf{0},{\tau }^2{\mathbf{\text{I}}}_{n\times n}\right),\end{array}$$ (8)

where

$$\begin{array}{l}{\mathbf{\text{V}}}^{\ast }=\left(\begin{array}{c}{1}_n\mathbf{\text{V}}{\mathbf{\text{V}}}_1\dots {\mathbf{\text{V}}}_q\end{array}\right)\text{and}\\ \mathit{\beta }\cdot ={\left(\begin{array}{c}{\beta }_0{\mathit{\beta }}_{joint}^T{\mathit{\beta }}_{indiv,1}^T\dots {\mathit{\beta }}_{indiv,q}^T\end{array}\right)}^T\cdot \end{array}$$ (9)

We use the following priors in our model for $\mathbf{\text{y}}$:

$$\begin{array}{l}{\mathit{\beta }}_{\cdot }\sim \text{Normal}\left(\mathbf{0},{\mathbf{\text{Σ}}}_{\beta }\right)\text{and}\\ {\tau }^2\sim \text{Inverse-Gamma}(a,b),\end{array}$$ (10)

where ${\mathbf{\text{Σ}}}_{\beta }=diag\left\{{\alpha }_0^2,{\alpha }^2{\mathbf{\text{I}}}_{r\times r},{\alpha }^2{\mathbf{\text{I}}}_{r_1\times r_1},\dots ,{\alpha }^2{\mathbf{\text{I}}}_{r_q\times r_q}\right\}$ and the hyperparameters, $a$, $b$, ${\alpha }_0^2$, and ${\alpha }^2$ are fixed constants. In our study of HIV-associated OLD (Section 4), we fixed ${\alpha }_0^2={1000}^2$ and ${\alpha }^2=1$. ${\alpha }_0^2$ was chosen to be uninformative and ${\alpha }^2$ was chosen to reflect the anticipated effect size of the factors. This model may be modified to suit the characteristics of a given outcome; for example, we have implemented an analogous model with a probit link for a binary outcome and applied it to gene expression, methylation, and miRNA data from the Cancer Genome Atlas in our Supplementary Materials (see Sections 2 and 5). We initialize as in the BSF model, and infer the full posterior for ${\mathbf{\text{U}}}_{\cdot }$, $\mathbf{\text{V}}$, ${\mathbf{\text{W}}}_{\cdot }$, ${\mathbf{\text{V}}}_{\cdot }$, ${\tau }^2$, and ${\mathit{\beta }}_{\cdot }$ via Gibbs sampling, which we describe in more detail in the Supplement. Inferring the factorization and prediction model simultaneously confers two advantages. First, it incorporates supervision by $\mathbf{\text{y}}$ for the latent structures, which may yield more phenotypically-relevant factors. Second, posterior uncertainty in these underlying factors is propagated through to the predictive model.

We treat $\mathbf{\text{y}}$ distinctly from the other $q$ sources because as a vector it does not have low-rank structure. We estimate the error variance in $\mathbf{\text{y}}$, ${\tau }^2$, explicitly, while we fix the error variance in ${\mathbf{\text{X}}}_{\cdot }$ to 1 after scaling. Simulations showed that initializing with $\mathbf{\text{y}}$ and scaling $\mathbf{\text{y}}$ by its estimated error standard deviation, as is done with each ${\mathbf{\text{X}}}_s$, yielded no improvement in recovery of the underlying structure or in prediction (see Supplementary Materials Section 3.3 for more information).

2.6. Identifiability

The joint and individual structures ${\mathbf{\text{J}}}_{\cdot }$ and ${\mathbf{\text{A}}}_{\cdot }$ in the posterior mode of the decomposition of ${\mathbf{\text{X}}}_{\cdot }$, which matches the solution to the nuclear-norm penalized objective in Equation 4, are uniquely identified by Theorem 1 of Lock et al. (2022). This theorem provides sufficient conditions for identifiability: the decomposition must minimize Equation 4 with $\text{λ}$ and ${\text{λ}}_s$ fixed as in Section 2.3, and for each source, $s=1,\dots ,q$, the columns of the loadings and scores in the joint and individual structures must be linearly independent. In general, the decomposition is not unique, as it is possible that ${{\mathbf{\text{J}}}_{\cdot }}^{(i)}+{{\mathbf{\text{A}}}_{\cdot }}^{(i)}={{\mathbf{\text{J}}}_{\cdot }}^{(j)}+{{\mathbf{\text{A}}}_{\cdot }}^{(j)}$ with ${{\mathbf{\text{J}}}_{\cdot }}^{(i)}\ne {{\mathbf{\text{J}}}_{\cdot }}^{(j)}$ and ${{\mathbf{\text{A}}}_{\cdot }}^{(i)}\ne {{\mathbf{\text{A}}}_{\cdot }}^{(j)}$; however, posterior samples will favor more efficient decompositions (with respect to the regularization induced by $\text{λ}$ and ${\text{λ}}_s$) that concentrate near the unique posterior mode.

A challenge in Bayesian factor models like ours is that the loadings and scores are not identifiable due to rotation, permutation, and sign invariance. Under rotation invariance, ${\mathbf{\text{U}}}_{\cdot }{\mathbf{\text{V}}}^T$, for example, is unchanged if we right-multiply ${\mathbf{\text{U}}}_{\cdot }$ and $\mathbf{\text{V}}$ by an orthogonal $r\times r$ matrix $\mathbf{\text{P}}$, i.e. ${\mathbf{\text{U}}}_{\cdot }\mathbf{\text{P}}{(\mathbf{\text{V}}P)}^T={\mathbf{\text{U}}}_{\cdot }\mathbf{\text{P}}{\mathbf{\text{P}}}^T{\mathbf{\text{V}}}^T={\mathbf{\text{U}}}_{\cdot }{\mathbf{\text{V}}}^T$. Under permutation invariance, the columns in ${\mathbf{\text{U}}}_{\cdot }$ and $\mathbf{\text{V}}$ can be reordered and yield the same decomposition. Likewise, under sign invariance, the signs in the columns of ${\mathbf{\text{U}}}_{\cdot }$ and $\mathbf{\text{V}}$ can be swapped. Rotation, permutation, and sign invariance obstruct interpretation of posterior summaries of the Gibbs samples for ${\mathbf{\text{U}}}_{\cdot }$, $\mathbf{\text{V}}$, ${\mathbf{\text{W}}}_{\cdot }$, ${\mathbf{\text{V}}}_{\cdot }$ , and ${\mathit{\beta }}_{\cdot }$. To address all three sources of non-identifiability, we use the MatchAlign algorithm (Poworoznek et al., 2021) which first orthogonalizes the loadings to load the observed features onto one or a few factors. Then, using a greedy matching algorithm and pivot, factors are iteratively matched to the positively- or negatively-signed pivot columns for which the $L_2$-normed difference is minimized.

We now describe our adaptation of the MatchAlign algorithm to accommodate multiple structures and our predictive model. The MatchAlign algorithm proceeds as follows: at each Gibbs sampling iteration after burn-in, $t=T_{\text{burn-in}},\dots ,T$, define ${\mathbf{\text{U}}}_{\cdot ,\mathit{\beta }}^{(t)}={\left(\begin{array}{c}{\mathbf{\text{U}}}_{\cdot }^{(t)T}{\mathit{\beta }}_{joint}^{(t)}\end{array}\right)}^T$ and ${\mathbf{\text{W}}}_{s,\beta }^{(t)}={\left(\begin{array}{c}{\mathbf{\text{W}}}_s^{(t)T}{\beta }_{indiv,s}^{(t)}\end{array}\right)}^T$ for $s=1,\dots ,q$. For each $t$, we apply a Varimax rotation, yielding ${\mathbf{\text{U}}}_{\cdot ,\beta }^{†(t)}$ and ${\mathbf{\text{W}}}_{s,\beta }^{†(t)}$ for $s=1,\dots ,q$. The resulting rotation is also applied to the scores, yielding ${\mathbf{\text{V}}}^{†(t)}$ and ${\mathbf{\text{V}}}_s^{†(t)}$ for $s=1,\dots ,q$. For pivot, we use ${\mathbf{\text{U}}}_{\cdot ,\beta }^{†(0)}={\left(\begin{array}{c}{\mathbf{\text{U}}}_{\cdot }^{†(0)T}{\beta }_{joint}^{†(0)}\end{array}\right)}^T$ and ${\mathbf{\text{W}}}_{s,\beta }^{†(0)}={\left(\begin{array}{c}{\mathbf{\text{W}}}_s^{†(0)T}{\beta }_{indiv,s}^{†(0)}\end{array}\right)}^T$, where ${\mathbf{\text{U}}}_{\cdot }^{†(0)}$, ${\beta }_{joint}^{†(0)}$, ${\mathbf{\text{W}}}_s^{†(0)}$, and ${\beta }_{indiv,s}^{†(0)}$ are chosen as the posterior sample with the median condition number after burn-in. The median condition number is defined as the posterior sample with the median largest singular value. (Poworoznek et al., 2021) Taking ${\mathbf{\text{U}}}_{\cdot ,\beta }^{†(t)}$ as an example, we start with the column with the largest norm and calculate the normed difference between each column in the pivot under positive and negative signage, ${\mathbf{\text{U}}}_{\cdot ,\beta }^{†(0)}[\cdot ,k]$ and $-{\mathbf{\text{U}}}_{\cdot ,\beta }^{†(0)}[\cdot ,k]$, $k=1,\dots ,r$. We match this column in ${\mathbf{\text{U}}}_{\cdot ,\beta }^{†(t)}$ to signed ${\mathbf{\text{U}}}_{\cdot ,\beta }^{†(0)}[\cdot ,k]$ with $k$ that yields the smallest norm. We proceed similarly with ${\mathbf{\text{W}}}_{s,\beta }^{†(t)}$ for $s=1,\dots ,q$. The scores are permuted and signed appropriately to match. For more information on the performance of the MatchAlign algorithm, we refer readers to Poworoznek et al. (2021).

2.7. Multiple Imputation

The Gibbs sampling algorithm for estimating the posterior of the latent variation structure and regression coefficients in BSFP naturally accommodates multiple imputation of missing values in ${\mathbf{\text{X}}}_{\cdot }$ and $\mathbf{\text{y}}$. We use the most up-to-date posterior samples to impute missing values at each iteration of the sampler. This can be done even when an entire sample is unobserved in a source (referred to as “blockwise” missingness). Imputations are generated from the posterior predictive distribution and can be studied using standard posterior summaries.

Let ${𝐼}_s^{(m)}=\left\{(j,i):{\mathbf{\text{X}}}_s[j,i]\text{missing}\right\}$ denote the set of bivariate indices for which entries in the source ${\mathbf{\text{X}}}_s$ are not observed. At the $t\text{th}$ iteration of the Gibbs sampler, we impute these entries by randomly generating for $(j,i)\in {𝐼}_s^{(m)}$:

$${\mathbf{\text{X}}}_s[j,i]={\mathbf{\text{U}}}_s^{(t)}[j,\cdot ]\mathbf{\text{V}}{[i,\cdot ]}^{(t)T}+{\mathbf{\text{W}}}_s^{(t)}[j,\cdot ]\mathbf{\text{V}}{[i,\cdot ]}_s^{(t)T}+{\mathbf{\text{E}}}_s[j,i],$$ (11)

where ${\mathbf{\text{E}}}_s[j,i]\sim \text{Normal}(0,1)$. Missing values in $y$ may be imputed in a similar manner. Let ${𝐼}_y^{(m)}=\left\{i:y_i\text{missing}\right\}$. For $i\in {𝐼}_y^{(m)}$, we may impute these values at iteration $t$ using:

$$y_i={\mathbf{\text{V}}}^{\ast (t)}[i,\cdot ]\beta {\cdot }^{(t)}+e_i,$$ (12)

where $e_i\sim \text{Normal}\left(0,{\tau }^{2(t)}\right)$.

3. Simulations

We consider two simulation studies to characterize the performance of BSF and BSFP. In Section 3.1, we compare BSFP to existing one- and two-step approaches for factorization and prediction. In Section 3.2, we compare BSF to existing single-imputation approaches using simulated multi-source datasets.

3.1. Model Comparison

We examine the ability of BSFP to recover latent variation structure and predict a continuous outcome under varying levels of signal-to-noise (s2n). We define s2n as the ratio of the variance in the latent structure $({\mathbf{\text{J}}}_{\cdot }+{\mathbf{\text{A}}}_{\cdot })$ to the variance in the error $\left({\mathbf{\text{E}}}_{\cdot }\right)$, i.e., $s2n=\text{Var}(\text{vec}({\mathbf{\text{J}}}_{\cdot })+\text{vec}({\mathbf{\text{A}}}_{\cdot }))/\text{Var}(\text{vec}({\mathbf{\text{E}}}_{\cdot }))$. We generated $q=2$ sources of data with 100 features each on $n=200$ samples, which were then split into a training and test set, 100 samples apiece, denoted ${\mathbf{\text{X}}}_{\cdot }^{train}$, ${\mathbf{\text{y}}}^{train}$, ${\mathbf{\text{X}}}_{\cdot }^{test}$, and ${\mathbf{\text{y}}}^{test}$. The true overall rank of the latent structure, the sum of the ranks of the joint and individual structures, was 3, where $r=1$ and $r_s=1$ for $s=1,2$. We generated ${\mathbf{\text{X}}}_{\cdot }$ according to the decomposition in Equation 6 where the entries of ${\mathbf{\text{U}}}_{\cdot }$, $\mathbf{\text{V}}$, ${\mathbf{\text{V}}}_{\cdot }$, and ${\mathbf{\text{W}}}_{\cdot }$ were generated iid from a Normal(0,1) distribution. We generated $\mathbf{\text{y}}$ from the model in Equation 8 where an intercept was generated from a Normal(0,10) distribution and ${\beta }_{joint}$, ${\beta }_{indiv,s}$ were generated iid Normal(0,1). Random noise in ${\mathbf{\text{X}}}_{\cdot }$ and $\mathbf{\text{y}}$ were generated iid Normal(0,1). We scaled ${\mathbf{\text{X}}}_{\cdot }$ and $\mathbf{\text{y}}$ to have s2n ratios 9, 3, 1, and $\frac{1}{3}$ and considered all 16 combinations of s2n. These s2n levels were chosen to reflect realistic signal levels in omics data. In practice, we expect an average s2n of approximately 1, and the levels 9, 3, and $\frac{1}{3}$ help illustrate how the methods perform with especially-high and especially-low signal in the data.

We considered UNIFAC, JIVE (Lock et al., 2013), MOFA (Argelaguet et al., 2018), sJIVE (Palzer et al., 2022), BIP (Chekouo and Safo, 2023), multiview (Ding et al., 2022), and IntegratedLearner (Mallick et al., 2023) as alternative methods. BIP and sJIVE perform simultaneous factorization and prediction and are the most natural comparisons to BSFP. For UNIFAC, JIVE, and MOFA, which do not directly accommodate prediction, we treated the estimated factors as fixed covariates in a Bayesian linear model for $\mathbf{\text{y}}$ as described in Section 2.5, following the two-step approach described in Section 1. UNIFAC was also an important comparison, as it matches the posterior mode of the data-decomposition in BSFP but error is not propagated from the estimated factorization to prediction. The multiview method does not estimate an underlying factorization to use within its predictive model. Instead, it uses the observed features in a predictive model for each source and encourages “agreement” between the predictions across datasets. The IntegratedLearner framework also does not estimate an underlying factorization, but constructs a weighted average of predicted outcomes based on each source using a user-selected learner. We considered Bayesian additive regression trees (BARTs) and the lasso as the per-source learners and aggregated the predictions using stacking and weights estimated by non-negative least squares.

We compared how well the models recovered the underlying structure in ${\mathbf{\text{X}}}_{\cdot }$ and how well each predicted a held-out $\mathbf{\text{y}}$ on the test data. The proposed model was fit on the full training and test ${\mathbf{\text{X}}}_{\cdot }$ with only access to ${\mathbf{\text{y}}}^{train}$. We used UNIFAC, JIVE, and MOFA to estimate the underlying structure on the full ${\mathbf{\text{X}}}_{\cdot }$. The scores corresponding to the test set were then used as covariates to predict ${\mathbf{\text{y}}}^{test}$. sJIVE and BIP were trained on ${\mathbf{\text{X}}}_{\cdot }^{train}$ and ${\mathbf{\text{y}}}^{train}$, as these methods do not accommodate prediction of unobserved outcomes. We then used the estimated loadings to predict ${\mathbf{\text{y}}}^{test}$ using ${\mathbf{\text{X}}}_{\cdot }^{test}$ . We ran each model under each s2n combination for 100 replications and compared their recovery of the underlying structure using the relative squared error (RSE):

$$RSE\left(\mathbf{\text{S}},{\widehat{\mathbf{\text{S}}}}^{mod}\right)=\frac{{\Vert \mathbf{\text{S}}-{\widehat{\mathbf{\text{S}}}}^{mod}\Vert }_F^2}{{\Vert \mathbf{\text{S}}\Vert }_F^2},$$ (13)

where $\mathbf{\text{S}}\in \{\mathbf{\text{J}},\mathbf{\text{A}}\}$ reflects the true joint or individual structure, respectively, and ${\widehat{\mathbf{\text{S}}}}^{mod}\in \left\{{\widehat{\mathbf{\text{J}}}}^{mod},{\widehat{\mathbf{\text{A}}}}^{mod}\right\}$ reflects the estimated joint or individual structures from model, mod. An RSE close to 0 suggests better performance. With the exception of UNIFAC, we also consider each method with ranks fixed at the truth. Since the multiview and IntegratedLearner methods do not estimate latent structure, we will not include them in our discussion of how well the compared methods recover underlying structure. We calculate coverage of the truth for BSFP, UNIFAC, JIVE, MOFA, and IntegratedLearner with BARTs using 95% credible intervals. We do not calculate coverage for sJIVE, BIP, multiview, and IntegratedLearner with lasso because these methods do not offer full posterior inference for the predictive model. In the Supplementary Materials, we include rank selection results for BSFP, UNIFAC, sJIVE, MOFA, JIVE, and BIP. Additional discussion on rank selection in integrative factorization is given in Palzer et al. (2022) and Gaynanova and Li (2019).

The RSEs averaged across simulation replications for recovery of the underlying joint and individual structures is shown in Figure 1 on a log-scale. BSFP, UNIFAC, MOFA, BIP, and sJIVE all performed similarly well in recovering the joint and individual structure, even compared to their performances given the true ranks. With high signal, JIVE overestimated the ranks, leading to poor estimation of the structure. The average relative RSE for recovery of $\mathbb{E}\left({\mathbf{\text{y}}}^{test}\mid {\mathbf{\text{X}}}_{\cdot }\right)$ is shown in Figure 2 on a log-scale for s2ns 9 and 1/3. All models performed comparably well, with BSFP, BIP, and sJIVE showing the smallest median RSE and lowest variability under high signal in ${\mathbf{\text{X}}}_{\cdot }$ and $\mathbf{\text{y}}$. This suggests a unique benefit to one-step approaches under certain conditions, though BSFP and the comparison approaches provided good prediction accuracy across a range of signal levels in the data and response. The multiview method was likely less competitive across these conditions because it does not explicitly estimate an underlying factorization. While the one-step approaches to factorization and prediction yielded comparable prediction accuracy to existing two-step approaches, the greatest benefit to a one-step approach is illustrated most clearly below.

Figure 1:

Comparing BSFP to existing methods on how well each recovers underlying joint and individual structure based on relative squared error (RSE). RSE values closer to 0 reflect better performance. Each panel reflects a different signal-to-noise (s2n) ratio in ${\mathbf{\text{X}}}_{\cdot }$. We do not differentiate according to s2n in $\mathbf{y}\cdot$ as results did not vary under differing levels of signal in the response. For each method and s2n, a boxplot shows the distribution of RSE values across replications; the middle line of the boxplot is the median RSE, and the upper and lower edges give the interquartile range.

Figure 2:

Comparing BSFP to existing methods on how well each recovers the $\mathbb{E}\left({\mathbf{\text{y}}}^{test}\mid {\mathbf{\text{X}}}_{\cdot }\right)$ based on relative squared error (RSE). RSE values closer to 0 reflect better performance. Each panel reflects a different signal-to-noise (s2n) ratio in ${\mathbf{\text{X}}}_{\cdot }$ and ${\mathbf{\text{y}}}^{test}$. We select only the highest and lowest s2n ratios for space considerations. For each method and s2n , a boxplot shows the distribution of RSE values across replications; the middle line of the boxplot is the median RSE, and the upper and lower edges give the interquartile range.

The most important benefit of BSFP is apparent when studying posterior coverage of $\mathbb{E}\left({\mathbf{\text{y}}}^{test}\mid {\mathbf{\text{X}}}_{\cdot }\right)$, shown in Figure 3 for s2ns 9 and 1/3. We expect to see coverage around 95%, which BSFP achieves across signal-to-noise levels while UNIFAC, JIVE, and MOFA fall short, even under high signal in the data and outcome. This is because BSFP propagates error from estimation of the underlying structure to the predictive model, yielding nominal coverage rates. This advantage is especially apparent when the signal in ${\mathbf{\text{X}}}_{\cdot }$ is low, leading to higher uncertainty in estimating the structure. BSFP marginalizes over this uncertainty when estimating the posterior distribution for $\mathbb{E}\left({\mathbf{\text{y}}}^{test}\mid {\mathbf{\text{X}}}_{\cdot }\right)$. IntegratedLearner with BARTs also has this advantage, though instead of propagating uncertainty from the estimated latent structure, it propagates uncertainty from the underlying regression trees to the ensemble predictions. UNIFAC, JIVE, and MOFA treat the identified factors as fixed, assuming there is no associated uncertainty. This affects our confidence in the predicted values for ${\mathbf{\text{y}}}^{test}$ and can affect posterior inference for the estimated regression coefficients. Despite similar performance in recovery of the underlying structure and prediction accuracy for ${\mathbf{\text{y}}}^{test}$, posterior inference on $\mathbb{E}\left({\mathbf{\text{y}}}^{test}\mid {\mathbf{\text{X}}}_{\cdot }\right)$ is unreliable when uncertainty in the estimated factorization is not properly accounted for. This suggests BSFP provides comprehensive posterior inference across varying levels of signal in the data and response. We show here results for the highest and lowest s2ns for space considerations but provide results for all s2ns in the Supplementary Materials.

Figure 3:

Comparing BSFP to existing methods on coverage of $\mathbb{E}\left({\mathbf{\text{y}}}^{test}\mid {\mathbf{\text{X}}}_{\cdot }\right)$ under the posterior. Each panel reflects a different signal-to-noise (s2n) ratio in ${\mathbf{\text{X}}}_{\cdot }$ and ${\mathbf{\text{y}}}^{test}$. Coverage was assessed using 95% credible intervals. We select only the highest and lowest s2n ratios for space considerations. For each method and s2n, a boxplot shows the distribution of coverage values across replications; the middle line of the boxplot is the median coverage, and the upper and lower edges give the interquartile range.

3.2. Missing Data Imputation

We then studied the imputation performance of BSF. Here, we do not consider prediction of an outcome and focus on how well each method imputes observations randomly removed from each ${\mathbf{\text{X}}}_s$. We generated $q=2$ sources of data, measured on $n=100$ samples, each with $p_s=100$ features, for $s=1,2$. Data were generated in the same manner as described in Section 3.1. We considered three types of missingness: (1) entrywise, in which 10% of entries in each source were randomly removed, (2) blockwise, in which 10 non-overlapping samples from each source were randomly removed, and (3) missingness-not-at-random (MNAR), in which the lowest 10% of samples in each source were removed to reflect a global, source-specific limit-of-detection. We varied the s2n across 9, 3, 1, and 1/3. We considered two settings for the true ranks: in setting 1, the overall rank was 15 where $r=r_s=5$ for $s=1,2$, and in setting 2, the overall rank was 3 as in Section 3.1. We focus on results from setting 1 and provide a discussion of setting 2 in the Supplement. For each condition (missingness type, s2n, and rank of the underlying structure), we considered 100 replications and averaged the results across sources and replications.

We compared BSF to five other single imputation approaches, including mean imputation and UNIFAC. We also compared to the iterative SVD imputation algorithm given by Fuentes et al. (2006) (referred to as the SVD) with a rank of 4, k-nearest neighbors (kNN) (Kowarik and Templ, 2016), and random forest (RF) (Stekhoven and Bühlmann, 2012). We applied the SVD, kNN, and RF to each source separately and to the sources combined. We evaluated the performance using the relative squared error of the imputed values compared to the true unobserved values, i.e. $\text{RSE}\left({\mathbf{\text{X}}}_s,{\widehat{\mathbf{\text{X}}}}_s^{mod}\right)={\Vert \left\{{\mathbf{\text{X}}}_s[j,i]|(j,i)\in {𝐼}_s^{(m)}\right\}-\left\{{\widehat{\mathbf{\text{X}}}}_{\text{s}}^{mod}[j,i]|(j,i)\in {𝐼}_s^{(m)}\right\}\Vert }_F^2/{\Vert {\mathbf{\text{X}}}_s[j,i]\mid (j,i)\in {𝐼}_s^{(m)}\Vert }_F^2$ where ${\widehat{\mathbf{\text{X}}}}_s^{mod}[j,i]$ denotes the imputation for ${\mathbf{\text{X}}}_s[j,i]$ given by model mod. For BSF, the RSE was calculated using the posterior mean of the imputed values across Gibbs samples. In the Supplement, we discuss uncertainty in multiple imputation when applying BSF under different missingness mechanisms and signal levels.

Imputation accuracy results under an assumed overall rank of 15 (i.e., $r=r_1=r_2=5$) are shown in Table 1. Models were compared using a t-test to assess whether performance was significantly different at the 0.05 level within each condition. Under high signal ($\text{s}2\text{n}=9$ and 3), both BSF and UNIFAC excel at predicting the unobserved values, with RF closely following in terms of performance.

Table 1:

Average RSEs for imputing missing values under varying levels of signal and missingness mechanisms under an assumed rank of 15. Bolded values correspond to the best-performing method under the given condition. If more than one value is bolded, model performance was not significantly different from the best-performing model.

	s2n = 9			s2n = 3			s2n = 1			s2n = 0.333
Model	Entrywise	Blockwise	MNAR	Entrywise	Blockwise	MNAR	Entrywise	Blockwise	MNAR	Entrywise	Blockwise	MNAR
BSF	0.226	0.638	0.572	0.460	0.764	0.871	0.790	0.917	1.361	0.977	0.992	1.262
UNIFAC	0.219	0.663	0.542	0.435	0.779	0.806	0.740	0.914	1.335	0.951	0.986	1.279
Mean Imputation	1.011	1.012	1.426	1.011	1.012	1.418	1.011	1.011	1.654	1.011	1.011	1.491
SVD (Combined)	0.634	0.992	0.835	0.711	1.023	0.983	0.840	1.077	1.394	0.980	1.166	1.403
SVD (Separate)	0.559		0.610	0.653		0.816	0.815		1.354	0.993		1.405
kNN (Combined)	0.729	1.022	0.889	0.854	1.095	0.986	1.050	1.200	1.388	1.213	1.274	1.410
kNN (Separate)	0.685	1.529	0.840	0.830	1.461	0.957	1.045	1.379	1.388	1.216	1.334	1.411
RF (Combined)	0.448	0.773	0.950	0.602	0.840	1.095	0.816	0.936	1.564	0.968	1.004	1.491
RF (Separate)	0.396	1.070	0.872	0.559	1.054	1.036	0.790	1.038	1.539	0.965	1.031	1.487

Under blockwise missingness, methods that do not estimate an underlying factorization and methods applied to each source separately are not expected to perform well, as there is no information available to predict the missing samples. In fact, the SVD algorithm would not run under this condition, denoted by blanks in Table 1. Under high signal, BSF, UNIFAC, and RF applied to the sources combined were the only methods yielding RSEs below 1 as they were able to impute values other than 0. As in the entrywise missingness case, these gains in performance dissipated as the signal in the data decreased.

Under MNAR and high signal, BSF, UNIFAC, and SVD applied to each source separately provided the best predictive accuracy. This is reasonable given recent results supporting the use of low-rank factorization methods to impute missing values under the MNAR assumption (Wang et al., 2021), and underscores the non-viability of methods like kNN and RF in this case.

4. Application to HIV-OLD Study

We applied BSFP to metabolomic and proteomic data collected from the Vancouver and Pittsburgh lung cohorts as part of a matched case-control study on HIV-associated obstructive lung disease (OLD) (Cribbs et al., 2016; Akata et al., 2022). We are interested in predicting lung function based on the metabolome and proteome in bronchoalveolar lavage lung fluid (BALF). In this study, 26 cases (those with OLD) were matched to 26 non-OLD controls based on age, antiretroviral treatment status, and smoking status. Lung function was measured as percent predicted forced expiratory volume in 1-second (FEV1pp). Our dataset contained 252 BALF metabolites and 4253 BALF proteins, which we used to predict FEV1pp. Prior to model fitting, we log-transformed the features in both datasets to normalize their distributions.

We first compared BSFP to sJIVE, JIVE, UNIFAC, MOFA, multiview, and IntegratedLearner under cross validation where we iteratively held out FEV1pp for each case-control pair. Due to computational barriers, we were unable to fit BIP. We fit BSFP using 5000 posterior samples with a 2500 sample burn-in, which was deemed sufficient based on monitoring of trace plots and the log-joint likelihood of the model. Each model was fit in the same manner as in Section 3. We also considered the lasso regression model (Tibshirani, 1996) on the combined metabolomic and proteomic data and on each separately. To fit the lasso model, we held-out the FEV1pp values and the metabolomic and/or proteomic observations for each case-control pair and trained the model on the remaining samples. We compared the correlation between the predicted and true held-out FEV1pp values. For BSFP, we used the posterior mean of the predicted FEV1pp values. The correlations and their p-values given by a Pearson correlation test were as follows: BSFP (0.4847, p = 0.0003), UNIFAC (0.4787, p = 0.0003), sJIVE (0.4425, p = 0.0010), IntegratedLearner (stacked BART) (0.4299, p = 0.0015), lasso (combined sources) (0.4029, p = 0.0031), multiview (0.4084, p = 0.0027), lasso (metabolite only) (0.3972, p = 0.0035), IntegratedLearner (stacked lasso) (0.3808, p = 0.0054), MOFA (0.3719, p = 0.0066), lasso (protein only) (0.3157, p = 0.0226), and JIVE (0.3131, p = 0.0238). BSFP and UNIFAC had comparable predictive performance, though BSFP has the advantage of providing a framework for uncertainty in the estimated factorization. In addition, models that distinguished between the joint and individual factors generally performed better, and considering both sources yielded better prediction than using only the metabolomic or proteomic data.

We now focus on the results from BSFP, which identified 14 joint, 11 metabolite-specific, and 16 protein-specific factors for a total rank of 41. It took approximately 2.3 hours to fit BSFP to this dataset on a single core of a Linux desktop with a 4.0 GHz i7–6900K processor. We visualize the estimated structures via heatmap (Figure 4), where the samples (columns) are ordered by FEV1pp. The joint structure, which explained 17.7% of variation in the metabolome (95% CI=(16.3%,18.8%)) and 21.3% in the proteome (19.9%, 22.3%) reveals a sample cluster driven by shared metabolomic and proteomic expression, highlighted in orange. The individual structures explained 54.6% (53.6%,55.5%) in the metabolome and 61.8% (60.9%,62.6%) in the proteome. The proportion of variance in FEV1pp explained by the joint factors was 2.3% (0.4%,6.5%), while the metabolomic factors explained 1.2% (0.3%,3.0%) and the proteomic factors explained 6.3% (1.5%,14.5%). We visualize the accuracy of the fitted FEV1pp vs. observed and the associated uncertainty in fitted FEV1pp in Figure 5. Observed FEV1pp was heterogeneous, ranging from 21 to 128% of predicted normal. The fitted FEV1pp ranged from 73 to 93% of predicted normal, reflecting the challenge of capturing the heterogeneity in a small sample. Posterior 95% credible intervals reflect this uncertainty. Seven samples colored in orange correspond to those which clustered together based on shared proteomic and metabolomic patterns in Figure 5. Clustering was done using k-means with $k=2$ on the joint structure across the posterior sampling iterations. Our choice of $k$ was motivated by previous analyses of data from this study in which K-means clustering with $k=2$ separated this unusual cluster from the rest of the cohort (Samorodnitsky et al., 2023). These clustered together over 90% of posterior sampling iterations and had lower fitted FEV1pp than the other 45 samples. This cluster reflects a multi-omic molecular subtype in the lung that is associated with poor lung function; further molecular association with FEV1pp are uncertain.

Figure 4:

Heatmap of the posterior mean of estimated joint and individual structures. Columns represent samples and rows represent proteins or metabolites. Samples are ordered by FEV1pp. Blue values reflect lower expression and red values reflect higher expression relative to the rowwise mean. The orange rectangle highlights a cluster of samples with low FEV1pp driven by joint metabolomic and proteomic expression.

Figure 5:

Plot of fitted vs. observed FEV1pp for each of 52 samples with 95% credible intervals. Those colored in orange correspond to the samples which clustered together based on joint metabolomic and proteomic expression.

We applied the alignment algorithm described in Section 2.6 to study the estimated factors using posterior summaries. Alignment results were similar when we considered matching to posterior samples around the chosen pivot. We visualize the posterior summaries of all factors in a Shiny app at https://sarahsamorodnitsky.shinyapps.io/BSFP_HIV_OLD/. We discuss here the joint factor which explained the most variation within the joint structure, which we refer to as “Joint Factor 1”. This factor is associated with the previously-identified cluster, based on the distinct scores for these samples. The loadings for 61/252 metabolites and 666/4253 proteins had 95% credible intervals which did not contain zero, suggesting they contribute “significantly” to this factor. We used these biomarkers in a pathway analysis using IMPaLA (Kamburov et al., 2011). The top pathways were neutrophil degranulation (FDR-adjusted p-value $<2e^{-6}$) and innate immunity (FDR-adjusted p-value $<2e^{-6}$), both of which are pertinent to OLD. Neutrophil activation and subsequent inflammation is a hallmark of the disease (Herrero-Cervera et al., 2022). In addition, there is evidence that multiple host defense mechanisms, including innate immunity, play a role in OLD pathogenesis (Agustí and Hogg, 2019).

The metabolomics and proteomics datasets did not contain missing values. However, in the Supplementary Materials we consider artificially inducing entrywise and blockwise missingness to examine how well each dataset can be used to impute missing values in the other.

5. Discussion

In this article, we proposed a Bayesian framework for exploratory integrative factorization (BSF) and simultaneous factorization and prediction (BSFP) of multi-omic data. BSF and BSFP provide a complete framework for uncertainty in the estimated factorization, predictive model (for BSFP), and imputed values.

We showed via simulation the importance of accounting for uncertainty in the estimated factors during prediction. Otherwise, posterior inference in the predictive model may be underconservative, especially when signal in the omics sources is low (but even when signal is high). We did not find that incorporating an outcome into the estimation of the factorization yielded improved accuracy in recovering the underlying structure. This is consistent with previous work (Palzer et al., 2022) which showed that incorporating an outcome into the latent factorization estimation yielded only modest benefits. We found the greatest benefit to simultaneous factorization and prediction was in uncertainty quantification in the latent factorization and predictive model. Additionally, our simulations suggested BSF can be used under various missingness patterns. When observations were randomly removed from each source, BSF was competitive against existing methods for imputing unobserved values. Under blockwise missingness and MNAR, BSF yielded gains in imputation accuracy. Further, BSF offers full posterior inference for the imputed values, which can be studied using posterior summaries, and posterior inference for the unknown parameters marginalizes over the uncertainty in the imputed values. In choosing an imputation approach, researchers may value the ability to perform multiple imputation over single imputation and have access to the posterior distributions for the imputed values.

Our data application revealed a cluster of participants with HIV-associated OLD driven by joint metabolomic and proteomic abundance patterns. While this sheds light on some possible disrupted disease pathways, more research is needed to validate if such an OLD subtype exists in other cohorts. While BSFP improved upon existing methods in terms of prediction, it was challenging to capture the heterogeneity in FEV1pp in this dataset. One challenge may have been the low sample size (52) relative to the rank of the estimated factorization (41). In the future, considering a steeper penalty in the initialization of the model (i.e., smaller prior variance on the factorization components) could yield a lower-ranked factorization with more distinguished factors.

Our approach directly extends the UNIFAC method to a fully Bayesian framework, and can also be viewed as an extension of PMF. There is a dynamic literature on alternative approaches to Bayesian matrix factorization, particularly with respect to infinite factorization models with structured shrinkage on the loadings (Bhattacharya and Dunson, 2011; Legramanti et al., 2020). Our model choice was chiefly motivated by computational efficiency and simplicity, with efficiency achieved via fast initialization at the posterior mode and conjugate priors for direct Gibbs sampling, and simplicity via the lack of hyperparameters to specify and a fixed finite model dimension. However, there are advantages of Bayesian infinite factorization approaches that are worth exploring in our context (e.g., to accommodate uncertainty in the shared and individual ranks).

There are many avenues for further development. Expanding this framework to accommodate zero-inflated or non-Gaussian data, such as SNP or imaging data would be valuable. However, this would require an alternative strategy to rank selection. Implementing a Bayesian approach would be a worthwhile solution. Another avenue is exploring alternatives to choosing the prior variance on the factorization structures, $\text{λ}$ and ${\text{λ}}_s$. One could impose priors or consider a cross validation approach. Accommodating “bidimensional” structure (multiple omics sources measured on multiple sample cohorts) would be worthwhile, as well as accommodating longitudinal data structures. It would also be straightforward to extend this framework to accommodate other parametric models for the outcome, such as a parametric survival model or Poisson model. Finally, future work could impose sparsity on the loadings or on the regression coefficients, though this would require careful consideration of the model identifiability.

Figure 6:

Scores for each sample for Joint Factor 1, the joint factor that explains the largest variation within the joint structure. Each point reflects the posterior mean score and the interval reflects the 95% credible interval. Intervals colored in orange correspond to samples that belong to the stand-out cluster.

Figure 7:

Loadings of each observed metabolite for Joint Factor 1. Each point reflects the posterior mean loading and the interval reflects the 95% credible interval. Intervals colored in orange correspond to those that do not contain 0.

Figure 8:

Loadings of each observed protein for Joint Factor 1. Each point reflects the posterior mean loading and the interval reflects the 95% credible interval. Intervals colored in orange correspond to those that do not contain 0.

Table 2:

Top pathways based on the metabolites and proteins with non-zero loadings on “Joint Factor 1”, which explained the most variation across the metabolomic and proteomic datasets.

Pathway	P-Value (Protein)	Q-Value (Protein)	P-Value (Metabolite)	Q-Value (Metabolite)	P-Value (Joint)	Q-Value (Joint)
Neutrophil degranulation	0.00000	0.00000	1.00000	1	0.00000	0.00000
Innate Immune System	0.00000	0.00000	0.70600	1	0.00000	0.00000
Immune System	0.00000	0.00000	0.92000	1	0.00000	0.00000
Neutrophil extracellular trap formation - Homo sapiens	0.00000	0.00019	1.00000	1	0.00000	0.00019
HDMs demethylate histones	0.00000	0.00019	1.00000	1	0.00000	0.00019
Transcriptional misregulation in cancer - Homo sapiens	0.00000	0.00114	1.00000	1	0.00000	0.00114
Complement and coagulation cascades - Homo sapiens	0.00000	0.00114	1.00000	1	0.00000	0.00114
RHO GTPases activate PKNs	0.00000	0.00114	1.00000	1	0.00000	0.00114
Regulation of Insulin-like Growth Factor (IGF) transport and uptake by Insulin-like Growth Factor Binding Proteins (IGFBPs)	0.00000	0.00160	1.00000	1	0.00000	0.00160
Signaling by Interleukins	0.00000	0.00201	1.00000	1	0.00000	0.00201