Multi-response Regression for Block-missing Multi-modal Data without Imputation

Abstract

Multi-modal data are prevalent in many scientific fields. In this study, we consider the parameter estimation and variable selection for a multi-response regression using block-missing multi-modal data. Our method allows the dimensions of both the responses and the predictors to be large, and the responses to be incomplete and correlated, a common practical problem in high-dimensional settings. Our proposed method uses two steps to make a prediction from a multi-response linear regression model with block-missing multi-modal predictors. In the first step, without imputing missing data, we use all available data to estimate the covariance matrix of the predictors and the cross-covariance matrix between the predictors and the responses. In the second step, we use these matrices and a penalized method to simultaneously estimate the precision matrix of the response vector, given the predictors, and the sparse regression parameter matrix. Lastly, we demonstrate the effectiveness of the proposed method using theoretical studies, simulated examples, and an analysis of a multi-modal imaging data set from the Alzheimer’s Disease Neuroimaging Initiative.

1. Introduction

With the prevalence of large-scale multi-modal data in various scientific fields, multi-response linear regression is attracting increasing attention in the statistics and machine learning communities (Rothman et al., 2010; Lee and Liu, 2012; Loh and Zheng, 2013). Although linear regressions with a scalar response are well studied, many applications may have a vector as the response, for example, in biological problems (Kim and Xing, 2012). For example, for multi-tissue joint expression quantitative trait loci (eQTL) mapping (Molstad et al., 2020), researchers predict gene expression values in multiple tissues simultaneously by using a weighted sum of eQTL genotypes. A separate prediction for each tissue is inefficient if the same genes in different tissues are correlated because of shared genetic variants or other unmeasured common regulators. In order to use data from all tissues simultaneously, Molstad et al. (2020) propose a joint eQTL model that considers cross-tissue expression dependence.

To apply variable selection methods to multi-response problems, one option is to separately fit each response using a single-response model. For example, the lasso is a well-studied variable selection method for single-response linear regression models (Tibshirani, 1996). However, although this is a straightforward method, it neglects the dependency structure between responses. Incorporating the dependency structure of the response vector enables us to obtain a more efficient multi-response linear regression approach in terms of estimation and prediction.

For multi-response regression problems, Breiman and Friedman (1997) proposed the curds and whey method to improve the prediction performance by using the dependencies between responses. Specifically, they first fit a single-response regression model for each response, and then modify the predicted values from these regressions by shrinking them using the canonical correlations between the response variables and the predictors. Another popular approach is to use dimension reduction. In particular, the reduced-rank regression (Izenman, 1975) minimizes the least squares criterion, subject to a constraint on the rank of the regression parameter matrix. Yuan et al. (2007) extended this method to include the high-dimensional settings, reducing the dimension by encouraging sparsity among the singular values of the parameter matrix. Nevertheless, although these methods achieve better prediction performance than when using a separate univariate regression, they do not address the problem of variable selection.

In order to handle correlated responses together with variable selection, we can estimate the precision matrix of the response vector, given the predictors, and the regression parameter matrix either separately or simultaneously (Lee and Liu, 2012). For a separate estimation, Cai et al. (2013) use a constrained ${\ell }_1$ minimization that can be treated as a multivariate extension of the Dantzig selector to estimate the regression parameter matrix. After removing the regression effect using the estimated regression parameter matrix, the precision matrix of the error terms can be estimated accordingly. A potential drawback of this indirect method is that it ignores the relationships between the responses, given the predictors, when estimating the regression parameter matrix. Thus, in order to use all information more efficiently, it may be better to estimate the precision matrix and regression parameter matrix simultaneously. Existing joint estimation techniques include those of Rothman et al. (2010), Yin and Li (2011), and Lee and Liu (2012) who formulate the multi-response regression problem in a penalized log-likelihood framework to estimate the parameter and precision matrices simultaneously. Using a similar idea, Chen et al. (2018) propose an estimation procedure that estimates the parameter and precision matrices simultaneously based on the generalized Dantzig selector.

However, most existing multi-response linear regression methods deal only with complete data without missing entries, even though multi-modal data are often incomplete in practice. For instance, studies on Alzheimer’s disease (AD) use data from different sources, including magnetic resonance imaging (MRI) of the brain, positron emission tomography (PET), and cerebrospinal fluid (CSF). In practice, observations of a certain modality can be missing completely, because patients drop out or other practical issues arise, leading to a block-wise missing data structure. Thus, it is important to integrate data from all modalities to improve model prediction and variable selection.

One way of handling incomplete multi-modal data is to simply remove observations with missing entries. However, this procedure may greatly reduce the number of observations and lead to loss of information. Another approach is to perform data imputation. However, existing imputation methods, such as matrix completion (Johnson, 1990) algorithms, may be unstable when the missing values occur in blocks. For such cases, Yu et al. (2020) proposed a direct sparse regression procedure using the covariance from the block-missing multi-modal data (DISCOM). They first use all available information to estimate the covariance matrix of the predictors and the cross-covariance vector between the predictors and the response variable, and then use these estimates and an extended Lasso-type estimator to estimate the coefficients. However, the DISCOM method considers only single-response regressions. Recently, Xue and Qu (2021) proposed the multiple block-wise imputation (MBI) method for a single-response regression when the data are block-wise missing. They developed an estimating equation approach to accommodate block-wise missing patterns in multi-modal data. The method is shown to have high selection accuracy and a low estimation error for a single-response regression with block-wise missing data. However, because their imputation method requires analyzing all combinations of blocks, it can be computationally expensive when the number of modalities is large.

Here, we consider a multi-response regression model for block-wise missing data. The main contribution of our method is to allow missing values in both the responses and the predictors, as well as correlations between responses. In contrast to most traditional methods, the proposed method can also be applied when no subject has complete observations. Our method includes two steps. The first step estimates each element of the covariance and cross-covariance matrices using all available observations without imputation. The second step uses a penalized approach to simultaneously estimate the sparse regression coefficient matrix and the precision matrix of the error terms. We show that this method exhibits estimation and model selection consistency in a high-dimensional setting. The results of our numerical studies and an analysis of Alzheimer’s Disease Neuroimaging Initiative (ADNI) data confirm that the proposed method performs competitively for block-wise missing data.

The remainder of the paper is organized as follows. In Section 2, we introduce the problem background and our model. In Section 3, we establish some theoretical properties of our proposed method, and in Sections 4 and 5, we present our simulation studies and a multi-modal ADNI data example, respectively.

2. Methodology

2.1. Problem setup and notation

Consider the following multi-response linear regression model:

$$\mathbf{Y}={\mathbf{X}\mathbf{B}}^{\mathrm{*}}+\mathcal{E},$$ (2.1)

where ${\mathbf{B}}^{\mathrm{*}}=(b_{jk})\in {\mathbb{R}}^{p\times q}$ is an unknown $p\times q$ parameter matrix, $\mathbf{Y}={({\mathbf{y}}_1,\dots ,{\mathbf{y}}_n)}^{\mathrm{\top }}$ is the $n\times q$ response matrix, $\mathbf{X}={({\mathbf{x}}_1,\dots ,{\mathbf{x}}_n)}^{\mathrm{\top }}$ is the $n\times p$ design matrix, and $\mathcal{E}={({\mathit{ϵ}}_1,\dots ,{\mathit{ϵ}}_n)}^{\mathrm{\top }}$ is an $n\times q$ error matrix. We assume that ${\{{\mathbf{x}}_i\}}_{i=1}^n$ are independent and identically distributed (i.i.d.) realizations of a random vector ${(X_1,\dots ,X_p)}^{\mathrm{\top }}$ with zero mean and covariance matrix ${\mathbf{\Sigma }}_{XX}=({\sigma }_{ij}^{XX})\in {\mathbb{R}}^{p\times p}$. We use ${\mathbf{\Sigma }}_{XY}=({\sigma }_{ij}^{XY})\in {\mathbb{R}}^{p\times q}$ to denote the cross-covariance matrix between ${\mathbf{x}}_i$ and ${\mathbf{y}}_i$. We assume that the predictors come from multiple modalities, and there are $p_k$ predictors in the kth modality. In addition, $\mathbf{X}$ has block-missing values. That is, for one sample, its measurements in one modality can be entirely missing. We assume the elements of $\mathbf{Y}$ can also be missing. The errors ${\mathbf{ϵ}}_i={({ϵ}_{i1},\dots ,{ϵ}_{iq})}^{\mathrm{\top }}$, for $i=1,\dots ,n$, are i.i.d. realizations from a random vector $\mathbf{ϵ}$ with zero mean and covariance matrix ${\mathbf{\Sigma }}_{ϵ}=({\sigma }_{ij}^{EE})\in {\mathbb{R}}^{q\times q}$. We let ${\mathit{C}}^{\mathrm{*}}={\mathbf{\Sigma }}_{ϵ}^{-1}$. Moreover, we assume ${\mathbf{x}}_i$ and ${\mathbf{ϵ}}_i$ are uncorrelated. Denote the support of ${\mathbf{B}}^{\mathrm{*}}$ and ${\mathbf{C}}^{\mathrm{*}}$ as $S_B=\{j:\mathrm{v}\mathrm{e}\mathrm{c}{({\mathbf{B}}^{\mathrm{*}})}_j\ne 0\}$ and $S_C=\{j:\mathrm{v}\mathrm{e}\mathrm{c}{({\mathbf{C}}^{\mathrm{*}})}_j\ne 0\}$, repectively, where “vec” denotes vectorization by a column operator. For a set $S$, we denote $|S|$ as its cardinality. Denote $s_B=|S_B|$, $s_C=|S_C|$, and $s=\mathrm{m}\mathrm{a}\mathrm{x}(s_B,s_C)$.

We employ the following notation throughout. The symbol ${\mathbb{S}}_{+}^{d\times d}$ denotes sets of $d\times d$ symmetric positive-definite matrices. For a square matrix $\mathbf{C}=(c_{i{i}^{\mathrm{\text{'}}}})\in {\mathbb{R}}^{p\times p}$, we denote its trace as $\mathrm{t}\mathrm{r}(\mathbf{C})={\sum }_i{c}_{ii}$ and its diagonal matrix as $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(\mathbf{C})$. For a matrix $\mathbf{A}=(a_{ij})\in {\mathbb{R}}^{p\times q}$, we define its entrywise ${\ell }_1$-norm as $\parallel \mathbf{A}{\parallel }_1={\sum }_{i,j}|a_{ij}|$, and its entrywise ${\ell }_{\mathrm{\infty }}$-norm as $\parallel \mathbf{A}{\parallel }_{\mathrm{\infty }}={\mathrm{m}\mathrm{a}\mathrm{x}}_{i,j}|a_{ij}|$. In addition, we define its matrix ${\ell }_1$-norm as $\parallel \mathbf{A}{\parallel }_{L_1}={\mathrm{m}\mathrm{a}\mathrm{x}}_j{\sum }_i|a_{ij}|$, the matrix ${\ell }_{\mathrm{\infty }}$-norm as $\parallel \mathbf{A}{\parallel }_{L_{\mathrm{\infty }}}={\mathrm{m}\mathrm{a}\mathrm{x}}_i{\sum }_j|a_{ij}|$, the spectral norm as $\parallel \mathbf{A}{\parallel }_2={\mathrm{m}\mathrm{a}\mathrm{x}}_{\parallel \mathbf{x}{\parallel }_2=1}\parallel \mathbf{A}\mathbf{x}{\parallel }_2$, the Frobenius norm as $\parallel \mathbf{A}{\parallel }_F=\sqrt{{\sum }_{i,j}{a}_{ij}^2}$, and the number of nonzero elements as $\parallel \mathbf{A}{\parallel }_0={\sum }_{i,j}\mathbb{I}(a_{ij}\ne 0)$. Denote the largest and smallest eigenvalues of $\mathbf{A}$ by ${\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}(\mathbf{A})$ and ${\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}(\mathbf{A})$, respectively. Denote the sub-matrix of $\mathbf{A}$ with row and column indices in $I_1$ and $I_2$ as ${\mathbf{A}}_{I_1{I}_2}$. For a vector $\mathbf{v}\in {\mathbb{R}}^p$, denote ${\mathbf{v}}_{I_1}$ as the sub-vector of $\mathbf{v}$ with indices in $I_1$, $\parallel \mathbf{v}{\parallel }_1={\sum }_i|v_i|$, $\parallel \mathbf{v}{\parallel }_{\mathrm{\infty }}={\mathrm{m}\mathrm{a}\mathrm{x}}_i|v_i|$, $\parallel \mathbf{v}{\parallel }_{\mathrm{m}\mathrm{i}\mathrm{n}}={\mathrm{m}\mathrm{i}\mathrm{n}}_i|v_i|$, and $\parallel \mathbf{v}{\parallel }_2=\sqrt{{\sum }_i{v}_i^2}$. For a function $h(X)$, we use ${\nabla }_{X}h$ to denote a gradient or subgradient of $h$ with respect to $X$, if it exists. Finally, we write $a_n\lesssim b_n$ if $a_n\le c{b}_n$ for some constant $c$, and write $a_n\asymp b_n$ if $a_n\lesssim b_n$ and $b_n\lesssim a_n$.

2.2. Proposed multi-DISCOM method

If we separately apply a least squares estimation with the ${\ell }_1$-norm penalty to each response, the multi-response linear regression model (2.1) essentially solves

$$\mathrm{a}\mathrm{r}\mathrm{g}{\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathbf{B}}\mathbb{E}[\parallel \mathbf{Y}-\mathbf{X}\mathbf{B}{\parallel }_F^2]+\lambda \parallel \mathbf{B}{\parallel }_1=\mathrm{a}\mathrm{r}\mathrm{g}{\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathbf{B}}\mathrm{t}\mathrm{r}(\frac{1}{2}{\mathbf{B}}^{\mathrm{\top }}{\mathbf{\Sigma }}_{XX}\mathbf{B}-{\mathbf{\Sigma }}_{XY}^{\mathrm{\top }}\mathbf{B})+\lambda \parallel \mathbf{B}{\parallel }_{1,}$$ (2.2)

where $\lambda$ is a tuning parameter. We refer to this method as the separate lasso, with the solution denoted as ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}^{LASSO}$. However, the approach fails to consider correlations between the responses, and may lead to poor predictive performance (see, e.g., Breiman and Friedman (1997)). To produce a better estimator, we propose incorporating ${\mathbf{\Sigma }}_{ϵ}$ into the estimation of ${\mathbf{B}}^{\mathrm{*}}$ and solving the following problem:

$${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}^0=\arg {\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathbf{B}}\mathrm{t}\mathrm{r}\left[{\mathbf{C}}^{\mathrm{*}}{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{\mathbf{Y}\mathbf{Y}}+{\mathbf{C}}^{\mathrm{*}}{\mathbf{B}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{\mathbf{X}\mathbf{X}}\mathbf{B}-2{\mathbf{C}}^{\mathrm{*}}{\mathbf{B}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{\mathbf{X}\mathbf{Y}}\right]+\lambda \parallel \mathbf{B}{\parallel }_1,$$ (2.3)

where $\lambda$ is a tuning parameter, and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{YY}$, ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}$, and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}$ are estimators of ${\mathbf{\Sigma }}_{YY}$, ${\mathbf{\Sigma }}_{XX}$, and ${\mathbf{\Sigma }}_{XY}$, respectively.

In practice, ${\mathbf{C}}^{\mathrm{*}}$ is usually unknown. In case, we first estimate ${\mathbf{C}}^{\mathrm{*}}$ using $\stackrel{\mathbf{ˆ}}{\mathbf{C}}$, and then plug this into (2.3) and solve the following problem:

$${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}^0=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathbf{B}}{\mathrm{m}\mathrm{i}\mathrm{n}}\mathrm{t}\mathrm{r}\left[\stackrel{\mathbf{ˆ}}{\mathbf{C}}{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{\mathbf{Y}\mathbf{Y}}+{\stackrel{\mathbf{ˆ}}{\mathbf{C}}\mathbf{B}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{\mathbf{X}\mathbf{X}}\mathbf{B}-2\stackrel{\mathbf{ˆ}}{\mathbf{C}}{\mathbf{B}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{\mathbf{X}\mathbf{Y}}\right]+\lambda \parallel \mathbf{B}{\parallel }_1.$$ (2.4)

We refer to this method as the two-step weighted lasso. As shown in But as shown by the toy example in Section 2.2.1, the separate lasso may outperform this method in some problems.

We propose estimating ${\mathbf{B}}^{\mathrm{*}}$ and ${\mathbf{C}}^{\mathrm{*}}$ simultaneously by solving the following optimization problem:

$$\begin{gathered} \begin{array}{l}(\widehat{\text{B}},\widehat{\text{C}})=\text{arg}\underset{\text{C}\in {\mathbb{\text{S}}}_{+}^{q\times q},\text{B}}{\text{min}}\mathrm{\text{tr}}\left[\text{C}{\widehat{\text{Σ}}}_{\text{YY}}+\text{C}{\text{B}}^{\top }{\widehat{\text{Σ}}}_{\text{XX}}\text{B}-2\text{C}{\text{B}}^{\top }{\widehat{\text{Σ}}}_{\text{XY}}\right]\\ \\ +{\lambda }_B\Vert \text{B}{\Vert }_1+{\lambda }_C\Vert \text{C}{\Vert }_1-\log \det \text{C},\end{array} \end{gathered}$$ (2.5)

where ${\lambda }_B$ and ${\lambda }_C$ are tuning parameters. When ${\lambda }_C$ is sufficiently large, Theorem 4 of Banerjee et al. (2008) implies that all off-diagonal entries in $\stackrel{\mathbf{ˆ}}{\mathbf{C}}$ become zero. Then, our proposed method (2.5) reduces to the separate lasso (2.2). For a univariate response regression problem, our proposed method (2.5) reduces to the DISCOM algorithm (Yu et al., 2020). When there are no missing entries, (2.5) reduces to the sparse conditional Gaussian graphical model of Yin and Li (2011).

In the toy example in Section 2.2.1, our joint estimation model (2.5) outperforms the two-step weighted lasso and the separate lasso.

2.2.1. Toy example

For illustration, we consider a toy example similar to that in Lee and Liu (2012). Assume $p=q=2$, ${\mathbf{X}}^{\mathrm{\top }}\mathbf{X}=\mathbf{I}$, and ${\mathbf{\Sigma }}_{ϵ}=\left(\begin{array}{ll}1& \rho \\ \rho & 1\end{array}\right)$, where $\rho$ is an unknown constant. We perform simulation studies for this example with 200 training samples, 300 tuning samples, and 1000 testing samples. Set ${\mathbf{B}}^{*}=\left(\begin{array}{ll}0& 0\\ 2& 3.5\end{array}\right)$ in Case 1, and $\left(\begin{array}{ll}0& 0\\ \mathrm{-2}& 3.5\end{array}\right)$ in Case 2. Figure 1 shows the estimation error for the separate lasso, two-step weighted lasso, and joint estimation model (2.5). In Case 1, the two-step weighted lasso has a smaller estimation error than that of the separate lasso when $\rho$ is positive. The reverse is true when $\rho$ is negative. In Case 2, the separate lasso has a smaller estimation error than that of the two-step weighted lasso when $\rho$ is positive. The joint estimation model performs best in all cases.

Figure 1:

Plots of the estimation errors for the separated lasso, two-step weighted lasso and joint estimation when ${\mathbf{\Sigma }}_{ϵ}=(\begin{array}{ll}1& \rho \\ \rho & 1\end{array})$. The left panel is for ${\mathbf{B}}^{\mathrm{*}}=(\begin{array}{cc}0& 0\\ 2& 3.5\end{array})$, and the right panel is for ${\mathbf{B}}^{\mathrm{*}}=(\begin{array}{cc}0& 0\\ -2& 3.5\end{array})$.

The simulation results can be explained by the following calculations. With the penalty parameter $\lambda$, the solution of the separate LASSO is given by ${\hat{B}}_{ij}^{\text{LASSO}}=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{ij}^S){[{\hat{B}}_{ij}^S-\lambda /2]}_{+}$, where $[u{]}_{+}=u$ if $u\ge 0$, $[u{]}_{+}=0$ if $u<0$, and ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}^S={({\mathbf{X}}^{\mathrm{\top }}\mathbf{X})}^{-1}{\mathbf{X}}^{\mathrm{\top }}\mathbf{Y}$.

We can show that the two-step weighted lasso (2.4) is equivalent to

$${\widehat{\text{B}}}^{2step}=\arg {\min }_B\left[{(\mathrm{vec}(\text{B})-\mathrm{vec}({\text{B}}^S))}^{\top }({\text{I}}_2\otimes \widehat{\text{C}})(\mathrm{vec}(\text{B})-\mathrm{vec}({\text{B}}^S))+\Vert \mathrm{vec}(\text{B}){\Vert }_1\right].$$ (2.6)

When the estimate $\stackrel{\mathbf{ˆ}}{\mathbf{C}}$ is accurate, ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}^{2step}$ should be very close to the solution of (2.3), where we use ${\mathbf{\Sigma }}_{ϵ}^{-1}$ as the weight. After we plug $\stackrel{\mathbf{ˆ}}{\mathbf{C}}={\mathbf{\Sigma }}_{ϵ}^{-1}$ into (2.6), the solution is given by ${\hat{B}}_{ij}^{2step}=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{ij}^S){[|{\hat{B}}_{ij}^S|-\lambda (1+\rho )/2]}_{+}$ when $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{i1}^S{\hat{B}}_{i2}^S)=1$, and ${\hat{B}}_{ij}^{2step}=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{ij}^S){[|{\hat{B}}_{ij}^S)-\lambda (1-\rho )/2]}_{+}$ when $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{i1}^S{\hat{B}}_{i2}^S)=-1$. Compared with ${\hat{B}}_{ij}^{\mathrm{L}\mathrm{A}\mathrm{S}\mathrm{S}\mathrm{O}}=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{ij}^S){[{\hat{B}}_{ij}^S-\lambda /2]}_{+}$, ${\hat{B}}_{ij}^{2\text{step}}$ differs only in the shrinkage amount for each entry. The shrinkage amounts for all entries of the separate lasso are the same, and depend only on the tuning parameter $\lambda$. The shrinkage amounts for all entries of the two-step weighted lasso depend on $\rho$, $\lambda$, and the sign of ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}^S$. Each entry of the two-step weighted lasso may have different shrinkage amounts.

We consider two cases of $\rho$ in Case 1, where ${\mathbf{B}}^{\mathrm{*}}=\left(\begin{array}{cc}0& 0\\ 2& 3.5\end{array}\right)$. Because $B_{21}^{\mathrm{*}}$ and $B_{22}^{\mathrm{*}}$ are far from zero, for simplicity, we assume that $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{21}^S)=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{22}^S)=1$.

1. Consider $\rho =-0.4$. When $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{11}^S{\hat{B}}_{12}^S)=-1$, the shrinkage amounts for ${\hat{B}}_{21}^{2step}$ and ${\hat{B}}_{22}^{2step}$ are $0.7\lambda$, and those for ${\hat{B}}_{11}^{2step}$ and ${\hat{B}}_{12}^{2step}$ are $0.3\lambda$. Thus, the shrinkage amounts for ${\hat{B}}_{21}^{2step}$ and ${\hat{B}}_{22}^{2\text{step}}$ are smaller than those for ${\hat{B}}_{11}^{2step}$ and ${\hat{B}}_{12}^{2step}$. Therefore, with the tuning parameter $\lambda$ that shrinks ${\hat{B}}_{11}^{2\text{step}}$ and ${\hat{B}}_{12}^{2step}$ to zero, the shrinkage amounts for ${\hat{B}}_{21}^{2step}$ and ${\hat{B}}_{22}^{2step}$ are also smaller than those for ${\hat{B}}_{21}^{LASSO}$ and ${\hat{B}}_{22}^{LASSO}$. Thus, the two-step weighted lasso has a smaller estimation error than that of the separate lasso in this scenario. When $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{11}^S{\hat{B}}_{12}^S)=1$, the shrinkage amounts for all entries in ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}^{2step}$ are equal.

2. Consider $\rho =0.4$. When $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{11}^S{\hat{B}}_{12}^S)=-1$, the shrinkage amounts for ${\hat{B}}_{21}^{2step}$ and ${\hat{B}}_{22}^{2step}$ are $0.3\lambda$, and those for ${\hat{B}}_{11}^{2step}$ and ${\hat{B}}_{12}^{2step}$ are $0.7\lambda$. Therefore, with the tuning parameter $\lambda$ that shrinks ${\hat{B}}_{11}^{2step}$ and ${\hat{B}}_{12}^{2step}$ to zero, the shrinkage amounts for ${\hat{B}}_{21}^{2step}$ and ${\hat{B}}_{22}^{2step}$ are larger than those for ${\hat{B}}_{21}^{LASSO}$ and ${\hat{B}}_{22}^{LASSO}$. Thus, the separate lasso is preferred to the two-step weighted lasso in this scenario. When $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\hat{B}}_{11}^S{\hat{B}}_{12}^S)=1$, all entries in ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}^{2step}$ have the same shrinkage amount.

In Case 2, where ${\mathbf{B}}^{\mathrm{*}}=\left(\begin{array}{cc}0& 0\\ -2& 3.5\end{array}\right)$, the two-step weighted lasso is preferred to the separate lasso only when $\rho$ is negative. In conclusion, the performance of the two-step weighted lasso compared with that of the separate lasso depends on the sign of ${\mathbf{B}}^{\mathrm{*}}$ and the covariance matrix ${\mathbf{\Sigma }}_{ϵ}$. In contrast, the joint estimation model (2.5) is more flexible. When ${\mathbf{\Sigma }}_{ϵ}$ and ${\mathbf{B}}^{\mathrm{*}}$ favor the separate lasso, the joint estimation model (2.5) performs better by choosing a large ${\lambda }_C$. Otherwise, it can perform better by choosing a relatively small ${\lambda }_C$, and thus performs competitively in all cases.

2.2.2. Covariance estimation

Now, we show how to obtain ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}$, ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}$, and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{YY}$ when the data exhibit block-missing values. The following notation is used throughout. For the $j$ th predictor, define $S_j^X=\{i:x_{ij}\mathrm{i}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\}\}$. For the $j$ th response, define $S_j^Y=\{i:y_{ij}\mathrm{i}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\}$. Define $S_{jk}^{XX}=\{i:x_{ij}\mathrm{a}\mathrm{n}\mathrm{d}{x}_{ik}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\}$, $S_{jk}^{XY}=\{i:x_{ij}\mathrm{a}\mathrm{n}\mathrm{d}{y}_{ik}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\}$, $S_{jkl}^{XX/Y}=\{i:x_{ij},x_{ik}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g},\mathrm{b}\mathrm{u}\mathrm{t}{y}_{il}\mathrm{i}\mathrm{s}\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\}$, $S_{jkl}^{XY/X}=\{i:x_{ij},y_{ik}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{b}\mathrm{u}\mathrm{t}{x}_{il}\mathrm{i}\mathrm{s}\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\}$, and $S_{jk}^{YY}=\{i:y_{ij}\mathrm{a}\mathrm{n}\mathrm{d}{y}_{ik}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}\}\}$. Denote the cardinality of $S_j^X$, $S_j^Y$, $S_{jk}^{XX}$, $S_{jk}^{XY}$, $S_{jkl}^{XX/Y}$, $S_{jkl}^{XY/X}$, and $S_{jk}^{YY}$ as $n_j^X$, $n_j^Y{n}_{jk}^{XX}$, $n_{jk}^{XY}$, $n_{jkl}^{XX/Y}$, $n_{jkl}^{XY/X}$, and $n_{jk}^{YY}$, respectively. Denote $n_X={\mathrm{m}\mathrm{i}\mathrm{n}}_j|S_j^X|$, $n_{XX}={\min }_{j,k}\left|S_{jk}^{XX}\right|$, $n_{XY}={\min }_{j,k}\left|S_{jk}^{XY}\right|$, $n_{YY}={\min }_{j,k}\left|S_{jk}^{YY}\right|$, $n_{XX/Y}={\mathrm{m}\mathrm{a}\mathrm{x}}_{j,k,l}|S_{jkl}^{XX/Y}|$ and $n_{XY/X}={\mathrm{m}\mathrm{a}\mathrm{x}}_{j,k,l}|S_{jkl}^{XY/X}|$.

We propose using the initial estimators of ${\mathbf{\Sigma }}_{XX}$, ${\mathbf{\Sigma }}_{XY}$, and ${\mathbf{\Sigma }}_{YY}$ as the sample covariance matrices from all available data, that is, ${\tilde{\mathbf{\Sigma }}}_{XX}=({\tilde{\sigma }}_{jt}^{XX})$, ${\tilde{\mathbf{\Sigma }}}_{XY}=({\tilde{\sigma }}_{jt}^{XY})$, ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{YY}=({\hat{\sigma }}_{jt}^{YY})$, where ${\tilde{\sigma }}_{jt}^{XX}={\sum }_{i\in S_{jt}^{XX}}{x}_{ij}{x}_{it}/n_{jt}^{XX}$, ${\tilde{\sigma }}_{jt}^{XY}={\sum }_{i\in S_{jt}^{XY}}{x}_{ij}{y}_{it}/n_{jt}^{XY}$, and

$${\widehat{\sigma }}_{jt}^{YY}=\frac{1}{n_{jt}^{YY}}\sum _{i\in S_{jt}^{YY}}{y}_{ij}{y}_{it}.$$ (2.7)

Note that our method requires that ${\tilde{\mathbf{\Sigma }}}_{XX}$, ${\tilde{\mathbf{\Sigma }}}_{XY}$, and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{YY}$ be unbiased estimators of their counterparts. When the missingness in $\mathbf{X}$ and $Y$ is completely at random, the unbiasedness assumption is satisfied. However, this assumption may also hold under other missing mechanisms. For our theory, we do not specify any particular missing mechanism, and the unbiasedness assumption suffices.

For block-missing data $\mathbf{X}$, the estimate ${\tilde{\mathbf{\Sigma }}}_{XX}$ can be ill-conditioned and have negative eigenvalues. Therefore, it may not be a good estimate of ${\mathbf{\Sigma }}_{XX}$, and cannot be used in (2.5) directly. Next, we introduce an estimator that is both well conditioned and more accurate than the initial estimate ${\tilde{\mathbf{\Sigma }}}_{XX}$. According to the partition of the predictors into $K$ modalities, ${\tilde{\mathbf{\Sigma }}}_{XX}$ can be partitioned into $K^2$ blocks, denoted by ${\tilde{\mathbf{\Sigma }}}^{k_1{k}_2}$, for $1\le k_1$, $k_2\le K$, where ${\tilde{\mathbf{\Sigma }}}^{k_1{k}_2}$ is a $p_{k_1}\times p_{k_2}$ matrix. We denote

$${\tilde{\mathbf{\Sigma }}}_I=\left(\begin{array}{cccc}{\tilde{\mathbf{\Sigma }}}^{11}& & & \\ & {\tilde{\mathbf{\Sigma }}}^{22}& & \\ & & \ddots & \\ & & & {\tilde{\mathbf{\Sigma }}}^{KK}\end{array}\right)\text{and}{\tilde{\mathbf{\Sigma }}}_C=(\begin{array}{cccc}0& {\tilde{\mathbf{\Sigma }}}^{12}& \dots & {\tilde{\mathbf{\Sigma }}}^{1K}\\ {\tilde{\mathbf{\Sigma }}}^{21}& 0& \dots & {\tilde{\mathbf{\Sigma }}}^{2K}\\ ⋮& ⋮& \ddots & ⋮\\ {\tilde{\mathbf{\Sigma }}}^{K1}& {\tilde{\mathbf{\Sigma }}}^{K2}& \dots & 0\end{array})\text{,}$$

where ${\tilde{\mathbf{\Sigma }}}_I$ is called the intra-modality sample covariance matrix and is a $p\times p$ block-diagonal matrix containing $K$ diagonal blocks of ${\tilde{\mathbf{\Sigma }}}_{XX}$, and ${\tilde{\mathbf{\Sigma }}}_C=\tilde{\mathbf{\Sigma }}-{\tilde{\mathbf{\Sigma }}}_I$ is called the cross-modality sample covariance matrix containing all off-diagonal blocks of ${\tilde{\mathbf{\Sigma }}}_{XX}$. Let ${\mathbf{\Sigma }}_I$ and ${\mathbf{\Sigma }}_C$ be the true intra-modality and cross-modality covariance matrices, respectively. For block-missing multi-modal data, the imbalanced sample sizes mean that the estimate ${\tilde{\mathbf{\Sigma }}}_I$ can be relatively accurate, while the estimate ${\tilde{\mathbf{\Sigma }}}_C$ can be inaccurate. In that case, we estimate ${\mathbf{\Sigma }}_{XX}$ using a linear combination of ${\tilde{\mathbf{\Sigma }}}_I$ and ${\tilde{\mathbf{\Sigma }}}_C$ with different weights. In addition, to ensure the positive definiteness of our estimation, we adopt the idea of a shrinkage estimation of the covariance matrix (Fisher and Sun, 2011) and add the diagonal matrix $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}({\tilde{\mathbf{\Sigma }}}_I)$ to our estimator,

$${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}={\alpha }_1{\tilde{\mathbf{\Sigma }}}_I+(1-{\alpha }_1)\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}({\tilde{\mathbf{\Sigma }}}_I)+{\alpha }_2{\tilde{\mathbf{\Sigma }}}_C,$$ (2.8)

where ${\alpha }_1,{\alpha }_2\in [\mathrm{0,1}]$ are two shrinkage weights. We add the diagonal matrix $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}({\tilde{\mathbf{\Sigma }}}_I)$ to ensure the diagonal entries of our estimator are not shrunk.

By Weyl’s theorem, the eigenvalues of our estimator are greater than or equal to ${\alpha }_1{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}({\tilde{\mathbf{\Sigma }}}_I)+(1-{\alpha }_1){\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}(\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}({\tilde{\mathbf{\Sigma }}}_I))+{\alpha }_2{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}({\tilde{\mathbf{\Sigma }}}_C)$. Because $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}({\tilde{\mathbf{\Sigma }}}_I)$ is a positive-definite matrix, we can guarantee that the eigenvalues of our estimator are positive by carefully selecting the tuning parameters ${\alpha }_1$ and ${\alpha }_2$.

As dicussed previously, our estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}$ is a shrinkage estimator. Using a similar idea, we use a shrinkage estimator to estimate ${\mathbf{\Sigma }}_{XY}$. That is, we propose estimating ${\mathbf{\Sigma }}_{XY}$ by

$${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}={\alpha }_3{\tilde{\mathbf{\Sigma }}}_{XY},$$ (2.9)

where ${\alpha }_3\in [\mathrm{0,1}]$ is the shrinkage weight. Here, we want to find the optimal linear combination ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}^{\mathrm{*}}={\alpha }_3^{\mathrm{*}}{\tilde{\mathbf{\Sigma }}}_{XY}$ that minimizes the expected quadratic loss $\mathbb{E}{\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}^{\mathrm{*}}-{\mathbf{\Sigma }}_{XY}\parallel }_F$.

Here, we consider only a relative low dimension of $Y$, with not too many incomplete observations, so we use ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{YY}$ defined in (2.7) directly. However, when the dimension of $Y$ is very high or there are many incomplete observations of $Y$, a shrinkage estimator of ${\mathbf{\Sigma }}_{YY}$ is recommended instead.

Denote ${\gamma }^{\mathrm{*}}={({\gamma }_1^{\mathrm{*}},\dots ,{\gamma }_K^{\mathrm{*}})}^{\mathrm{\top }}={(\mathrm{t}\mathrm{r}({\mathbf{\Sigma }}^{11})/p_1,\dots ,\mathrm{t}\mathrm{r}({\mathbf{\Sigma }}^{KK})/p_K)}^{\mathrm{\top }}$, ${\delta }_I=\sqrt{\mathbb{E}{\parallel {\tilde{\mathbf{\Sigma }}}_I-{\mathbf{\Sigma }}_I\parallel }_F^2}$, ${\delta }_C=\sqrt{\mathbb{E}{\parallel {\tilde{\mathbf{\Sigma }}}_C-{\mathbf{\Sigma }}_C\parallel }_F^2}$, ${\delta }_{XY}=\sqrt{\mathbb{E}{\parallel {\tilde{\mathbf{\Sigma }}}_{XY}-{\mathbf{\Sigma }}_{XY}\parallel }_F^2}$ and $\theta ={\parallel \mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}({\tilde{\mathbf{\Sigma }}}_I)-{\mathbf{\Sigma }}_I\parallel }_F$. The optimal choice for the weights of ${\alpha }_1$, ${\alpha }_2$, and ${\alpha }_3$ is stated in Proposition 2.1.

Proposition 2.1.

The solutions to the two optimization problems

$$({\alpha }_1^{\mathrm{*}},{\alpha }_2^{\mathrm{*}})=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\alpha }_1,{\alpha }_2}{\mathrm{m}\mathrm{i}\mathrm{n}}\mathbb{E}{\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}-{\mathbf{\Sigma }}_{XX}\parallel }_F^2$$ (2.10)

$${\alpha }_3^{\mathrm{*}}=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\alpha }_3}{\mathrm{m}\mathrm{i}\mathrm{n}}\mathbb{E}{\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}-{\mathbf{\Sigma }}_{XY}\parallel }_F^2$$ (2.11)

are

$${\alpha }_1^{*}=\frac{{\theta }^2}{{\theta }^2+{\delta }_I^2},{\alpha }_2^{*}=\frac{{\Vert {\Sigma }_C\Vert }_F^2}{{\Vert {\Sigma }_C\Vert }_F^2+{{\delta }_C}^2},\text{and}{\alpha }_3^{*}=\frac{{\Vert {\Sigma }_{XY}\Vert }_F^2}{{\Vert {\Sigma }_{XY}\Vert }_F^2+{{\delta }_{XY}}^2}.$$

In addition, for ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}^{\mathrm{*}}={\alpha }_1^{\mathrm{*}}{\tilde{\mathbf{\Sigma }}}_I+(1-{\alpha }_1^{\mathrm{*}})\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}({\tilde{\mathbf{\Sigma }}}_I)+{\alpha }_2^{\mathrm{*}}{\tilde{\mathbf{\Sigma }}}_C$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}^{\mathrm{*}}={\alpha }_3^{\mathrm{*}}{\tilde{\mathbf{\Sigma }}}_{XY}$, we have

$$\begin{array}{c}\mathbb{E}{\Vert {\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}^{\mathrm{*}}-{\Sigma }_{XX}\Vert }_F^2=\frac{{\delta }_I^2{\theta }^2}{{\delta }_I^2+{\theta }^2}+\frac{{\delta }_C^2{\Vert {\Sigma }_C\Vert }_F^2}{{\delta }_C^2+{\Vert {\Sigma }_C\Vert }_F^2}\le {\delta }_I^2+{\delta }_C^2=\mathbb{E}{\Vert {\tilde{\mathbf{\Sigma }}}_{XX}-{\tilde{\mathbf{\Sigma }}}_{XX}\Vert }_F^2,\\ \mathbb{E}{\Vert {\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}^{\mathrm{*}}-{\Sigma }_{XY}\Vert }_F^2=\frac{{\delta }_{XY}^2{\Vert {\Sigma }_{XY}\Vert }_F^2}{{\delta }_{XY}^2+{\Vert {\Sigma }_{XY}\Vert }_F^2}\le {\delta }_{XY}^2=\mathbb{E}{\Vert {\tilde{\mathbf{\Sigma }}}_{XY}-{\tilde{\mathbf{\Sigma }}}_{XY}\Vert }_F^2.\end{array}$$

Define the ${\ell }_2$-error of the estimators ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}$ as $\mathbb{E}{\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}-{\mathbf{\Sigma }}_{XX}\parallel }_F^2$ and $\mathbb{E}{\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}-{\mathbf{\Sigma }}_{XY}\parallel }_F^2$, respectively. Proposition 2.1 shows that our estimator is more accurate than the sample covariance matrix.

Proposition 2.1 is closely related to Proposition 1 of Yu et al. (2020). They calculated the optimal weight and estimation error for their proposed estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX,DISCOM}^{\mathrm{*}}$ of ${\mathbf{\Sigma }}_{XX}$, where the estimation error is

$$\mathbb{E}{\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX,DISCOM}-{\mathbf{\Sigma }}_{XX}\parallel }_F^2=\frac{{\delta }_I^2{\tilde{\theta }}^2}{{\delta }_I^2+{\tilde{\theta }}^2}+\frac{{{\delta }_C}^2{\parallel {\mathbf{\Sigma }}_C\parallel }_F^2}{{{\delta }_C}^2+{\parallel {\mathbf{\Sigma }}_C\parallel }_F^2},$$

and ${\tilde{\theta }}^2={\parallel \mathrm{t}\mathrm{r}(\mathbf{\Sigma }){\mathbf{I}}_{\mathbf{p}}/p-{\mathbf{\Sigma }}_I\parallel }_F^2$. Here, our estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}$ has a smaller ${\ell }_2$-error than that of their estimator, and our weighted estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}$ is more accurate than the sample covariance matrix.

2.3. Computational algorithm

In this section, we describe the computational algorithm used to solve the optimization problem (2.5). Because (2.5) is a bi-convex problem, the standard approach to solving it is to use the alternating minimization method. In particular, starting with some given initial point $({\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0,{\stackrel{\mathbf{ˆ}}{\mathbf{C}}}_0)$ at the tth iteration, we solve solving the following problems:

$${\widehat{\text{B}}}_t=\arg {\min }_{\text{B}}\mathrm{tr}\left[{\widehat{\text{C}}}_{t-1}{\widehat{\mathrm{\Sigma }}}_{YY}+{\widehat{\text{C}}}_{t-1}{\text{B}}^{\top }{\widehat{\mathrm{\Sigma }}}_{XX}\text{B}-2{\widehat{\text{C}}}_{t-1}{\text{B}}^{\top }{\widehat{\mathrm{\Sigma }}}_{XY}\right]+{\lambda }_B\parallel \text{B}{\parallel }_1,$$ (2.12)

$${\widehat{\text{C}}}_{\text{t}}=\arg {\min }_{\text{C}\in S_{+}^{q\times q}}\mathrm{tr}\left[\text{C}{\widehat{\mathrm{\Sigma }}}_{YY}+\text{C}{\widehat{\text{B}}}_{t-1}^{\top }{\widehat{\mathrm{\Sigma }}}_{XX}{\widehat{\text{B}}}_{t-1}-2\text{C}{\widehat{\text{B}}}_{t-1}^{\top }{\widehat{\mathrm{\Sigma }}}_{XY}\right]+{\lambda }_C\parallel \text{C}{\parallel }_1-\log \det \text{C}.$$ (2.13)

In each iteration of our algorithm, given ${\stackrel{\mathbf{ˆ}}{\mathbf{C}}}_{t-1}$, we first update the estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_t$ by solving (2.12). Because (2.12) is quadratic in $\mathbf{B}$, we use the coordinate descent algorithm to solve it. Then, we adopt the graphical lasso method of Friedman et al. (2008) to solve (2.13). We summarize the above procedures in Algorithm 1.

3. Theoretical study

We establish the following theoretical results. First, we prove in Theorem 3.1 that the proposed estimators ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}$, ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{YY}$ are consistent with high probability. We then show the convergence rate of our proposed estimators $\stackrel{\mathbf{ˆ}}{\mathbf{B}}$ and $\stackrel{\mathbf{ˆ}}{\mathbf{C}}$ in Theorem 3.4. Finally, the selection consistency of our proposed method is shown in Theorem 3.5. The technical assumptions (A1) to (A5), and all proofs are provided in the Supplementary Material. In the following analysis, we allow $p$ and $q$ to diverge as $n_{XX}$, $n_{XY}$ and $n_{YY}$ increase.

In Theorem 3.1, we prove the large deviation bounds for our proposed estimators ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}$, ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{YY}$.

Theorem 3.1.

Suppose $1-{\alpha }_1=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_X})$, $1-{\alpha }_2=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_{XX}})$, and $1-{\alpha }_3=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}pq/n_{XY}})$. If Conditions (A1) and (A2) hold, there exists positive constants $v_1^{\mathrm{\prime }}$, $v_2^{\mathrm{\prime }}$, and $v_3^{\mathrm{\prime }}$ such that

$$P\left({\parallel {\widehat{\mathrm{\Sigma }}}_{XX}-{\mathrm{\Sigma }}_{XX}\parallel }_{\infty }\ge v_1^{\prime }\sqrt{\frac{\log p}{n_{XX}}}\right)\le \frac{4}{p},$$ (3.1)

$$P\left({\parallel {\widehat{\mathrm{\Sigma }}}_{XY}-{\mathrm{\Sigma }}_{XY}\parallel }_{\infty }\ge v_2^{\prime }\sqrt{\frac{\log (pq)}{n_{XY}}}\right)\le \frac{4}{pq},$$ (3.2)

$$P\left({\parallel {\widehat{\mathrm{\Sigma }}}_{YY}-{\mathrm{\Sigma }}_{YY}\parallel }_{\infty }\ge v_3^{\prime }\sqrt{\frac{\log q}{n_{YY}}}\right)\le \frac{4}{q}.$$ (3.3)

If we only use samples with complete observations, sample covariance estimators ${\tilde{\mathbf{\Sigma }}}_{XX,\text{complete}}$, ${\tilde{\mathbf{\Sigma }}}_{XX\text{,complete}}$ and ${\tilde{\mathbf{\Sigma }}}_{XX\text{,complete}}$ have the following convergence rates

$${\parallel {\tilde{\mathbf{\Sigma }}}_{XX,\text{complete}}-{\mathbf{\Sigma }}_{XX}\parallel }_{\mathrm{\infty }}=O_p\left(\sqrt{(\mathrm{l}\mathrm{o}\mathrm{g}p)/n_{\text{complete}}}\right),$$

$${\parallel {\tilde{\mathbf{\Sigma }}}_{XY,\text{complete}}-{\mathbf{\Sigma }}_{XY}\parallel }_{\mathrm{\infty }}=O_p\left(\sqrt{(\mathrm{l}\mathrm{o}\mathrm{g}(pq))/n_{\text{complete}}}\right),$$

$${\parallel {\tilde{\mathbf{\Sigma }}}_{YY,\text{complete}}-{\mathbf{\Sigma }}_{YY}\parallel }_{\mathrm{\infty }}=O_p\left(\sqrt{(\mathrm{l}\mathrm{o}\mathrm{g}q)/n_{\text{complete}}}\right),$$

where $n_{\text{complete}}$ is the number of samples with complete observations; see Yu et al. (2020). For block-missing data, $n_{\text{complete}}$ can be much smaller than $n_{XX}$, $n_{XY}$ and $n_{YY}$.

Next, we give the properties of initial estimators ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{C}}}_0$. The following lemma describes estimation consistency of the initial estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0$.

Lemma 3.2.

Suppose Conditions (A1)-(A4) hold, $1-{\alpha }_1=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_X}),1-{\alpha }_2=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_{XX}})$, and $1-{\alpha }_3=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}pq/n_{XY}})$. If we choose ${\lambda }_{B_0}=C{(\mathrm{l}\mathrm{o}\mathrm{g}(pq)/\mathrm{m}\mathrm{i}\mathrm{n}(n_{XY},n_{XX}))}^{\frac{1}{2}}{\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}$ for some large enough constant $C$, then with probability at least $1-4/p-4/(pq)$, the initial estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0=\mathrm{a}\mathrm{r}\mathrm{g}{\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathbf{B}}\mathrm{t}\mathrm{r}[{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{YY}+{\mathbf{B}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XX}\mathbf{B}-2{\mathbf{B}}^{\mathrm{\top }}{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{XY}]+{\lambda }_B\parallel \mathbf{B}{\parallel }_1$ satisfies

$$\begin{array}{l}{\parallel {\widehat{B}}_0-B^{*}\parallel }_F\lesssim \sqrt{q{s}_B}{\parallel {\widehat{\mathrm{\Sigma }}}_{XY}-{\widehat{\mathrm{\Sigma }}}_{XX}{B}^{*}\parallel }_{\infty }\\ \lesssim {\parallel B^{*}\parallel }_{L_1}\sqrt{\frac{q{s}_B\log (pq)}{\min (n_{XX},n_{XY})}}.\end{array}$$

Cai et al. (2013) showed that when there is no missing data and the true coefficient ${\mathbf{B}}^{\mathrm{*}}$ is exactly sparse, their estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_{Cai}$ has the convergence rate of ${\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_{Cai}-{\mathbf{B}}^{\mathrm{*}}\parallel }_F=O_p(N_p\sqrt{q{s}_B\mathrm{l}\mathrm{o}\mathrm{g}(pq)/n})$, where $n$ is the sample size of the data and $N_p$ is the upper bound of ${\parallel {\mathbf{\Sigma }}_{XX}^{-1}\parallel }_{L_{\mathrm{\infty }}}$. When there is no missing data, our initial estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0$ has the convergence rate of $\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0-{\mathbf{B}}^{\mathrm{*}}{\parallel }_F=O_p({\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}\sqrt{q{s}_B\mathrm{l}\mathrm{o}\mathrm{g}(pq)/n})$. If we assume ${\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}\asymp {\parallel {\mathbf{\Sigma }}_{XX}^{-1}\parallel }_{L_{\mathrm{\infty }}}$, the convergence rate of ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0$ is the same as that of ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_{Cai}$. When the data are block-wise missing, and we only use complete samples to estimate ${\mathbf{B}}^{\mathrm{*}}$, we will have ${\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0-{\mathbf{B}}^{\mathrm{*}}\parallel }_F=O_p({\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}\sqrt{q{s}_B\mathrm{l}\mathrm{o}\mathrm{g}(pq)/n_{\text{complete}}})$, which can be much slower than the rate in Lemma 3.2 as $n_{\text{complete}}$ is typically much smaller than $n_{XX}$ and $n_{XY}$ for block-wise missing data.

For the single-response regression with block-wise missing data, the result in Lemma 3.2 is the same as Theorem 2 in Yu et al. (2020) and the estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0$ performs well when the dimension of $\mathbf{Y}$ is small. But when the dimension of $\mathbf{Y}$ becomes large, the estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0$ may perform poorly.

The following lemma describes consistency of our initial estimator ${\stackrel{\mathbf{ˆ}}{\mathbf{C}}}_0$.

Lemma 3.3.

Suppose Conditions (A1)-(A4) hold, $1-{\alpha }_1=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_X}),1-{\alpha }_2=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_{XX}})$, $1-{\alpha }_3=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}pq/n_{XY}})$. If we choose ${\lambda }_{C_0}=C{\parallel {\mathbf{C}}^{\mathrm{*}}\parallel }_2^2{\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}({\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}+s_B\sqrt{q}){(\mathrm{l}\mathrm{o}\mathrm{g}(pq)/\mathrm{m}\mathrm{i}\mathrm{n}(n_{XX},n_{XY}))}^{1/2}$ for a large enough $C$, it holds with probability at least $1-4/p-4/(pq)-4/q$ that

$$\begin{gathered} {\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{C}}}_0-{\mathbf{C}}^{\mathrm{*}}\parallel }_F\lesssim \sqrt{s_C}{\parallel {\mathbf{C}}^{\mathrm{*}}\parallel }_2^2{\parallel {\mathbf{\Sigma }}_{ϵ}-{\stackrel{\mathbf{ˆ}}{\mathbf{C}}}_0^{-1}\parallel }_{\mathrm{\infty }} \\ \lesssim {\parallel {\mathbf{C}}^{\mathrm{*}}\parallel }_2^2{\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}({\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}+s_B\sqrt{q})\sqrt{\frac{s_C\log (pq)}{\mathrm{m}\mathrm{i}\mathrm{n}(n_{XX},n_{XY})}}. \end{gathered}$$

There are two terms in the estimation error bound of ${\stackrel{\mathbf{ˆ}}{\mathbf{C}}}_0$. The first term ${\parallel {\mathbf{C}}^{\mathrm{*}}\parallel }_2^2{\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}^2\sqrt{\frac{s_C\mathrm{l}\mathrm{o}\mathrm{g}(pq)}{\mathrm{m}\mathrm{i}\mathrm{n}(n_{XX},n_{XY})}}$ comes from the error induced by using incomplete observations to estimate ${\mathbf{\Sigma }}_{XX}$ and ${\mathbf{\Sigma }}_{XY}$. The second term ${\parallel {\mathbf{C}}^{\mathrm{*}}\parallel }_2^2{\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}{s}_B\sqrt{\frac{s_{C}q\mathrm{l}\mathrm{o}\mathrm{g}(pq)}{\mathrm{m}\mathrm{i}\mathrm{n}(n_{XX},n_{XY})}}$ comes from the estimation error of ${\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0$.

We next derive the convergence rates of $\stackrel{\mathbf{ˆ}}{\mathbf{B}}$ and $\stackrel{\mathbf{ˆ}}{\mathbf{C}}$. The convergence rates are related to $n_{XX/Y}$ and $n_{XY/X}$, which are fractions of $n_{XX}$ and $n_{XY}$ respectively. Hence, we let $n_{XX/Y}\asymp n_{XX}^{T_1}$ and $n_{XY/X}\asymp n_{XY}^{T_2}$ with ${\tau }_1,{\tau }_2\in \{-\mathrm{\infty }\}\cup [\mathrm{0,1}]$. When the responses are complete while the covariates have missing entries, $n_{XX/Y}=0$ and ${\tau }_1=-\mathrm{\infty }$, $n_{XY/X}>0$ and ${\tau }_2\in [\mathrm{0,1}]$. When the covariates are complete while the responses have missing entries, $n_{XY/X}=0$ and ${\tau }_2=-\mathrm{\infty }$, $n_{XX/Y}>0$ and ${\tau }_1\in [\mathrm{0,1}]$. When both the responses and covaraites are complete, $n_{XX/Y}=n_{XY/X}=0$ and ${\tau }_1={\tau }_2=-\mathrm{\infty }$. Theorem 3.4 below establishes the consistency of proposed estimators $\stackrel{\mathbf{ˆ}}{\mathbf{B}}$ and $\stackrel{\mathbf{ˆ}}{\mathbf{C}}$ in (2.5).

Theorem 3.4.

Suppose Conditions (A1)–(A4) hold, $1-{\alpha }_1=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_X}),1-{\alpha }_2=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_{XX}})$, $1-{\alpha }_3=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}(pq)/n_{XY}})$. If we choose ${\lambda }_B$ and ${\lambda }_C$ satisfying ${\lambda }_B=C((\mathrm{l}\mathrm{o}\mathrm{g}p{)}^{1/2}/\mathrm{m}\mathrm{i}\mathrm{n}(n_{XX}^{1-{\tau }_1/2},n_{XY}^{1-{\tau }_2/2}){\parallel {\mathbf{B}}^{\mathrm{*}}{\mathbf{C}}^{\mathrm{*}}\parallel }_{L_1}+\{{(\mathrm{l}\mathrm{o}\mathrm{g}(pq)/n_{XY}\}}^{1/2})$ and ${\lambda }_C=C{\parallel {\mathbf{C}}^{\mathrm{*}}\parallel }_2^2\left[{\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}^2+s_B{\parallel {\mathbf{B}}^{\mathrm{*}}{\mathbf{C}}^{\mathrm{*}}\parallel }_{L_1}/\mathrm{m}\mathrm{i}\mathrm{n}\left(n_{XX}^{1/2-{\tau }_1/2},n_{XY}^{1/2-{\tau }_2/2}\right)\right]{(\mathrm{l}\mathrm{o}\mathrm{g}(pq)/\mathrm{m}\mathrm{i}\mathrm{n}(n_{XX},n_{XY}))}^{1/2}$ for a large enough $C$, then it holds with probability at least $1-4/p-4/(pq)-4/q$ that

$${\parallel \widehat{\text{B}}-{\text{B}}^{*}\parallel }_F\lesssim \sqrt{s_B}\left(\frac{{\parallel {\text{B}}^{*}{\text{C}}^{*}\parallel }_{L_1}{(\log (pq))}^{1/2}}{\min (n_{XX}^{1-{\tau }_1/2},n_{XY}^{1-{\tau }_2/2})}+{\{\frac{\log (pq)}{n_{XY}}\}}^{1/2}\right),$$

$${\parallel \widehat{\text{C}}-{\text{C}}^{*}\parallel }_F\lesssim \sqrt{s_C}{\parallel {\text{C}}^{*}\parallel }_2^2\left(\frac{s_B{\parallel {\text{B}}^{*}{\text{C}}^{*}\parallel }_{L_1}{(\log (pq))}^{1/2}}{\min \left(n_{XX}^{1-{\tau }_1/2},n_{XY}^{1-{\tau }_2/2}\right)}+\frac{{\parallel {\text{B}}^{*}\parallel }_{L_1}^2{(\log (pq))}^{1/2}}{\min \left(n_{XX}^{1/2},n_{XY}^{1/2}\right)}\right)$$

$${\parallel \widehat{\text{B}}-{\text{B}}^{*}\parallel }_1\lesssim s_B\left(\frac{{\parallel {\text{B}}^{*}{\text{C}}^{*}\parallel }_{L_1}{(\log (pq))}^{1/2}}{\min \left(n_{XX}^{1-{\tau }_1/2},n_{XY}^{1-{\tau }_2/2}\right)}+{\left\{\frac{\log (pq)}{n_{XY}}\right\}}^{1/2}\right),$$

$${\parallel \widehat{\text{C}}-{\text{C}}^{*}\parallel }_1\lesssim s_C{\parallel {\text{C}}^{*}\parallel }_2^2\left(\frac{s_B{\parallel {\text{B}}^{*}{\text{C}}^{*}\parallel }_{L_1}{(\log (pq))}^{1/2}}{\min \left(n_{XX}^{1-{\tau }_1/2},n_{XY}^{1-{\tau }_2/2}\right)}+\frac{{\parallel {\text{B}}^{*}\parallel }_{L_1}^2{(\log (pq))}^{1/2}}{\min \left(n_{XX}^{1/2},n_{XY}^{1/2}\right)}\right).$$

Next, we discuss some direct implications of Theorem 3.4. First, we show that our estimators are at least as good as the initial estimators under some conditions. Since ${\tau }_1,{\tau }_2\le 1$ as $n_{jkl}^{XX/Y}\le n_{jk}^{XX}$ and $n_{jkl}^{XY/X}\le n_{jk}^{XY}$, the convergence rate of ${\parallel \stackrel{\mathbf{ˆ}}{\mathbf{B}}-{\mathbf{B}}^{\mathrm{*}}\parallel }_F$ is no slower than $O_p(\mathrm{m}\mathrm{a}\mathrm{x}({\parallel {\mathbf{B}}^{\mathrm{*}}{\mathbf{C}}^{\mathrm{*}}\parallel }_{L_1},1))\sqrt{s_{\text{B}}\mathrm{l}\mathrm{o}\mathrm{g}(pq)/\mathrm{m}\mathrm{i}\mathrm{n}(n_{XX},n_{XY})})$. Similarly, the convergence rate of ${\parallel \stackrel{\mathbf{ˆ}}{\mathbf{C}}-{\mathbf{C}}^{\mathrm{*}}\parallel }_F$ is no slower than $O_p(\sqrt{s_{\text{C}}}{\parallel {\mathbf{C}}^{\mathrm{*}}\parallel }_2^2({\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}^2+s_{\text{B}}{\parallel {\mathbf{B}}^{\mathrm{*}}{\mathbf{C}}^{\mathrm{*}}\parallel }_{L_1})\sqrt{\frac{\mathrm{l}\mathrm{o}\mathrm{g}(pq)}{\mathrm{m}\mathrm{i}\mathrm{n}(n_{XX},n_{XY})}})$. Here the two slowest convergence rates are achieved when ${\tau }_1={\tau }_2=1$. If we assume ${\parallel {\mathbf{B}}^{\mathrm{*}}{\mathbf{C}}^{\mathrm{*}}\parallel }_{L_1}=O({\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}\sqrt{q})$, the upper bounds of ${\parallel \stackrel{\mathbf{ˆ}}{\mathbf{B}}-{\mathbf{B}}^{\mathrm{*}}\parallel }_F$ and ${\parallel \stackrel{\mathbf{ˆ}}{\mathbf{C}}-{\mathbf{C}}^{\mathrm{*}}\parallel }_F$ are at least as tight as ${\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0-{\mathbf{B}}^{\mathrm{*}}\parallel }_F$ and ${\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{C}}}_0-{\mathbf{C}}^{\mathrm{*}}\parallel }_F$.

On the other hand, if ${\parallel {\mathbf{B}}^{\mathrm{*}}{\mathbf{C}}^{\mathrm{*}}\parallel }_{L_1}=o({\parallel {\mathbf{B}}^{\mathrm{*}}\parallel }_{L_1}\sqrt{q})$ or $\mathrm{m}\mathrm{a}\mathrm{x}({\tau }_1,{\tau }_2)<1$ and ${\parallel {\mathbf{B}}^{\mathrm{*}}{\mathbf{C}}^{\mathrm{*}}\parallel }_{L_1}^2=o(\mathrm{m}\mathrm{i}\mathrm{n}(n_{XX}^{1/2-{\tau }_1/2},n_{XY}^{1/2-{\tau }_2/2}))$, the upper bounds of ${\parallel \stackrel{\mathbf{ˆ}}{\mathbf{B}}-{\mathbf{B}}^{\mathrm{*}}\parallel }_F$ and ${\parallel \stackrel{\mathbf{ˆ}}{\mathbf{C}}-{\mathbf{C}}^{\mathrm{*}}\parallel }_F$ are strictly tighter than that of ${\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{B}}}_0-{\mathbf{B}}^{\mathrm{*}}\parallel }_F$ and ${\parallel {\stackrel{\mathbf{ˆ}}{\mathbf{C}}}_0-{\mathbf{C}}^{\mathrm{*}}\parallel }_F$. One example is when $\mathrm{v}\mathrm{a}\mathrm{r}({ϵ}_j)>\frac{1}{\sqrt{q}}$ for all $j\le q$ and $cov({ϵ}_j,{ϵ}_k)=0$ for $j\ne k$. Another example is when $n_{XX/Y}=o(n_{XX}),n_{XY/X}=o(n_{XY})$, and ${\parallel {\mathbf{B}}^{\mathrm{*}}{\mathbf{C}}^{\mathrm{*}}\parallel }_{L_1}^2=o(\mathrm{m}\mathrm{i}\mathrm{n}(n_{XX}^{1/2-{\tau }_1/2},n_{XY}^{1/2-{\tau }_2/2}))$.

When $\mathbf{Y}$ is complete while $\mathbf{X}$ has missing entries, ${\tau }_1=-\mathrm{\infty }$ and ${\tau }_2\in [\mathrm{0,1}]$. Then convergence rate of $\stackrel{\mathbf{ˆ}}{\mathbf{B}}$ in Theorem 3.4 becomes

$${\parallel \stackrel{\mathbf{ˆ}}{\mathbf{B}}-{\mathbf{B}}^{\mathrm{*}}\parallel }_F\lesssim \sqrt{s_B}(\frac{{\parallel {\mathbf{B}}^{\mathrm{*}}{\mathbf{C}}^{\mathrm{*}}\parallel }_{L_1}(\mathrm{l}\mathrm{o}\mathrm{g}(pq){)}^{1/2}}{n_{XY}^{1-{\tau }_2/2}}+{\left\{\frac{\mathrm{l}\mathrm{o}\mathrm{g}(pq)}{n_{XY}}\right\}}^{1/2}).$$

When $\mathbf{X}$ are complete while $\mathbf{Y}$ have missing entries, ${\tau }_2=-\mathrm{\infty }$ and ${\tau }_1\in [\mathrm{0,1}]$. In this case, we can set ${\alpha }_1={\alpha }_2=1$ and have

$${\parallel \stackrel{\mathbf{ˆ}}{\mathbf{B}}-{\mathbf{B}}^{\mathrm{*}}\parallel }_F\lesssim \sqrt{s_B}\left(\frac{{\parallel {\mathbf{B}}^{\mathrm{*}}{\mathbf{C}}^{\mathrm{*}}\parallel }_{L_1}(\mathrm{l}\mathrm{o}\mathrm{g}(pq){)}^{1/2}}{n_{XX}^{1-{\tau }_1/2}}+{\left\{\frac{\mathrm{l}\mathrm{o}\mathrm{g}(pq)}{n_{XY}}\right\}}^{1/2}\right).$$

When both $\mathbf{X}$ and $\mathbf{Y}$ are complete, ${\tau }_1={\tau }_2=-\mathrm{\infty }$. In this case, we can set ${\alpha }_1={\alpha }_2={\alpha }_3=1$ and have

$${\parallel \stackrel{\mathbf{ˆ}}{\mathbf{B}}-{\mathbf{B}}^{\mathrm{*}}\parallel }_F\lesssim \sqrt{s_B\mathrm{l}\mathrm{o}\mathrm{g}(pq)/n},$$ (3.4)

where $n$ is the sample size. The error bound in (3.4) is the minimax rate of the ${\ell }_1$-penalized estimator as shown in Raskutti et al. (2011).

In Theorem 3.5 below, we show that our proposed method is model selection consistent.

Theorem 3.5.

Assume that Conditions (A1)-(A5) hold. Suppose $1-{\alpha }_1=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_X})$, $1-{\alpha }_2=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_{XX}})$, $1-{\alpha }_3=O(\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}(pq)/n_{XY}})$. If ${(\mathrm{l}\mathrm{o}\mathrm{g}(pq)/n_{XY})}^{\frac{1}{2}-{\gamma }_2}/{\lambda }_B=o(1)$, ${\lambda }_B{\parallel {({(C^{\ast }\otimes {\mathrm{\Sigma }}_{XX})}_{S_B{S}_B})}^{-1}\parallel }_{L_{\infty }}/{\min }_{j\in S_B}\left|{\beta }_j^{\ast }\right|=o(1)$, $s_B{\parallel {({(C^{\ast }\otimes {\mathrm{\Sigma }}_{XX})}_{S_B{S}_B})}^{-1}\parallel }_{L_{\infty }}{(\log p/n_{XX})}^{\frac{1}{2}-{\gamma }_2}=o(1)$, and ${s_B(\mathrm{l}\mathrm{o}\mathrm{g}p/n_{XX})}^{\frac{1}{2}-{\gamma }_1-{\gamma }_2}/{\lambda }_B=o(1)$, then with probability at least $1-4/p-4/(pq)-4/q$, there exists a solution $\stackrel{\mathbf{ˆ}}{\mathbf{B}}$ to (2.5) such that $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(\stackrel{\mathbf{ˆ}}{\mathbf{B}})=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}({\mathbf{B}}^{\mathrm{*}})$.

4. Numerical study

In this section, we examine the performance of our proposed method (Multi-DISCOM) in terms of ${\mathbf{\Sigma }}_{ϵ}$, the signal-to-noise ratio, and the distribution of the error $\mathit{ϵ}$ using numerical studies. We compare the efficiency of our proposed method with that of the following methods: (1) the complete lasso, which separately applies the lasso to each response using only samples with complete observations (both $X$ and $Y$ have no missing values); (2) the imputed lasso, which separately applies the lasso to each response using all samples, where missing data are imputed using the soft-thresholded SVD method; (3) the MBI, which separately applies the MBI (Xue and Qu, 2021) to each response using all samples, and the missing data are imputed using multiple block-wise imputation; (4) DISCOM, which separately applies the DISCOM method (Yu et al., 2020) to each response; and (5) the imputed-MRCE, which runs the MRCE (Rothman et al., 2010) using all samples, with missing data imputed using the soft-thresholded SVD method.

In all examples, we set $q=4$ and ${\mathbf{x}}_i={(x_{i1},\dots ,x_{ip})}^{\mathrm{\top }}~N(0,\mathbf{\Sigma })$, with ${\sigma }_{jt}={0.6}^{|j-t|}$. The ith row of the coefficient matrix ${\mathbf{B}}^{\mathrm{*}}$ is (1,1.5, 1, 1.5), for $i=1,p_1+1,p_1+p_2+1$, and zero otherwise. The response $\mathbf{Y}$ has entries missing completely at random, with the missing proportion 0.01.

For each example, the data are generated from three modalities, with dimensions $p_1$, $p_2$, and $p_3$, respectively. The training data set contains $n_1$ samples with complete observations, $n_2$ samples from the third modality, $n_3$ samples from the first and third modalities, and $n_4$ samples from the first modality. The tuning data set contains 75 samples with complete observations, and the testing data set includes 300 samples with complete observations. For each method, we train our model with different tuning parameters on the training data set. Then we choose the optimal tuning parameter by minimizing the mean squared error (MSE) on the tuning data set.

For each example, we repeat the simulation 50 times. To evaluate the selection performance of the algorithm, we use the false-positive rate (FPR) and false-negative rate (FNR) as criteria: $\mathrm{F}\mathrm{P}\mathrm{R}=\mathrm{F}\mathrm{P}/(\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{N})$ and $\mathrm{F}\mathrm{N}\mathrm{R}=\mathrm{F}\mathrm{N}/(\mathrm{F}\mathrm{N}+\mathrm{T}\mathrm{P})$, where FN represents the number of coefficients wrongly detected as zero, TN is the number of coefficients correctly detected as zero, TP are is the number of coefficients correctly detected as nonzero and FP is the number of coefficients wrongly detected as nonzero. Furthermore, to evaluate the accuracy of our estimators, we use the MSE on the testing data set and the ${\ell }_2$-distance ${\parallel \stackrel{\mathbf{ˆ}}{\mathbf{B}}-{\mathbf{B}}^{\mathrm{*}}\parallel }_F$ as criteria.

In Example 1, we examine our method related to ${\mathbf{\Sigma }}_{ϵ}$. Let $n_1=n_2=n_3=n_4=30$ and $p_1=p_2=p_3=30$. We set the error ${\mathit{ϵ}}_i=({ϵ}_{i1},\dots ,{ϵ}_{iq})~N(0,{\mathbf{\Sigma }}_{ϵ})$, with ${\mathbf{\Sigma }}_{ϵ}=3{\mathbf{I}}_2\otimes \left(\begin{array}{ll}1& \rho \\ \rho & 1\end{array}\right)$. We choose $\rho$ between −0.4 and 0.4.

In Example 2, we examine the performance of our method related to the signal-to-noise ratio. Let $n_1=n_2=n_3=n_4=30$ and $p_1=p_2=p_3=30$. We set the error ${\mathit{ϵ}}_i=({ϵ}_{i1},\dots ,{ϵ}_{iq})~N(0,{\mathbf{\Sigma }}_{ϵ})$, with ${\mathbf{\Sigma }}_{ϵ}=\alpha {\mathbf{I}}_2\otimes \left(\begin{array}{cc}1& -0.4\\ -0.4& 1\end{array}\right)$, and choose $\alpha$ between one and five.

In Example 3, we examine the robustness of our method when the error follows a heavy-tailed distribution. Let $n_1=n_2=n_3=n_4=30$ and $p_1=p_2=p_3=30$. We set the error ${ϵ}_i=({ϵ}_{i1},\dots ,{ϵ}_{iq})~t_{10}(0,{\mathbf{\Sigma }}_{ϵ})$, where ${\mathbf{\Sigma }}_{ϵ}=3{\mathbf{I}}_2\otimes (\begin{array}{cc}1& -0.4\\ -0.4& 1\end{array})$, and $t_{\nu }(0,{\mathbf{\Sigma }}_{ϵ})$ refers to Student’s $t$ distribution with location vector 0 and scale matrix ${\mathbf{\Sigma }}_{ϵ}$.

To demonstrate the results, we focus on Example 1. We report the results of the other examples in the Supplementary Material.

The results in Table 1 indicate that the multi-DISCOM method delivers the best performance in all settings. Specifically, the multi-DISCOM method produces a smaller MSE and estimation errors than those of the other methods in all settings, especially when there are large correlations between the responses. In addition, the lasso method with imputed data may deliver worse selection performance, possibly because of the randomness in the imputation of the block-missing data. The results in Table 4 in the Supplementary Material indicate that the multi-DISCOM method has a greater advantage when the signal-to-noise ratio is small. When the ratio is smaller, the noise has a stronger effect on $\mathbf{Y}$, and hence considering the precision matrix is more more helpful for our estimation.

Table 1:

Performance comparison for the methods in Example 1 with different ρ. The values in parentheses are the standard errors of the measures.

	Method	${\parallel \widehat{\mathbf{B}}-{\mathbf{B}}^{*}\parallel }_F$	MSE	FPR	FNR
ρ = −0.4	Lasso	1.51(0.06)	3.70(0.06)	0.09(0.02)	0.00(0.00)
	Imputed-Lasso	1.73(0.06)	3.57(0.06)	0.11(0.01)	0.00(0.00)
	MBI	2.10(0.08)	4.26(0.09)	0.12(0.02)	0.11(0.03)
	DISCOM	1.44(0.04)	3.56(0.06)	0.05(0.00)	0.05(0.01)
	Imputed-MRCE	1.53(0.05)	3.72(0.08)	0.17(0.03)	0.08(0.02)
	Multi-DISCOM	1.40(0.04)	3.39(0.08)	0.02(0.01)	0.09(0.02)
ρ = 0.4	Lasso	1.55(0.06)	3.77(0.06)	0.11(0.02)	0.00(0.00)
	Imputed-Lasso	1.75(0.06)	3.61(0.06)	0.13(0.01)	0.00(0.00)
	MBI	2.14(0.08)	4.30(0.09)	0.13(0.02)	0.11(0.03)
	DISCOM	1.46(0.04)	3.59(0.06)	0.06(0.00)	0.05(0.01)
	Imputed-MRCE	1.54(0.05)	3.73(0.08)	0.19(0.03)	0.09(0.02)
	Multi-DISCOM	1.43(0.04)	3.44(0.08)	0.04(0.01)	0.07(0.02)

5. Application to the ADNI study

We apply the multi-DISCOM method to data from the ADNI study (Mueller et al., 2005), and compare it with several existing approaches. A primary goal of this analysis is to identify biological markers and neuropsychological assessments to measure the progression of mild cognitive impairment (MCI) and early AD. We are interested in predicting the mini mental-state examination (MMSE), ADAS1, and ADAS2, which are common diagnotic scores for AD. The data processing steps are summarized in the Supplementary Material.

After data processing, we have 93 features from MRI, 93 features from PET, and five features from CSF. There are 805 subjects in total, including 199 subjects with complete MRI, PET, and CSF features, 197 subjects with MRI and PET features only, 201 subjects with MRI and CSF features only, and 208 subjects with MRI features only.

In our analysis, we divide the data into training, tuning, and testing sets. The training set consists of all subjects with incomplete observations and 40 randomly selected subjects with complete features. The tuning set consists of another 40 randomly selected subjects with complete observations. The testing set contains the remaining 119 subjects with complete observations. We train our model using different tuning parameters on the training set, choosing the tuning parameter that minimizes the MSE on the tuning set. The testing set is used to evaluate the methods. We use all methods shown in the simulation study to predict the MMSE score. For each method, the analysis is repeated 30 times using different partitions of the data. In addition to the sum of the MSE of all three responses, we compare the MSEs for each response (${\mathrm{M}\mathrm{S}\mathrm{E}}_{MMSE},{\mathrm{M}\mathrm{S}\mathrm{E}}_{ADAS1}$, and ${\mathrm{M}\mathrm{S}\mathrm{E}}_{ADAS2}$) as criteria. We also compare the number of features selected by each method.

As shown in Table 2, the multi-DISCOM method outperforms all other methods. The DISCOM method has a similar overall MSE to that of the multi-DISCOM method, but worse ${\mathrm{M}\mathrm{S}\mathrm{E}}_{ADAS1}$ and ${\mathrm{M}\mathrm{S}\mathrm{E}}_{ADAS2}$. One possible reason for this is that ADAS1 and ADAS2 are highly correlated, which means considering the precision matrix can help. Because there are 208 subjects with MRI features only, the MBI method may not impute those 208 subjects accurately. As a result, the MBI method may not perform well in this case.

Table 2:

Performance comparison for the ADNI data.

Method	Overall MSE	MSEMMSE	MSEADAS1	MSEADAS2	# of Selected Features
Lasso	93.37(3.82)	5.31(0.19)	29.84(1.35)	58.23(2.40)	54.20
Imputed-Lasso	80.40(1.62)	4.54(0.12)	25.80(0.51)	50.07(1.15)	165.00
MBI	91.84(3.02)	5.13(0.14)	28.43(1.17)	58.29(2.16)	59.87
DISCOM	67.47(1.33)	4.26(0.11)	21.76(0.51)	41.45(0.86)	72.87
Imputed-MRCE	67.41(2.02)	4.29(0.10)	21.61(0.65)	41.50(1.33)	218.50
Multi-DISCOM	65.82(1.21)	4.22(0.12)	21.18(0.46)	40.41(0.80)	89.67

With regard to model selection, both the DISCOM method and the multi-DISCOM method deliver relatively simple models. Figure 2 shows the selection frequency of the 191 features when predicting ADAS1. The selection frequency of each feature is defined as the number of times of it is selected in the 30 replications. As shown in Figure 2, for our method, some features are often selected, and many other features are rarely selected. Thus our method delivers robust model selection. However, the features selected by the imputed lasso method vary across replications. One possible reason for the unstable performance in terms of model selection is the randomness in the imputation of the block-missing data. Hippocampus formation left (69th region) and amygdale right (83th feature) are frequently selected by our method, and have been shown to be highly correlated with AD and MCI (Jack et al., 1999; Misra et al., 2009; Zhang and Shen, 2012); however, the DISCOM method rarely selects these features.

Figure 2:

Selection frequency of 191 features for prediction of ADAS1 score.

6. Conclusion

We have proposed a joint estimation method in a penalized framework with an entry-wise ℓ₁-regularization using block-missing multi-modal predictors. We first estimate the covariance matrix of the predictors using a linear combination of the estimates of the variance of each predictor, the estimates of the intra-modality covariance matrix, and the cross-modality covariance matrix. The proposed estimator of the covariance matrix can be positive semidefinite and more accurate than the sample covariance matrix. In the second step, we use the estimated covariance matrix and a penalized estimator to deliver a sparse estimate of the coefficients in the optimal linear prediction. We also establish the theory for the estimation and feature selection consistency. Extensive simulation studies indicate that our method exhibits promising performance in terms of estimation, prediction, and model selection for block-missing multi-modal data. Finally, we apply the multi-DISCOM method to the ADNI data set, showing that our model has good prediction power and meaningful interpretation.