Low-rank regression models for multiple binary responses and their applications to cancer cell-line encyclopedia data

Abstract

In this paper, we study high-dimensional multivariate logistic regression models in which a common set of covariates is used to predict multiple binary outcomes simultaneously. Our work is primarily motivated from many biomedical studies with correlated multiple responses such as the cancer cell-line encyclopedia project. We assume that the underlying regression coefficient matrix is simultaneously low-rank and row-wise sparse. We propose an intuitively appealing selection and estimation framework based on marginal model likelihood, and we develop an efficient computational algorithm for inference. We establish a novel high-dimensional theory for this nonlinear multivariate regression. Our theory is general, allowing for potential correlations between the binary responses. We propose a new type of nuclear norm penalty using the smooth clipped absolute deviation, filling the gap in the related non-convex penalization literature. We theoretically demonstrate that the proposed approach improves estimation accuracy by considering multiple responses jointly through the proposed estimator when the underlying coefficient matrix is low-rank and row-wise sparse. In particular, we establish the non-asymptotic error bounds, and both rank and row support consistency of the proposed method. Moreover, we develop a consistent rule to simultaneously select the rank and row dimension of the coefficient matrix. Furthermore, we extend the proposed methods and theory to a joint Ising model, which accounts for the dependence relationships. In our analysis of both simulated data and the cancer cell line encyclopedia data, the proposed methods outperform the existing methods in better predicting responses.

1. Introduction

Logistic regression is the standard approach to analyzing the relationship between a binary response variable and a set of explanatory variables. For high-dimensional data, where the number of covariates often exceeds the sample size, a number of penalized methods (Lee and Silvapulle, 1988; Cessie and Houwelingen, 1992; Meier et al., 2008; Li et al., 2015) have been developed to estimate the model parameters. These methods have been applied to analyze biomedical data, such as cancer classification and gene interaction detection; see Zhu and Hastie (2004), Park and Hastie (2008), Hastie et al. (2009), and Hung and Wang (2012) for extensions.

Although most of the proposed methods consider one binary response, multiple binary outcomes need to be predicted by a common set of predictors in an increasing number of applications (Chen and Huang, 2012; Martin et al., 2021). For multiple binary responses arising from many applications such as the cancer cell line data analysis, where the treatment responses from multiple drugs are measured from each cell line, standard logistic regression can be applied by combining multiple binary outcomes into a single categorical response (Greenlund et al., 2005; Hayes et al., 2006). The main limitation of this approach is that it may hide the potential outcome-specific covariate effects, leading to inefficient estimates. Another approach is to fit separate logistic models for each response (Koziol-McLai et al., 2001; Cumyn et al., 2009). Although the effects of the covariates on each response can be distinguished, this approach does not consider the potential relationship among multiple responses. When analyzing the Cancer Cell Line Encyclopedia (CCLE) (Garnett et al., 2012) data, which is presented in detail in Section 4.1, if the binary outcomes of multiple drugs are considered on the same cell line, they are likely to be correlated within a cell line (Fitzmaurice et al., 1995; France and Velanovich, 2009; Carpenter, 2010). Because multiple drug responses are available for each individual cancer cell line, there could be an information gain when jointly considering all the drug responses for the same cell line. Thus, one could consider multiple response logistic regression models that exploit information on the common gene expression and the multiple drug response data derived from each cell line. Despite this potential advantage, multiple binary response regression based on a reduced set of predictors has not been rigorously investigated in the literature.

To address the limitations of the existing methods for multiple binary responses, we establish a high-dimensional multivariate logistic regression to estimate the coefficient matrix by imposing a structural assumption on this matrix. For the analysis of the CCLE data, it could be reasonable to assume that the coefficient matrix has a certain structure, e.g., specific genes could have similar effects on multiple responses. Because the dimension of the CCLE data is large because of the large number of genes as well as the number of drugs tested, standard statistical inference may be unstable. It is important to develop methods that are more efficient in dealing with a matrix parameter estimation. In this study, we assume that the underlying coefficient matrix is both row-wise sparse and low-rank to capture the potential similarities among the effects of different drugs to better predict treatment responses.

Such low-rank and row-wise sparse assumptions are commonly used in a wide variety of matrix-related estimation problems to reduce the effective number of model parameters and to achieve better statistical performance. Among numerous approaches for low-rank matrix estimation, nuclear norm regularization is one popular approach including multitask regression (Yuan et al., 2007; Bach, 2008; Rohde and Tsybakov, 2011; Negahban and Wainwright, 2011; Chen et al., 2013; She and Chen, 2017; Cho and Park, 2022), matrix completion (Candés and Recht, 2009; Keshavan et al., 2010; Koltchinskii et al., 2011; Negahban and Wainwright, 2012), and collaborative filtering (Srebro et al., 2005). The nuclear norm is the sum of the singular values of a matrix; so nuclear norm regularization encourages many zeros in singular values for the estimated matrix, i.e., it leads to a low-rank estimate. Chen et al. (2013) proposed an adaptive nuclear norm penalization approach in high dimensional multivariate linear regression.

Another line of work, called reduced-rank regression (RRR) (Izenman, 1975; Reinsel and Velu, 1998), is based on a restriction imposed on the rank of the regression coefficient matrix; it has proven to be effective for the prediction of multiple responses from common predictors. As for the estimation of this rank-deficient coefficient matrix parameter in RRR, the minimax optimal estimation has been studied in the literature (Bunea et al., 2011; Giraud, 2011; Bunea et al., 2012; She, 2017; Bing and Wegkamp, 2019; She and Tran, 2019). Bunea et al. (2011) proposed a rank selection criterion for selecting the optimal reduced rank estimator of the coefficient matrix in multivariate response regression models. She (2017) proposed selective RRR for constructing optimal explanatory factors. She and Tran (2019) developed a predictive information criterion to achieve the minimax optimality when the coefficient matrix parameter is both reduced rank and row-wise sparse. For the variable selection methods in RRR, see Turlach et al. (2005); Chun and Keles (2010); Peng et al. (2010). Most related studies including the cited papers above have focused on linear regression type problems with continuous multivariate responses. For multivariate categorical responses, the class of vector generalized linear models has been widely used and examined only in computational and applied studies, e.g., Yee (2015).

In this paper, we establish high-dimensional multivariate logistic regression and general asymptotic theory with low-rank and row-wise sparse assumptions. This facilitates suitable statistical estimation and inferences of the multiple binary responses such as in the CCLE data application. To the best of our knowledge, this study is one of the first to develop rigorous multivariate regression theory for discrete responses such as binary variables. We propose new non-convex penalized estimation methods under both marginal and joint model set-ups, and we conduct novel theoretical analyses under weak dependencies of the binary responses. The proposed penalization uses the sums of the Smoothly Clipped Absolute Deviation (SCAD) (Fan and Li, 2001) transformations of singular values and Euclidean norm of rows, respectively. The first part, i.e., the sum of the SCADs of the singular values, is interpreted as a non-convex variant of the nuclear norm penalization. Despite the remarkable popularity for a parsimonious model selection, such non-convex nuclear norm penalizations via SCAD have been not studied in the high-dimensional matrix literature. For an overview of the non-convex penalization research, see Fan and Lv (2010) and references therein. It is worthy noting that most existing literature on non-convex penalties analyzes coordinate-separable penalty functions satisfying $R(\beta )=\sum _{j}R({\beta }_j)$, e.g., see Loh and Wainwright (2014, 2017). In contrast, our penalty functions for rank selection are not coordinate-separable, which poses challenges in developing theoretical properties. By addressing theoretical and computational issues due to the non-convexity of the nuclear penalization, we prove that the proposed optimization has a unique stationary point so that it avoids the potential multiple local minima problem in non-convex optimizations. That is, the global minimizer $\widehat{B}$ is indeed the unique stationary point of the proposed non-convex problem in multivariate logistic regression models.

We establish high dimensional statistical theory of the unique stationary point $\widehat{B}$. Specifically, we show that $\widehat{B}$ enjoys a tight estimation error bound in terms of an operator norm with a rate of ${\Vert \widehat{B}-B^{\ast }\Vert }_{op}\lesssim {\{\max (r_S,q)/n\}}^{1/2}$, where $S\subseteq \{1{,}^{\dots },p\}$ represents an index set of non-zero rows of the matrix $B^{\ast },r_S=\text{rank}(X_{.,S})\le s=|S|$, and q is the number of responses. Additionally, we can achieve a true rank $r\le \min (p,q)$ of the underlying coefficient matrix $B^{\ast }\in {ℝ}^{p\times q}$ and the true row support set S at the same time via the developed method. Moreover, we propose a new data-adaptive selection criterion to choose the rank and row-dimension of a coefficient matrix simultaneously. We prove that it provides an estimate with rank consistency as well as support consistency of rows. Compared with the theoretical analysis in the existing multivariate regression literature in the high dimensional setting (Chen and Huang, 2012; Bunea et al., 2012; She and Chen, 2017; Bing and Wegkamp, 2019; She and Tran, 2019; Zou et al., 2021), our results are new in that we consider discrete multiple responses, i.e., multivariate logistic regression, and non-convex version of the nuclear norm penalization, and we develop a new data-adaptive choice for consistent simultaneous selection of the rank and non-zero rows of a matrix parameter. For detailed comparisons to the existing literature, we refer to Sections 2.2.1 and 3.4. Further, we develop an alternating direction method of multipliers (ADMM) (Boyd et al., 2011) algorithm in our implementation to solve the proposed constrained non-convex optimization.

The remainder of this paper is organized as follows. In Section 2, we introduce the proposed marginal and joint models, respectively, to predict multiple binary responses simultaneously, and develop statistical procedures and theory. In Section 3, we perform simulation studies to evaluate the finite sample performance of the developed methods. In Section 4, we apply the developed methods to the CCLE data to illustrate the advantage of our method over the existing methods. All the technical proofs are deferred to the Supplementary materials.

2. Model and Method

2.1. Marginal Logistic Regression Models

We consider a high-dimensional multivariate marginal logistic regression, where for units i = 1, …, n with a covariate vector $x_i\in {ℝ}^p$, there exist q binary responses $y_i^k\in \{-1,1\}$ for $k=1,\mathrm{\dots },q$ satisfying

$$\text{logit}(p_i^k)=\log \left(\frac{p_i^k}{1-p_i^k}\right)=x_i^T{({\beta }^{\ast })}^k(i=1,\dots ,n;k=1,\dots ,q),$$ (1)

where $p_i^k=\text{pr}(y_i^k=1|x_i)$ is the conditional probability of $y_i^k=1$ given x_i, and both dimensions p and q are allowed to increase with n. Such datasets are widely encountered in an increasing number of biomedical studies. For example, for the CCLE data, the gene expression levels x_i and the treatment responses $y_i^k$ corresponding to drugs k = 1, …, 24, were measured for each of the 482 cell lines i = 1, …, 482. As the 24 treatment outcomes are likely to be correlated within a cell line, we need to simultaneously consider a logistic modeling of multiple responses to exploit information on the common gene expression. We also allow dependence between responses, i.e., $y_i^k$ for k = 1, …, q may satisfy $\mathrm{cov}(y_i^k,y_i^{k^{\prime }}|x_i)\ne 0$ for $k\ne k^{\prime }$ given x_i.

With multiple responses, the model parameters can be written in matrix-form, namely $B^{\ast }=[{({\beta }^{\ast })}^1,{({\beta }^{\ast })}^2,\dots ,{({\beta }^{\ast })}^q]\in {ℝ}^{p\times q}$, and we impose both low-rank and row-wise sparse constraints on this high dimensional matrix B*. That is, we assume that the row support set of B* is $S=\{1,\dots ,s\}$ with $s\ll p$, and $\text{rank}(B^{\ast })=r\le \min (s,q)$. These are plausible in the CCLE data application: some responses (drugs) may be similar, i.e., their associations with covariates (gene expression levels) are similar, and only a few variables (genes) are associated with responses (drugs), i.e., only the first s rows of B* can have non-zero values. The low-rank assumption can also be interpreted as that the underlying coefficients ${({\beta }^{\ast })}^k$ ’s lie inside a lower dimensional space (Rohde and Tsybakov, 2011; Bunea et al., 2012; Chen and Ye, 2014). In our theoretical analyses, the number of covariates p, the true row model size s, and the number of responses q are allowed to increase with the sample size n.

For the matrix notation, define the binary response matrix $Y=[y^1,\dots ,y^q]\in {ℝ}^{n\times q}$ and the covariate matrix $X={[x_1^T,\dots ,x_n^T]}^T\in {ℝ}^{n\times p}$. Let $\ell ({\beta }^k|x_i,y_i^k)=-\log \text{pr}(y_i^k|x_i,{\beta }^k)=\log \{1+\exp (-y_i^k{x}_i^T{\beta }^k)\}(i=1,\dots ,n;k=1,\dots ,q)$ be the negative log-likelihood function for conditional distribution $y_i^k|x_i$ of each drug response y_k given covariates x_i. Following Meier et al. (2008), we assume that the probability $p_i^k$ in Model (1) satisfies

$${ϵ}_n\le p_i^k\le 1-{ϵ}_n(i=1,\dots ,n;k=1,\dots ,q)$$ (2)

for some ${ϵ}_n\in (0,0.5)$ so that we can consider the settings where the probability $p_i^k$ approaches to zero or one by allowing ${ϵ}_n\to 0$. Note that (2) is equivalent to $|\text{logit}(p_i^k)|\le -\text{logit}({ϵ}_n)$ or ${\Vert X{B}^{\ast }\Vert }_{\max }\le -\text{logit}({ϵ}_n)$ by (1) where $\Vert A{\Vert }_{\max }={\max }_{i,j}|A_{i,j}|$ for a matrix A. Because ${\Vert X{B}^{\ast }\Vert }_{\max }\le \Vert X{\Vert }_{\max }\sum _j{\Vert B_{j,\cdot }^{\ast }\Vert }_2$, where $B_j^{\ast },\cdot$ is the jth row of B*, the inequality $\sum _j{\Vert B_{j,.}^{\ast }\Vert }_2\le -\text{logit}({ϵ}_n)/\Vert X{\Vert }_{\max }$ is a sufficient condition for (2). Motivated by this, we consider the following feasible set: $\text{Ω}≔\{B=[{\beta }^1,\dots ,{\beta }^q]\in {ℝ}^{p\times q}:\sum _j{\Vert B_{j,.}\Vert }_2\le b\}$ for some b > 0.

One could estimate the coefficient matrix B* by using the likelihood principle, i.e., minimizing the negative marginal likelihood $\sum _{i=1}^n\sum _{k=1}^q\ell ({\beta }^k|x_i,y_i^k)$, subject to low-rank and row-sparse space constraints on B, i.e., $B\in \{[{\beta }^1,\dots ,{\beta }^q]\in \text{Ω}:\text{rank}(B)\le \tilde{r},B_{{\tilde{s}}^c,.}=0\}$, where $\tilde{r}\le \min \{p,q\}$ and $\tilde{S}\subseteq \{1,2,\dots ,p\}$ are pre-specified. Denote this estimator, that is, the constrained Maximum Likelihood Estimator (MLE) by ${\tilde{B}}_{\tilde{r},\tilde{S}}$. The negative logistic likelihood $\sum _{i=1}^n\sum _{k=1}^q\ell ({\beta }^k|x_i,y_i^k)$ is obtained under the assumption that the multiple binary variables $y_i^k,k=1,\dots ,q$ are independent. It is also used as the marginal negative likelihood for the models allowing for correlated multiple binary responses such as in the CCLE data application. Usually, it is quite complicated to model the dependence relationships between variables, in particular in high dimension as in our CCLE application. If the (conditional) marginal distribution $y^k|x$ for k = 1, …, q is of major interest, the marginal likelihood may be conveniently used, which has been employed in various problems with dependent variables (Wilson, 1989; Welsh et al., 2002; Jentsch et al., 2021; Lee and Park, 2021). Refer to Section 2.4 for the joint model setting of the multiple responses involving the dependence structures.

Because it is computationally infeasible to choose a rank $\tilde{r}$ and row-wise space $\tilde{S}$ together, especially for high dimension, we develop a novel penalized approach for the simultaneous choice of a matrix rank and row space, and coefficient matrix estimation in this work. More specifically, we consider the following optimization with penalty functions $P_{{\lambda }_1}(\cdot )$ and $P_{{\lambda }_2}(\cdot )$ :

$$\widehat{B}={\text{argmin}}_{B\in \text{Ω}}\frac{1}{n}\sum _{i=1}^n\sum _{k=1}^q\ell ({\beta }^k|x_i,y_i^k)+\sum _{l=1}^{p\wedge q}{P}_{{\lambda }_1}\{{\sigma }_l(B)\}+\sum _{j=1}^p{P}_{{\lambda }_2}({\Vert B_{j,.}\Vert }_2),$$ (3)

where ${\lambda }_1,{\lambda }_2>0$ are regularization parameters, ${\sigma }_j(B)$ is the jth largest singular value of B, and $P_{{\lambda }_k}(\cdot )$’s are SCAD functions (Fan and Li, 2001) with regularization parameters ${\lambda }_1,{\lambda }_2,a>0$ :

$$P_{\lambda }(x)=\{\begin{array}{cc}\lambda |x|\hfill & \text{if}\left|x\right|\le \lambda \hfill \\ (2a\lambda \left|x\right|-x^2-{\lambda }^2)/\{2(a-1)\}\hfill & \text{if}\lambda <\left|x\right|\le a\lambda \hfill \\ {\lambda }^2(a+1)/2\hfill & \text{otherwise}.\hfill \end{array}$$

We set a = 3.7 as suggested by Fan and Li (2001); Tang et al. (2022), which is known to be optimal based on cross-validated empirical studies (Loh and Wainwright, 2014). In (3), two different penalty terms are included. The first one, i.e., $\underset{l=1}{\overset{p\wedge q}{\Sigma }}{P}_{{\lambda }_1}\{{\sigma }_l(B)\}$, aims to obtain low-rank estimation. The second one, i.e., $\underset{j=1}{\overset{p}{\Sigma }}{P}_{{\lambda }_2}({\Vert B_{j,.}\Vert }_2)$, aims to obtain row-wise sparse estimation. Then, S and r are estimated by the row support set and rank of the estimate $\widehat{B}$, i.e.,

$$\widehat{S}=\{j:{\widehat{B}}_{j,\cdot }\ne 0\},\widehat{r}=\text{rank}(\widehat{B})={\text{arg max}}_k\left\{{\sigma }_k(\widehat{B})>0\right\}.$$

Note that the proposed optimization (3) is non-convex due to the non-convex SCAD penalty terms, which can involve multiple local minimum points. In Subsection 2.2, we prove that the proposed optimization (3) actually has a unique stationary point under some conditions, i.e., a global minimum point is a unique stationary point. We also derive the estimation error bound of the proposed estimator (3) and show the rank consistency and row-selection consistency for the proposed estimator.

2.2. Assumptions and Results

Throughout the theoretical considerations, let $\text{vec}(\cdot ):{\text{R}}^{p\times q}\to {\text{R}}^{pq}$ be the transformation of a matrix into a vector, by stacking its columns into a column vector, and $\text{mat}(\cdot ):{\text{R}}^{pq}\to {\text{R}}^{p\times q}$ be its inverse operation. For a function f, let ∇f denote a gradient or subgradient, if it exists. Let $L_n(B)=\underset{i=1}{\overset{n}{\Sigma }}\underset{k=1}{\overset{q}{\Sigma }}\ell ({\beta }^k|x_i,y_i^k))/n$ for $B=[{\beta }^1,\dots ,{\beta }^q]\in {ℝ}^{p\times q}$. We can view $L_n(B)$ as $L_n(\mathrm{vec}(B))$. Further, we let $\nabla L_n(B)=\text{mat}(\nabla L_n(\text{vec}(B)))$, which is a p × q matrix. Define a function $\psi (t)=\log \{1+\exp (t)\}$. Thus, ${\psi }^{\prime }(t)=\exp (t)/\{1+\exp (t)\}$ and ${\psi }^{\prime \prime }(t)=\exp (t)/{\{1+\exp (t)\}}^2$. Throughout the paper, $\Vert A{\Vert }_{op}$ represents an operator norm of a matrix A, i.e., the maximum singular value of A, and ${\lambda }_{\text{max}}(B)$ is a maximum eigenvalue of a symmetric matrix B. We write $a\lesssim b$ if $a\le C_{1}b$ for some positive constant C₁, write $a\asymp b$ if $a\lesssim b$ and $b\lesssim a$, and use $a\vee b$ and $a\wedge b$ to denote $\max (a,b)$ and $\min (a,b)$, respectively. For a matrix A, let $\text{row-support}(A)=\{j:A_{j,\cdot }\ne 0\}$ be the index set of non-zero rows of the matrix A, and $\text{support}(A)=\{(j,k):A_{j,k}\ne 0\}$ be the index set of non-zero entries of the matrix A. For a vector x, let $\Vert x{\Vert }_0$ be the number of non-zero elements in the vector x. For a matrix $M=(M_{ij})$, let $\Vert M{\Vert }_F={[\sum _i\sum _j{M}_{ij}^2]}^{1/2},\Vert M{\Vert }_1=\sum _i\sum _j|M_{ij}|$, and $\Vert M{\Vert }_{\max }={\text{max}}_{i,j}|M_{ij}|$. For a matrix A and index sets of rows and columns, say S₁ and S₂, let A_S1,⋅ and A_⋅,S2 be the sub-matrices of A with rows in S₁ and columns in S₂, respectively. Let $r_X\le n\wedge p$ and $r_S\le n\wedge s$ be the rank of X and X_⋅,s respectively. A stationary point $\tilde{B}=[{\tilde{\beta }}^1,\dots ,{\tilde{\beta }}^q]$ of the optimization (3) is any point in the feasible set Ω satisfying

$$\langle \nabla L_n(\tilde{B})+\nabla \sum _{l=1}^{n\wedge q}{P}_{{\lambda }_1}\{{\sigma }_l(\tilde{B})\}+\nabla \sum _{j=1}^p{P}_{{\lambda }_2}({\Vert {\tilde{B}}_{j,\cdot }\Vert }_2),B-\tilde{B}\rangle \ge 0$$

for all B ∈ Ω, where $\langle A,B\rangle =\mathrm{trace}(A^{T}B)$. Let $B_{S,\cdot }^{\ast }=UD{V}^T$ be the compact singular value decomposition of $B_{S,\cdot }^{\ast }$, i.e., $U\in {ℝ}^{s\times r},V\in {ℝ}^{q\times r}$, and $U^{T}U=V^{T}V=I$. Let denote $U_{\perp }\in {ℝ}^{s\times (s-r)}$ and $V_{\perp }\in {ℝ}^{q\times (q-r)}$ any orthogonal complements of U and V, respectively. Let $Q_s^{\ast }=\text{diag}({(Q_S^{\ast })}_1,\dots ,{(Q_S^{\ast })}_q)\in {ℝ}^{sq\times sq}$, where ${(Q_S^{\ast })}_k=\sum _{i=1}^n{\psi }^{\prime \prime }\{x_i^T{({\beta }^{\ast })}^k\}{x}_{i,S}{x}_{i,S}^T/n$ represents the Fisher information matrix related to the kth response.

We make the following assumptions to facilitate our technical derivations.

Assumption 1. [On the likelihood] It holds that for some positive constants α_k and γ_k,

$$\langle \nabla L_n(B+\text{Δ})-\nabla L_n(B),\text{Δ}\rangle \ge \{\begin{array}{l}{\alpha }_1\Vert \text{Δ}{\Vert }_F^2-{\gamma }_1(\log p/n)\Vert \text{Δ}{\Vert }_1^2\text{for all}\Vert \text{Δ}{\Vert }_{\text{F}}\le 1;\hfill \\ {\alpha }_2\Vert \text{Δ}{\Vert }_F-{\gamma }_2{(\log p/n)}^{1/2}\Vert \text{Δ}{\Vert }_1\text{for all}\Vert \text{Δ}{\Vert }_{\text{F}}>1,\hfill \end{array}$$

where ${\alpha }_1>9/\{8(a-1)\}$.

Assumption 2. [On the correlations between the drug responses] Denote the conditional covariance matrix of the multiple responses $y_i={(y_i^1,\dots ,y_i^q)}^{\top }$ given $X_i=x$ by ${\text{Σ}}_x≔\mathrm{Cov}(y|x)\in {ℝ}^{q\times q}$. Then, the largest eigenvalue of Σ_x is uniformly bounded, i.e., $\underset{x\in {ℝ}^p}{\sup }{\lambda }_{\max }({\text{Σ}}_x)\le q_y$ for some $0<q_y<\infty$.

Assumption 3. [On the coefficient matrix] The underlying coefficient matrix B* satisfies $\sum _j{\Vert B_{j,\cdot }^{\ast }\Vert }_2\le 0.5b,{\min }_{j\in S}{\Vert B_{j,\cdot }^{\ast }\Vert }_{2,\cdot }/{\{(q\vee r_s)\log (q\vee r_X)/n\}}^{1/2}\to \infty$, and ${\sigma }_r(B^{\ast })/[(q\wedge s){\{(q\vee r_s)/n\}}^{1/2}]\to \infty$ and, where $b=o({\{n/(q\log p)\}}^{1/2})$.

Assumption 4. [On the eigenvalues] The eigenvalues of $X_{\cdot ,S}^T{X}_{\cdot ,S}/n$ is bounded from above by some constant ${\text{Γ}}_{\max }(S)$, i.e., ${\text{Γ}}_{\max }(S)≔\underset{\delta \in {ℝ}^s,\delta \ne 0}{\sup }{\delta }^T{X}_{\cdot ,S}^T{X}_{\cdot ,S}\delta /(n\Vert \delta {\Vert }_2^2)<\infty$.

Assumption 5. [On the covariates] For $j\in S^c$, it holds that $\underset{\delta \in {ℝ}^s}{\sup }s\sum _{i=1}^n{\left\{X_{ij}{X}_{iS}\delta \right\}}^2/(n\Vert \delta {\Vert }_2^2)=O(1)$.

Assumption 1 is referred as a restricted strong convexity condition that is also imposed in Raskutti et al. (2010); Loh and Wainwright (2014, 2017), where α_k represents a curvature parameter and γ_k represents a tolerance parameter for k = 1, 2. It is related to the lower and upper bounds of the conditional probabilities $p_i^k$, and it holds when the $p_i^k$ are uniformly bounded away from zero and one. Assumption 2 is interpreted as dependence conditions imposed on the covariance matrix $y_i^k$. Note that $q_y\le q$ by simple calculations. If the responses $y_i^k$ for k = 1, …, q are uncorrelated given x_i, then it is satisfied, e.g., by taking q_y = 1 / 4. More generally, it holds if the covariance matrix Σ_x is sparse in the sense that there exists a function $c:\mathbb{Z}\to ℝ$ satisfying $\sum _k|c(k)|\le q_y<\infty$ and $Cov(y_i^k,y_i^{k^{\prime }}|x)\le c(k-k^{\prime })$ for any $i\in \{1,\dots ,n\},k,k^{\prime }\in \{1,\dots ,q\}$, and x. Thus, one easily sees that Assumption 2 is fulfilled for weakly dependent binary variables such as m-dependent and autoregressive logistic processes (e.g., Sim, 1993). Assumption 3 guarantees the feasibility of the underlying B*, and gives beta-min type conditions imposed on the non-zero singular value and non-zero rows of B*, respectively. A similar condition is also imposed in Bunea et al. (2012); Chen et al. (2013); She and Tran (2019). Assumption 4 is an eigenvalue condition, which is commonly assumed in the literature of high-dimensional models (Zheng et al., 2015). Assumption 5 is also imposed in Zheng et al. (2015) with a slight modification to prove the oracle property of their estimate, which restricts the correlations between relevant covariates and irrelevant covariates.

Note that the proposed method enjoys the rank consistency without the following irrepresentable condition: $\Vert \widetilde{\text{Γ}}{\Vert }_{op}<1$, where $\mathrm{vec}(\widetilde{\text{Γ}})={\{{(V_{\perp }\otimes U_{\perp })}^T{(Q_S^{\ast })}^{-1}(V_{\perp }\otimes U_{\perp })\}}^{-1}{(V_{\perp }\otimes U_{\perp })}^T{(Q_S^{\ast })}^{-1}(V\otimes U)\mathrm{vec}(I))$. The rank consistency based on the conventional nuclear norm penalization is known to require an additional irrepresentable condition, which can be found in Bach (2008) and Kong et al. (2020). Our results show that the proposed method based on SCAD-type penalty for singular value will guarantee the rank consistency even in the absence of the irrepresentable condition, illustrating the theoretical advantage of the SCAD in the rank selection context. Similar advantage of SCAD over ℓ₁ norm penalization (Lasso) has also been investigated in high dimensional univariate regression (e.g., Loh and Wainwright, 2014, 2017). The following theorem presents the theoretical properties of the proposed method.

Theorem 1. Suppose that Assumptions 1-5 hold, $sq\log p=o(n)$, and ${\lambda }_{\max }(Q_S^{\ast })/{\lambda }_{\min }(Q_S^{\ast })=o(\sqrt{n/sq})$. If ${\lambda }_1$ and ${\lambda }_2$ satisfy ${\lambda }_1\asymp {\{(q\vee r_s)/n\}}^{1/2}$ and ${\lambda }_2\asymp {\{q\log (q\vee p)/n\}}^{1/2}$, then the program (3) has a unique stationary point $\widehat{B}$ and it satisfies

$$\begin{gathered} {\Vert \widehat{B}-B^{\ast }\Vert }_F=O_p\left[{\{r(q\vee r_S)/n\}}^{1/2}\right],{\Vert \widehat{B}-B^{\ast }\Vert }_{op}=O_p\left[{\{(q\vee r_S)/n\}}^{1/2}\right], \\ P(\widehat{S}=S)\to 1,P(\widehat{r}=r)\to 1\text{as}n\to \infty . \end{gathered}$$

Theorem 1 shows that if the penalty parameters are appropriately chosen, then the proposed method achieves the same convergence rate as the oracle estimator ${\tilde{B}}_{r,S}$, that is, the constrained MLE using the true rank r and row-support set S, in terms of the Frobenius norm $\Vert \cdot {\Vert }_F$. See Theorem S1 in the Supplementary materials for theoretical properties of the constrained MLE. Moreover, it implies that it leads to consistent simultaneous selections of the rank $\widehat{r}$ and row space $\widehat{S}$ of relevant covariates. One also sees that although the proposed optimization (3) is non-convex, there exists a unique solution $\widehat{B}$, more specifically, the proposed estimate $\widehat{B}$ is the unique stationary point of the non-convex problem (3). It is worth noting that the growth rate conditions $sq\log p=o(n)$ and ${\lambda }_{\max }(Q_S^{\ast })/{\lambda }_{\min }(Q_S^{\ast })=o(\sqrt{n/sq})$ are only required to prove the uniqueness of the stationary point as in Kim and Kwon (2012); Loh and Wainwright (2014, 2017) and rank consistency as in Bach (2008); Kong et al. (2020). A similar condition can also be found in Loh and Wainwright (2017) in high dimensional univariate regression contexts. Without this growth rate condition, the local optimality is only guaranteed for $\widehat{B}$. The growth rate conditions have been assumed in many published studies about the global optimality of nonconvex penalized methods in the other settings (Fan and Li, 2001; Wang et al., 2012; Loh and Wainwright, 2014).

2.2.1. Comparisons with related results

Here, we provide detailed theoretical error rate comparisons to related literature, including the existing multivariate reduced rank regression papers and extensions of some penalizations to our multiple logistic settings. We would like to note that a main goal of our study is to achieve selection consistency (of rank and row support set), which generally needs stronger assumptions than to attain optimal estimation rates.

Even though our main interest, penalization techniques, and model set-ups are different, we consider estimation of XB, i.e., ${\Vert X\widehat{B}-X{B}^{\ast }\Vert }_F$ which is the main focus in some literature (Giraud, 2011; Bunea et al., 2012) in the multivariate response linear regression setting. Bunea et al. (2012) achieved an error rate of ${(sr\log p+qr)}^{1/2}$ and She (2017); She and Tran (2019) achieved an error rate of ${\{(qr+sr+s\log p)\}}^{1/2}$ under the setting that the underlying coefficient B* is low-rank and row-wise sparse. Theorem 1 stated above implies that our estimator $\widehat{B}$ satisfies ${\Vert X\widehat{B}-X{B}^{\ast }\Vert }_F=O_p\left[{\{r(q+r_S)\}}^{1/2}\right]$, which is tighter than the obtained rate in the aforementioned literature because Theorem 1 assumes stronger conditions. If we are interested in estimation consistency instead of selection consistency, a slight modification of the proposed method can lead to ${\Vert X\widehat{B}-X{B}^{\ast }\Vert }_F=O_p\left[{\{(qr+sr+s\log p)\}}^{1/2}\right]$ as in She (2017); She and Tran (2019) under more mild conditions. We refer to Theorem S2 in the Supplementary materials for more details. Under the setting that the underlying coefficient B* is just low-rank, Bunea et al. (2011); Bing and Wegkamp (2019) achieved a minimax error rate of ${[r(q+r_X)]}^{1/2}$. It is worth noting that the obtained error bound ${\Vert X\widehat{B}-X{B}^{\ast }\Vert }_F=O_p\left[{\{r(q+r_s)\}}^{1/2}\right]$ corresponds to this minimax error rate when only relevant covariates in X, i.e., X_,S, are included in the low-rank matrix estimation problem.

Theorem 1 also implies that the proposed method has theoretical advantage than extensions of some existing penalizations to our multivariate response logistic regression setting: regular logistic regression with ℓ₁ penalty (RL) to individual responses; and multiple response logistic regression with group Lasso and nuclear norm penalties (MLL), which are also considered in the simulation analysis as in Section 3. Using the existing theories applied to Lasso, we can prove that RL achieves ${\Vert \widehat{B}-B^{\ast }\Vert }_F=O_p\left[{\{sq/n\}}^{1/2}\right]$. This implies that the proposed method provides a tighter estimation error bound because it always holds that $r(q\vee r_s)\le sq$. Specifically, if the rank r is small, i.e., $r\ll q\wedge s$, the proposed method can provide a substantial rate improvement over RL by considering multiple responses jointly through the low-rank and row-wise sparse structural assumption. Additionally, through modification of theories used in Theorem 1, it can be shown that MLL attains the same rate ${\Vert \widehat{B}-B^{\ast }\Vert }_F=O_p\left[{\{r(q\vee r_S)/n\}}^{1/2}\right]$, but with an additional irrepresentable condition as mentioned in Subsection 2.2. Refer to Section 3 for numerical comparisons with RL and MLL.

2.3. Simultaneous selection of rank and row space

We now propose a data-dependent way to select the regularization parameters λ₁ and λ₂ for the proposed method (3), and prove that the proposed selection criteria jointly choose the underlying rank r and row support set S with probability tending to one. Recently, the generalized information criterion (GIC), including the Bayesian information criterion, has been used to select the tuning parameter of penalized methods for variable selection in high dimensional models, e.g., Chen and Chen (2008), Fan and Tang (2013), Lee et al. (2014), and Zheng et al. (2015). Accordingly, we define an information criterion suitable for our simultaneous choice problem by modifying the GIC. See also Nishii (1984). Let ${\widehat{B}}_{{\lambda }_1,{\lambda }_2}$ be the proposed estimate using penalty parameters ${\lambda }_1$ and ${\lambda }_2$. We propose to select ${\lambda }_1$ and ${\lambda }_2$ as the minimizers of the following information criterion:

$$\mathrm{GIC}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})=2L_n({\widehat{B}}_{{\lambda }_1,{\lambda }_2})+\widehat{r}(\widehat{s}+q)g_n^{(1)}+\widehat{s}q{g}_n^{(2)},$$ (4)

where $\widehat{s}=|\text{row}-\text{support}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})|$ and $\widehat{r}=\text{rank}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})$. Here, we can see that r (s + q) and sq are the degrees of freedom corresponding to the choices of the rank r and the row sparsity s, respectively. There are similar computations of the degrees of freedom in the literature, e.g., Bai and Ng (2002) and Huang et al. (2012).

For selection of the row support set S, we set an upper bound of the size of the row support set, denoted by U(n), with s < U(n) < p, i.e., we only consider B satisfying $|\text{row}-\text{support}(B)|\le U(n)$. Similar assumptions of restricting the model dimension have been widely used in related studies., e.g., Fan and Tang (2013) and Zheng et al. (2015). For d ≤ p, define $r_d={\text{max}}_{A\subseteq \{1,\cdots ,p\}:|A|=d}\text{rank}(X_{\cdot ,A})$. It holds that $r_d\le r_X\wedge d$. Following Zheng et al. (2015), let Γ_max be the uniform upper bound of the eigenvalues of $X_{\cdot ,\tilde{S}}^T{X}_{\cdot ,\tilde{S}}/n$ with $|\tilde{S}|\le U(n)$ i.e., ${\text{Γ}}_{\max }≔\underset{\delta \in {ℝ}^p,\Vert \delta {\Vert }_0\le U(n),\delta \ne 0}{\sup }{\delta }^T{X}^{T}X\delta /(n\Vert \delta {\Vert }^2)<\infty$. To discuss selection consistency of the proposed information criterion (4), we introduce some notations for mis-specified models for (r, S), that is, overfitted and underfitted models. We define an overfitted model when $({\lambda }_1,{\lambda }_2)\in Z_1\cup Z_2$, where

$$\begin{gathered} Z_1≔\{({\lambda }_1,{\lambda }_2):\text{rank}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})>r,\text{row-support}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})\supseteq S,|\text{row-support}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})|\le U(n)\}, \\ {Z}_2≔\{({\lambda }_1,{\lambda }_2):\text{rank}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})\ge r,\text{row-support}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})⊋S,|\text{row-support}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})|\le U(n)\}. \end{gathered}$$

Similarly, we define an under-fitted model when $({\lambda }_1,{\lambda }_2)\in Z_3\cup Z_4$, where

$$\begin{gathered} Z_3≔\{({\lambda }_1,{\lambda }_2):\text{rank}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})<r,|\text{row-support}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})|\le U(n)\}, \\ {Z}_4≔\{({\lambda }_1,{\lambda }_2):S\backslash \text{row-support}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})\ne \varnothing ,|\text{row-support}({\widehat{B}}_{{\lambda }_1,{\lambda }_2})|\le U(n)\}. \end{gathered}$$

Theorem 2 implies that λ₁ and λ₂ chosen from the proposed information criterion provides the estimate with the correct rank r and correct row support set S with high probability.

Theorem 2. Suppose that the conditions in Theorem 1,

$${\min }_{j\in S}{\Vert B_j^{\ast }\Vert }_{2,\cdot }/{(\{U(n)+s+q\}q\log p/n)}^{1/2}\to \infty ,\text{and}{\sigma }_r(B^{\ast })/{\{(r_{U(n)}\vee q)r\log p/n\}}^{1/2}\to \infty \text{hold}.$$

Let ${\widehat{B}}_{{\lambda }_1^{\ast },{\lambda }_2^{\ast }}$ be the proposed estimate using the optimal penalty parameters ${\lambda }_1^{\ast }$ and ${\lambda }_2^{\ast }$ satisfying conditions in Theorem 1. If $g_n^{(1)}\to 0,g_n^{(2)}\to 0,g_n^{(1)}n/\log p\to \infty$, and $g_n^{(2)}n/\log p\to \infty$, we have as $n,p,q\to \infty$, $pr\left\{{\text{min}}_{({\lambda }_1,{\lambda }_2)\in \underset{k=1}{\overset{4}{\cup }}{Z}_k}GIC({\widehat{B}}_{{\lambda }_1,{\lambda }_2})>GIC({\widehat{B}}_{{\lambda }_1^{\ast },{\lambda }_2^{\ast }})\right\}\to 1.$

Theorem 2 suggests that wide ranges of $g_n^{(1)}$ and $g_n^{(1)}$ work in theory. This includes the following specifications as special cases: $g_n^{(1)}=\log (\log n)\log p/n,g_n^{(2)}=\log (\log n)\log p/n$. In this case, the GIC in (4) can be be written as

$$GIC({\widehat{B}}_{{\lambda }_1,{\lambda }_2})=2L_n({\widehat{B}}_{{\lambda }_1,{\lambda }_2})+\frac{\widehat{r}(\widehat{s}+q)\log (\log n)\log p}{n}+\frac{\widehat{s}q\log (\log n)\log p}{n}.$$ (5)

In our simulation and real data examples, we set $g_n^{(1)}=g_n^{(2)}=\log (\log n)\log p/n$ and we observed that these choices perform well.

The beta-min (minimum signal strength) conditions and the restrictions on the rank r and the sparsity parameter s in Theorem 2 are essential for rank and row selection consistency of the GIC-type penalty. Note that She (2017) and She and Tran (2019) proposed a new class of information criteria called PIC, which achieved the minimax optimal error rate without the beta-min conditions. However, they also require the beta-min conditions to guarantee the row support recovery via PIC as in Theorem 5 of She and Tran (2019). Among different types of PIC, we choose the following criterion to compare it with our GIC:

$$PIC({\widehat{B}}_{{\lambda }_1,{\lambda }_2})=2L_n({\widehat{B}}_{{\lambda }_1,{\lambda }_2})+\frac{A_1\widehat{r}(\widehat{s}+q-\widehat{r})}{n}+\frac{A_2\widehat{s}\log (ep/\widehat{s})}{n}$$

for some absolute constants A₁, A₂ > 0, which can be determined by Monte Carlo experiments. Following She and Tran (2019), we set A₁ = 2 and A₂ = 1.8, and we check that this specification works well in our numerical experiments. For comparisons of the proposed GIC and PIC via simulation analyses, see Subsection 3.4.

2.4. Joint Ising models for multiple binary responses

In this subsection, we consider the joint Ising models (Ising, 1925) for multivariate binary data. The models account for the conditional dependency relationships between the outcome variables as well as for the marginal models as in Section 2.1. Following Cheng et al. (2014), we assume a sparse covariate dependent Ising model to study both the conditional dependency within the binary data and its relationship with the additional covariates. Let $p_i^k=P(y_i^k=1|y_i^{-k},x_i)$, where $y_i^{-k}=(y_i^1,\cdots ,y_i^{k-1},y_i^{k+1},\cdots ,y_i^q)$. We assume that the log-odds depend on the covariates through

$$\log \left(\frac{p_i^k}{1-p_i^k}\right)=x_i^T{({\beta }^{\ast })}^k+\sum _{l\ne k}{x}_i^T{({\theta }^{\ast })}^{k,l}{y}_i^l.$$ (6)

Compared to model (1), model (6) includes the additional term $\sum _{l\ne k}{x}_i^T{({\theta }^{\ast })}^{l,k}{y}_i^l$, from which dependencies between responses are considered. Let

$${({\theta }^{\ast })}^k={[{({({\theta }^{\ast })}^{k,1})}^T,\cdots ,{({({\theta }^{\ast })}^{k,k-1})}^T,{({({\theta }^{\ast })}^{k,k+1})}^T,\cdots ,{({({\theta }^{\ast })}^{k,q})}^T]}^T\in {ℝ}^p(q-1),{\text{Θ}}^{\ast }=[{({\theta }^{\ast })}^1,\cdots ,{({\theta }^{\ast })}^q]\in {ℝ}^{p(q-1)\times q}.$$

Specifically, the vector ${({\theta }^{\ast })}^{k,l}$ being zero implies that $y_i^k$ and $y_i^l$ are conditionally independent given x, and $y_i^t$ for $t\ne k,l$, and ${({\theta }^{\ast })}_j^{k,l}=0$ implies that the conditional association between $y_i^k$ and $y_i^l$ does not depend on x_ij (Cheng et al., 2014). We assume that $S=\text{row-support}(B^{\ast })=\{1,\cdots ,s\}$ with $|S|=s\ll p,\text{rank}(B^{\ast })=r\le \min (s,q)$, and $F=\text{support}({\text{Θ}}^{\ast })$ with $|F|=f\ll p(q-1)q.$. Note that a sufficient condition for (2) is ${\Vert X{B}^{\ast }\Vert }_{\max }\le -\text{logit}({ϵ}_n)/2$ and $\sum _{l\ne k}{\Vert X{({\theta }^{\ast })}^{k,l}\Vert }_{\max }\le -\text{logit}({ϵ}_n)/2$ for all $k=1,\cdots ,q$. Because ${\Vert X{B}^{\ast }\Vert }_{\max }\le \Vert X{\Vert }_{\max }\sum _j{\Vert B_{j,\cdot }^{\ast }\Vert }_2$ and ${\Vert X{({\theta }^{\ast })}^{k,l}\Vert }_{\max }\le \Vert X{\Vert }_{\max }{\Vert {({\theta }^{\ast })}^{k,l}\Vert }_1$, satisfying $\sum _j{\Vert B_{j,\cdot }^{\ast }\Vert }_2\le -0.5\text{logit}({ϵ}_n)/\Vert X{\Vert }_{\max }$ and ${\Vert {({\theta }^{\ast })}^k\Vert }_1\le -0.5\text{logit}({ϵ}_n)/\Vert X{\Vert }_{\max }$ for all k is a sufficient condition for (2). Thus, we consider the following feasible set: $\text{Ω}≔\{B\in {ℝ}^{p\times q},\text{Θ}\in {ℝ}^{p(p-1)\times q}:\sum _j{\Vert B_{j,\cdot }\Vert }_2\le b,{\Vert |\text{Θ}|\Vert }_1\le b\}$ for some b > 0, where ${\Vert |\mathrm{\Theta }|\Vert }_1={\max }_k\sum _j|{\text{Θ}}_{jk}|$. Let $\ell ({\beta }^k,{\theta }^k|x_i,y_i^{-k})=-\log \text{pr}(y_i^k|x_i,y_i^{-k})=\log [1+\exp (-y_i^k\{x_i^T{\beta }^k+\sum _{l\ne k}{x}_i^T{\theta }^{k,l}{y}_i^l\})](i=1,\dots ,n;k=1,\dots ,q)$ be the negative log-likelihood function for conditional distribution $y_i^k|x_i,y_i^{-k}$ of each drug response $y_i^k$ given x_i and $y_i^{-k}={(y_i^1,\dots ,y_i^{k-1},y_i^{k+1},\dots ,y_i^q)}^{\top }$. We can consider the following optimization:

$$(\widehat{B},\widehat{\mathrm{\Theta }})={\text{argmin}}_{(B,\text{Θ})\in \text{Ω}}\frac{1}{n}\sum _{i=1}^n\sum _{k=1}^q\ell ({\beta }^k,{\theta }^k|x_i,y_i^{-k})+\sum _{l=1}^{p\wedge q}{P}_{{\lambda }_1}\{{\sigma }_l(B)\}+\sum _{j=1}^p{P}_{{\lambda }_2}({\Vert B_{j,\cdot }\Vert }_2)+\sum _{k=1}^q\sum _{j=1}^{p(q-1)}{P}_{{\lambda }_3}(|{\text{Θ}}_{jk}|),$$ (7)

where ${\lambda }_1,{\lambda }_2,{\lambda }_3>0$ are regularization parameters, and Θ_jk is the (j, k)th component of a matrix Θ. For a penalty function $P_{{\lambda }_j}(\cdot )$, we consider the SCAD functions. For choices of $\lambda =({\lambda }_1,{\lambda }_2,{\lambda }_3)$, we consider the following GIC, which generalizes the GIC in (5):

$$\text{GIC}({\widehat{B}}_{\lambda },{\widehat{\mathrm{\Theta }}}_{\lambda })=2L_n({\widehat{B}}_{\lambda },{\widehat{\mathrm{\Theta }}}_{\lambda })+\frac{\widehat{r}(\widehat{s}+q)\log (\log n)\log p}{n}+\frac{\widehat{s}q\log (\log n)\log p}{n}+\frac{\widehat{f}\log (\log n)\log p}{n},$$

where $\widehat{s},\widehat{r}$, and $\widehat{f}$ are the estimates of s, r, and f obtained from ${\widehat{B}}_{\lambda }$ and ${\widehat{\mathrm{\Theta }}}_{\lambda }$.

The following theorem presents the theoretical properties of the proposed penalization incorporating the Ising models. We make new assumption for the joint Ising models that replace Assumptions 1 in the marginal models. Assumption S1, presented in the Supplementary materials, is the restricted strong convexity condition in the Ising models, which corresponds to Assumption 1 in the proposed marginal models. Let $Q_{11}^{\ast }=\text{diag}(Q_{11,1}^{\ast },\dots ,Q_{11,q}^{\ast })\in {ℝ}^{sq\times sq}$, where $Q_{11,k}^{\ast }=\sum _{i=1}^n{\psi }^{″}\{x_i^T{({\beta }^{\ast })}^k+\sum _{l\ne k}{x}_i^T{({\theta }^{\ast })}^{k,l}{y}_i^l\}{x}_{i,S}{x}_{i,S}^T/n$ represents the Fisher information matrix related to the kth response.

Theorem 3. Suppose Assumptions 2-5 and Assumption S1 hold, $(sq\vee f)\log p=o(n),{\lambda }_{\max }(Q_{11}^{\ast })/{\lambda }_{\min }(Q_{11}^{\ast })=o(\sqrt{n/sq})$ and ${\min }_{(j,k)\in F}|{\text{Θ}}_{jk}^{\ast }|\gtrsim {\{\log (p\vee q)/n\}}^{1/2}$. If ${\lambda }_1,{\lambda }_2$, and ${\lambda }_3$ satisfy ${\lambda }_1\asymp {\{(q\vee r_S)/n\}}^{1/2}$, ${\lambda }_2\asymp {\{q\log (p\vee q)/n\}}^{1/2}$, and ${\lambda }_3\asymp {\{\log (p\vee q)/n\}}^{1/2}$, then the program (7) has a unique stationary point $(\widehat{B},\widehat{\mathrm{\Theta }})$, and it satisfies

$$\begin{gathered} {\Vert \widehat{B}-B^{\ast }\Vert }_F^2+{\Vert \widehat{\mathrm{\Theta }}-{\text{Θ}}^{\ast }\Vert }_F^2=O_p\left[\{r(q\vee r_S)/n\}+f/n\right],{\Vert \widehat{B}-B^{\ast }\Vert }_{op}=O_p\left[{\{(q\vee r_S)/n\}}^{1/2}\right], \\ {\Vert \widehat{\mathrm{\Theta }}-{\text{Θ}}^{\ast }\Vert }_{\max }=O_p{[\log (p\vee q)/n\}}^{1/2}],P(\widehat{S}=S)\to 1,P(\widehat{r}=r)\to 1,P(\widehat{F}=F)\to 1\text{as}n\to \infty . \end{gathered}$$

Theorem 3 shows that if the penalty parameters are appropriately chosen, then the proposed method incorporating the Ising models leads to consistent simultaneous selections of the rank $\widehat{r}$, row space $\widehat{S}$, and support set $\widehat{F}$. Although the optimization (7) is non-convex, the proposed estimate $(\widehat{B},\widehat{\mathrm{\Theta }})$ is the unique stationary point of the non-convex problem (7).

3. Simulation Study

3.1. Methods for Comparison

We conduct simulation studies to investigate the finite sample performance of the proposed method. Specifically, to illustrate the advantages of the proposed estimator in the multivariate logistic regression framework, we compare it to two other logistic regression methods:

Multiple response Logistic regression with SCAD penalties (MLS)

Multiple response Logistic regression with group Lasso and nuclear norm penalties (MLL)

Regular Logistic regression with ℓ₁ penalty (RL).

Here, MLS is the proposed estimate in (3), MLL is the estimate using nuclear norm and group Lasso penalties, i.e., estimate of (3) with a penalty ${\lambda }_1\sum _l{\sigma }_l(B)+{\lambda }_2\sum _j{\Vert B_{j,\cdot }\Vert }_2$, and RL is the logistic regression with Lasso penalty by considering each response separately. We consider a setting where the underlying B* is both low-rank and row-wise sparse. For regularization parameters in MLL and MLS, we choose λ₁ and λ₂ using the proposed generalized information criterion as in Subsection 2.3. We set $U(n)=[n/\log p]$ and $b=-\log \text{it}(0.01)/\Vert X{\Vert }_{\max }$, which satisfies (2) with ${ϵ}_n=0.01$. Note that RL involves only one regularization parameter λ. We choose λ using the Bayesian information criterion. Using this comparison, we can assess the performance of each method with the appropriately chosen regularization parameters.

3.2. Simulation Model

In our simulation study, we consider the multiple response logistic model, where the underlying coefficient matrix $B^{\ast }={[{({\beta }_0^{\ast })}^T,{({\beta }_1^{\ast })}^T,\dots ,{({\beta }_p^{\ast })}^T]}^T$ is both low-rank and row-wise sparse, where ${\beta }_0^{\ast }$ is the intercept coefficient, the first s + 1 rows of B* are non-zero, and the remaining p − s rows of B* are set to zero. We set ${\beta }_0^{\ast }=(0.5,\dots ,0.5)\in {ℝ}^q$, and the non-zero entries in ${[{({\beta }_1^{\ast })}^T,\dots ,{({\beta }_s^{\ast })}^T]}^T$ are generated by the singular value thresholding of ${\tilde{B}}^{\ast }\in {ℝ}^{s\times q}$, such that the rank of B*, except ${\beta }_0^{\ast }$, is r, where each entry in ${\tilde{B}}^{\ast }$ follows the mixture distribution $\frac{1}{2}\text{Uniform}(0,5)+\frac{1}{2}\text{Uniform}(-5,0)$. Specifically, for the singular value decomposition of ${\tilde{B}}^{\ast }=UD{V}^T,U\in {ℝ}^{s\times (s\wedge q)}$ and $V\in {ℝ}^{q\times (s\wedge q)}$ satisfy $U^{T}U=V^{T}V=I$, and $D=\text{diag}(d_1,\dots ,d_{s\wedge q})\in {ℝ}^{(s\wedge q)\times (s\wedge q)}$ is a diagonal matrix whose diagonal components are singular values of ${\tilde{B}}^{\ast }$ sorted in non-increasing order. We set ${[{({\beta }_1^{\ast })}^T,\dots ,{({\beta }_s^{\ast })}^T]}^T=U_r{D}_r{V}_r^T$, where U_r and V_r are the first r columns of U and V, respectively, and D_r is the upper-left r by r sub-matrix of D. The dimensions are chosen from $n\in \{400,800\},p\in \{400,800\},s\in \{10,20\},r\in \{5,10\}$, and q = 25. Note that we consider the datasets $\tilde{X}=[1_n,X]\in {ℝ}^{n\times (p+1)}$ and $Y\in {ℝ}^{n\times q}$, where $P(Y_{ik}=1|\tilde{X})=\text{logit}({\tilde{X}}_{i,\cdot }{B}_{\cdot ,k}^{\ast })$, i.e., in a matrix form $P(Y=1|\tilde{X})=\mathrm{logit}(\tilde{X}{B}^{\ast })\in {ℝ}^{n\times q}$. When generating X, each row of X follows a multivariate normal distribution, $MN(0_p,\text{Σ})$, where $\text{Σ}={({0.5}^{|i-j|})}_{1\le i,j\le p}$. Thus, a total of 16 different simulation cases are considered, with varying data distributions and model parameters.

3.3. Simulation Results

To compare the performance of different methods, we consider the identification accuracy of the non-zero rows and rank, and estimation accuracy for the estimates in terms of the following criteria:

• Relative estimation error ${\Vert \widehat{B}-B^{\ast }\Vert }_F/{\Vert B^{\ast }\Vert }_F$ between the underlying B* and the estimate $\widehat{B}$

• PUS: Percentage of Under-fitted models in terms of the row Support set, i.e., $S\backslash \widehat{S}\ne \varnothing$

• PCS: Percentage of Correctly fitted models in terms of the row Support set, i.e., $\widehat{S}=S$

• POS: Percentage of Over-fitted models in terms of the row Support set, i.e., $S⊊\widehat{S}$

• PCR: Percentage of Correctly fitted models in terms of the Rank, i.e., $\widehat{r}=r$

• Average of the estimated rank $\widehat{r}$

The performance measures are averaged over 100 replications for each specification. It is worth noting that we consider 100 cases in which the true support set and true rank are included in a sequence of candidate models. The proposed information criterion is used to select the tuning parameters for MLS and MLL.

Table S1 in the Supplementary materials displays the average of the relative estimation errors with the corresponding standard deviations within parentheses for the three methods over the 16 different cases. We can see that the proposed method MLS achieves smaller relative estimation errors compared with MLL and RL for most cases. This suggests that MLS with appropriately chosen regularization parameters can improve the estimation accuracy compared to the other two methods when the underlying coefficient matrix is both row-wise sparse and low-rank. Table 1 and Table S2 in the Supplementary materials record the performances of the row support identification for MLS, MLL, and RL, when n = 400 and n = 800, respectively. We can see that RL tends to produce an over-fitted model because RL does not incorporate multiple responses. Overall MLS has the best performance in the row support set estimation accuracy, which demonstrates that the reduced rank-related smoothly clipped absolute deviation penalty does not affect negatively to the sparse row-related smoothly clipped absolute deviation penalty. This also demonstrates that by accumulating information across responses and using SCAD penalties for row support selection, the proposed method may boost the power of correct model selection.

Table 1.

Support set accuracy for the three methods when n = 400

Parameters				MLS			MLL			RL
n	p	s	r	PUS(%)	PCS(%)	POS(%)	PUS(%)	PCS(%)	POS(%)	PUS(%)	PCS(%)	POS(%)
400	400	10	5	19	71	10	31	50	19	29	40	31
			10	17	70	13	28	51	21	27	38	35
		20	5	19	68	13	27	48	25	26	32	42
			10	17	70	13	25	48	27	22	34	44
	800	10	5	27	62	12	40	43	27	38	20	42
			10	28	65	7	37	43	20	36	24	40
		20	5	33	57	10	26	35	19	50	13	37
			10	33	60	7	49	38	13	59	15	26

Table 2 and Table S3 in the Supplementary materials record the performances of the rank estimates for MLS and MLL, when n = 400 and n = 800, respectively. For the average rank, we also record standard deviations within parentheses. Because RL does not involve rank-based regularization, it is not included in this comparison. We can see that MLS gives more accurate rank estimates on average compared with MLL for all cases. MLS outperforms MLL also in correct rank accuracy for all cases. This demonstrates that MLS with appropriately chosen regularization parameters improves the rank accuracy.

Table 2.

Rank estimate accuracy for the two methods when n = 400

Parameters				Mean rank		PCR(%)
n	p	s	r	MLS	MLL	MLS	MLL
400	400	10	5	4.25 (1.32)	9.23 (3.32)	84	40
			10	8.42 (3.20)	18.32 (6.31)	79	31
		20	5	6.02 (1.40)	13.23 (6.33)	80	33
			10	11.44 (2.25)	21.23 (7.83)	78	31
	800	10	5	5.92 (1.88)	12.22 (5.92)	68	17
			10	11.71 (2.53)	23.82 (8.31)	61	14
		20	5	7.72 (3.15)	20.75 (7.78)	64	17
			10	12.78 (4.01)	29.79 (9.41)	65	12

3.4. Comparisons of GIC and PIC

In this subsection, we compare the proposed GIC and the existing PIC in terms of rank and row support selection. For simulation settings, we generate data as in Subsection 3.2, and follow She and Tran (2019) when generating an underlying coefficient matrix B*. Specifically, we set the coefficient matrix $B^{\ast }={[A_0{A}_1^T,0_{q,p-s}]}^T$, where $A_0\in {ℝ}^{q\times r},A_1\in {ℝ}^{s\times r}$, and entries in A₀ and A₁ are i.i.d. standard Gaussian. We consider the following six different simulation settings, with varying model parameters: (a) $n=400,p=200,q=10,s=10,r\in \{3,5\}$; (b) n = 400, p = 400, q = 10, s = 15, r ∈ {3, 5}; (c) n = 400, p = 600, q = 15, s = 20, r ∈ {3, 7}.

Tables S4 and S5 in the Supplementary materials record the comparisons of GIC and PIC in terms of the row support accuracy and rank estimate accuracy, respectively. Overall, both GIC and PIC provide accurate row support accuracy, whereas PIC tends to overfit compared to GIC. Regarding rank estimate, GIC provides accurate results except when (n, p, q, s, r) = (400, 600, 15, 20, 7). In this case, GIC tends to underestimate the underlying rank r, which may be due to the fact that the term log p in GIC causes overpenalizing the rank.

4. Applications to cancer cell line data

4.1. The dataset

With the development of high-throughput technologies in recent decades, various studies have been conducted to build genomic predictors of drug response from large panels of cancer cell lines (Staunton et al., 2001; Barretina et al., 2012; Garnett et al., 2012). The CCLE dataset offers a comprehensive collection of gene expression and anti-cancer drug responses across cell lines (Barretina et al., 2012; Garnett et al., 2012). This dataset contains drug treatment responses for 24 drugs on 482 cancer cell lines, where the transcription profile of each cell line is characterized by the measured expression levels of 18,988 probes. These data have an important role in drug discovery for the screening of drug candidates (Blanco et al., 2018). They are widely used to understand cancer biology and to test the efficacy of novel therapies (Sharma et al., 2010) due to their cost-effectiveness and unlimited replicative nature (Ferreira et al., 2013).

For the cancer cell line data analysis, standard logistic regression can be applied by combining multiple binary outcomes into a single categorical response (Greenlund et al., 2005; Hayes et al., 2006). The main limitation of this approach is that it may hide the potential outcome-specific covariate effects, leading to inefficient estimates. Another approach is to fit separate logistic models for each response (Koziol-McLai et al., 2001; Cumyn et al., 2009). Although the effects of the covariates on each response can be distinguished, this approach does not consider the potential relationships among multiple responses. When analyzing the cancer cell line encyclopedia data, if the binary outcomes of multiple drugs are considered, they are likely to be correlated within the same cell line (Fitzmaurice et al., 1995; France and Velanovich, 2009; Carpenter, 2010). Because multiple drug responses are available for each individual cancer cell line, there could be an information gain when jointly considering all the drug responses for the same cell line. Thus, one could consider multiple response logistic regression models that exploit information on the common gene expression and the multiple drug response data derived from each cell line. Despite this potential advantage, multiple response regression based on a reduced set of predictors has not been investigated for logistic modeling, either theoretically or practically, in the literature.

In this analysis, we consider binary drug responses defined as being either sensitive or resistant between a drug and the treated cell line. We first select the 3,000 probes with the largest variances across the 482 cell lines and then choose 450 probes having the largest correlation coefficients with the IC50 values. Therefore, we consider n = 482 cell lines, p = 450 features, and q = 24 drugs, respectively. We divide the cell lines into two groups based on their continuous growth inhibition values (IC50) such that the cell lines with their IC50 value less than 0.5 are assigned to the “sensitive” group, and the other cell lines are assigned to the “resistant” group.

4.2. Results

For prediction accuracy, we consider the following five methods including the three logistic regression methods considered above, i.e., MLS, MLL, RL, and adding Linear Support Vector Machine (SVM) and Support Vector Machine with a Gaussian kernel (SVMG) using the optimal parameter. To evaluate the prediction accuracy of each method, we perform 10-fold CV. The cancer cell lines are randomly divided into 10 mostly equal-sized subsets, then nine subsets are used as the training set to estimate the model, and the remaining subset is used as the test set to assess the prediction accuracy. The performance of each method is presented by the proportion of the accurately predicted cell lines (PA) and area under the ROC curve (AUC) to investigate the trade-off between the true positive rate and the false positive rate. Figure 1 displays the overall ROC curve, and average values of AUC, and PA with one standard deviation for the five methods. Here, we include all the cancer cell lines and drugs to compute the performance measures. We observe that the proposed multiple logistic regression, i.e., MLS, has higher AUC and PA values than the other methods, suggesting its better performance on average.

Fig. 1.

Performances of the five methods. All cancer cell lines and drugs are included when computing the performance measures.

Figure 2 illustrates the average prediction accuracy for each drug. Note that when computing AUC for each drug, the test data set involving only a single response value for the corresponding drug is not included because the ROC curve and AUC cannot be computed for that case. Based on the AUC, MLS and SVMG have superior predictions compared to the other methods for the majority of drugs. Specifically, MLS, SVMG, and SVM perform best in 13, 7, and 4 of the 24 drugs, respectively. Based on PA, MLS generally outperforms the other methods, suggesting that it may provide superior prediction for the extremely imbalanced test data that are excluded from AUC calculation.

Fig. 2.

Performances of the five classification methods. Responses of each drug across all cancer lines are included when computing performance measures.

Next, we investigate the performance for each pair of drug and cancer type separately. In particular, we consider the following 16 cancer types that include at least 10 cell lines in the CCLE data: Kidney, Liver, Pancreas, Ovary, Lung, Skin, Lymphoid, Intestine, Stomach, Nervous, Oesophagus, Bone, Breast, Endometrium, Ganglia, and Soft. For this study, we only consider PA because for a specific cancer type and drug combination, the test data frequently include a single response value with respect to a drug such that AUC cannot be computed. Figures 3 and S1–S3 in the Supplementary materials display the heatmaps of the average PA values for each pair of cancer type and drug. We observe that MLS achieves greater or similar PA values compared to the other methods in many cases, which suggests the benefit of prediction by jointly combining different drugs.

Fig. 3.

Performances of the five methods for the four selected cancer types, ‘Kidney’, ‘Liver’, ‘Pancreas’, and ‘Ovary’. For each pair of drug and cancer type, the corresponding cancer cell lines and drugs are included when computing performance measures.

We observe that for certain compounds (e.g., 17-AAG, Topotecan, PD-0325901), all the classification methods consistently have smaller PA values across the 16 cancer types, suggesting that the prediction accuracy may depend on the specific compound. Figure S4 displays the standard deviations of the PA values over 16 cancer types. We observe that the standard deviation for ‘17-AAG’, ‘Irinotecan’, and ‘AZD6244’ is near 0.1, whereas the standard deviation of ‘PHA-665752’, ‘Nutlin-3’, and ‘PD-0332991’ is near 0.01 for all the methods. These results imply that the variations of the prediction accuracy across different cancer types depend on the drugs considered.

Among the drugs considered, ‘Lapatinib’ is an orally active drug for breast cancer, and its treatment has been found to induce prevalent resistance (Liu et al., 2009; Wang et al., 2011). We observe that the proposed MLS provides a more accurate prediction for the breast cancer cell lines treated with ‘Lapatinib’. ‘17-AAG’ is another effective compound for breast cancer (Modi et al., 2011). We also observe that MLS provides a higher PA for the breast cancer cell lines for this compound. In addition, ‘PD-0332991’ can also be used for the treatment of ER-positive and HER2-negative breast cancer. In this case, we see that all the methods predict the response well for the breast cancer cell lines.

Regarding drugs for lung cancer treatment, ‘PF2341066’, an anti-cancer drug acting as an ALK (anaplastic lymphoma kinase) and ROS1 (c-ros oncogene 1) inhibitor (Forde and Rudin, 2012; Roberts, 2013), is approved for lung cancer in the US and other countries. ‘TAE684’ is also effective to inhibit cell proliferation and used for the treatment of lung cancer. For the lung cancer cell lines, MLS and SVMG predict the responses of these two compounds well compared with other methods. Regarding other drugs related to the other cancer types, ‘Erlotinib’ is used for both lung cancer and pancreatic cancer. For the lung and pancreatic cancer cell lines, all the methods predict the response of ‘Erlotinib’ well. As studied in Scotlandi et al. (2005), ‘AEW541’ is one of the growth factor-I receptor kinase inhibitors applied to musculoskelet al tumors such as cancer that develops in bone or soft tissue. For the bone and soft cancer cell lines, all the methods have similar prediction accuracy for the response of ‘AEW541’. As demonstrated in Petrucci et al. (2012), ‘LBW242’ can be used for the treatment of ovarian cancer. For the ovary cancer cell lines, we observe that all the methods similarly predict the response of ‘LBW242’. For pancreatic cancer, the administration of ‘AZD6244’ can significantly inhibit tumor growth in the pancreatic tumor xenograft model (Yeh et al., 2007). MLS and SVMG predict the response of ‘AZD6244’ well compared with the other methods for the pancreatic cancer cell lines.

Note that some of the 24 drugs in the CCLE data have been used for multiple cancer types. For example, ‘Topotecan’ is used to treat ovarian cancer when other treatments have failed and is also used to treat lung cancer. For the lung cancer cell lines, MLS predicts the response of ‘Topotecan’ well compared with the other methods. For the ovarian cancer cell lines, SVMG predicts the response of ‘Topotecan’ well; MLS is the second best. ‘Paclitaxel’ is used for the treatment of breast, ovarian, lung, and other types of solid tumor cancers. For the breast cancer cell lines, MLS predicts the response of ‘Paclitaxel’ well; for the lung cancer patients, all the methods predict similarly; however, for the ovarian cancer cell lines, SVM predicts well compared with the other methods. Overall, for the drugs known to be effective for the treatment of certain cancer types, we observe that the proposed MLS predicts the response of the drug well in many cases or similarly compared with the other methods.

Further, we apply the Ising model proposed in Subsection 2.4 to investigate the dependency structures among drugs. Because the prediction accuracy of the proposed Ising model is similar to MLS, we mainly consider conditional associations between drugs. Specifically, for each pair of drugs k and l, we record $\Vert {\widehat{\theta }}^{k,l}\Vert =\sqrt{\sum _{j=1}^p{({\widehat{\theta }}_j^{k,l})}^2}$, which represents the degree of dependencies between drugs k and l, across all the genes. These values are shown as a heatmap in Figure 4, where white colors between two drugs indicate weak associations whereas black colors indicate strong associations.

Fig. 4.

Heatmap of drug dependencies

Specifically, TAE684 and PF2341066 have an estimated dependency of 0.97, which is the highest value. Both TAE684 and PF2341066 are Kinase Inhibitors, their mechanism of action is known to ALK Inhibitor, and were synthesized following published procedures (Galkin et al., 2007; Li et al., 2011). RAF265 and L-685458, both Kinase Inhibitors, have the second highest dependency value of 0.88. Both TAE684 and AEW541 are also Kinase Inhibitors, and they have the third highest dependency 0.85. For ZD-6474, only Erlotinib has significant non-zero value. Both ZD-6474 and Erlotinib are EGFR inhibitors, and that ZD-6474 is only grouped with Erlotinib according to the results in Wei et al. (2019). For PD-0332991, only Nilotinib has a non-zero value. This implies that PD-0332991 may only have an association with Nilotinib, which is consistent with the clustering results in Wei et al. (2019). For PHA-665752, only PF2341066 and Topotecan have non-zero values. Note that both PHA-665752 and PF2341066 are small molecule inhibitors related to c-met (Zillhardt et al., 2010), and PHA-665752 and Topotecan are grouped together in Wei et al. (2019). For Lapatinib, AZD0530 has the highest value 0.68. Note that these drugs are EGFR inhibitors. For AZD0530, Nilotinib has the highest value 0.82. Both Nilotinib and AZD0530 are related to BCR-ABL (Sabitha, 2012). Specifically, Nilotinib inhibits several tyrosine kinases including BCR-ABL, and AZD0530 targets BCR-ABL kinase activity and reduces the leukaemic maintenance by BCR-ABL.

5. Discussion

Although recent years have seen the adoption of SCAD as a popular penalization approach for a more parsimonious and consistent model selection in a wide range of high dimensional settings, there has been little work for rank selection using SCAD in the high-dimensional matrix literature. In this paper, we considered the use of a SCAD-type penalization coupled with information criterion for both rank and row selection consistency. More specifically, we have introduced a novel sparse reduced rank regression with correlated multiple binary responses. We have proposed a new penalized likelihood method using SCAD version of the nuclear norm, developed associated novel high-dimensional theory, and conducted comprehensive data analyses. We found that the SCAD penalty function may also benefit low-rank matrix estimation in other high dimensional settings. Our theoretical work needs more mild conditions rather than strong irrepresentable condition assumed for the ordinary nuclear norm penalization (Bach, 2008; Kong et al., 2020). To account for the dependencies among binary responses, we also developed a novel estimation procedure and theory for an Ising model in the multiple response logistic regression setting.

As matrix responses are commonly encountered in a wide range of applications as also seen in Kong et al. (2020), a natural extension of our work is to develop both low-rank and row-sparse procedures for high-dimensional matrix responses. Moreover, it is also of interest to extend the sure independence screening procedure of Kong et al. (2020) to binary matrix response settings, which may help ultra-high dimensional data analysis.