A Unified Model for Joint Normalization and Differential Gene Expression Detection in RNA-Seq data

Abstract

The RNA-sequencing (RNA-seq) is becoming increasingly popular for quantifying gene expression levels. Since the RNA-seq measurements are relative in nature, between-sample normalization of counts is an essential step in differential expression (DE) analysis. The normalization of existing DE detection algorithms is ad hoc and performed once for all prior to DE detection, which may be suboptimal since ideally normalization should be based on non-DE genes only and thus coupled with DE detection. We propose a unified statistical model for joint normalization and DE detection of log-transformed RNA-seq data. Sample-specific normalization factors are modeled as unknown parameters in the gene-wise linear models and jointly estimated with the regression coefficients. By imposing sparsity-inducing L1 penalty (or mixed L1/L2 penalty for multiple treatment conditions) on the regression coefficients, we formulate the problem as a penalized least-squares regression problem and apply the augmented lagrangian method to solve it. Simulation studies show that the proposed model and algorithms perform better than or comparably to existing methods in terms of detection power and false-positive rate. The performance gain increases with increasingly larger sample size or higher signal to noise ratio, and is more significant when a large proportion of genes are differentially expressed in an asymmetric manner.

1. INTRODUCTION

Ultra high-throughput sequencing of transcriptomes (RNA-seq) is a widely used method for quantifying gene expression levels due to its low cost, high accuracy and wide dynamic range for detection [1]. As of today, modern ultra high-throughput sequencing platforms can generate hundreds of millions of sequencing reads from each biological sample in a single day. RNA-seq also facilitates the detection of novel transcripts [2] and the quantification of transcripts on isoform level [3], [4]. For these reasons, RNA-seq has become the method of choice for assaying transcriptomes [5].

One major limitation of RNA-seq is that it only provides relative measurements of transcript abundances due to difference in library size (i.e., sequencing depth) between samples [6]. Normalization of RNA-seq read counts is required in gene differential expression analysis to correct for such variation between samples. A popular form of between-sample normalization is achieved by scaling raw read counts in each sample by a sample-specific factor related to library size [6], [7]. This include CPM/RPM (counts/reads per million) [8], quantile normalization [9],

[10], upper-quartile normalization [11], trimmed mean of M values [8] and DESeq normalization [12]. Also, commonly used gene expression measures, e.g., TPM (transcript per million) [13], and RPKM/FPKM (reads/fragments per kilo-base of exon per million mapped reads) [1], [2], also correct for difference in gene length within a sample [14] (the so-called within-sample normalization). In particular, the CPM/RPM (counts/reads per million) [8], TPM (transcript per million) [13], and RPKM/FPKM (reads/fragments per kilobase of exon per million mapped reads) [1], [2] for the i-th gene from the j-th sample are respectively defined as

$$\begin{gathered} {\text{cpm}}_{ij}={10}^6\frac{c_{ij}}{N_j} \\ {\text{fpkm}}_{ij}={10}^9\frac{c_{ij}}{{\mathcal{l}}_i\cdot N_j} \\ \text{tpm}{}_{ij}={10}^6\frac{c_{ij}/{\mathcal{l}}_i}{{\sum }_i{c}_{ij}/{\mathcal{l}}_i} \end{gathered}$$ (1)

where c_ij is the observed read count for gene i from the j-th sample, N_j= ∑_ic_ij is the sequencing depth in the j-th sample, and ℓ_i be the length of gene i. In this work we focus on between-sample normalization.

In traditional count-based RNA-seq analysis methods, the read counts for each gene are assumed to follow a Poisson [15] or negative binomial (NB) distribution. One issue with the count-based RNA-seq analysis methods is that their procedures are complicated and contain many ad hoc heuristics. Moreover, the Poisson or NB distributions of counts are mathematically less tractable than the normal distribution [16], [17]. This makes count-based methods difficult to generalize to new data. Moreover, commonly used statistical methods for microarray data analysis, e.g., quality weighting of RNA samples, addition of random noise to generate technical replicates, and gene set test [16] have been designed for normally distributed data and it is unclear whether we can adapt them to count data. Also the presence of outliers is an issue that is not well addressed (addressed in a very ad hoc manner) by existing methods. To handle that, the authors of [16] take the logarithm of the raw count of reads and apply normal distribution-based statistical methods to analyze them. Note that by logarithmic transformation, the dynamic range of the RNA-seq counts is compressed such that the outlier counts are largely transformed into “normal” data. As a result, sophisticated way to detect and discard outliers [18], [19], [20] is not required.

In this paper, like in [16], [17] we work with logtransformed gene expression values and propose a unified statistical model for differential gene expression. Different from [16], [17], we model sample-specific scaling factors for between-sample normalization as unknown parameters and incorporate them into the gene-wise linear models. By imposing the sparsity-inducing penalty (L1 penalty for single treatment factor and mixed L1/L2 penalty for multiple treatment factors) on the regression coefficients and carefully choosing the tuning parameter, the model is able to achieve joint accurate detection of DE genes and between-sample normalization. To fit the model, we first eliminate sample-specific parameters using optimization argumentation to formulate the problem as a penalized linear regression problem, and then solve it with the alternating direction method of multipliers algorithm (ADMM), which is known for its fast convergence to modest accuracy [21]. Regarding the choice of tuning parameter, we theoretically derive the smallest tuning parameter α_max that leads to all-zero solution, and thereby find a proper tuning parameter within [0, α_max].

Note that our work is preceded by [22] which address the differential expression problem in a similar way. The difference is that the model of [22] considers only categorical or qualitative predictor/explanatory variables (treatment conditions). For example, label “0” is assigned to samples from the control group and label “1” to samples from the treatment group. While in our model, the predictor/explanatory variables can take arbitrary numeric values, and is thus a generalization of [22] from discrete to continuous predictorvariable model case. Note that the algorithm in [22] does not apply to the current numeric variable model at hand, because (i) applicability: it requires that multiple samples are present in each group but in the continuous-predictor model the concept of “group” no longer exists, or more precisely, each group is formed by only one sample; (ii) algorithmic complexity: it requires an p-dimensional exhaustive search, where p is the number of treatment conditions. When p > 1 (see Section 4), the algorithm is computationally very expensive.

The remainder of the paper is organized as follows. In Section 2, we formulate the problem in the context of a single treatment factor. In Section 3, we formulate the problem as a penalized simple regression problem and derive efficient ADMM algorithm to solve it, together with the estimation of noise variance and tuning parameter. In Section 4, we extend the simple regression model to multiple linear regression model. Comparison with existing methods is presented in Section 5, followed by discussions in Section 6.

2. DATA MODEL AND PROBLEM FORMULATION

Throughout the paper, the subscript and superscript are used to index the vectors for rows and columns of a matrix, respectively. For example, the i-th row and j-th column vector of a matrix A is denoted as a_i and a^j, respectively. Note that this does not conform to conventional notations where the subscript is used to index the columns of a matrix and the superscript is to index the rows.

2.1. Data model

Suppose there are a total of m genes measured in n samples. Let y_ij, i = 1, 2, …, m, j=1, 2, …, n, be the log-transformed gene expression measurements (a small positive number is usually added before taking logarithm) for the i-th gene from the j-th sample. The following statistical model is assumed

$$y_{ij}={\beta }_{i0}+{\beta }_i{x}_j+d_j+{\epsilon }_{ij},$$ (2)

where β_i0 is the y-intercept for gene i, x_j, j = 1, 2, …, n, is the predictor variable that represents the treatment condition (e.g., drug dosage) for sample j, β_i is the slope or regression coefficient representing log-fold-change of expression levels of gene i with unit change of x_j, d_j is the scaling factor (e.g., log (sequencing depth) or log (library size)) for sample j for between-sample normalization [6], and ${\epsilon }_{ij}~\mathcal{N}\left(0,{\sigma }_i^2\right)$ models the measurement noise. We assume that the error terms ε_ij are uncorrelated with the predictor variable and uncorrelated with each other (across both gene i and sample j).

In (2), we consider a single treatment condition. Extension to models with multiple treatment conditions will be discussed in Section 4.

Our main interest is to detect differentially expressed (DE) genes, i.e., whether β_i is equal to zero. If β_i ≠ 0 gene i is differentially expressed across the n samples; otherwise it is not.

Remark 2.1. Since β_i0 and d_j in (2) respectively model gene-specific factor (e.g., gene length) and sample-specific factor, model (2) is able to work with any log-transformed gene expression measures in the form of

$$y_{ij}=\text{log}\frac{c_{ij}}{{\mathcal{l}}_i\cdot q_j},$$ (3)

where c_ij is the raw counts, ℓ_i is the length of gene i and q_j is the normalization factor of the j-th sample, since ℓ_i and q_j can be absorbed into β_i0 and d_j, respectively. Note that gene expression measures of form c_ij/(ℓ_i q_j) include the raw counts (with ℓ_i= q_{j =}1), measures based on between sample normalization only(ℓ_i=1) [6], and FPKM and TPM which are shown in (1) and involve both between and within-sample normalization.

2.2. Penalized likelihood

The likelihood function based on the measured data is given by

$$L\left({\beta }_0,\beta ,d;y\right)=\prod _{i=1}^m\prod _{j=1}^n\frac{1}{\sqrt{2\pi {\sigma }_i^2}}\text{exp}\left\{-\frac{{\left(y_{ij}-{\beta }_{i0}-{\beta }_i{x}_j-d_j\right)}^2}{2{\sigma }_i^2}\right\},$$ (4)

where

$$\beta =\left(\begin{array}{c}{\beta }_1\\ {\beta }_2\\ ⋮\\ {\beta }_m\end{array}\right).$$

Assume that ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$ are known, maximization of (4) is equivalent to minimizing the negative log-likelihood:

$$\mathcal{l}\left({\beta }_0,\beta ,d\right)=\sum _{i=1}^m\sum _{j=1}^n\frac{1}{2{\sigma }_i^2}{\left(y_{ij}-{\beta }_{i0}-{\beta }_i{x}_j-d_j\right)}^2,$$ (5)

where we have the irrelevant constant.

In practice, we solve for ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$ using an ad hoc approach, which will be described in Section 3.4.

We introduce a L1 penalty on the β_i′s,

$$p(\beta )=\alpha \Vert \beta {\Vert }_1:=\alpha \sum _{i=1}^m\left|{\beta }_i\right|.$$ (6)

It is well known that L1 the penalty favors sparse solutions (forces some coefficients to be exactly zero) [23]. This is reasonable since in practice many genes are not differentially expressed.

The objective function to be minimized is

$$f\left({\beta }_0,\beta ,d\right)=\sum _{i=1}^m\sum _{j=1}^n\frac{1}{2{\sigma }_i^2}{\left(y_{ij}-{\beta }_{i0}-x_j{\beta }_i-d_j\right)}^2+\alpha \sum _{i=1}^m\left|{\beta }_i\right|.$$ (7)

3. ALGORITHM DEVELOPMENT

3.1. Formulation of (7) as Penalized Simple Linear Regression Model

It can be proved that the optimization problem in (7) is jointly convex in (β₀, β, d). Therefore, the minimizer of (7) is the stationary point.

The derivative of f (β₀, β, d with respect to d_j, j=1, 2, …, n, is

$$\frac{\partial f}{\partial d_j}=\sum _{i=1}^m-\frac{1}{{\sigma }_i^2}\left(y_{ij}-{\beta }_{i0}-x_j{\beta }_i-d_j\right).$$ (8)

Setting to zero gives

$$d_j=\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}\left(y_{ij}-{\beta }_{i0}-x_j{\beta }_i\right).$$ (9)

Model (2) is non-identifiable because we can simply add any constant to all the d_j′s, and subtract the same constant from all the β_i0′s, while having the same fit. To resolve this issue, we fix d_{1 =} 0. Therefore

$$d_j=d_j-d_1=\left({\overline{y}}_{\cdot j}^{(w)}-{\overline{y}}_{\cdot 1}^{(w)}\right)-\left(x_j-x_1\right){\overline{\beta }}^{(w)},$$ (10)

where

$${\overline{y}}_{\cdot j}^{(w)}:=\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{y}_{ij},\text{ for }j=1,2,\dots ,n,$$ (11)

$${\overline{\beta }}^{(w)}:=\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i.$$ (12)

Here, the superscript ^(w) indicates that the mean is a weighted mean instead of an unweighted one.

On the other hand, from

$$\frac{\partial f}{\partial {\beta }_{i0}}=-\frac{1}{{\sigma }_i^2}\sum _{j=1}^n\left(y_{ij}-{\beta }_{i0}-x_j{\beta }_i-d_j\right)=0,$$ (13)

we have

$${\beta }_{i0}=\frac{1}{n}\sum _{j=1}^n\left(y_{ij}-x_j{\beta }_i-d_j\right)={\overline{y}}_{i.}-\overline{x}{\beta }_i-\frac{1}{n}\sum _{j=1}^n{d}_j,$$ (14)

where

$${\overline{y}}_{i.}:=\frac{1}{n}\sum _{j=1}^n{y}_{ij},\text{ for }i=1,2,\dots ,m,$$ (15)

$$\overline{x}:=\frac{1}{n}\sum _{j=1}^n{x}_j.$$ (16)

From (10) we have

$$\frac{1}{n}\sum _{j=1}^n{d}_j=\left({\overline{y}}^{(w)}-{\overline{y}}_{\cdot 1}^{(w)}\right)-\left(\overline{x}-x_1\right){\overline{\beta }}^{(w)},$$ (17)

where

$${\overline{y}}^{(w)}:=\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\left(\frac{1}{{\sigma }_i^2}\cdot \frac{1}{n}\sum _{j=1}^n{y}_{ij}\right).$$ (18)

Substituting (17) into (14) yields

$${\beta }_{i0}={\overline{y}}_{i.}+{\overline{y}}_{\cdot 1}^{(w)}-{\overline{y}}^{(w)}+\left(\overline{x}-x_1\right){\overline{\beta }}^{(w)}-\overline{x}{\beta }_i.$$ (19)

Without loss of generality, we make the following two assumptions:

Assumption 3.1.

$$\sum _{j=1}^n{x}_j=n\overline{x}=0,\sum _{j=1}^n{x}_j^2=1.$$ (20)

These assumptions are reasonable since in the model (2) the center and scaling factor of x_j’s can be absorbed into β_i0 and β_i, respectively.

Then (19) simplifies to

$${\beta }_{i0}={\overline{y}}_{i.}+{\overline{y}}_{\cdot 1}^{(w)}-{\overline{y}}^{(w)}-x_1{\overline{\beta }}^{(w)}.$$ (21)

The sum of (10) and (21) yields

$${\beta }_{i0}+d_j={\overline{y}}_{i.}+{\overline{y}}_{\cdot j}^{(w)}-{\overline{y}}^{(w)}-x_j{\overline{\beta }}^{(w)}.$$ (22)

Substituting (22) into (7), the latter simplifies to

$$f(\beta )=\sum _{i=1}^m\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left({\tilde{y}}_{ij}-x_j{\beta }_i+x_j{\overline{\beta }}^{(w)}\right)}^2+\alpha \sum _{i=1}^m\left|{\beta }_i\right|,$$ (23)

where

$${\tilde{y}}_{ij}:=y_{ij}-{\overline{y}}_{i.}-{\overline{y}}_{\cdot j}^{(w)}+{\overline{y}}^{(w)}.$$ (24)

It can be shown by straightforward calculation that $\left\{{\tilde{y}}_{ij}\right\}$

$$\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\tilde{y}}_{ij}=0,$$ (25)

$$\sum _{j=1}^n{\tilde{y}}_{ij}=0.$$ (26)

3.2. Model Fitting by ADMM

We propose to use the alternating direction method of multipliers (ADMM) [21] to solve (23). Although ADMM can be very slow to converge to high accuracy, it is often the case that ADMM converges to modest accuracy very fast (within a few tens of iterations) [21].

To apply the ADMM, the problem (23) is reformulated as

$$f(\beta )=\sum _{i=1}^m\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left({\tilde{y}}_{ij}-x_j{\beta }_i+x_j{\delta }_0\right)}^2+\alpha \sum _{i=1}^m\left|{\beta }_i\right|,$$ (27a)

subject to

$$\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i={\delta }_0.$$ (27b)

The augmented Lagrangian of (27) is (28) at the bottom of the page.

Step 1: Update β_i, i =1, 2, …, m:

The derivative of (28) with respect to β_i is

$$\begin{gathered} \frac{\partial L_{\rho }}{\partial {\beta }_i}=\frac{1}{{\sigma }_i^2}\sum _{j=1}^n-x_j\left({\tilde{y}}_{ij}-x_j{\beta }_i+x_j{\delta }_0\right)+\alpha \partial \left|{\beta }_i\right| \\ +\frac{1}{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\frac{1}{{\sigma }_i^2}\mathrm{\lambda }+\rho \frac{1}{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\frac{1}{{\sigma }_i^2}\left(\frac{1}{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\sum _{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}{\beta }_{\mathcal{l}}-{\delta }_0\right), \end{gathered}$$ (29)

where ∂ |β_i| is the subgradient of|β_i| with respect to β_i and is defined as

$$\partial \left|{\beta }_i\right|=\{\begin{array}{ll}1,\hfill & {\beta }_i>0\hfill \\ -1,\hfill & {\beta }_i<0\hfill \\ [-1,1],\hfill & {\beta }_i=0\hfill \end{array}$$

Setting (29) equal to zero gives (30) at the bottom of the page, where T is the soft-thresholding operator:

$$T_{{\sigma }_i^2\alpha }[x]:=\text{sign}(x){\left(\left|x\right|-{\sigma }_i^2\alpha \right)}_{+}=\{\begin{array}{ll}x-{\sigma }_i^2\alpha ,\hfill & x>{\sigma }_i^2\alpha \hfill \\ x+{\sigma }_i^2\alpha ,\hfill & x<-{\sigma }_i^2\alpha \hfill \\ 0,\hfill & -{\sigma }_i^2\alpha \le x\le {\sigma }_i^2\alpha \hfill \end{array}$$

Step 2: Update δ₀:

The derivative of (28) with respect to δ₀ is

$$\begin{gathered} \frac{\partial L_{\rho }}{\partial {\delta }_0}=\sum _{i=1}^m\frac{1}{{\sigma }_i^2}\sum _{j=1}^n{x}_j\left({\tilde{y}}_{ij}-x_j{\beta }_i+x_j{\delta }_0\right) \\ -\mathrm{\lambda }+\rho \left({\delta }_0-\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i\right). \end{gathered}$$ (31)

Setting (31) equal to zero gives

$$\begin{gathered} {\delta }_0=\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}+\rho }\left(\mathrm{\lambda }-\sum _{i=1}^m\frac{1}{{\sigma }_i^2}\sum _{j=1}^n{x}_j{\tilde{y}}_{ij}\right)+\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i \\ =\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}+\rho }\mathrm{\lambda }+\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i, \end{gathered}$$ (32)

where the second equality is due to (32) where the second equality is due to (25). Step 3: Update λ:

$${\mathrm{\lambda }}^{\text{ new }}={\mathrm{\lambda }}^{\text{ old }}+\rho \left(\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i-{\delta }_0\right)$$ (33)

The model fitting algorithm is described in Algorithm 1.

3.3. Estimation of Tuning Parameter α

Eq. (23) can be expressed in matrix form as

$$f(\beta )=\frac{1}{2}{\Vert {\Sigma }^{1/2}\left(\tilde{Y}-M\beta x^{\text{T}}\right)\Vert }_{\text{F}}^2+\alpha \Vert \beta {\Vert }_1,$$ (34)

where

$$\Sigma =\text{diag}\left\{\sigma \right\},$$ (35)

with

$$\sigma ={\left(\begin{array}{cccc}1/{\sigma }_1^2& 1/{\sigma }_2^2& \cdots & 1/{\sigma }_m^2\end{array}\right)}^{\text{T}},$$ (36)

and

$$\begin{array}{l}M=\left(\begin{array}{cccc}1& 0& \cdots & 0\\ 0& 1& \cdots & 0\\ 0& 0& \ddots & 0\\ 0& 0& \cdots & 1\end{array}\right)-\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\left(\begin{array}{cccc}1/{\sigma }_1^2& 1/{\sigma }_2^2& \cdots & 1/{\sigma }_m^2\\ 1/{\sigma }_1^2& 1/{\sigma }_2^2& \dots & 1/{\sigma }_m^2\\ ⋮& ⋮& \ddots & ⋮\\ 1/{\sigma }_1^2& 1/{\sigma }_2^2& \cdots & 1/{\sigma }_m^2\end{array}\right)\end{array}=I_m-\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}{1}_m{\sigma }^{\text{T}}.$$ (37)

After expansion, (34) becomes

$$f(\beta )=\frac{1}{2}{\Vert {\Sigma }^{1/2}\tilde{Y}\Vert }_{\text{F}}^2-{\beta }^{\text{T}}{M}^{\text{T}}\Sigma \tilde{Y}x+\frac{1}{2}{\beta }^{\text{T}}{M}^{\text{T}}\Sigma \mathit{M}\mathit{\beta }+\alpha \Vert \beta {\Vert }_1,$$ (38)

where we exploit assumption x^Tx = 1.

Since $\frac{1}{2}{\beta }^{\text{T}}{M}^{\text{T}}\Sigma \mathit{M}\mathit{\beta }\ge 0$ with equality occurring at β = 0, it is shown that $\widehat{\beta }=0$ is the minimizer of f(β)when

$$\alpha \ge {\Vert M^{\text{T}}\mathbf{\Sigma }\tilde{Y}x\Vert }_{\infty }:=\underset{1\le i\le m}{\text{max}}\left|m^{i^{\text{T}}}\mathbf{\Sigma }\tilde{Y}x\right|,$$ (39)

where mⁱ denotes the i-th column of M in (37).

$$L_{\rho }\left(\beta ,{\delta }_0,\mathrm{\lambda }\right)=\sum _{i=1}^m\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left({\tilde{y}}_{ij}-x_j{\beta }_i+x_j{\delta }_0\right)}^2+\alpha \sum _{i=1}^m\left|{\beta }_i\right|+\mathrm{\lambda }\left(\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i-{\delta }_0\right)+\frac{\rho }{2}{\left(\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i-{\delta }_0\right)}^2$$ (28)

$${\beta }_i=\frac{{\sigma }_i^2{\left({\sum }_{\mathcal{l}=1}^m{\sigma }_{\mathcal{l}}^{-2}\right)}^2}{{\sigma }_i^2{\left({\sum }_{\mathcal{l}=1}^m{\sigma }_{\mathcal{l}}^{-2}\right)}^2+\rho }{T}_{{\sigma }_i^2\alpha }\left[\left(\sum _{j=1}^n{x}_j{\tilde{y}}_{ij}+{\delta }_0\right)-\frac{\rho }{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\left(\frac{1}{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\sum _{\mathcal{l}\ne i}\frac{1}{{\sigma }_{\mathcal{l}}^2}{\beta }_{\mathcal{l}}-{\delta }_0+\frac{\mathrm{\lambda }}{\rho }\right)\right]$$ (30)

Note that

$$M^{\text{T}}\Sigma \tilde{\mathit{Y}}=\left(I_m-\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sigma 1_m^{\text{T}}\right)\Sigma \tilde{\mathit{Y}}=\Sigma \tilde{\mathit{Y}},$$ (40)

where the last equality holds because $1_m^{\text{T}}\Sigma \tilde{\mathit{Y}}=\mathbf{0}$ due to (25)

Substituting (40) into (39) yields

$${\alpha }_{\text{max}}=\Vert \Sigma \tilde{\mathit{Y}}\mathit{x}{\Vert }_{\infty }=\underset{i}{\text{max}}\left|\frac{1}{{\sigma }_i^2}{x}^{\text{T}}{\tilde{y}}_i\right|.$$ (41)

Our strategy is to first sort $\left|\frac{1}{{\sigma }_1^2}{x}^{\text{T}}{\tilde{y}}_1\right|,\left|\frac{1}{{\sigma }_2^2}{x}^{\text{T}}{\tilde{y}}_2\right|,\dots ,\left|\frac{1}{{\sigma }_{\text{N}}^2}{x}^{\text{T}}{\tilde{y}}_m\right|$ in ascending order, from least to greatest, and then set α as the P-th percentile (0 < P < 100) of the m ordered value. We set P=5 in Section 5.

3.4. Maximum likelihood estimation of ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$

To solve for ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$, consider the negative log-likelihood function in (4) with ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$ being unknown parameters as well:

$$\mathcal{l}\left({\beta }_0,\beta ,d,{\left\{{\sigma }_i^2\right\}}_{i=1}^m\right)=\sum _{i=1}^m[\frac{n}{2}\text{log}\left(2\pi {\sigma }_i^2\right)+\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left(y_{ij}-{\beta }_{i0}-x_j{\beta }_i-d_j\right)}^2$$ (42)

Taking the partial derivatives of ℓ(^.) with respect to d_j and β_i0 and setting the results to zero, we arrive at (10) and (21) respectively. The sum of (10) and (21) gives (22).

Taking the partial derivative of ℓ(^.) with respect to β_i and setting the result to zero, we have

$${\beta }_i=\sum _{j=1}^n{x}_j{y}_{ij}-\sum _{j=1}^n{x}_j\left({\beta }_{i0}+d_j\right).$$ (43)

Substituting (22) into (43) yields

$${\beta }_i=\sum _{j=1}^n{x}_j{y}_{ij}-\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}\sum _{j=1}^n{x}_j{y}_{ij}+{\overline{\beta }}^{(w)},$$ (44)

where $\overline{\beta }(w)$ is defined in (12).

Taking the partial derivative of ${\sigma }_i^2$ and setting the result to zero gives

$${\sigma }_i^2=\frac{1}{n}\sum _{j=1}^n{\left(y_{ij}-{\beta }_{i0}-x_j{\beta }_i-d_j\right)}^2.$$ (45)

Substituting (22) into (45) yields

$${\sigma }_i^2=\frac{1}{n}\sum _{j=1}^n{\left(y_{ij}-{\overline{y}}_{i.}-{\overline{y}}_{\cdot j}^{(w)}+{\overline{y}}^{(w)}-x_j{\beta }_i+x_j{\overline{\beta }}^{(w)}\right)}^2,$$ (46)

where ${\overline{y}}_{i\cdot }$, ${\overline{y}}_{\cdot j}^{(w)}$ and ${\overline{y}}^{(w)}$ are defined in (15), (11) and (18), respectively.

Given initial estimates for ${\overline{\beta }}^{(w)}$ and ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$ we can alternate equations (44), (46) and (12) iteratively to gradually refine the estimates for β_i and ${\sigma }_i^2$, as shown in Algorithm 2.

To obtain a robust estimate for ${\sigma }_i^2$, we further take the weighted average of ${\widehat{\sigma }}_i^2$ and the estimated mean variance across all the genes. That is

$${\widehat{\sigma }}_i^{\prime 2}=(1-w){\widehat{\sigma }}_i^2+w\overline{{\widehat{\sigma }}^2}$$ (47)

where

$$\overline{{\widehat{\sigma }}^2}=\frac{1}{m}\sum _{i=1}^m{\widehat{\sigma }}_i^2,$$ (48)

and the weight w is calculated using the following formula which is derived based on an empirical Bayes approach [24]

$$w=\frac{2(m-1)}{n+1}\left(\frac{1}{m}+\frac{{\left(\overline{{\widehat{\sigma }}^2}\right)}^2}{{\sum }_{i=1}^m{\left({\widehat{\sigma }}_i^2-\overline{{\widehat{\sigma }}^2}\right)}^2}\right).$$ (49)

This kind of variance estimation approach is widely used in differential gene expression analysis with small sample sizes [25], [26]. The estimated variances ${\widehat{\sigma }}_i^{\prime 2}$ can then used in Algorithm 1 to solve for ${\left\{{\beta }_i\right\}}_{i=1}^m$Remark 3.1. In the special case of ${\sigma }_1^2={\sigma }_2^2=\cdots ={\sigma }_m^2={\sigma }^2$, it no longer requires to estimate σ² since the unknown σ² in (7) can be absorbed into the tuning parameter α.

4. EXTENSION TO MULTIPLE LINEAR REGRESSION MODEL AND ALGORITHM DEVELOPMENT

In the multiple linear regression model, each response or outcome is modeled by p > 1 predictors:

$$y_{ij}={\beta }_{i0}+{\beta }_i^{\text{T}}{x}_j+d_j+{\epsilon }_{ij},$$ (50)

where

$${\beta }_i=\left(\begin{array}{c}{\beta }_{i1}\\ {\beta }_{i2}\\ ⋮\\ {\beta }_{ip}\end{array}\right)\in {ℝ}^{p\times 1}$$ (51)

is a vector of regression coefficients representing log-fold-change of expression levels of gene i between treatment conditions, and

$$x_j=\left(\begin{array}{c}{x}_{j1}\\ x_{j2}\\ ⋮\\ x_{jp}\end{array}\right)\in {ℝ}^{p\times 1}$$ (52)

is a vector of predictors representing the treatment conditions (drug dosage, blood pressure, age, BMI, etc.) for sample j, and β_i0, d_j and ${\epsilon }_{ij}~\mathcal{N}\left(0,{\sigma }_i^2\right)$ are the y-intercept, scaling factor for sample j and measurement noise, respectively. We assume that the error terms ε_ij are uncorrelated with all the predictor variables and uncorrelated with each other.

The likelihood function based on the observed data is given by

$$L\left({\beta }_0,{\left\{{\beta }_i\right\}}_{i=1}^m,d;Y\right)=\prod _{i=1}^m\prod _{j=1}^n\frac{1}{\sqrt{2\pi {\sigma }_i^2}}\text{exp}\left\{-\frac{{\left(y_{ij}-{\beta }_{i0}-{\beta }_i^{\text{T}}{x}_j-d_j\right)}^2}{2{\sigma }_i^2}\right\}.$$ (53)

Assume that ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$ are known, maximization of (53) leads to minimizing the negative log-likelihood:

$$\mathcal{l}\left({\beta }_0,{\left\{{\beta }_i\right\}}_{i=1}^m,d\right)=\sum _{i=1}^m\sum _{j=1}^n\frac{1}{2{\sigma }_i^2}{\left(y_{ij}-{\beta }_{i0}-{\beta }_i^{\text{T}}{x}_j-d_j\right)}^2$$ (54)

The objective function to be minimized is

$$f\left({\beta }_0,\left\{{\beta }_i\right\},d\right)=\sum _{i=1}^m\sum _{j=1}^n\frac{1}{2{\sigma }_i^2}{\left(y_{ij}-{\beta }_{i0}-x_j^{\text{T}}{\beta }_i-d_j\right)}^2+\sum _{i=1}^{m}p\left({\beta }_i\right).$$ (55)

Below we introduce two types of penalty function p(β_i).

1. Type I penalty:

$$p\left({\beta }_i\right)=\alpha \left|{\beta }_{ip}\right|.$$ (56)

Gene i is differentially expressed if β_ip ≠ 0 and not otherwise. This penalty is for the applications where one covariate is of main interest (e.g., treatment) while we want to adjust for all possible effects of other confounding covariates (e.g., age, gender, etc).

1. Type II penalty:

$$p\left({\beta }_i\right)=\alpha \Vert {\beta }_i\Vert .$$ (57)

Gene i is differentially expressed if β_i≠ 0 and not otherwise. This penalty is for the applications where all covariates are of interest and we want to identify the genes for which at least one covariate has an effect.

It can be proved that the optimization problem (55) with penalty (56) or (57) is jointly convex in (β₀, {β_i} ,d).

Assume that

$$\sum _{j=1}^n{x}_j=0,$$ (58)

and set d₁=0. using similar argumentation as in Section 3.1 to eliminate β₀ and d, we simplify (55) to

$$f\left(\left\{{\beta }_i\right\}\right)=\sum _{i=1}^m\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left({\tilde{y}}_{ij}-x_j^{\text{T}}{\beta }_i+x_j^{\text{T}}{\overline{\beta }}^{(w)}\right)}^2+\sum _{i=1}^{m}p\left({\beta }_i\right),$$ (59)

where ${\tilde{y}}_{ij}$ is the same as that in (24), and

$${\overline{\beta }}^{(w)}:=\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i.$$ (60)

4.1. Regression with type I penalty: Model fitting by ADMM

To apply the ADMM, we reformulate the Type I penalized regression problem as

$$f\left(\left\{{\beta }_i\right\},{\delta }_0\right)=\sum _{i=1}^m\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left({\tilde{y}}_{ij}-x_j^{\text{T}}{\beta }_i+x_j^{\text{T}}{\delta }_0\right)}^2+\alpha \sum _{i=1}^m\left|{\beta }_{ip}\right|,$$ (61a)

subject to

$$\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i={\delta }_0.$$ (61b)

The augmented Lagrangian of (61) is (62) at the bottom of the page.

Step 1: Update β_i, i 1, 2, …, m:

Taking the partial derivative of (62) with respect to β_i and setting the result to zero gives

$$\left[\frac{1}{{\sigma }_i^2}{X}^{\text{T}}X+\frac{\rho }{{\sigma }_i^4}\frac{1}{{\left({\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}\right)}^2}{I}_p\right]{\beta }_i+\alpha \left(\begin{array}{c}0\\ ⋮\\ 0\\ \partial \left|{\beta }_{ip}\right|\end{array}\right)=v_i,$$ (63)

where

$$X=\left(\begin{array}{c}{x}_1^{\text{T}}\\ x_2^{\text{T}}\\ ⋮\\ x_n^{\text{T}}\end{array}\right)=\left(\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right)\in {ℝ}^{n\times p},$$ (64)

∂|β_ip| is the subgradient of |β_ip| with respect to β_ip,a nd

$$\begin{gathered} v_i=\frac{1}{{\sigma }_i^2}\left(\sum _{j=1}^n{x}_j{\tilde{y}}_{ij}+X^{\text{T}}X{\delta }_0\right) \\ -\frac{\rho }{{\sigma }_i^2}\frac{1}{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\left(\frac{1}{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\sum _{\mathcal{l}\ne i}\frac{1}{{\sigma }_{\mathcal{l}}^2}{\mathit{\beta }}_{\mathcal{l}}-{\mathit{\delta }}_0+\frac{\mathbf{\lambda }}{\rho }\right). \end{gathered}$$ (65)

Given matrix partition in the following form:

$$\begin{gathered} \frac{1}{{\sigma }_i^2}{X}^{\text{T}}X+\frac{\rho }{{\sigma }_i^4}\frac{1}{{\left({\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}\right)}^2}{I}_p=Q=\left(\begin{array}{cc}{Q}_{11}& q\\ q^{\text{T}}& q_{pp}\end{array}\right), \\ {\beta }_i=\left(\begin{array}{c}{\beta }_i^{-}\\ {\beta }_{ip}\end{array}\right),v_i=\left(\begin{array}{c}{v}_i^{-}\\ v_{ip}\end{array}\right), \end{gathered}$$

where Q₁₁ is the submatrix‚ of Q „with last‚ row and last column deleted, from (63) we have

$$Q_{11}{\beta }_i^{-}+q{\beta }_{ip}=v_i^{-}$$ (66)

$$q^{\text{T}}{\beta }_i^{-}+q_{pp}{\beta }_{ip}+\alpha \partial \left|{\beta }_{ip}\right|=v_{ip}.$$ (67)

From (66) it follows

$${\beta }_i^{-}=Q_{11}^{-1}\left(v_i^{-}-q{\beta }_{ip}\right).$$ (68)

Substituting (68) into (67) yields

$${\beta }_{ip}=\frac{1}{q_{pp}-q^{\text{T}}{Q}_{11}^{-1}q}{T}_{\alpha }\left[v_{ip}-q^{\text{T}}{Q}_{11}^{-1}{v}_i^{-}\right].$$ (69)

Step 2: Update δ₀:

Taking the derivative of (62) with respect to δ₀ and setting the result to zero gives

$${\delta }_0={\left(\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{X}^{\text{T}}X+\rho I_p\right)}^{-1}\mathbf{\lambda }+\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i,$$ (70)

where we have exploited (25).

Step 3: Update λ:

$${\mathbf{\lambda }}^{\text{ new }}={\mathbf{\lambda }}^{\text{ old }}+\rho \left(\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\mathit{\beta }}_i-{\mathit{\delta }}_0\right).$$ (71)

The model fitting algorithm is described in Algorithm 3.

4.2. Regression with type II penalty: Model fitting by ADMM

The Type II penalized regression problem is reformulated as

$$f\left(\left\{{\beta }_i\right\},{\delta }_0\right)=\sum _{i=1}^m\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left({\tilde{y}}_{ij}-x_j^{\text{T}}{\beta }_i+x_j^{\text{T}}{\delta }_0\right)}^2+\alpha \sum _{i=1}^m\Vert {\beta }_i\Vert ,$$ (72a)

subject to

$$\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i={\delta }_0.$$ (72b)

The augmented Lagrangian of (72) is (73) at the bottom of the page.

Step 1: Update β_i, i 1, 2, …, m:

The relevant terms to compute the derivatives of (73) with respect to β_i is (74) at the bottom of the page, where c is an irrelevant constant which does not depend on β_i, and v_i is defined in (65).

$$L_{\rho }\left(\left\{{\beta }_i\right\},{\delta }_0,\mathbf{\lambda }\right)=\sum _{i=1}^m\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left({\tilde{y}}_{ij}-x_j^{\text{T}}{\beta }_i+x_j^{\text{T}}{\delta }_0\right)}^2+\alpha \sum _{i=1}^m\left|{\beta }_{ip}\right|+{\mathbf{\lambda }}^{\text{T}}\left(\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i-{\delta }_0\right)+\frac{\rho }{2}{\Vert \frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i-{\delta }_0\Vert }^2.$$ (62)

It can be shown that when ‖υ_i‖ ≤ α then β_{i =} 0; otherwise denote the eigendecomposition of X^TX as X^TX=UDU^T, we have that minimization of (74) is equivalent to

$$\underset{{\beta }_i}{\text{min}}\frac{1}{2}{\Vert Z_i{\beta }_i-b_i\Vert }^2+\alpha \Vert {\beta }_i\Vert ,$$ (75a)

where

$$Z_i={\left[\frac{1}{{\sigma }_i^2}D+\frac{\rho }{{\sigma }_i^4}\frac{1}{{\left({\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}\right)}^2}{I}_p\right]}^{1/2}{U}^{\text{T}},$$ (75b)

$$b_i={\left[\frac{1}{{\sigma }_i^2}D+\frac{\rho }{{\sigma }_i^4}\frac{1}{{\left({\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}\right)}^2}{I}_p\right]}^{-1/2}{U}^{\text{T}}{v}_i.$$ (75c)

As in [27], we use a coordinate descent procedure to optimize (75). For each s, given the estimate of ${\left\{{\widehat{\beta }}_{i\mathcal{l}}\right\}}_{\mathcal{l}\ne s}$ can be estimated by solving

$$\underset{{\beta }_{is}}{\text{min}}\frac{1}{2}{\Vert z^s{\beta }_{is}-r_i^{(s)}\Vert }^2+\alpha \sqrt{{\beta }_{is}^2+\sum _{\mathcal{l}\ne s}{\widehat{\beta }}_{i\mathcal{l}}^2},$$ (76)

where

$$r_i^{(s)}=b_i-\sum _{\mathcal{l}\ne s}{z}^{\mathcal{l}}{\widehat{\beta }}_{i\mathcal{l}}.$$ (77)

We solve (76) via a one-dimensional search. Note that the solution to (76) falls between 0 and ${\beta }_{i\mathcal{l}}^{\text{o}}=z^{s\text{T}}{r}_i^{(s)}/{\Vert z^s\Vert }^2$, the ordinary least-squares estimate. We can use the optimize function in the R package, or fminbnd function in MAT-LAB, which performs one-dimensional search based on golden section search and successive parabolic interpolation.

After updating ${\left\{{\beta }_i\right\}}_{i=1}^m$, the updates of δ₀ and λ out to be the same as that in Section 4.1. The model fitting algorithm is described in Algorithm 4.

4.3. Estimation of Tuning Parameter α

Eq. (59) can be expressed in matrix form as

$$f(B)=\frac{1}{2}{\Vert {\Sigma }^{1/2}\left(\tilde{Y}-MB{X}^{\text{T}}\right)\Vert }_{\text{F}}^2+p(B),$$ (78)

where M and X are respectively defined in (37) and (64), and

$$B=\left(\begin{array}{c}{\beta }_1^{\text{T}}\\ {\beta }_2^{\text{T}}\\ ⋮\\ {\beta }_m^{\text{T}}\end{array}\right)=\left(\begin{array}{cccc}{\beta }_{11}& {\beta }_{12}& \cdots & {\beta }_{1p}\\ {\beta }_{21}& {\beta }_{22}& \cdots & {\beta }_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ {\beta }_{m1}& {\beta }_{m2}& \cdots & {\beta }_{mp}\end{array}\right)\in {ℝ}^{m\times p},$$ (79)

and p(B)is the penalty function.

The derivative of f(B)with respect to B is

$$\frac{\partial f}{\partial B}=M^{\text{T}}\Sigma \mathit{M}\mathit{B}{X}^{\text{T}}X-M^{\text{T}}\Sigma \tilde{\mathit{Y}}\mathit{X}+\frac{\partial p(B)}{\partial B}.$$ (80)

4.3.1. Type I Penalty

When p (B) $p(B)=\alpha {\sum }_{i=1}^m\left|{\beta }_{ip}\right|$, its derivative with respect to B is

$$\frac{\partial p(B)}{\partial B}=\alpha \left(\begin{array}{cccc}0& \cdots & 0& \partial \left|{\beta }_{1p}\right|\\ 0& \cdots & 0& \partial \left|{\beta }_{2p}\right|\\ ⋮& \ddots & ⋮& ⋮\\ 0& \cdots & 0& \partial \left|{\beta }_{mp}\right|\end{array}\right)=\left(\begin{array}{cc}{0}_{m\times (p-1)}& \alpha \frac{\partial {\Vert {\beta }^p\Vert }_1}{\partial {\beta }^p}\end{array}\right).$$ (81)

Denote

$$\begin{array}{l}X=\left[\begin{array}{cccc}{x}^1& \cdots & x^{p-1}& x^p\end{array}\right]=\left[\begin{array}{cc}{X}_1& x^p\end{array}\right],\\ B=\left[\begin{array}{llll}{\beta }^1\hfill & \cdots \hfill & {\beta }^{p-1}\hfill & {\beta }^p\hfill \end{array}\right]=\left[\begin{array}{ll}{B}_1\hfill & {\beta }^p\hfill \end{array}\right].\end{array}$$

Setting (80) equal to zero gives

$$M^{\text{T}}\Sigma \mathit{M}\left(B_1{X}_1^{\text{T}}+{\beta }^p{x}^{p\text{T}}\right)X_1=M^{\text{T}}\Sigma \tilde{Y}{X}_1$$ (82)

$$M^{\text{T}}\Sigma \mathit{M}\left(B_1{X}_1^{\text{T}}+{\beta }^p{x}^{p\text{T}}\right)x^p+\alpha \frac{\partial {\Vert {\beta }^p\Vert }_1}{\partial {\beta }^p}=M^{\text{T}}\Sigma \tilde{Y}{x}^p.$$ (83)

Since M^TΣM is rank deficient², the solution to (82) is not unique. We apply the pseudoinverse of M^TΣM to obtain the minimum-norm solution to (82):

$$B_1={\left(M^{\text{T}}\Sigma \mathit{M}\right)}^{†}(M^{\text{T}}\Sigma \tilde{Y}-M^{\text{T}}\Sigma \mathit{M}{\beta }^p{x}^{p\text{T}})X_1{\left(X_1^{\text{T}}{X}_1\right)}^{-1}.$$ (84)

Substituting (84) into (83) yields

$$\begin{gathered} M^{\text{T}}\Sigma \mathit{M}{\beta }^p{x}^{p\text{T}}\left[I_n-X_1{\left(X_1^{\text{T}}{X}_1\right)}^{-1}{X}_1^{\text{T}}\right]x^p+\alpha \frac{\partial {\Vert {\beta }^p\Vert }_1}{\partial {\beta }^p} \\ =M^{\text{T}}\Sigma \tilde{\mathit{Y}}\left[I_n-X_1{\left(X_1^{\text{T}}{X}_1\right)}^{-1}{X}_1^{\text{T}}\right]x^p. \end{gathered}$$ (85)

2. Simple analysis shows that the rank of M^TΣM is m-1.

$$L_{\rho }\left(\left\{{\beta }_i\right\},{\delta }_0,\mathbf{\lambda }\right)=\sum _{i=1}^m\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left({\tilde{y}}_{ij}-x_j^{\text{T}}{\beta }_i+x_j^{\text{T}}{\delta }_0\right)}^2+\alpha \sum _{i=1}^m\Vert {\beta }_i\Vert +{\mathbf{\lambda }}^{\text{T}}\left(\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\beta }_i-{\delta }_0\right)+\frac{\rho }{2}{\Vert \frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}{\mathit{\beta }}_i-{\mathit{\delta }}_0\Vert }^2$$ (73)

$$\begin{gathered} L_i\left(\left\{{\beta }_i\right\},{\delta }_0,\mathbf{\lambda }\right)=\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left({\tilde{y}}_{ij}-x_j^{\text{T}}{\beta }_i+x_j^{\text{T}}{\delta }_0\right)}^2+\alpha \Vert {\beta }_i\Vert +{\mathbf{\lambda }}^{\text{T}}\frac{1}{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\frac{1}{{\sigma }_i^2}{\beta }_i+\frac{\rho }{2}{\Vert \frac{1}{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\frac{1}{{\sigma }_i^2}{\beta }_i+\frac{1}{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\sum _{\mathcal{l}\ne i}\frac{1}{{\sigma }_{\mathcal{l}}^2}{\beta }_{\mathcal{l}}-{\delta }_0\Vert }^2 \\ =\frac{1}{2}{\beta }_i^{\text{T}}\left(\frac{1}{{\sigma }_i^2}{X}^{\text{T}}X+\frac{\rho }{{\sigma }_i^4}\frac{1}{{\left({\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}\right)}^2}{I}_p\right){\beta }_i-{\beta }_i^{\text{T}}{v}_i+\alpha \Vert {\beta }_i\Vert +c \end{gathered}$$ (74)

Note that to arrive at (85), we have exploited the fact that (M^TΣM)(M^TΣM)^†M^TΣ=M^TΣ which is due to that M^TΣM=M^TΣ according to the the definition of M in (37) and definition of the pseudoinverse of a matrix.

Since the coefficient matrix of β^p, i.e., $M^{\text{T}}\Sigma \mathit{M}\left(x^{p\text{T}}\left[I_n-X_1{\left(X_1^{\text{T}}{X}_1\right)}^{-1}{X}_1^{\text{T}}\right]x^p\right)$ is positive semidefinite, (85) implies that when

$$\begin{gathered} \alpha \ge \Vert M^{\text{T}}\Sigma \tilde{\mathit{Y}}[I_n-X_1{\left(X_1^{\text{T}}{X}_1\right)}^{-1}{X}_1^{\text{T}}]x^p{\Vert }_{\infty } \\ ={\Vert \Sigma \tilde{\mathit{Y}}\left[I_n-X_1{\left(X_1^{\text{T}}{X}_1\right)}^{-1}{X}_1^{\text{T}}\right]x^p\Vert }_{\infty } \\ =\underset{i}{\text{max}}|\frac{1}{{\sigma }_i^2}{\tilde{y}}_i^{\text{T}}\left[I_n-X_1{\left(X_1^{\text{T}}{X}_1\right)}^{-1}{X}_1^{\text{T}}\right]x^p|, \end{gathered}$$ (86)

where the next to last equality is due to (40) we obtain zero solution.

4.3.2. Type II Penalty

The derivative of $p(B)=\alpha {\sum }_{i=1}^m\Vert {\beta }_i\Vert$ with respect to B is

$$\frac{\partial p(B)}{\partial B}=\alpha \left(\begin{array}{c}\frac{\partial \Vert {\beta }_1\Vert }{\partial {\beta }_1^{\text{T}}}\\ \frac{\partial \Vert {\beta }_2\Vert }{\partial {\beta }_2^{\text{T}}}\\ ⋮\\ \frac{\partial \Vert {\beta }_m\Vert }{\partial {\beta }_m^{\text{T}}}\end{array}\right),$$ (87)

when $\frac{\partial \Vert {\beta }_i\Vert }{\partial {\beta }_i}=\frac{{\beta }_i}{\Vert {\beta }_i\Vert }$ if β_i ≠ 0 and $\Vert \frac{\partial \Vert {\beta }_i\Vert }{\partial {\beta }_i}\Vert \le 1$ otherwise[27],[28]

Setting (80) equal to zero yields

$$X^{\text{T}}X{B}^{\text{T}}{M}^{\text{T}}\Sigma m^i-X^{\text{T}}{\tilde{Y}}^{\text{T}}\Sigma m^i+\alpha \frac{\partial \Vert {\beta }_i\Vert }{\partial {\beta }_i}=0_{p\times 1},$$ (88)

for i=1, 2, …, m, where mⁱ is the i-th column of M in (37) The minimizer to f (B) is a zero matrix when

$$\alpha \ge \underset{i}{\text{max}}\Vert X^{\text{T}}{\tilde{Y}}^{\text{T}}\Sigma m^i\Vert .$$ (89)

Note that

$${\tilde{Y}}^{\text{T}}\Sigma m^i={\tilde{Y}}^{\text{T}}\Sigma \left(e_i-\frac{1}{{\sum }_{\mathcal{l}=1}^m\frac{1}{{\sigma }_{\mathcal{l}}^2}}\frac{1}{{\sigma }_i^2}{1}_m\right)={\tilde{Y}}^{\text{T}}\Sigma e_i=\frac{1}{{\sigma }_i^2}{\tilde{y}}_i,$$ (90)

where the next to last equality is due to (25) Substituting (90) into (89) yields

$${\alpha }_{\text{max}}=\underset{i}{\text{max}}\Vert \frac{1}{{\sigma }_i^2}{X}^{\text{T}}{\tilde{y}}_i\Vert .$$ (91)

4.4. Maximum likelihood estimation of ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$

To solve for ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$ consider the negative log-likelihood function with ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$ being unknown parameters as well:

$$\mathcal{l}\left({\mathit{\beta }}_0,{\left\{{\beta }_i\right\}}_{i=1}^m,d,{\left\{{\sigma }_i^2\right\}}_{i=1}^m\right)=\sum _{i=1}^m[\frac{n}{2}\text{log}\left(2\pi {\sigma }_i^2\right)+\frac{1}{2{\sigma }_i^2}\sum _{j=1}^n{\left(y_{ij}-{\beta }_{i0}-x_j^{\text{T}}{\beta }_i-d_j\right)}^2].$$ (92)

Taking the partial derivatives of ℓ(^.)with respect to d_j and β_i0 and setting the result to zero, we arrive at

$$d_j=d_j-d_1=\left({\overline{y}}_{\cdot j}^{(w)}-{\overline{y}}_{.1}^{(w)}\right)-{\left(x_j-x_1\right)}^{\text{T}}{\overline{\beta }}^{(w)},$$ (93)

$${\beta }_{i0}={\overline{y}}_{i.}-\frac{1}{n}\sum _{j=1}^n{x}_j^{\text{T}}{\beta }_i-\frac{1}{n}\sum _{j=1}^n{d}_j={\overline{y}}_{i.}+{\overline{y}}_{\cdot 1}^{(w)}-{\overline{y}}^{(w)}-x_1^{\text{T}}{\overline{\beta }}^{(w)},$$ (94)

where to derive the second equality we have exploited assumption (58).

The sum of (93) and (94) gives

$${\beta }_{i0}+d_j={\overline{y}}_{i.}+{\overline{y}}_{\cdot j}^{(w)}-{\overline{y}}^{(w)}-x_j^{\text{T}}{\overline{\beta }}^{(w)}.$$ (95)

Taking the partial derivative of ℓ(^.) with respect to β_i and setting the result to zero, we have

$${\beta }_i=\sum _{j=1}^n{x}_j{y}_{ij}-\sum _{j=1}^n{x}_j\left({\beta }_{i0}+d_j\right).$$ (96)

Substituting (95) into (96) yields

$${\beta }_i={\left(X^{\text{T}}X\right)}^{-1}\left[\sum _{j=1}^n{x}_j{y}_{ij}-\frac{1}{{\sum }_{i=1}^m\frac{1}{{\sigma }_i^2}}\sum _{i=1}^m\frac{1}{{\sigma }_i^2}\sum _{j=1}^n{x}_j{y}_{ij}\right]+{\overline{\beta }}^{(w)},$$ (97)

where β^(w) is defined in (60).

Taking the partial derivative of ${\sigma }_i^2$ and setting the result to zero gives

$${\sigma }_i^2=\frac{1}{n}\sum _{j=1}^n{\left(y_{ij}-{\beta }_{i0}-x_j^{\text{T}}{\beta }_i-d_j\right)}^2.$$ (98)

Substituting (95) into (98) yields

$${\sigma }_i^2=\frac{1}{n}\sum _{j=1}^n{\left(y_{ij}-{\overline{y}}_{i.}-{\overline{y}}_{\cdot j}^{(w)}+{\overline{y}}^{(w)}-x_j^{\text{T}}{\beta }_i+x_j^{\text{T}}{\overline{\beta }}^{(w)}\right)}^2,$$ (99)

where ${\overline{y}}_{i\cdot }$, ${\overline{y}}_{\cdot j}^{(w)}$ and ${\overline{y}}^{(w)}$ are defined in (15), (11) and (18), respectively.

Given initial estimates for ${\overline{\beta }}^{(w)}$ and ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$ estimates for β_i and ${\sigma }_i^2$ can then be iteratively updated using equations (97), (99), and (60) and until convergence.

After estimating ${\sigma }_i^2\prime \text{s}$ s, they can then be shrinked (squeezed) toward the common noise variance to obtain robust estimates for ${\sigma }_i^2$, as done in Section 3.4.

Given initial estimates for ${\overline{\beta }}^{(w)}$ and ${\left\{{\sigma }_i^2\right\}}_{i=1}^m$, estimates for β_i and ${\sigma }_i^2$ can then be iteratively updated using equations (97), (99), (60) until convergence, as shown in Algorithm 5.

5. EXPERIMENTS

We evaluate the performance of the proposed algorithm (referred to as ELMSeq, short for extended linear model for RNA-seq data analysis). To save space, we only verify the proposed algorithm the for simple regression model (2). We use the 5^th percentile to set the tuning parameter α (see Section 3.3).

We compare our method with the state-of-the-art methods for detecting differential gene expression from RNA-seq data: edgeR-robust [20], [29], DESeq2 [18], and limma-voom [16], [17].

5.1. Simulations on Synthetic Data

We simulate RNA-seq data with a total of m 1000 genes and n = 15 samples. The data generation is described in Table 1.

Table 1:

Synthetic data generation process and parameters

ℓi ~ 2unif(5,10)	gene length of gene i
${\beta }_{i0}~\mathcal{N}(0,1)$	other log scaling factors of gene i
βi = 0	log-fold change for non-DE genes
${\beta }_i~\mathcal{N}(2,1)$	log-fold change for up-regulated DE genes
${\beta }_i~\mathcal{N}(-2,1)$	log-fold change for down-regulated DE genes
$x_j~\mathcal{N}(0,1)$	condition data of sample j
Nj ~ unif(2, 3) × 106	library size of sample j
$d_j~\mathcal{N}(0,1)$	other log scaling factors of sample j
${\mu }_{ij}=N_j\frac{{\mathcal{l}}_i}{{\sum }_{i=1}^m{\mathcal{l}}_i}{\text{e}}^{{\beta }_{i0}+{\beta }_i{x}_j+d_j}$	expected RNA-seq read counts of gene i from sample j
$c_{ij}=\lceil {\text{e}}^{\text{log}\mathcal{N}\left({\mu }_{ij},0.1\right)}\rceil$	read counts
yij = log cij	log-transformed gene expression

We first examine whether the proposed algorithm can accurately estimate the log-fold changes (or slopes) β_i′s. For ease of illustration, we set the true slopes for DE ones as β_i= ±2 instead of ${\beta }_i~\mathcal{N}(\pm 2,1)$ We start with 300 DE and 700 non-DE genes. Among DE genes 50% are up-regulated while the remaining 50% are down-regulated. The fitted ${\left\{{\beta }_i\right\}}_{i=1}^m$ using ELMSeq are plotted in Figure 1(a). We see that the estimated slopes are centered around the true ones: the estimated β_i′s of the DE genes are centered around ±2, while those of the non-DE genes are close to zero. In Figure 1(b) and Figure 1(c), we increase the percent of up-regulated DE genes to 70% and 90%, respectively. Our method still accurately retrieves all non-zero β_i′s while shrinking all other β_i′s to zero.

Figure 1:

Estimated β_i in the simple linear regression model from simulated RNASeq data, where the number of genes is m=1000 and number of samples is n= 15. The number of DE genes varies from 300 to 700, and the percentage of up-regulated DE genes varies from 50% to 90%. Along the horizontal axis, from left to right: up-regulated genes (β_i= 2), down-regulated genes (β_i= −2) and non-DE genes (β_i=0).

In Figure 1(d–f), we increase the number of DE genes to 500, among which 50%, 70% or 90% are up-regulated while others are down-regulated. Our method still achieves accurate estimates. In Figure 1(g–h), we further increase the number of DE genes to 700 among which 50% or 70% are up-regulated, for which our method still achieves accurate estimates when. Only when we simulate with 700 DE genes among which 90% are up-regulated, our method fails to distinguish between DE and non-DE genes since the estimated regression coefficients of the latter are not zero either [Figure 1(i)]. A theoretical explanation of Figure 1(i) has been provided in the supplementary material.

Using a different gene expression measure such as CPM, RPKM or TPM values computed with formulas in (1) yields essentially the same result.

Using the algorithm in Algorithm (1), we estimate the regression coefficient ${\widehat{\beta }}_i$ for each gene i. We decide there is a linear relationship between the predictor variable x_j and the expression data y_ij if ${\widehat{\beta }}_i\ne 0$ The larger $\left|{\widehat{\beta }}_i\right|$ is, the stronger the relationship. We then sort the genes in descending order of their $\left|{\widehat{\beta }}_i\right|$ vary the threshold to construct the receiver operating characteristic (ROC) curve and to calculate the area under the ROC curve (AUC).

The AUCs for DE gene detection using all four methods are summarized in Table 2. We see that the ELMSeq performs better than or comparbly to other three methods, regardless of how many genes are differentially expressed and whether they are expressed in a symmetric manner or not. In challenging cases where a large proportion of genes are differentially expressed in an asymmetric manner (e.g., 50% DE genes among which 90% are up-regulated or 70% DE genes among which 70% are up-regulated), the performance gain of the ELMSeq over completing methods is more significant.

Table 2:

AUC comparison of edgeR-robust, DESeq2, limma voom and ELMSeq in log-normally distributed data. Number of samples: n= 15, log-fold change for DE genes: ${\beta }_i~\mathcal{N}(\pm 2,1)$, and noise level: σ_i= 0.1. The table shows the percent of DE genes (DE %), percent of up-regulated genes among the DE genes (Up %), as well as the mean AUCs for all four methods measured using 10 simulated replicates. The standard errors of the mean AUCs are given in parentheses.

DE (%)	Up (%)	edgeR	DESeq2	voom	ELMSeq
10	50	0.9903 (0.0016)	0.6068 (0.0807)	0.991 (0.0018)	0.9914 (0.0017)
10	70	0.9935 (0.0021)	0.4527 (0.0638)	0.9941 (0.0021)	0.9943 (0.0021)
10	90	0.9869 (0.0028)	0.6878 (0.0637)	0.9875 (0.0024)	0.9897 (0.0022)
30	50	0.9898 (0.001)	0.5508 (0.0883)	0.99 (0.001)	0.99 (0.001)
30	70	0.9891 (0.0014)	0.7946 (0.064)	0.9897 (0.0014)	0.991 (0.0011)
30	90	0.9788 (0.0023)	0.6114 (0.0805)	0.9796 (0.0022)	0.9795 (0.0014)
50	50	0.9917 (8e-04)	0.429 (0.0797)	0.9916 (8e-04)	0.9917 (8e-04)
50	70	0.9748 (0.0026)	0.4923 (0.081)	0.9754 (0.0026)	0.9826 (0.0015)
50	90	0.8717 (0.0133)	0.4697 (0.0667)	0.8801 (0.0119)	0.9662 (0.002)
70	50	0.9907 (9e-04)	0.5572 (0.1027)	0.9915 (8e-04)	0.9923 (7e-04)
70	70	0.8564 (0.018)	0.5307 (0.0588)	0.8696 (0.0148)	0.9591 (0.0034)
70	90	0.3375 (0.0108)	0.4808 (0.0192)	0.3204 (0.0154)	0.4718 (0.0124)

In Table 3, we decrease the log-fold change of the DE genes as ${\beta }_i~\mathcal{N}(\pm 0.2,0.1)$ while keeping all other data generation parameters (including the noise level) the same as those in Table 2. We see that all methods suffer a degradation in AUC performance; but again, the ELMSeq consistently perform better than or comparably to all other methods.

Table 3:

AUC comparison of edgeR-robust, DESeq2, limma voom and ELMSeq in log-normally distributed data. The data generation parameters are the same as those in Table 2 except that the log-fold changes for DE genes decrease to: ${\beta }_i~\mathcal{N}(\pm 0.2,0.1)$.

DE (%)	Up (%)	edge	DESeq2	voom	ELMSeq
10	50	0.8055 (0.0089)	0.5241 (0.0142)	0.8224 (0.0095)	0.8232 (0.0095)
10	70	0.8086 (0.009)	0.4846 (0.0126)	0.8212 (0.0095)	0.8234 (0.0101)
10	90	0.7867 (0.0084)	0.5078 (0.0084)	0.7955 (0.0104)	0.8024 (0.0106)
30	50	0.8087 (0.005)	0.497 (0.0119)	0.8158 (0.0054)	0.8157 (0.0054)
30	70	0.7848 (0.0052)	0.5471 (0.0211)	0.7949 (0.0052)	0.8013 (0.0055)
30	90	0.7398 (0.0059)	0.5329 (0.0181)	0.7505 (0.0059)	0.773 (0.0054)
50	50	0.8143 (0.0061)	0.4931 (0.0137)	0.8265 (0.0049)	0.8268 (0.0051)
50	70	0.7611 (0.0054)	0.5061 (0.0155)	0.7704 (0.0054)	0.7752 (0.0056)
50	90	0.6451 (0.006)	0.5017 (0.0102)	0.6503 (0.0059)	0.6793 (0.0025)
70	50	0.8149 (0.0022)	0.5231 (0.0273)	0.8261 (0.003)	0.8267 (0.0028)
70	70	0.7271 (0.0074)	0.5093 (0.01)	0.7354 (0.0086)	0.7388 (0.0083)
70	90	0.5449 (0.0066)	0.5158 (0.0089)	0.5505 (0.0081)	0.5434 (0.0069)

Note that when more samples are available, the performance gain of the ELMSeq over completing methods becomes even more significant. The results for various sample sizes n=5, 8, 25, 50, 100 are provided in the supplementary materials (Tables S1–S5 for genes with high expression profiles ${\beta }_i~\mathcal{N}(\pm 2,1)$ and Tables S6–S10 for genes with low expression profiles ${\beta }_i~\mathcal{N}(\pm 0.2,0.1)$).

We also performed simulations with the multiple linear regression model in Section 4, and the preliminary results are similar to that obtained for the simple regression model. Note that unlike the simple regression model and type I penalized multiple linear regression model, the type II penalized multiple linear regression model does not allow to define up-and down-regulated genes as multiple regression coefficients are tested simultaneously.

5.2. An application to a real RNA-Seq dataset

We further evaluate our algorithm on a prostate adenocarcinoma (PRAD) RNA-Sequencing dataset published as part of The Cancer Genome Atlas (TCGA) project [30]. The RNA-Seq datasets of 20531 genes from 187 samples were downloaded from the TCGA data portal (https://tcga-data.nci.nih.gov/tcga). We desire to identify genes that are associated with pre-operative prostate-specific antigen (PSA), an important risk factor for prostate cancer. The gene expression data were preprocessed by the TCGA consortium. Tissue samples from 333 PRAD patients were sequenced using the Il-lumina sequencing instruments. The raw sequencing reads were processed and analyzed using the SeqWare Pipeline 0.7.0 and MapspliceRSEM workflow 0.7 developed by the University of North Carolina, and then aligned to the human reference genome using MapSplice[31]. The gene expression distributions of all samples are normalized to have the same 75th percentile expression values (1,000).

Using the algorithm in Algorithm 1, we obtain the estimated between-sample normalization factors ${\widehat{d}}_j\prime s$ and regression coefficient ${\widehat{\beta }}_i$ for each gene i. We then substitute ${\widehat{d}}_j\prime s$ into model (2), and for each gene i compute the p-value by testing the null hypothesis that the slope of the regression line is equal to zero, i.e., β_{i =} 0. We determine a gene is differentially expressed if the p value associated with its linear regression model is less than 0.05/m. Here the threshold 0.05/m is determined using the Bonferroni correction to adjust for multiple significant tests and to achieve a desired family-wise error rate of 0.05. The relations between the sets of differentially expressed genes selected by edgeR, DESeq2, limma-voom and ELMSeq are depicted in Fig. 2.

Figure 2:

Venn diagram showing the relation between the set of differentially expressed genes detected by edgeR, DESeq2, limma-voom and ELMSeq.

Nine genes are uniquely detected by ELMSeq: RIC3, ALDH1A2, BCL11A, CDH3, DIRAS3, EPHA5, CEACAM1, PRSS16, and AJAP1. For most of these genes, evidence has also been reported in the literature on their association with prostate cancer. For example, the genes ALDH1a2 [32] and CEACAM1 [33] are reported to be tumor suppressors in prostate cancer: underexpression of these genes promote prostate cancer cell proliferation.

Twelve genes are detected by all four methods: KANK4, RHOU, TPT1, SH2D3A, EEF1A1P9, ZCWPW1, ZNF454, RACGAP1, PTPLA, POC1A, AURKA and TIMM17A. The common genes detected by three methods are: six genes CDK1, FAM111B, MLF1IP, PRC1, DTL, RAD54B by edgeR, DESeq2, and limma-voom, three genes SH3RF2, ATCAY and PCP4 by edgeR, DESeq2 and ELMSeq, three genes FERMT1, FOXA3 and LRAT by edgeR, limma-voom and ELMSeq, and one gene IPO9 by DESeq2, limma-voom and ELMSeq. For most of these genes, evidence has also been reported in the literature on their association with prostate cancer. For example, the silencing of gene RHOU decreases the invasion, proliferation and motility of prostate cancer cells[34].

6. DISCUSSION

A unified statistical model is proposed for joint between sample normalization and DE detection of RNA-seq data. The sample-specific normalization factors are modeled as unknown parameters and jointly estimated together with DE detection. As a result, the model is robust against normalization errors and is independent of the units (i.e., counts, CPM/RPM, RPKM/FPKM or TPM) in which gene expression levels are summarized.

For the model with a single treatment condition, we introduce the L1 penalty to the linear regression model. The L1 penalty favors sparse solutions (forces some coefficients to be exactly zero). This is desirable since many genes are not differentially expressed. From a Bayesian point of view, the lasso penalty corresponds to a Laplace (double exponential centred at zero) prior over the regression coefficients. By contrast, existing methods do not exploit the sparsity information. We also extend the simple linear regression model to multiple linear regression model to accommodate multiple treatment conditions. Two types of penalty functions are introduced. In the first one only one covariate is of interest while all other covariates are treated as confounding factors. We are interested in testing whether that specific covariate is associated with differential expression. In the second case all covariates are of interest (there are no confounding covariates) and we are interested in testing whether any covariate affects the differential expression of a gene.

Simulation studies show that the proposed methods always perform better than or comparably to existing methods in terms of AUC. The performance gain increases with a larger sample size or higher signal to noise ratio, and is more significant when a large proportion of genes are differentially expressed in an asymmetric manner.

The R codes of the algorithms described in the paper are available for download at http://www-personal.umich. edu/~jianghui/lr-ADMM/.

Biographies

Kefei Liu received B.Sc. in mathematics and Ph.D. in Electronic Engineering from Wuhan University in 2006 and City University of Hong Kong in 2013, respectively. He is currently a Postdoctoral Research Fellow at the Center for Computational Biology and Bioinformatics, Indiana University School of Medicine. Before joining IU, he worked as a Postdoctoral Research Associate at The Biodesign Institute of Arizona State University and the Department of Computational Medicine and Bioinformatics of University of Michigan. His current research interests include machine learning, optimization, tensor decompositions and their applications in biomedical data analysis.

Jieping Ye received the Ph.D. degree in computer science from the University of Minnesota, Twin Cities, MN, USA, in 2005.

He is an Associate Professor of Department of Computational Medicine and Bioinformatics and Department of Electrical Engineering and Computer Science at the University of Michigan, Ann Arbor, MI, USA. His research interests include machine learning, data mining, and biomedical informatics. Dr. Ye has served as Senior Program Committee/Area Chair/Program Committee Vice Chair of many conferences including NIPS, ICML, KDD, IJCAI, ICDM, SDM, ACML, and PAKDD. He serves as a PC Co-Chair of SDM 2015. He serves as an Associate Editor for IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING and IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, and serves as an Action Editor for Data Mining and Knowledge Discovery. He won the NSF CAREER Award in 2010. His papers have been selected for the outstanding student paper at ICML in 2004, the KDD best research paper honorable mention in 2010, the KDD best research paper nomination in 2011 and 2012, the SDM best research paper runner up in 2013, the KDD best research paper runner up in 2013, and the KDD best student paper award in 2014.

Yang Yang is a Ph.D. candidate at Beihang University. Now he is a visiting student under the supervision of distinguished professor Philip S. Yu at the University of Illinois at Chicago. He got his bachelor’s and master’s degree from Xidian University. His research interests are social network analysis, machine learning, and complex networks.

Li Shen holds a B.S. degree from Xi’an Jiao Tong University, an M.S. degree from Shanghai Jiao Tong University, and a Ph.D. degree from Dartmouth College, all in Computer Science. He is an Associate Professor of Radiology and Imaging Sciences at Indiana University School of Medicine. His research interests include medical image computing, bioinformatics, data mining, network science, systems biology, brain imaging genomics, and brain connectomics.

Hui Jiang is an Assistant Professor in the Department of Biostatistics at University of Michigan. He received his Ph.D. in Computational and Mathematical Engineering from Stanford University in 2009. He received his B.S. and M.S. in Computer Science from Peking University. Before joining the University of Michigan in 2011, he was a postdoctoral scholar in the Department of Statistics and Genome Technology Center at Stanford University. He is interested in developing statistical and computational methods for the analysis of large-scale biological data generated using modern high-throughput technologies.