Likelihood-Based Approach to Gene Set Enrichment Analysis with a Finite Mixture Model

Abstract

In this paper, we study a parametric modeling approach to gene set enrichment analysis. Existing methods have largely relied on nonparametric approaches employing, e.g., categorization, permutation or resampling-based significance analysis methods. These methods have proven useful yet might not be powerful. By formulating the enrichment analysis into a model comparison problem, we adopt the likelihood ratio-based testing approach to assess significance of enrichment. Through simulation studies and application to gene expression data, we will illustrate the competitive performance of the proposed method.

1 Introduction

Differential gene expression data analysis is a mainstream of microarray experiments. The classical statistical method is to test one gene at a time, compute a p-value for each gene and then adjust to a multiple comparison through controlling the familywise error rate or false discovery rate (FDR, [3]). Although single gene analysis gives many important insights, it has a few limitations [25]. A number of genes which contribute to subtle changes in expression may not be detected because cut-off is determined after a correction for multiple testing. On the other hand, statistical analysis results in a long list of significant genes, and it is not easy to interpret and figure out any genetic patterns. Often a set of genes jointly influence a biological process or a critical function of metabolic pathways, and single-gene manner may ignore these. Recently many researchers have proposed methods to address challenges of gene set-based analyses. These approaches are often based on gene sets which have already been annotated by functional categories and yield more biologically interpretable result. One of the main research questions for gene set inference is called gene set enrichment analysis (GSEA): we want to evaluate whether the gene set is enriched in terms of certain characteristic of our interest (e.g., differential expression) relative to the other (random) gene sets.

A widely used approach starts from the list of differentially expressed genes derived from single gene analysis, and then evaluates over-representation of a gene set within a list of genes using Fisher’s exact test, hypergeometric test, or other independent tests in a 2 × 2 contingency table. This approach has been modified by many authors (see, e.g., [13] for a review), but the results of significance could be highly dependent on the selected cutoff value and we may lose information as a result of discretizing continuous values. An alternative approach is based on distribution comparisons. Typically a gene score, known as the local statistic for each gene that measures the difference of that gene’s expression across different experimental conditions, is computed. Then a gene set score (global statistic) associated with local statistics within a gene set is compared to those of its complement. Several different variations of testing methods have been developed (see, e.g., [2, 19, 22, 25] and [6]). Among the existing methods, the random set-based methods proposed by [9] and [20] have standardized test statistics, which are then compared to random gene sets with significance assessed by permutation and random sampling. These random set-based methods are state-of-the-art currently in the field. In this paper, we will approach the GSEA under a likelihood-based testing framework, and develop a parametric statistical method for enrichment analysis, which could offer very competitive performance by combining information across all genes.

The rest of the paper is organized as follows. Statistical methods are introduced in Sect. 2, and we develop efficient numerical algorithms for model estimation in Sect. 3. Section 4 is devoted to simulation studies and Sect. 5 discusses applications to a leukemia and p53 gene expression data. We end the paper with a discussion in Sect. 6. All technical details are delegated to the Appendix.

2 Statistical Methods

Gene pathway typically consists of a set of genes that jointly influence the system function. Genes are often divided into sets with similar functions based on their annotation information (e.g., the Gene Ontology, [1]). For the following discussion, we will summarize them as providing the gene set information.

Consider a two-class microarray data, and denote the normal transformed two-sample t -statistics for testing differential expressions as z_i for gene i = 1, …, m. We propose to model z_i with the following finite normal mixture model:

$${\displaystyle \sum _{k=0}^K{\theta }_k{f}_k(z),f_k=N({\mu }_k,{\sigma }_0^2),{\theta }_k>0,{\displaystyle \sum _{k=0}^K{\theta }_k=1}.}$$ (1)

Here the first component, $N({\mu }_0,{\sigma }_0^2)$, empirically models null genes, which is different from theoretical null (standard normal distribution) and could take into account the potential dependence among genes [7]. We can interpret θ₀ as the proportion of null genes, and θ_k the proportion of genes with μ_k magnitude of differential expression. In principle, the collection of all μ_k will capture the heterogeneity of differential expressions across all genes. We choose K based on BIC [23].

In enrichment analysis, we try to test whether a given gene set A is significantly different from any random gene set. Note that a random gene set can be treated as a random sampling from all genes. Thus comparing the given set A to a random set is equivalent to comparing A to all genes, which is again equivalent to comparing A to other genes (since A is a subset of all genes). Conceptually the (modified) two-sample t -statistics of genes in a given set can be modeled by a similar finite mixture model with different proportions of each component,

$${\displaystyle \sum _{k=0}^K{\nu }_k{f}_k(z),{\displaystyle \sum _{k=0}^K{\nu }_k}=1.}$$ (2)

Under no enrichment, the gene set A and any random gene set have the same proportion of differentially expressed genes. Therefore gene set A and all the other genes (denoted as A^c) can be modeled, respectively, with

$${\nu }_0{f}_0(z)+{\displaystyle \sum _{k=1}^K{\nu }_{jk}{f}_k(z),{\nu }_0+{\displaystyle \sum _{k=1}^K{\nu }_{jk}=1,j=1,2.}}$$ (3)

Under enrichment, the gene sets A and A^c have different proportions of differentially expressed genes, and hence can be modeled separately with

$${\displaystyle \sum _{k=0}^K{\eta }_{jk}{f}_k(z),{\displaystyle \sum _{k=0}^K{\eta }_{jk}=1,j=1,2.}}$$ (4)

Enrichment analysis corresponds to evaluating η₁₀ = η₂₀, which can be tested by likelihood ratio statistic, e_A, comparing models (3) and (4). The significance of e_A can be approximately assessed using chi-square distribution with one degree of freedom. Enrichment analysis is an one-sided test: whether gene set A is enriched with more differentially expressed genes compared to a random set. Therefore we adjust the p-value calculation as 0.5 + F(e_A; 1)/2 when η̂₁₀ ≥ η̂₂₀, and 0.5 − F(e_A; 1)/2 otherwise, where F·(·; df) is the ${\chi }_{df}^2$ distribution function.

In the proposed model, we have assumed that the variance of individual gene test statistics z_i conditional on the mean expression is fixed and the different mixing proportions will capture the varying variation of different gene sets. Therefore it is important that we allow the individual mixing proportions to vary across different gene sets.

In the following we discuss estimation of the empirical null distribution, and EM algorithms [5] for solving the proposed models (1), (2), and (3).

3 Model Estimation

3.1 Empirical Null Distribution Estimation and Finite Mixture Model Fitting

Efron [8] proposed two methods for estimating (θ₀, μ₀, σ₀): the geometric and analytical approaches. The geometric approach approximates the marginal log density with a quadratic curve near zero. The analytical approach is based on a truncated normal model by assuming non-null distribution has zero support in a pre-chosen small interval around zero. The geometric approach yields almost unbiased estimates if θ₀ exceeds 0.9, but it has large variation for estimating μ₀. The analytical approach generally gives more stable estimates while it depends on the pre-chosen interval. Both methods have been implemented in the R package, locfdr. In our simulation studies, we have observed that the analytical approach gives satisfactory results.

Given K and estimated empirical null distribution parameters $({\widehat{\theta }}_0,{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)$, we can estimate (μ_k, θ_k) for model (1) iteratively based on the EM algorithm as follows (see the Appendix for technical details)

$${\theta }_k\Leftarrow (1-{\widehat{\theta }}_0)\frac{{\displaystyle {\sum }_{i=1}^m{T}_{k,i}}}{{\displaystyle {\sum }_{j=1}^K{\displaystyle {\sum }_{i=1}^m{T}_{j,i}}}},{\mu }_k\Leftarrow \frac{{\displaystyle {\sum }_{i=1}^m{T}_{k,i}{z}_i}}{{\displaystyle {\sum }_{i=1}^m{T}_{k,i}}},k=1,\dots ,K,$$

where

$$T_{k,i}=\frac{{\theta }_k{f}_k(z_i;{\mu }_k,{\widehat{\sigma }}_0^2)}{{\widehat{\theta }}_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{j=1}^K{\theta }_j{f}_j(z_i;{\mu }_j,{\widehat{\sigma }}_0^2)}},k>0,$$

$$f_j(z;\mu ,{\sigma }^2)=\frac{1}{\sigma }\varphi \left(\frac{z-\mu }{\sigma }\right).$$

Here ϕ (·) is the standard normal distribution density function.

Occasionally, analytical approach implemented in locfdr gives abnormal estimate of θ̂₀ which is larger than 1. We then estimate θ₀ in the EM algorithm together with other parameters as follows:

$${\theta }_k\Leftarrow \frac{1}{m}{\displaystyle \sum _{i=1}^m{T}_{k,i},}k\ge 0,{\mu }_k\Leftarrow \frac{{\displaystyle {\sum }_{i=1}^m{T}_{k,i}{z}_i}}{{\displaystyle {\sum }_{i=1}^m{T}_{k,i}}},k>0,$$

where

$$T_{0,i}=\frac{{\theta }_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)}{{\theta }_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{j=1}^K{\theta }_j{f}_j(z_i;{\mu }_j,{\widehat{\sigma }}_0^2)}}.$$

3.2 Gene Set Model Fitting

Given estimated $({\widehat{\mu }}_0,{\widehat{\sigma }}_0^2,{\widehat{\mu }}_k)$ based on all genes, we can estimate the individual model (2) for a given set A iteratively as follows (see the Appendix for technical details)

$${\nu }_k\Leftarrow \frac{{\displaystyle {\sum }_{i\in A}{T}_{k,i}}}{m_A},$$

where m_A is the size of set A and for gene i in set A

$$T_{k,i}=\frac{{\nu }_k{f}_k(z_i;{\widehat{\mu }}_k,{\widehat{\sigma }}_0^2)}{{\displaystyle {\sum }_{j=0}^K{f}_j(z_i;{\widehat{\mu }}_j,{\widehat{\sigma }}_0^2)}}.$$

3.3 Model Fitting for a Gene Set and All the Other Genes Under no Enrichment

Under no enrichment, we can similarly estimate the mixture model (3) using the EM algorithm. Denote by A^c the complement of set A. Let

$$T_{0,i}=\frac{{\nu }_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)}{{\nu }_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{l=1}^K{\nu }_{1l}{f}_l(z_i;{\widehat{\mu }}_l,{\widehat{\sigma }}_0^2)}},i\in A,$$

$$T_{k,i}=\frac{{\nu }_{1k}{f}_k(z_i;{\widehat{\mu }}_k,{\widehat{\sigma }}_0^2)}{{\nu }_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{l=1}^K{\nu }_{1l}{f}_l(z_i;{\widehat{\mu }}_l,{\widehat{\sigma }}_0^2)}},i\in A,k>0,$$

$$T_{0,j}=\frac{{\nu }_0{f}_0(z_j;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)}{{\nu }_0{f}_0(z_j;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{l=1}^K{\nu }_{2l}{f}_l(z_j;{\widehat{\mu }}_l,{\widehat{\sigma }}_0^2)}},j\in A^c,$$

$$T_{k,j}=\frac{{\nu }_{2k}{f}_l(z_j;{\widehat{\mu }}_k,{\widehat{\sigma }}_0^2)}{{\nu }_0{f}_0(z_j;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{l=1}^K{\nu }_{2l}{f}_l(z_j;{\widehat{\mu }}_l,{\widehat{\sigma }}_0^2)}},j\in A^c,k>0.$$

We then iteratively solve parameters as follows (see the Appendix for technical details)

$${\nu }_0\Leftarrow \frac{1}{m}\left({\displaystyle \sum _{i\in A}{T}_{0,i}}+{\displaystyle \sum _{j\in A^c}{T}_{0,j}}\right),$$

$${\nu }_{1k}\Leftarrow (1-{\nu }_0)\frac{{\displaystyle {\sum }_{i\in A}{T}_{k,i}}}{{\displaystyle {\sum }_{l=1}^K{\displaystyle {\sum }_{i\in A}{T}_{l,i}}}},$$

$${\nu }_{2k}\Leftarrow (1-{\nu }_0)\frac{{\displaystyle {\sum }_{j\in A^c}{T}_{k,j}}}{{\displaystyle {\sum }_{l=1}^K{\displaystyle {\sum }_{j\in A^c}{T}_{l,j}}}}.$$

Next we conduct a simulation study to compare the proposed likelihood-based method (denoted as Lrt) to the GSA approach (using the maxmean test statistic) studied at [9].

4 Simulation Study

For 2 × 10⁴ genes from two groups each with n samples, we simulate their expressions based on the conditional normal distribution. Expression variance ${\sigma }_j^2$ is simulated individually for each gene from a χ² distribution with 10 degrees of freedom. This mimics the commonly observed large variation of gene variances in microarray data. We simulate the dependence by dividing genes into 200 blocks each with m_g = 100 genes and within-block pairwise gene correlation being ρ_g. Gene block correlation parameter ρ_g is randomly simulated from a Beta distribution, Beta(2, 2). We randomly set m_gθ₀ genes in each block as null. The standardized differences of non-null genes, (μ_1j − μ_2j)/σ_j are randomly simulated from a mixture of two scaled Beta distributions, 0.5 + Beta(2, 2) and −0.5−Beta(2, 2), with equal probabilities.

We consider three types of gene set, each with m_e genes and different dependence structures. The first type has similar dependence structure as all genes and is sampled from all G = 200 blocks. The other two types of gene set exhibit relatively stronger dependence and are sampled from the first G = 30 and 50 blocks respectively. This mimics the commonly observed gene pathways with genes highly interacting with each other. For every type of gene set, we consider two enrichment scenarios. Firstly, the non-null genes in the gene set are randomly sampled from all differentially expressed genes. Secondly, the non-null genes in the gene set are all up-regulated (i.e., the gene set is enriched with different differential expression categories compared to all the other genes).

For size evaluation, we randomly sample m_eθ₀ null and m_e(1−θ₀) non-null genes, and compute the enrichment p-values based on Lrt and GSA in each simulation. For power comparison, we consider gene set with randomly sampled m_eθ_e null and m_e (1 − θ_e) non-null genes.

In the simulation, we set n = 15, θ₀ = 0.9, and consider two sets of scenarios: (1) θ_e = (0.86, 0.82, 0.78) and m_e = (100, 200, 300), and (2) θ_e = (0.8, 0.7, 0.6), and m_e = (10, 20, 50), which will investigate the performance under different gene set sizes. In the second scenario with relatively small gene set, θ_e is selected to define a meaningful number of differentially expressed genes.

The proposed Lrt performs better than GSA under all simulation settings, and we have observed similar patterns. Here we report the results for m_e = (100, 200, 300) and non-null genes are sampled from all differentially expressed genes. The complete results are provided at the supplementary materials.

Table 1 summarizes the estimated sizes for true Type I error α = (0.01, 0.05, 0.10) over 1000 simulations. We can see that both methods have approximately the right size. The proposed Lrt in general is more conservative compared to GSA, which could over estimate the Type I error under relatively large significance level.

Table 1.

Estimated type I error of Lrt and GSA over 1000 simulations (listed within parentheses are the standard errors). Non-null genes are randomly sampled from all differentially expressed genes

			α̂
	α		0.01	0.05	0.1
me = 100	G = 200	Lrt GSA	0.002 (5e-5) 0.009 (3e-4)	0.014 (4e-4) 0.076 (2e-3)	0.038 (1e-3) 0.176 (5e-3)
	G = 50	Lrt GSA	0.005 (2e-4) 0.013 (4e-4)	0.019 (6e-4) 0.085 (2e-3)	0.046 (1e-3) 0.172 (5e-3)
	G = 30	Lrt GSA	0.005 (2e-4) 0.012 (4e-4)	0.026 (8e-4) 0.071 (2e-3)	0.068 (2e-3) 0.156 (4e-3)
me = 200	G=200	Lrt GSA	0.001 (3e-5) 0.012 (4e-4)	0.011 (3e-4) 0.077 (2e-3)	0.035 (1e-3) 0.166 (4e-3)
	G = 50	Lrt GSA	0.002 (6e-5) 0.012 (4e-4)	0.034 (1e-3) 0.066 (2e-3)	0.071 (2e-3) 0.173 (5e-3)
	G = 30	Lrt GSA	0.012 (4e-4) 0.009 (3e-4)	0.046 (1e-3) 0.067 (2e-3)	0.089 (3e-3) 0.140 (4e-3)
me = 300	G = 200	Lrt GSA	0.001 (3e-5) 0.007 (2e-4)	0.009 (3e-4) 0.072 (2e-3)	0.030 (9e-4) 0.158 (4e-3)
	G = 50	Lrt GSA	0.007 (2e-4) 0.006 (2e-4)	0.028 (9e-4) 0.060 (2e-3)	0.072 (2e-3) 0.149 (4e-3)
	G = 30	Lrt GSA	0.020 (6e-4) 0.008 (3e-4)	0.051 (2e-3) 0.055 (2e-3)	0.105 (3e-3) 0.126 (3e-3)

Figures 12 and 3 summarize the power averaged over 1000 simulations for m_e = (300, 200, 100), respectively. The red solid/dashed/dotted lines are estimated power for Lrt under θ_e = (0.86, 0.82, 0.78), and black lines are the corresponding power for GSA. Overall we can see that the proposed Lrt has very competitive performance compared to GSA under all settings. In general both methods have reduced power with increasing gene interactions within a given set and decreasing gene set size m_e. With increasing gene set size m_e, we observe relatively larger performance difference between the two methods.

Fig. 1.

Power of Lrt and GSA averaged over 1000 simulations for m_e = 300. The horizontal axis corresponds to type I error

Fig. 2.

Power of Lrt and GSA averaged over 1000 simulations for m_e = 200. The horizontal axis corresponds to type I error

Fig. 3.

Power of Lrt and GSA averaged over 1000 simulations for m_e = 100. The horizontal axis corresponds to type I error

Next we analyze a leukemia and p53 gene expression microarray data to illustrate the relative performance of the proposed likelihood-based method and GSA.

5 Application to Leukemia and p53 Gene Expression Data

The leukemia gene expression data reported at [15] measured the expressions of 45101 genes from five paired controls and Meis1-knockdown cases. We identified 522 gene pathways from C2 functional collection in the Molecular Signature Database [25]. Pathway sizes range from 2 to 365 genes. We analyze in total 357 pathways that have more than 10 genes.

To improve the accuracy of the normal distribution approximation, we apply the empirical Bayes modeling approach of [24], which computed a moderated t -statistic, t_i, for gene i by pooling information across all genes for an improved sample variance estimate (implemented in the R package, limma). We then apply the normal distribution transformation to the moderated t -statistic, z_i = ϕ⁻¹ (T_d (t_i)), where ϕ (·) is the standard normal distribution function and T_d (·) is the t -distribution function with d degrees of freedom. Here, the degree of freedom d is estimated from all genes using the empirical Bayes modeling approach.

When applied to the leukemia microarray data, controlling FDR at 0.05/0.1, the proposed Lrt detected 29/51 significant gene sets, while no gene pathway is identified as significant with GSA. Figure 4 shows the number of significant pathways versus the estimated FDR for Lrt and GSA.

Fig. 4.

The number of significant pathways versus FDR for the leukemia data

Table 2 lists the top 29 significant pathways identified by the proposed method. Many of them are closely related to cancer development. For example, several identified pathways are related to cell cycle, which is known to play an important role in cancer development: cell cycle machinery controls cell proliferation, and cancer is a disease of inappropriate cell proliferation [4]. The atrbrcaPathway is also closely related to cell cycle and cancer. Specifically the ATR gene serves as a checkpoint kinase that halts cell cycle progression and induces DNA repair when DNA is damaged. Loss of ATR results in a loss of checkpoint control in response to DNA damage, leading to cell death (see http://www.biocarta.com/pathfiles/h_ATRBRCAPATHWAY.asp). Liu et al. [18] have shown the important role of ATR in cell cycle control in MLL/Meis1 leukemia. The DNA damage signaling pathway is linked to DNA repair, cell-cycle control, growth arrest, and plays an important role in cancer development.

Table 2.

Top 29 most significant pathways identified with the proposed likelihood-based method

Pathway	# genes	p-value
Cell_Cycle	73	2E-13
CR_CELL_CYCLE	74	5E-11
atrbrcaPathway	18	7E-07
CR_REPAIR	35	4E-06
GLUT_DOWN	230	5E-06
cell_cycle_checkpoint	22	1E-05
DNA_DAMAGE_SIGNALING	85	1E-05
HTERT_UP	94	2E-05
CR_DNA_MET_AND_MOD	20	3E-05
LEU_DOWN	130	3E-05
cell_cycle_regulator	20	6E-05
rbPathway	11	0.0001
cell_cycle_arrest	27	0.0005
hdacPathway	28	0.0008
RAP_DOWN	169	0.0010
SA_REG_CASCADE_OF_CYCLIN_EXPR	12	0.0011
il7Pathway	16	0.0015
mRNA_processing	40	0.0018
shh_lisa	15	0.0019
GLUCOSE_DOWN	122	0.0020
MAP00020_Citrate_cycle_TCA_cycle	16	0.0022
cellcyclePathway	22	0.0022
mRNA_splicing	45	0.0023
SIG_IL4RECEPTOR_IN_B_LYPHOCYTES	26	0.0025
caspasePathway	21	0.0028
crebPathway	25	0.0030
eif4Pathway	24	0.0034
MAP00240_Pyrimidine_metabolism	38	0.0035
nfatPathway	49	0.0040

The p53 expression data are available at http://www.broadinstitute.org/gsea/datasets.jsp, and consists of 12625 genes from 33 p53 mutant and 17 p53+ cancer cell lines. We analyze in total 453 pathways that have more than 10 genes from the C2 functional collection.

Controlling FDR at 0.01/0.05, the proposed Lrt detected 26/50 significant gene sets, and GSA detected 3/8 significant gene sets. Figure 5 shows the number of significant pathways versus the estimated FDR for Lrt and GSA. Table 3 listed the top ranked pathways by Lrt and GSA (controlling FDR at 0.05).

Fig. 5.

The number of significant pathways versus FDR for the p53 data

Table 3.

Significantly enriched pathways for the p53 data identified by Lrt and GSA (FDR ≤ 0.05)

Lrt
Pathway	# genes	p-value
P53_UP	49	6.3E-09
p53Pathway	43	1.2E-08
rasPathway	41	4.3E-08
GLUT_UP	294	4.0E-07
SA_PROGRAMMED_CELL_DEATH	24	5.5E-07
mitochondriaPathway	32	6.4E-07
HTERT_UP	135	6.7E-07
p53hypoxiaPathway	36	7.3E-07
SA_G1_AND_S_PHASES	26	2.3E-06
ceramidePathway	48	1.2E-05
radiation_sensitivity	61	1.3E-05
fmlppathway	65	1.3E-05
hivnefPathway	100	3.0E-05
DNA_DAMAGE_SIGNALING	154	3.0E-05
hsp27Pathway	34	3.9E-05
XINACT_MERGED	26	3.9E-05
insulinPathway	44	1.1E-04
badPathway	43	1.3E-04
integrinPathway	60	1.6E-04
igf1Pathway	47	2.9E-04
g2Pathway	41	3.4E-04
atmPathway	43	3.6E-04
tcrPathway	85	3.6E-04
Glycogen_Metabolism	50	4.6E-04
tsp1Pathway	17	4.6E-04
tall1Pathway	23	5.0E-04
cdmacPathway	32	8.2E-04
metPathway	71	1.0E-03
at1rPathway	64	1.2E-03
ngfPathway	36	1.3E-03
cxcr4Pathway	41	1.3E-03
bcl2family_and_reg_network	50	1.4E-03
eif2Pathway	12	1.5E-03
mef2dPathway	27	1.5E-03
spryPathway	27	1.8E-03
eea1Pathway	12	1.8E-03
CR_DEATH	114	2.1E-03
pgc1aPathway	35	2.6E-03
relaPathway	26	2.7E-03
rnaPathway	17	3.2E-03
ecmPathway	36	3.3E-03
INSULIN_2F_UP	200	3.5E-03
pyk2Pathway	57	3.7E-03
SA_FAS_SIGNALING	14	4.0E-03
chemicalPathway	46	4.0E-03
deathPathway	56	4.5E-03
Cell_Cycle	115	4.6E-03
breast_cancer_estrogen_signaling	162	4.7E-03
tollPathway	45	5.1E-03
SA_B_CELL_RECEPTOR_COMPLEXES	46	5.3E-03

GSA
Pathway	# genes	p-value	Lrt rank
P53_UP	49	8.0E-6	1
p53Pathway	43	9.0E-6	2
p53hypoxiaPathway	36	1.0E-5	8
badPathway	43	1.1E-4	18
radiation_sensitivity	61	3.3E-4	11
SA_PROGRAMMED_CELL_DEATH	24	3.5E-4	5
rasPathway	41	7.2E-4	3
SA_G1_AND_S_PHASES	26	7.4E-4	9

We can see that the eight significant pathways identified by GSA are all detected by Lrt. Many of the identified pathways by Lrt are Biocarta pathways, e.g., ceramidePathway, fmlppathway, hivnefPathway, hsp27Pathway, insulinPathway, badPathway, integrinPathway, igf1Pathway, g2Pathway, and atmPathway etc. Most of them have been studied and shown related to the p53 gene (see, e.g., [10–12, 14, 16, 17, 21]). For example, the ATM gene interacts with p53 gene to cause the disease ataxia telangiectasia which involves an inherited predisposition to some cancers (http://www.biocarta.com/pathfiles/h_atmPathway.asp). The hsp27 gene modulated the p53 signaling [21]. The igf1Pathway highly interacts with the p53 signaling pathway and they regulate cell growth, proliferation, and death [16]. The g2Pathway consists of genes involved in the cell cycle G2/M checkpoint event, and the p53 gene plays an important role (http://www.biocarta.com/pathfiles/h_g2Pathway.asp).

6 Discussion

The GSEA approach firstly proposed and studied at [19] and [25] provides a very novel way to interpret the large-scale gene expression data. Compared to individual gene oriented analysis, gene set-based inference can often produce meaningful and easy to interpret results and provide additional insights into the underlying biological processes. Many simple and ad hoc statistical methods based on categorization are becoming routinely used in practice (e.g., the widely used hypergeometric testing approach) for gene set significance assessment. Nonparametric methods based on permutation and random sampling have been proposed and proven to be more powerful but might be quite computing intensive. We approach the GSEA from a likelihood framework and transform it into a model comparison problem, which can be addressed using the powerful likelihood ratio test approach. Through applications and simulation studies we have demonstrated the competitive performance of the proposed method. An interesting extension is to develop similar method for multi-group comparison problems, which can be approached using a finite chi-square distribution mixture model. We will report the results elsewhere in the future.

Appendix

EM Algorithm for Estimating the Finite Mixture Model

We begin with the finite mixture model in (1) given $({\widehat{\theta }}_0,{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)$ and K. Define indicators w_ik ∈ {0, 1} following a multinomial distribution, Pr(w_ik = 1) = θ_k, ${\displaystyle {\sum }_{k=0}^K{w}_{ik}}=1$, and conditionally we assume z_i |w_ik = 1 ~ f_k. The complete data likelihood function for (z_i, w_ik) can be written as

$${\displaystyle \prod _{i=1}^m{\{{\widehat{\theta }}_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)\}}^{w_{i0}}}{\displaystyle \prod _{k=1}^K{\{{\theta }_k{f}_k(z_i;{\mu }_k,{\widehat{\sigma }}_0^2)\}}^{w_{ik}}}.$$

In the E-step, the conditional probabilities can be checked to be

$$T_{0,i}=\frac{{\widehat{\theta }}_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)}{{\widehat{\theta }}_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{k=1}^K{\theta }_k{f}_k(z_i;{\mu }_k,{\widehat{\sigma }}_0^2)}},$$

$$T_{k,i}=\frac{{\theta }_k{f}_k(z_i;{\mu }_k,{\widehat{\sigma }}_0^2)}{{\widehat{\theta }}_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{j=1}^K{\theta }_j{f}_j(z_i;{\mu }_j,{\widehat{\sigma }}_0^2)}}.$$

In the M-step, the conditional expected log likelihood can be checked to be proportional to

$${\displaystyle \sum _{i=1}^m\left\{T_{0,i}\left(\text{log}{\widehat{\theta }}_0-\text{log}{\widehat{\sigma }}_0-\frac{{(z_i-{\widehat{\mu }}_0)}^2}{2{\widehat{\sigma }}_0^2}\right)+{\displaystyle \sum _{k=1}^K{T}_{k,i}}\left(\text{log}{\theta }_k-\text{log}{\widehat{\sigma }}_0-\frac{{(z_i-{\mu }_k)}^2}{2{\widehat{\sigma }}_0^2}\right)\right\},}$$

which can be easily verified to be maximized by

$${\widehat{\theta }}_k=(1-{\widehat{\theta }}_0)\frac{{\displaystyle {\sum }_{i=1}^m{T}_{k,i}}}{{\displaystyle {\sum }_{j=1}^K{\displaystyle {\sum }_{i=1}^m{T}_{j,i}}}},{\widehat{\mu }}_k=\frac{{\displaystyle {\sum }_{i=1}^m{T}_{k,i}{z}_i}}{{\displaystyle {\sum }_{i=1}^m{T}_{k,i}}},k\ge 1.$$

Given only $({\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)$ with θ₀ also being a parameter, we have

$$({\widehat{\theta }}_0,\dots ,{\widehat{\theta }}_K)=\underset{{\theta }_k}{\text{arg max}}{\displaystyle \sum _{k=0}^K{\displaystyle \sum _{i=1}^m{T}_{k,i}\text{log}{\theta }_k},}$$

$${\widehat{\mu }}_k=\underset{{\mu }_k}{\text{arg min}}{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{k=1}^K{T}_{k,i}{(z_i-{\mu }_k)}^2}},k>0.$$

We can easily check that

$${\widehat{\theta }}_k=\frac{1}{m}{\displaystyle \sum _{i=1}^m{T}_{k,i}},k\ge 0,{\widehat{\mu }}_k=\frac{{\displaystyle {\sum }_{i=1}^m{T}_{k,i}{z}_i}}{{\displaystyle {\sum }_{i=1}^m{T}_{k,i}}},k>0.$$

EM Algorithm for Estimating the Gene Set Model

The complete data likelihood function for a gene set A given $({\widehat{\theta }}_0,{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2,{\widehat{\mu }}_k)$ is

$${\displaystyle \prod _{i\in A}{\displaystyle \prod _{k=0}^K{\{{\nu }_k{f}_k(z_i;{\widehat{\mu }}_k,{\widehat{\sigma }}_0^2)\}}^{w_{ik}}}}.$$

The conditional expected log likelihood can easily be checked to be

$${\displaystyle \sum _{i\in A}{\displaystyle \sum _{k=0}^K{T}_{k,i}\text{log}{\nu }_k}},T_{k,i}=\frac{{\nu }_k{f}_0(z_i;{\widehat{\mu }}_k,{\widehat{\sigma }}_0^2)}{{\displaystyle {\sum }_{0=1}^K{\nu }_j{f}_j(z_i;{\widehat{\mu }}_j,{\widehat{\sigma }}_0^2)}},k\ge 0.$$

We can easily verify that

$${\widehat{\nu }}_k=\frac{{\displaystyle {\sum }_{i\in A}{T}_{k,i}}}{m_A},k\ge 0.$$

EM Algorithm for Estimating the Model Under no Enrichment

The complete data likelihood can be written as

$${\displaystyle \prod _{i\in A}{\{{\nu }_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)\}}^{w_{i0}}}{\displaystyle \prod _{k=1}^K{\{{\nu }_{1k}{f}_k(z_i;{\widehat{\mu }}_k,{\widehat{\sigma }}_0^2)\}}^{w_{ik}}}{\displaystyle \prod _{j\in A^c}{\{{\nu }_0{f}_0(z_j;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)\}}^{w_{j0}}}\times {\displaystyle \prod _{k=1}^K{\{{\nu }_{2k}{f}_k(z_j;{\widehat{\mu }}_k,{\widehat{\sigma }}_0^2)\}}^{w_{jk}}},$$

where ${\nu }_0+{\displaystyle {\sum }_{k=1}^K{\nu }_{lk}}=1,l=1,2$. The conditional expected log likelihood can be easily checked to be

$${\displaystyle \sum _{i\in A}\left\{T_{0,i}\log {\nu }_0+{\displaystyle \sum _{k=1}^K{T}_{k,i}}\log {\nu }_{1k}\right\}}+{\displaystyle \sum _{j\in A^c}\left\{T_{0,j}\log {\nu }_0+{\displaystyle \sum _{k=1}^K{T}_{k,j}}\log {\nu }_{2k}\right\}},$$

where

$$T_{0,i}=\frac{v_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)}{{\nu }_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{l=1}^K{\nu }_{1l}{f}_l(z_i;{\widehat{\mu }}_l,{\widehat{\sigma }}_0^2)}},i\in A,$$

$$T_{k,i}=\frac{{\nu }_{1k}{f}_k(z_i;{\widehat{\mu }}_k,{\widehat{\sigma }}_0^2)}{{\nu }_0{f}_0(z_i;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{l=1}^K{\nu }_{1l}{f}_l(z_i;{\widehat{\mu }}_l,{\widehat{\sigma }}_0^2)}},i\in A,k>0,$$

$$T_{0,j}=\frac{{\nu }_0{f}_0(z_j;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)}{{\nu }_0{f}_0(z_j;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{l=1}^K{\nu }_{2l}{f}_l(z_j;{\widehat{\mu }}_l,{\widehat{\sigma }}_0^2)}},j\in A^c,$$

$$T_{k,j}=\frac{{\nu }_{2k}{f}_k(z_j;{\widehat{\mu }}_k,{\widehat{\sigma }}_0^2)}{{\nu }_0{f}_0(z_j;{\widehat{\mu }}_0,{\widehat{\sigma }}_0^2)+{\displaystyle {\sum }_{l=1}^K{\nu }_{2l}{f}_l(z_j;{\widehat{\mu }}_l,{\widehat{\sigma }}_0^2)}},j\in A^c,k>0.$$

To maximize the conditional log likelihood, we use the Lagrange multiplier method

$$Q={\displaystyle \sum _{i\in A}\left\{T_{0,i}\log {\nu }_0+{\displaystyle \sum _{k=1}^K{T}_{k,i}\log {\nu }_{1k}}\right\}+{\displaystyle \sum _{j\in A^c}\left\{T_{0,j}\log {\nu }_0+{\displaystyle \sum _{k=1}^K{T}_{k,j}\log {\nu }_{2k}}\right\}}-{\lambda }_1\left({\nu }_0+{\displaystyle \sum _{k=1}^K{\nu }_{1k}}-1\right)-{\lambda }_2}\left({\nu }_0+{\displaystyle \sum _{k=1}^K{\nu }_{2k}}-1\right).$$

Setting the gradient vector ∇Q = 0 yields the following equations:

$$\frac{\partial Q}{\partial {\nu }_0}=\frac{{\displaystyle {\sum }_{i\in A}{T}_{0,i}}}{{\nu }_0}+\frac{{\displaystyle {\sum }_{j\in A^c}{T}_{0,h}}}{{\nu }_0}-{\lambda }_1-{\lambda }_2=0,$$

$$\frac{\partial Q}{\partial {\nu }_{1k}}=\frac{{\displaystyle {\sum }_{i\in A}{T}_{k,i}}}{{\nu }_{1k}}-{\lambda }_1=0,k>0,$$

$$\frac{\partial Q}{\partial {\nu }_{2k}}=\frac{{\displaystyle {\sum }_{j\in A^c}{T}_{k,j}}}{{\nu }_{2k}}-{\lambda }_2=0,k>0,$$

$$\frac{\partial Q}{\partial {\lambda }_1}={\nu }_0+{\displaystyle \sum _{k=1}^K{\nu }_{1k}}-1=0,$$

$$\frac{\partial Q}{\partial {\lambda }_2}={\nu }_0+{\displaystyle \sum _{k=1}^K{\nu }_{2k}}-1=0.$$

From the first three equations we can obtain

$${\nu }_0=\frac{{\displaystyle {\sum }_{i\in A}{T}_{0,i}+{\displaystyle {\sum }_{j\in A^c}{T}_{0,j}}}}{{\lambda }_1+{\lambda }_2},{\nu }_{1k}=\frac{{\displaystyle {\sum }_{i\in A}{T}_{0,i}}}{{\lambda }_1},$$

$${\nu }_{2k}=\frac{{\displaystyle {\sum }_{j\in A^c}{T}_{0,j}}}{\lambda 2},k>0.$$

When plugging these into the last two equations, we obtain

$${\widehat{\nu }}_0=\frac{{\displaystyle {\sum }_{i\in A}{T}_{0,i}+{\displaystyle {\sum }_{j\in A^c}{T}_{0,j}}}}{m},$$

and

$${\widehat{\nu }}_{1k}=(1-{\widehat{\nu }}_0)\frac{{\displaystyle {\sum }_{i\in A}{T}_{k,i}}}{{\displaystyle {\sum }_{l=1}^K{\displaystyle {\sum }_{i\in A}{T}_{l,i}}}},{\widehat{\nu }}_{2k}=(1-{\widehat{\nu }}_0)\frac{{\displaystyle {\sum }_{j\in A^c}{T}_{k,j}}}{{\displaystyle {\sum }_{l=1}^K{\displaystyle {\sum }_{j\in A^c}{T}_{1,j}}}}.$$