A Unified Mixed Effects Model for Gene Set Analysis of Time Course Microarray Experiments

Abstract

Methods for gene set analysis test for coordinated changes of a group of genes involved in the same biological process or molecular pathway. Higher statistical power is gained for gene set analysis by combining weak signals from a number of individual genes in each group. Although many gene set analysis methods have been proposed for microarray experiments with two groups, few can be applied to time course experiments. We propose a unified statistical model for analyzing time course experiments at the gene set level using random coefficient models, which fall into the more general class of mixed effects models. These models include a systematic component that models the mean trajectory for the group of genes, and a random component (the random coefficients) that models how each gene's trajectory varies about the mean trajectory.

We show that the proposed model (1) outperforms currently available methods at discriminating gene sets differentially changed over time from null gene sets; (2) provides more stable results that are less affected by sampling variations; (3) models dependency among genes adequately and preserves type I error rate; and (4) allows for gene ranking based on predicted values of the random effects. We describe simulation studies using gene expression data with “real life” correlations and we demonstrate the proposed random coefficient model using a mouse colon development time course dataset. The agreement between results of the proposed random coefficient model and the previous reports for this proof-of-concept trial further validates this methodology, which provides a unified statistical model for systems analysis of microarray experiments with complex experimental designs when re-sampling based methods are difficult to apply.

1. INTRODUCTION

To understand the temporal nature of gene regulatory mechanisms, time course microarray experiments have been used to monitor mRNA transcript abundance of many genes over time. To identify genes that are differentially expressed over time, a variety of statistical models have been proposed for analysis at the single gene level (Luan et al. 2004; Park et al. 2003; Storey et al. 2005). To help with interpretation of the results, following analyses for each gene, Fisher’s exact test is often used to test whether differentially expressed genes are significantly overrepresented in a priori defined functional groups. A popular example for the functional groups is the GO categories defined in the Gene Ontology (Ashburner et al. 2000) database, which are structured controlled vocabularies (ontologies) that describe gene products in terms of their biological processes, cellular components and molecular functions. For simplicity we use the terms “pathways”, “biological processes”, and “gene sets” interchangeably for these gene groups, although they may not be strictly equivalent. The Fisher’s exact method is implemented in several software packages such as GENMAPP (Dahlquist et al. 2002), ONTO-TOOLS (Draghici et al. 2003), WebGestalt (Zhang et al. 2005), GOTM (Zhang et al. 2004) and JMP Genomics (http://www.jmp.com/genomics). Because each gene is analyzed individually first and information from all genes in the gene set is then combined, this approach has been called a “bottom-up” approach (Liu et al. 2007). Despite the popularity of “bottom-up” approaches, they have some limitations: the assumption that genes are independent may not be tenable for tightly co-regulated gene sets; the selection of significant genes is often based on an arbitrary cutoff; and information is lost by not using continuous information in p-values measuring differential expression.

In contrast, a “top down” approach for pathway analysis requires no threshholding of gene-wise significance levels; the gene expression values from a group of genes are combined to estimate a test statistic for the gene set. By borrowing information and combining weak signals across genes in the same gene set, improved power is gained for pathway-based analysis methods.

For experiments with two groups, many top-down methods have been proposed (Barry et al. 2005; Chen et al. 2008; Goeman et al. 2004; Kim et al. 2005; Mootha et al. 2003); however, few methods can be used to analyze time course microarray experiments, especially those with multiple groups and covariate information. Recently, in an interesting paper, Hummel et al. (2008) proposed the GlobalANCOVA test which uses a permutation test coupled with gene-wise linear models to identify gene sets associated with time while accounting for the experimental design. Although the global null hypothesis:

H_0G: no gene in the group exhibits differential expression over time

is certainly useful, in practice we have found it may be rejected for too many gene sets for some microarray experiments. For example, in our application of the GlobalANCOVA test to a mouse colon development time course experiment (Section 3.2), when 10,000 permutations were used, among the 522 GO terms tested, there were 287 GO terms with permutation p-value of 0. Among the gene sets tested, larger gene sets were in particular more likely to be rejected for H_0G because they are more likely to include at least one differentially expressed gene. Furthermore, in situations where a significant gene belongs to multiple gene sets, the results may be difficult to interpret because nearly all these categories will be rejected for the global null hypothesis.

To address this difficulty, in this paper, we propose a parametric approach for testing the central null hypothesis:

H_0C: the average gene expression of a gene group is not differentially expressed over time.

We test H_0C via a class of statistical models called random coefficient models (Littell et al. 2006). These models estimate a common mean trajectory over time for all genes in the gene set and fall into the more general mixed models framework. Mixed models have been shown to be a very effective tool for the analysis of microarray experiments; see for example (Chu et al. 2002; Wang et al. 2008; Wolfinger et al. 2001).

In Section 2, we provide details for the proposed random coefficient models, including our proposals for modeling the heterogeneous correlations between genes using random effects, and the ranking of individual genes on their contributions to the gene set signal using empirical BLUP (Best Linear Unbiased Prediction) shrinkage estimation under the mixed models framework. In addition, to compare the proposed random coefficient models with currently available methods in terms of false positive rate, power, and stability, we describe the design of a simulation experiment using real microarray data where genes have “real life” correlations. In Section 3, we show (1) our random coefficient models outperform GlobalANCOVA and Fisher’s exact test at discriminating gene sets changed over time from null gene sets in terms of sensitivity and specificity; (2) the proposed models account for dependency among genes adequately and preserve the type I error rate for testing the central null hypothesis; (3) statistical inference based on these models is less affected by sampling variations. We apply the proposed random coefficient model to a mouse colon development time course microarray dataset in which gene expressions were measured in two strains of mice daily from embryonic day E13.5 to E18.5. In addition to identifying biologically meaningful GO categories that are differentially expressed over time, we also illustrate ranking of individual genes on their contributions to the gene set signal. In Section 4, we provide some concluding comments.

2. METHODS

2.1. The Random Coefficient Model

The first step in pathway analysis is to link gene identifiers in expression dataset with pre-defined biological processes or pathways such as those defined by Gene Ontology, so that genes are grouped into different gene sets according to GO terms. After pre-processing gene expression values (see details in Section 3.2), we obtain one value for each gene. To identify gene sets differentially expressed over time, for each GO term, we construct a random coefficient model:

$$\text{Model}\hspace{0.17em}1:Y_{ijk}=({\beta }_0+b_{0i})+({\beta }_1+b_{1i}){\mathit{\text{time}}}_{\text{j}}+{\mathit{\text{Array}}}_k+r_{1i}+\dots +r_{pi}+{\varepsilon }_{ijk}$$

where

• Y_ijk = log transformed value for gene i from array k at time j;

• p = is the rank of the gene-gene covariance matrix Σ;

• β₀, β₁ are fixed effects that model the mean intercept and slope for the group of genes defined by the same GO term, they describe the mean trajectory for the group. The central null hypothesis tests H₀ : β₁ = 0;

• (b_0i, b_1i)^t ∼ N(0, G) is a vector of random effects, they model how the intercept and slope for ith gene deviates from intercept and slope of the mean trajectory. These are the random coefficients; we assume an unstructured covariance matrix $\begin{array}{l}\mathit{\text{G}}=\left(\begin{array}{ll}{\sigma }_0^2\hfill & {\sigma }_{01}\hfill \\ {\sigma }_{01}\hfill & {\sigma }_1^2\hfill \end{array}\right)\\ \end{array}$ for the random intercept and slope deviations, allowing an arbitrary correlation between random intercept and slope. In this paper, since we are mainly interested in monotonic changes in groups of genes, only linear terms are specified here. However, Model 1 can be easily augmented with additional quadratic or cubic polynomial terms;

• Array₁, Array₂, …, Array_Narray ∼ independent $N(0,{\sigma }_a^2)$ are random effects that model sample variations, allowing inferences to be made to the entire population of samples from which the observed samples arise;

• r₁, …, r_p ∼ independent $N(0,{\sigma }_r^2)$ are random effects included to account for the heterogeneous covariance structure between genes (see details in Sec 2.2) and ɛ_ijk ∼ N(0, σ²) represent variations due to measurement error.

Because both fixed and random effects are included, the random coefficient model falls into the more general class of linear mixed effects model. Parameters in the linear mixed model are estimated using restricted maximum likelihood (REML) along with appropriate standard errors (Littell et al, 2006).

2.2. Modeling the Covariance Structure between Genes

One important but challenging issue in modeling gene expression from a group of genes such as those belonging to the same GO category is accommodating the dependency among genes. We propose modeling the heterogeneous covariance structure between genes using random effects {Array_k; k = 1, …, N_array} and {r₁, …, r_p} in Model 1. Note that a general representation of the linear mixed model is

$$\begin{array}{l}\mathit{\text{Y}}=\text{X}\beta +\text{Zu}+\text{e}\hfill \\ \text{u}\sim \text{N}(0,\text{G})\hfill \\ \text{e}\sim \text{N}(0,\text{R})\hfill \\ \text{Cov}[\text{u},\text{e}]=0\hfill \end{array}$$

where X, Z are design matrices for the fixed and random effects, β, u are vectors of parameters for the fixed and random effects, and e is the error term. The marginal model for Y is then Y ∼ N(Xβ, ZGZ^t + R).

In Model 1, for each Array random effect, the corresponding column in the design matrix Z is constructed as the indicator variable for the array, so that the array random effect accounts for the homogeneous covariance among all observations in the same array. The random effects r₁, …, r_p were included to account for different amount of dependencies between pairs of genes within a geneset. The design matrix corresponding to the random effects r₁, …, r_p is motivated by the theorem on Spectral Decomposition (Jolliffe 2002) which states that under regularity conditions, for any symmetric matrix Σ (with rank p), we have

$$\Sigma ={\lambda }_1{\alpha }_1{\alpha }_1^t+{\lambda }_2{\alpha }_2{\alpha }_2^t+\dots +{\lambda }_p{\alpha }_{\text{p}}{\alpha }_p^t,$$

where α_l and λ_l (l = 1, …, p) are l-th eigenvector and eigenvalue of Σ. The eigenvectors and eigenvalues of Σ are defined as vectors α_l and scalars λ_l such that Σα_l = λ_lα_l, l = 1, …, p.

In Model 1, to apply the Spectral Decomposition theorem to account for heterogenous covariance structure between genes, let Σ be the gene-gene covariance matrix, we specify the column in the design matrix corresponding to r_l to be $\sqrt{{\widehat{\lambda }}_l}{\widehat{\alpha }}_l$ where α̂_l = estimated l-th eigenvector of Σ and λ̂_l = estimated l-th eigenvalue of Σ. Figure 1 illustrates the computation of the design matrix corresponding to {r_l; l = 1, ..., 3} for a gene set with 3 genes using SAS procedure PRINCOMP.

Figure 1.

An illustration for the computation of the design matrix for the random effects {r_l; l = 1, …, p} in Model 1. This gene set has 3 genes (variables) and the dataset has 12 samples (observations). Covariance Matrix = estimated gene-gene covariance matrix ∑̂. Under “Eigenvectors”, Prin 1 = the estimated first eigenvector α̂₁ of ∑̂, and λ̂₁ = 0.09802458 is the estimated first eigenvalue of ∑̂. The column in design matrix corresponding to r₁ is then $\sqrt{0.098}{\widehat{\alpha }}_1$, note that they vary according to genes, so the random effects have sub-index i in Model 1.

Now let Z₁ = the sub-matrix of the design matrix Z corresponding to the random effects r₁, …, r_p. Assume r₁, …, r_p ∼ independent $N(0,{\sigma }_r^2)$, we then have

$$\begin{array}{l}{\mathit{\text{Z}}}_1=\left[\sqrt{{\widehat{\lambda }}_1}{\widehat{\alpha }}_1 \sqrt{{\widehat{\lambda }}_2}{\widehat{\alpha }}_2 \hspace{0.17em}\dots \hspace{0.17em} \sqrt{{\widehat{\lambda }}_p}{\widehat{\alpha }}_p\right]\\ \text{G}=\left[\begin{array}{cccc}{\sigma }_r^2& 0& \dots & 0\\ 0& {\sigma }_r^2& \dots & 0\\ ⋮& ⋮& ⋮& ⋮\\ 0& 0& \dots & {\sigma }_r^2\end{array}\right]\end{array}$$

and the contributions of the r₁, …, r_p to the covariance matrix of Y in the marginal model would be

$${\mathit{\text{Z}}}_1\mathit{\text{G}}{\mathit{\text{Z}}}_1^t={\sigma }_r^2{\widehat{\lambda }}_1{\widehat{\alpha }}_1{\widehat{\alpha }}_1^t+\dots +{\sigma }_r^2{\widehat{\lambda }}_p{\widehat{\alpha }}_p{\widehat{\alpha }}_p^t$$

Next, we show the approximation of gene-gene covariance matrix Σ using ${\mathit{\text{Z}}}_1\mathit{\text{G}}{\mathit{\text{Z}}}_1^t$ based on the estimated eigenvectors and eigenvalues is asymptotically unbiased. To see this, note that

$$\begin{array}{l}E\left({\widehat{\lambda }}_l{\widehat{\alpha }}_l{\widehat{\alpha }}_l^t\right) l=1,\dots ,p\hfill \\ =E({\widehat{\lambda }}_l)E\left({\widehat{\alpha }}_l{\widehat{\alpha }}_l^t\right)\hspace{1em}{\widehat{\lambda }}_l\hspace{0.17em}\text{and}\hspace{0.17em}{\widehat{\alpha }}_l\hspace{0.17em}\text{are}\hspace{0.17em}\text{independent}\hspace{0.17em}(\text{Jolliffe}\hspace{0.17em}2002,\hspace{0.17em}\text{p}48)\hfill \\ ={\lambda }_l\left\{E\left({\widehat{\alpha }}_l\right)E\left({\widehat{\alpha }}_l\right)+\mathit{\text{Var}}\left({\widehat{\alpha }}_l\right)\right\}\hfill \\ ={\lambda }_l\left\{{\alpha }_l{\alpha }_l^t+O\left(1/n_{\mathit{\text{sample}}}\right)\right\}\hspace{1em}(\text{Jolliffe}\hspace{0.17em}2002,\hspace{0.17em}\text{p}48)\hfill \\ \to {\lambda }_l{\alpha }_l{\alpha }_l^t\hspace{0.17em}\text{as}\hspace{0.17em}{n}_{\mathit{\text{sample}}\hspace{0.17em}}\to \infty \hfill \end{array}$$

Now letting ${\sigma }_r^2=1$, we then have

$$E({\mathit{\text{Z}}}_1\mathit{\text{G}}{\mathit{\text{Z}}}_1^t)\to {\lambda }_1{\alpha }_1{\alpha }_1^t+\dots +{\lambda }_p{\alpha }_{\text{p}}{\alpha }_p^t=\Sigma \hspace{0.17em}\text{as}\hspace{0.17em}{n}_{\mathit{\text{sample}}}\to \infty$$

In practice, to increase the goodness-of-fit of Model 1, we include ${\sigma }_r^2$ as an unknown parameter and use restricted maximum likelihood to obtain its estimate. In addition, since Σ is not known, we replace Σ with its unbiased estimate Σ̂.

In summary, the steps for fitting the proposed model are as follows:

1. For each gene, to remove the mean trend, we fit the linear model $Y_{ijk}={\beta }_{0i}+{\beta }_{1i}{\mathit{\text{time}}}_{\text{j}}+{\mathit{\text{Array}}}_k+{\varepsilon }_{ijk}^{\text{'}}$ where Y_ijk denotes log transformed value for gene i from array k at time j. From this model, the studentized residuals (Littell et al. 2006, p415), which are residuals divided by an estimate of its standard deviation are then computed.

2. Let Σ̂ be the sample gene-gene covariance matrix (an unbiased estimate of Σ), calculated based on the studentized residuals from the gene-wise linear models in (1). Specify the column in design matrix corresponding to r_l to be $\sqrt{{\widehat{\lambda }}_l}{\widehat{\alpha }}_l$ where α̂_l = estimated l-th eigenvector of ∑̂ and λ̂_l = estimated l-th eigenvalue of ∑̂.

Another way to model covariance between genes is to include a single random effect r and specify the design matrix corresponding to r using the studentized residuals (Littell et al. 2006, p415) from the gene-wise linear models directly. Then, the correlations between genes would depend on direction and magnitude of the difference in their observed and expected trajectories. Two genes are highly positively correlated if both genes deviate from their mean trajectories in the same direction and by a large amount. We compare sensitivity, specificity and type I error rates of models with r₁, …, r_p (MMevct model) or r (MMrstudent model) as random effects using simulation studies in Section 3.1.

2.3. Ranking of Individual Gene’s Contribution to the Gene Set Signal

Because gene sets are defined based on existing knowledge in biological processes and pathways without considering biological context such as tissue type or environment, when they are put into specific conditions such as those defined in a microarray experiment, typically not all member genes of a perturbed biological process or pathway are responsive to the specific conditions. For gene sets differentially expressed over time, it is thus helpful to identify the subset of genes contributing to the gene set significance. Toward this end, we define the influential subset of genes, which are those genes that contribute most to the gene set signal, to be those genes with estimated trajectory β̂₁ + b̂_1i (Model 1 in Section 2.1) more extreme than the estimated mean trajectory β̂₁ for the gene set. Recall in Model 1, b_1i models the deviation of each gene’s slope from the group mean slope β̂₁. For example, for gene sets with positive mean trajectory (β̂₁ > 0 ), the influential subset includes all genes with b̂_1i > 0 or equivalently β̂₁ + b̂_1i > β̂₁. On the other hand, for gene sets with negative mean trajectory (β̂₁ < 0), the influential subset includes all genes with b̂_1i < 0 or equivalently β̂₁ + b̂_1i < β̂₁.

To further identify genes for follow up experiment with alternative platform (e.g. real time PCR), we rank these influential genes by their estimated individual gene trajectories over time β̂₁ + b̂_1i. Under the mixed model framework, these estimates are called empirical Best Linear Unbiased Predictors (BLUPs); they are shrinkage estimates that borrow information across all genes in the gene set and naturally fall into the hierarchical empirical Bayes framework. We illustrate ranking and selection of the influential genes in Sec 3.2 with a mouse colon development dataset.

2.4. Design of Simulation Experiment

We conducted a simulation experiment to compare the performance of the proposed random coefficient models with GlobalANCOVA and Fisher’s exact test for testing the central and global null hypotheses. To obtain genes with “real life” correlations, we used 12 samples of microarray data from a real time course experiment (Section 3.2). First, after mapping the gene expression data with the Gene Ontology database, we obtained 522 sets of genes corresponding to 522 GO terms. For each gene set, fixing the gene expression data, we then generated values for “pseudo time” so that the status of the gene sets in Table 1 would be satisfied. More specifically, for gene sets 1–174, for each sample, we let pseudo time equal the average gene expression values of genes in the gene set, therefore, by design of the experiment, both global and central null hypotheses are false for these gene sets. For gene sets 349–522, pseudo times were generated from a normal distribution with the same mean as average of all genes values. Because these pseudo times were generated independently of the gene expression values, both global and the central null hypotheses are true for these gene sets. Finally, for gene sets 175–348, in addition to generating a pseudo time unrelated to the gene expression data (as in gene sets 349–522), for each gene set, we also generated a pseudo gene with the same value as pseudo time, and added this pseudo gene to the gene expression data set. Therefore, in gene sets 175–348, the pseudo time is related to only one gene (i.e. the pseudo gene), but not the average gene expression of genes in the gene set, in other words, the global null hypothesis is false but the central null hypothesis is true for these gene sets. Therefore, in this simulation study, by fixing gene expression profiles and generating “pseudo time”, we preserved the correlation patterns of real gene expression data.

Table 1.

Status of the Gene Sets by Design of the Simulation Experiment.

GO terms (sorted alphabetically)	True Status of Gene Sets
	H0G: no gene is related to pseudo time	H0C: av. gene exp is not related to pseudo time
1 – 174	False	False
175 – 348	False	True
349 – 522	True	True

Given the known status of the gene sets according to the experimental design (Table 1), for each of the four methods compared, we next calculated Area under the Receiver Operating Characteristic Curves (AUC). The receiver operating characteristic (ROC) curves show a trade-off between sensitivity and specificity as the significance cutoff is varied. AUC assesses the overall discriminative ability of the methods at determining whether a given gene-set is associated with time over all possible cutoffs. In addition, we calculated power based on a nominal p-value of 0.05, that is, the probability of declaring a gene set being significant at p = 0.05 for true positive gene sets. Finally, we calculated the test sizes of each method (the proportions of p-values less than 0.05 for null gene-sets). Because under the null hypothesis we expect the p-values to follow a uniform distribution, a method with test size roughly equal to or less than the significance cutoff (e.g. 0.05) is desirable.

While AUC, power, and test sizes evaluate critical statistical properties of the methods, another important aspect is the stability of the results. A different sample would give a different result, the difference being due to sampling variation. To compare stability of the methods, we took sub-samples from the 12 original samples, and evaluated stability based on changes in rank ordering of the p-values for each method using Spearman correlation coefficients. More specifically, the above experiment was repeated 12 times, each time leaving out one sample in turn. Then, for each method, we have 12 sets of p-values, one for each repetition. In Table 2, we show for each method, the average pairwise rank correlations for the 12 sets of p-values.

Table 2.

Results of Simulation Experiment. Gene sets = relevant gene sets used in the computation of evaluation measures (test size, area under ROC curve, power or stability); Ho Tested = the null hypothesis being tested; MMevct = mixed model with eigenvectors as random effects to account for the covariance structure between genes; MMrstudent = mixed model with studentized residuals as random effects; globalANCOVA = the globalANCOVA method (Hummel et al. 2008); Fisher05 = Fisher’s exact test with nominal p-value 0.05 as threshold for selecting significant genes. See text for details of the simulation experiment.

Gene Sets	Ho Tested	MMevct	MMrstudent	globalANCOVA	Fisher05
Test Size
175–522	central	0.049	0.106	0.351	0.06
349–522	global	0.052	0.109	0.04	0.034
Power
1–174	central	0.73	0.736	0.833	0.299
1–348	global	0.388	0.42	0.747	0.193
Area Under ROC Curve
1–522	central	0.951	0.911	0.785	0.696
1–522	global	0.783	0.759	0.954	0.791
Stability (av. pair-wise spearman rank correlations)
1–522		0.902	0.964	0.881	0.702

2.5. Software Implementation

The R packages (http://www.r-project.org/) globalANCOVA and fisher.test were used for Fisher’s exact method and globalANCOVA method. The proposed method can be easily implemented in any common statistical software packages. We used SAS PROC MIXED for the random coefficient models analysis. To increase computational efficiency, orthogonal polynomial scores, which are linear transformations of the natural polynomial scores, were used to test for a linear time trend. To help with convergence, the PROC MIXED statement parms /ols; can be used when the default procedure produces poor starting values for the optimization process. Estimates of BLUPs for single genes were requested by specifying the solution option in the random statement. The eigenvectors r₁, …, r_p were computed using SAS PROC PRINCOMP.

3. RESULTS

3.1. Simulation Study

Table 2 shows results of the simulation experiment comparing Fisher’s exact test with nominal p-value of 0.05 as threshold for selecting significant genes (Fisher05), globalANCOVA (globalANCOVA), the random coefficient models with eigenvectors (MMevct) or studentized residuals (MMrstudent) as random effects to model the covariance structure between genes. As expected, for testing the global null hypothesis that no gene in the gene set is associated with pseudo time, globalANCOVA maintained reasonable test size (0.040) and performed best with highest power (74.7%) and largest AUC (0.954). The random coefficient models and the “bottom up” approach using Fisher’s exact test were conservative for testing the global null hypothesis.

On the other hand, for testing the central null hypothesis, while the mixed model (MMevct) and Fisher’s exact test maintained reasonable test sizes (0.049, 0.060), globalANCOVA and the mixed model (MMrstudent) with simpler covariance structure did not. In particular, globalANCOVA rejected (35%) of the null gene sets. Therefore, globalANCOVA is not recommended for testing shift of mean trajectory for a group of genes. In contrast, Fisher’s exact test was very conservative with 29.9% power and 0.696 AUC, because information is lost with thresholding significance levels of the genes. For the two random coefficient models, although both models had similar power (73% and 73.6%), MMevct performed better than MMrstudent with higher AUC (0.951 vs. 0.911). Among all models, MMevct had the best sensitivities across all levels of specificity (Figure 2).

Figure 2:

ROC Curves for Testing the Central Null Hypothesis H_0C: the average gene expression of a gene group is not differentially expressed over time. The receiver operating characteristic (ROC) curves show a trade-off between sensitivity and 1-specificity as the significance cutoff is varied. Among all models, the random coefficient model MMevct had the best sensitivities across all levels of specificity, the model rstudent performed comparably, Fisher’s exact test lacked sensitivity while globalANCOVA lacked specificity.

Table 2 also shows the results of comparison on stabilities of the methods. For each group of gene sets, the “top down” approaches, globalANCOVA and the random coefficient models MMrstudent and MMevct, were more stable over changes in samples than the “bottom up” approach Fisher’s exact test. This shows the gene set estimate from aggregating gene expressions from a group of genes in the “top down” approaches is more robust to variability due to sampling than single gene estimates in the “bottom up” approach. P-values from the random coefficient models (MMrstudent, MMevct) were more stable than the globalANCOVA p-values. This confirms our hypothesis that modeling the mean trajectory rather than maximum or minimum trajectory of a group of genes results in more stable p-values. Between the two random coefficient models, results of the simpler model MMrstudent, which uses one random effect (studentized residuals, Section 2.2) rather than multiple random effects (eigenvectors) as in model MMevct, was more stable. This is in agreement with the bias-variance tradeoff principal, which states that estimates from simpler models have more bias, but less variance while estimates from more complex models have less bias but more variance. In our study, results of the more complex model MMevct were more accurate with larger AUC while results of the simpler model MMrstudent were more stable with higher correlations among sub-sample p-values.

In terms of computational efficiency, on a single CPU PC with a 2.00GHz processor and 2 GB of memory, for processing 522 genesets, computing times were 155 seconds for mixed model estimation and testing, 197 seconds for globalANCOVA with 10,000 permutations and 223 seconds for Fisher’s exact test. Therefore, all three methods were computationally efficient for practical use.

3.2. Analysis of a Mouse Colon Development Time Course Experiment

To study the regulatory genetic programs underlying the morphological changes during mouse colonic development from E13.5 to E18.5, we conducted a time course microarray study using two strains of mice, outbred CD-1 and inbred C57BL/6. Twelve microarray samples, one for each time by strain combination were used. To identify the biological processes underlying colonic development and maturation, we used the proposed random coefficient model.

C57BL/6 (Jackson Laboratories, Bar Harbor, ME) and CD-1 (Charles River Laboratories, Wilmington, MS) mice were used in this microarray study. Embryonic colon collection and RNA preparation were performed as previously described (Park et al. 2005). RNA samples were submitted to the Vanderbilt Microarray Shared Resource (VMSR, http://array.mc.vanderbilt.edu), where RNA was hybridized to the Affymetrix Mouse Genome 430 2.0 GeneChip Expression Arrays (Santa Clara, CA) according to manufacturer’s instructions. Microarray data were normalized using the Robust MultiChip Average (RMA) algorithm (Irizarry et al. 2003) as implemented in Bioconductor (Gentleman et al. 2004). Probe set identifiers (IDs) were mapped to Ensembl Gene IDs based on the mapping provided by Ensembl V49 (http://www.ensembl.org). Median expression levels from multiple probe sets corresponding to the same gene were calculated to represent the gene expression level.

After this step, we were left with 15548 genes. To homogenize variances for all the genes included in the mixed model and to help with interpretation, we standardized values for each gene by subtracting their value at baseline and dividing by their group specific standard deviation. The standardized gene expression values then represent the number of standard deviations away from the baseline gene expression values. We next mapped these genes to gene sets generated based on the biological process categories of Gene Ontology. We focused on GO categories with 5 to 200 genes. In order to reduce the redundancy in GO, we removed all child-categories if corresponding parent-category was within the size limitation. After the above processes, we were left with 522 gene sets.

We applied the random coefficient models to gene expression values from each gene set. First, to test for differential trajectory over time for the two groups of mice, we used the random coefficient model

$$\text{Model}\hspace{0.17em}2\hspace{0.17em}:Y_{\mathit{\text{hijk}}}=({\beta }_{0h}+b_{0i})+({\beta }_{1h}+b_{1i}){\mathit{\text{time}}}_{\text{j}}+{\mathit{\text{Array}}}_k+r_{1i}+\dots +r_{\mathit{\text{pi}}}+{\varepsilon }_{\mathit{\text{ijk}}}$$

where

• Y_hijk = log transformed value for gene i from array k at time j for a mouse from group h;

• h = 1, 2 for CD-1 and C57BL/6 groups respectively and all other parameters are defined the same as in Model 1 (Section 2.1). For each pathway, we tested the null hypothesis

• H₀ : β₁₁ = β₁₂ which corresponds to a time by strain interaction effect. Rejection of H₀ would indicate different patterns of change over time for the two strains of mice. After computing FDR (false discovery rate) however, all gene sets had adjusted p-value of 1. The 12 samples were then pooled to identify gene sets differentially expressed over time using Model 1 (Section 2.1) for subsequent analysis.

When we applied globalANCOVA with 10,000 permutations, there were 287 GO terms with FDR (False Discovery Rate) adjusted p-value of 0. In particular, all gene sets with more than 50 genes had p-value of 0. Therefore, results of testing global null hypotheses were difficult to interpret for this dataset.

In contrast, at a 1% FDR level, the random coefficient model MMevct from Section 2.1 identified 60 gene sets that were significantly differentially expressed over time (Table 3 and Supplementary Tables), among which 20 gene sets showed a trend of down-regulation across the developmental time course whereas 40 showed a trend of up-regulation. Interestingly, almost all of the down-regulated gene sets were related to cell proliferation and genetic information processing, such as cell cycle checkpoint, DNA replication, RNA splicing, and transcription initiation etc. On the other hand, the up-regulated gene sets were mostly related to metabolic process and are related to differentiated cellular processes (including steroid metabolic process, polyol metabolic process, coenzyme metabolic process, monosaccharide metabolic process, cytokine metabolic process, fatty acid metabolic process, sulfur metabolic process, glycoprotein metabolic process, pyruvate metabolic process, vitamin metabolic process, carbohydrate catabolic process, alcohol catabolic process), transport (including hydrogen transport, lipid transport, carbohydrate transport, organic acid transport, sodium ion transport, peptide transport, transition meta lion transport, inorganic anion transport), and stress response (such as response to extracellular stimulus, innate immune response, induction of programmed cell death etc). These results suggested that the embryonic colon is committed to growth early on. After cellular proliferation, the colon up-regulates genes involved in metabolism, transport, and stress response. This is consistent with prior observations on embryonic colon (Park et al. 2005) and kidney (Stuart et al. 2001).

Table 3.

Results of the Mixed Models Analysis of the Colon Development Dataset. Top five most significantly up-regulated and down-regulated GO Bio-logical Process Terms. Size = number of genes in gene set; raw_p = nominal p-value; fdr_p = FDR adjusted p-value; n_inf = number of influential genes identified; Influential Genes = five influential genes contributing most to gene set significance signal, with most influential gene ranked first. See Supplementary Materials for full list of significantly changed gene sets.

GO Term	size	raw_p	fdr_p	Description	n_inf	Influential Genes (Most Influential First)
Up-regulated
GO_0006818	62	3.912E-09	3.403E-07	hydrogen transport	40	Atp1a1;Atp2c2;Clca3;Atp2a3;Atp6v0b;
GO_0010324	157	3.110E-08	2.030E-06	membrane invagination	89	Cubn;Elmo3;Cav1;Cav2;Fcer1g;
GO_0008202	137	1.572E-07	8.206E-06	steroid metabolic process	78	Cubn;Nr1h4;Atp1a1;Hsd17b2;Stard4;
GO_0019751	38	1.798E-07	8.534E-06	polyol metabolic process	22	4833409A17Rik;Gdpd1;Gyk;Gpd1l;Psmb10;
GO_0006732	140	7.984E-07	3.206E-05	coenzyme metabolic process	85	Rnasel;Aldh1l1;Hnf4a;Folr1;Clca3;
Down-regulated
GO_0008380	140	4.001E-15	2.086E-12	RNA splicing	97	Snrpd1;Ncbp2;1110037F02Rik;Sfrs7;Dhx15;
GO_0015931	58	8.301E-14	2.167E-11	nucleobase, nucleoside, nucleotide and nucleic acid transport	35	Ncbp2;Nup62;Nup93;Magoh;Qk;
GO_0006403	52	1.462E-13	2.542E-11	RNA localization	31	Ncbp2;Nup62;Nup93;Magoh;Qk;
GO_0006260	137	5.296E-13	6.908E-11	DNA replication	88	Orc6l;Mcm4;Dut;Dbf4;Mcm6;
GO_0000087	171	4.907E-10	5.123E-08	M phase of mitotic cell cycle	105	Bub1;Tubb5;Cenph;Pds5a;Anapc1;

Because gene sets are defined based on existing knowledge in biological processes and pathways without considering specific biological context, not all genes in a significant gene set will follow the trend of up- or down-regulation during mouse colon development. It is of particular interest to identify influential genes that follow the main trend and rank individual genes by their contributions to the gene set significance. Using the method described in Section 2.3, we identified the subset of influential genes for each significant gene set. Tables 3 and 4 show the total number of the influential genes within the gene set and the top five genes contributing most to the gene set significance signal.

4. DISCUSSION

In this paper, we have proposed a unified strategy for systems analysis of time course microarray experiments at the gene set level using random coefficient models. Several features of the proposed models make them especially attractive in this setting:

First, in the analysis of gene sets, to avoid inflation of false positive rate, great care must be taken to account for correlations between genes in the same pathway. For testing the central null hypothesis, we have shown both theoretically (Section 2.2) and empirically (Section 3.1) that the proposed random coefficient model adequately captures the primary covariance structure between genes and preserves type I error rate. On the other hand, permutation tests rely on exchangeability of the permuted units, careful consideration are required to account for data structure in complex study designs to avoid misleading results (Churchill et al. 2008). For example, Xu and Hsu (2007) showed that when comparing mean expression levels of a set of genes between two groups of subjects, permutation tests based on permuting sample labels across groups may not preserve type I error rate, when the joint distributions of gene expression levels differ between the groups.

Second, using gene expression data with “real life” correlations, we have shown in comparison to currently available methods, by modeling the mean trajectory and testing the central hypothesis for a group of genes, the proposed random coefficient models yield statistical inferences for gene sets that are more stable (less affected by sampling variation) and more interpretable (allowing ranking of individual genes that contribute most to the gene set signal). In the analysis of the mouse embryonic colon dataset, the agreement between results of the random coefficient models and previous reports further validates the proposed method.

Finally, the proposed method is a general methodology that operates within a well-established statistical framework. This flexible, unified and practical approach can be easily implemented in common statistical software packages. By including design factors and covariate effects, the random coefficient models can be augmented easily to handle more complex designs with multiple sources of variation such as those with biological replicates or even spatial effects. For example, when arrays are processed in multiple batches, a batch effect can be added to the model to adjust for systematic effects from different batches. Similarly, other random effects from blocks and sites where the experiments were performed can also be incorporated into the model. Littell et al. (2006) provides a comprehensive set of examples covering a wide range of mixed models and related covariance structures.