Detecting strong signals in gene perturbation experiments: An adaptive approach with power guarantee and FDR control

Abstract

The perturbation of a transcription factor should affect the expression levels of its direct targets. However, not all genes showing changes in expression are direct targets. To increase the chance of detecting direct targets, we propose a modified two-group model where the null group corresponds to genes which are not direct targets, but can have small non-zero effects. We model the behavior of genes from the null set by a Gaussian distribution with unknown variance τ². To estimate τ², we focus on a simple estimation approach, the iterated empirical Bayes estimation. We conduct a detailed analysis of the properties of the iterated EB estimate and provide theoretical guarantee of its good performance under mild conditions. We provide simulations comparing the new modeling approach with existing methods, and the new approach shows more stable and better performance under different situations. We also apply it to a real data set from gene knock-down experiments and obtained better results compared with the original two-group model testing for non-zero effects.

1. Introduction

The transcriptional regulatory networks, formed by transcription factors(TFs) and their targets, are believed to play an important role regulating embryonic stem(ES) cell pluripotency(Niwa et al. (1998, 2000); Chambers and Smith (2004); Loh et al. (2006); Kim et al. (2008); Chen et al. (2008)). A multitude of inference methods exist in the literature for the identification of such networks using observational gene expression data(Friedman et al. (2000); Murphy et al. (1999); Kim et al. (2004); Lebre et al. (2010)). On the other hand, there is also intense interest in using perturbation experiments in the study of gene regulation. For example, the TF knock-down experiment is expected to very informative identifying potential targets of a TF because it depicts a less complex picture and can provide evidence for causal relationships(Geier et al. (2007); Werhli et al. (2006)).

Traditionally, potential targets of the TF are usually identified as the subset of differentially expressed genes between the control and experiment group. However, when an important TF has been knocked down, it is almost always the case that the proportion of significantly changed genes is much larger than expected(Ivanova et al. (2006); Zhou et al. (2007)). As a concrete example, consider the data set analyzed in this study, which is from the knockdown experiment for two TFs which play an important role regulating embryonic stem (ES) cell pluripotency(see section 5 for details). In this data set, the number of differentially expressed genes is very large, while the number of likely direct targets (from external CHIP-seq data assessing TF binding) is significantly smaller.

There are two popular explanations for this phenomenon:

1. The theoretic null distribution of the test statistics(often z-score and other analogous quantities) for zero effect is not accurate.

2. There are a large number of genes showing non-zero but small changes of gene expression level, as effects of the perturbation.

Proposed solutions include modifying the null distribution of z-score empirically(Efron (2007, 2008)) and applying a cut-off to fold-change as a second-layer filter; the latter has been extremely popular in practice(Nichols et al. (1998); Zhou et al. (2007); Vaes et al. (2014)). While both of these approaches can narrow down the selected, the former tackles the problem mainly based on the first explanation while the latter adopts the second implicitly, and results can be different in general(Witten and Tibshirani (2007)). For the knockdown experiment, the latter seems preferable because it considers both the change magnitude and the non-zero significant level, which is more related to what scientists care about; however, this approach lacks a natural quantitative justification.

Here, we propose a simple model to combine these two perspectives. By using a Gaussian distribution with unknown variance to describe the underlying behavior of genes in the null group, our model assumes that there can be relatively small non-zero effects even for the null genes. Assuming that the number of genes with large effect size is small, we test for the presence of such large effects relative to the background null variance. Although this model is motivated by the knockdown experiment, it can be applied to more general multiple-testing setting where both the significance and the effect size matter.

Our approach is related to the method of maximal agreement cut by Henderson and Newton (2015). They similarly pointed out that testing approaches which measure evidence against the null hypothesis tend to over-populate the candidate list with those associated with small variance, while approaches that consider only the magnitude will overlook the noise. While sharing the same spirit, our method does not aim to find the top α% subset of genes maximizing the expected overlap with the truth, with some assumed prior for all genes. Instead, we are interested in identifying the subset of genes which could not be described well by the prior describing the majority. In the setting of knock-down experiment, our model describes a scenario different from the one assumed in approaches testing for differentially expressed genes. In our model, it is assumed that, perhaps due to propagation through the gene regulatory network, when we collect the data, a lot and even all genes may have been influenced once a TF has been knocked down, and we take this possibility into consideration. We show that in this scenario, it is still possible to test for strong effect if (1) the direct target tends to have larger effect size, and (2) there are enough null hypotheses to estimate the variance under the null.

The paper is organized as follows: We describe our model in section 2.1 and the procedure to estimate the null variance in section 2.2. In section 3, we study the properties of the estimating procedure of τ²; in section 4, we extended the model to the non-centered case and the case of two sample testing with unequal variance. We provide simulations in section 5 and real data examples in section 6.

2. Statistical model and estimation procedure

2.1. Statistical model

We assume there is a control group with m₀ replicates and an experiment group with m₁ replicates after knocking down one TF of interest. The expression levels for N genes are measured for each replicate. Let x_{i, j} be the measurement for gene i in replicate j from the experiment group, and z_{i, j} be the measurement for gene i in replicate j from the control group. Without loss of generality, assume the mean level of z_{i, j} is 0 and the mean level of x_{i, j} is μ_i:

$$x_{i,j}\sim N({\mu }_i,{\sigma }_i^2),{\forall }_i=1,2,\dots ,N,j=1,2,\dots ,m_1$$

$$z_{i,j}\sim N(0,{\sigma }_i^2),{\forall }_i=1,2,\dots ,N,j=1,2,\dots ,m_0$$

To do inference on μ_i, we can look at the two-sample test statistics,

$${\overline{x}}_i-{\overline{z}}_i=\frac{\sum _{j=1}^{m_1}{x}_{i,j}}{m_1}-\frac{\sum _{j=1}^{m_0}{z}_{i,j}}{m_0}\sim N({\mu }_i,{\sigma }_i^2(\frac{1}{m_1}+\frac{1}{m_0})),\forall i=1,2,\mathrm{\dots },N$$

Or the paired sample test statistics to remove the batch effect (m₀ = m₁),

$$\overline{x_i-z_i}=\frac{\sum _{j=1}^{m_1}(x_{i,j}-z_{i,j})}{m_1}\sim N({\mu }_i,{\sigma }_i^2\frac{1}{m_1}),\forall i=1,2,\mathrm{\dots },N$$

As there is no fundamental difference between these two tests in our later analysis, we will omit the notation z_i and use the following common notations for simplicity:

$${\overline{x}}_i\sim N({\mu }_i,{\sigma }_{{\overline{x}}_i}^2),\forall i=1,2,\dots ,N$$

$${\widehat{\sigma }}_{{\overline{x}}_i}^2\sim {\sigma }_{{\overline{x}}_i}^2\frac{{\chi }_{m-k}^2}{m-k},\forall i=1,2,\dots ,N$$

where ${\sigma }_{{\overline{x}}_i}^2=\frac{{\sigma }_i^2}{n}$, with n being the effective sample size and ${\widehat{\sigma }}_{{\overline{x}}_i}^2$ is the usual unbiased variance estimate of ${\overline{x}}_i$. In the two-sample case, $n=\frac{m_1{m}_0}{m}$, m = m₁ + m₂, k = 2 and in the paired sample case, n = m = m₁, k = 1. Following the widely used two group model(Efron (2008)), let A₀, A₁ denote the sets of nulls and non-nulls respectively and $\gamma =\frac{|A_1|}{N}$ denote the proportion of non-nulls. We assume that the μ_is in A₀ and A₁ are generated from different distributions:

$${\mu }_i\sim \{\begin{array}{ll}N(0,{\tau }^2)& \forall i\in A_0\\ g_i(.)& \forall i\in A_1\end{array}$$

For each gene i ∈ A₁, g_i is some unknown density function. The parameter t can be viewed as describing the range of normal behaviour.

In contrast to the original two-group model, which corresponds to t = 0, we allow t to take positive values. By relaxing this assumption on t, we are able to detect relatively abnormal behavior compared with the background signal. If we know t, the p-value for the new null hypothesis for gene i can be derived.

Let ${\mu }_i=\frac{\overline{x}}{\sqrt{{\tau }^2+{\widehat{\sigma }}_{{\overline{x}}_i}^2}},{\overline{x}}_i\sim N(0,{\tau }^2+{\sigma }_{{\overline{x}}_i}^2)$, under the null hypothesis, u_i is a Welch statistics(Welch (1947)) in the limit case with the degree of freedom for the first “variance estimate” τ² being ∞. Usual analysis controlling False discovery rate (FDR) or family wise error rate (FWER) carries through under our extended model in this case.

We emphasize that the parameter t itself is informative because it characterizes how influential a stimulus is – in our case, how dramatically the whole system changes after we have knocked down a TF. The value of t reflects the importance of the TF: it can be set according to either prior knowledge or estimated from the data. The second approach is usually more feasible, as we lack a quantitative characterization of this kind of importance, and it can vary under different environments even for the same TF.

2.2. Estimation procedure

For $i\in A_0,{\overline{x}}_i\sim N(0,{\sigma }_{{\overline{x}}_i}^2+{\tau }^2)$ marginalizing out μ_i. If A₀ is known, the empirical Bayes estimate of τ² with estimated variance ${\widehat{\sigma }}_{{\overline{x}}_i}^2$ for ${\sigma }_{{\overline{x}}_i}^2$ is given by ${\widehat{\tau }}_{A_0}^2=\frac{1}{|A_0|}\sum _{i\in A_0}({\overline{x}}_i^2-{\widehat{\sigma }}_{{\overline{x}}_i}^2)$. Let δ be a pre-determined small value, the adjusted form ${\widehat{\tau }}_{A0}^2=[\frac{1}{|A|}\sum _{i\in A_0}{\overline{x}}_i^2-(1+\delta ){\widehat{\sigma }}_{{\overline{x}}_i}^2{]}_{+}$, is often preferred to reduce the error for small τ² and to ensure non-negativity. Similar form of estimate is analyzed by Johnstone (2001b,a) in the context of estimating non-centrality of χ² - distribution with known variance.

Let F_i(.) denote the distribution of $\frac{x_i^{-2}}{{\tau }^2+{\widehat{\sigma }}_{{\overline{x}}_i}^2}$ when i ∈ A₀ which is the distribution of the square of a Welch’s statistic as mentioned before, and ${\tilde{F}}_i(x)=1-F_i(x)$. The Iterated empirical Bayes estimation(ITEB) procedure is given below, which starts from the whole set (as A₀ ∪ A₁) and then iteratively remove potential outliers on the tail based on the current estimate of τ². It stops when no point needs to be further removed.

Iterated empirical Bayes estimation(ITEB) of τ²

Input: $\{({\overline{x}}_i,{\widehat{\sigma }}_{{\overline{x}}_i}^2),\forall i=1,2,\dots ,N\}$, significance level α₁, α₂ and δ. By default, α₁ = 0.1, α₂ = 0.01 and $\delta =\sqrt{\frac{8}{N}}$.

Output: ${\widehat{\tau }}^2$, the estimated τ².

Initialization: S₀ = {1,2,…,N} be the initial estimate of the null set, and ${\widehat{\tau }}_{S_0}^2=\frac{[\sum _{i=1}^N{\overline{x}}_i^2-(1+\delta )\sum _{i=1}^N{\widehat{\sigma }}_{{\overline{x}}_i}^2{]}_{+}}{N}$.

For k = 1,2,…, do

1. Update the p value for each gene $p_i={\tilde{F}}_i(\frac{{\overline{x}}_i^2}{{\widehat{\tau }}_{S_{k-1}}^2+{\widehat{\sigma }}_{{\overline{x}}_i}^2})$. The ordered p values from small to large are p₍₁₎, p₍₂₎,…, p_(N). Let i* be the largest index, such that $p_{(i^{*})}\le \frac{i^{*}}{N}{\alpha }_1$.

2. Let $J_k^1=\{i\in A:p_i\le p_{(i^{*})}\}$, and $J_k^2=\{i\in A:p_i\le {\alpha }_2\}$ and remove $J_k:=J_k^1\cap J_k^2$. Update S_k = S_k−1 \ J_k and ${\widehat{\tau }}_{S_k}^2=\frac{[\sum _{i\in S_k}{\overline{x}}_i^2-(1+\delta )\sum _{i\in S_k}{\widehat{\sigma }}_{{\overline{x}}_i}^2{]}_{+}}{|S_k|}$.

3. If S_k = S_k−1, return ${\widehat{\tau }}^{\text{2}}={\widehat{\tau }}_{S_k}^2$

We have also described two other estimation methods and compared them with ITEB in the Supplement: the truncated MLE method and the central matching(CM) method. The detailed descriptions of the truncated MLE and the CM estimator are given in Appendix C. These two methods have also been applied to estimating the empirical null distribution in the traditional two group test problem(Efron (2012); Efron et al. (2001)), and we have adapted them to our problem here. In Appendix D, we compare performances of the three estimators in different scenarios and discuss their strengths and weaknesses. ITEB is most computationally efficient and is found to have better performance overall when the non-null proportion γ is small. Thus, we will focus on ITEB and we provide detailed analysis of its properties.

3. Properties of ITEB

We study the estimation quality of ITEB as the number of hypotheses N → ∞ For simplicity, we analyze the algorithm under following mild conditions and notations with δ = 0. Let $\lambda (\alpha ):={\max }_i{\tilde{F}}_i^{-1}(\alpha )$. The degree of freedom for the variance estimate is m for all i, and K := the number of iterations needed for the algorithm to stop. Since in this section we only use the mean level ${\overline{x}}_i$ and its estimated variance ${\widehat{\sigma }}_{{\overline{x}}_i}^2$, with slight abuse of notation, let $x_i:={\overline{x}}_i,{\widehat{\sigma }}_i^2:={\widehat{\sigma }}_{{\overline{x}}_i}^2,{\sigma }_i^2:={\sigma }_{{\overline{x}}_i}^2$. For the two levels α₁ and α₂ in the ITEB algorithm, we let $0<{\alpha }_1<\frac{1}{2e}$ be a fixed value, and let α₂ → 0 at a slow rate to simplify the notations in the proof (we always let $\frac{N{\alpha }_2}{{\log }^{2}N}$ bounded away from 0).

Assumption 3.1. The degree of freedom for variance estimates m ≥ 5 is a constant and the non-null proportion γ < 1 − c for some positive constant c. The ratio of variances of different genes is bounded: there exists a positive constant C such that $\frac{{\max }_i{\sigma }_i^2}{{\min }_i{\sigma }_i^2}\le C$.

Without loss of generality, we rescale ${\min }_i{\sigma }_i^2=1$, then $C={\max }_i{\sigma }_i^2$. We do not require τ² to be positive or a constant. It can be 0 or decay to 0 as N → ∞.

Assumption 3.2. There exist constants L and ϵ, such that $\forall i\in A_1,E[x_i^2-(1+ϵ){\widehat{\sigma }}_i^2|x_i^2-(1+ϵ){\widehat{\sigma }}_i^2\le L({\tau }^2+1)]\ge (1+ϵ){\tau }^2$

Remark 3.3. If i ∈ A₀, we have $E[x_i^2-{\widehat{\sigma }}_i^2]={\tau }^2$ Assumption 3.2 states that, for i ∈ A₁, $x_i^2-{\widehat{\sigma }}_i^2$ has expectation non-negligibly bigger than τ², and it is not purely driven by observations from its tail.

Assumption 3.4. The non-null proportion γ → 0 as N → ∞.

Theorem 3.5. (Lower bound of the variance estimate) Let R_K :=|J_K ∩ A₁|, where J_K is the rejected set at the last step k = K from ITEB. Let ${\Delta }_1=\sqrt{\frac{\log N}{N}}({\tau }^2+C),t_l=\max (\frac{{\log }^{2}N}{N},\min (\frac{l}{N},2{\alpha }_2)),{\Delta }_{2,l}=3({\tau }^2+C)t_l\log \frac{1}{t_l}\mathrm{and}{\tau }_l^2={[{\tau }^2-{\Delta }_1-{\Delta }_{2,l}]}_{+}$. Under Assumption 3.1 and 3.2, we have $P({\cup }_{l=0}^{N\gamma }\{R_K=l,{\widehat{\tau }}^2\ge {\tau }_l^2\})\to 1$.

As $\frac{{\Delta }_1+{\Delta }_{2,l}}{{\tau }^2+C}\to 0$ for all l, Corollary 3.6 is a direct result of Theorem 3.5.

Corollary 3.6. Under Assumption 3.1 and 3.2, for any $\delta >0,{\mathit{lim}}_{N\to \infty }P({\widehat{\tau }}^2\ge {[{\tau }^2-\delta ({\tau }^2+C)]}_{+})=1$.

Theorem 3.7. (Upper bound of the variance estimate) Under Assumption 3.1 and 3.4, suppose α₁ > 0 is fixed and α₂ → 0 at a rate slow enough: γλ(α₂) → 0. Then, for any δ > 0, we have $\underset{N\to \infty }{\lim }P({\widehat{\tau }}^2\le {\tau }^2+\delta ({\tau }^2+C))=1$.

We next show that these results can usually lead to good performance in the follow-up analysis in practice. Theorem 3.8 states that our estimate of τ² can successfully control the FDR if we reject the hypotheses in the set J_K.

Theorem 3.8. (FDR control) Under Assumption 3.1 and 3.2, if we reject all hypothesis in J_K, we have $\underset{N\to \infty }{\lim }FDR\le {\alpha }_1$.

Remark 3.9. Note that at the given level α₁, α₂ in the ITEB algorithm, $J_K^1$ will correspond to the set of rejections using the BFI(Benjamini-Hochberg procedure)(Benjamini and Hochberg (1995)) and J_K will correspond to the set of rejections which are both rejected by the BH procedure and with p-values no greater than α₂. The extra requirement that the p-value is no greater than a reasonable small value α₂ is desirable in many large scale hypotheses testing settings, including the knock-down experiment.

Theorem 3.10. (Power analysis) Let ${\varphi }_{i,\alpha }=1_{p_i\le \alpha }$ and let ${\varphi }_{i,\alpha }^{*}$ be the oracle decision rule knowing τ²: for any level α, ${\varphi }_{i,\alpha }^{*}=1_{x_i^2>{\tilde{F}}_i^{-1}(\alpha )({\widehat{\sigma }}_i^2+{\tau }^2)}$. Let $z_i^2=\frac{x_i^2}{{\tau }_i^2+{\sigma }_i^2}$ where ${\tau }_i^2=E[{\mu }_i^2]$ for i ∈ A₁. Under Assumption 3.1 and 3.4, if we further assume that the density of $z_i^2$ is upper bounded by a constant, and the tail probability for $z_i^2$ decays sufficiently fast:

$$\underset{w\to \infty }{\lim }\underset{\delta >0}{\sup }\underset{i\in A_1}{\sup }\frac{P(z_i^2\le w(1+\delta ))}{P(z_i^2\le w(1+\delta ))}\le 1$$

Then we have $\underset{N\to \infty }{\lim }\underset{i\in A_1}{\text{inf}}\underset{\alpha \ge 0}{\text{inf}}(P({\varphi }_{i,\alpha }=1)-P({\varphi }_{{}_{i,\alpha }}^{*}=1))\ge 0$ .

Remark 3.11. Recall that the follow-up p value p_i for hypothesis i is ${\tilde{F}}_i(\frac{x_i^2}{{\widehat{\tau }}^2+{\widehat{\sigma }}_i^2})$. Thus Theorem 3.10 says that the test ϕ_i based on the estimated variance ${\widehat{\tau }}^2$ is asymptotically as powerful as the optimal test based on the (unknown) true variance τ².

Proofs of Theorem 3.5, 3.7, 3.8 and 3.10 are given in the Appendix A.

4. Extensions

4.1. Non-centered null distribuion

We have been assuming that μ_i in the null set is generated according to N(0, τ²) . The ITEB estimation approach is easily extended to the setting where the null distribution might not be centered and μ_i ~ N(ϵ, τ²) with a small non-centrality ϵ for i ∈ A₀. In ITEB, ϵ can be approximated by

$$\widehat{ϵ}=\frac{\sum _{i=1}^N{\overline{x}}_i^{\mathit{I}}[{\overline{x}}_i\in (-{\delta }_0,{\delta }_0)]}{\sum _{i=1}^N{\mathit{I}}_{[{\overline{x}}_i\in (-{\delta }_0,{\delta }_0)]}}$$

where δ₀ > 0 is a reasonable cut-off and we treat x_i ∈ (−δ₀, δ₀) to be from A₀. We can form an ITEB estimation for the non-centered case by replacing ${\overline{x}}_i$ with ${\overline{x}}_i-ϵ$ in the ITEB algorithm.

4.2. Two sample test with unequal variance

For hypothesis i, the observations from the experiment and control groups, x_{i, j} and z_{i, j}, can have different variances. It is straightforward to generalize ITEB to this situation if we want to perform a two-sample test. We know that ITEB takes in {${\overline{x}}_i-{\overline{z}}_i$} and {${\widehat{\sigma }}_{{\overline{x}}_i-{\overline{z}}_i}^2$}. In the unequal variance setting, we can estimate ${\sigma }_{{\overline{x}}_i-{\overline{z}}_i}^2$ by:

$${\widehat{\sigma }}_{{\overline{x}}_i-{\overline{z}}_i}^2=\frac{\sum _{j=1}^{m_1}{(x_{i,j}-{\overline{x}}_i)}^2}{m_1(m_1-1)}+\frac{\sum _{j=1}^{m_0}{(z_{i,j}-{\overline{z}}_i)}^2}{m_0(m_0-1)}$$

whose degree of freedom is approximated by

$$d{f}_{\sigma }=\frac{(\frac{\sum _{j=1}^{m_1}{(x_{i,j}-{\overline{x}}_i)}^2}{m_1}+\frac{\sum _{j=1}^{m_0}{(z_{i,j}-{\overline{z}}_i)}^2}{m_0}{)}^2}{\frac{1}{(m_1-1)}(\frac{\sum _{j=1}^{m_1}{(x_{i,j}-{\overline{x}}_i)}^2}{m_1}{)}^2+\frac{1}{(m_0-1)}(\frac{\sum _{j=1}^{m_0}{(z_{i,j}-{\overline{z}}_i)}^2}{m_0}{)}^{{}_2}}$$

We approximate F_i(.), the distribution of the test statistics $\frac{{({\overline{x}}_i-{\overline{z}}_i)}^2}{{\tau }^2+{\sigma }_{{\overline{x}}_i-{\overline{z}}_i}^2}$, by F_1,df(.), the F distribution with degree of freedoms (1, df), where df is approximated by $df={(\frac{{\tau }^2}{{\widehat{\sigma }}_{{\overline{x}}_i-{\overline{z}}_i}^2}+1)}^{2}d{f}_{\sigma }$ (Satterthwaite (1946)).

5. Simulation: Detection of large signal

We consider the two-sample setting with equal-variance and generate data under various values of t and non-null proportion $\gamma =\frac{|A_1|}{N}$. Specifically, we fix N = 15000, m = 5 for both the control and experiment group, for any given t and γ, where γ = 1%,5% and τ = 0, 0.1,…,1, 1.5, 2, 2.5, 3 , we generate the true mean and variance as below.

1. Let μ_i = 0 in the control group, and in the experiment group, we generate them as following:

$${\mu }_i\sim \{\begin{array}{ll}N(0,{\tau }^2)& \forall i\in A_0\\ \pm U[1,\text{max}(3,10\tau )]& \forall i\in A_1\end{array}$$

where U[1, max(3, 10τ)] is the uniform distribution between 1 and max(3, 10τ), and the signs of μ_is will be half positive and half negative.

1. We sample the variances ${\sigma }_i^2$ from its empirical distribution from the real data set, and we scale them to have mean level 1.

We compare the following approaches:

• ITEB estimate of τ², followed the Welch’s t-test.

• t-test with the null hypothesis testing for non-zero effect.

• EBarray(Kendziorski et al. (2003); Yuan and Kendziorski (2006)), which is a two group empirical Bayes method providing a posterior probability for having non-zero effect. We choose the “LNNMV” method to fit the data as suggested by the authors.

• Fold-change rankings with a threshold for t-test p-values being 10⁻⁵ (referred to as “fchange”). For hypotheses that do not pass this cut-off, we rank them based on p values from the t-test, after those who have passed the cut-off.

• Rvalue ranking, which finds the α% of genes maximizing the overlap between this gene list and gene list with differential change above the upper α% quantile of a prior normal distribution(Henderson and Newton (2015)).

The proposed model, the rvalue ranking and the fchange all consider the magnitude and significance explicitly. If the p-value cut-off is correctly set for the fchange, we expect these three methods to share more in common in terms of their ROC curves (average sensitivity versus FDP), while only the proposed model allows you to set a cut-off based the significance level of whether a hypothesis is different from the background. LNNMV models observations from the control and the experiment directly instead of modeling their differences, also, although it smoothes the variance estimations with an inverse Chi-square prior, the prior is not related to the effect sizes across hypotheses.

Figures 1 and 2 provide ROC curves across 20 repetitions as the significance level in the testing step is varied. We see that the new approach performs the best or one of the best across different experiments. When t = 0, ITEB procedure and t-test behave similarly, and ITEB becomes almost identical as the fchange when t is large. ITEB and rvalue result in very similar results across different settings (ITEB is slightly better in the more sparse case with γ = 1%), however, besides that ITEB provides information about where to set a cut-off, the ITEB approach is much faster considering the run time. All methods except for the t-test are stable across a wide range of t.

Figure 1.

ROC curve for γ = 1 %.

Figure 2.

ROC curve for γ = 5%.

6. Real Data Examples

In this section, we apply our approach to data from two knock-down experiments described below. The quality of the results are evaluated by the enrichment of ChIP-seq peaks (for the perturbed TF) in active enhancers/promoters for the selected genes. Note that the ChIP-seq data for ES cells are external to the data used to select the genes, and it provides an orthogonal information to access how likely the selected genes are direct targets of the TFs.

We perform gene knock-down experiments on 2 TFs on the mouse ES cell line R1. For each TF, RNA interference (RNAi) delivered using nucleofection was used to knock down its expression. Puromycin selection was introduced 18 h later at 1 μ g/ml, and the medium was changed daily. 30 h, 48 h, and 72 h after puromycin selection, the cells were collected for RNA isolation. After the experiments, Microarray hybridizations were performed on the MouseRef-8 v2.0 expression beadchip arrays (lllumina, CA). More details of the experiments can be found in Appendix E. Quantile normalization is performed in the first step to reduce the batch effect, and for the same reason, for each sample in the experiment group, we consider the paired test statistics with each pair being a pair of independent experiment and control samples from the same batch and time point. We have 8 paired observations for both POU5F1 and NANOG, and we take the log difference between the gene expression levels in a knock-down sample and its corresponding control sample to further reduce the batch effect. Table 1 summarizes the data we have at different time points. Figure 3 shows results from nine random realizations of the t-SNE(Maaten and Hinton (2008)) plot using the top 1000 genes with largest variance across experiments. Each data point in the t-SNE plot represents one sample (paired) in the experiment. We use the colors black, red and green to represent data at time points 30 hr, 48 hr and 72 hr respectively. From the results, we see that differences between time points within the same knock-down experiment is comparable to the differences across batches, and they are very small compared with the differences between two knock-down experiments. To compare the targets of two TFs, we will regard different times points in the same knock-down experiment as replicates of each other.

Table 1.

Information of the knock-down data sets.

	30 hr	48 hr	72 hr
POU5F1	4 pairs	4 pairs	–
NANOG	2 pairs	4 pairs	2 pairs

Figure 3.

Clustering with t-SNE algorithm. The nine plots here are 9 different random tsne-plot realizations. Black, red, and green colors represent experiments from 30 hr, 48 hr and 72 hr, respectively. Note that there are two replicates from the same batch in a single experiment (same day, same TF, same batch) that are almost identical to each other.

ChIP-seq data and enhancer-gene association data:

To evaluate the quality of the selected gene set, we utilize two external data sets: the ChIP-seq data is from Chen et al. (2008) and the enhancer-gene association data is from Mumbach et al. (2017). The ChIP-seq data contains results using chromatin immunoprecipitation coupled with ultra-high-throughput DNA sequencing (ChIP-seq) to map the binding locations of 13 sequence-specific TFs, including POU5F1 and NANOG. The enhancer-gene association data is generated by using the HiChIP method where the authors performed H3K27ac HiChIP in mouse ES cells. H3K27ac is a histone modification mark characteristic of active enhancers and promoters in the cell. HiChIP using H3K27ac mark as bait will provide enhancer-gene interaction information. We can evaluate the quality of the selected gene set by examining whether the binding sites of POU5F1 and NANOG are enriched near the active enhancers/promoters of the selected genes in the ES cell.

Let us call the approach based on p-value using a simple t-test S₀, the approach based on EBarray S₁, and the approach based on p-value using ITEB S₂. We will focus on those genes with significant decrease in their expression levels after POU5F1/NANOG knock-down. For each method, we set the cut-off using the BH procedure with targeted FDR level at 0.01.

Accordingly, we set the the cut-off level α₁ = 0.01 and we select 87 genes after knocking down POU5F1 and 43 genes after knocking down NANOG using S₂. These numbers are 2274 and 1267 using S₀, 2362 and 886 using S₁. Neither S₀ nor S₁ provides informative candidate lists with this criterion. To have a meaningful comparison, we also consider the case where we control FWER at 0.01, which is quite a stringent criterion and under which, S₀ selects 144 genes for POU5F1 and 49 genes for NANOG, S₁ selects 1091 genes for POU5F1 and 254 genes for NANOG.

We say that there is supporting evidence of a gene being a direct target of a TF if this TF has at least one ChIP-seq peak within x kilobase(kb) away from the gene’s active enhancers/promoter. As we change x in a large range of value, Figure 4 gives the percentages of genes with this supporting evidence in the selected gene sets using S₀, S₁ with FWER control and S₂ with FDR control. Figure 4 also shows the percentages of all genes with this supporting evidence (referred to as “all” in the figure), and the percentages of bottom 2000 genes with this supporting evidence (referred to as “bottom”). The bottom 2000 genes are those showing less significance after the knock-down based on the ITEB p-value (we consider this set to be the negative control). From Figure 4, we see that the ChIP-seq enrichment is quite significant for the selected genes comparing with both the negative controls and all genes. The selected gene set using S₂ is significantly better than the gene set selected using S₁ and S₀. Table 2 shows the result with x = 20.

Figure 4.

Percent of genes with ChIP-seq nearby versus x for the selected gene sets (S₀/S₁ genes selected using t-test/EBarray with FWER control, S₂ genes selected based on ITEB with FDR control, bottom: the bottom 2000 genes which show the smallest changes, all: all genes). The x-axis is the threshold we use to define whether a gene has a ChIP-seq peak near its enhancer/promoter and the y-axis is the percentage of selected genes with ChIP-seq nearby.

Table 2.

S₀, S₁, S₂ results with FDR/FWER level set at 0.01

	TF	percent(negative control)	size(S0)	percent(S0)	size(S1)	percent(S1)	size(S2)	percent(S2)
FDR	POU5F1	0.21	2274	0.48	2362	0.48	87	0.73
	NANOG	0.32	1267	0.62	886	0.61	43	0.81
FWER	POU5F1	0.21	144	0.62	1091	0.53	31	0.74
	NANOG	0.32	49	0.74	254	0.64	20	0.8

The third, fifth, seventh and nineth columns are the percent of genes with Chip-seq+Hi-C support in the negative control set, the gene set selected by S₀ (t-test), the gene sets selected by S₁ (EBarray) and S₂ (ITEB t-test) respectively.

Figure 5 provides quality evaluation of the top K genes in different ranking lists respectively. The vertical grey line is where the top 50 genes is. Besides S₀, S₁ and S₂, we also include the results using rvalues and fchange. Comparing S₂ with S₀ and S₁, not only S₂ provides a candidate gene set with much smaller size and higher quality, it provides a gene ranking list with higher quality. In terms of the gene ranking list, the rvalue provides a similar list as that from ITEB (~90% overlappings in the top 200 genes for both TFs).

Figure 5.

Percent of genes with ChIP-seq nearby versus selected gene size. The x-axis is the threshold of the ranking, and we only consider the top k genes from each ranking list.

7. Discussion

In this paper, motivated by the problem of identifying TF targets based on data from the knock-down experiment, we have proposed to test for large effect size instead of non-zero effect size in the two-group model where a Gaussian distribution with non-zero variance is used for the effect in the null group. We have considered three approaches(ITEB, truncated MLE and CM) to estimate this non-zero variance adaptively, and recommend ITBE for its computational efficiency, strong performance in simulation and attractive theoretical properties. Although we have focused on the Gaussian setting here, the idea of testing for strong signal and the approaches to estimate the null distribution can be applied to problems involving other data types. The model itself is related to the “g-modeling”(Carroll and Hall (1988); Efron (2014)), the “ϵ-contamination”(Huber (1964); Chen et al. (2016)) and the “robust Bayesian analysis”(Berger and Berliner (1986); Gaver and O’ Muircheartaigh (1987); Berger et al. (1994)). However, it should be noted that our approach has a different goal from the g-modellings. We only estimate the shape of the null distribution of the mean parameter while the g-modeling models the marginal distribution of the mean parameter considering both the nulls and the non-nulls. Our purpose for estimating the null effect distribution is to set a cut-off for the strong signals adaptively, which is not the case for the g-modelling. Although our model can be considered as a special case of the ϵ-contamination model in the parameter space and a special case of the robust empirical Bayesian analysis, the use of these models for large effects have not been studied and, to the best of our knowledge, methods with provable guarantees on power and FDR have not been demonstrated previously.