Selecting ChIP-seq normalization methods from the perspective of their technical conditions

Abstract

Chromatin immunoprecipitation with high-throughput sequencing (ChIP-seq) provides insights into both the genomic location occupied by the protein of interest and the difference in DNA occupancy between experimental states. Given that ChIP-seq data are collected experimentally, an important step for determining regions with differential DNA occupancy between states is between-sample normalization. While between-sample normalization is crucial for downstream differential binding analysis, the technical conditions underlying between-sample normalization methods have yet to be examined for ChIP-seq. We identify three important technical conditions underlying ChIP-seq between-sample normalization methods: balanced differential DNA occupancy, equal total DNA occupancy, and equal background binding across states. To illustrate the importance of satisfying the selected normalization method’s technical conditions for downstream differential binding analysis, we simulate ChIP-seq read count data where different combinations of the technical conditions are violated. We then externally verify our simulation results using experimental data. Based on our findings, we suggest that researchers use their understanding of the ChIP-seq experiment at hand to guide their choice of between-sample normalization method. Alternatively, researchers can use a high-confidence peakset, which is the intersection of the differentially bound peaksets obtained from using different between-sample normalization methods. In our two experimental analyses, roughly half of the called peaks were called as differentially bound for every normalization method. High-confidence peaks are less sensitive to one’s choice of between-sample normalization method, and thus could be a more robust basis for identifying genomic regions with differential DNA occupancy between experimental states when there is uncertainty about which technical conditions are satisfied.

Introduction

In recent decades, high-throughput sequencing has become one of the most popular methods for data generation in genomics, epigenomics, and transcriptomics [1]. A popular method of high-throughput sequencing is chromatin immunoprecipitation with high-throughput sequencing (ChIP-seq). ChIP-seq typically involves shearing the DNA via sonication before conducting immunoprecipitation with an antibody that is known a priori to bind with the protein of interest (i.e. transcription factor, histone mark, etc.). Ideally, only the DNA fragments that are occupied by the protein of interest will remain after immunoprecipitation. These remaining fragments are then purified and sequenced before being aligned to a reference genome [2]. The aligned reads are then used to characterize the amount of occupancy of the protein of interest within a specific genomic region [3]. ChIP-seq experiments can be used as a binary measure of whether the protein of interest is bound or unbound at a genomic location. However, we focus on a different goal of many ChIP-seq experiments: comparing the amount of binding (at a particular site) across two experimental states (differential binding analysis). While identifying binding sites is the most common goal of ChIP-seq experiments, many researchers investigate questions about differential binding using ChIP-seq data (e.g. see [4–15]).

Given that ChIP-seq experiments are conducted to assess which genomic regions are truly differentially occupied by the protein of interest, regions enriched with DNA binding, peaks, typically serve as the unit of interest in ChIP-seq data analysis. In this paper, we refer to DNA occupancy (per cell) as the population parameter that we aim to estimate using ChIP-seq analysis and the sample estimate of the DNA occupancy (per cell) as the DNA binding (per cell). In this sense, a peak is considered differentially bound if there is a statistically significant difference in the amount of DNA binding in the peak region between the experimental states [16]. [In current literature, the population parameter and its sample estimate are usually both referred to as “DNA binding (per cell).” However, we use distinct terms in this paper to disambiguate when we are referring to the population parameter (DNA occupancy) versus its sample estimate (DNA binding)]. The raw amount of DNA binding within a given peak is calculated by counting the number of reads aligned to the specific genomic region. On average, genomic regions with higher read counts, i.e. more DNA binding (per cell), are expected to have a higher amount of DNA occupancy (per cell) [16].

That said, since ChIP-seq data are collected experimentally, there are expected differences in the observed DNA binding between the experimental states, even if there is the same amount of DNA occupancy in the states. Thus, hypothesis tests are essential for determining whether there is sufficient evidence that a difference in DNA binding between experimental states reflects a true difference in DNA occupancy between the states. The process of performing hypothesis tests to identify statistically significant differences in the DNA binding between experimental states is called differential binding analysis and is popular for analyzing other types of high-throughput data beyond ChIP-seq. For instance, Cleavage Under Targets and Release Using Nuclease (CUT&RUN) also leverages differential binding analysis to assess changes in protein–DNA interaction between experimental states [16, 17]. (CUT&RUN produces very similar data to ChIP-seq. The difference between the techniques is that in CUT&RUN, DNA fragments are generated by adding a protein A/G-micrococcal nuclease fusion protein after the primary antibody is added to the target of interest, rather than by sonicating the DNA as in ChIP-seq. After binding to the primary antibody, the fusion protein cuts the DNA around the target protein, producing sequenced fragments in CUT&RUN. CUT&TAG is a variation that utilizes tagmentation to generate cut fragments rather than using micrococcal nuclease).

Performing accurate differential binding analysis requires the raw read counts in each peak to be normalized between samples, since the raw read counts can be affected by experimental artifacts, such as variations in the amount of DNA loaded or the quality of the antibody used between samples [18]. Such experimental artifacts have the potential to influence the sequencing (or read) depth (i.e. the total number of reads in the entire sample) for the entire sample [16], meaning that differences in the raw read counts for a given peak can arise between experimental states even when there is no difference in the DNA occupancy between the states. (Note that other normalization methods, e.g. those that address GC content bias, should also be used when the analysis goals are not focused on comparing DNA occupancy of a single region across different experimental conditions [19], but in our work we focus only on between-sample normalization methods).

There are various between-sample ChIP-seq normalization methods available to researchers (e.g. spike-in methods, background-bin methods, and peak-based methods [20]). While the role of between-sample normalization methods has been analyzed through the lens of their technical conditions for RNA-seq [21], another popular type of high-throughput sequencing, there has been no parallel analysis of the between-sample normalization methods for ChIP-seq. However, ChIP-seq and RNA-seq data differ from each other in crucial ways. For one, RNA-seq focuses on characterizing (differential) gene expression, while ChIP-seq focuses on characterizing (differential) DNA occupancy. As a result, ChIP-seq data do not have predefined genomic regions of interest, whereas genes serve as the genomic regions of interest in RNA-seq data. Indeed, the genomic regions of interest in ChIP-seq are defined through the process of peak calling, which leverages hypothesis testing (or other statistical tools, such as Hidden Markov Models [22]) to identify genomic regions that are significantly enriched with DNA binding [23]. Further, because the experimental processing of ChIP-seq samples involves many steps over multiple days, and because both antibody quality and cell number contribute to the level of background noise, the signal-to-noise ratio can be quite variable between samples in ChIP-seq data compared with that in RNA-seq data [24]. For these reasons, we cannot directly apply the results from previous RNA-seq between-sample normalization analyses to ChIP-seq between-sample normalization methods. Rather, between-sample normalization methods available for ChIP-seq data and other similarly structured types of high-throughput data must be explicitly analyzed through the lens of their own technical conditions.

In this paper, we identify three key technical conditions underlying between-sample normalization methods for ChIP-seq and similarly structured types of high-throughput data: (1) balanced differential DNA occupancy, (2) equal total DNA occupancy across experimental states, (3) equal background binding across experimental states. We simulate ChIP-seq read count data to demonstrate how violating these technical conditions can substantially impact the accuracy of downstream differential binding, leading to differential binding analysis with higher empirical false discovery rates (FDRs) and lower power, on average. We also use experimental CUT&RUN [25] and ChIP-seq [12] data to validate that normalization methods that rely on the same technical condition yield similar “normalized” results. Additionally, the experimental data analyses demonstrate the applicability of using a high-confidence peakset, which only contains the peaks identified as differentially bound regardless of the between-sample normalization method used. Based on our results, we recommend that researchers use their understanding of the experiment at hand to guide their choice of ChIP-seq between-sample normalization method when possible. Alternatively, researchers could conduct analogous data analyses in which they only vary the selected between-sample normalization method when there is uncertainty about which technical conditions are satisfied. A separate differentially bound peakset would then be generated for each between-sample normalization method. The researcher could then use an intersection of these peaksets to create a high-confidence peakset that would be more robust to violations of the between-sample normalization technical conditions than the differentially bound peakset associated with any singular between-sample normalization method. This, in turn, would limit the impact of the choice of between-sample normalization method on the biological conclusions drawn from downstream differential binding analysis.

Normalization and differential binding analysis

In a typical ChIP-seq workflow, samples are cross-linked with formaldehyde to fix DNA-bound proteins, and the chromatin is sheared. Antibodies are added to the protein of interest to immunoprecipitate bound DNA, which is then separated from the proteins and sequenced. Input DNA is collected prior to immunoprecipitation as a control. Additionally, some researchers will use an irrelevant IgG antibody for immunoprecipitation as an extra control. After the DNA is sequenced, it is aligned to the reference genome and analyzed using a peak caller. Peak callers typically compare the amount of DNA immunoprecipitated with the antibody against the protein of interest to the DNA collected from the input control or the IgG control antibody to call regions of the genome with statistically significant pile-ups of reads, or peaks for the immunoprecipitated DNA [24]. These regions with statistically significant DNA binding are assumed to be bound by the protein of interest. Additional controls with knockout strains can help to remove any spurious peaks from the dataset. Once the peaks are identified for a given experimental state, researchers might wish to interrogate whether the DNA occupancy for the protein of interest varies between experimental states [4–15].

In ChIP-seq differential binding data analysis, between-sample normalization is performed on the consensus peakset across experimental states, i.e. the subset of peaks that are consensus peaks in at least a certain number of experimental states. For a peak to be considered a consensus peak within an experimental state, it must be called as a peak within at least a certain number (or proportion) of replicates [26]. After the consensus peaks across experimental states have been identified, a read count matrix can be generated, where each entry corresponds to the number of reads aligned to a given consensus peak within a given sample [20]. The raw read counts are then normalized between samples and sometimes also within samples. For between-sample normalization, the normalized read count for consensus peak in sample () is typically computed as the raw read count for consensus peak in sample () divided by the sample-wide size factor for sample (). Note that some between-sample normalization methods that we consider in this paper, such as MAnorm2 (Common Peaks) and Loess Adjusted Fit (Reads in Peaks), normalize the raw reads by scaling them via (local) regression instead of generating a sample-wide size factor for each replicate.

(1)

The overarching goal of between-sample normalization for ChIP-seq is to eliminate discrepancies in the read counts between sample groups that are due to experimental artifacts rather than genuine changes in the DNA occupancy between experimental states. Experimental artifacts, such as the amount of DNA loaded into the sequencer, the quality of the antibody, the starting cell number, and how many reads are returned from the sequencer, can influence how many reads are aligned to a particular genomic region within a given sample [16]. As a result, between-sample normalization is considered to have correctly normalized the ChIP-seq data between experimental states when the relationship between the samples’ normalized read counts tracks the true relationship between the states’ DNA occupancy levels. In other words, between-sample normalization is considered correct if peaks that do not have differential DNA occupancy between samples have the same normalized read counts across experimental states, on average, and peaks that do exhibit differential DNA occupancy between states have different normalized read counts across experimental states, on average. Figure 1 demonstrates the difference between incorrect and correct ChIP-seq between-sample normalization through a toy example, where we assume peak calling has already been performed (e.g. using input or IgG controls) (figure adapted from Evans et al. [21]). Comparing (a) and (c) in Figure 1, we see that the total library size is the same across states A and B, even though state B has more total DNA occupancy per cell. Correct normalization returns the true fold change A/B relationship in the DNA occupancy for all three peaks, i.e. the fold change A/B relationship in (a). In contrast, Library Size (Reads in Peaks) normalization does not return the true fold change relationship from (a) and, instead, incorrectly indicates that there is a change in the DNA occupancy per cell in all three peaks across states A and B.

Figure 1.

A toy example showing the importance of correct between-sample normalization to downstream differential binding analysis. (a) the true DNA occupancy per cell for the three peaks across experimental states A and B; note only Peak 3 (in purple) has differential DNA occupancy between A and B, with a higher amount of DNA occupancy per cell in state B. (b) the proportional shares of DNA occupancy in the three peaks in the experimental states. (c) the reads aligned to each peak in each state. (d) the normalized read count associated with each peak with no normalization (i.e. the normalized reads are simply the raw reads), Library Size (Reads in Peaks) normalization, and correct normalization. (e) the fold change A/B associated with each peak for the aforementioned normalization methods. Figure adapted from Evans et al. [21].

Importance of correct normalization to differential binding analysis

Correct between-sample normalization is crucial for meaningful differential binding analysis. Indeed, Wu et al. [3] note that out of all the ChIP-seq data analysis steps they analyzed, between-sample normalization has the greatest potential to influence the discovery of differentially bound regions. (Relatedly, Reske et al. [27] demonstrate that normalization method can significantly affect differential accessibility analysis and interpretation in ATAC-seq data). Our focus in this paper is to interrogate common between-sample normalization methods after peaks have already been called using appropriate controls. One important experimental method for normalizing between ChIP-seq samples is through the addition of spike-in DNA controls, where an equal amount of DNA that does not correspond to the genome of interest is added to each experimental sample prior to the immunoprecipitation step, which can help control for differences in genomic shearing or immunoprecipitation efficiency [28]. However, spike-in DNA is not always included for every ChIP-seq experimental dataset. Figure 1 walks through a toy example with no spike-in data, where a standard normalization technique, Library Size (Reads in Peaks), leads to incorrect between-sample normalization, and consequently incorrect differential binding analysis results. In the toy example, there is unequal DNA occupancy between experimental states A and B, with state B having a higher amount of DNA occupancy than state A. Additionally, only Peak 3 exhibits differential DNA occupancy between the experimental states, with it having double the amount of DNA occupancy in state B compared with in state A. As we highlight later on, Library Size (Reads in Peaks) relies on the technical condition that there is equal DNA occupancy across the experimental states. As a result, when the read counts in the toy example are normalized via Library Size (Reads in Peaks), all three peaks appear to be differentially bound between states A and B, even though only Peak 3 has differential DNA occupancy between the states (see Figure 1). The relationship between the normalized reads for Peak 3 also does not align with the relationship between the DNA occupancy for Peak 3 when Library Size (Reads in Peaks) normalization is used. Hence, the toy example underscores that understanding the technical conditions of the selected between-sample normalization method and whether they are violated in a particular instance is crucial to ensuring accurate differential binding analysis results.

In the appendix, we detail frequently assumed technical conditions and connect them to common ChIP-seq between-sample normalization methods (see Table 2 for a summary), many of which are available through the DiffBind R package, which is specifically designed for identifying differentially bound peaks between sample groups [20].

Table 2.

Summary of technical conditions underlying different between-sample normalization methods.

Technical condition	Normalization methods
Balanced Differential DNA Occupancy	TMM (Reads in Peaks)
	RLE (Reads in Peaks)
	MAnorm2 (Common Peaks)
Equal Total DNA Occupancy	Library Size (Reads in Peaks)
	Loess Adjusted Fit (Reads in Peaks)
Equal Background Binding	TMM (Background Bins)
	RLE (Background Bins)
	Library Size (Background Bins)

Simulations

To demonstrate the importance of satisfying the technical conditions of the selected between-sample normalization method on downstream differential binding analysis, we simulate ChIP-seq read count data across two experimental states under eight distinct conditions (see Table 1). Our ChIP-seq read count data simulation primarily builds on code from Lun and Smyth [29] as well as from Evans et al. [21]. The code for the ChIP-seq simulations, result figures, experimental data analysis, and all toy examples is available in the GitHub repository associated with this paper (https://github.com/scolando/ChIP-Seq-norm).

Table 1.

Summary of the eight unique simulation conditions.

Balanced	Equal occupancy	Equal background	% Peaks with more Occ. (A)	% Peaks with More Occ. (B)	FC (A)	FC (B)	Avg. % More Background/Repa
✓	✓	✓
✓	X	✓
X	✓	✓
X	X	✓
✓	✓	X
✓	X	X
X	✓	X
X	X	X

^a The experimental state with more background binding per replicate is chosen at random in each simulation iteration. FC = fold change in DNA occupancy when comparing non-differentially occupied peaks to peaks with more DNA occupancy in the given state. Occ. = DNA Occupancy.

Simulation details

We simulate a total of eight distinct conditions, which we summarize in Table 1. The eight simulation conditions cover all possible combinations of whether the three primary technical conditions—balanced differential occupancy, equal DNA occupancy across experimental states, and equal background binding across the experimental states—are met or violated.

Our simulations focus on the occupancy of a simple transcription factor on an artificial chromosome that is roughly 150 million base pairs long. Note that simple transcription factors often generate narrower and taller peaks than other proteins that might be of interest in ChIP-seq experiments, such as histone marks [26]. Each simulation consists of two experimental states, denoted by A and B, and three replicates per experimental state (see the Appendix for simulation results with two replicates per experimental state).

Leveraging the simple transcription factor simulation code from Lun and Smyth [29], we initialize SAM files for each replicate. Within each SAM file, we simulate 10,000 narrow peaks (with peak width of 200 base pairs) whose summit locations are 15,000 base pairs apart from each other, on average, with the total proportion of peaks with differential DNA occupancy between states A and B ranging from 0.05 to 0.95 (with a step-size of 0.05). These reads are sampled from a negative binomial distribution, parametrized by the mean and dispersion [21, 29]. The mean of the negative binomial distribution represents the average amount of DNA occupancy within a given genomic region. Meanwhile, the dispersion represents the variability in the amount of DNA occupancy within a given genomic region. Like Lun and Smyth [29], we generate this dispersion parameter by sampling from an inverse chi-square distribution with 10 degrees of freedom. However, we vary the mean of the negative binomial distribution according to the specific simulation condition (see the simulation code for details). Once the 10,000 peaks have been simulated for each replicate, we add background reads to the SAM files by sampling from a uniform distribution (the minimum and maximum of the uniform distribution also vary according to the simulation condition). We create an input control for each replicate by sampling background reads from a uniform distribution and adding them to a new SAM file that contains no simulated peaks.

After the SAM files are created for both the replicates and their corresponding input controls, we convert them to BAM files using the xscss R package (https://bioinf.wehi.edu.au/csaw/). We then perform peak calling with MACS2 on the BAM files using the parameters -g 300734518 --mfold 2 50 --keep-dup=all -f BAM, and supply the corresponding input control for each replicate to increase the peak-calling specificity [23]. Using DiffBind, we identify the consensus peaksets within experimental states A and B for the given simulation iteration using a minimum overlap of 75%. We combine these peaksets such that any peak that appears in at least one consensus peakset within an experimental state is included in the consensus peakset across states, which we use for downstream differential binding analysis. We use MACS2 as the peak caller in our simulation for two reasons. First, our focus is on normalization methods that occur after peak calling; thus, adding in different peak callers as an additional variable would make it harder to directly compare how different normalization methods affect the results. Second, the experimental datasets presented in our analysis both used MACS2 as the peak caller. Using the same peak caller for both the simulated and experimental datasets makes it easier to compare how the normalization method affects the results for both the experimental and simulated data. That said, we recognize that there are other peak callers (e.g. see [30]) that are suitable for analyzing broad, flat peaks, or other more complex DNA occupancy events rather than the DNA-sequence specific, high, sharp peaks for which MACS2 is optimized [23].

Next, we perform between-sample normalization using the DiffBind-supported between-sample normalization methods as well as MAnorm2, a package developed by Tu et al. [31], which uses the common peakset across experimental states for normalization [20, 31]. For the DiffBind-supported normalization methods, we conduct our differential binding analysis in DiffBind [20]. We use the differential binding analysis developed in conjunction with the MAnorm2 (Common Peaks) normalization method to perform differential binding analysis on the MAnorm2-normalized read counts. For both differential binding analysis procedures, we set the FDR threshold to 0.05, meaning that we only classify peaks with Benjamini-Hochberg adjusted -values below 0.05 as differentially bound. For each simulation iteration, we return the number of true positive, true negative, false positive, and false negative differentially bound peaks associated with each normalization method, along with metadata about the simulation iteration. We iterate through this entire data-generation and differential binding analysis workflow 100 times for each combination of simulation condition and proportion of peaks with differential DNA occupancy.

To evaluate the performance of each between-sample normalization method, we devise an omniscient Oracle normalization method, which derives its sample-wide size factors directly from the simulation parameters (described in more detail in the appendix). We use the Oracle normalization method as the basis of comparison since peaks could be incorrectly identified as differentially occupied, or differentially occupied peaks could remain undiscovered, even with perfect normalization. For example, a peak with differential DNA occupancy might not be called as a peak during peak calling due to low signal. Alternatively, a peak with true differential occupancy might have different normalized read counts between experimental states but have an adjusted -value above the pre-specified FDR threshold due to a low number of replicates in the experimental states.

We directly compare the sample-wide size factors generated by the Oracle to those generated by other between-sample normalization methods by computing the average absolute size factor ratio relative to the Oracle. Let denote the sample-wide size factor corresponding to sample for normalization method , and denote the sample-wide size factor corresponding to sample for the Oracle. Then, the absolute size factor ratio, which we denote as , is defined by the following piecewise function:

(2)

The average absolute size factor ratio relative to the Oracle for normalization method () for a given simulation condition is then defined as the mean of the absolute size factor ratios for normalization method relative to the Oracle across all samples:

(3)

We use the metric, along with the average empirical FDR and average power, to assess the relative performance of the different between-sample ChIP-seq normalization methods in the subsequent section.

The three simulation conditions that we assess are (1) balanced differential DNA occupancy, (2) equal total DNA occupancy, and (3) equal background binding. Balanced differential occupancy refers to the setting where the number of peaks with more DNA occupancy in state A is the same as the number of peaks with more DNA occupancy in state B (see Figure S4 in the supplementary materials). Equal total DNA occupancy refers to the setting where the total DNA occupancy in states A and B are the same (see Figure S3 in the supplementary materials). Equal background binding refers to the setting where there is an equal number and distribution of rogue reads (i.e. binding in regions that are not truly occupied by the protein of interest) across states A and B (see Figure S2 in the supplementary materials).

Simulation results

Table 2 connects each normalization method covered in our simulation study to the appropriate technical condition for that method. Figure 2 shows the average empirical FDR across our eight simulation conditions for each between-sample normalization method. (n.b., the average FDR will naturally decrease as the proportion of peaks with differential DNA occupancy increases since there are fewer peaks to falsely discover as differentially bound when the proportion of peaks differentially occupied between states increases). Notably, we see that when the technical conditions of a normalization method are violated, the average empirical FDR increases, becoming larger than the average empirical FDR associated with the Oracle—our omniscient normalization method. In other words, a higher proportion of the peaks that are identified as differentially bound do not have true differential DNA occupancy between the experimental states when the technical conditions for the selected normalization method are violated. For example, columns 3 and 4 of Fig. 2 show that when the technical assumption of balanced differential DNA occupancy is violated, TMM (Reads in Peaks), RLE (Reads in Peaks), and MAnorm2 (Common Peaks) perform poorly, with their average empirical FDR diverging from that of the Oracle. Meanwhile, when there is balanced differential occupancy between the experimental states (see columns 1 and 2 in Fig. 2), these normalization methods have average empirical FDR values that track closely to that of the Oracle. Likewise, between-sample normalization methods that rely on the technical condition of equal DNA occupancy between experimental states, such as Library Size (Background Bins), Library Size (Reads in Peaks), and Loess Adjusted Fit (Reads in Peaks), have an increased average empirical FDR when there is unequal DNA occupancy between the experimental states, diverging from the Oracle’s average empirical FDR curve (see columns 2 and 4 of Fig. 2). On the other hand, when there is equal DNA occupancy between experimental states, these methods track closely to the Oracle’s average empirical FDR curve (see columns 1 and 3 of Fig. 2). Finally, Fig. 2 demonstrates that normalization methods that use background bins rather than peaks to normalize the read counts between experimental states are affected by both unequal background binding and unequal DNA occupancy across experimental states. We hypothesize that unequal DNA occupancy impacts the average empirical FDR of between-sample normalization methods that use background bins because highly unequal DNA occupancy, particularly when there is a high proportion of peaks with differential occupancy, would likely cause non-trivial differences in the background binding between the experimental states.

Figure 2.

The average empirical FDR for each normalization method, faceted by simulation condition (three replicates per experimental state and 100 simulation iterations per combination of simulation condition and proportion of peaks with differential DNA occupancy). The dashed horizontal line denotes the pre-specified FDR threshold of We consider normalization methods that track with the Oracle (the solid black line) to be performing well with respect to the average empirical FDR.

To further understand how violating the technical conditions underlying the selected between-sample normalization method impacts differential binding analysis results, we next examine the average power. Figure 3 shows that the average power associated with a normalization method’s performance also tends to decrease when its technical conditions are violated. That is, peaks with differential DNA occupancy are generally less likely to be identified as differentially bound when the technical conditions for the selected normalization method are violated. For example, the average power decreases as the proportion of peaks with differential DNA occupancy increases for methods that use the mean or median peak reads to normalize between samples—namely, TMM (Reads in Peaks), RLE (Reads in Peaks), and MAnorm2 (Common Peaks)—when there is unbalanced differential occupancy between the experimental states (see columns 3 and 4 of Fig. 3). Thus, we would expect to have low sensitivity in downstream differential binding analysis if we were to use these normalization methods when there is unbalanced differential DNA occupancy between experimental states.

Figure 3.

The average power for each normalization method, faceted by simulation condition (three replicates per experimental state and 100 simulation iterations per combination of simulation condition and proportion of peaks with differential DNA occupancy). The nearer a curve is to the dashed horizontal black line at , the higher power the associated normalization method has (on average) for the given proportion of peaks with differential DNA occupancy.

Importantly, Fig. 3 highlights that the average power does not universally decrease when the technical conditions for the selected normalization method are violated. In particular, normalization methods that use background bins for normalization or that normalize using library size maintain high average power even when there is unequal total DNA occupancy between experimental states. This high average power, coupled with the high average empirical FDR that we observe in Fig. 2, suggests that the methods that use background bins or library size to normalize between states lead to low specificity in identifying peaks that are differentially occupied in cases with unequal DNA occupancy between experimental states.

In the subsequent subsections, we delve deeper into how violating each of the three technical conditions—balanced differential DNA occupancy, equal total DNA occupancy, and unequal background binding—affects the performance of between-sample normalization methods, as measured by the average empirical FDR, power, and absolute size factor ratio relative to the Oracle. In particular, we demonstrate how violating the technical conditions for selected normalization methods leads to incorrect between-sample normalization, which ultimately results in the misidentification of peaks with differential DNA occupancy between experimental states.

All technical conditions met

Per Fig. 4, all normalization methods achieve an average empirical FDR close to that of the Oracle when the three technical conditions are met. Notably, MAnorm2 (Common Peaks) has an average empirical FDR lower than the Oracle’s (and below the pre-specified FDR threshold of 0.05) regardless of the proportion of peaks with differential DNA occupancy. However, the middle panel of Fig. 4 indicates that MAnorm2 (Common Peaks) also has the uniformly lowest average power of all normalization methods, providing evidence that MAnorm2 (Common Peaks) leads to more conservative differential binding analysis than the other normalization methods we tested.

Figure 4.

Simulation results when all three technical conditions are met (three replicates per experimental state and 100 iterations per simulation condition and proportion of peaks with differential DNA occupancy). In the left panel, the dashed horizontal black line represents the pre-specified FDR threshold of . In the middle panel, the dashed horizontal black line at represents the highest possible average power. In the right panel,the solid horizontal line at represents the Oracle’s average absolute size factor ratio with itself. Thus, a normalization method tracks closely with the Oracle if its size factor curve is close to the solid horizontal black line at . Recall that MAnorm2 (Common Peaks) and Loess Adjusted Fit (Reads in Peaks) do not generate sample-wide size factors and so do not have curves in the right panel.

Additionally, the right panel of Fig. 4 shows that the average size factors corresponding to the background binding normalization methods—RLE (Background Bins), TMM (Background Bins), and Library Size (Background Bins)—are different from the Oracle’s size factors even when all three technical conditions are met, with an average absolute size factor ratio relative to the Oracle ranging between roughly 1.12 and 1.15. Despite the size factor discrepancy, normalization methods that use background bins still maintain an average empirical FDR and power that track closely to the Oracle’s. As such, we posit that the size factor discrepancy between normalization methods and the Oracle is due to these methods using background bins instead of peaks as the unit of normalization.

Effect of unbalanced differential DNA occupancy

When the technical condition of balanced differential DNA occupancy between experimental states is violated, but the other two technical conditions are still met, TMM (Reads in Peaks), RLE (Reads in Peaks), and MAnorm2 (Common Peaks) diverge from the Oracle across all three evaluation metrics. Indeed, as shown in the left and middle panels of Fig. 5, RLE (Reads in Peaks), TMM (Reads in Peaks), and MAnorm2 (Common Peaks) all fail to control the average empirical FDR and exhibit a lower average power than the Oracle, particularly as the proportion of peaks with differential DNA occupancy increases. An explanation for this is that the peak with the mean or median read count, which these methods use to normalize between samples (see Appendix), is more likely to be a differentially occupied peak when the proportion of peaks with differential DNA occupancy increases.

Figure 5.

Simulation results when only balanced differential DNA occupancy between experimental states is violated (three replicates per experimental state and 100 iterations per simulation condition and proportion of peaks with differential DNA occupancy). In the left panel, the dashed horizontal black line represents the pre-specified FDR threshold. In the middle panel, the dashed horizontal black line at represents the highest possible average power. In the right panel, the solid horizontal line at represents the Oracle’s average absolute size factor ratio with itself. Thus, a normalization method tracks closely with the Oracle if its size factor curve is close to the solid horizontal black line at in the right panel. Recall that MAnorm2 (Common Peaks) and Loess Adjusted Fit (Reads in Peaks) do not generate sample-wide size factors and so do not have curves in the right panel.

The average size factors associated with RLE (Reads in Peaks) and TMM (Reads in Peaks) also diverge from the Oracle’s as the proportion of peaks with differential DNA occupancy increases (see the right panel of Fig. 5). Note that MAnorm2 (Common Peaks) is not depicted in the right panel of Fig. 5 since it does not generate sample-wide size factors [31]. The divergence in the average size factors generated by RLE (Reads in Peaks) and TMM (Reads in Peaks) from the Oracle verifies that these methods lead to inaccurate between-sample normalization when there is unbalanced differential DNA occupancy between states.

Effect of unequal total DNA occupancy

TMM (Background Bins), Library Size (Background Bins), RLE (Background Bins), Library Size (Reads in Peaks), and Loess Adjusted Fit (Reads in Peaks) all fail to maintain an average empirical FDR close to the Oracle’s when there is unequal total DNA occupancy between experimental states (see the left panel of Fig. 6). That said, in the middle panel of Fig. 6, we see that all of these normalization methods still maintain a high average power. This high average power, coupled with the high average empirical FDR, indicates that TMM (Background Bins), Library Size (Background Bins), RLE (Background Bins), Library Size (Reads in Peaks), and Loess Adjusted Fit (Reads in Peaks) result in over-calling peaks as differentially bound, leading to low specificity in differential binding analysis, when their technical condition of equal DNA occupancy between states is violated.

Figure 6.

Simulation results when only equal total DNA occupancy between experimental states is violated (three replicates per experimental state and 100 iterations per simulation condition and proportion of peaks with differential DNA occupancy). In the left panel, the dashed horizontal black line represents the pre-specified FDR threshold. In the middle panel, the dashed horizontal black line at represents the highest possible average power. In the right panel, the horizontal black line at represents the Oracle’s average absolute size factor ratio with itself. Thus, a normalization method tracks closely with the Oracle if its size factor curve is close to the solid horizontal black line at in the right panel. Recall that MAnorm2 (Common Peaks) and Loess Adjusted Fit (Reads in Peaks) do not generate sample-wide size factors and so do not have curves in the right panel

While average absolute size factor ratio relative to the Oracle increases for all normalization methods with the proportion of peaks with differential DNA occupancy in Fig. 6, normalization methods that use background bins uniformly have the largest average absolute size factor ratio relative to the Oracle. This could be because when the proportion of peaks with differential occupancy increases, so does the difference in the binding in background bins across the experimental states. As a result, the normalization methods that use background bins diverge further from the Oracle’s and the average size factors for background binding normalization methods were already different from the Oracle’s to begin with, even when all three technical conditions are met.

Effect of unequal background binding

Finally, when there is unequal background binding between experimental states, the between-sample normalization methods that use background bins have a higher average empirical FDR and lower average power than the Oracle (see the left and middle panel of Fig. 7). Meanwhile, the average absolute size factor ratio relative to the Oracle is still approximately constant for TMM (Background Bins), RLE (Background Bins), and Library Size (Background Bins) even as the proportion of peaks with differential DNA occupancy increases (see the right panel of Fig. 7). Given that the size factor ratio relative to the Oracle is roughly constant, we posit that the average empirical FDR decreases for background binding normalization methods as the proportion of peaks with differential DNA occupancy increases, not because the methods improve as the proportion of peaks with differential DNA occupancy increases, but instead, because there are fewer peaks that can be falsely discovered as differentially bound.

Figure 7.

Simulation results when only equal background binding between experimental states is violated (three replicates per experimental state and 100 iterations per simulation condition and proportion of peaks with differential DNA occupancy). In the left panel, the dashed horizontal black line represents the pre-specified FDR threshold. In the middle panel, the dashed the horizontal black line at represents the highest possible average power. In the right panel the solid horizontal line at represents the Oracle’s average absolute size factor ratio with itself. Thus, a normalization method tracks closely with the Oracle if its size factor curve is close to the solid horizontal black line at in the right panel. Recall that MAnorm2 (Common Peaks) and Loess Adjusted Fit (Reads in Peaks) do not generate sample-wide size factors and so do not have curves in the right panel.

Experimental results

In this section, we analyze two experimental datasets (one CUT&RUN and one ChIP-seq) to illustrate how selecting different between-sample normalization methods can affect the results of differential binding analysis, as well as the similarity between different normalization methods that rely on the same underlying technical conditions. Note that spike-in DNA is processed in the CUT&RUN data (Example 1), so we include spike-in normalization methods in our analysis. However, spike-ins are not included in the ChIP-seq data (Example 2), and therefore we cannot incorporate spike-in normalization methods into our analysis of this dataset.

Analysis considerations

While most of the normalization methods analyzed in our simulation use the exact same regions of interest, the regions of interest for MAnorm2 (Common Peaks) are computed slightly differently, and are thus not identical to those generated by the other methods. Because the simulated data consist of sharp, narrow peaks, the regions of interest for MAnorm2 (Common Peaks) and the other methods align perfectly. However, in the experimental data examples, the regions of interest do not align well. In particular, MAnorm2 (Common Peaks) profiles smaller regions of interest than DiffBind. This makes comparing results between MAnorm2 (Common Peaks) and other methods more difficult for experimental data because the regions of interest being compared are not the same. For this reason, we left the MAnorm2 (Common Peaks) method out of the analysis on experimental data, and focused on the methods that use identical regions of interest to ensure a fair apples-to-apples comparison.

Another key consideration for analyzing ChIP-seq data is how to incorporate controls in the analysis pipeline. Our simulation analysis includes simulated input data that are utilized by MACS2 to determine whether a particular genomic region is called as a peak. If IgG data are available, they can also be analyzed as an experimental sample with input controls using MACS2 to identify spurious peaks. In the experimental results, both input and IgG controls are used to generate greylists that identify and filter out spurious peaks from the consensus peakset. Additionally, knockout data can be used as a control to validate peaks, since a valid peak should disappear when the gene corresponding to the protein of interest has been removed from the genome (e.g. see [32]). Thus, peaks that appear in both the wildtype and knockout samples should be eliminated as spurious. CUT&RUN experiments do not have input data because sequences unbound by the protein of interest remain in the nucleus and are not sequenced [17]. In these cases, IgG data can instead be used by MACS2 (or an alternate peak-calling algorithm) as the control sample, as we do here with experimental CUT&RUN data. Once the regions of interest are identified by the peak-calling algorithm, downstream analysis focuses on determining whether the identified peaks have differential occupancy between experimental states. Input and IgG controls do not play a role in this latter part of the analysis, which determines the peaks with differential occupancy between states.

Further, researchers must determine which peak-caller algorithm to use in ChIP-seq analysis. The MACS2 algorithm, which we use for both simulated data and the experimental datasets, is optimized for identifying sequence-specific DNA-binding factors that bind to a narrow region of DNA. The MACS2 algorithm, however, also contains functionality for identifying broad peaks, which was used for the experimental CUT&RUN data [25]. For experimentalists studying histone marks that occur over broader genomic regions, a different peak-calling algorithm, such as SICER [33], which is optimized for detecting broad, low peaks, might be more suitable.

Reconciling peaks called between different replicates in an experimental state is also an important consideration in ChIP-seq analysis. In our simulations and experimental analyses, we use the narrowest region of intersection between all replicates to define the peak width for a particular peak of interest (the default in DiffBind [20]). However, this could have downstream consequences for differential occupancy analysis if a large number of peaks results in a wider width for each merged peak. If visualization of peak width indicates that the width is changing significantly between experimental states, an alternate strategy such as Hidden Markov Model to identify differential occupancy might be more appropriate, e.g. using the program ODIN [22].

Example 1: dynamic bromodomain protein 3 occupancy during African trypanosome differentiation

The first experimental dataset we analyze, from Ashby et al. [25], characterizes the dynamic occupancy of the bromodomain protein 3 (Bdf3) in the eukaryotic protozoan parasite Trypanosoma brucei during differentiation from bloodstream to insect forms. The dataset was generated by endogenously tagging a chromatin interacting bromodomain protein (Bdf3) with hemagglutinin (HA) and performing CUT&RUN using an anti-HA antibody [25, 34] to investigate whether Bdf3 chromatin occupancy changes as parasites transition from the mammalian bloodstream phase of the life cycle to the procyclic gut stage found in the parasite tsetse fly vector. Samples were harvested from bloodstream parasites and from several time points after differentiation to produce the data. Spike-in reads from yeast DNA were added to allow for spike-in normalization during analysis. Previous analysis by Ashby et al. [25] generated a high-confidence differentially bound peakset using Library Size (Spike-in), RLE (Background Bins), RPKM, and RLE (Spike-in) between-sample normalization methods. This analysis showed that occupancy of Bdf3 is altered at hundreds of sites as parasites transition from bloodstream to procyclic forms.

The single-end sequencing of CUT&RUN libraries resulted in reads per library. The number of mapped reads ranged from 70%–74%, possibly due to the quality of the genome sequence. To analyze the CUT&RUN data, raw Fastq files were trimmed for quality, and adapter sequences were removed. The resulting sequences were aligned to the Tb927v5.1 trypanosome genome using bowtie [35], requiring unique alignments using the parameters: -best -strata -t -v 2 -a -m 1. Spike-in reads were aligned to the yeast genome (sacCer3/R64). BAM files from the alignment were then analyzed using MACS2 in broad-peak mode to conduct peak calling on each replicate with these parameters: -g 23650671, --keep-dup all, --nomodel, --broad. Since there are no input controls for CUT&RUN data (the sequences unbound by the protein of interest remain in the nucleus and are not sequenced) [17], we use an IgG sample in MACS2 as the control sample to increase peak-calling specificity. The Fraction of Reads in Peaks (FRiP) scores ranged between 0.31 and 0.36, exceeding the ENCODE standard of 0.1 [26]. The Fastq files are available in the SRA database under project number PRJNA795567 (https://www.ncbi.nlm.nih.gov/sra/?term=PRJNA795567).

To provide a simple comparison between the different between-sample normalization methods available for ChIP-seq and other similarly structured types of high-throughput data, such as CUT&RUN, we examine the similarity in the normalized log2 fold change between experimental states (i.e. the log2 fold changes changes after between-sample normalization has been performed). For our analysis, we use the 3 hour (3h) mark after differentiation as our comparison point to the bloodstream (0h) samples, as it is when the greatest change in DNA occupancy is observed relative to the bloodstream samples [25]. Using MACS2 with the parameters specified above, 793 consensus peaks were identified in the bloodstream 0h samples, and 576 consensus sites were identified in the 3h differentiated samples (using a minimum overlap of three replicates). We next applied a greylist generated from the IgG control to filter out spurious peaks using the GreyListChIP R package [36]. After filtering out spurious peaks, we implemented the same between-sample normalization methods as we used in our simulation, as well as spike-in normalization methods, since spike-in DNA was also processed in the experiment. MAnorm2 (Common Peaks) was not included for the reasons cited above. Finally, we identified differentially bound regions between bloodstream 0h and 3h samples using DiffBind (FDR < 0.05).

We compare the normalized log2 fold changes between experimental states for each between-sample normalization method using principal component analysis (PCA) (Fig. 8). An UpSet plot of the differentially bound peaksets was also generated with each between-sample normalization method (Fig. 9). Since we cannot display every possible distinct intersection of the 10 differentially bound peaksets in the UpSet plot, Fig. 9 provides information on the twenty largest groups of distinct overlaps between the differentially bound peaksets.

Figure 8.

PCA of the log2 fold changes after between-sample normalization has been performed between the bloodstream parasites and those induced to transition into procyclic forms at the 3h timepoint.

Figure 9.

UpSet plot (for the 20 largest distinct groupings) comparing the peaks identified as differentially bound between the bloodstream parasites and those induced to transition into procyclic forms at the 3h timepoint after selecting different between-sample normalization methods. One hundred and seventy peaks were identified as differentially bound by all ten between-sample normalization methods that we tested.

The PCA results support our simulation findings that between-sample normalization methods with the same technical conditions yield very similar normalization outcomes. Notably, the first principal component, which explains nearly all of the variability in the normalized log2 fold changes (99.45%), distinguishes peak-based normalization methods from those that use spike-ins or background bins for between-sample normalization. Within the peak-based normalization methods, Loess Adjusted Fit (Reads in Peaks) and Library Size (Reads in Peaks) produce extremely similar normalized log2 fold changes (see Fig. 8). This is expected since both normalization methods require an equal amount of total DNA occupancy between states, as we observed in our simulations (see Fig. 6). The second principal component, which explains 0.40% of the variability in the normalized log-2 fold changes, separates the background binding normalization methods from spike-in normalization methods. The primary technical conditions for spike-in normalization methods are that the spike-in DNA comes down in the immunoprecipitation at the same frequency regardless of the experimental states and that experimental artifacts have the same effects on spike-in DNA and the experimental samples. On the other hand, background binding normalization methods rely on equal background binding across experimental states for accurate normalization (see Fig. 7).

In total, between 270 and 457 peaks were called as differentially bound for each normalization method in this dataset, which verifies our simulation results that the choice of between-sample normalization has a substantial impact on which peaks are identified as differentially bound, especially when not every technical condition is satisfied. A total of 170 peaks, or between 38% and 63% of the differentially bound peaks found by each method, were identified as differentially bound by every selected normalization method. Leveraging the language from Ashby et al. [25], these 170 peaks in the distinct intersection between all normalization methods constitute a high-confidence peakset, as they were found to have differential DNA binding between the 3h differentiation and the 0h bloodstream state regardless of the between-sample normalization method used. As Ashby et al. [25] emphasize, the choice of normalization method has an “outsized effect” on which regions are identified as differentially bound between experimental states. The high-confidence peakset circumvents this problem by containing only differentially bound peaks that are robust to the choice of between-sample normalization method.

Example 2: differential estrogen receptor- binding in breast cancer cell lines

For our second experimental analysis, we use a subset of ChIP-seq data from Ross-Innes et al. [12], which profiles estrogen receptor- (ER) binding in tamoxifen-resistant ER-positive cell lines (in T-47D, ZR75-1, and MCF-7 tissues) versus tamoxifen-responsive ER-positive cell lines (in TAM-R and BT-474 tissues) for breast cancer patients with distinct clinical outcomes. As previously mentioned, this dataset does not include spike-in controls. In our analysis of the data, we aim to identify genomic regions with differential occupancy of the ER transcription factor between tamoxifen-resistant and tamoxifen-responsive cell types. Specifically, we want to detect differentially bound genomic regions that are robust to tissue-specific variability. Thus, we aggregate replicates from the different tissue types into the broader categories of tamoxifen-resistant and tamoxifen-responsive cell lines for our analysis.

Each cell line has at least two replicates and a corresponding input sample, yielding four tamoxifen-resistant replicates (2 TAM-R and 2 BT-474) and seven tamoxifen-responsive replicates (2 T-47D, 2 ZR75-1, and 3 MCF-7). Ross-Innes et al. processed the cell line material by cross-linking the proliferating cells using an anti-ER (SC-543) antibody and following the ChIP protocol described in Schmidt et al. [12, 37]. The resulting sequences were then processed with the Illumina analysis pipeline (version 1.6.1) and aligned to the hg19 Human Reference Genome (NCBI build 36.1, March 2008) using BWA (version 0.5.5) [12]. The single-end sequencing of the ChIP-seq libraries resulted in roughly reads per library. Additional information on how the experimental data was processed is available in Ross-Innes et al. [12] and in the GEO database under accession number GSE32222 (https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE32222). The Fastq files for the full data across all chromosomes are also available under the same accession number.

To simplify our analysis, we use the subset of the sequences that were aligned to chromosome 18 in the hg19 Human Reference Genome. The chromosome 18 observations were processed for the DiffBind vignette and are accessible through the DiffBind R package [20]. The peaks in each replicate were identified using MACS2 on the BAM files with the parameters: -g 2.7e9 --nomodel --extsize 137 --bw 300 --mfold 5 20 -p 1e-5 -f BAM. The corresponding input sample for each cell line was also supplied to MACS2 in order to increase peak-calling specificity [23]. After peak calling, the peaks were then subsetted to only those on chromosome 18. Six out of seven FRiP scores for the subsetted data exceeded the ENCODE standard of 0.1 (ranging from 0.10 to 0.31). However, one replicate from the TAM-R tissue had an FRiP score of 0.06, which is below the ENCODE standard. A blacklist based on the hg19 Human Reference Genome and greylists based on the input controls were next created via DiffBind to further cull the set of peaks considered in downstream analysis. After applying the blacklist and greylists, 1336 consensus peaks were identified in the tamoxifen-resistant cell line samples, and 2453 consensus peaks were identified in the tamoxifen-responsive cell line samples (at least two replicates within an experimental state had to identify the region as significantly enriched with DNA-protein binding for it to be considered a consensus peak within the experimental state). Taking the union of these peaksets yielded 2698 consensus peaks across the experimental states. We then applied the same between-sample normalization methods as we used in our simulation (except for MAnorm2, for reasons mentioned above) to normalize the read counts in each consensus peak. Finally, we determined which genomic regions were differentially bound using DiffBind (FDR < 0.05) [20].

Like we see in the CUT&RUN experimental analysis, the between-sample normalization methods that rely on the same technical conditions generate normalized log2 fold changes that are similar to one another. Indeed, the first principal component, which explains 91.23% of the variability in the normalized log2 fold changes, separates background binding normalization methods from the peak-based normalization methods (see Fig. 10). The peak-based normalization methods that rely on the technical condition of equal total DNA occupancy between states—Library Size (Reads in Peaks) and Loess Adjusted Fit (Reads in Peaks)—are distinguished from those that rely on the balanced differential DNA occupancy—TMM (Reads in Peaks) and RLE (Reads in Peaks)—by the second principal component, which accounts for 4.91% of the variability in the normalized log2 fold changes.

Figure 10.

PCA of the log2 fold changes after between-sample normalization has been performed between the tamoxifen-resistant and tamoxifen-responsive cell lines.

Between 245 and 440 peaks were called as differentially bound for each normalization method for this dataset, with 209 peaks being considered high-confidence since they were identified as differentially bound regardless of the selected between-sample normalization method. Between 48% and 85% of peaks found by each individual method were considered high-confidence differentially bound (see Fig. 11). This result has a broader range than the 38%–63% of peaks found with each individual method that were classified as high-confidence in the trypansome dataset (see Fig. 9). However, for both datasets, high-confidence peaks capture at least the top 38% of all peaks identified as differentially bound by the various between-sample methods in the two experimental datasets analyzed (with most methods having greater than 50% of their differentially bound peakset belonging to the high-confidence peakset).

Figure 11.

UpSet plot (for the 20 largest distinct groupings) comparing the peaks identified as differentially bound between the tamoxifen-resistant and tamoxifen-responsive cell lines after selecting different between-sample normalization methods. 209 peaks were identified as differentially bound by all of the seven between-sample normalization methods that we tested.

Discussion

Our results show that when performing differential binding analyses, the preceding between-sample normalization step is crucial to get right as the technical conditions underlying the selected between-sample normalization method often drive the accuracy of the downstream differential binding analysis. Indeed, our simulation results demonstrate that the performance of differential binding analysis declines when the technical conditions for the selected between-sample normalization method are violated, as measured by the average empirical FDR, power, and sample-wide size factors relative to the omniscient Oracle method. The experimental data externally validate our simulation results. In particular, normalization methods that rely on the same technical conditions cluster together in the PCA plot of the normalized log2 fold changes in both experimental analyses, suggesting that between-sample normalization methods that rely on the same technical conditions produce similar normalization results.

In some experiments, a researcher will have a sense of whether and how the technical conditions underlying a particular between-sample normalization method are met. For example, if one experimental state causes cell damage or death, the researcher might expect an increase in chromatin shearing, and with that, a potential increase in background binding within that specific experimental state. Alternatively, if one experimental state makes it difficult to obtain sufficient numbers of cells to begin the experiment, there could also be an increase in the amount of noise, or background binding, in that particular experimental state. There is also sometimes evidence of global changes in DNA occupancy, such as with a global decrease in H3K27me3 levels [38], or a global decrease in CTCF occupancy during the transition from pluripotency to early neuronal lineage commitment for neonatal mouse brains [39].

In cases where the researcher does not have a preconceived notion of any potential technical condition violations, a potential analysis might include implementing multiple between-sample normalization methods on the data, like Ashby et al. to determine a high-confidence differentially bound peakset that is less sensitive to one’s choice of between-sample normalization method [25]. If multiple normalization methods yield the same set of differentially bound peaks, there is strong evidence that the differential binding is the result of genuine differences in DNA occupancy between the experimental states, rather than the consequence of experimental or statistical artifacts. In the experimental data we analyzed, roughly half of the called peaks were called as differentially bound for every normalization method. In other words, the high-confidence peakset is made up of roughly half of the differentially bound peaks for each method.

Our analysis in this paper focuses on the differential binding analysis of transcription factors, which typically produce sharp, narrow peaks [26]. However, other popular proteins of interest in ChIP-seq experiments, such as histone marks, are characterized by broader, lower-intensity peaks [26], or other more complex occupancy behavior. In principle, the same downstream methods described in this paper for detecting differentially bound regions after peak calling could be used following the peak-calling step for such proteins. However, because our simulations did not model data with a broad and low genomic distribution, it is possible that some of the technical condition violations explored here might not apply in exactly the same way. An analysis of between-sample normalization methods and the impact of relevant technical conditions in cases with broader, lower intensity DNA occupancy could be an avenue for future work that builds upon our study.

Key Points

• Correct between-sample normalization of ChIP-seq data is important for accurate downstream differential binding analyses.

• Different between-sample normalization methods rely on different technical conditions that describe the underlying structure of the data and how it differs between experimental states.

• Using a high-confidence differentially bound peakset can make differential binding analysis results less sensitive to one’s choice of between-sample normalization method.

Author contributions

Sara Colando: Formal analysis, Investigation, Methodology, Software, Visualization, Writing—original draft, Writing—review & editing. Danae Schulz: Data curation, Investigation, Software Writing—original draft, Writing—review & editing. Johanna Hardin: Conceptualization, Formal analysis, Project administration, Writing—original draft, Writing—review & editing.

Competing interests

No competing interest is declared.

Funding

S.C. was supported in part by grants from the Pomona College SURP program and Kenneth Cooke Summer Research Fellowship. D.S. was supported in part by NSF CAREER grant 2041395. J.H. was supported in part by NIH GM112625.

Data availability

The data that support this paper are openly available. CUT & RUN fastq files are available in the SRA database under project number PRJNA795567. Alignment and peak data from the estrogen study are deposited in the GEO archive (accession number GSE32222).