Detecting differentially methylated regions with multiple distinct associations

Abstract

Aim:

We evaluated five methods for detecting differentially methylated regions (DMRs): DMRcate, comb-p, seqlm, GlobalP and dmrff.

Materials & methods:

We used a simulation study and real data analysis to evaluate performance. Additionally, we evaluated the use of an ancestry-matched reference cohort to estimate correlations between CpG sites in cord blood.

Results:

Several methods had inflated Type I error, which increased at more stringent significant levels. In power simulations with 1–2 causal CpG sites with the same direction of effect, dmrff was consistently among the most powerful methods.

Conclusion:

This study illustrates the need for more thorough simulation studies when evaluating novel methods. More work must be done to develop methods with well-controlled Type I error that do not require individual-level data.

DNA methylation (DNAm) is one of the most widely studied epigenetic marks, and advances in DNAm microarrays have enabled convenient and affordable measurements of DNAm at hundreds of thousands of CpG sites in large cohorts. The Illumina MethylationEPIC BeadChip (EPIC array) measures DNAm at 853,307 CpG sites in the human epigenome [1]. Its predecessor, the Illumina Infinium HumanMethylation450 BeadChip (450K array) measured DNAm at 482,421 CpG sites [2]. These data are often used to conduct an epigenome-wide association study (EWAS), testing for an association between DNAm and a phenotype of interest at each CpG site separately [3].

There is evidence that the standard EWAS approach is underpowered to detect the small effect sizes seen in epigenetic studies, particularly in studies of complex traits and smaller cohorts [4–6]. However, even CpG sites with small effect sizes can be replicated in independent cohorts and populations, suggesting that they are true associations which may be important to understanding disease etiology [7]. Furthermore, previous work has found that regional DNAm profiles can be more predictive in determining disease status than individual CpG sites [8]. Searching for differentially methylated regions (DMRs), clusters of neighboring CpG sites associated with a phenotype, has been suggested as a way to increase power by both reducing multiple testing burden and aggregating evidence of association across multiple CpG sites in a region.

There are many existing methods to detect DMRs [9–15]. Here, we perform a simulation study to evaluate the power (true positive rate) and Type I error (false positive rate) of single-site EWAS as well as five existing DMR detection methods: DMRcate [10], comb-p [12], seqlm [15], GlobalP [13] and dmrff [14]. Additionally, we explore the use of external data for estimating correlation between CpG sites, a common approach in genetic studies to estimate correlation between neighboring genetic markers when analyzing published summary statistics [16–19], which has been suggested for DNAm studies [14]. Finally, we perform a real data analysis, searching for DMRs in cord blood associated with birth weight using all methods with well-controlled Type I error.

Materials & methods

Study cohort

Our study cohort consisted of 446 infants from the Genetics of Glucose Regulation in Gestation and Growth (Gen3G) cohort, a prospective population-based cohort in Sherbrooke, Canada [20]. Expectant mothers of at least 18 years of age with singleton pregnancies and no history of diabetes, either known prepregnancy or detected at first prenatal visit, were recruited to participate in the study during the first trimester of pregnancy [20]. All study participants provided informed consent, and the study was approved by the Centre Hospitalier Universitaire de Sherbrooke ethics review board [20].

We restricted this analysis to infants with available data on the following variables: cord blood DNAm, sex, birth weight, maternal smoking and gestational age at delivery (Table 1). All infants were born from mothers who self-reported their ancestry as European descent (White). Infant ancestry was not collected at birth in this cohort.

Table 1. . Characteristics of 446 infants in the Genetics of Glucose Regulation in Gestation and Growth cohort with complete covariate information.

	Mean ± SD or n (%)
Male	233 (47.7%)
Birth weight (g)	3438.9 ± 423.7
Gestational age (weeks)	39.51 ± 1.04
Any smoking during pregnancy	40 (9.0%)
Estimated cell proportions
CD8+ T-cells	0.13 ± 0.04
CD4+ T-cells	0.17 ± 0.07
Monocytes	0.10 ± 0.03
Natural killer cells	0.02 ± 0.03
B-cells	0.10 ± 0.03
Nucleated red blood cells	0.12 ± 0.07

SD: Standard deviation.

Reference cohort

A second cohort, Project Viva [21], was used as a reference cohort to estimate correlation between CpG sites in cord blood. Project Viva is a prospective birth cohort in Eastern Massachusetts. Expectant mothers with singleton pregnancies were recruited during their first prenatal visit at participating Atrius Harvard Vanguard Medical Associates offices [21]. The study was approved by the institutional review boards of all participating institutions, and all participating mothers provided informed consent. Cord blood DNAm was measured in the children whose mothers additionally provided genetic consent [22].

Because it is important in genetic studies to match the ancestry of the reference cohort to the study cohort when computing correlation between genetic markers [23,24], we restricted to Project Viva infants reported as White by their mothers (n = 309).

DNAm normalization

Cord blood DNAm data from Gen3G, measured on the EPIC array, were normalized to correct for technical variation using FunNorm [25]. Regression on correlated probes was used to adjust for bias introduced by the two different probe designs on the EPIC array [26]. Data were adjusted for batch effects with ComBat [27], using plate as the batch variable. In order to account for cell type heterogeneity and avoid spurious results due to cell-type-specific methylation patterns, cord blood cell proportions were estimated from DNAm levels using the deconvolution method proposed by Houseman et al. and a cord blood reference panel [28,29].

Cord blood DNAm data from Project Viva, measured on the 450K array, were first normalized using normal-exponential out-of-band background subtraction with dye-bias normalization [30]. Beta-Mixture Quantile normalization was then performed to adjust for the two different probe designs on the 450K array [31]. Data were adjusted for batch effects with ComBat [27], using plate as the batch variable.

Methods evaluated

We evaluated five DMR detection methods in this study: DMRcate [10], comb-p [12], seqlm [15], GlobalP [13] and dmrff [14]. All of the methods we evaluated were designed for microarray data and able to accommodate continuous phenotypes. With the exception of seqlm, all of these methods combine EWAS summary statistics to create regional test statistics. All methods except GlobalP use the observed data to define DMR boundaries, while GlobalP tests a user-defined set of regions for associations with a trait.

Seqlm first divides the genome into segments based on distances between consecutive probes (default = 1000 bp). Segments are then further divided based on consistent methylation profiles of consecutive probes. Finally, a linear mixed model is used to test for association between methylation in each segment and a phenotype of interest, which can be binary or continuous. Seqlm does not allow for covariates in the model, which presents challenges for analysis of heterogeneous tissues, where it is important to adjust for cell composition [32].

DMRcate uses a Gaussian kernel smoothing to adjust squared EWAS t-statistics at each CpG site, based on the squared t-statistics of neighboring CpG sites. These smoothed statistics are then used to recalculate p-values for each CpG site, and significant CpG sites within a certain distance of one another are combined to form DMRs. DMRcate is computationally efficient, but does not account for correlation between neighboring sites and was found to have high false positive rates in regions with high correlation between sites [14].

Similarly to DMRcate, comb-p combines EWAS summary statistics from neighboring CpG sites. After estimating spatial auto-correlation at different distance lags, p-values for each CpG site are adjusted using the p-values of neighboring CpG sites. A peak-finding algorithm is then used to identify regions enriched for low p-values. One main benefit of comb-p is that the only required input is the chromosome, position and p-value of each CpG site, making it possible to search for DMRs in meta-analysis results or published summary statistics.

Dmrff employs an inverse-variance weighted meta-analysis of EWAS effect sizes, accounting for correlation between CpG sites. Candidate regions are defined by grouping all nominally significant (p < 0.05) CpG sites with the same direction of effect and within a given distance of one another (default = 500 bp) into regions. Meta-analysis test statistics are calculated for each region, which are also then divided into greedily selected sub-regions. The authors of dmrff also provide a framework for DMR meta-analysis, including functions for individual cohorts to compute both summary statistics and correlation between sites [14]. Although they evaluated the use of one cohort as a reference to estimate correlation between sites in their real data analysis, eliminating the need for individual data, they did not include this reference cohort approach in their Type I error simulations.

The GlobalP test statistic is z^TΣ⁻¹z, where z is a vector of EWAS z-scores for every CpG site in the region and Σ is the partial correlation between CpG sites, adjusting for all covariates in the model [13]. Multicollinearity in a region can lead to a singular Σ or unstable estimate of Σ⁻¹, which can result in previously observed inflated Type I error [13]. To address this, we introduced a pruning parameter to GlobalP, κ (the condition number of Σ), and iteratively pruned CpG sites until the condition number of each region was less than κ [33]. Higher values of κ indicate more severe collinearity in a region, with κ = 1 when all CpG sites are independent. Unlike other methods, GlobalP does not define region boundaries from the data but instead uses a predefined list of annotations and the CpG sites belonging to them. This allows researchers to test any set of CpG sites, not just continuous segments of the genome.

Phenotypes & covariates

The phenotype of interest in the real data analysis was birth weight (g), while the phenotypes were simulated in the Type I error and power simulations. The models in the real data analysis and simulations were adjusted for the same set of covariates.

For all methods except seqlm, the following covariates were included in the model: infant sex, gestational age in weeks, maternal smoking during pregnancy (yes/no) and estimated cord blood cell proportions (monocytes, natural killer cells, B cells, CD4⁺ T cells, CD8⁺ T cells and nucleated red blood cells) [28,29]. Because seqlm does not allow for covariate adjustment, we ran seqlm two different ways: once with the phenotype of interest, denoted seqlm (phenotype) and once with residuals calculated from a linear model predicting the phenotype of interest from all of the model covariates listed above, denoted seqlm (residuals). Figure 1 shows an overview of the study design, including all covariates and phenotypes of interest.

Figure 1. . Study design models.

DNA methylation ~1 + EWAS phenotype + covariates. Arrows leading to simulated and imputed variables indicate which observed variables were used in the simulation/imputation. Dimensions for all data types are included below the variable name.

EWAS: Epigenome-wide association study; T1E: Type 1 error.

Model parameters

For all methods that use EWAS summary statistics for each CpG site as input, EWAS summary statistics were computed using linear models with DNAm as the outcome, implemented with the lmFit and eBayes functions in the limma R package, adjusting for all covariates described above [34,35].

In our GlobalP models, we tested for associations between DNAm and the phenotype of interest in 99,015 regions, defined by the CpG islands (+/-4 kb to include shores and shelves) and gene features in the Illumina EPIC array annotation files (Table 2). For all other methods, we used the default parameter settings (Table 3) to define region boundaries, assuming that the average user will use the default parameters and deferring to the original publication for the most appropriate parameter setting for each method.

Table 2. . GlobalP annotations present in Genetics of Glucose Regulation in Gestation and Growth cohort by category.

	n (%)
Transcription start site to 1500 bp upstream	12,186 (12.3%)
5′ UTR	14,448 (14.6%)
Gene body	22,127 (22.3%)
3′ UTR	24,414 (24.7%)
CpG island (including shores and shelves)	25,840 (26.1%)
Total	99,015

Table 3. . Default parameters used to define regions.

Method	Parameter descriptions	Default values
DMRcate	lambda: maximum distance between two consecutive probes in a region (bp) C: scaling factor for Gaussian kernel smoothing, used to adjust epigenome-wdide association study summary statistics using information from neighboring sites	lambda = 1000 C = 2
comb-p	seed: CpG site p-value required to start a candidate region dist: maximum distance between two consecutive probes in a region (bp)	seed = 1e-3 dist = 200
seqlm	max_block_length: maximum number of CpG sites in a region max_dist: maximum distance between two consecutive probes in a region (bp)	max_block_length = 50 max_dist = 1000
dmrff	max_gap: maximum distance between two consecutive probes in a region (bp) p_cutoff: CpG site p-value required to start a candidate region	maxgap = 500 p.cutoff = 0.05

We chose four pruning thresholds to evaluate for GlobalP: κ<100, κ<20, κ<10 and κ<5. In a region with only two CpG sites, these thresholds are equivalent to a Pearson correlation pruning threshold of approximately 0.98, 0.91, 0.82 and 0.67, respectively.

We also evaluated two different approximations for the GlobalP partial correlation matrix, Σ: the marginal correlation between each pair of CpG sites in the study cohort (Gen3G), and the marginal correlation between each pair of CpG sites in the external reference cohort (Project Viva). Because of the difference in arrays between our study cohort (EPIC array) and reference cohort (450K), the tests where Σ was estimated using the reference cohort had fewer CpG sites per region.

Multiple testing correction

Multiple testing correction for the number of regional tests (or CpG site tests in the single-site EWAS) differed by method. We used a Bonferroni correction to account for the number of tests in the single-site EWAS, GlobalP, dmrff and seqlm, while comb-p uses a Sidak correction [36] to correct for the number of regional tests. As the default, DMRcate controls Type I error by limiting the number of DMRs reported to the number of CpG sites with EWAS false discovery rate (FDR) <0.05 (pcutoff = ‘fdr’). Therefore, if there are no individual CpG sites meeting the FDR <0.05 criteria, no DMRs are reported by DMRcate, and the R package includes a warning that changing this pcutoff threshold can lead to increased Type I error.

Type I error simulation

To evaluate Type I error under the null hypothesis of no association at any CpG sites, we performed a simulation study with 200 phenotypes simulated to be associated with sex and gestational age but not associated with DNAm (Figure 1). We only simulated the phenotypes; DNAm and covariate values were taken from observed data.

The effect sizes for gestational age and sex were based on relationships observed in the real dataset between birth weight, sex and gestational age. In the Gen3G cohort, gestational age explains 14.2% of the variability in birth weight after adjusting for sex, and sex explains 1.6% of the variability in birth weight after adjusting for gestational age. These values were used as the variance explained by sex and gestational age in the simulations.

For all methods, we compared the number of false positive DMRs identified at the 0.05 significance level after adjusting for multiple testing. Additionally, we computed the observed Type I error over the expected Type I error at a range of significance levels for EWAS, GlobalP and seqlm. Because comb-p, DMRcate and dmrff do not report results from all tests performed across the epigenome, we were unable to compute the genome-wide observed versus expected Type I error at different significance levels.

Power simulation

In order to evaluate power, we simulated phenotypes associated with gestational age, sex and DNAm in a randomly selected CpG island on chromosome 7 (chr7:55,072,266–55,072,622) for each infant in Gen3G (Figure 1). Including shores and shelves, this region spanned from 55,068,266 to 55,076,622 on chromosome 7. After quality control, this region contained 8 CpG sites measured in both Gen3G and Project Viva.

We performed simulations for 8 different sets of causal CpG sites in this region (Table 4). Each causal set had either one or two CpG sites, and each causal set explained 4% or 6% of the variation in the phenotype, which is consistent with previous work showing that individual CpG sites can explain from <1% up to 11% of the variation in a phenotype [37–40]. For causal sets with 2 sites, we chose pairs of CpG sites that were weakly (Pearson correlation = 0.10), moderately (Pearson correlation = 0.39) or strongly (Pearson correlation = 0.86) correlated with each other in the observed Gen3G DNAm data. All causal CpG sites were simulated to have the same direction of effect.

Table 4. . Causal CpG site(s) and variance explained by DNA methylation for each power simulation scenario.

Scenario	CpG 1	CpG 2	Correlation	Variance explained
1	cg07313064	-	-	4%
2	cg07313064	-	-	6%
3	cg07313064	cg00832581	0.10	4%
4	cg07313064	cg00832581	0.10	6%
5	cg07313064	cg09508476	0.39	4%
6	cg07313064	cg09508476	0.39	6%
7	cg07313064	cg13490352	0.86	4%
8	cg07313064	cg13490352	0.86	6%

In addition to the variance explained by DNA methylation at the causal sites, 16% of the variance in the simulated phenotypes was explained by sex and gestational age.

For each of the eight scenarios, we simulated 200 phenotypes associated with sex, gestational age and DNAm. As in the Type I error simulations, the variance explained by gestational age (14.2%) and sex (1.6%) were derived from the relationship between gestational age, sex and birth weight in the observed data. Additionally, 4–6% of the variance in the simulated phenotypes was explained by DNAm at 1–2 causal CpG sites in the simulated DMR. Table 4 shows the causal sites for each scenario and proportion of variance explained by each causal set.

Because methods with inflated Type I error will identify true positives more often just by chance, we limited the power simulations to methods with well-controlled Type I error. For DMR methods, we calculated the proportion of simulations in which a region overlapping with the simulated CpG island association was identified at the α = 0.05 significance level, after adjusting for multiple testing. For the single-site EWAS, we calculated how often the causal site(s) were identified as well as how often any of the 8 sites in the region were identified.

Real data analysis

For all methods with well-controlled Type I error, we conducted a real data analysis to identify DMRs associated with birth weight in Gen3G (n = 446). As with the simulations, the model was adjusted for infant sex, gestational age in weeks, any maternal smoking during pregnancy and estimated cord blood cell proportions (CD8⁺ T cells, CD4⁺ T cells, monocytes, natural killer cells, B cells and nucleated red blood cells) [28,29].

Results

Type I error simulation

The Type I error rate of GlobalP was well controlled when using the partial correlation from the study sample to estimate Σ, but inflated when using either approximation for Σ. Table 5 shows the mean ratio of observed to expected Type I error at six different significance levels, α, across all 200 epigenome-wide simulations, with ratios greater than 1 indicating inflated Type I error. When approximating Σ, as the significance level decreased, the observed Type I error rate became more inflated. While CpG site pruning improved the Type I error rate when using approximations for Σ, even the most stringent pruning criteria (κ <5) still resulted in inflated Type I error. Using a reference cohort to approximate Σ resulted in more severe inflation than using the marginal correlation in the study cohort to approximate Σ. When using the partial correlation in the study cohort to estimate Σ, the ratio of observed to expected Type I error rates ranged from 0.61 to 0.95 and was not sensitive to the choice of pruning threshold (Table 5).

Table 5. . Observed Type I error rate divided by expected Type I error rate across 200 epigenome-wide simulations, by method and significance level.

Method	α
	5 × 10-2	1 × 10-2	1 × 10-3	1 × 10-4	1 × 10-5	1 × 10-6
EWAS	0.97	0.94	0.88	0.81	0.75	0.71
seqlm (phenotype)	1.41	1.61	2.15	3.52	6.69	14.14
seqlm (residuals)	0.45	0.35	0.26	0.2	0.18	0.14
GlobalP, Σ = Gen3G partial
No pruning	0.94	0.88	0.79	0.74	0.61	0.76
κ <100	0.94	0.89	0.8	0.74	0.61	0.76
κ <20	0.94	0.89	0.8	0.74	0.62	0.76
κ <10	0.94	0.89	0.8	0.75	0.63	0.86
κ <5	0.95	0.9	0.82	0.79	0.66	0.81
GlobalP, Σ = Gen3G marginal
No pruning	2.43	5.41	23.69	130.87	812.86	5451.16
κ <100	2.34	4.79	16.95	70.43	318.92	1539.9
κ <20	1.84	2.93	6.75	18.61	57.78	200.78
κ <10	1.52	2.08	3.77	8.21	21.03	59.59
κ <5	1.22	1.42	1.93	3.02	5.74	11.92
GlobalP, Σ = reference marginal
No pruning	3.77	10.32	54.22	333.36	2265.88	16479.24
κ <100	3.72	9.98	49.75	282.2	1728.15	11095.4
κ <20	2.95	6.36	22.1	85.89	365.16	1659.67
κ <10	2.23	3.97	10.4	31.08	102.87	368.39
κ <5	1.57	2.24	4.16	8.85	21.19	56.2

Values >1 indicate inflated Type I error.

EWAS: Epigenome-wide association study; Gen3G: Genetics of Glucose Regulation in Gestation and Growth.

Running seqlm without adjusting for covariates also resulted in inflated Type I error, which was more extreme at more stringent significant levels (Table 5). At α = 0.05, we observed 1.41-times as many significant regions as expected by chance on average, while at α = 1 × 10^-6 we observed 14.14-times as many. As a proxy for including covariates in the model, which is not possible with seqlm, we modeled the simulated phenotypes as a function of all covariates and used the residuals from these models as input to seqlm. Using this approach, Type I error was well controlled, but the method appeared to be overly conservative, especially at more stringent significance levels. At α = 0.05, we observed 0.45-times as many significant regions as expected by chance on average, while at α = 1 × 10^-6 we observed 0.14-times as many.

Because we were not able to compute the ratio of observed to expected Type I error for all methods at different significance levels, we compared the number of false positive regions identified at the 0.05 significance level after adjusting for multiple testing for all methods. Table 6 shows a summary of the number of false positive results across all 200 simulations by method. If Type I error was well controlled, we would expect to detect false positives in about 5% of simulations. EWAS (individual CpG-by-CpG), DMRcate, dmrff, GlobalP (with Σ estimated from partial correlation in the study cohort) and seqlm (with covariate-adjusted residuals) detected false positives in no more than 5% of simulations. In contrast, comb-p identified at least one false positive DMR in 74% of simulations, and in one simulation identified 29 false positives. Running seqlm with the phenotype of interest and no covariate adjustment resulted in at least one false positive in 38.5% of simulations, with a maximum of 32 false positive DMRs in a single simulation. Both approximations for Σ with GlobalP resulted in an extreme number of false positive DMRs, with nearly 100% of simulations identifying at least one false positive, while the gold standard (Σ estimated from partial correlation) identified false positives in 4–5% of simulations as expected.

Table 6. . Distribution of the number of false positive results across 200 Type I error simulations at the 0.05 significance level, after adjusting for multiple testing, by method.

	Min	Q1	Median	Q3	Max	% Any FP
EWAS	0	0	0	0	1	2.5%
seqlm (phenotype)	0	0	0	1	32	38.5%
seqlm (residuals)	0	0	0	0	1	1.0%
GlobalP, Σ = Gen3G partial
No pruning	0	0	0	0	2	4.0%
κ <100	0	0	0	0	2	4.0%
κ <20	0	0	0	0	2	4.0%
κ <10	0	0	0	0	2	4.0%
κ <5	0	0	0	0	2	5.0%
Global P, Σ = Gen3G marginal
No pruning	310	412.75	461	532.5	885	100.0%
κ <100	53	92.75	111	138.25	351	100.0%
κ <20	3	9	13	17	62	100.0%
κ <10	0	2	4	5	19	97.0%
κ <5	0	0	0	1	6	49.0%
GlobalP, Σ = reference marginal
No pruning	887	1099	1253	1403.5	2691	100.0%
κ <100	452	640.75	788	944	2182	100.0%
κ <20	27	66.75	92	135	524	100.0%
κ <10	2	11	17	28	147	100.0%
κ <5	0	1	2	4	26	85.0%
DMRcate	0	0	0	0	0	0.0%
comb-p	0	0	2	3	29	74.0%
dmrff	0	0	0	0	2	4.5%

EWAS: Epigenome-wide association study; FP: False positive.

Based on these results, we determined that Type I error was well controlled by EWAS, DMRcate, dmrff, GlobalP (with Σ estimated from partial correlation in the study cohort) and seqlm (with covariate-adjusted residuals). We evaluated each of these methods in the power simulation and real data analysis.

Power simulation

In all but two scenarios, dmrff had the most power to detect the simulated DMR (Figure 2). In most scenarios, DMRcate and GlobalP with the most stringent pruning threshold (κ <5) had the lowest power.

Figure 2. . Power simulation results.

Estimated statistical power of single-site EWAS, dmrff, GlobalP (Σ = Gen3G partial correlation), seqlm (residuals) and DMRcate across eight scenarios with 200 simulations each.

EWAS: Epigenome-wide association study.

When only one causal CpG site was simulated (scenarios 1–2), dmrff and EWAS outperformed other methods and performed similarly to each other, with dmrff identifying the causal site in only one additional simulation in each scenario compared with EWAS. EWAS and dmrff also performed similarly when two strongly correlated CpG sites (r = 0.86) were simulated as the causal sites (scenarios 7–8).

When two moderately correlated CpG sites (r = 0.39) were chosen as the causal sites, dmrff performed best, followed by EWAS and then GlobalP.

When two weakly correlated CpG sites (r = 0.10) were simulated as the causal sites (scenarios 3–4), GlobalP had the highest power, but power was low for all methods. In scenario 3 (2% variance explained by each CpG site), GlobalP without any CpG site pruning had an estimated power of 2%, compared with 1.5% for dmrff. In scenario 4 (3% variance explained by each CpG site), GlobalP with no CpG site pruning had an estimated power of 13%, compared with 9.5% for dmrff.

Real data analysis

In the real data analysis (n = 446), we tested for CpG sites or DMRs associated with birth weight, adjusting for infant sex, gestational age, maternal smoking status and estimated cord blood cell proportions as covariates. We included only methods with well-controlled Type I error. There were no CpG sites where DNAm was associated with birth weight using the standard single-site EWAS approach, and there were no DMRs detected by DMRcate, dmrff or seqlm (residuals). Using GlobalP (Σ = Gen3G partial), one CpG island (chr6: 37,137,070–37,139,434) was consistently identified as a DMR except with the most stringent pruning threshold (k <5).

There were 22 CpG sites measured in the associated CpG island on chromosome 6, including shores and shelves (chr6: 37,137,070–37,139,434 +/-4000 bp). Of these 22 CpG sites, six were nominally significant (p < 0.05) in the single-site EWAS, but none were significant after multiple testing (minimum p = 0.0014). These six sites were weakly correlated, with a maximum pairwise partial correlation of 0.28, suggesting multiple independent, weak associations in the region. Figure 3 shows the EWAS p-values, GlobalP p-values (Σ = Gen3G partial, no pruning) and pairwise partial correlation in the region [41].

Figure 3. . Birth weight differentially methylated region.

EWAS p-values, GlobalP (Σ = Gen3G partial correlation, no CpG site pruning) p-values and partial correlation between CpG sites in Gen3G in the PIM1 locus. There were 5 annotations for the GlobalP tests in this locus: PIM1 TSS to 1500 bp upstream, PIM1 5′ UTR, PIM1 Body, PIM1 3′ UTR and the CpG island surrounding PIM1. The CpG island was found to be significantly (Bonferroni p < 0.05) associated with birth weight. No individual CpG sites were significantly associated with birth weight after correcting for multiple testing.

EWAS: Epigenome-wide association study.

Discussion

We evaluated multiple DMR detection methods with both a simulation study and real data analysis. In addition to comparing methods, we evaluated the impact of using a reference cohort to estimate correlation between CpG sites and pruning out highly correlated CpG sites on both Type I error and power.

Our Type I error simulations confirmed a previous report that comb-p has inflated Type I error when applied to EWAS summary statistics [14]. Additionally, we found that seqlm had inflated Type I error without covariate adjustment. Because seqlm does not directly allow for covariates in the model, we ran a linear model predicting the phenotype of interest from all covariates and used the residuals from this model as input to seqlm, which adequately controlled Type I error. We found that GlobalP only had well-controlled Type I error when estimating Σ using the partial pairwise correlation between CpG sites, adjusting for all covariates in the model. Using the marginal correlation in either the study cohort or an external reference cohort resulted in inflated Type I error.

The discrepancy between Type I error of GlobalP when estimating Σ from the study cohort marginal correlation compared with the reference cohort marginal correlation, particularly at more stringent significance levels (Table 5), suggests that the idea of using a reference panel to estimate correlations between neighboring sites is not as straightforward in epigenetic studies as it is genetic studies. In genetic studies, finding a genetic dataset of the same ancestry is sufficient, regardless of platform or tissue source, to accurately estimate correlation between genetic variants and perform follow-up analyses of published summary statistics [19,23,24,42]. Here, we show that a tissue- and ancestry-matched reference cohort is not in itself sufficient for accurately estimating correlation between sites in DNAm studies. However, because we only evaluated a single reference cohort, which differed from our study cohort in several ways (geographic location, array, normalization methods, ascertainment of ancestry), we are not able to determine which factors contributed to the more extreme Type I error when using a reference cohort to estimate Σ. Further investigation into whether it is possible to accurately estimate Σ from an external reference cohort is warranted. For instance, it may be more accurate to use a reference cohort with available covariate data and estimate Σ from the partial correlation in the reference cohort, which we were not able to evaluate in this study. Resources and methods for accurate estimation of correlation between CpG sites without individual data would be beneficial to researchers aiming to search for DMRs in meta-analysis results or published summary statistics. Furthermore, it is important to extend this investigation of reference cohorts to more than one population so that more diverse cohorts may implement these methods.

Of the methods with well-controlled Type I error, all of which used individual data, dmrff had the most consistently high power across different scenarios, but GlobalP performed better in power simulations with weak, independent associations and identified a known birth weight locus [43] in our real data analysis not found by any other method, including dmrff.

The birth weight DMR identified by GlobalP in the real data analysis is a CpG island on chromosome 6, including its shores and shelves (+/-4000 bp). This CpG island is located in the promoter region of the oncogene PIM1, which has previously been implicated in birth weight [43–47]. In a mouse knockout study, mice deficient in all three PIM kinase genes had significantly reduced body weight, both at birth and throughout life [45]. Another mouse knockout study of PIM1 specifically found that PIM1-deficient mice had significantly reduced body weight from birth to week 12 [46]. In humans, both epigenetic and genetic associations with birth weight have been reported in the PIM1 locus. A GWAS of nearly 300,000 women of European ancestry found a significant association between rs2395668, an intergenic variant 30 kb upstream of PIM1, and birth weight [47]. In the context of cord blood DNAm, two previous studies have identified associations between birth weight and cg25325512, located in the 3′ UTR of PIM1 and the south shelf of the CpG island identified by GlobalP in this study [43,44]. In the POSEIDON study (n = 311), cord blood DNAm at cg25325512 was found to mediate the effect of maternal smoking on birth weight [44]. More recently, a large meta-analysis by the Pregnancy and Childhood Epigenetics Consortium with 8825 cord blood samples from 24 cohorts (all using 450K array), including 162 Gen3G samples, identified a strong association between DNAm at cg25325512 and birth weight after adjusting for maternal smoking status [43]. Interestingly, although cg25325512 was nominally significant in our birth weight EWAS (p = 0.0015), the strongest association in our single-site EWAS was cg04373205 (p = 0.0014), which is located near the transcription start site of PIM1, is independent of cg25325512 in Gen3G (Pearson correlation = 0.01), and was not significant in the Pregnancy and Childhood Epigenetics meta-analysis (p = 0.78) [43]. This suggests that there are either multiple distinct, weak associations in this region or multiple CpG sites tagging a single CpG site not present on the array. This DMR was not identified by GlobalP when using the most stringent pruning threshold (κ <5), which was shown to have low power in our simulations. This DMR was also not detected by dmrff, despite dmrff having the best performance in most of our power simulations. Upon further investigation, we found that many of the 22 CpG sites in the region are negatively correlated with one another and have different directions of effect (Figure 3). Since dmrff requires that all CpG sites in a candidate region have the same direction of effect, it split the 6 nominally significant CpG sites with alternating directions of effect into six separate candidate regions containing 1 CpG site each instead of aggregating evidence of association across the entire region.

Importantly, this study illustrates the need for more thorough simulation studies in epigenomics research. Previous DMR simulation studies have been performed on a small number of samples and/or a small subset of CpG sites in the genome [10,14,48,49]. We performed simulations across the whole genome, included over 400 cord blood samples in our simulations, and included potential confounders in our models. Our 200 simulated phenotypes for evaluating Type I error resulted in over 158 million single-site tests in the EWAS (791,324 sites per simulation) and 19 million GlobalP regional tests (99,016 annotations per simulation), allowing the evaluation of GlobalP down to the α = 10^-6 significance level. We observed that for methods with inflated Type I error, inflation was more severe at lower significance levels. Because published associations are generally adjusted for multiple testing across the epigenome, it is important to understand how methods perform at more stringent significance levels.

Conclusion

This study demonstrates the benefit of existing DMR detecting methods. In power simulations, dmrff performed at least as well as the standard single-site EWAS in every scenario, even when there was only one causal CpG site in the region. In a real data analysis, GlobalP identified a DMR at a known birth weight locus (based on EWAS and GWAS of much larger sample sizes) not identified with traditional single-site EWAS or other regional approaches. These methods are especially useful for identifying loci with multiple weak, distinct signals, which many cohorts are underpowered to detect with EWAS due to sample size limitations.

Future perspective

All of the methods found to have well-controlled Type I error in this study require individual-level DNAm data and covariate information. This is useful for individual cohorts but presents challenges in detecting DMRs using summary statistics from meta-analysis results or publications. In these instances, a DMR calling method that does not require individual data and does not have an inflated Type I error rate is needed. More work must be done to rigorously evaluate DMR calling methods and develop new methods to meet the needs of researchers.

Summary points.

• There are great differences in the performance of existing approaches (DMRcate, comb-p, seqlm, GlobalP and dmrff) to detect differentially methylated regions.

• All of the methods found to have well-controlled Type I error in this study require individual-level DNA methylation (DNAm) data and inclusion of covariate information.

• Only two methods in this study did not require individual-level DNAm data or covariate information, comb-p and GlobalP using an external reference panel to approximate correlation between sites, and both had inflated Type I error.

• Researchers should adjust phenotypes for all covariates before running seqlm to avoid false positive associations.

• GlobalP was most powerful in scenarios where the two causal CpG sites were weakly correlated; dmrff was most powerful in all other scenarios.

• GlobalP identified a birth weight differentially methylated region in a modestly sized cohort (n = 446) at the PIM1 locus, which has been previously identified in a large cord blood DNAm meta-analysis.