ChromDL: a next-generation regulatory DNA classifier

Abstract

Motivation

Predicting the regulatory function of non-coding DNA using only the DNA sequence continues to be a major challenge in genomics. With the advent of improved optimization algorithms, faster GPU speeds, and more intricate machine-learning libraries, hybrid convolutional and recurrent neural network architectures can be constructed and applied to extract crucial information from non-coding DNA.

Results

Using a comparative analysis of the performance of thousands of Deep Learning architectures, we developed ChromDL, a neural network architecture combining bidirectional gated recurrent units, convolutional neural networks, and bidirectional long short-term memory units, which significantly improves upon a range of prediction metrics compared to its predecessors in transcription factor binding site, histone modification, and DNase-I hyper-sensitive site detection. Combined with a secondary model, it can be utilized for accurate classification of gene regulatory elements. The model can also detect weak transcription factor binding as compared to previously developed methods and has the potential to help delineate transcription factor binding motif specificities.

Availability and implementation

The ChromDL source code can be found at https://github.com/chrishil1/ChromDL.

1 Introduction

The availability of large genome sequencing datasets has led to the development of numerous computational algorithms for the prediction of chromatin features and gene regulatory elements (referred to as regulatory features here) in the human genome in the past two decades (International Human Genome Sequencing Consortium 2001, Kent et al. 2002, ENCODE Project Consortium 2012, Rosenbloom et al. 2013, Roadmap Epigenomics Consortium et al. 2015, Davis et al. 2018, Nassar et al. 2023). In particular, the advent of deep learning (DL) algorithms has been shown to be effective in the prediction of DNase-I hyper-sensitive sites (DHSs), histone modifications (HMs), and transcription factor binding sites (TFBSs) (LeCun et al. 2015, Angermueller et al. 2016, Min et al. 2017, Eraslan et al. 2019). The neural network architectures commonly take genomic sequences as input and apply a series of non-linear transformations before computing probability scores for a series of target labels, each corresponding to a specific regulatory feature. These prediction efforts have become increasingly feasible with the improvement of machine-learning libraries, increased GPU speeds, and sophisticated optimization algorithms, all of which yield convergence to a high-performing solution in a fraction of time compared to earlier generations of neural networks.

One of the first-generation regulatory feature prediction models was DeepSEA (Zhou and Troyanskaya 2015), which was constructed using three convolutional neural networks (CNNs) (Supplementary Fig. S1). DeepSEA paved the way for next-generation neural networks in attempting to improve prediction accuracy of transcription factor (TF) binding, open chromatin, and HMs (919 regulatory features in total) and allowed the prioritization of expression quantitative trait loci and disease associated variants (Zhou and Troyanskaya 2015).

Subsequent models investigated the effectiveness of recurrent neural networks (RNNs) in addition to CNNs (LeCun et al. 2015, Angermueller et al. 2016, Min et al. 2017, Eraslan et al. 2019). These RNN layers share parameters between their units as CNNs do, but also contain internal memory components that allow them to “remember” previous passes over the input. RNNs are generally considered to be more powerful when dealing with sequential data processing, whereas CNNs are typically used to capture recurring patterns and motifs within the whole input data. Since the sequences fed into these models are represented as one-hot encoded matrices, two RNN classes can be combined in a larger architecture to improve performance when processing the inputs: long short-term memory (LSTM) and gated recurrent units (GRUs) that act in a similar fashion and often improve performance as compared to traditional RNNs (Hochreiter and Schmidhuber 1997, Gers et al. 2000, Cho et al. 2014, Eraslan et al. 2019).

While both RNNs and LSTMs contain feedback loops that help them “remember” the previous iteration, LSTMs contain an additional memory unit, which gives them more control about when information is remembered, when it is utilized, and when it is forgotten, making them more flexible than a traditional RNN (Hochreiter and Schmidhuber 1997, Gers et al. 2000). GRUs conversely contain an update and a reset gate that control what information is retained but lack an output gate, instead feeding the input and the reset gate back into the update gate. Both layers attempt to combat the vanishing gradient problem, a well-documented issue that prevents weights from changing when gradients become increasingly smaller with each epoch of training in traditional RNNs (Bengio et al. 1994).

Another feature of these RNN classes is their ability to be bidirectional, where two identical hidden layers read the input in a forward and reverse direction simultaneously to give the model context of the past, present, and future input (Hochreiter and Schmidhuber 1997, Gers et al. 2000, Graves et al. 2013). Two models introduced by Quang and Xie (2016), denoted as DanQ and DanQ-JASPAR, found success with these bidirectional RNNs as well as initializing their model with motifs from the JASPAR database (Mathelier et al. 2016). The DanQ neural network was constructed with one convolutional layer that mirrored DeepSEA’s input processing, followed by a bidirectional long short-term memory unit (BiLSTM) layer, before using DeepSEA’s exact dense and output layer architecture, subsequently improving performance metrics (Supplementary Fig. S2). DanQ-JASPAR increased the number of filters in the convolutional layer and number of units in the BiLSTM, and together with the initialized motifs observed improved performance. In this way, the authors proved that BiLSTMs could be used in regulatory feature prediction contexts using only DNA sequences as input, and opened up the possibility of combining CNNs and RNNs in further intricate architectures to improve performance metrics.

Here, we propose ChromDL, a neural network architecture that improves on previously developed models using a combination of CNNs and RNNs. ChromDL combines bidirectional gated recurrent units (BiGRU) layers, BiLSTM layers, and CNN layers in an architecture that differs quite substantially from both DeepSEA and DanQ. ChromDL surpasses these methods in accuracy of regulatory feature prediction. The model also detects a significantly higher proportion of weak TFBS ChIP-seq peaks and demonstrates the potential to more accurately predict TF-binding affinities. We also show that the TREDNet DL second-phase enhancer classifier (Hudaiberdiev et al. 2022) built on top of ChromDL is capable of detecting regulatory variants according to validation using reporter assay quantitative trait loci (raQTLs). We believe ChromDL will allow accurate predictions of gene regulatory mechanisms, modeling gene regulatory networks, and detection of disease-causative non-coding variants with utmost precision.

2 Materials and methods

2.1 Model architecture and training

ChromDL is a DL neural network containing two internal CNNs and three bidirectional RNNs. The neural network in its entirety extracts sequence features using weight matrices in sequential layers to minimize errors in prediction of the outputs, in this case being the 919 regulatory feature labels.

This model contains 11 layers, with 10 of these being distinct (Fig. 1). The 1 kb input sequence, encoded in one-hot representation (1000×4), was fed into a 128-unit BiGRU layer, which was then fed into a separable convolutional layer with 750 filters and a kernel size of 16, followed by a standard convolutional layer with 360 filters and a kernel size of 8. Both CNN layers had L1/L2 regularization of 1E-08 and 5E-07, respectively, and rectified linear unit (ReLU) activation before being maximally pooled with a 1 × 4 window. Next, the data were fed into a 128-unit BiLSTM layer with 20% dropout and applied batch normalization, before an average pooling layer with a 1 × 8 window. Finally, a second 128-unit BiLSTM layer was applied before the data was flattened and fed into the 919 length dense layer with sigmoid activation that produced the scores for all of the labels included in the dataset. In total, the model contained 10 414 957 parameters, with 512 non-trainable parameters. The exact technical specifications of these layers can be found in the online supplementary material (see “ChromDL Layer Specifications”).

Figure 1.

Schematic representation of ChromDL’s deep learning architecture.

This model utilized the same dataset and target feature labels described in DeepSEA in its training, testing, and validation (Zhou and Troyanskaya 2015) (Supplementary Table S10). Briefly, the human GRCh37 reference genome was split into 200-bp bins, and each TF-bound bin was labeled with 919 regulatory features. These labels include 690 TFBS, 125 DHS, and 104 HM regions. Each 200-bp bin included in the dataset was extended to 1 kb to include the flanking regions before being fed to the models in one-hot representation. Four thousand regions from Chromosome 7 were used as the validation set, the entirety of Chromosomes 8 and 9 as the test set, and the rest of the autosomes as the training set. All available training/validation regions were included in training this model, and all available testing regions were utilized for analyzing performance. The models were evaluated using area under the receiver operating characteristic curve (auROC) and area under the precision recall curve (auPRC) metrics. The average of the model outputs for forward and reverse strand sequences of the input regions were used as probabilities and compared to the true labels.

To train this model, we minimized the binary cross-entropy loss function between the true labels and the predicted labels after the sequences were processed by the model. This loss function can be expressed as: where N is the output size (the 919 predicted labels), ${\widehat{y}}_i$ is the ith scalar value in the model’s output, and $y_i$ is the corresponding target value.

$$\lambda =\frac{-1}{N}\sum _{i=1}^N{y}_i\hspace{0.17em}\text{log}({\widehat{y}}_i)+(1-y_i)\hspace{0.17em}\text{log}(1-{\widehat{y}}_i),$$

The Adaptive Moment Estimation (Adam) optimizer was ultimately chosen in the training of the model due to its properties of both the Adagrad and RMSprop optimizers, and it is relatively rapid convergence to solutions (Kingma et al. 2014, Reddi et al. 2019) (Supplementary Table S1). The optimizer utilizes a stochastic gradient descent method that is based on the adaptive estimation of first and second order moments. We trained our models using the Adam optimizer with a learning rate of 1E-03 and the parameters ${\beta }_1$ = 0.9, ${\beta }_2$ = 0.999, $ϵ$ = 1E-08 (Chollet et al. 2015, Abadi et al. 2016). A minibatch of size 500 was utilized in each training step.

The models were trained for 100 epochs or until stagnation of validation loss values. ChromDL models stagnated within the first 20 epochs. Each epoch took ∼16 h on a single core NVIDIA Tesla K80 GPU processor with 24 GB RAM. All our models were implemented in Tensorflow v2.4.0 with built-in Keras libraries (Chollet et al. 2015).

2.2 TF ChIP-seq signal intensity analysis

For this analysis, we collected signal values from the 690 TF ChIP-seq peak datasets to compare correlations between these values and ChIP-seq predictions for four models: ChromDL, DeepSEA, DanQ, and DanQ-JASPAR. We limited our investigation to Chromosomes 8 and 9, as the model had been trained and validated using the TFBS ChIP-seq peak data from all other autosomes. Ten equal percentile bins were created from the signal values of the individual peaks for each cell line. For every peak in the dataset, the center of the peak was taken and flanked with 500 bp windows on either side to obtain a 1 kb sequence, which was then fed through the respective model. The prediction value for each peak was isolated from the 919-output label vector and paired with the signal values from the published data. The 3%, 5%, and 10% false positive rate (FPR) thresholds were calculated for each model using the original DeepSEA testing dataset and used to determine whether a given peak was correctly predicted for the corresponding TFBS label. These proportions predicted correctly were pooled for all 690 cell lines for each of the 10 bins and plotted for each of the 4 models. To measure the significance of the pooled data, three separate one-sided independent t-tests were conducted in each percentile bin using the SciPy stats library to calculate the P-value of ChromDL having a higher average proportion than each of the three other models of interest (Virtanen et al. 2020). P-values $<$1E-05 were denoted as significant.

2.3 Motif/co-factor performance

We used the de novo motif discovery mode of MEME-ChIP (Bailey and Elkan 1994, Bailey 2009, Heinz et al. 2010) for detecting motifs in the TF ChIP-seq peak files. The motif detection was run on the ChIP-seq peaks, which were predicted as positive by all four models (ChromDL, DeepSEA, DanQ, and DanQ-JASPAR) at their respective 3% FPR threshold cutoffs. For each run, up to 20 of the most enriched motifs were detected. As a background, the merged peaks of all the available TF ChIP-seq assays in HepG2, K562, and H1-hESC cells obtained from the ENCODE Project (ENCODE Project Consortium 2012, Davis et al. 2018) were used, with an assumption that these are empirically observed TF-binding regions. For each trial, any overlaps in the control file with the target regions were removed from the background. We utilized experimentally characterized PWMs from the UniPROBE PBM database (Hume et al. 2015) and SELEX assays of TFs from the Catalog of Inferred Sequence Binding Preferences Database (Weirauch et al. 2014) for the comparison with the ChIP-seq extracted PWMS. The MEME-ChIP PWMs were then compared to these published PWMs independently for the ChromDL, DeepSEA, DanQ, and DanQ-JASPAR peak files. Following this, on each of the top 20 MEME-ChIP PWMs, the TOMTOM Motif Comparison Tool (Gupta et al. 2007) was run to assess alignment with the corresponding published TF, and the similarity between the top 20 PWMs and the matching TOMTOM PWM was computed [−log(p); Supplementary Table S11]. MEME Version 5.1.0 was used in these investigations.

2.4 Enhancer scoring validation

For our enhancer validation, we utilized the CNN architecture from the TREDNet DL second-phase enhancer classifier and attached it to our regulatory feature model (Hudaiberdiev et al. 2022). This classifier took the 919 output labels from our feature prediction model and produced a single output enhancer prediction score. This model’s architecture contained three CNN layers of increasing filter size with ReLU activation, a 180-unit dense layer, a batch normalization layer, two max pooling and dropout layers, and a LeakyReLU activation function before the single neuron dense output layer (Supplementary Fig. S3).

The mutagenesis for the enhancer validation was conducted using the ChIP-seq peaks of EP300 (optimal IDR peaks of the ENCODE ChIP-seq were used, 5526 peaks in total) and inputting 4000 regions for each individual enhancer. Each region was in silico mutated by looking at 1 bp of the 1 kb in the sequence and generating the four alternate alleles that could be recognized by the model (A, C, G, T, and N), with the rest of the sequence remaining unchanged. The one-hot encoded representations of these 4000 sequences were run through ChromDL and this secondary enhancer model, and the scores were then compared with the wild-type enhancer score.

We then established eight enhancer definitions for the training of this model: (i) ChromHMM Strong Enhancer States, (ii) ChromHMM Weak Enhancer States, (iii) H3K27ac peaks, (iv) H3K4me1 peaks, (v) Overlapping regions between H3K27ac and H3K4me1 peaks, (vi) Overlapping regions between H3K27ac and H3K4me1 with any H3K4me3 peaks removed, (vii) EP300 ENCODE Project IDR conservative ChIP-seq peaks, and (viii) EP300 ENCODE Project IDR optimal ChIP-seq peaks. For each definition, the following steps were taken to create positive and negative samples for the model training. For positive labels, each individual cell line DHS regions were taken and intersected with its respective definition peak file using the BEDTools command line tool (Quinlan and Hall 2010). Any regions overlapping with promoter regions spanning 1.5 kb upstream and 0.5 kb downstream from each alternative transcription start site were removed (as defined in knownGene table of UCSC Genome browser, Supplementary Table S18). The remaining regions were centered on the peak of the intersecting DHS sequence and extended to 1 kb, and assigned a one in the target label vector. DHS regions from all cell lines established the control set, and any overlaps with any of the defined enhancer regions were removed. Chromosomes 8 and 9 were used exclusively for testing and removed from training and validation. Chromosome 7 was used for validation, and all other autosomes were used in the training dataset. ChromHMM Strong Enhancer States and ChromHMM Weak Enhancer States were extracted from the NIH Roadmap Epigenomics project (Roadmap Epigenomics Consortium et al. 2015) (Supplementary Table S12). The DHS, H3K27ac, H3K4me1, and H3K4me3 peak regions were obtained from the UCSC ENCODE database (International Human Genome Sequencing Consortium 2001, Kent et al. 2002, ENCODE Project Consortium 2012, Rosenbloom et al. 2013, Davis et al. 2018, Nassar et al. 2023) (Supplementary Tables S13 and S15–S17). The EP300 IDR optimal and conservative ChIP-seq peak call sets came from the ENCODE portal (Supplementary Table S14) (https://www.encodeproject.org/) with the following identifiers: ENCFF087TMV, ENCFF843SYD, ENCFF583XDA, ENCFF784YVX, ENCFF054JWD, ENCFF259JWD, ENCFF821GNB, ENCFF549TYR, ENCFF476RII, ENCFF222XPQ, and ENCFF227EUK.

2.5 Experimental raQTL validation

raQTL experimental data were pulled from the Open Science Framework raQTL list (Supplementary Table S19), where 14 183 single nucleotide polymorphisms (SNPs) from the HepG2 cell line and 19 237 SNPs from the K562 cell line were identified as significantly impacting the activity of regulatory regions in MPRA assays. The eight enhancer definitions and corresponding control sets were created with the same previously outlined methodology used in “Enhancer Scoring Validation” for these two cell lines.

We performed in silico mutagenesis on the position of raQTL (for both reference and alternative alleles), by calculating the change of enhancer score on a 1 kb DNA segment centered at the raQTL. These two sequences were scored by the enhancer model, and the expression: was taken to yield the delta score for each individual raQTL in the two datasets. The entire dataset of raQTLs was then separated into five equal bins based on the percentile of delta scores for each cell line (20% each). Next, the raQTL density in enhancer regions was calculated by taking the number of raQTLs in each percentile bin that overlapped with the defined enhancers divided by the total number of enhancers. For the control group, 100 random samples were taken where 20% of the raQTLs and a random selection of DHS regions equal to the number of enhancer regions and of equal length were taken, and this same calculation was performed. This one-fifth of raQTLs was meant to standardize for the raQTL binning distribution.

$${\hspace{0.17em}\text{log}}_2\left(\frac{\text{ref score}}{\text{alt score}}\right)$$

These calculations were done for all delta scores and positive delta scores (SNPs that decreased enhancer score) for each of the eight enhancer definitions. This control measurement was used to calculate the fold change for each of the experimental group bins, defined as the density of raQTL SNPs in the defined enhancer regions for that given bin divided by the density of raQTL SNPs in the control set. An extensive collection of these plots can be found in Supplementary Figs S4–S6.

3 Results

3.1 ChromDL architecture discovery

Identification of this model architecture involved investigating numerous different combinations of convolutional and RNN layers, as well as pooling, normalization, and regularization alterations using random sampling and semi-supervised architecture design. The first ∼1100 models generated were constructed similar to DeepSEA and traditional CNN architectures like AlexNet (Krizhevsky et al. 2017) with variable convolutional filter sizes, layers of max pooling, and batch normalization. We found that these models trained with the Adam optimizer yielded prediction auROCs of about 0.918 on average, so we explored alternative methods beyond that of the original DeepSEA architecture. We investigated if improvements could be achieved by implementing hybrid neural networks, incorporating GRUs, LSTMs, and simple RNN layers into our CNN model (about 6000 models). With this implementation, we saw an improvement in classification metrics yielding best case auROCs of ∼0.92–0.925. We next moved to including several of these layers either of the same or different type in a single model. We found that the inclusion of multiple of these layers in the final iterations of the models as well as introducing bidirectionality within the RNN layers (about 705 models) generated the best output metrics (average auROC scores in the high 0.93 range when run for 20 epochs), with ChromDL emerging as the clear best model in terms of both median auROC (0.961) and auPRC (0.402).

ChromDL exhibited an accuracy of 0.970, 0.936, and 0.864 in median auROC for ChIP-seq TFBS, DHS, and HM, respectively (Fig. 2). We also observed that ChromDL outperforms DeepSEA in 95% of the 919 labels, DanQ—in 88%, and DanQ-JASPAR—in 85% (auROC; Table 1, Supplementary Table S2, and Supplementary Figs S7–S9) and outperforms these DL models across all categories with the single exception of DanQ-JASPAR in HM labels (Tables 2 and 3). A 100 000 run bootstrapping comparison of auROC and auPRC values across the four models showed that the improvement in accuracy achieved by ChromDL is statistically significant (auROC P-value $<$1E-3 and auPRC P-value $<$1E-2, respectively, Supplementary Fig. S10 and Supplementary Tables S3 and S4). We attribute the superior accuracy of ChromDL predictions to the presence of the BiGRU input layer and two BiLSTM layers, as the other network architecture elements are similar between ChromDL and the previously published models.

Figure 2.

ChromDL ROC curves for TFBS, DHSs, and HMs for each available chromatin feature file across the three categories. The red line in each plot represents the average of the ROC curves across every dataset available in the trials, with each gray line representing one dataset.

Table 1.

The number and proportion of labels for each of the four categories of labels (All, TFBS, DHS, and HM) that ChromDL has a higher auROC in a label-by-label comparison with each model individually.

	DeepSEA	DanQ	DanQ-JASPAR
All (919)	875 (0.9521)	812 (0.8835)	790 (0.8596)
TFBS (690)	677 (0.9812)	640 (0.9275)	636 (0.9217)
DHS (125)	125 (1.0)	122 (0.976)	120 (0.96)
HM (104)	73 (0.7019)	50 (0.4808)	34 (0.3269)

Table 2.

Median auROC for the four studied models for All 919 DeepSEA dataset labels, 690 TFBS labels, 125 DHS labels, and 104 HM labels.

	ChromDL	DeepSEA	DanQ	DanQ-JASPAR
All (919)	0.961	0.944	0.951	0.953
TFBS (690)	0.97	0.958	0.963	0.965
DHS (125)	0.936	0.924	0.928	0.932
HM (104)	0.864	0.856	0.864	0.868

Table 3.

Median auPRC for the four studied models for All 919 DeepSEA dataset labels, 690 TFBS labels, 125 DHS labels, and 104 HM labels.

	ChromDL	DeepSEA	DanQ	DanQ-JASPAR
All (919)	0.402	0.344	0.372	0.383
TFBS (690)	0.372	0.303	0.334	0.343
DHS (125)	0.498	0.448	0.465	0.477
HM (104)	0.359	0.338	0.356	0.359

To identify critical features of the ChromDL architecture, we first performed an iterative single removal of the nine internal layers and trained the models until stagnation of validation loss, analogous to ChromDL. It is important to note that while these individual layers are crucial for the model to achieve the performance metrics it does, the entire model and each layer’s variables work together to yield its final predictions. With this in mind, our analysis showed that the removal of any internal layer led to a decrease in scoring metrics, with the BiGRU input layer having the most pronounced impact on performance (Supplementary Table S5). Next, we created 10 subset architectures of ChromDL with pairs of the recurrent and convolutional layers removed (Supplementary Table S6). We then trained the corresponding models until stagnation in five independent trials, randomly taking 20% of the validation and testing sets, and 30% of the training set. Similarly, our analysis showed that the removal of any layer pair led to a decrease in overall performance (Supplementary Fig. S11), with the most significant decrease in metric again being observed with the absence of the BiGRU input layer, highlighting the potential importance of using these units for modeling regulatory sequence information.

3.2 Accurate prediction of TF binding

Next, we investigated the correlation between the prediction accuracy of TF-bound regions and TF ChIP-seq signal intensity. The TF ChIP-seq signal intensity reflects the propensity of TF binding, so we were interested in addressing how our model would detect the regions that had a lower binding affinity as compared to the previously developed models. We observed that ChromDL predicts a statistically significantly higher proportion of TFBS correctly in the lowest ChIP-seq signal strength bin compared to DeepSEA, DanQ, and DanQ-JASPAR (P-value $<$1E-05; one-tailed t-test). This statistically significant improvement remains present for all incremental bins of signal strength (ranging from 0%–10% to 90%–100%) and across 3%, 5%, and 10% FPR thresholds except for the highest two scoring bins in the 10% FPR calculations, suggesting that ChromDL scores can yield an elevated correlation with the presence of TFBSs at low ChIP-seq signal levels (Fig. 3, Supplementary Figs S12 and S13, and Supplementary Tables S7–S9).

Figure 3.

The proportion of TFBS ChIP-seq peaks in Chromosomes 8 and 9 predicted correctly based on the ChIP-seq signal strength. The 3% FPR thresholds were calculated from the DeepSEA dataset and used to generate the plots for each of the 690 available ChIP-seq peak files.

Some of the weak signal TF ChIP-seq peaks represent sequencing errors and transient TF binding. As a result, their detection can often be difficult using predictive models due to noise associated with the corresponding signal levels as compared to the strong signal TF ChIP-seq peaks. It is therefore noteworthy that our model could detect and predict these weak signal TFBS regions with higher accuracy as compared to previously developed models. This elevated sensitivity could allow the detection of sequencing and mapping errors in existing datasets, as well as open the possibility of detecting transiently bound regions with improved accuracy.

To assess ChromDL’s ability to predict TF-binding motifs, we extracted 9 PBM and 48 SELEX motif PWMs that correspond to a total of 306 TF ChIP-seq assays in our feature set (Weirauch et al. 2014, Hume et al. 2015). These ChIP-seq datasets contain on average 2273 peaks per file, and the average percentage of these peaks predicted correctly was 88% for ChromDL, 80% for DeepSEA, 81% for DanQ, and 83% for DanQ-JASPAR. We next applied MEME-ChIP to perform de novo motif discovery on the peaks independently predicted as positive by each of the four models (with FRP 5% threshold). We then utilized TOMTOM to select the motif closest to the PBM/SELEX motif for each of our 306 ChIP-seq files out of the top 20 most enriched motifs detected by MEME-ChIP, and performed a pairwise comparison between the published motifs and the assigned motifs. Our analysis found that the motifs detected from ChromDL predicted peaks showed the highest similarity score to the PBM/SELEX motifs, albeit by a small margin (Supplementary Fig. S14; P-value =0.03, Wilcoxon rank-sum test). Overall, we found a number of examples of agreement between the PWMs predicted by the four models and those derived from PBM/SELEX experiments (Fig. 4C). We also discovered in our investigation a number of notable cases where the ChromDL PWM showed much higher similarity to the PBM/SELEX PWM than the PWMs predicted by the other three models. For instance, although the PBM motif of c-Jun in HepG2 cells was detected in the positive peaks reasonably well by all four models, only ChromDL accurately captured the G/C dinucleotide ambiguity between the TGA and TCA parts of the motif (Fig. 4A). Similarly, the SELEX motif of HSF1 was correctly identified only by ChromDL, with the other three models missing significant components of the HSF1 binding specificity (Fig. 4B).

Figure 4.

Motif discovery using ChromDL, DeepSEA, DanQ, and DanQ-JASPAR MEME PWM mappings (bottom four rows) compared to PBM and SELEX PWM discovery results (top row). PBM c-Jun in HepG2 cells (A), SELEX HSF1 in HepG2 cells (B), and SELEX ELF1 in A561 cells (C). Boxed in the PBM Jun and SELEX HSF1 columns are base pairs of ChromDL/DeepSEA of interest in alignment with the PBM/SELEX motif.

It is important to note here that these examples of ChromDL outperforming the other comparison models in primary motif detection do not occur within every TF peak file, we investigated. There existed a fair number of occurrences in our dataset where all four models were in agreement with the published motif, but we found it encouraging that these examples exist in which ChromDL has the ability to outperform these models.

3.3 ChromDL-based enhancer predictor

Next, we used the TREDNet DL second-phase classifier CNN architecture with the output of the ChromDL model serving as input to compute an enhancer score of an arbitrary input DNA region. The two-stage model yielded respectable performance, with an average auROC of 0.89 across eight enhancer definitions (Supplementary Fig. S15 and Supplementary Table S20). We took a particular interest in the HepG2 and K562 cell lines for subsequent analysis and obtained high auROC (average 0.967 and 0.949) and auPRC metrics (average 0.313 and 0.352) in the EP300 enhancer trials as compared to other definitions (Supplementary Figs S16 and S17).

When we examined the trends across all available cell lines and the eight enhancer definitions, we noticed several key differences. We found that the lowest scoring auPRC enhancer group is the regions centered on overlapping H3K27ac and H3K4me1 peaks without H3K4me3 peak overlaps, with an average auPRC of 0.28. Closely behind it is the ChromHMM Strong Enhancer State definition with an average auPRC of 0.33. We attribute this poor relationship between the precision and recall to the enhancer definitions, as these two groups in particular have sequence features as compared to the other groups which lead them to be previously termed “stronger” enhancers. On the other hand, the “weaker” enhancer groups, which have been demonstrated to lead to elevated enhancer activity (Kwasnieski et al. 2014), resulted in more accurate ChromDL classification according to the auPRC metric. It has also been experimentally demonstrated that weaker regions with fewer H3K27ac peaks and more H3K4me1 peaks lead to an increased enhancer activity, and we would thus expect it to lead to a better auPRC metric. Indeed, we find this to be true in our overlapping H3K27ac and H3K4me1 peak enhancers with an average auPRC of 0.40, and our H3K4me1 peak enhancers with an average auPRC of 0.55. These results are encouraging, since the high auROC ($>$0.84 average across all definitions) suggests that ChromDL is able to provide enough contextual information to this enhancer model across multiple cell lines.

3.4 Reporter assay QTL validation of ChromDL regulatory variant predictions

After including the TREDNet second-phase enhancer classifier for the prediction of enhancer sequences using the output context from ChromDL as input, we decided to address if the enhancer scores are sensitive to point mutations in the input enhancer regions. We found that indeed, this two-stage model can be directly used to quantify the impact of DNA sequence mutations on enhancer activity by computing the difference in enhancer scores corresponding to two different alleles of the mutated nucleotide.

We performed an in silico mutagenesis of every nucleotide of HepG2 EP300 enhancers (optimal IDR) using the ChromDL enhancer model and observed that 52.8% of enhancer mutations show a detrimental impact on enhancer activity with 2.2% of enhancer mutations being capable of enhancer deactivation (defined as at least a 2-fold enhancer score decrease), which is in line with previously reported results of deleterious enhancer mutations (Li and Ovcharenko 2015, Li et al. 2017, 2022, Alvarez et al. 2018). Additionally, 47.2% of enhancer mutations are strengthening enhancer activity and 0.65% of mutations increase the ChromDL enhancer score over 2-fold. These results extend the scope of the previously reported enhancer gain mutations for a single ZRS enhancer to the whole-genome scale and identify a small set of key mutations with the most pronounced impact on enhancer activity (Kvon et al. 2020).

To validate the accuracy of ChromDL regulatory variant predictions, we used the results of experimental characterization regulatory effects at SNP positions in the human genome using the raQTL technique (van Arensbergen et al. 2019), which mapped 14 183 and 19 237 regulatory SNP mutations in HepG2 and K562 cell lines, respectively.

For each raQTL, two sequences were created and fed into the pre-trained enhancer model attached to ChromDL, a 1 kb sequence centered at the wild-type allele and a 1 kb sequence centered at the same allele with the mutant base pair in its place. Those raQTLs with the greatest delta scores (corresponding to the greatest predicted impact on enhancer activity) were found to be the most enriched in enhancer regions (Fig. 5). The enrichments were significant between the DHS control group and each of the percentile bins (P-value $<$0.001, two-sided binomial test). These results indicate that the ChromDL enhancer classifier can be applied for the accurate prediction of regulatory mutations and the degree of enhancer modulation by these mutations. Our model has been experimentally validated through (i) the trend of increasing magnitude raQTL score correlating with higher density in enhancer sequences and (ii) the proportion of enhancers that were deactivated (2-fold score decrease) in the mutagenesis trials aligning with previously published results.

Figure 5.

raQTL enrichment binned by DL delta score. All enrichments are statistically significant according to the two-sided Binomial Test. ***P-value $<$0.001.

4 Discussion

By testing thousands of various DL architectures, we have built an accurate DL predictor of regulatory activity in the human genome named ChromDL. ChromDL improves the accuracy of prediction of regulatory features as compared to DeepSEA, DanQ, and DanQ-JASPAR. The optimal ChromDL model highlights the benefits of RNN layers for prediction of genomic features as compared to the traditional CNN architecture. We also attribute the presence of the two BiLSTM layers and a BiGRU layer to the improved predictive ability of the developed classifier.

In some cases, ChromDL demonstrates an improved prediction accuracy in detecting low-affinity TF binding and TF-binding motif identification when compared to DeepSEA and DanQ/DanQ-JASPAR. Evidence for this was observed through the analysis of in vitro derived PBM and SELEX motifs, which are used to measure the protein-DNA-binding affinity in several orders of magnitude, meaning that they are a very powerful tool in the comparison of DNA-binding specificity. It is notable as a result that ChromDL has the capacity to predict these key base pairs with higher accuracy as compared to DeepSEA and DanQ/DanQ-JASPAR models.

In addition, we find that ChromDL can be utilized with an enhancer model such as the TREDNet second-phase classifier for the prediction of enhancer sequences that are sensitive to regulatory region mutations as validated through experimental raQTL point mutations. This illustrates the possibility of very effective enhancer classifiers using ChromDL’s output context and deep learning neural networks for the accurate prediction of regulatory mutations and to quantify the enhancer modulation of these mutations.

Further updates to this model could investigate lowering the training speed to analyze model iterations more quickly, initializing the model with motifs before training to see if scores improve, and training the model on outside datasets as they become available to make the model more versatile for the prediction of regulatory features. We expect to see further hybrid architectures of varying size, speed, and composition as effective in exploring human regulatory DNA using similar exploration techniques with the availability of faster GPUs and more sophisticated layer implementations.

We have built what we believe to be the most accurate regulatory classifier to date in ChromDL, with the ability to predict TFBSs and DHSs with performance metrics beyond that of all studied predecessors. We believe ChromDL’s sensitivity to low-affinity TF binding, its flexibility with secondary classifiers while retaining strong performance metrics, and its base pair sensitivity to point mutations make it a powerful tool to aid in the development of new technologies and assist in further novel discoveries in the field of epigenetics.

Supplementary data

Supplementary data is available at Bioinformatics online.

Conflict of interest

None declared.

Funding

This work was supported by the Intramural Research Program of the National Library of Medicine, National Institutes of Health.