A Biologically Informed and Efficient DNA Sequence Learner for Predicting Functional Genomics Events

Abstract

Elucidating the functional mechanisms underlying most associations between phenomes and genomes uncovered by genome-wide association studies remains a challenging problem. Deep neural networks that excel in feature learning from sequential data have recently emerged as promising approaches to addressing this challenge by mapping sequence patterns in DNA to functional genomic events. Despite the impressive progress made in this regard, the existing studies are largely limited to examining a type of network architecture that primarily consists of simple stacked convolutional layers of filters of a uniform size. These networks lack the consideration of specifics in the mapping of DNA sequences to functional genomic events, thereby impairing the learning efficiency of these networks. To address this problem, in this article, we propose an efficient DNA sequence learner (EDSL), a novel biologically informed architecture that (1) introduces filters of varying sizes in the first convolutional layer to enhance the learning of sequence patterns of diverse sizes and (2) utilizes dense connections to facilitate the participation of sequence patterns at varying levels in prediction. Our results regarding both synthetic data and a dataset consisting of 367 experimentally derived functional genomic profiles demonstrate the effectiveness of the proposed design choices and the superiority of the EDSL over existing networks in terms of both prediction performance and sequence pattern learning. Moreover, our ablation study indicates that both the proposed design choices enhance learning—importantly, in a differential and complementary manner.

1. INTRODUCTION

Almost all common human diseases have a genetic component, from diabetes to a variety of forms of cancers, and psychiatric disorders, including addictions. To localize the underlying genetic variants, genome-wide association studies (GWASs) have been widely and fruitfully applied. According to the human GWAS Catalog, over 1300 GWASs have been performed for approximately 3200 different traits, with the majority related to diseases, and leading to the discovery of over 110,000 phenome–genome associations. However, GWASs offer no mechanistic understanding of the identified associations, thereby hampering the translation of these discoveries into actionable targets for clinical intervention. The vast majority (approximately 95%) of the associated variants identified by GWASs are located in noncoding genomic regions, and their precise biological functions remain unclear (Schipper and Posthuma, 2022).

Considering the large number of genetic variants and cell types, experimentally elucidating the functional roles of each variant implicated by GWASs in biological labs is cost-prohibitive, thus necessitating cost-effective alternatives. Advances in high-throughput sequencing biotechnologies have enabled the accumulation of massive genome-wide profiles of functional genomic events across a wide range of cell types—for example, chromatin accessibility, histone modification, DNA methylation, transcription factor (TF) binding, and gene expression, among others. These profiles, available through large repositories—such as ENCODE (Feingold et al., 2004), Roadmap (Kundaje et al., 2015), and NCBI GEO—combined with the adoption of Graphics Processing Units for deep learning, have created opportunities for training deep neural networks (DNNs) as cost-effective alternatives to decipher genetic codes embedded in DNA sequences.

Indeed, DNNs have been recently trained to predict several different functional genomic events in varying cell lines and a wide range of primary tissues (Angermueller et al., 2017; Chen et al., 2022; Hong et al., 2020; Singh et al., 2019; Zeng and Gifford, 2017; Zhou et al., 2018; Zhou and Troyanskaya, 2015). Despite these impressive efforts, the network design in existing studies generally lacks the adaptation of specifics in functional biology, thus resulting in suboptimal performance and learning efficiency (refer to the Supplementary Data for detailed discussion). In the present study, we introduce the efficient DNA sequence learner (EDSL), a biologically informed architecture designed to address existing inefficiencies in DNN applications for functional genomic event prediction. First, it is known that functional elements in DNA sequences differ in terms of their length. As illustrated in Figure 1, the length of vertebrate protein binding motifs ranges from 6 bp to 24 bp, with 12 bp being the median. We introduce filters of varying sizes in the first convolutional layer of the EDSL to enable it to learn functional elements with improved precision and efficiency. Second, in contrast with other domains, such as computer vision (Zeiler and Fergus, 2014), filters beginning from the very first layer characterize sequence patterns that are potentially functional and predict genomic events. Thus, dense connections are introduced to facilitate their direct participation in the final prediction and, thus, enhance the learning efficiency of EDSL.

FIG. 1.

The distribution of the size in base pair of 786 known motifs of vertebrates (including human ones). Data were obtained from JASPAR (Castro-Mondragon et al., 2022).

In this study, we evaluated the effectiveness of the proposed design choices using both synthetic data and a dataset consisting of 367 experimentally derived functional genomic profiles. The results obtained from the experimentally derived data demonstrate the effectiveness and superiority of the EDSL over existing architectures, achieving an improvement of 6.5% in area under the precision–recall curve (AUPRC) over the second-best performing one, and learning sequence patterns that better align with known biology. In the study with the synthetic data, a design with varying filter sizes in the first convolutional layer is also presented with improved effectiveness in sequence pattern learning compared to networks with fixed filter sizes. Additionally, the EDSL maintained a higher AUPRC when the training sample size was reduced, thereby showcasing its efficiency compared to its closest competitor. Moreover, we conducted an ablation study to assess the individual effect of the two proposed design choices using the experimentally derived dataset and found that they both helped to improve the performance of the EDSL and, importantly, functioned in a differential and complementary manner.

2. METHODS

2.1. Convolution operation in DNA sequences

Convolutional neural networks (CNNs) excel in tasks that require understanding the order and arrangement of elements in sequences, such as learning from DNA sequences. The most fundamental aspect of CNNs is the convolutional operation that enables the extraction of local patterns by sliding filters through sequences. These filters are much shorter in length than the sequences to which they are applied. Taking as an example the convolution operation in the first convolutional layer in a typical CNN trained for predicting functional genomic events, both the input and output of such an operation are sequences. The one-hot encoding is the most commonly utilized approach to numerically encode the input DNA sequence. Let $X\in {\mathbb{R}}^{l\times 4}$ be the one-hot encoded input DNA sequence with a length of l bp. Then, each element in the output sequence $y\in {\mathbb{R}}^{l-f+1}$ from applying a filter that has a size of f can be calculated as $y_i=X_i\odot W$. Here, $W\in {\mathbb{R}}^{f\times 4}$ contains the learnable parameters that dictate the sequence patterns represented by the filter. $X_i\in {\mathbb{R}}^{f\times 4}$ is a section of $X$ that is f bp in length, beginning from position i, and $\odot$ is the dot product of two matrices defined as the summation of their element-wise product.

2.2. Efficient DNA sequence learner

The design of the proposed EDSL is motivated by two observations from exploring the biology that underlies functional genomic events, such as TF binding, chromatin accessibility, and histone modification. One is the large variation seen in the size of sequence patterns that attract DNA-binding proteins, as demonstrated in Figure 1. Functional genomic events are known to be regulated by the binding of certain proteins at specific genome locations. Disruption in sequence patterns may lead to changes in DNA–protein binding and, subsequently, a dysfunction in related genomic events. Therefore, precisely characterizing the functional sequence patterns is critical to deciphering the genetic code that underlies genomic events. The other observation is that short-sequence patterns (beginning from those characterized by filters in the first convolutional layer of a typical CNN) regulate DNA–protein binding, thus having direct predictive power for genomic events. This is a unique aspect in the prediction of functional genomic events compared to other domains, such as computer vision, where low-level features typically have minimal direct predictive power related to a target.

As illustrated in Figure 2, the overall architecture of the EDSL can be decomposed into two major components, an encoder and a collection of classification heads. Each head is used to produce profile-specific predictions when jointly training for multiple function genomic profiles. Following previous studies (Avsec et al., 2021; Chen et al., 2022; Zhou et al., 2018; Zhou and Troyanskaya, 2015), all models share the same encoder when joint training is applied with the assumption that the governing genetic code underlying different types of genomic events in distinct cell types is common. Such joint training is expected to improve the generalizability of trained models, as demonstrated in a wide range of machine learning literature. Overall, the encoder takes a one-hot encoded DNA fragment — that is, $X\in {\mathbb{R}}^{l\times 4}$ — that produces the embedding that represents the fragment in a vector space. This embedding is subsequently used as the input for classification heads. Each head predicts the presence of a specific type of genomic event at the center of the input fragment in cells of a specific type.

FIG. 2.

EDSL architecture. The model consists of an encoder and multiple classification heads. The encoder begins with three sets of convolutional filters of sizes 9, 17, and 25 to capture sequence patterns of varying lengths. The outputs are concatenated and passed through three convolutional blocks, each followed by pooling, normalization, and activation layers. Dense connections aggregate the features from all blocks using max and average pooling. The final embedding is fed into task-specific classification heads for multi-label prediction. EDSL, efficient DNA sequence learner.

More specifically, the encoder consists of four convolutional blocks, after which there are two fully connected layers. Uniquely, to promote both efficiency and effectiveness in characterizing sequence patterns, we used three sets of filters with differing sizes in the first convolutional layer. The three different filter sizes are 9, 17, and 25, and the number of filters in each size is 128, 640, and 64, respectively. These numbers were specially selected to approximately align with the distribution of motif length among known TF binding motifs, as demonstrated in Figure 1. More specifically, there are approximately 128 motifs that have a length that does not exceed 9 bp, 64 motifs with a length of over 17 bp, and 640 motifs with a length between 9 bp and 17 bp. After applying batch normalization (BN) and nonlinear activation, the output maps of the three sets of filters are concatenated and fed into another BN layer to produce the final activation of the first convolutional block (Conv1). Note that given the large number of functional motifs and the wide variation in their length, it is computationally prohibitive to optimize the size and quantity of filters as free parameters. In addition, the use of only three different sizes of filters in our design is to prevent overfitting in the JASPAR database.

The activation map of the first convolutional block is not only used as the input to the second convolutional block (Conv2) but also concatenated with the activation map of other convolutional blocks to produce the final embedding. To reduce the dimensionality of the embedding, the concatenation is done before an aggregation operation that involves a max pooling and a global average pooling, in sequence. This is another unique design of EDSL, which aims to facilitate the participation of sequence patterns learned at different levels in the final prediction. The remaining three convolutional blocks all have the same composition, with only the difference being their filter size, each beginning with a max pooling layer followed by convolution, layer normalization, and nonlinear activation in sequence. Another notable design choice of the encoder is the use of the exponential function (Expon) as the nonlinear activation, which has been shown to improve the efficacy of the characterization of sequence patterns (Koo and Ploenzke, 2021).

2.3. Loss function

To handle the large imbalance in the class label distribution in the experimentally derived profiles (most of the examples are negatives), the weighted cross-entropy, as defined in Equation (1), was used in training networks.

$$L=-\frac{1}{N}{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{t=1}^T{\displaystyle \sum _{c=1}^2{w}_c^{(t)}}}}{y}_{ic}^{(t)}\hspace{0.17em}\log \left({\widehat{y}}_{ic}^{(t)}\right),$$ (1)

where i and c are index examples and class labels (i.e., positive and negative), respectively; T represents the number of profiles in joint training; N is the total number of DNA fragment; $y_{ic}^{(t)}$ and ${\widehat{y}}_{ic}^{(t)}$ represent the true and predicted labels, respectively; and $w_c^{(t)}$ is the weight associated with individual classes to address class imbalance. The value of $w_c^{(t)}$ is profile-specific and determined by the inverse frequency method. Specifically, for each profile t, $w_c^{(t)}$ is calculated as $w_c=(1/N_c^{(t)})(N/{\max }_{c^{\prime }}(N_{c^{\prime }}^{(t)}))$, where $N_c^{(t)}$ and $N_{c^{\prime }}^{(t)}$ represent the number of samples from class c and $c^{\prime }$, respectively, in the profile; moreover, ${\max }_{c^{\prime }}(N_{c^{\prime }}^{(t)})$ finds the number of examples in the major class, which is defined as the class that has the largest sample size. This approach ensures that classes with fewer samples are assigned higher weights, thereby encouraging the model to pay more attention to these underrepresented classes during training.

3. RESULTS AND DISCUSSION

To validate our hypothesis that the uses of filters of varying sizes and dense connections improve learning efficiency, we first conducted simulation studies on a synthetic dataset. Two separate studies were performed, one for revealing the impact of the use of filters of varying sizes and the other for demonstrating the value of dense connections (refer to Supplementary Data for description and results). We then evaluated our proposed architecture, as illustrated in Figure 2, using an aggregated dataset that includes a total of 367 functional genomic event profiles and has been previously employed as well (Shiri and Sun, 2022). These profiles include three types of functional genomic events: chromatin accessibility, histone modification, and TF binding in a wide range of cell lines and primary tissues related to the human brain (refer to Shiri and Sun, 2022) for a detailed description). After preprocessing (see Shiri and Sun, 2022 for procedures), each profile essentially contains a set of genomic regions, referred to as peaks, that indicate locations in the genome where corresponding genomic events were observed.

To generate examples for network training and evaluation from these profiles, we first tiled the reference genome into 200-bp bins. Then, we assigned a set of binary labels to each bin, one for every profile in accordance with the overlap between the bins and peaks in the corresponding profile. More specifically, a bin was labeled as positive for a profile if the bin overlaps with any peak in the profile by at least 50 bp or otherwise negative. To reduce the number of examples, we retained only those bins that had a positive label in at least one of the profiles. Last, each bin was extended from both sides to obtain a DNA fragment that was 1001 bp in length; this served as the prediction input. These procedures eventually yielded a labeled dataset that includes a total of 10,386,311 examples, each of which has 367 binary labels (+/−). The network training details, such as data partition and hyperparameter settings, are presented in the Supplementary Data.

To demonstrate the superiority of the proposed design choices, we compared the EDSL to several existing architectures, including DeepSEA (Zhou and Troyanskaya, 2015), DeeperDeepSEA (Zhou et al., 2018), Basset (Kelley et al., 2016), DanQ (Quang and Xie, 2016), ExplaiNN (Novakovsky et al., 2023), and Sei (Chen et al., 2022). A brief introduction to these methods is presented in the Supplementary Data. All compared architectures, except Sei, are representatives of existing ones in the sense that they neither use filters of varying sizes in the first convolutional layer nor do they contain dense connections, as all the rest do. Compared to others, these networks have relatively small sizes, with the total number of trainable parameters being similar to that in the EDSL or approximately three times more. Despite there being skip connections employed in Sei, their primary role is the same as that in standard residual networks—facilitating the training of very deep networks. Without dense connections, such as those used in EDSL, sequence features characterized by lower network layers are still required to effectively go a long way to reach the classification head to contribute to the final prediction, which could greatly impair the learning efficiency.

Following existing studies (Angermueller et al., 2017; Chen et al., 2022; Zeng and Gifford, 2017; Zhou and Troyanskaya, 2015), we computed the area under the receiver operating characteristic curve (AUROC), accuracy, and F1-score, with the major class considered as positive, to evaluate the performance of all models. Here, the major class refers to the class label that has more examples than the other one, which is referred to as the minor class. It is known that these metrics may inflate the corresponding results, thus overestimating the performance when there is a large imbalance among class labels (Dodlapati et al., 2022). Considering that the vast majority of the examples (over 98%) in most profiles are negative, we also compute the AUPRC and the F1-score, with the minor class considered positive, to obtain a full view of the performance of models.

3.1. Prediction performance

The performance of all compared networks measured by five different metrics is provided in Table 1. ROC and PR curves are presented in Figure S3 in Supplementary Data. According to the results in this table, the EDSL outperformed all compared architectures. Even though the improvement in AUROC, accuracy, and F1-score of the major class (−) is relatively small, a substantial increase is evident in both the AUPRC and F1-score of the minor class (+) by 6.5% and 3.5%, respectively, over the second-best performing network. Considering the rather large class imbalance, these results are significant. Despite being the most recently proposed architecture, ExplaiNN exhibits the lowest performance compared to others. Moreover, it is interesting to note that even though DeeperDeepSEA has double the number of convolutional layers compared to DeepSEA, it performs mostly in the same manner as DeepSEA. Due to the added complexity, Sei has a performance that is the closest to the EDSL compared to others. However, in contrast, again, because of the added complexity, its training time is also substantially higher than that of all others—almost 10 times higher than that of DeepSEA, which has the lowest training cost (Table 1). Compared to DeepSEA, the EDSL has a higher training time, but this increase is substantially smaller—only approximately 30%. Consistent with the observations in our simulation study, the results for the experimentally derived data again demonstrate the superior efficiency and prediction performance of the EDSL as compared to existing approaches. Additionally, to validate the effectiveness of the use of filters of three different sizes in the first convolutional layer, we explored a variant of EDSL, which includes nine different filter sizes (refer to the Supplementary Data for details). Its suboptimal prediction performance and significantly increased training time, as presented in Table 1, suggest that the choice of three different filter sizes in the EDSL is a good balance between learning flexibility and customization toward known biology.

Table 1.

Comparison of the Efficient DNA Sequence Learner and Previous Networks Using Experimentally Derived Data

Network	Time (h)	AUROC	AUPRC	Accuracy	F1-score
					(−)	(+)
DeepSEA	26	0.910	0.327	0.967	0.981	0.357
DeeperDeepSEA	53	0.906	0.326	0.968	0.983	0.357
Sei	326	0.914	0.337	0.967	0.981	0.367
DanQ	40	0.895	0.298	0.962	0.978	0.332
Basset	37	0.910	0.329	0.969	0.984	0.359
ExplaiNN	64	0.884	0.252	0.964	0.981	0.302
EDSLa	59	0.918	0.354	0.969	0.982	0.377
EDSL	34	0.918	0.359	0.972	0.985	0.380
			$\uparrow 6.5$%			$\uparrow 3.5$%
			( $4e-93$)			( $10e-40$)

The bold values indicate the best performance achieved for each metric.

^aA variation of the EDSL that has filters of nine different sizes. The reported results were averaged across all profiles. The bottom two rows present the performance improvement in percentage by the EDSL over the second-best performing architecture (i.e., Sei) and p values of the tests for statistical significance of the improvement.

AUROC, area under the receiver operating characteristic curve; AUPRC, area under the precision–recall curve; EDSL, efficient DNA sequence learner.

3.2. Motif analysis

We performed a motif analysis to explore the short DNA sequence patterns characterized by convolutional filters in the trained networks of all compared architectures. For all architectures, except Sei, we studied the filters in the first convolutional layer, as there is a max-pooling operation before the second convolutional layer in these architectures, which introduces spatial uncertainty and messes up the sequence characterization. For Sei, we studied the filters in the fourth convolutional layer because there is neither max-pooling nor a dilated convolutional operation among its first four convolutional layers. These filters have a receptive field of 33, which implies that they are capable of representing any sequence patterns that have a length of up to 33 bp. Considering the distribution of the size of the known motifs, as in Figure 1, Sei is capable of precisely characterizing all known motifs.

The sequence patterns obtained by each architecture were compared to the known protein-binding motifs in the JASPAR database. A match fraction was calculated for each architecture; such a fraction is defined as the proportion of the sequence patterns learned by the respective architecture that can find a match in the reference database (i.e., JASPAR). The EDSL achieved the highest match fraction of 71.8%, which is higher than that of all other architectures by a very large margin; the next largest match fraction was 52.8% for DanQ, followed by 50.8% for Sei. Rather low match fraction as compared to those of the remaining models, uniform small-size filters not only failed to learn motifs that are relatively long, as expected, but were also struggling with characterizing short ones. This is also well demonstrated in the visualization of the learned sequence patterns. As depicted in Figure 3, where five representative reference motifs of varying sizes are plotted, they either failed to identify the short motif (SOX6) at all or made an inaccurate characterization (POU6F1). In the case of relatively long motifs (NFIB and CTCF), when a partial match exists, it is rather inaccurate. The large filter size (26) in the first convolutional layer of DanQ and the large receptive field of Sei before any max pooling operation enabled the two to achieve a much higher match fraction. However, due to their learning inefficiency, as discussed previously, the matches are imprecise and noisy (Fig. 3). The unique design choices of the EDSL prompt it to learn sequence patterns that have clean and precise matches. Similar to the observations in our simulation study, the results here again demonstrate the superiority of the EDSL in the learning of DNA functional elements over existing approaches.

FIG. 3.

Motif learning comparison between the EDSL and the compared network architectures on experimentally derived data. The match fraction associated with each architecture is given in parentheses of respective column labels. The + or − at the end of a motif name indicates its strand. Empty entries indicate no sequence patterns learned by the corresponding network that match well with the respective reference motif. EDSL, efficient DNA sequence learner.

3.3. Ablation study

Compared to existing network architectures—such as DeepSEA, DeeperDeepSEA, and Sei—there are two major design choices that are different in EDSL: one is the use of filters of varying sizes in the first convolutional layer and the other is the addition of dense connections. Both these design choices may have contributed to the improved performance of EDSL. To understand the impact of each individual network architecture, we conducted an ablation study using the experimentally derived data. Three networks were implemented by modifying from EDSL, with variation in the use of the two design choices: (1) baseline, which includes neither filters of varying sizes nor dense connections; (2) including only varying size filters; and (3) including only dense connections. The obtained results of this ablation study—together with that of the full model, that is, the EDSL—are provided in Table 2. These results suggest that both design choices significantly contributed to the performance improvement in EDSL. Importantly, they contributed in a differential and complementary way, since the combination of the two led to the best performance.

Table 2.

Performance of Networks Involved in the Ablation Study Using Experimentally Derived Data

Network	AUROC	AUPRC	Accuracy	F1-score
				(−)	(+)
Baseline	0.907	0.325 (–, –)	0.968	0.981	0.353 (–, –)
Varying size filters only	0.916	0.350 ( $\uparrow 7.7\%$, $4e-91$)	0.971	0.985	0.376 ( $\uparrow 6.5\%$, $1e-68$)
Dense connections only	0.914	0.343 ( $\uparrow 5.5\%$, $9e-74$)	0.968	0.982	0.370 ( $\uparrow 4.8\%$, $1e-50$)
EDSL	0.918	0.357 ( $\uparrow 9.8\%$, $7e-112$)	0.972	0.985	0.380 ( $\uparrow 7.6\%$, $1e-87$)

The reported results are averaged across all profiles. Improvement over baseline and p value of the tests for statistical significance of the improvement are provided in parentheses. AUROC, area under the receiver operating characteristic curve; AUPRC, area under the precision–recall curve; EDSL, efficient DNA sequence learner.

4. CONCLUSIONS

In this study, we introduced EDSL, a novel biologically informed network architecture. Through the introduction of filters of varying sizes in the first convolutional layer and dense connections, the EDSL aimed to enhance the learning that links DNA sequence patterns to functional genomic events. The empirical results for both synthetic data and a dataset consisting of 367 experimentally derived profiles validated the effectiveness of EDSL. Our findings emphasize the advantage of biologically adapted network architectures for the prediction of functional genomic events from DNA sequences, thereby offering effective design choices that can be utilized in future research in the field.

5. LIMITATIONS AND FUTURE WORK

While the EDSL achieves state-of-the-art performance in predicting functional genomic events and learning biologically meaningful sequence patterns, there are limitations that warrant consideration. First, the selection of only three filter sizes in the first convolutional layer, although biologically informed and computationally efficient, may limit the model’s ability to capture rare or atypical sequence motifs beyond the predefined size range. Future work could explore dynamic or learnable filter size mechanisms that are more efficient in accommodating such a variation. In addition, the evaluation of the EDSL was conducted primarily on functional profiles related to the human brain. Despite the fact that its success is expected when applied to other tissues, organisms, or disease contexts, the actual results remain to be seen. Finally, this study is limited to the development of the EDSL and the validation of its efficacy and efficiency. The application of trained models in practical biomedical problems, such as prioritizing genetic variants identified in various GWASs for experimental verification or annotating genetic variants with functional impact, in general, has not been explored. This will be done in our future work.

AUTHORS’ CONTRIBUTIONS

M.S.: Designing and implementation of the methodology, investigation, and writing—original draft. J.S.: Designing of the methodology, supervision, writing—original draft, and funding acquisition.

AUTHOR DISCLOSURE STATEMENT

The authors declare that they have no conflicts of interest relevant to this study.

FUNDING INFORMATION

This work was supported by National Institutes of Health grants: R01HL157519 and R15DA055906.