LOGOWheat: deep learning–based prediction of regulatory effects for noncoding variants in wheats

Abstract

Identifying the regulatory effects of noncoding variants presents a significant challenge. Recently, the accumulation of epigenomic profiling data in wheat has provided an opportunity to model the functional impacts of these variants. In this study, we introduce Language of Genome for Wheat (LOGOWheat), a deep learning–based tool designed to predict the regulatory effects of noncoding variants in wheat. LOGOWheat initially employs a self-attention-based, contextualized pretrained language model to acquire bidirectional representations of the unlabeled wheat reference genome. Epigenomic profiling data are also collected and utilized to fine-tune the model, enabling it to discern the regulatory code inherent in genomic sequences. The test results suggest that LOGOWheat is highly effective in predicting multiple chromatin features, achieving an average area under the receiver operating characteristic (AUROC) of 0.8531 and an average area under the precision–recall curve (AUPRC) of 0.7633. Two case studies illustrate and demonstrate the main functions provided by LOGOWheat: assigning scores and prioritizing causal variants within a given variant set and constructing a saturated mutagenesis map in silico to discover high-impact sites or functional motifs in a given sequence. Finally, we propose the concept of extracting potential functional variations from the wheat population by integrating evolutionary conservation information. LOGOWheat is available at http://logowheat.cn/.

Introduction

Crops, of which continuous and sustainable productivity improvement have been ongoing for almost 10 000 years, need to feed a faster-growing global population under future climate changes. Among these crops, bread wheat (Triticum aestivum) is the most widely grown worldwide, providing ~20% of the calories and proteins of the human diet [1, 2]. By 2050, the world population is projected to increase by 2 billion people, requiring a 60% increase in wheat yield to meet the demand of the growing population [3].

The generation and recombination of natural variation form the genetic basis for crops to respond to various environmental conditions and agricultural practices. Mapping the causal alleles among natural variations and decoding these longstanding genetic changes are central to understanding crop evolution and improving crop breeding. Quantitative trait locus (QTL) analysis and genome-wide association study (GWAS) have become powerful and routine approaches for dissecting the genetic basis of complex traits in crops [4–6]. However, despite discovering numerous genomic regions associated with complex traits through these methods, only a few have been linked to mechanistic hypotheses regarding their molecular and cellular functions, let alone practical applications. One major reason is that >90% of GWAS signals associated with complex traits and disease-related phenotypes often lie outside protein-coding exons, falling within cis-regulatory elements (CREs) such as promoters, enhancers, and insulators [7, 8]. Given the dynamic distances between regulatory regions and genes, as well as the spatiotemporal specificity of gene regulation, explaining the underlying mechanisms of variation in noncoding regions is more challenging than in protein-coding regions. As of May 2020, 364 causal genes have been identified and validated in the six crops (rice, maize, wheat, tomato, soybean, and barley), but only a third of causal variants fall in noncoding regions [9]. Even when QTLs are fine-mapped to small regions or GWAS achieves high resolution, determining actual causal variants from numerous natural variations based solely on association results remains difficult [10–14]. GWAS mapping resolution or QTL detection power is considerably affected by linkage disequilibrium (LD) [15], which is influenced by genetic forces such as recombination, selection, mutation, and drift [16, 17]. In species with relatively small genome sizes, such as rice and maize, LD decays over short distances, facilitating the identification of candidate genes [18, 19]. However, hexaploid wheat possess a large genome size (~16 Gb) and high level of selfing, resulting in unexpected distances of LD even reached to 20 cM in centromeric regions, significantly hindering the efficiency of GWAS and QTLs to identify candidate genes [20, 21].

The interaction between different CREs types and epigenetic modifications has become clear. H3K4me3 is predominantly enriched in promoter regions, whereas H3K4me1 and H3K27ac serve as characteristic markers for active enhancers in human [22, 23]. Active markers such as H3K4me3, H3K9ac, and H3K27ac are found in active CREs, whereas the repressive mark H3K27me3 indicates the poised or repressed state of CREs in plants [24, 25]. Extensive experimental mapping [26–28] and integration of histone modification patterns and DNA accessibility have significantly contributed to the annotation of various types of CREs in human [29, 30]. These maps help elucidate the molecular mechanisms underlying complex traits by identifying the preferential enrichment of genetic variants associated with traits in tissue-specific gene regulatory elements [31, 32] and by fine-mapping potential causal genetic variants within enriched annotations [33, 34]. Recently, with the completion of a high-quality bread wheat reference genome, an increasing number of wheat epigenetic maps have been produced under diverse developmental and environmental conditions [35–41], facilitating the accurate narrowing down of the locations of candidate functional regions and providing a valuable data resource for modeling the regulatory code inherent.

At the same time, deep learning has not only advanced natural language processing (NLP) and computer vision (CV) but also led to significant breakthroughs in bioinformatics. Several models have been constructed and successfully utilized to extract complex features from genomic sequences and predict epigenetic states in human [42–46]. DeepSEA is one of the classic examples, which has applied deep convolutional neural network (CNN) architecture to a large-scale chromatin-profiling dataset comprising 919 chromatin features, including transcription factor binding, DNA accessibility, and histone-mark profiles [46]. This model improves the prioritization of functional variants, including expression QTLs (eQTLs) and disease-associated variants. Inspired by DeepSEA, plantDeepSEA employing the same model architecture was proposed to predict chromatin accessibility in six plant species (excluding wheat) [47] and had been effectively used to prioritize regulatory causal variants and identify high-impact sites. In 2017, the Transformer emerged as a state-of-the-art architecture in sequence modeling, relying on the self-attention mechanism to relate different positions across a sequence and compute a contextualized representation [48]. While CNNs excel at capturing locality, Transformer can handle long-range dependency without position bias, surpassing CNNs in numerous global sequence classification tasks. Since then, a new wave of pretrained language models using self-supervision techniques has emerged as a core trend in NLP, with Bidirectional Encoder Representations from Transformers (BERT) [49] and its lite version, A Lite BERT (ALBERT) [50], being representative models. Considering the inherent sequential nature of the genome, which is analogous to documents composed of words, characters, and phrases, Language of Genome (LOGO), a lighter ALBERT-like structure, was proposed to learn contextualized representations of the reference genome hg19 [45]. By integrating convolution and self-attention, LOGO surpassed DeepSEA as the state-of-the-art multitask predictive model on thousands of chromatin features in human.

However, advanced deep learning tools in wheat research are lacking. This study initially collected, downloaded, and processed epigenomic profiling data using a unified pipeline to obtain baseline data for model training. To assess the functional impact of genetic variation in wheat, we then developed and trained models using various parameter configurations of both the DeepSEA and LOGO architectures. The optimal model, named Language of Genome for Wheat (LOGOWheat), was selected to assess the functional impact of genetic variation. The capabilities of LOGOWheat were further illustrated through two case studies, prioritizing functional variants proximal to the wheat gene TaSNAC8-6A [51] and discovering high-impact sites within the promoter region of wheat gene VRN1 [52]. Finally, we explored the distribution of variant scores across different genomic regions and proposed integrating evolutionary conservation information to identify potential functional variants in wheat populations.

Overall, LOGOWheat is an accurate and efficient tool for predicting the regulatory effects of noncoding variants, holding significant scientific research and application value.

Results

Overview of LOGOWheat

To capture the regulatory effects of noncoding variants, we developed a deep learning–based pipeline named LOGOWheat, which comprises three phases: pre-training, fine-tuning, and prediction (Fig. 1).

Figure 1.

Schematic overview of the LOGOWheat pipeline, a strategy for predicting regulatory effects of noncoding variants. This approach comprises three phases: (A) the pretraining phase, which uses wheat reference genome as input to perform unsupervised learning tasks; (B) the fine-tuning phase, which utilizes epigenome data to train the model for predicting chromatin states; and (C) the prediction phase, where LOGOWheat predicts multiple chromatin states for each input sequence. It provides two main functions based on the prediction results: assigning scores and prioritizing causal variants and constructing a saturated mutagenesis map in silico to discover high-impact sites or functional motifs. (D) The deep learning architecture of LOGOWheat is based on a Transformer-based bidirectional encoder model.

In the pretraining phase (Fig. 1A), we treated the wheat reference genome as a corpus containing complex life information specific to wheat. With five types of nucleotides (A, T, C, G, and N for ambiguous cases), we considered adjacent k bases (k-mer) as a single word, ultimately choosing k = 5 for this study. We established a token vocabulary to encompass all possible k-mers, resulting in a token vocabulary size of 5⁵ when employing the k-mer (k = 5) strategy. Subsequently, we partitioned both the forward and complementary sequences of the genome into 100-bp nonoverlapping bins, yielding ~300 million segments. To generate training samples, we extended 1000-bp forward along the genome for each bin, akin to input sentences in the NLP field.

During the fine-tuning stage, we constructed a multilabel classification task to predict seven chromatin states of the wheat genome, including chromatin accessibility from assay for transposase accessible chromatin with high-throughput sequencing (ATAC-seq) and six histone modification states from chromatin immunoprecipitation coupled to massively parallel DNA sequencing (ChIP-seq). We divided the wheat genome into nonoverlapping 200-bp bins, using only bins with at least one chromatin feature for fine-tuning the model, resulting in 6 374 930 bins. We also padded 400-bp flanking regions on both sides to generate a 1000-bp context sequence suitable for the model’s input.

In the model prediction phase, we used fine-tuned models to assess the regulatory impact of genetic variation. Specifically, we input the reference sequence and the variation sequence into the model, calculate the Euclidean distance between the predicted results, and define this as the variation score to represent the comprehensive effect of sequence variation on the seven chromatin states. Using variant scores, LOGOWheat offers two primary functions: prioritizing causal variants within a given variant set and conducting an in silico saturated mutagenesis scan of all potential nucleotide substitutions to identify high-impact sites and potential functional motifs within a sequence of interest.

The backbone of LOGOWheat processing flow is a bidirectional encoder model based on Transformer architecture, which harnesses the encoder component of the Transformer architecture to acquire representations of input sequences through a multihead self-attention mechanism (Fig. 1D). In our study, we employed two identical Transformer encoder layers (L = 2), each consisting of two sublayers: a multihead self-attention layer (the number of heads A = 8) followed by a position-wise fully connected feed-forward layer (the hidden size H = 256). A residual connection is added to each sublayer, followed by layer normalization. Before inputting into the Transformer network, each 1000-bp sequence is tokenized using the k-mer representation. Furthermore, to prevent biased segmentation and further augment training data, we slide 1 bp for each 1000-bp sequence, generating multiple sets (n = k) of k-mer tokens as input training instances. Token embeddings (embedding size E = 128), together with position embeddings, constitute the ultimate input to the Transformer network. During the pretraining stage, we employ a masked language model (MLM) task with a 15% masking ratio to characterize the wheat genome. In the fine-tuning stage, the final hidden state corresponding to the initial input token [Classification (CLS)] is considered as the aggregate representation of chromatin state prediction task, followed by a classification layer with sigmoid activation to output a 7D vector from 0 to 1 that represents the likelihood of sequence belonging to the seven chromatin states.

A comprehensive epigenomic mapping of wheat

With the completion of a high-quality bread wheat reference genome, an extensive collection of epigenome data has been generated, providing valuable data resources for modeling chromatin states. Here, we present a comprehensive epigenomic mapping of wheat, manually curated and uniformly processed from public datasets National Center for Biotechnology Information (NCBI) and China National Center for Bioinformation (CNCB). These data were generated by high-throughput sequencing experiments, including ChIP-seq and ATAC-seq. The resulting compendium includes seven epigenetic features, chromatin accessibility and six histone modifications (H3K27ac, H3K27me3, H3K36me3, H3K4me1, H3K4me3, and H3K9ac) across various cultivars (Chinese Spring, Paragon, Aikang58, etc.) and nine tissues (Fig. 2A). After manual screening and quality filtering, we obtained a total of 222 epigenomic datasets (Table S1, Table S2). Among them, the histone marks H3K4me3 and H3K27me3 had the highest number of datasets, with 60 and 53 datasets, respectively, while H3K4me1 had only 7 datasets. Recently, it has been observed that H3K4me3 and H3K27me3 Hi-ChIP loops regulate distal accessible chromatin regions [53], making them commonly used as active and repressive marks for CREs.

Figure 2.

Comprehensive epigenomic mapping of wheat. (A) The tree map displays the distribution of 222 epigenetic datasets across seven epigenetic marks and nine tissues. (B) The peak distribution of each mark surrounding various genomic regions, with “distal” denoting distances >2 kb from the TSSs. (C) The radar chart presents the proportion of genome coverage by peaks for each marker. (D) Overlap between ATAC-seq and the other six marks. (E) The density plot illustrates the distance distribution between the seven marks and the nearest TSSs. (F) The Circos plot summarizes the chromosomal distribution of epigenetic marks. The outermost circle depicts the ideograms of each chromosome, while the second outermost circle represents gene density, with blue and white indicating high and low density, respectively. Bar plots in the inner circles show the density of epigenetic marks, where the letters c, d, e, f, g, h, and i represent ATAC-seq, H3K27ac, H3K27me3, H3K36me3, H3K4me1, H3K4me3, and H3K9ac, respectively.

Our analysis identified 1 466 775 genomic regions containing at least one of seven epigenetic features, accounting for 6.33% of the whole genome and representing a comprehensive collection of potentially functional REs across the common wheat genome (Fig. 2C, Table S6). All seven epigenetic features are predominantly located interstitial and distal end regions with high gene densities (Fig. 2F) and exhibit the highest density enrichment at transcription start sites (TSSs) (Fig. 2E).

Based on the location relative to genes, we categorized peaks as distal (>2 kb), proximal (≤2 kb), 5′UTR, Coding Sequence (CDS), intronic, and 3′UTR (Fig. 2B, Table S3). Among these seven epigenetic features, over half of the peaks were identified in intergenic regions, a proportion significantly higher than that observed in previously well-studied model plant species with relatively small genomes such as Arabidopsis and rice. For instance, 56.58% of the ATAC-seq peaks are located >2 kb away from the TSSs of wheat genes. Notably, in Arabidopsis with a genome size of 125 Mb, the median distance between ATAC-seq peaks and TSSs was 129 bp [54], while in wheat with a genome size of ~14 Gb, this distance increased to 4937 bp.

A recent study characterizing histone marks and accessible chromatin regions in 13 plant species reported limited association between accessible chromatin and histone marks in distal regions [54]. We assessed the overlap between accessible chromatin and six other histone marks in wheat distal regions (Table S4). Our findings reveal that 79.39% of ATAC-seq peaks overlapped with at least one histone mark. As expected, the repressive mark H3K27me3 exhibited the lowest overlap proportion with ATAC-seq peaks (11.68%), while the activating mark H3K4me3 showed the highest overlap proportion (48.93%) (Fig. 2D). Furthermore, correlation analysis revealed that the densities of the active marks H3K9ac and H3K4me3 were highly correlated (r = 0.95) [38]. Our compendium also indicates a significant overlap between the active histone marks H3K9ac (78%) and H3K4me3 (100%) across the whole genome (Table S5).

These findings suggest that the genomic regions characterized by these markers may play significant regulatory roles. Our comprehensive epigenomic mapping provide rich resources for deep exploration of wheat regulomes and for learning the wheat genomic regulatory code through modeling. In human studies, the EpiMap compendium has collected 10 000 epigenomic maps [55], supporting the construction of models to predict each epigenetic feature for every sample [45, 46]. However, the wheat compendium in this study includes limited epigenetic datasets, such as only having two datasets for H3K9ac in sheath (Fig. 2A). Therefore, we integrated the epigenetic data from all tissues for each feature, trading off tissue specificity in favor of enhanced generalization capabilities.

Performance evaluation and comparison

In this section, we evaluate and compare the predictive performance of four models, simpleCNN, DeepSEA, LOGOWheat (3-mer), and LOGOWheat (5-mer), using six evaluation indices: accuracy, precision, recall, F1 score, auPRC, and auROC (Fig. 3A–D, Table S7–11). Among these models, simpleCNN and DeepSEA are CNN architectures, comprising one convolutional layer and three convolutional layers, respectively. Meanwhile, LOGOWheat (3-mer) and LOGOWheat (5-mer) are based on Transformer structures, with k-mers set to 3 and 5, respectively. Compared to traditional CNN architectures, LOGOWheat exhibits a significantly reduced number of parameters, ~33 times fewer than DeepSEA, and 2.5 times fewer than the relatively straightforward simpleCNN (Fig. 3E). In terms of runtime, LOGOWheat is significantly faster than DeepSEA but slower than simpleCNN, despite simpleCNN having a higher parameter count than LOGOWheat. This difference may be attributed to the extensive data augmentation strategies used by LOGOWheat, which substantially increase the size of the training dataset (Fig. 3F).

Figure 3.

Evaluation of prediction performance and model comparison. (A) Comparison of models based on the average values of evaluation metrics across seven chromatin features in the validation set. The prediction performance of LOGOWheat (5-mer) for each chromatin feature in the test set is assessed using (B) accuracy, precision, recall, and F1 score; (C) AUROC; and (D) AUPRC. Additionally, (E) the parameter size and (F) training time for four models are presented.

The epigenomic dataset (6 374 930) is partitioned into nonoverlapping training (4 620 894, 72.49%), validation (760 865, 11.94%), and test sets (993 171, 15.58%) based on chromosomes. Notably, chromosomes 6 and 7 were excluded from the training set. Chromosome 6 served as the validation set for parameter tuning and model comparison, whereas chromosome 7 was employed for final testing.

LOGOWheat (5-mer) exhibited the best predictive performance on the validation set, with the following average values across seven chromatin features: accuracy 81.20%, precision 74.24%, recall 65.15%, F1 score 68.97%, auPRC 77.53%, and auROC 85.27% (Fig. 3A). Therefore, we selected k = 5 as the default parameter for LOGOWheat in subsequent analyses. On the test set, LOGOWheat achieved similar predictive performance, with average values across seven chromatin features as follows: accuracy 82.04%, precision 73.12%, recall 64.62%, F1 score 68.26%, auPRC 76.33%, and auROC 85.31% (Fig. 3B–D).

Case study 1: Prioritizing functional variants proximal to the wheat gene TaSNAC8-6A

NAM, ATAF and CUC (NAC) proteins constitute a large family of plant-specific TFs involved in various regulatory and developmental processes. Several NAC proteins have been extensively validated through transgenic overexpression to enhance crop drought tolerance and improve yield or biomass [56]. A recent study identified the drought-responsive gene TaSNAC8-6A through a candidate gene association analysis of the stress-responsive NAC gene subfamily in a wheat panel comprising 700 varieties collected worldwide [51]. Further analysis involved resequencing a 3310-bp genomic DNA fragment corresponding to the full-length TaSNAC8-6A locus in 109 variation panel accessions, revealing 43 Single Nucleotide Polymorphism (SNPs) and 8 InDels. Among these 51 variations, the natural variant InDel-313 was validated through transgenic overexpression for its significant contribution to drought resistance in wheat.

In this section, we mapped these 51 variants to the wheat reference genome and extracted a 1000-bp reference sequence and allele sequence centered on each variant from the wheat reference genome (Fig. 3A). Subsequently, all sequences were input into LOGOWheat to obtain the variant scores (Fig. 3B). The results revealed that the natural variant InDel-313 had the highest variant score (0.0734), significantly surpassing the random expectation (1 million 1000-bp sequences were randomly extracted from the wheat reference genome as reference sequences. The allele sequences were generated by replacing the middle three bases of each reference sequence with “NNN.” The obtained variant scores were then sorted in ascending order, with a variant score of 0.0734 falling within the top 5%).

Next, we fed the 1000-bp sequence centered on InDel-313 into LOGOWheat to obtain a saturated mutagenesis profile. The results revealed significant variant scores for multiple SNPs within the Abscisic Acid (ABA)-responsive elements (ABRE, TACGTG), accompanied by a notable peak (Fig. 3C).

Case study 2: Discovering high-impact sites within the promoter region of wheat gene VRN1

The VERNALIZATION1 (VRN1) locus plays a pivotal role in the vernalization requirement of wheat, encoding an MADS-box transcription factor homologous to the product of the AP1 gene in Arabidopsis thaliana [57]. In common wheat, the homoeologous loci VRN-A1, VRN-B1, and VRN-D1 are situated on the long arm of chromosome 5, within the A, B, and D subgenomes. All three homoeologs are functional, collectively contributing significantly to the biodiversity observed in winter and spring forms of wheat.

Various dominant alleles of VRN1 loci are primarily associated with mutations in the promoter regulatory regions. A study analyzed alleles in the VRN-A1 promoter region of spring and winter T. monococcum accessions, revealing that mutations in the CArG-like motif may be the main determinants contributing to the spring type [52]. Additionally, the VRN-box (TTAAAAACCCCTCCCC) located directly 5′-adjacent to the CArG-box appears to have the greatest impact on vernalization sensitivity [58]. It has recently been established that even point mutations in the VRN-box are sufficient to transform T. durum into the spring type, and the combination of these mutations can modulate the response to vernalization [59].

In this section, three 1000-bp sequences were extracted from the VRN1 promoter region of the A, B, and D subgenomes of common wheat, with the VRN-box as the focal point. Subsequently, three deletion variants from T. monococcum and three SNPs from T. durum were mapped onto these three sequences. Using LOGOWheat, the variant scores of these six variants were calculated in the backgrounds of the different subgenomes, and a saturated mutagenesis profile was constructed for each of these three sequences (Fig. 5). The results indicate that these six variants exhibit relatively high variant scores in at least one subgenome background, although each variant has different variant scores in different subgenome backgrounds (Fig. 5B). The saturated mutagenesis maps show prominent peaks for both the VRN-box and the CArG-box. The A-tract of the VRN-box exhibits significant peaks in various subgenome backgrounds, while the C-rich region shows significant peaks only in the B subgenome. However, an SNP (T- > C) of T. durum in the C-rich region almost consistently demonstrates significantly high variant scores (with a variant score of 0.0450 in the A subgenome, very close to the threshold of 0.0493) (Fig. 5C).

Figure 5.

Discovering high-impact sites within the promoter region of the wheat gene VRN1. (A) Six variants in the promoter region of VRN1 contributing significantly to the biodiversity observed in winter and spring wheat forms. This region contains two motifs: the VRN-box, which includes an A-tract and a C-rich segment, and the CArG-box. (B) The variant scores for these six variants. (C) Saturated mutagenesis profiles of these variants in the contexts of different subgenomes.

Extracting variants with dual identities by incorporating evolutionary conservation information

Natural populations have undergone thousands of years of evolution, accumulating longstanding variation that affects crops’ response to various environmental conditions. This reservoir of variation provides an unparalleled resource for exploring functional variants. The two case studies above demonstrate that LOGOWheat can help identify causal variations and functional motifs. Additionally, it is well known that evolutionarily constrained noncoding regions in the genome often possess significant regulatory functions. In this section, we propose a method that combines de novo sequence–based chromatin effect predictions with evolutionary conservation information to extract natural variations with dual identities from natural populations (Fig. 6A).

Figure 6.

Extracting potential functional variations from the wheat population by incorporating evolutionary conservation information. (A) The strategy that combines sequence-based chromatin effect predictions with CNSs to identify dual-identity variants. (B) The distribution of variant scores from one million randomly selected variants across the whole genome, natural populations, gene promoter regions, and CNSs. (C) Three saturated mutagenesis profiles centered on naturally occurring SNPs in the population. The black border marks the SNP position, while the green background highlights CNSs. The title in the top-left corner provides the genomic coordinates of the SNP, the mutation type, the nearest gene, and its distance.

The natural populations used in this study included 827 Watkins landraces and 208 modern cultivars, with ~262 million high-quality SNPs [60]. The conserved noncoding sequences (CNSs) obtained from another of our studies (not yet published) identified around 780 000 CNSs through multiple-genome alignments of 15 Triticeae genomes. We investigated the distribution of variant scores for different types of genomic variations, using 1 million randomly selected variants from the whole genome, natural populations, gene promoter regions, and CNSs. The results demonstrate that, as expected, promoter sequences and CNSs tend to have higher variant scores compared to the whole genome (Fig. 6B). Interestingly, variant scores from natural populations show significant differences compared to the whole genome, exhibiting a bimodal distribution with a preference for lower variant scores. This implies that mutations are more likely to occur in less functional regions of genome. The axiom that mutations occur randomly with respect to their consequences has been a fundamental principle in biology and evolutionary theory since the first half of the twentieth century. From this perspective, this observation is believed to result solely from selection acting on these random mutations. However, recent research has proposed another explanation: epigenome-related mutation bias reduces the occurrence of harmful mutations in A. thaliana, challenging the longstanding paradigm of mutation randomness [61].

Finally, we present three saturated mutagenesis profiles centered on natural variations in the population, all of which possess dual identities: exhibiting higher variant scores and being present on CNSs (Fig. 6B). We found that these CNSs accumulate a large number of high-impact sites and exhibit significant peaks, implying the presence of functional motifs. We believe that these dual-identity SNPs, as reliable candidate functional sites, are necessary and valuable for interpreting their genetic basis and utilizing them for crop improvement.

Discussion

Natural populations have accrued abundant genetic and phenotypic diversity over thousands of years of evolution. Mapping the QTLs and ultimately pinpointing the causal quantitative trait nucleotides underlying complex traits are essential for the genetic improvement of crops. Advances in epigenomic sequencing technologies and deep learning techniques in NLP and CV have enabled us to curate a comprehensive epigenomic map of wheat and develop a deep learning framework called LOGOWheat to decode epigenetic regulatory codes.

Compared to classical CNN architectures, LOGOWheat achieves higher predictive performance with significantly fewer parameters (~1.6 million) than DeepSEA (~52 million) and even the simpler simpleCNN (~4 million). This reflects the advantage of pretrained models, indicating that unsupervised learning during the model’s pretraining stage on the raw reference genome has successfully improved downstream tasks of predicting epigenetic states.

Although LOGOWheat has successfully predicted the epigenetic states of wheat, it is currently limited to seven chromatin feature categories, whereas human studies have identified 919 categories. This limitation is primarily due to the scarcity of wheat epigenetic data. We believe that the increasing availability of wheat epigenetic data in the future will gradually address this shortfall. Additionally, continuous advancements in deep learning techniques, such as recent updates to MoD [62], life2vec [63], and Infini-attention [64], have improved the performance of Transformer models, which holds promise for enhancing chromatin state classification tasks.

Using the perturbation of LOGOWheat prediction results caused by sequence alterations, we defined a variant score to quantify the impact of variants on regulatory functionality. We introduced two main functions provided by LOGOWheat: prioritizing causal variants and discovering high-impact sites. These functionalities were illustrated and demonstrated using two case studies. Whether near the drought-resistant gene TaSNAC8-6A or in the promoter region of the vernalization gene VRN1, LOGOWheat successfully identified validated causal variant sites and functional motifs. Notably, in addition to these already published functional variants, the saturated mutational profile also reveals other variants and peaks with high variant scores (Figs 4C and 5C).

Figure 4.

Prioritizing functional variants proximal to the wheat gene TaSNAC8-6A. (A) The VCF file shows genome coordinates and base mutation types for 43 SNPs and 8 InDels. (B) The line graph shows the 51 variants arranged in descending order by their variant scores, with the functionally validated natural variant InDel-313 having the highest variant score. (C) The heatmap displays a saturated mutagenesis profile centered on the InDel-313 within a 100-bp sequence. In the upper half of the heatmap, deeper shades of red indicate higher variant scores for each base mutation type, while white represents a variant score of 0 for the base in the reference sequence, and the blue outer border signifies a significant deviation from random expectation. The lower half features a line graph showing cumulative variant scores for all mutation types at each base position, with the red dashed line indicating positions with scores significantly higher than random expectation. The black dashed line marks the position of InDel-313 within the sequence, and the green background highlights the ABRE motif.

Natural variants present in populations represent only the tip of the iceberg of the theoretical variation space, despite existing for tens of millions of years. Genome editing and de novo sequence design have significantly broadened the potential variation space at rates much faster than natural evolution. Our saturated mutational profile serves as a guide for precise genome editing, holding great promise for addressing challenges posed by current and emerging agricultural paradigms.

The continuous harnessing of natural alleles is essential for discovering new targets and increasing the predictability of new-to-nature design [9]. The approach proposed in this study combines de novo sequence–based chromatin effect predictions with evolutionary conservation information to obtain dual-identity variants, which greatly reduces the candidate space for variants to be verified. We believe that decoding the genotype–phenotype associations of these dual-identity variants will be necessary and valuable in the future.

Whether it’s prioritizing natural variants in populations or finding high-impact sites from sequences for genome editing, we anticipate that LOGOWheat will accelerate the identification of functional variants and crop improvement.

Materials and methods

Benchmark datasets

Raw sequencing data for ChIP-seq and ATAC-seq samples (with accession numbers starting with SRR, ERR, or CRR) were obtained from the NCBI Sequence Read Archive (SRA) (https://www.ncbi.nlm.nih.gov/sra) and CNCB (https://www.cncb.ac.cn/) databases. The selection criteria were as follows: the organism must be T. aestivum, the sequencing platform must be Illumina, and the library layout must be paired-end. Each dataset was meticulously curated to remove inaccurate comments and identify the specific investigated factor (e.g. histone modification mark) and experimental type (such as ChIP or ATAC) based on the metadata and original publications.

We followed the ChIP-seq and ATAC-seq data analysis protocols endorsed by the Encyclopedia of DNA Elements (ENCODE) project to construct a computational pipeline for uniformly processing the raw sequencing data, which included quality control, read mapping, and peak calling. Raw data in SRA format were converted to FASTQ format using the “fastq-dump” command from the SRA Toolkit (version 3.0.0) with the “--split-files” option for paired-end reads. Sequencing reads were cleaned using Fastp (version 0.23.2), which removed bases with low quality scores (parameters: -q 20), detected and removed potential sequencing adapters (parameters: --detect_adapter_for_pe), and filtered out short reads (parameters: -l 20) [65]. The cleaned reads were then mapped to the International Wheat Genome Sequencing Consortium (IWGSC) reference sequence (version 1.0) using Bowtie2 (version 2.4.5) with the parameters “bowtie2 -q --no-unal --sensitive -S” [66]. Aligned reads were sorted using Samtools (version 1.6) [67], and redundant reads and Polymerase Chain Reaction (PCR) duplicates were marked using Picard (version v2.27.4; http://broadinstitute.github.io/picard/) with the parameters: --VALIDATION_STRINGENCY LENIENT --ASSUME_SORTED true --REMOVE_DUPLICATES false.

To ensure homolog specificity and accuracy for subsequent analyses, the mapping results were stringently filtered with “samtools view -q 20 -F 1804 -f 0x2” as follows: (i) removing reads with mapping quality <20 to ensure mostly uniquely mapped reads; (ii) removing all supplementary alignments and duplicated reads mapped to the same position (SAM flag = 1804); and (iii) keeping only correctly mapped read pairs.

Peak calling was performed with MACS2 (version 2.2.7.1) using the parameters “-g 14e9 --nomodel -q 0.05.” Duplicated reads were not considered (“--keep-dup = 1”) during peak calling to achieve better specificity. To identify narrow peaks in ATAC-seq data, the parameters “--shift -100 --extsize 200 --call-summits” were used. For broad histone modification marks, broad peaks were identified by enabling the “--broad” parameter in MACS2. To mitigate experimental noise from different laboratories, antibody lots, and reagent batches, only peaks present in at least 20% of the samples for each epigenetic feature were retained and merged for the chromatin feature prediction task.

Finally, to prepare the training samples for fine-tuning the model, the IWGSC reference genome (version 1.0) was partitioned into 200-bp bins, which served as the core region for each sample. An additional 400-bp flanking region was added on both sides, resulting in a 1000-bp context sequence to provide extra contextual information. Each core 200-bp bin was labeled as 1 if more than half of the bin overlapped with a peak region; otherwise, it was labeled as 0. Only 200-bp bins containing at least one chromatin feature were used.

Model architecture and training

In this study, we constructed and trained four deep learning models: simpleCNN, DeepSEA, LOGOWheat(3-mer), and LOGOWheat(5-mer). The simpleCNN and DeepSEA are built on the CNN framework, while LOGOWheat(3-mer) and LOGOWheat(5-mer) are based on the Transformer architecture.

CNN is a common deep learning architecture widely used in image recognition. Its basic layers include convolutional layers, pooling layers, and fully connected layers. For simpleCNN, we used a single convolutional layer with 128 filters of size 8, followed by a max-pooling layer and a fully connected layer containing 128 neurons. Before the fully connected layer, a dropout rate of 50% was applied. For DeepSEA, we utilized its default parameters, which include three convolutional layers with 320, 480, and 960 filters of size 8, followed by a fully connected layer with 925 neurons. Both simpleCNN and DeepSEA take one-hot encoded sequences as input and use the sigmoid activation function to output likelihood values of various chromatin states. During training, the loss function (binary cross-entropy) is optimized using stochastic gradient descent with learning rates adapted via Adam.

In contrast to the semantically weaker representation of one-hot encoding, LOGOWheat(3-mer) and LOGOWheat(5-mer) tokenize sequences using k-mers to increase information content, with k set to 3 and 5, respectively. The higher the value of k, not only the greater the information content but also the greater the computational resources required. Token embeddings, with an embedding size of 128, are combined with position embeddings and then input into the Transformer encoder network. LOGOWheat uses two identical layers of Transformer encoders, each layer comprising a multihead self-attention layer with eight heads and a position-wise fully connected feed-forward layer with 128 neurons.

In the pretraining stage, we use the MLM task to define self-supervised loss. Specifically, we randomly mask 15% of k-mer tokens for prediction, where 80% are substituted with [MASK], 10% are replaced by a random token from the vocabulary, and the remaining 10% are left unchanged. During the fine-tuning stage, we append a classification layer with a sigmoid activation function after the Transformer encoder, transforming the [CLS] representation learned from the last hidden layer into likelihood values of chromatin states. The loss function is calculated using binary cross-entropy.

Both deep learning frameworks used in this study were implemented using the TensorFlow platform and trained on an Nvidia Tesla V100s 32G Graphics Processing Unit (GPU).

Scoring variants and in silico saturated mutagenesis analysis

To calculate the predicted chromatin effects of variants using the LOGOWheat model, we extracted the 1000-bp context sequence centered on each SNP from the wheat reference genome, ensuring the variant was positioned at the 500th nucleotide. We then constructed pairs of 1000 bp sequences for both the reference and alternative alleles at each variant position. These sequences were input into LOGOWheat to compute the probabilities for all chromatin features. The variant score was defined as the Euclidean distance between the predicted results of the reference and alternative sequences.

To identify informative sequence features, we conducted in silico saturated mutagenesis scanning. This involved evaluating the effect of mutating each base of the input sequence (four substitutions per base) on chromatin feature predictions. The sum of the variant scores for all mutation types at each base position represented the importance of that base position.

To determine whether a given variant score exceeded random expectation, we randomly sampled 1 million 1000-bp sequences from the wheat reference genome and calculated variant scores for these SNPs to generate an empirical background distribution. The significance threshold for variant scores was defined as the top 5% among these 1 million variants.

Key Points

• We present a comprehensive epigenomic map of wheat, meticulously curated and uniformly processed from public datasets, covering 6.33% of the whole genome.

• A deep learning–based tool, LOGOWheat, is proposed after evaluating multiple model architectures, which is designed to predict seven epigenetic features, including chromatin accessibility and six histone modifications.

• We apply LOGOWheat to assess the regulatory effects of noncoding variants and use two case studies to demonstrate its main functions: prioritizing causal variants and constructing a saturated mutagenesis map in silico to discover high-impact sites or functional motifs.

• A method that can combine de novo sequence–based chromatin effect predictions with evolutionary conservation information to obtain dual-identity variants is proposed, which greatly reduces the candidate space for variants to be verified.

Funding

This work was supported by the Research Grants for Postdocs Settle in Shenzhen from Shenzhen Government to B.S.

Data availability

LOGOWheat (http://logowheat.cn/) is freely available to all users. The peaks files of all chromatin features, model weight file, and scripts are available at https://github.com/mrklp/LOGOWheat.

Author contributions

L.K. designed this work, coded the program, performed all computational experiments, and wrote the manuscript. L.K., H. C., K.Z., and B.S. revised the manuscript together. K.Z. designed and developed web application. All authors discussed the results and commented on the manuscript.