Benchmarking transcription factor binding site prediction models: a comparative analysis on synthetic and biological data

Abstract

Transcription factors (TFs) are essential regulatory proteins controlling the cellular transcriptional states by binding to specific DNA sequences known as transcription factor binding sites (TFBSs) or motifs. Accurate TFBS identification is crucial for unraveling regulatory mechanisms driving cellular dynamics. Over the years, various computational approaches have been developed to model TFBSs, with position weight matrices (PWMs) being one of the most widely adopted methods. PWMs provide a probabilistic framework by representing nucleotide frequencies at every position within the binding site. While effective and interpretable, PWMs face significant limitations, such as their inability to capture positional dependencies or model complex interactions. To address these, advanced methods, like support vector machine (SVM)–based, and deep learning (DL)–based models, have been introduced. Leveraging human ChIP-seq data from ENCODE, we systematically benchmarked the predictive performance of PWM, SVM-, and DL-based models across different scenarios. We evaluate the impact of key factors such as training dataset size, sequence length, and kernel functions (for SVMs) on models’ performance. Additionally, we explore the impact of synthetic versus real biological background data during model training. Our analysis highlights strengths and limitations of each approach under different conditions, providing practical guidance for selecting and tailoring models to specific biological datasets. To complement our analysis, we present a comprehensive database of pretrained SVM models for TFBS detection, trained on human ChIP-seq data from diverse cell lines and tissues. This resource aims to facilitate broader adoption of SVM-based methods in TFBS prediction and enhance their practical utility in regulatory genomics research.

Introduction

Transcription factors (TFs) are key regulatory proteins that orchestrate different cellular processes, including transcriptional regulation, cellular differentiation, and development [1–3]. TFs execute their regulatory function by binding to short DNA sequences called transcription factor binding sites (TFBSs). TFBSs, often also referred to as ‘motifs’, are distributed across different genomic regions, encompassing promoters, enhancers, silencers, or insulators, and even within coding sequences [4–7]. The precise configuration of TFBS sequences governs TFs’ regulatory function [8, 9], highlighting the significance and importance of identifying putative binding sites for elucidating the intricate regulatory mechanisms underlying the cellular dynamics.

Different experimental methods have been developed to identify TFBSs both in living cells (in vivo) and controlled environments using purified components [10]. Earlier assays, such as the electrophoretic mobility shift assay [11] and footprinting [12] focused on small datasets of target sequences, while high-throughput techniques like protein binding microarrays (PBM), SELEX, and ChIP [13–16] enabled broader analyses.

ChIP assays [15, 16] transformed TFBS research, allowing genome-wide identification of bound regions in vivo. In ChIP, TF–DNA complexes are cross-linked with formaldehyde, fragmented, and immunoprecipitated using TF-specific antibodies. Following cross-link reversal, the DNA fragments are amplified and analyzed via microarray hybridization (ChIP-on-Chip [15]) or sequencing (ChIP-seq [16, 17]). Fragments are then mapped to the genome, where peak-calling algorithms [18–20] identify binding regions by detecting significant enrichment compared to control samples. ChIP-seq produces large datasets, often covering tens of thousands of genomic regions, each ranging from hundreds to thousands of nucleotides. These data enable the inference of potential TFBS locations, establishing ChIP-seq as the current in vivo ‘gold-standard’ method. However, it has limitations: ChIP-seq may detect indirect binding from other TFs, include false positives due to antibody quality, and yield low-resolution data with excess flanking nucleotides [21–23]. ChIP-exo [24] addresses this by using a lambda exonuclease to trim excess flanking nucleotides. Since most TFs bind to target sequences in open chromatin regions, methods like DNase-seq [25] can be used to identify potential TFBS in vivo, particularly when the target TF is unknown.

Databases like ENCODE, Cistrome, and GTRD [26–28] offer extensive collections of ChIP-seq datasets across various organisms, tissues, and cell types. ENCODE [26] provides a collection of ChIP-seq and DNase-seq data on human functional elements from different tissues and cell types.

Despite these resources, identifying precise binding sites within sequences remains challenging [29, 30]. Motif discovery algorithms [23, 31, 32] help bridge this gap by analyzing sequence datasets to learn TF binding affinities and identify potential TFBS locations. The learned binding affinities are then encoded in computational models representing the discovered TFBSs. Position weighted matrices (PWMs) [33, 34] are robust and interpretable models quantifying the contribution of each nucleotide at each motif position. Despite their effectiveness and widespread adoption, PWMs have some limitations. They assume positional independence within TFBS motifs, accommodating different binding site configurations [35] but potentially increasing susceptibility to noise, which can result in false-positive predictions of binding sites [23]. Alternative models have been proposed, aiming to address these limitations.

Support vector machines (SVMs) [36] have been employed to address different challenges in bioinformatics [37], including motif discovery and modeling. SVM-based motif models [23, 38] leverage k-mers frequencies within input sequences to create a classification space, distinguishing bound and unbound sequences. While studies demonstrated that SVM-based models can outperform PWMs in predictive tasks [39, 40], they come with limitations. SVM-based models are limited to capturing short k-mers (10–12 bp), and their performance heavily relies on training data quality [23].

Deep learning (DL)–based methods gained prominence in computational biology [41, 42] due to their ability to capture complex patterns. Convolutional neural networks (CNNs) have been applied to model TF–DNA interactions by learning sequence features underlying TFBSs, as demonstrated by tools like DeepBind [43] or Basset [44]. Recently, Transformer-based architectures [45] have become state of the art in various domains, including genomics, thanks to their ability to model long-range dependencies. Nucleotide Transformer [46] and DNABERT [47] have shown strong performance across a range of sequence-based predictive tasks.

However, DL-based models require large training datasets and substantial computational resources. Their limited interpretability [48] also makes it challenging, deriving biological insights. Transformers further increase computational demands due to extensive pretraining and their quadratic scaling with sequence length. In contrast, PWM- and SVM-based models offer more scalable and hardware-independent solutions. PWMs provide an interpretable framework facilitating biological interpretation.

Motif discovery algorithms face challenges when applied to large datasets, including scalability issues and noise susceptibility. To mitigate these issues, motif discovery tools are run on subsets of the input sequences [49, 50]. While this technique can reduce computational demands, it may produce inaccurate or biased models [51]. Another technique to minimize noise in ChIP-seq sequences involves restricting the width of the input sequences, focusing on regions surrounding the peak summit [52]. However, this method may overlook TFBS occurrences distal from the peak summit. For methods employing SVMs, an additional factor is the selection of the kernel function, as it determines the model’s ability to learn the patterns defining TFBSs [53].

Several studies have evaluated the performance of TFBS models [54–57], but many rely on using synthetic datasets [54] or simpler genomes [55] in their evaluations, limiting their applicability to complex regulatory landscapes.

In this paper, we compare the predictive performance of PWM, SVM- and DL-based motif models (sections Position Weight Matrix Motif Models, Support Vector Machine–Based Motif Models, and Deep Learning–based Models) trained using four widely used motif discovery and sequence analysis tools: MEME [58–60], STREME [61], LS-GKM [62], and Nucleotide Transformer [46]. Models were trained on human ChIP-seq data from ENCODE [26], focusing on TFs representing all DNA-binding domains (DBDs) [1] (section Analysis Workflow for Training and Evaluating Position Weight Matrix–, Support Vector Machine–, and Deep Learning–Based Motif Models). We corroborate the impact of training dataset size and sequence width, kernel functions and the impact of background data on models’ accuracy, comparing synthetic (shuffled input sequences) and biologically feasible (DNase-seq regions) backgrounds (section Benchmarking Position Weight Matrix, Support Vector Machine, and Deep Learning Motif Models’ Performance). We further explore the impact of dataset imbalance on models’ performance, simulating a ‘needle in the haystack’ scenario [63].

While SVM-based motif models potentially offer notable advantages, their adoption has been hindered by the lack of accessible pretrained models, unlike PWMs, which benefit from established resources. To address this gap, we present a comprehensive database of SVM-based motif models trained on human ChIP-seq data obtained on diverse cell lines and tissues from ENCODE (section Discussion).

Through these comparisons, we deliver valuable guidance for selecting the most suitable motif model based on data characteristics and research goals while introducing a resource enhancing the accessibility and utility of SVM-based models.

Position weight matrix motif models

PWMs [33, 34] are widely recognized as the predominant models for encoding TFs’ binding preferences. PWMs provide an additive framework describing the contribution of each binding site position to TF-DNA binding events. PWMs achieve this by tallying the frequency of each nucleotide at every TFBS position, derived from computing an ungapped alignment profile constructed using the sequences identified as potential motifs by motif discovery algorithms [23]. PWMs are statistically evaluated using metrics, such as nucleotide conservation or information content [64]. PWMs are often visualized as sequence logos [65], where the height of each nucleotide corresponds to its information content., PWMs have become the standard model for representing and visualizing TFBS motifs, with databases like JASPAR [66] or HOCOMOCO [67], providing annotated PWMs covering a broad spectrum of known binding sites. However, PWMs come with limitations. PWMs assume positional independence within the motif, potentially causing the identification of false-negative or false-positive TFBSs [23, 35]. Despite this, several algorithms have been proposed to discover TFBS in genomic sequences, constructing PWMs to represent the identified motifs. Methods utilizing enumeration or alignment of motif candidate sequences have gained attention due to their balance between computational efficiency and model interpretability. While more complex methods, such as those leveraging deep neural networks (DNNs) [42] can yield PWMs, they come with certain trade-offs between computational requirements for training and model interpretability. On the other hand, enumerative and alignment-based methods maintain interpretability while offering a more computationally scalable framework [68].

Enumerative motif discovery for position weight matrix training: an example with STREME

The recently introduced algorithm STREME [61] employs an enumerative-based approach to discover TFBS motifs in DNA sequences (Supplementary File Section 1). Enumeration-based motif discovery methods search for overrepresented sequence patterns within the input sequence dataset, relative to a background set of sequences [23]. By counting the approximate frequency of each sequence fragment in the foreground (positive) and background (negative) datasets, these methods evaluate whether the observed differences in occurrence are statistically significant, enabling the discovery of enriched motifs. Enumerating subsequences and comparing their frequencies can be computationally demanding. Therefore, many enumerative algorithms employ optimized data structures like suffix trees (STs), to enhance this process. STREME leverages STs to enhance computational efficiency and perform approximate sequence matching to discover enriched patterns. By constructing an ST from positive sequences, STREME identifies overrepresented seed words, subsequently extended to form motif candidates. This seed-based approach reduces search space, thereby mitigating computational complexity. A key advantage of STREME is its ability to discover motifs of different lengths in a single ST traversal. To statistically evaluate seed word enrichment in foreground sequences, STREME uses either Fisher’s exact test or the binomial distribution, depending on dataset composition (see Supplementary File Section 1). Once statistically significant words are identified, STREME counts approximate matches to derive PWMs. By using significant motif candidates, STREME constructs alignment profiles and iteratively refines them, assessing their statistical significance throughout the process. Profile refinement ensures the selection of the most significant alignments, improving PWMs’ accuracy.

Alignment-based motif discovery for position weight matrix training: An example with MEME

MEME [58–60] stands as a fundamental tool for motif discovery, employing an alignment-based approach to discover TFBS motifs (Supplementary File Section 2). Alignment-based methods construct alignment profiles by selecting motif candidate sequences from the input dataset and then evaluating them using nucleotide conservation, information content, or profile’s statistical significance [23, 32]. However, finding the best alignment according to the chosen metric is computationally challenging, since it implicitly requires combining all possible subsequences from the input dataset in the alignment profile. Heuristic techniques, such as expectation maximization (EM) [58], help overcome this limitation by constructing the best alignment without exploring the entire solution space. MEME employs EM to construct PWMs while exploring the solution search space. Unlike methods using greedy approaches [69], which may focus on partial solutions, MEME ensures a more comprehensive search. For a motif of length M, MEME initializes an alignment profile by selecting one subsequence of length M from each input sequence. The initial profile is then iteratively refined using an EM. In the E-step, MEME scores each subsequence of length M based on its similarity to the current profile. During the M-step, MEME assigns a weight to each subsequence in the profile proportional to the scores computed in the E-step. Low-scoring subsequences are then replaced with better-fitting ones, updating the profile. The refinement process continues until convergence (i.e. the profile is no longer updated) or after a predefined number of iterations. MEME integrates various background models to assess the profile significance of each constructed profile, measuring the likelihood of obtaining a profile with similar information content by chance.

Support vector machine–based motif models

SVMs successfully address biological challenges, including learning TFBSs’ structure [23, 38]. By analyzing k-mers (DNA fragments of length k) frequencies in positive and negative datasets, SVM-based models train kernels to capture structural features of binding sites. Input sequences are partitioned into k-mers, and their frequencies are used to create feature vectors serving as inputs for training SVM kernels. The choice of kernel function is critical, as it determines how SVMs represent TF binding preferences [38]. Two primary methods are typically employed: (i) binding sites are represented as a list of k-mers, each with an associated weight indicating their contribution to the motif, and (ii) as a list of support vectors characterizing positive and negative datasets. One key advantage of SVMs over PWMs is their ability to also capture higher-order dependencies between neighboring nucleotides (k-th order dependencies) [23, 38]. For visualization purposes, SVM-based models are often reduced to PWMs, obtained by aligning the most informative k-mers, though this conversion often results in information loss. Various kernels have been proposed to better capture motif structures in DNA sequences. Early methods, like the spectrum kernel [70], count exact k-mer matches, while weighted versions [39] incorporate positional information. However, these are limited to short k-mers (~10 bp) and struggle with large datasets. More flexible approaches, such as mismatch, wildcard, and dinucleotide mismatch kernels [71–73], allow for mismatches and local dependencies but still face challenges in modeling longer motifs and avoiding overfitting due to sparse feature vectors with large k. To address these limitations, gapped k-mers [74] allow degenerate positions within motifs, enabling scalable analysis of longer TFBSs. Gkm-SVM [40] leverages gapped k-mers frequencies to train SVMs for TF-DNA interaction prediction but lacks scalability beyond ~10 000 sequences. LS-GKM [62] extends this framework with improved efficiency and additional kernel functions, making it suitable for large datasets and more complex motif learning.

LS-GKM

LS-GKM [62] efficiently analyzes large-scale sequence datasets, leveraging gapped k-mers to train SVM-based motif models (Supplementary File Section 3). By comparing datasets of bound (positive) and unbound (negative) sequences, LS-GKM generates feature vectors by counting gapped k-mer occurrences in both datasets. To compute gapped k-mer frequencies, LS-GKM builds a k-mer’s tree [40], whose height is k with a total of leaves, each corresponding to a k-mer. To derive feature vectors for each sequence, LS-GKM traverses the tree using a depth-first search. LS-GKM implements parallel searches on the k-mer tree by dividing it into smaller subtrees, processing each subtree independently. The resulting feature vectors are used to train the gapped k-mer kernel, which computes support vectors to distinguish between bound and unbound sequences. To ensure scalability with large datasets, LS-GKM integrates the gapped k-mer kernel with decomposition techniques for SVM training [62]. To compute the SVM hyperplane, the feature vectors for each sequence are mapped to the normalized frequencies of distinct gapped k-mers. LS-GKM offers different kernel functions to compute both linear and nonlinear hyperplanes and weight the training features (Supplementary File Section 3). Among these, the gkmrbf kernel applies a radial basis function (RBF) to the space of gapped k-mer frequency vectors. The center weighted gkm (wgkm) kernel assigns different weights to gapped k-mers based on their distance from the sequence center, reflecting the observation that ChIP-seq signals are often concentrated in the central regions of peaks. The wgkmrbf kernel combines the features of both the gkmrbf and wgkm kernels, blending their strengths for flexible and accurate modeling.

Deep learning–based models

DNNs have been widely used in computational biology [23, 41, 42], due to their ability to learn complex, nonlinear patterns from large-scale omics data. Convolutional neural networks (CNNs) have been used to model TF-DNA interactions by treating nucleotide sequences as structured, multi-channel inputs [42]. This representation enables TFBS prediction to be addressed as a binary classification task. Models such as DeepBind [43] or Basset [44] have shown robust performance in motif discovery, though they are limited by data quality and their constraints in modeling long-range dependencies.

Transformer architectures have recently been employed in genomic analysis [45]. By leveraging self-attention mechanisms to capture long-range dependencies in DNA sequences, transformers overcome the locality constraints of CNNs. Transformer-based models can achieve state-of-the-art performance in predicting different regulatory elements, including TFBSs, by leveraging their ability to integrate both short-range motif information and long-range genomic interactions over kilobase-scale inputs.

This advancement aligns with a broader shift in machine learning toward large-scale architectures pretrained on massive, unlabeled datasets and adaptable to diverse downstream tasks [75]. In natural language processing, models like BERT and GPT [76] leverage masked language modeling [77] to learn contextual token embeddings by predicting randomly masked tokens from their surrounding context. For DNA, the attention mechanism enables transformers to assign importance to nucleotides at varying distances, allowing them to model the hierarchical and distributed nature of gene regulation. As a result, transformers not only improve predictive accuracy but also provide a flexible framework for analyzing sequence function in complex genomic landscapes. For example, Nucleotide Transformer (NT) [46] has demonstrated strong performance in predicting regulatory elements such as TFBSs.

Nucleotide transformer

NT [46] is a family of large-scale language models pretrained on genomic sequences using masked language modeling (MLM) (Supplementary File Section 4). Based on the BERT architecture [47], NT employs an encoder-only Transformer to learn contextual representations of DNA sequences. Sequences are first tokenized into 6-mers, transformed into vector embeddings, and augmented with positional encodings to preserve nucleotide order. Embeddings are then processed through stacked transformer layers, where self-attention mechanisms capture both local and long-range sequence dependencies.

MLM pretraining involves randomly masking a subset of tokens and training the model to predict them using the surrounding context. Through this process, NT captures biologically meaningful features from sequences. The model has demonstrated strong performance across a range of genomics tasks, including the prediction of TFBSs. NT models range from 50 million to 2.5 billion parameters and have been trained on diverse genomic data. For our benchmark study, we used NT-500 M pretrained on the human reference genome.

Analysis workflow for training and evaluating position weight matrix–, support vector machine–, and deep learning–based motif models

In this study, we benchmarked models’ performance in predicting TFBS by curating a diverse set of human TF ChIP-seq peak data from ENCODE. This evaluation framework incorporated synthetic and biologically feasible background data, enabling a comprehensive assessment of model robustness across different genomic contexts. Building on this foundation, we employed a systematic approach integrating model training and evaluation to assess the robustness and efficacy of PWM- and SVM-based motif models in TFBS prediction. We analyzed each model’s adaptability across diverse dataset conditions, accounting for the complexities and potential noise in biological data. Our evaluation focused on key aspects of model performance, including robustness, generalizability, and reliability, under diverse training and testing scenarios, providing a comprehensive assessment of each model’s predictive capability.

ChIP-seq sequence benchmark datasets

To benchmark models’ performance, we downloaded optimal IDR thresholded ChIP-seq peaks from ENCODE (snapshot November 2023) for 59 distinct transcription factors (TFs). To strike a balance between comprehensive TF representation and computational efficiency, we selected either a representative pair or the single available TF corresponding to each DNA-binding domain proposed in [1] (Table 1).

Table 1.

Summary statistics of the 59 ChIP-seq experiments used in the benchmark study. For each experiment, the table reports the target transcription factor, the DNA binding domain of the TF, the cell line used to perform the experiment, the number of peak sequences contained in each experiment, the total length (in bp) of peak sequences, the peaks average and median lengths, and the GC content

DNA binding   domain	Experiment	Target	Biosample	Number of   peaks	Total size (bp)	Average   length (bp)	Median (bp)	GC content (%)
Arid-bright	ENCFF477OUI	ARID5B	Cell line—HepG2	58 725	20 826 315	354.6	436	50.42
	ENCFF536DJU	ARID3A	Cell line—HepG2	22 098	7 988 472	361.5	376	45.01
At-hook	ENCFF614IHQ	HMGA2	Cell line—WCT11	30 003	11 102 945	370.1	396	32.7
Bed-zf	ENCFF753UAX	ZBED1	Cell line—GM12878	14 705	7 276 487	494.8	504	47.61
Bhlh	ENCFF445XYV	BHLHE40	Cell line—GM12878	49 176	17 203 875	349.8	404	55.94
	ENCFF713RHL	MAX	Cell line—A549	43 747	19 864 596	454.1	524	56.05
Bzip	ENCFF149ZWX	CEBPB	Cell line—A549	33 140	13 761 267	415.2	456	44.39
	ENCFF658HLG	JUND	Cell line—A549	21 652	5 720 961	264.2	270	55.56
C2h2-zf	ENCFF761YYL	REST	Cell line—K562	46 684	16 047 649	343.8	410	51.06
	ENCFF796WRU	CTCF	Cell line—GM12878	44 217	8 003 778	181	216	51.75
	ENCFF174JHS	PATZ1	Cell line—HepG2	44 172	15 021 066	340.1	400	58.25
C2h2-zf-homeodomain	ENCFF060IPX	ZEB1	Cell line—HepG2	14 122	7 181 581	508.5	530	53.86
Cbf-nf-y	ENCFF166QJY	NFYA	Cell line—HepG2	17 610	7 373 260	418.7	444	58.75
Csl	ENCFF762NCU	RBPJ	Cell line—HepG2	29 838	14 047 851	470.8	540	55.97
Cut-homeodomain	ENCFF919ZSN	ONECUT1	Cell line—HepG2	36 880	11 422 683	309.7	340	44.28
Cxxc-at-hook	ENCFF644EPI	KMT2A	Cell line—HepG2	23 015	11 418 906	496.2	580	58.71
E2f	ENCFF786NAO	E2F6	Cell line—A549	18 252	8 081 468	442.8	480	61.03
	ENCFF115KRO	E2F1	Cell line—MCF-7	34 657	21 076 131	608.1	650	58.26
Ebf1	ENCFF996MFL	EBF1	Cell line—GM12878	38 808	11 950 492	307.9	320	48.3
Ets	ENCFF889XXJ	GABPA	Cell line—K562	33 640	24 148 852	717.9	750	54.55
	ENCFF479HJF	ELF1	Cell line—GM12878	34 707	13 838 703	398.7	456	53.46
Ets-at-hook	ENCFF600JNK	ELF3	Cell line—HepG2	39 379	13 894 418	352.8	420	52.74
Forkhead	ENCFF125JED	FOXA1	Cell line—A549	33 937	8 882 303	261.7	280	43.92
	ENCFF605GXY	FOXK1	Cell line—HepG2	26 581	9 453 067	355.6	404	53.32
Gata	ENCFF409YKQ	GATA2	Cell line—HepG2	22 236	7 718 388	347.1	370	47.14
	ENCFF657CTC	GATA1	Cell line—K562	14 676	3 526 317	240.3	250	46.52
Gtf2i-like	ENCFF866OZW	GTF2I	Cell line—K562	23 275	9 125 007	392.1	404	51.22
Hmg-sox	ENCFF453YPR	SOX13	Cell line—HepG2	49 059	14,999,666	305.7	356	49.73
	ENCFF726SWQ	TCF7L2	Cell line—HepG2	36 726	15 758 780	429.1	450	52.93
Homeo-domain	ENCFF345BUG	PBX2	Cell line—K562	33 022	12 362 945	374.4	416	45.07
	ENCFF753OCA	HOXA3	Cell line—HepG2	40 285	17 383 257	431.5	476	57.22
Homeodomain-pou	ENCFF934JFA	POU2F2	Cell line—GM12878	27 932	8 295 289	297	320	52.84
Irf	ENCFF992KZC	IRF2	Cell line—HepG2	30 647	12 031 721	392.6	460	53.91
	ENCFF113VGD	IRF4	Cell line—GM12878	23 043	6 802 891	295.2	320	48.28
Mads-box	ENCFF884QQW	MEF2B	Cell line—GM12878	39 364	20 154 511	512	580	48.76
	ENCFF432PLY	SRF	Cell line—GM12878	18 489	6 785 804	367	396	46.54
Mbd	ENCFF788YHU	MBD2	Cell line—K562	15 309	5 970 715	390	416	58.19
	ENCFF812YDP	SETDB1	Cell line—HEK293	37 296	25 216 497	676.1	716	40.89
Myb-sant	ENCFF904DOZ	MYBL2	Cell line—HepG2	24 426	11 100 651	454.5	520	53.39
Nuclear-receptor	ENCFF361TSX	NR3C1	Cell line—A549	32 987	14 115 408	427.9	470	44.73
	ENCFF746GDG	NR2F1	Cell line—K562	36 225	15 737 646	434.4	496	50.01
Paired-box	ENCFF192WNJ	PAX5	Cell line—GM12878	24 349	6 825 221	280.3	304	53.41
Pipsqueak	ENCFF453UVV	LCOR	Cell line—HepG2	36 722	12 451 254	339.1	390	53.19
	ENCFF606UCO	LCORL	Cell line—HepG2	37 417	21 609 600	577.5	590	54.3
Rel	ENCFF687TJX	NFKB2	Cell line—HepG2	21 913	11 835 017	540.1	560	61.78
	ENCFF781YDP	RELA	Cell line—GM12878	17 781	6 440 289	362.2	376	47.41
Rfx	ENCFF782YIG	RFX1	Cell line—MCF-7	24 688	10 011 002	405.5	440	51.88
	ENCFF438ZEN	RFX5	Cell line—HeLa-S3	24 379	7 779 542	319.1	324	48.8
Runt	ENCFF346JDW	RUNX3	Cell line—GM12878	70 592	21 672 321	307	370	47.5
Sand	ENCFF365ETH	GMEB1	Cell line—K562	29 365	16 385 797	558	600	54.59
Smad	ENCFF232ZHD	SMAD4	Cell line—K562	36 210	18 183 986	502.2	540	54.14
	ENCFF489IWB	NFIB	Cell line—MCF-7	17 741	6 305 825	355.4	416	50.21
Stat	ENCFF186VMW	STAT1	Cell line—HeLa-S3	19 092	8 418 546	440.9	460	50.16
	ENCFF746VUX	STAT3	Cell line—HeLa-S3	17 089	5 619 944	328.9	336	42.72
T-box	ENCFF170IZO	TBX21	Cell line—GM12878	40 199	16 522 444	411	484	44.41
	ENCFF524ZER	MGA	Cell line—K562	6389	2 872 070	449.5	421	58.67
Tbp	ENCFF363NQY	TBP	Cell line—HeLa-S3	23 582	8 388 076	355.7	380	51.06
Tea	ENCFF501XJP	TEAD4	Cell line—K562	35 971	11 042 074	307	364	48.04
	ENCFF128AZO	TEAD1	Cell line—HepG2	35 097	11 400 581	324.8	376	48.84

For model training and performance evaluation, it is crucial to include negative sequences alongside positive ChIP-seq peak sequences. To address this, we employed two complementary strategies employing synthetic and biological background datasets. Supplementary File Section 5 describes the procedure employed to create the background datasets. Synthetic backgrounds were generated by shuffling the nucleotides of positive sequences while preserving first-order dependencies (Fig. 1a). This approach ensures that nucleotide pairs remain intact, thereby creating more biologically plausible backgrounds [78]. For biologically feasible background data, we used a comprehensive collection of DNase I hypersensitive sites (DNase-seq) derived from various tissue types [79]. To ensure that background and ChIP-seq sequences were distinct, we selected DNase-seq regions not overlapping, even partially, any of the peaks in the corresponding ChIP-seq dataset (Fig. 1a). To ensure consistent genomic contexts, we restricted DNase-seq regions to match chromosomes on positive peaks. To split positive and negative datasets into training and testing sets, we allocated all sequences on chromosome 2 to the test sets, leaving the rest for model training. The chromosome-stratified partitioning provides a large, representative test set for robust statistical assessment. Supplementary File Section 6 details the rationale and highlights its minimal impact on performance impact on performance variation. Synthetic and biological background datasets introduced two testing scenarios, enabling a nuanced evaluation of models’ performance. Synthetic background data test the models’ ability to distinguish true TFBS from artificially generated negative sequences. This approach creates a controlled environment emphasizing motif-recognize capability, while excluding the influence of complex genomic features. Conversely, biologically feasible background data simulate genomic regions with accessible DNA but not necessarily bound by the investigated TF. In this scenario, the challenge for the model is to distinguish between TF binding events from regions of open chromatin that lack TF binding. To further assess models’ predictive ability in identifying TFBS within large genomic contexts, we constructed an additional test dataset with a 10-times higher proportion of DNase-seq sequences compared to positive sequences. This scenario provides insights into models’ robustness under realistic genomic conditions, where biologically active but nonbinding regions are prevalent, resulting in class imbalance.

Figure 1.

Analysis workflow to train and evaluate PWM, SVM- and DL-based motif models. (a) Positive sequences are derived from ChIP-seq peaks, representing genomic regions enriched for transcription factor binding. Negative sequences are generated using two approaches: (i) synthetic construction by shuffling dinucleotides from the positive sequences to preserve local sequence composition, and (ii) selection from DNase-seq hypersensitive sites that do not overlap with ChIP-seq peaks, providing real-world background data. These datasets reflect diverse biological and computational scenarios for model evaluation. (b) Motif models are trained using PWM-based approaches, including STREME and MEME, SVM-based methods like LS-GKM, as well as DL-based methods, such as NT. PWMs capture nucleotide frequencies at each motif position, while SVM-based models leverage kernel functions to identify complex sequence patterns and positional dependencies. DL-based methods exploit self-attention mechanisms to learn long-range dependencies in DNA sequences. (c) The performance of each model is assessed using multiple metrics, including area under the receiver operating characteristic (AUROC), area under the precision–recall curve (AUPRC), and F1-score. The metrics measure a model’s ability to accurately distinguish true binding motifs from background sequences across varying conditions. (d) Key factors influencing model performance are systematically explored, including dataset size, sequence width, and kernel type for SVM-based models. This analysis identifies optimal configurations for each model and highlights their strengths and limitations across different data properties and biological contexts.

Model training and testing

Training was conducted independently for SVM and PWM models, with MEME/STREME used to train PWMs and LS-GKM for generating SVM-based motifs (Fig. 1b). Supplementary File Section 6 details the training procedure. Each model was evaluated across four test conditions to assess models’ robustness, adaptability, and susceptibility to overfitting. (i) Training and testing on ChIP-seq sequences as positives and synthetic negative datasets provide a controlled environment to establish a baseline for model performance in the presence of simple, random noise. (ii) Training and testing on ChIP-seq peak sequences as foreground and DNase-seq background data replicate a realistic biological setting, testing each model’s ability to function under complex, biologically relevant background conditions where TF binding might plausibly occur [80, 81]. (iii) Training on ChIP-seq sequences as positives and synthetic negatives and testing on ChIP-seq sequences and DNase-seq data is a technique often employed [81–83] and introduce a background complexity shift evaluating whether models trained on simpler data could adapt effectively to biologically complex contexts. (iv) Training on ChIP-seq peaks sequences as foreground and DNase-seq background sequences and testing on ChIP-seq sequences and synthetic data as negatives reverse the previous scenario, examining whether models trained in a complex biological context could retain accuracy and generalizability when evaluated against a simpler background. For tests involving biological background data, two scenarios were evaluated: (i) a balanced setup with a 1:1 ratio of positive to negative sequences and (ii) an imbalanced setup with a 1:10 ratio. The latter simulated real-world conditions, where unbound genomic regions outnumber bound regions. These test conditions provide a comprehensive assessment of each model’s generalizability, resilience to varying background complexities, and suitability for TFBS prediction in realistic genomic settings. To address the potential confounding of homologous sequences biasing models training and inflating their performance [84, 85], we assessed sequence similarity between training and test sets in all scenarios. Across the 59 datasets, we found that homologous sequences represented <1% in nearly all experiments, suggesting minimal risk of performance inflation due to homology (Supplementary File Section 7). After training, models were employed to score each test sequence and assess the likelihood of TF binding. For SVM-based models, scores were generated using LS-GKM’s gkmpredict function [62], while for PWMs generated by MEME/STREME, scoring was performed using FIMO [86] (see Supplementary File Section 8 for details). For NT, we used the output logits as scores for test sequences. Following sequence scoring with the trained models, we evaluated their predictive performance across different testing scenarios using multiple metrics (Fig. 1c).

Model evaluation criteria: the impact of dataset size, sequence width, and kernel functions

To comprehensively assess the predictive performance and robustness of SVM and PWM models, we performed models training across different experimental conditions (Fig. 1d). Each setting was chosen to explore key factors that may influence model accuracy and generalizability, such as the number of training sequences, sequences width, and the choice of kernel function, for SVM models. The first analyzed factor was training dataset size (Fig. 1d). Models were trained on datasets of varying sizes (500, 1000, 2000, 5000, 10 000, and the full dataset), prioritizing the most reliable sequences (peaks sorted by q-value and ChIP enrichment score). This analysis aimed to determine the effect of dataset size on model performance. While smaller datasets can reduce computational requirements, they may not capture the full binding sites complexity, potentially impacting models’ performance [49, 50]. On the other hand, larger datasets contain more information, but they may introduce noise from less informative sequences [87]. Next, the sequence width was systematically varied to evaluate its influence on model performance (Fig. 1d). Sequences of different lengths (50, 100, 150, 200, and full length) were extracted around the ChIP-seq peak summit and used for training and evaluation. This analysis illustrates how training sequences’ length influences the surrounding genomic context that may either support or interfere with accurate motif models’ training [52]. Shorter sequences may focus the model on the core binding site, reducing noise from surrounding regions, but could also omit important flanking information that might contribute to TF’s binding specificity [88]. On the other hand, longer sequences include flanking regions, potentially introducing noise, but provide a richer context for motif training. Additionally, for SVM-based models, the choice of sequence start position may impact their performance, as different starting points can produce distinct k-mer sets, potentially influencing performance. Although not studied in this study, this parameter warrants further investigation. Lastly, for SVM-based models, we compared different kernel functions available in LS-GKM (Supplementary File Section 3): these included the gapped k-mer kernel, k-mer frequency counts, the standard gkm kernel, the gkm kernel combined with an RBF, and the center-weighted variants of both the gkm and gkm-RBF kernels (Fig. 1d). Additionally, configurations better performing by tool and by individual experiment were evaluated to provide a broader perspective on model performance.

Model performance metrics

In our study, we defined true positives (TPs) as test sequences from ChIP-seq datasets that were correctly predicted by the model to be bound by the investigated TF. False positives (FPs) were sequences from the negative test dataset that the model incorrectly predicted as bound by the TF. Conversely, sequences in the positive dataset that were predicted as unbound were labeled as false negatives (FNs). Given the difficulty in accurately determining the total number of negative binding sites for each TF across the genome [89], we opted to exclude performance metrics that depend on true-negative (TN) counts [90]. As a result, we evaluated model performance using precision (), recall (), and F1-score (). Additionally, to further assess model performance, we computed the area under the precision–recall curve (AUPRC) and the area under the receiver operating characteristic curve (AUROC) for each model and test dataset (Fig. 1c).

Benchmarking position weight matrix, support vector machine, and deep learning motif models’ performance

In this section, we compare the performance of models trained using MEME, STREME, LS-GKM, and NT by analyzing key performance metrics, such as AUPRC, AUROC, and F1-score under multiple training and testing scenarios. We explore how these metrics evolve as experimental settings change, leveraging both synthetic and biologically feasible background training data.

By analyzing how varying training dataset size and background data affect model performance, we observed distinct trends across methods. Since MEME ignores background data to train PWMs, its performance remains stable regardless of background changes (Fig. 2a and b). Across 59 datasets, MEME performs best at distinguishing positive (ChIP-seq) sequences from synthetic negatives when trained on the most reliable 5000 sequences (AUPRC = 0.7). Interestingly, when distinguishing positives from biological negatives (1:1 ratio), MEME performs better when trained on the top 1000 sequences (AUPRC = 0.72). However, performance drops when tested on the needle-in-the-haystack scenario (1:10 positive–negative ratio, AUPRC = 0.28), but 1000 sequences still yield the best results. Similar trends are observed for AUROC (Supplementary Fig. 1a and b), while MEME performs best at 5000 sequences on all three test scenarios in terms of F1-score (Supplementary Fig. 2a and b). STREME shows different trends depending on the training background (Fig. 2c and d). Using synthetic background data, when tested on distinguishing positive from synthetic negatives, STREME achieves the best results at 10 000 sequences (AUPRC = 0.72). Even when tested on biological negatives, STREME remains competitive (AUPRC = 0.77), performing best at 10 000 sequences. However, in the needle-in-the-haystack scenario, performance declines sharply (AUPRC = 0.35). AUROC and F1-score follow similar trends (Supplementary Figs 1c and 2c). When trained using a biological background, STREME’s performance on synthetic test data generally decreases, but it improves significantly on biologically feasible negatives (AUPRC = 0.82 for 1:1 and AUPRC = 0.43 for 1:10), with dataset sizes of 10 000 and full yielding the best performance. AUROC and F1-score confirm that larger training sets enhance STREME’s performance in this scenario (Supplementary Figs 1d and 2d). SVM and NT models’ performance is strongly influenced by the type of background data used for training (Fig. 2e–h). Both perform best on synthetic test data when trained with synthetic backgrounds (SVM AUPRC = 0.97; NT AUPRC = 0.88) and achieve higher accuracy on biologically feasible test datasets when trained using biological negatives (SVM AUPRC = 0.96 for 1:1, 0.8 for 1:10; NT AUPRC = 0.70 for 1:1, 0.61 for 1:10). Larger training datasets consistently improve performance, reaching peak performance when trained on full datasets. Notably, both models remain robust in the needle-in-the-haystack scenario when trained on biological negatives (SVM AUPRC = 0.8 versus AUPRC = 0.3 for synthetic-trained models; NT AUPRC = 0.61 versus AUPRC = 0.28 for synthetic-trained models). AUROC and F1-score analyses confirm that larger training sets and careful choice of background training data enhance SVMs’ performance (Supplementary Figs 1e–h and 2e–h).

Figure 2.

Average AUPRC values from models trained across 59 datasets with varying background and dataset sizes. (a, b) AUPRC performance of MEME’s PWMs trained on synthetic (a) or biologically feasible (b) background data, tested on synthetic negative sequences and biological negatives (1:1 and 1:10 positive–negative ratios). (c, d) AUPRC performance of STREME’s PWMs trained on synthetic (c) or biologically feasible (d) background data, evaluated under the same testing conditions. (e, f) AUPRC performance of SVM-based models trained on synthetic (e) or biological (f) background data, tested similarly. (g, h) AUPRC performance of NT trained on synthetic (g) or biological (h) background data, tested in the same way. Labels in bold mark the best performing training dataset size on each testing scenario.

Examining how training sequence width affects model performance revealed distinct trends across methods. MEME performs best on narrow sequences, peaking at 100 bp when tested on synthetic (AUPRC = 0.71) and biological negatives (1:1 ratio; AUPRC = 0.75) while excelling in the needle-in-the-haystack scenario at 50 bp (Fig. 3a and b; Supplementary Figs 3a and b and 4a and b). STREME performs better on wider inputs, yielding best results at 150–200 bp, depending on the training and test backgrounds (Fig. 3c and d). In the needle-in-the-haystack scenario, STREME performs better when trained on full-length sequences and using biological background (AUPRC = 0.43 versus AUPRC = 0.37 for synthetic-trained models). AUROC and F1-score trends align with these findings (Supplementary Figs 3c and d and 4c and d). SVMs are more sensitive to sequence width, requiring full-length sequences for optimal performance when trained and tested on biological backgrounds (AUPRC = 0.96 and 0.8, respectively; Fig. 3e and f). NT shows consistent improvement with longer inputs, achieving top performance with full-length training sequences and biologically realistic backgrounds (AUPRC = 0.70 and 0.61, respectively; Fig. 3g and h; Supplementary Figs 3g and h and 4g and h). Overall, longer training sequences tend to enhance SVM and NT models, while MEME and STREME perform better on moderate lengths, balancing signal and noise.

Figure 3.

Average AUPRC values from models trained across 59 datasets with varying background and sequence widths. (a, b) AUPRC performance of MEME’s PWMs trained on synthetic (a) or biologically feasible (b) background data, tested on synthetic negative sequences and biological negatives (1:1 and 1:10 positive–negative ratios). (c, d) AUPRC performance of STREME’s PWMs trained on synthetic (c) or biologically feasible (d) background data, evaluated under the same testing conditions. (e, f) AUPRC performance of SVM-based models trained on synthetic (e) or biological (f) background data, tested similarly. (g, h) AUPRC performance of NT trained on synthetic (g) or biological (h) background data, tested in the same way. Labels in bold mark the best performing training sequence width on each testing scenario.

We also evaluated the impact of kernel functions on SVM-based model performance (Fig. 4a and b). When trained and tested using synthetic background data (Fig. 4a), gapped k-mer, gkm, and gkmrbf kernels outperform the classical k-mer counting and weighted kernels across all test scenarios. When tested on biological negatives, instead, the weighted gkm and gkmrbf kernels achieve the best performance (AUPRC = 0.78 for 1:1 ratio, 0.3 for 1:10 ratio). Similarly, when trained using biologically feasible background sequences (Fig. 4b), weighted kernels consistently outperform the other kernels on all test scenarios, indicating that positional weighting enhances model robustness, particularly when analyzing biologically feasible data (AUPRC = 0.8 for both wgkm and wgkmrbf). AUROC and F1-score analyses confirm these trends (Supplementary Figs 5a and b and 6a and b).

Figure 4.

Average AUPRC values for SVM-based models trained across 59 datasets, comparing different LS-GKM kernel functions and optimized training configurations. (a, b) performance of kernel functions available in LS-GKM when trained on synthetic (a) and biologically feasible (b) background sequences. Labels in bold mark the best performing training kernel function on each testing scenario. (c) Comparison of model performance using two optimization strategies: A globally optimized tool-specific configuration versus per-experiment fine-tuning.

To assess the impact of configuration optimization on model performance, we applied two optimization strategies: (i) selecting the best-performing dataset size, sequence width, and kernel (for SVMs) per tool across all experiments and (ii) selecting the best configuration individually for each dataset and tool (Fig. 4c). By analyzing average performance across AUPRC, AUROC, and F1-score (Fig. 4c and Supplementary Figs 5c and 6c), we generally observed an improvement in performance compared to those obtained in nonoptimized scenarios. However, we observed no significant differences between the two optimization approaches across training scenarios (Supplementary Tables 1–3). This suggests that a globally optimized, tool-specific configuration is sufficient for robust performance, eliminating the need for fine-tuning on a per-experiment basis.

We further explored model performance by evaluating scenarios that combine synthetic and biological negative sequences during models’ training and testing (see Supplementary File Section 9 for details).

To investigate the sequence patterns driving models’ prediction, we analyzed the positional distribution of motif scores on positive test sequences for the best- and worst-performing TFs across all methods, except for NT (Fig. 5 and Supplementary Fig. 7). We excluded NT because, unlike the other models, it lacks the ability to meaningfully score isolated sequence fragments or individual tokens. For each TF, models were applied to score positive test sequences (from test sets) at each position. Scores measure the likelihood that the binding motif of the investigated TF occurs at a specific position within the sequence. MEME performed best in predicting CTCF [91] (experiment ENCFF796WRU) binding (Fig. 5a and b), showing motif occurrences near peak summits even when trained using synthetic background sequences. STREME exhibited a similar but stronger trend. LS-GKM displayed a sharper signal localization, aligning with its higher AUPRC compared to PWMs (Fig. 5a). Training models using biologically feasible background sequences (Fig. 5b) yielded a minor improvement for LS-GKM, suggesting that it benefits from training with biologically relevant backgrounds but remains robust across training conditions. For MEME and STREME PWMs, the improvement was more marked. Precision–recall (PRC) and receiver operating characteristic (ROC) curves further support these findings, confirming SVM’s predictive advantage.

Figure 5.

Comparison of transcription factor binding predictions by MEME, STREME, and LS-GKM trained on synthetic (left) and biological (right) background sequences. The panels present results for the top-performing TFs of each tool: (a, b) CTCF for MEME (experiment ENCFF796WRU), (c, d) HMGA2 for STREME (experiment ENCFF614IHQ), and (e, f) SETDB1 for LS-GKM (experiment ENCFF812YDP). Heatmaps illustrate the positional distribution of motif scores across test sequences, highlighting motif enrichment patterns near peak summits. ROC (top left) and precision-recall (top right) curves summarize predictive performance.

STREME excelled in predicting HMGA2 [92] (experiment ENCFF614IHQ) binding (Fig. 5c and d), particularly when trained using biological negatives. In this scenario, STREME’s PWM showed a strong motif signal around the peak summit (Fig. 5d). However, the signal was less pronounced than with LS-GKM, which maintained the highest signal concentration around the peak summit. MEME displayed a dispersed signal instead. When trained employing synthetic background sequences, MEME and STREME performed near-randomly (Fig. 5c), failing to detect binding signals. Similarly, LS-GKM’s SVM model performed less effectively in this scenario. PRCs and ROCs reinforce these trends, highlighting LS-GKM’s consistent advantage and STREME’s dependence on training background sequences.

LS-GKM achieved its highest performance predicting SETDB1 [93] (experiment ENCFF812YDP) binding (Fig. 5e and f), outperforming MEME and STREME when trained using biologically feasible background sequences (Fig. 5f). However, its predictions exhibited broader signal distribution compared to the previous TFs, potentially indicating that the model captured secondary, functionally relevant sequences beyond the binding site itself. This suggests that LS-GKM may incorporate additional regulatory elements into its predictions, a hypothesis supported by PRC and ROC curves, which reflect the overall challenges in predicting SETDB1 binding.

An analysis of the experiments with the weakest predictive performance revealed consistent trends across the three tools. MEME showed the lowest AUPRC when predicting MYBL2 [94] (experiment ENCFF904DOZ) binding (Supplementary Fig. 7a and b). Heatmaps show that none of the models, including the SVM-based model, captured clear positional enrichment when trained using synthetic background data. Similar results were observed when training models with biological negatives, where MEME and STREME failed to display motif aggregation (Supplementary Fig. 7a). However, LS-GKM exhibited slight improvement, with stronger signal concentration near the peak summit (Supplementary Fig. 7b) and increased AUPRC and AUROC.

STREME performed the worst when predicting SOX13 [95] (experiment ENCFF453YPR) binding (Supplementary Fig. 7c and d). When trained using synthetic background sequences, it showed no clear positional weighting (Supplementary Fig. 7c), while training with biological background data led to a light improvement (Supplementary Fig. 7d. LS-GKM, in contrast, maintained strong performance across both training conditions, consistently assigning higher weights to regions near the peak summit. MEME behaved randomly in both conditions.

LS-GKM struggled most with TEAD4 [96] (experiment ENCFF501XJP) binding (Supplementary Fig. 7e and f). Heatmaps reveal that PWMs failed to learn informative patterns, showing no motif enrichment near the peak summit. However, LS-GKM still outperformed MEME and STREME, displaying robustness even in challenging prediction scenarios. This suggests that LS-GKM may better handle complex binding grammars, albeit with some limitations.

Analyzing the tool’s scaling behaviors, they displayed different trends as dataset size and sequence width increase (Supplementary Tables 4 and 5). LS-GKM experiences a sharp rise in processing time with larger datasets and wider sequences, leading to high computational costs. MEME also slows considerably under these conditions. In contrast, STREME is the fastest for smaller datasets but becomes increasingly sensitive to input size, eventually surpassing MEME in processing time. Among LS-GKM’s kernel functions, the gkm kernel is the most efficient, while the classical k-mer counting kernel (estlmer) is the slowest. Computational performances were measured on a Linux PC with an AMD Ryzen Threadripper 3970X CPU with 32 cores and 64 GB of RAM. NT was excluded from this comparison, as it was executed on a GPU, making runtime results not directly comparable.

SVM-based motif models show strong predictive power for TF binding and maintain robustness to class imbalance. However, their wider adoption has been hindered by limited interpretability and the lack of publicly available model repositories. To bridge this gap, we constructed a comprehensive database of 2393 optimized SVM-based models trained on human ENCODE ChIP-seq data. Supplementary File Section 8 details the models’ construction pipeline and performance evaluation.

Discussion

In this study, we performed a comprehensive benchmark to compare the performance of PWMs, SVM-, and DL-based motif models in encoding TFBSs. To thoroughly evaluate the models at both theoretical and practical levels, we designed multiple test scenarios. Specifically, we used synthetic background data to measure the theoretical predictive ability of the models and DNase-seq data as a background to provide a more realistic assessment of their performance in biologically relevant contexts.

Our results demonstrate that SVM-based models generally outperform PWMs in most scenarios, particularly when tested on imbalanced datasets. NT models show similar performance when trained on biologically feasible backgrounds. However, we also observed that PWMs can still serve as the preferred model under certain conditions. Table 2 discusses models’ performance based on input dataset characteristics.

Table 2.

Model selection guide based on dataset characteristics. This table provides guidance on selecting the better model [PWM (MEME or STREME), SVM (LS-GKM), or DL (NT)] based on key dataset attributes, including dataset size, sequence width, and background training data type. The column ‘dataset size’ refers to the number of sequences in the training dataset (small: Fewer than 5000 sequences; large: >5000 sequences). ‘Sequence width’ indicates the length of each sequence in the training dataset (narrow: <200 bp wide; long: >200 bp wide). ‘Background type’ describes the type of background data used for training. The column ‘recommended model’ highlights the model expected to perform best for each combination of characteristics, with a rationale provided for each recommendation

Dataset size	Sequence width	Background type	Recommended model	Rationale
Small	Narrow	Synthetic	PWM (MEME)	MEME performs better on small datasets of narrow sequences, especially when background data are unavailable, such as in SELEX or PBM experiments, where sequences lack genomic context (Figs 2a and 3a). Since
		Biologically feasible	PWM (STREME)	MEME does not require background data to train PWM models, it handles synthetic backgrounds better, making it ideal for cases where no biologically meaningful background can be recovered. In contrast, STREME benefits from real biological background data (e.g. DNase-seq) and performs well when genomic context matters, such as in ChIP-exo experiments (Figs 2d and 3d). When meaningful background sequences are available, STREME is the better choice.
Large	Narrow	Synthetic	PWM (MEME)	MEME performs well on large datasets of narrow sequences, particularly when background data are unavailable and synthetic backgrounds need to be computed (Figs 2a and 3a). Since MEME does not require background data to
		Biologically feasible	PWM (STREME)	train PWM models, it is better suited for cases where only synthetic background data may be used, avoiding potential biases from biologically implausible controls. STREME, in contrast, excels with large datasets of narrow sequences, such as ChIP-exo data (Figs 2d and 3d). Its ability to integrate biologically feasible background data (e.g. DNase-seq) enhances its ability to distinguish bound from unbound sequences.
Small	Long	Synthetic	SVM (LS-GKM)	LS-GKM SVM-based models perform well on small datasets with long sequences, particularly when trained on biologically feasible background data, making it suitable for high-quality but limited ChIP-seq data where biological
		Biologically feasible	SVM (LS-GKM)	context enhances models’ performance (Fig. 2f). SVMs leverage the amount of information carried by long sequences. When biological background is unavailable, LS-GKM remains applicable using synthetic background data (Fig. 2e). However, STREME PWMs, while slightly outperformed by LS-GKM in this synthetic background setting, remain competitive (Fig. 2c).
Large	Long	Synthetic	SVM (LS-GKM)	LS-GKM and NT perform well on large datasets with long sequences, particularly when trained on biological background data. This makes them well-suited for ChIP-seq,
		Biologically feasible	SVM (LS-GKM), DL (NT)	where both binding sites and regulatory flanking regions contribute to signal (Figs 2f and h and 3f and h). When only synthetic background data is available, LS-GKM remains effective, performing strongly in this setting (Figs 2e and 3e).

PWMs trained using alignment-based methods like MEME are particularly valuable for small datasets with highly reliable bound sequences. These methods do not require background data for training, making them ideal for scenarios with high-confidence positive data and where background datasets cannot be recovered or computed, such as PBM or SELEX [13, 14] experiments (Table 2).

PWM training methods based on enumeration, such as STREME, are also well suited for small, reliable datasets and can handle larger datasets more effectively than alignment-based methods (Table 2). Unlike MEME, enumerative methods can employ background sequences during training. Our analysis shows that employing biologically feasible background data, such as DNase, rather than synthetic negative sequences, significantly improves their ability to distinguish between bound and unbound sequences. However, we observed that the performance of STREME is influenced by the input dataset size and the length of training sequences. Since PWMs focus on learning the binding site itself and not the surrounding sequence context [97, 98], longer sequences can introduce noise, reducing predictive performance. Consequently, STREME is particularly effective for large datasets with narrow sequences, such as those generated by ChIP-exo [24].

When working with large datasets containing long sequences, as is common in ChIP-seq experiments, SVM-based models appear better suited due to their ability to learn both core TFBSs and surrounding sequence context. As a result, they are robust to noise from flanking regions and perform well even in imbalanced datasets, where negative samples greatly outnumber positives. However, they incur a significant increase in computational time for larger datasets (Table 2). Similarly, NT models perform well when trained on biologically feasible datasets, thanks to their ability to model complex sequence dependencies (Table 2). Yet, they require substantial computational resources and scale quadratically with sequence length, limiting their scalability in practical applications.

However, SVM-based and NT models lack interpretability, making it challenging to determine which sequence features the models learned [23, 41, 48, 99, 100]. In scenarios where understanding the learned features is crucial, PWMs remain the preferred choice due to their simplicity and straightforward visualization. Nevertheless, analyzing SVM weight assignments to k-mers suggests that these models possibly recognize secondary regulatory sequences beside motifs. Furthermore, SVM training often requires significantly more time compared to PWM training methods like STREME or MEME, making PWMs a more practical option for time-sensitive applications.

In this study, we selected evaluation metrics that do not rely on assumptions about TN sequences. Accurately identifying sequences that are unbound by a TF in real genomic datasets is inherently challenging. For instance, sequences classified as negatives, such as DNase-seq peaks that do not overlap with ChIP-seq peaks, may remain unbound under the specific experimental conditions tested. However, these same sequences might exhibit TF binding activity under alternative environmental conditions or in different cell types or tissues [101–103].

This ambiguity underscores the importance of prioritizing metrics that do not explicitly depend on the definition of true negatives when assessing model performance, such as AUPRC. By doing so, we mitigate potential biases introduced by incorrect or context-dependent classification of negatives.

Additionally, we emphasize the value of metrics that penalize false negatives, as the identification of bound sequences is often supported by high-confidence experimental data, such as ChIP-seq, that validate TF binding. The confidence in positive sequences contrasts sharply with the uncertainties associated with negatives, making it essential to minimize errors in identifying true binding events. FNs can have significant downstream consequences, particularly in applications such as regulatory network reconstruction [104, 105], or noncoding variants impact prediction [23], where missing key binding events can distort biological insights.

The reliance on experimental data, such as ChIP-seq, for positive sequences raises additional considerations. While these datasets are generally robust, they are not immune to biases, including experimental noise or limitations in sensitivity [1, 106, 107]. Moreover, ChIP-seq peaks may represent indirect binding events mediated by protein complexes rather than direct TF-DNA interactions [1, 108]. This complexity adds another layer of difficulty in evaluating models, highlighting the need for complementary validation strategies.

On the other hand, the inherent uncertainty in negative sequences poses challenges in developing and validating predictive models. This issue is exacerbated in highly dynamic regulatory systems, where TF binding is modulated by temporal, spatial, and environmental factors [1, 109, 110]. Incorporating such variability into models remains an open problem. Future approaches might leverage multi-omics data, such as ATAC-seq or single-cell transcriptomics, to refine the definition of both positive and negative sequences and improve model training as well as their evaluation.

Key Points

• Position weight matrices (PWMs) have been broadly employed due to their simplicity and interpretability. However, they fall short in capturing positional dependencies and intricate sequence interactions. Support vector machine (SVM)–based and deep learning (DL)–based models have emerged as powerful alternatives.

• PWMs perform optimally with small datasets containing high-confidence sequences, especially when background data are unavailable. In contrast, SVM- and DL-based models excel in handling large datasets and long sequences, effectively capturing sequence context despite requiring extended training times.

• The inclusion of biologically relevant background data, such as DNase-seq, enhances model accuracy compared to synthetic alternatives. While SVM- and DL-based models offer superior predictive performance, they lack interpretability, whereas PWMs provide clear motif representations.

• A curated repository of pretrained SVM models trained on human ChIP-seq data has been developed, facilitating broader accessibility and application in regulatory genomics research.

Funding

This work has been funded by the European Union - NextGenerationEU through the Italian Ministry of University and Research under PNRR - M4C2-I1.3 Project PE_00000019 ‘HEAL ITALIA’ CUP B33C22001030006.

Data availability

The source code to reproduce the analyses presented in this paper is available at https://github.com/InfOmics/PWM-SVM-benchmark. The datasets used in this benchmark can be accessed at https://doi.org/10.5281/zenodo.15052718. Additionally, the collection of SVM models trained on ENCODE ChIP-seq dataset is available at https://doi.org/10.5281/zenodo.15056257.