Evaluating methods for the prediction of cell-type-specific enhancers in the mammalian cortex

Summary

Identifying cell-type-specific enhancers is critical for developing genetic tools to study the mammalian brain. We organized the “Brain Initiative Cell Census Network (BICCN) Challenge: Predicting Functional Cell Type-Specific Enhancers from Cross-Species Multi-Omics” to evaluate machine learning and feature-based methods for nominating enhancer sequences targeting mouse cortical cell types. Methods were assessed using in vivo data from hundreds of adeno-associated virus (AAV)-packaged, retro-orbitally delivered enhancers. Open chromatin was the strongest predictor of functional enhancers, while sequence models improved prediction of non-functional enhancers and identified cell-type-specific transcription factor codes to inform in silico enhancer design. This challenge establishes a benchmark for enhancer prioritization and highlights computational and molecular features critical for identifying functional cortical enhancers, advancing efforts to map and manipulate gene regulation in the mammalian cortex.

Graphical abstract

Highlights

• •Community challenge benchmarked cortical enhancer prediction methods
• •Top methods leveraged single-cell ATAC-seq specificity for cell type enhancer ranking
• •Sequence models improved identification of non-functional enhancers
• •Combining chromatin and sequence data enhanced prediction accuracy

Johansen et al. report the results of a community challenge to predict functional enhancers targeting specific brain cell types. By comparing multi-omics machine learning approaches using in vivo data, they define critical features for enhancer prediction, improving genetic tool design for cell type targeting in the mammalian cortex.

Introduction

The mammalian neocortex, responsible for higher-order cognitive functions and sensorimotor processing, includes the primary motor cortex (M1), which facilitates fine motor movement and is composed of diverse cell types with distinct molecular signatures.^1,2 Some neurodegenerative diseases, including Parkinson’s disease, Huntington’s disease, and amyotrophic lateral sclerosis, affect specific M1 cell types and result in impaired coordination and dexterity.³ There is an urgent need for genetic tools to selectively access vulnerable cell populations to probe cortical circuit function and treat disease. To this end, cell-type-selective candidate enhancers have been identified based on single-cell genomic profiling of the mouse cortex and have been used to create recombinant adeno-associated virus (AAV) vectors to drive exogenous transgene expression in the predicted target cell types.^4,5,6 With increasingly comprehensive molecular phenotyping of the brain,^7,8,9,10,11 there is the prospect of developing AAV tools to target a wide diversity of cell types across the brain.

Identification of cell-type-specific viral tools remains challenging because experimental validation is low throughput and expensive. A recent study by Ben-Simon et al.⁶ tested 825 enhancers selected based on specificity of open chromatin within a cortical cell type of interest and achieved an overall average success rate of 30%. Consequently, the field needs new computational approaches to improve functional enhancer prediction⁹ and accelerate viral tool development that integrates comprehensive molecular profiling within and across species. However, it remains poorly understood how well different approaches predict functional over non-functional enhancer genomic sequences. Community-guided challenges have demonstrated their effectiveness in rigorously evaluating novel computational methods and advancing knowledge in the genomics field.¹² To date, there has been no community-led effort to address the prioritization of cell-type-specific enhancers using cutting-edge cell-type-resolved atlases from multiple species.

In this study, we present the Brain Initiative Cell Census Network (BICCN) Challenge, where six teams from computational biology labs across the world participated to predict functional cortical cell-type-specific enhancers. We introduce a community-driven benchmark and metrics to identify top-performing approaches that prioritize functional, cell-type-specific viral tools. By investigating the computational approaches and biological priors used by high-performance methods, we aim to contribute to the refinement of functional enhancer prediction to selectively target cell types in the mammalian cortex.

Results

A community challenge to predict functional enhancers

We provided teams with a comparative multi-omics study of M1 by Zemke et al.¹³ that measured the molecular profiles of individual nuclei using single-cell multi-omics and single-cell methyl-Hi-C in human, macaque, marmoset, and mouse (Figure 1A). Multiple species were included because molecular and DNA sequence patterns associated with enhancer activity are conserved in the mammalian brain.^1,14 Teams were asked to combine cross-species genomics measurements and biological priors to prioritize cell-type-specific and functional enhancer elements (Figure 1B). We asked teams to provide the top 10,000 putative enhancers for each cell type to increase the likelihood of validated enhancers being included in the rankings.

Figure 1.

Overview of the enhancer prioritization challenge

(A) Single-nucleus multi-omics data from the M1 of human, macaque, marmoset, and mouse. mya, million years ago.

(B) Schematic of the computational challenge to prioritize candidate cell-type-specific enhancers.

(C) Overview of AAV construction, cell-type ATAC-seq specificity, and screening of in vivo activity in the mouse brain for three candidate L5 extratelencephalic-projecting ( ET) enhancers.

(D) Teams predicted and ranked 10,000 candidate enhancers for each of 19 cortical cell types and were scored based on prioritization of strong, on-target enhancers.

(E) Combinations of data and methods for top team submissions.

(F) Normalized benchmark metrics (STAR Methods) based on epifluorescence and SSv4 from in vivo screening.

Teams’ predictions were evaluated against 677 recombinant AAV vectors among 825 vectors that were designed to label 19 M1 cell subclasses and assessed for in vivo enhancer specificity and brightness in the mouse brain.⁶ Validation data for 148 enhancers were released after the challenge and were therefore excluded from the analysis. The cell type specificity in the neocortex of each validated enhancer virus was assessed through manual evaluation of epifluorescence images, following the methodology outlined in Ben-Simon et al.⁶ The enhancers were classified into four groups: (1) on target (N = 202), specific labeling of the targeted cell type; (2) off target (N = 96), labeling of non-targeted cell types in the neocortex; (3) mixed target (N = 100), labeling of targeted and non-targeted cell type(s); and (4) no labeling (N = 279), no fluorescence in the mouse neocortex (Figure 1C, Table S1). To further quantify the in vivo activity of 191 enhancers, SYFP2-positive cells were extracted from the primary visual cortex (V1) and analyzed with Smart-seq v.4 (SSv4) sequencing.⁶

The challenge was run over a 3-month period. Participants submitted enhancer lists at several intervals, and performance was reported on a public leaderboard.¹⁵ In the final evaluation round, teams provided a detailed description of their approach (STAR Methods). To rigorously evaluate methods, we developed a benchmark metric that is optimized when enhancers with on-target activity are ranked highest and mixed-target, off-target, and no-labeling enhancers are ranked lowest for the respective cell type (Figure 1D; STAR Methods). Final scores per method were computed based on both epifluorescence and SSv4 metrics using the entire corpus of validated enhancers. The benchmark metric and an automated scoring tool are hosted via GitHub¹⁵ for the community to fairly evaluate novel enhancer prioritization methods.

Top-performing submissions focused on ATAC-seq specificity

Challenge participants from 5 teams contributed 79 submissions that comprised 16 unique enhancer prioritization methods, and the ArchR¹⁶ method was included as a performance baseline due to its popularity as a single-cell ATAC-seq processing pipeline. We grouped the methods into five broad categories based on the included enhancer features: (1) ATAC-seq (meta), (2) enhancer codes, (3) feature ranking, (4) sequence model, and (5) integration model (Figure 1E). The top three teams (Aerts, ArchR, and PeakRankR) achieved comparable performance with normalized scores between 0.36 and 0.41, while the remaining teams achieved scores below 0.27 (Figure 1F). Comparing methods based on precision and recall identifies that even top performing-teams are only able to achieve moderate accuracy in recovering on-target enhancers (F1 score: Aerts, 0.58; ArchR, 0.57; PeakRankR, 0.57). Teams submitted diverse approaches and markedly improved during the challenge (Figures 1E, 1F, and S1; Table S2). The final challenge ranks were statistically robust (p < 0.05) for 55 of 79 submissions based on a bootstrapping analysis (Figure S2; Table S2).

High-performing teams used similar approaches that leveraged ATAC-seq features, including differential chromatin accessibility (specificity) and signal strength at a given enhancer genomic location. The top-performing submission gained a slight advantage by including RNA sequencing (RNA-seq) and leveraging single-cell regulatory network inference and clustering (SCENIC)+¹⁷ to predict cell-type-specific transcription factor (TF)-enhancer-gene triplets. The runner-up baseline method, ArchR, used careful selection of background cells to minimize biases such as transcription start site (TSS) enrichment when performing pairwise statistical tests. The third-ranking team, PeakRankR, calculated three ATAC-seq metrics—specificity, magnitude, and coverage—that were combined to discriminate cell-type-selective enhancers.

Comparison of team submissions highlighted the genomics data, biological priors, and methodology that enabled accurate prediction of functional cell type enhancers (Figure S3, Table S2). While all submissions used mouse ATAC-seq data, they varied in how the signal was normalized to handle batch effects and how cell type specificity was calculated. Surprisingly, inclusion of DNA-methylation, chromatin folding (Hi-C), or primate data decreased performance, potentially due to increased model complexity and overfitting. Also, the Tanaka team used all data types but retained only the highest-magnitude open chromatin features that left primarily promoters, not enhancers, for the method to prioritize (supplemental information). Notably, a high-performance, runner-up method (Aerts CREsted) employed deep learning models, trained with the CREsted package,¹⁸ that learned to predict open chromatin from DNA sequence. Sequence models are of particular interest, since they have the potential to identify TF motifs that make up cell-type-specific enhancer codes,^19,20 including repressive elements.

Meta-analysis of performance across teams and cell types

To assess whether teams identified similar or distinct enhancers, we computed pairwise intersections of submissions from the top-performing methods (Figure 2A). Methods with similar performance predicted similar enhancers, and the top three methods (Aerts scATAC_triplet, ArchR, and PeakRankR) had the highest agreement. Many on-target enhancers were identified across several approaches (Figure 2B), yet some were not recovered by any method, likely due to low ATAC-seq signal (Figure S4).

Figure 2.

Comparison of team enhancer rankings

(A) Average proportion of ranked enhancers that overlap between pairs of team submissions for all cell types.

(B) Upset plot showing the number of validated enhancers that were identified by sets of submissions.

(C and D) Rates of identification of (C) on-target and (D) mixed-target, off-target and no-labeling enhancers.

(E) Comparison of methods based on distributions of normalized enrichment scores (NES). For each method and cell-type-specific ranking, NES measures the area under the recovery curve (AUC) up to the 1,000^th element compared to a random ranking.

(F) Heatmap ordered by Aerts scATAC_triplet scoring of L5 ET enhancers and summary of validation results. Examples of a strong (AiE0456m) and weak (AiE0460m) enhancer with Pou3f1 motifs identified by the CREsted model in the highlighted region. AiE0456m was also validated with SSv4.

The Aerts, PeakRankR, and ArchR methods consistently included on-target enhancers in the top rankings (Figure 2C), while the Aerts method deprioritized more mixed target, off-target and no-labeling enhancers (Figure 2D). From these recovery curves, we calculated the normalized enrichment score (NES) that quantifies enrichment of functional enhancers at the top of a method’s ranking (Figure 2E). Interestingly, compared to the top-performing methods, methods that used DNA sequence (Aerts CREsted) and Hi-C (Gillis) data better prioritized enhancers for L6b neurons, and the CREsted method better prioritized enhancers for low-abundance Sst Chodl inhibitory neurons (Figures 2E, S5, and S6; Table S2). In addition, Aerts scATAC_triplet and Aerts CREsted were better than ArchR and PeakRankR at deprioritizing lower-quality enhancers for most cell types (Figures S7 and S8).

Next, we compared performance in prioritizing L5 ET enhancers, the largest set of validated enhancers. The top five strongest and most specific on-target L5 ET enhancers were ranked highly by most methods, with somewhat lower rankings from Gillis (Figure 2F). Most methods correctly scored a strong L5 ET enhancer, AiE0456m, higher than a weak enhancer, AiE0460m, likely due to higher ATAC-seq signal in AiE0456m and the presence of multiple POU3F1 motifs (Figure 2F), a canonical TF of L5 ET neurons.¹ The strong enhancer AiE0463m had low ATAC-seq signal but multiple POU3F1 motifs and was correctly prioritized only by the Aerts CREsted model (Figures 2F and S9). Thus, methods that can learn from DNA sequence have an advantage to prioritize strong and specific enhancers that are associated with chromatin regions with limited accessibility.

Additional genomic features help predict enhancer activity

Since chromatin accessibility sometimes failed to predict cell-type-specific activity, we tested whether prediction accuracy could be improved by including additional enhancer features that were not provided to teams in the challenge. These features included (1) H3K27ac, a histone modification found at active enhancers²¹ from mouse cortical paired-tag data^13,22; (2) activity-by-contact (ABC) scores, the product of chromatin accessibility and contact frequency from Hi-C^13,23; and (3) conservation of chromatin accessibility between human and mouse.¹³ Like chromatin accessibility, cell-type-specific H3K27ac and ABC scores were significantly higher for on-target validated enhancers compared to mixed-target, off-target, and no-labeling enhancers and a random control (Figure 3A). On-target enhancers had significantly higher conservation of accessibility (Figure 3A) and sequence (Figure S10) compared with no-labeling and random but not mixed-target or off-target enhancers. Thus, conservation of accessibility predicted overall enhancer activity, while H3K27ac and ABC predicted cell-type-specific enhancer activity.

Figure 3.

Enhancer features predictive of functional activity

(A) Comparison of molecular features between on target and other enhancer categories. ∗∗p < 0.001, ∗∗∗p < 0.0001 Wilcoxon rank-sum test, two sided, unpaired.

(B) Correlation of H3K27ac and ATAC-seq specificity for astrocyte enhancers.

(C) Examples of astrocyte enhancers with in vivo activity that is better predicted by H3K27ac than ATAC-seq signal.

(D) Summary of informative features from a random forest model predicting enhancer activity. ANOVA with Tukey post hoc tests, Bonferroni-corrected p values.

(E) Schematic of ATAC-seq peak quantification based on cut sites or coverage and boxplot comparison of peak specificity for all on-target enhancers between different preprocessing methods. Adjusted p values were obtained through t tests, ∗p < 0.05, ∗∗p < 0.01.

(F) Overall enhancer activity prediction performance from peak specificity for the different methods for on-target (left) and all except off-target (right) enhancers.

For validated enhancers, cell-type-specific chromatin accessibility was notably correlated with cell-type-specific H3K27ac (r = 0.57) and ABC scores (r = 0.38) but not epigenetic conservation (r = 0.07) (Figures 3B and S11). For example, the on-target enhancer AiE2121m had high chromatin accessibility and H3K27ac specifically in astrocytes (Figures 3B and 3C). In contrast, no-labeling element AiE0358h had high chromatin accessibility but not H3K27ac specifically in astrocytes (Figures 3B and 3C). These examples demonstrate a potential for H3K27ac signal to improve the accuracy of enhancer predictions by distinguishing functional from non-functional chromatin-accessible candidate enhancers. Additionally, we trained a binary random forest classification model to predict on-target enhancers from all enhancers that were off target or had no labeling. This approach made use of labeled data and, thus, is a different approach than the BICCN Challenge submissions described in this manuscript. In the held-out test set, this model resulted in an area under the recovery curve (AUC) of 0.75. This model is better able to identify enhancers that are off target or have no labeling (precision 0.77, recall 0.80) than on-target enhancers (precision 0.52, recall 0.47). While the precision is relatively low, this model outperformed the initial enhancer selection, where only 30% of screened enhancers were scored as on target.⁶ The most informative features can be leveraged in future enhancer prediction models, including open chromatin metrics (ATAC-seq strength or Z score and cell type specificity), linear sequence conservation (% matching base pairs), and GC content (Figures 3D and S12).

Finally, we compared differences in ATAC-seq preprocessing methods, as there was some variability in data processing for the top-performing differential accessibility rankings. We compared counts-per-million (CPM) normalized coverage and cut-site tracks (Figure S13) and found that cell-type-specific peak heights based on accumulation of the signal inside a peak are significantly (p_adj < 0.05) more specific in on-target enhancers if they are obtained from cut-site tracks (Figure 3E). Moreover, additional normalization methods to scale peak heights across cell types (STAR Methods) further increase specificity. Using specificity as a metric to score enhancer activity in a multilabel classification setting (Figure 3F), we confirmed the increased performance of cut site-based peaks compared to coverage peaks both for on-target and all validated enhancers except off-target enhancers. These results demonstrate the benefit of peak normalization, which ensures better comparability of peak heights across cell types.

Rescored enhancers validate model predictions

To further investigate differences in predicted versus in vivo activity for the 677 tested enhancers, we manually inspected the single-cell ATAC (scATAC)-seq signals from Aerts scATAC_triplet and prediction and nucleotide contribution scores from Aerts CREsted. These top-performing models used the same CPM-normalized ATAC-seq coverage tracks. We classified each enhancer into one of five categories. “Explainable positives” (48%) have in vivo activity and high ATAC specificity and CREsted scores in the corresponding cell type. “Explainable negatives” (25.4%) have no in vivo activity and low ATAC or CREsted specificity. “Unexplainable positives” (6.2%) have activity but low ATAC and CREsted specificity. “Unexplainable negatives” (13.9%) have no reported activity but high ATAC and CREsted scores. The remaining enhancers were classified as “missing data” (6.5%). We visualized enhancers based on the peak-scaled ATAC signal and labeled by the targeted cell type (Figure 4A). Explainable on-target enhancers were well segregated and overlapped with some unexplainable no-labeling (negative) enhancers. This suggested that some negative enhancers may have weak in vivo activity that was missed in the initial evaluation.

Figure 4.

Refinement of models and enhancer screening results

(A) t-distributed stochastic neighbor embedding (t-SNE) plots of enhancers based on ATAC-seq specificity and labeled by the targeted cell type. On-target and no-labeling enhancers had explainable or unexplainable cell type labeling patterns based on ATAC-seq and DNA sequence (CREsted) model predictions.

(B) River plots of enhancer activity, predictions, and rescoring of experimental validation data.

(C and D) Model scores, predicted TF motifs, and SYFP fluorescence for two oligo enhancers with epifluorescence strengths (C) strong on-target and (D) no-labeling rescored to weak on-target activity.

(E) Performance of enhancer ranking methods using the rescored enhancer activities. AP, average precision. scATAC included two normalizations: count-normalized coverage pseudobulk or peak scaled.

(F) CREsted model scores for strong and weak on-target enhancers grouped by cell type. Mean ± SEM.

(G) Comparison of models at identifying No-Labeling enhancers. ∗p < 0.05, ∗∗∗p < 0.001, Wilcoxon rank-sum test, Bonferroni-corrected p values.

We re-evaluated validation data from all 267 no-labeling enhancers and found that 66 enhancers weakly drove SYFP2 expression in the brain (Figure 4B). 52 of 66 enhancers had weak expression in the targeted cell type and were rescored to on target. These enhancers were enriched for unexplainable versus explainable negatives, which supports our predictions of enhancer activity. Interestingly, among the 140 explainable negatives that were confirmed to have no activity after rescoring, 45.7% had ATAC-seq signal in their targeted cell type but no support from the CREsted model. Conversely, only two enhancers (1.3%) contained a specific CREsted prediction and no ATAC-seq peak. The remaining 74 enhancers (53%) were not predicted to have cell-type-specific activity by either ATAC-seq or CREsted. Thus, considering DNA sequence along with ATAC-seq signal can help avoid testing inactive enhancers.

To illustrate the rescoring process, we show two enhancers that had scATAC-seq, CREsted, and H3k27ac scores that were specific to oligodendrocytes (oligos). First, an on-target oligo enhancer was predicted by the CREsted model to include two important candidate TF binding sites: one for the established oligo TF SOX10^24,25 and one for CREB5, known to play an important role in oligo myelin synthesis and differentiation (Figure 4C). Second, a no-labeling enhancer was rescored as weakly on target for oligos, and this was consistent with predictions from all modalities and a SOX10 binding site (Figure 4D). To better understand sequence patterns that define cell-type-specific on-target enhancers, we calculated contribution scores for all on-target enhancers in their corresponding cell types and identified frequently occurring motifs²⁶ and clustered them across cell types¹⁸ (Figure S14). For example, we found distal-less homeobox (DLX)-and LIM homeobox (LHX)-like motifs in interneurons, motocyte enhancer factor 2 (MEF)-like motifs and enhancer box (E box) motifs in neurons, SRY-box transcription factor (SOX)-like motifs in oligos and oligo precursors (OPCs), and nuclear factor 1 (NF1)-like motifs in astrocytes, oligos, OPCs, and Vip and L6b neurons, consistent with previously reported TF motif enrichments in cortical cell types.⁹

Using the rescored enhancers, we assessed the multi-label classification performance of enhancer activity by calculating precision and recall at different score thresholds. ATAC-seq modalities scored higher (average precision [AP] = 0.51 using peak-scaled normalization, AP = 0.52 using ArchR ReadInTSS normalization) than the CREsted sequence model (AP = 0.42) and H3K27ac (AP = 0.25) (Figure 4E). A receiver operating characteristic curve highlights the same trends for the different modalities (Figure S15). Interestingly, combining outputs from the CREsted and scATAC-seq models improves prediction (AP = 0.54), demonstrating that the modalities contain complementary information.

Next, we examined whether the CREsted model captured sequence differences that were associated with the magnitude of in vivo activity of on-target enhancers. On average, the model predicted higher scores for strong versus weak enhancers for 9 of 14 cell types (Figures 4F and S16) in contrast to the magnitude of the ATAC-seq signal (Figure S17). Additionally, we investigated how well models identified explainable negative enhancers. The CREsted model significantly (p < 0.05) outperformed scATAC approaches in avoiding false positives (Figure 4G) and false negatives (Figure S18). Overall, these analyses underscore the benefits of using sequence models to improve non-functional enhancer prediction, prioritize strong enhancers, and lay the groundwork for understanding cell type enhancer codes.

Discussion

The BICCN Challenge provides a valuable benchmark for future computational methods aimed at predicting functional cell-type-specific enhancers. With 677 enhancers validated on a standardized pipeline and scoring criteria, this represents the largest collection of its kind.⁶ Importantly, both data and code are publicly available, facilitating further research and method refinement. Notably, a group that did not participate in the main challenge recently submitted enhancer ranking results, achieving top predictive performance (F1 score = 0.62) (Dai et al., personal communication).

The validation data from this challenge reveal which features are critical for predicting enhancer function, and the highest-performing methods primarily leveraged the specificity of ATAC-seq peaks in the mouse dataset. Interestingly, the top submission combined RNA-seq with ATAC-seq to predict cell-type-specific TF-enhancer-gene triplets. Additionally, incorporating Hi-C data was valuable for predicting enhancers in specific neuron types, such as L6b. H3K27ac and ABC scores tended to be higher in on-target enhancers but with lower recall and precision compared to cell-type specific ATAC-seq signals. Moreover, biological priors played a crucial role in the success of these models. For instance, including moderately sized ATAC-seq peaks was critical to predict candidate enhancers, since the largest peaks often represent promoters.

Despite initial expectations that deep learning and DNA sequence models would outperform, the results showed that simpler ranking of ATAC-seq peaks by their cell-type-specific signal performed similarly. However, the top model did not consistently excel across all cell types and enhancer categories. The distinct methodologies used by the top three teams indicate that there are opportunities to refine enhancer prioritization. For example, the CREsted sequence model avoided mixed-target or off-target enhancers and predicted a strong oligo enhancer by finding a coactivating TF motif AP-1 near a motif for SOX10, an oligo marker.^24,25 We further showed that combining cell-type-specific predictions from the sequence model together with cell-type-specific peaks from scATAC-seq data provides the best predictive factor of enhancer function and specificity (Figure 4E). Future challenge submissions that incorporate more diverse molecular profiling of cortical cells will be critical in evaluating the importance of open chromatin data for enhancer prediction.

Looking forward, a major goal of the field is to develop a comprehensive toolkit to target cell types across the brain. Improving data coverage and quality is crucial, as many on-target enhancers were missed due to low ATAC-seq signal. Fortunately, ongoing work supported by the NIH BRAIN Initiative is focused on whole-brain atlasing of cell types, including high-read-depth single-cell ATAC-seq, Hi-C, and histone marker profiling. Sequence models will enable the rational design of enhancers tailored to cell types or groups of types, a strategy successfully applied in fruit fly models.²⁰ Cross-species sequence models were under-represented in this challenge, but on-target enhancers tend to have conserved open chromatin and sequence, and TF regulatory networks show conservation across primates and rodents.¹ Additionally, sequence models trained on ATAC-seq data from multiple species enhance the prediction of chromatin accessibility.¹³ Given these observations, we propose that decoding conserved enhancer codes presents a promising avenue for developing robust and precise tools to target cell types across many species.

It will be essential to expand validation experiments to include more brain regions and cell types and to quantify specificity and completeness of labeling using automated scoring of whole-brain imaging and single-cell sequencing. Community sharing of raw data will enable re-evaluation with future algorithms and comprehensive whole-brain analysis, while negative results will provide valuable insights into DNA repressor codes and enhancer function. Cross-species testing, including in non-human primates, will help assess the predictive power of these models across model organisms and will bolster confidence in translational applications for humans.

Integrating modeling and experimental approaches will be key to advancing enhancer tool development. Selecting enhancers that offer the most informative data for models will refine predictions and improve success rates. Testing enhancers with conflicting predictions from different modalities, such as ATAC-seq versus DNA sequence, could yield critical insights even if they do not result in the most effective tools. Additionally, reinterpreting experiments with no labeling that have strong model support may uncover challenges in transducing some cell types, as reported for microglia.²⁷ Ultimately, we need interpretable models that help to generate viral tools to target cell types and provide insights into cell type identity and genetic regulation in the context of human evolutionary specializations and disease.

Limitations of the study

Several limitations temper the conclusions of this study and suggest directions for future improvement. First, enhancer predictions were trained on molecular data from the mouse motor cortex and validated in the V1, raising concerns about regional heterogeneity despite broad molecular conservation across neocortical subclasses.^7,28 Second, the use of PHP.eB AAVs, while enabling broad cortical transduction, introduces bias due to non-uniform tropism across cell types,^6,29 potentially confounding enhancer activity with viral infectivity. Third, manual epifluorescence scoring provides a coarse estimate of cell type specificity, relying on morphological and spatial cues that may underrepresent weak or sparse enhancer activity. Fourth, tested enhancers were often identified near cell type markers and may have biased validation away from more cell-type-specific or distal regulatory elements. Finally, moderate model performance (F1 score < 0.6) likely reflects limitations of feature selection, primarily chromatin accessibility, while more informative features, such as H3K27ac, ABC scores, and TF motifs were often unavailable or not systematically integrated. Future efforts should include multi-regional, quantitative validation and models that incorporate viral tropism and integrate diverse epigenomic and sequence features across species to improve enhancer prediction accuracy.

Resource availability

Lead contact

Requests for further information, resources, and reagents should be directed to and will be fulfilled by the lead contact, Trygve E. Bakken (trygveb@alleninstitute.org).

Materials availability

This study did not generate new unique reagents.

Data and code availability

• •Sequencing data and enhancer validation data used in this project are all from previous studies and are publicly available. Accession numbers and links to the datasets are listed in the key resources table.
• •All original code has been deposited at GitHub and is publicly available as of the date of publication. GitHub URLs and DOIs are listed in the key resources table.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

Acknowledgments

This publication was supported and coordinated through the NIH BRAIN Initiative Cell Census Network (BICCN) and Armamentarium for Precision Brain Cell Access (https://braininitiative.nih.gov/armamentarium). Research reported in this publication was supported by the Allen Institute for Brain Science and by NIH grants RF1MH121274 (to B.T.), 1UF1MH128339-01 (to B.T., T.E.B., T.L.D., B.P.L., and J.T.T.), R01MH113005 (to R.L. and J.G.), and RF1MH114126 and UG3MH120095 (to E.S.L., J.T.T., and B.P.L.). The authors also acknowledge Research Foundation - Flanders (FWO) PhD fellowship strategic basic research (1SH6J24N to N.K.); FWO PhD fellowship fundamental research (1191323N to S.D.W.); European Research Council Advanced Grant (ERC AdG 101054387), Chan Zuckerberg Initiative (CZI DI2-0000000068), FWO Strategic Basic Research (SBO) Programme (S005024N), FWO (G0I2722N EOS ID 40007513), FWO (G094121N), and FWO (G044124N) (to S.A.); and the “Pioneer” and “Leading Goose” R&D Program of Zhejiang (2024SSYS0032 to K.Z.). The authors thank the founder of the Allen Institute, Paul G. Allen, for his vision, encouragement, and support.

Author contributions

Viral tool testing, B.P.L., B.T., D.D., J.K.M., J.T.T., M.H., T.L.D., and Y.B.-S.; data analysis, B.L., D.A., D.D., D.M., E.C.E., E.J.A., F.W., G.H., G.P., J.D.M., J.G., J.T.T., K.Z., M.H., N.H., N.J.J., N.K., N.R.Z., R.L., S.A., S.D.W., S.S., T.E.B., V.K., Y.B.-S., and Y.T.; data interpretation, B.L., B.R., B.T., D.A., D.D., D.M., E.C.E., E.J.A., E.S.L., F.W., G.H., G.P., J.D.M., J.G., J.M., J.R.E., J.T.T., K.Z., M.H., N.H., N.J.J., N.K., N.R.Z., R.L., S.A., S.D.W., S.S., T.E.B., V.K., Y.B.-S., and Y.T.; writing of the manuscript, B.L., B.T., D.D., F.W., J.G., J.T.T., K.Z., M.H., N.J.J., N.K., N.R.Z., R.L., S.A., S.S., T.E.B., Y.B.-S., and Y.T.

Declaration of interests

The authors declare no competing interests.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
Motor cortex multiomic data	This paper	Zenodo: https://doi.org/10.5281/zenodo.15151311
Software and algorithms
Enhancer Benchmark	This paper	https://github.com/AllenInstitute/EnhancerBenchmark; Zenodo: https://doi.org/10.5281/zenodo.15243842
cisMultiDeep	This paper	https://github.com/ytanaka-bio/cisMultiDeep; Zenodo: https://doi.org/10.5281/zenodo.15243865
PeakRankR	This paper	https://github.com/AllenInstitute/PeakRankR; Zenodo: https://doi.org/10.5281/zenodo.15243876
CREsted	Kempynck et al.18	https://github.com/aertslab/CREsted; RRID:SCR_026617
pycisTopic	González-Blas et al.17	https://github.com/aertslab/pycisTopic; RRID:SCR_026618
SCENIC+	González-Blas et al.17	https://github.com/aertslab/scenicplus; RRID:SCR_026702

Method details

Single nucleus molecular profiling

10x multiome ATAC + Gene Expression and methyl-3C-sequencing (snm3C-seq) experiments were carried out on the same tissue samples from human, macaque, marmoset, and mouse M1, as described.¹³

In vivo enhancer validation data

Detailed experimental methods are described in the companion paper⁶ and other recent viral tools publications.^4,5,25

Primary screen scoring

Each enhancer vector was screened and scored based on the labeling pattern it produced across the entire brain, with additional emphasis on cortical populations. First, each region of the brain where labeling of cell somata was observed was manually scored based on the labeling brightness and density, classifying each into either low or high. In addition, we created 11 categories of cell populations within the neocortex that could be visually distinguished one from the other. Whenever labeling was observed in one or more of these cortical populations, each population was individually evaluated based on its own brightness and density. Whereas brightness was classified based on whether the labeling was stronger or weaker than the common brightness observed across all experiments, density was evaluated based on the expected density of cells for each of the scored regions or populations, using the nuclear markers as reference. To determine target specificity, we aligned each target cell population with the labeled population which best matches its known anatomical location, distribution, and morphological characteristics. We determined an enhancer to be “On-Target” if the target population aligned with the labeled population, “Mixed-Target” if labeling was observed in populations other populations, in addition to the target one, “Off-Target” if labeling was observed exclusively in population/s other than the target population, and “No-Labeling” if no labeling was observed in the neocortex, regardless of whether labeling was observed in other brain regions.

Cell-type enhancer activity quantified by single cell RNA-seq

SSv4 data for cortical enhancers were generated from mouse primary visual cortex (V1/VISp) and mapped to the Allen Institute AIT2.1.1 VISp taxonomy as described in Ben-Simon et al.⁶ The data were reused in this study to establish ground truth cell type specificity in combination with primary screening expression analysis.

Quantification and statistical analysis

Benchmark metrics of enhancer prioritization methods

We defined an interpretable benchmark metric based on the experimentally validated enhancer collection reported in Ben-Simon et al.⁶ This benchmark metric is available to the community at: https://github.com/AllenInstitute/EnhancerBenchmark. The benchmark metric is a composite of an epifluorescence imaging score and SSv4 quantifications score per enhancer that captures different properties of enhancer activity. Epifluorescence validation of an enhancer captures broad patterns such as cortical layer labeling and provides cell-type specificity when cellular morphologies are known. Additionally, the intensity of SYFP2 fluorescence provides a measure of enhancer strength. We scored each validated enhancer based on the cell-type-specific rank and placed more weight on strong enhancers being enriched in the top ranks. However, the epifluorescence validation does not provide a quantitative measure of specificity for each cell type targeted by the enhancer virus. To achieve this, we used SSv4 to quantify the transcriptome of cells with high SYFP2 fluorescence and mapped these cells to a cortical taxonomy to determine accurate abundances of each cell type targeted by the enhancer.

The epifluorescence metric was designed to be minimized when teams arranged On-Target enhancers in the top ranks per-cell type and Mixed Target enhancers relatively lower down the ranks as these enhancers both target the intended cell type as well as additional unintended cell types. Negative enhancer categories including Off-Target and No-Labeling enhancers optimize the metric when placed at the bottom of the ranked lists per-celltype. For each validated enhancer $E_i\in E_{validated}$ in the ranked list across cell types we multiply the predicted rank $R_{E_i}$ of the enhancer with an indicator variable $I_{Category}$ which encodes enhancer categories (On-Target, Mixed-Target, Off-Target or No-Labeling). Then the metric is weighted by an indicator variable $I_{Strength}$ encoding the strength of SYFP2 epifluorescence (Strong, Weak or None).

$${Epifluorescence}_{metric}=\sum _{E_i}^{E_{validated}}\left({Rank}_{E_i}\ast I_{Category{E}_i}\right)\ast I_{Strength{E}_i}$$ (Equation 1)

where:

$$I_{Category}=\{\begin{array}{ll}-1& On-Target\\ 0.5& Mixed-Target\\ 1& Off-Target|No-Labeling\end{array}$$

$$I_{Strength}=\{\begin{array}{cc}2& Strong\\ 1.5& Weak\\ 1& None\end{array}$$

The SSv4 metric was designed to be minimized and replaces epifluorescence strength $I_{Strength}$ with a quantification of cell-type specificity computed the fraction of cells labeled as the intended target cell type $SSv{4}_{E_i,celltyp{e}_{target}}\left[0-1\right]$. We then compute the SSv4 metric for each validated enhancer $E_i\in E_{validated}$ by multiplying the predicted rank $R_{E_i}$ with the associated category $I_{Category}$ and the fraction of cells labeled as the targeted cell type $SSv{4}_{E_i,celltyp{e}_{target}}$.

$${SSv4}_{metric}=\sum _{E_i}^{E_{validated}}\left({Rank}_{E_i}\ast I_{Category{E}_i}\right)\ast {SSv4}_{E_i,{celltype}_{target,}}$$ (Equation 2)

The composite benchmark score is then computed as the unweighted summation of Epi_metric and SSv4_metric.

$${Benchmark}_{metric,}={Epifluorescence}_{metric}+{SSv4}_{metric}$$

To define a normalized benchmark score ranging between 0 (worse) and 1 (better) we created an optimally ranked enhancer list per cell type and computed the associated benchmark metric. Then the benchmark metric score is divided by this optimal score and subtracted from 1 to achieve a normalized metric with the desired directionality.

We defined precision, recall and F1-score for each team based on the following measures: true positive (TP) (On-Target, Mixed-Target), false positive (FP) (Off-Target, No-Labeling), true negative (Off-Target, No-Labeling), false negative (On-Target, Mixed-Target).

To prevent overfitting during the challenge, we held back enhancers targeting excitatory cell types to ensure that no team could gain an advantage by learning patterns in the hidden enhancer validation data over the course of multiple challenge submissions.

Statistical quantification of benchmark metrics

To assess the significance of the ranking of the given methods we set up a statistical test in which we defined.

• (1)Null Hypothesis: The rank of the team is unstable and occurs uniformly across the possible ranks.
• (2)Alternative Hypothesis: The rank of the team is stable (e.g., significantly clustered around its observed rank).

In which significantly clustered is defined as +/− 2 ranks of the teams rank from the full benchmark score. p-value on rank stability per-team were then empirically computed by randomly subsampling to 90% the validated enhancer database, 10,000 times.

Recovery curves of functional enhancers

To assess the recovery of validated cell type specific enhancers based on the cell type specific rankings of each submission a recovery approach was used. In brief, the set of ground truth cell type specific enhancers was defined as the genomic regions (candidate enhancers) for which the specificity was classified as "On-Target". Mouse genomic coordinates were used for candidate enhancers originating from the mouse genome, and coordinates lifted over to the mouse genome were used for candidate enhancers originating from the human genome. The genomic regions in each cell type specific ranking were intersected with the ground truth cell type specific enhancers using pyranges identifying hits along the ranking (genomic regions in the ranking overlapping with multiple ground truth enhancers were only counted once). Recovery curves were drawn for each submission by calculating the cumulative sum of the union of hits along all cell type specific rankings per cell type. In cases where the ranking was shorter than 10,000 elements, the ranking was padded with non-hits up to a length of 10,000 elements. Normalized enrichment scores (NES) per cell type specific ranking per submission were calculated as the area under the curve of the recovery curve (AUC) up to the 1,000th element and dividing this by the AUC up to the 1,000th element of the average recovery curve of 100 random rankings.

Manual annotation of validated enhancers

We manually inspected a subset of the validated regions by classifying them into five categories: explainable positives, unexplainable positives, explainable negatives, unexplainable negatives, and undetermined. Explainable positives are defined as functional (both ‘strong’ and ‘weak’, and both ‘On-Target’ and ‘Mixed-Target’) enhancers that have either a strong accessibility peak in the scATAC-seq data, a strong prediction from the CREsted model, or both, in their intended target cell type(s). Unexplainable positives are functional enhancers that do not have a clear peak and/or prediction in their target cell types, but still show activity for those cell types, or they do have a strong peak and/or prediction in their targets, but show activity in other cell types. For validated enhancer candidates that did not show functionality (negatives), we identify explainable negatives as regions that either do not have a specific peak, a specific prediction, or both, in the target cell type. Unexplainable negatives have either a peak, prediction, or both in their target cell types, as well as the presence of positive contribution scores in one or more motifs obtained from the CREsted model. Undetermined regions do not contain enough decisive information to classify them into one of the other four categories, often because of their target cell types not being included in the original dataset.

Precision-recall curves of different modalities

To compare the different modalities directly on the 677 validated enhancers, we scored each enhancer per modality by taking per cell type the score over the sum of all scores. We generated a target binary matrix based on the On- and Mixed-Target and No-Labeling enhancers, combined with the SSv4 results, to label the targeted cell types on enhancer functionality, and calculated per cell type the precision and recall over different prediction thresholds. We excluded off-target enhancers because of uncertainty for the actual targeted cell types.

For the scATAC-seq and H3k27ac data, we took the average counts over the exact region. The peak-scaled scATAC data was obtained by scaling the peak heights per cell type through the normalization factor obtained from the CREsted package. We then applied the specificity metric to the resulting target vectors per modality.

For the CREsted model the enhancer regions were put in the center of a 2114 bp background region, since the model takes in a fixed 2114 bp sequence. Then, we took the predictions scores per enhancer and applied our specificity metric to obtain final scores. The CREsted model is available at https://crested.readthedocs.io/en/latest/models/biccn.html.

For the combined CREsted and scATAC prediction scores, we averaged their specificity scores and recalculated the average precision and recall.

Motif enrichment in on-target enhancers

We calculated the most frequently occurring patterns per cell type in On-Target enhancers with tfmodisco-lite, based on contribution scores obtained from the CREsted model (Figure S14). We then matched patterns across cell types with the CREsted pattern matching function crested.tl.modisco.process_patterns (sim_threshold = 4.25, trim_ic_threshold = 0.1) We used the frequency of the patterns per sequence as pattern_parameter per cell type, and plotted the results using the crested.pl.patterns.clustermap_with_pwm_logos function with the importance threshold set to 0.12.

Random forest modeling of enhancer features

To examine the importance of additional variables determining enhancer specificity, we modeled per enhancer metrics, including distance to the nearest transcription start site, mouse-human enhancer sequence conservation, and published ChIP seq data. ChIP seq data that overlapped enhancers were found using the GenomicRanges package in R, and all overlapping bins for each enhancer were summed to get values used for downstream analysis. Random forest models were constructed after removing enhancers that were ‘Mixed Target’ using scikit-learn version 1.3.0. The “% matching bps” measure was determined by calculating the percentage of the initial mouse enhancer length that is preserved upon examination of the best BLAST alignment with the human genome. For random forest model development, we used a 70/30 split for training and a held-out test set, respectively. Prior to testing on the held-out test set, models were constructed and validated using 10-fold cross-validation using GridSearchCV. The importance and statistical significance of variables is reported in Figure 3D and S10. Statistical significance was calculated using ANOVA, followed by a Tukey post hoc test.

Stein Aerts team methods

For the blind challenge, we first preprocessed the raw sequencing files and re-analyzed the human and mouse scATAC-seq datasets using cisTopic (based on fragment counts; see details below). This validated the provided cell type annotation with respect to 2D dimensionality reduction (Figure S19A). Next, we employed four different strategies to generate 4 types of rankings (each type may have one or more variants), each containing 19 ranked lists of the top 10,000 scoring intervals, for each of the given 19 cell types. The first strategy uses only cell-type specific chromatin accessibility to prioritize candidate enhancers per cell type by calculating per region the Gini index over the accessibility profile of all cell types. As a variant of this approach we took the pseudobulk profiles per cell type are derived from three datasets,^13,17,30 instead of only the provided mouse dataset, followed by the application of a peak-scaling factor (see details below). These ATAC-only models resulted in very high performances according to the BICCN challenge scoring (see Figure S1, Table S2): 0.4027 for the single dataset ATAC rankings, the best ranking in the entire challenge, and 0.4052 for the merged datasets.

The remaining three strategies all use sequence-based models. In the second strategy (Figure S19C), we trained two types of convolutional neural network models to predict, from the sequence of the genomic interval as input, chromatin accessibility across cell types: CREsted peak regression and topic classification. We also trained analogous models on the human scATAC-seq clusters and used them to score and rank the mouse genomic regions, followed by an order-statistics integration of the mouse- and human-based rankings (see details below). Among the sequence-based models, the best performing model is a peak regression model (referred to in main text as Aerts CREsted), trained only on the mouse scATAC-seq data. The 10k rankings sorted on Gini score based on the model’s predictions had a final score of 0.3556. In the third strategy, we combined the scATAC-seq and scRNA-seq data and inferred enhancer gene regulatory networks (eGRNs) using SCENIC+.¹⁷ This provides, as one of its outputs, triplet scores for TF-region-gene trios (Figure S19B). This resulted in the best score in the challenge (Table S2). In the fourth strategy, we generated SHAP-based explainability profiles for each candidate region, followed by seqlet clustering, and re-annotation of the regions using TFMoDisco-derived patterns³¹ (Figure S19D). Ranking was then performed using a heterogeneity score that measures the complexity of these patterns in each sequence (see details below). Combined with the scATAC-seq rankings, this resulted in the second highest score in the challenge (Table S2). Finally, we also combined multiple strategies, using order-statistics, but overall these ensemble rankings did not outperform the ATAC-only or the CREsted approaches.

ATAC-based rankings

We investigated the performance of rankings purely based on scATAC-seq data. We processed fragment files through pycisTopic¹⁷:, a tool that aggregates counts in cells per cell type (subclass level) and CPM-normalizes them to account for the variability in cell numbers per type. For all consensus peaks, we obtained the mean accessibility per cell type from the pseudobulked cell-type-specific accessibility tracks. We ranked regions based on the Gini index of their accessibility profile over all cell types to obtain the highest and most specific peaks per cell type. Using the Gini index only provides one value per region, so we assigned that value to the cell type with the highest peak value and gave a zero score to all the other cell types in that region. To further optimize this approach, we augmented the scATAC-seq data by merging the provided mouse dataset with publicly available mouse motor cortex datasets.^17,30 We merged datasets by manually matching corresponding cell types, and weighted peak heights across datasets by using the number of cells per matched cell type.

Normalization of peak heights across cell types

Since peak heights across cell types are not always in the same scale after count normalization because of a potential difference in the total amount of accessible regions, we implemented an additional normalization method to create peak-scaled scATAC tracks across cell types. We used the CREsted peak normalization functionality with default parameters, a method that is aimed to alleviate this problem. This method takes thetop 1% (based on peak strength/height) of peaks per cell type, and only retains peaks which are generally accessible (Gini index <0.25). This is done based on the assumption that strong generally accessible peaks should be in a similar range across all cell types. Based on those strong, generally accessible peaks, we compared the mean peak height per cell type and calculated scalars per cell type that ensure all mean peak heights become equal. These scalars are multiplied with the peak heights of their corresponding cell types to put peaks across different cell types in a more comparable range.

Sequence-based deep learning models

We trained two types of convolutional-based sequence-based accessibility prediction models from the CREsted package. The first is a peak regression model, a 9-layer convolutional neural network (CNN) model, trained on scATAC-seq data for chromatin accessibility analysis. It uses 2,114 bp DNA sequences as input and predicts the average accessibility signal per cell type on the center 1,000 bp of that region, inspired by ChromBPNet.³² As a loss function for these models, we chose to take the sum of the mean squared error (MSE) and the negative cosine similarity (‘CosineMSE Loss’ in CREsted). We first pretrained these models on all the given consensus peaks, and we further finetuned them on differentially accessible regions (DARs). DARs were determined through different methods, through regions which had a ratio higher than 2 between their highest and second highest peak scalar value, or by taking regions which had a Gini index higher than the mean plus one standard deviation of all regions in the peak set. The finetuning ensures that the models learn cell-type specific features, since most consensus peaks are not specifically accessible in a low number of cell types. A second approach is topic classification, which was the modeling method used in previously published models.^9,33,34,35 We analyzed the scATAC-seq data using pycisTopic¹⁷ to obtain topics per region required for training such models. Transfer learning to DARs made it possible to obtain cell-type-specific predictions, as was done in Hecker & Kempynck et al. 2024.⁹

We mostly focused on the mouse scATAC-seq data for training our model. For the regression models, we also trained a model on the human scATAC-seq data and did cross-species predictions to obtain the mouse rankings required for the challenge. For the topic models, we trained a model on all four species. We did topic modeling after integrating all four datasets, to obtain topics representing shared regulatory features. We finetuned that model on mouse DARs.

We used two methods of ranking all the consensus peaks for these models. The first one was using the Gini index on the prediction scores over the cell types per region. The second one was calculating per region, for each prediction per cell type, the product of the difference between a given cell type prediction and the highest prediction in any other cell type, and the ratio between them. This gave a specificity ranking per cell type, per region, which could be sorted to generate global rankings.

Pattern-based enhancer scoring

We reasoned that regions with heterogeneous motif content were more likely to be functional enhancers. For this purpose, we generated SHapley Additive exPlanations (SHAP) values for the top 10,000 regions per cell type from the CRESted model, generating explanations for the prediction score of that cell type. Of these, the top 5,000 regions per cell type were used to identify recurring, important patterns using the tfmodisco-lite package.³¹ To rank regions based on motif diversity, we calculated the Shannon diversity index based on the pattern hits. This index was multiplied by a signal over noise metric that was defined, per region, as the number of seqlets (short stretches of DNA with a high importance score) having a pattern hit divided by all seqlets.

Scenic+ triplet scores

We ran SCENIC+ on the provided mouse multiome dataset (scRNA-seq + scATAC-seq) using default parameters.¹⁷ Based on these results we generated a ranking of all transcription factor (TF)-region-gene triplets, which is the aggregated ranking (see below) of the TF- and region-to-gene importance scores and TF-to-region ranking based on the motif score of all motifs annotated to that TF. To make this ranking cell-type-specific we combined it with the gini-based cell-type-specific ATAC ranking.

Aggregate rankings of multiple methods

To combine multiple rankings of different methods, we used OrderStatistics.³⁶ In brief, rankings were generated based on the score of each method, ties were broken by assigning incremental rankings to tied regions based on the order in which they happen to occur. Next, rank-ratios were calculated as the ranking divided by the number of regions in each ranking and combined in a single ranking using the formula described in [ref] and implemented in the SCENIC+ package.¹⁷

ArchR method

ArchR was run with default parameters following the tutorial: https://www.archrproject.com/bookdown. Briefly, we used ArchR to: (1) create pseudobulk replicates in which careful selection of background cells occurs; (2) assign of cell type labels to the ATAC-seq nuclei based on the RNA-seq component of the 10x Multiome; (3) call peaks for each cell type group; (4) use the getMarkerFeatures function to identify cell type specific peaks and associated summary stats; (5) organize 10,000 putative cell -ype-specific peaks for each cell type by sorting by log2FC > 2 and FDR <0.05.

PeakRankR method

PeakRankR is an R function (https://github.com/AllenInstitute/PeakRankR) designed for prioritizing and ranking cell-type-specific peaks based on chromatin accessibility data. PeakRankR seeks to identify a minimal set of ATAC-seq features – namely, specificity, magnitude, sensitivity, and shape-related features like modality, skewness, and kurtosis (see details below) – for each enhancer. Together, these features can be used to prioritize peaks simply and efficiently. To assess the performance of PeakRankR, we investigated how well the method organized On-Target enhancers in the top predicted ranks and Mixed-Target, Off-Target and No-Labeling enhancers in lower ranks. Scores for all PeakRankR submissions, both pre and post challenge, are included in the challenge GitHub repository.

To evaluate the importance of each ATAC-seq feature in predicting the validated enhancers, we re-prioritized and evaluated peaks for the universe peak set using each PeakRankR feature separately. We find that specificity and magnitude using the bigWigSummary tool from UCSC optimize benchmark scores most efficiently, indicating that these aspects of peaks are the most important for prioritizing functional enhancers. To determine whether peak shape impacts performance we additionally included modality, skewness, and kurtosis (see details below) in a revised PeakRankR challenge submission. Inclusion of these additional features did not improve upon the model leveraging specificity, magnitude, and sensitivity. Identifying the most parsimonious set of features is needed for solving an optimization problem.

Enhancer scoring

PeakRankR employs a linear model approach to calculate the combined effects of positive features associated with a given peak in a cell type, which are then used to rank the peaks.

The features computed by PeakRankR include.

• (1)Specificity: Calculated using an R version of `multiBigWigSummary` from deeptools (https://deeptools.readthedocs.io/en/develop/).
• (2)Sensitivity: Determined by counting the number of cell types that have a peak at the genomic coordinates of the enhancer.
• (3)Magnitude: Estimated using MACS2 to assess the ATAC-Seq signal at the genomic coordinates of the enhancer for the cell type of interest.

These three features are combined to generate a score for each peak as a weighted linear sum of the features:

$${PeakRankR}_{Score}={\omega }_{Specificity}{Specificity}_{{Peak}_i}+{\omega }_{Sensitivity}{Sensitivity}_{{Peak}_i}+{\omega }_{Magnitude}{Magnitude}_{{Peak}_i}$$

where w stands for the weight of each feature. By default, all weights are set to 1, indicating equal importance for each feature.

Function: Peak_RankeR (tsv_file_df, group_by_column_name, background_group, bw_table, rank_sum, weights)

• (1)tsv_file_df: A tab-separated data frame of enhancer coordinates and peak groups.
• (2)group_by_column_name: The column name in `tsv_file_df` containing the groups of the enhancers.
• (3)background_group: The group against which the enhancer should be prioritized for the group of interest.
• (4)bw_table: A two-column table with the group BigWig file paths and group names.
• (5)rank_sum: If TRUE, the sum of the feature scores is included in the output file.
• (6)weights: Coefficients for the features, representing the influence of each predictor on the peak rank.

The function returns an object that includes the input peak coordinates, peak rank, and their corresponding scores.

• (1)peakRankR_rank: Prioritized rank assigned to each peak
• (2)rank_sum: Aggregate sum of scores for the various peak features

We enhanced PeakRankR to include additional features that characterize the shape of a peak: (1) Skew, which assesses the asymmetry of read pileups in an enhancer, with higher rankings given to those closer to symmetry (skewness near 0); (2) Kurtosis, which evaluates the dispersion of reads between the center and tails of an enhancer, with a higher rank for greater dispersion; (3) Modality, indicating the number of peaks within an enhancer, with unimodal peaks ranked higher. These features were calculated using functions from the `modes` package in R (https://www.rdocumentation.org/packages/modes/versions/0.7.0).

Jesse Gillis team method

We employed a multi-modal approach to predict cell-type-specific enhancers in the mouse brain, utilizing three distinct data types: single-cell RNA sequencing (scRNA-seq), single-cell Assay for Transposase-Accessible Chromatin sequencing (scATAC-seq), and meta-Hi-C (Figure S20). For scATAC-seq data, we obtained binarized MACS2 peak scores from three publicly available sources.^7,13,30 These scores were then z-scored at each genomic bin to capture cell-type specificity accurately. To enhance the power of scATAC-seq data further, we aggregated the binarized peaks from the three scATAC-seq sources to create a robust scATAC-seq matrix. Again, this matrix was z-scored at each bin to enhance cell-type specificity.

In the case of meta-Hi-C, we used our in-house method to obtain cell-type-specific contact vectors by combining meta-Hi-C data with scRNA-seq marker genes. Previously, we aggregated hundreds of Hi-C contact matrices for mouse Hi-C to generate meta-Hi-C maps available at https://labshare.cshl.edu/shares/gillislab/resource/HiC/.³⁷ Similarly, we utilized scRNA-seq data to obtain reliable markers for each brain cell type in mice.³⁸ The cell-type-specific profile was derived by computing the mean of the top markers for each cell type and smoothing the Hi-C contact vector using various marker subsets. The resulting meta-Hi-C matrix was z-scored at each bin to enhance cell-type specificity. Only intra-chromosomal contact matrices were used for this analysis. The Hi-C vectors are slightly updated from the time of submission to the challenge; however, this does not alter the validation results significantly.

To integrate the meta-ATAC-seq and meta-Hi-C data, we multiplied the ATAC-seq peak scores with the Hi-C contact scores for each cell type, resulting in the metaATAC X metaHi-C matrix. This matrix was also z-scored at each bin to capture cell-type-specific enhancers accurately.

Evaluation of models

We assessed the accuracy of our methods predictions using a set of experimentally validated enhancers from Ben-Simon et al.⁶ Our evaluation focused on determining the presence of cell-type-specific enhancers based on evaluating accessibility and activity per cell type.

We evaluated our methods against the validation set using the area under the Receiver Operating Characteristic (AUROC) curve and Precision-Recall (PR) curve as performance metrics for this multi-label classification task (Figure S21). Higher values of AUROC and PR area indicate stronger performance for each of our predictive models. We find that meta-ATAC-seq demonstrates marginally superior performance compared to leveraging individual ATAC-seq experiments, showcasing the predictive capability of robust ATAC-seq signals in identifying cell-type-specific enhancers. Interestingly, metaATAC combined with metaHi-C showed a marginal improvement over meta-ATAC-seq in PR curves. This suggests that incorporating ATAC-seq data along with contact density at enhancer regions is an important feature to enhancer prediction. Our finding aligns with observations from analogous activity by contact (ABC)²³ models for predicting gene expression. However, the ATAC-seq signal itself remains the most important feature when predicting cell-type specificity of enhancers. Additionally, we examined the AUC and AUPRC scores for each subclass and found that predicting enhancer specificity for non-neuron subclasses is relatively easier compared to neurons.

Kai Zhang team method

We developed a machine learning algorithm leveraging the pre-trained Enformer model¹⁹ to predict gene expression from DNA sequence and chromatin accessibility around TSS regions (Figure S22). Using the genome annotation downloaded from Gencode, for each gene we extracted the 196,608 bp DNA sequences centered around its TSS. We then applied the Enformer model¹⁹ to these sequences to derive sequence embeddings, followed by attention pooling to further reduce the dimensionality. Consequently, the processed sequences were represented as 896 vectors, each with 48 dimensions, corresponding to 128-bp segments of the initial sequence. Additionally, we quantified the ATAC-seq fragments overlapping these segments, adjusting for reads per kilobase million (RPKM).

To identify candidate enhancers, we developed a machine learning framework that integrates ATAC-seq signals and the aforementioned sequence embeddings to predict gene expression profiles. We first transformed normalized counts of ATAC-seq fragments into four-dimensional vectors using convolutional and self-attention layers. These ATAC embeddings were then merged with sequence embeddings and fed into a sequence of linear layers aimed at predicting levels of gene expression.

To train the model, we partitioned the data into training, validation, and testing sets using an 80:10:10 split. We employed the Adam optimizer with a learning rate of 0.0001 and trained the model over 10 epochs, after which the model’s performance plateaued.

To determine the enhancer activity score for a specific region, we modified its ATAC signal by setting its count value to zero. The trained model was then used to make two predictions: one for the original (unmasked) input and another for the modified (masked) input. The enhancer activity score was calculated as the difference in the model’s predicted gene expression between these two inputs, serving as an indicator of the ATAC signal’s impact on the expression of adjacent genes. For each cell type, the top 10,000 highest scored peaks were chosen for submission.

Evaluation of model

Our model achieved great accuracy in predicting gene expression at varying abundance (PCC = 0.91, Figure S23A). To identify candidate enhancers, we performed in-silico perturbation of chromatin accessibility at each ATAC peak, and used the model to compute the changes of predicted gene expression before and after perturbation. These values were named enhancer scores. We used the experimentally validated cell-type-specific enhancers to assess the accuracy of our predictions (Figure S23B). We find that the average AUC in all cell types is 0.52. We selected the top 10,000 peaks with the highest enhancer scores as candidate enhancers in each of the 19 cell types. We found out that this set covered about 18% of the validated enhancers, and the coverage increased to 84% when we retained the top 100,000 peaks from each cell type (Figure S23C)

To further evaluate the correlation between enhancers and other information of peaks, we selected mouse enhancers with cell type specific ATAC-seq, identifying a total of414 enhancers. We calculated the AUC for these enhancers using ABC score (Figure S23D), H3K27ac signal (Figure S23E), and ATAC signal (Figure S23F) of peaks for different cell types, giving an average AUC of 0.82, 0.82, and 0.93, respectively. Of all the validation enhancers, 328 enhancers were in the regions around genes’ TSS.

Yoshiaki Tanaka team method

The cell type is determined by expression of a specific set of genes that are governed by cis-regulatory elements (CREs), such as promoters and enhancers. CREs are characterized with open chromatin structure, in which transcription factors (TFs) are accessible. TF-bound CREs physically contact the target genes by forming chromatin loops, and transfer the regulatory information to the genes.³⁹ Importantly, the expression patterns of the cell-type-specific genes and TFs are conserved across species,^40,41 whereas cell-type-specific CREs are more susceptible to evolutionary divergence,^42,43 although some highly regulatory CREs are conserved.⁴⁴ In addition, the cell-type-specific gene expression also coincides with epigenetic modifications, such as DNA methylation (mCG and mCH). In particular, recent studies indicated that intragenic DNA methylation is negatively correlated with the gene expression.^45,46 These observations suggest that the cell-type-specific transcriptional programs are associated with various determinants, including conservation, chromatin looping, and open chromatin, and the framework to integrate such multiple omics data is essential.

Here, we introduce a pipeline, cisMultiDeep (Figure S24), that employs automatically-tuned deep learning with Shapley Additive exPlanation (SHAP) feature importance assessment in cross-species single-cell multi-omics data: RNA, ATAC, mCG, mCH, and Hi-C. First, this method identifies ‘conserved’ cell-type-specific genes from RNA, mCG, and mCH profiles. Subsequently, ‘conserved’ and ‘non-conserved’ cell-type-specific CREs are identified from ATAC profiles and linked with ‘conserved’ cell-type-specific genes using deep learning. Finally, the contribution of each CRE to the cell-type-specific gene expression is assessed by SHAP value and Hi-C contact map. cisMultiDeep is a repository (https://github.com/ytanaka-bio/cisMultiDeep) of R and Python scripts and command lines that identify functional CREs from single-cell multi-omics profiles.

Given high conservation of the cell-type-specific genes, we first obtained an orthologous gene list from Biomart (https://ensembl.org/info/data/biomart/).⁴⁷ On the other hand, the peak conservation was defined by UCSC LiftOver function in KentUtils and Bedtools (v2.30.0).⁴⁸ Subsequently, the cell type specificity in each gene and peak was estimated in RNA, mCG, mCH, and ATAC profiles by Wilcoxon rank-sum test. Here, the cell type specificity was calculated only in orthologous genes, whereas both conserved and non-conserved peaks were used for the assessment of the cell type specificity. In each cell type, the top 1,000 genes and 10,000 peaks were selected for subsequent deep learning analyses.

To ask if the selected genes/peaks are sufficient to define the cell types, we employed automatically tuned deep neural network that was designed by Tensorflow python library (v2.9.0) with Keras Tuner API (v1.1.2).^49,50 Briefly, at first, dimensionality of the input data (RNA, mCG, mCH, and ATAC profiles) was reduced into 200 by principal component analysis (PCA). Then, the sequential neural network model was built with seven tunable hyperparameters: i) the number of layers (2–10 with increment of 1), ii) the number of nodes in hidden layers (50–500 with increment of 50), iii) dropout rates (0–0.5 with increment of 0.1), iv) activation functions (e.g., sigmoid), v) optimizers (e.g., Adam), vi) learning rate (e.g., 1e−1, 1e−2, 1e−3, 1e−4, 1e−5), and vii) loss functions (e.g., mean squared error loss function). Once the neural network model was optimized, mean absolute SHAP value, which represents the impact of each gene or peak on the cell type determination, was calculated by DeepSHAP that is a technique that can handle the complex and non-linear interactions across features and is optimized to calculate SHAP value for deep neural network.⁵¹

Chromatin looping enables distal CREs to contact their target genes. Here, we hypothesized that the cell-type-specific CREs are physically contacted with various cell-type-specific genes. Thus, we ranked the peaks by the sum of the mean absolute SHAP values of the contacted genes by Hi-C loop. If the peak is conserved, we also added up the sum of the mean absolute SHAP values in other species.

Preparation of gene coordinates and orthologous gene list

Gene coordinates (GTF file) in human, macaque, marmoset, and mouse were obtained from EBI (Gencode v33), Ensembl (Release 110), UCSC (RefSeq), and EBI (Gencode M22) database, respectively. GTF files were then converted into BED format file with removing duplicated genes by a custom script. Orthologous gene pairs across the species were obtained from Biomart (https://ensembl.org/info/data/biomart/).⁴⁷

Classification of conserved and non-converted peaks

Peak conservation was defined by cross-species mapping of genomic coordinates. Briefly, chain format files of mouse with human (mm10ToHg38), macaque (mm10ToRheMac10), and marmoset (mm10ToCalJac4) were first downloaded from UCSC Genome Browser. Using liftOver (KentUtils 453) and Bedtools (v2.30.0),⁴⁸ orthologous regions of mouse ATAC peaks were identified in the other species genomes. If orthologous regions were identified and overlapped with ATAC peaks in the other species, we defined these peaks as conserved peaks. If not, the peaks were defined as non-conserved peaks.

Identification of top cell-type-specific gene and peak

Cell type specificity in each gene (RNA, mCG, and mCH profile) and peak (ATAC profile) was estimated by Wilcoxon rank-sum test. Recent studies have suggested that the cell-type-specific CREs are not fully conserved,⁴³ whereas the expression patterns of the cell-type-specific genes are evolutionarily conserved.⁴¹ Thus, the cell type specificity calculation was conducted only in orthologous genes, whereas both conserved and non-conserved peaks was used for the calculation of the cell type specificity. Finally, we selected top 1,000 “conserved” genes and top 10,000 “conserved” and “non-conserved” peaks for subsequent deep learning analysis.

Parameter optimization of deep neural network

The contribution of genes and peaks to cell type identity was then evaluated by an automatically tuned deep neural network. Main python scripts were designed by Tensorflow (v2.9.0)⁴⁹ and Keras Tuner API (v1.1.2).⁵⁰ Briefly, at first, dimensionality of the input data (RNA, mCG, mCH, and ATAC profiles) was reduced into 200 by principal component analysis (PCA). The sequential neural network model was then built with seven tunable hyperparameters: i) the number of layers (2–10 with increment of 1), ii) the number of nodes in hidden layers (50–500 with increment of 50), iii) dropout rates (0–0.5 with increment of 0.1), iv) activation functions (e.g., sigmoid), v) optimizers (e.g., Adam), vi) learning rate (e.g., 1e−1, 1e−2, 1e−3, 1e−4, 1e−5), and vii) loss functions (e.g., mean squared error loss function). The parameter search was conducted with 50 trials. After the parameter optimization, the neural network model was trained with 80% training and 20% validation datasets with 10 epochs.

Ranking genes and peaks by SHAP values

After the model building, the contribution of each gene or peak on the cell type identity was estimated by mean absolute SHAP value. This value was calculated by DeepSHAP library (v0.2.1) that handles the complex and non-linear interactions across features and is optimized to calculate SHAP value for deep neural network.⁵¹ After SHAP value calculation, each peak was connected to genes Hi-C chromatin loops. If peak was physically contacted with certain genes by Hi-C loop, we summed the mean absolute SHAP values of the connected genes to the peak. In addition, if the peak is conserved, Hi-C-adjusted mean absolute SHAP value was added up to the peak. Finally, 10,000 mouse peaks were ranked by the sum of the mean absolute SHAP value.

Evaluation of model

To evaluate our method we first intersected the validated enhancer collection with our top 10,000 CRE list. Amongst the 677 validated enhancers 27 peaks are unique to humans and do not have liftOver mouse genomic coordinates and were removed from subsequent assessment (Figure S25A). In addition, 131 peaks were also removed, since their coordinates were not overlapped with any mouse ATAC peaks from Zemke et al. 2023.¹³ in the remaining 519 validated enhancers, only 92 (17.8%) were detected in our top 10,000 CRE list. If we expanded the per-cell type ranked list to top 50,000 we detect 333 (64.1%).

To dissect the characteristics of CREs predicted by our model, we analyzed: (1) conservation, (2) magnitude, (3) distance from TSS, (4) specificity, (5) H3K27ac, and (6) ABC score.²³ We found that our model preferentially detected the conserved peaks (p < 3.68e−275 by hypergeometric test) (Figure S25B) with high magnitude (p < 2.2e−16 by T test) (Figure S25C). We note that the magnitude of the validated peaks was much lower than that of the predicted CREs (p = 7.95e−10 by T test). Usually, ATAC peaks in proximal CREs to genes are likely to display higher magnitude than those in distant CREs. Thus, we also compared the distance of the predicted CREs from transcription start sites (TSSs), and found that our model preferentially identified proximal CREs, whereas many validated enhancers are distal CREs (p < 2.2e−16 by T test) (Figure S25D). Furthermore, the rank of our top 10,000 CRE list was positively correlated with specificity and H3K27ac signal, but not ABC score (Figure S25E). Taken together, these assessments indicated that our model preferentially detected cell-type-specific promoter elements and requires refinement to capture distal CREs.

Published: May 21, 2025