Ranking reprogramming factors for directed differentiation

Abstract

Transcription factor over-expression is a proven method for reprogramming cells to a desired cell type for regenerative medicine and therapeutic discovery. However, a general method for the identification of reprogramming factors to create an arbitrary cell type is an open problem. Here we examine the success rate of methods and data for directed differentiation by testing the ability of nine computational methods (CellNet, GarNet, EBSeq, AME, DREME, HOMER, KMAC, diffTF, and DeepAccess) to discover and rank candidate factors for eight target cell types with known reprogramming solutions. We compare methods that utilize gene expression, biological networks, and chromatin accessibility data and comprehensively test parameter and pre-processing of input data to optimize performance. We find the best factor identification methods can identify an average of 50–60% of reprogramming factors within the top 10 candidates, and methods that use chromatin accessibility perform the best. Among the chromatin accessibility methods, complex methods DeepAccess and diffTF have higher correlation with the ranked significance of transcription factor candidates within reprogramming protocols for differentiation. We provide evidence that AME and diffTF are optimal methods for transcription factor recovery which will allow for systematic prioritization of transcription factor candidates to aid in the design of novel reprogramming protocols.

Introduction

Our ability to precisely discover, define, and characterize cell types has improved with the advent of new molecular technologies in the past decade^1–4. Advances in single cell sequencing methods has even made it possible to define individual cell types by their expression^5–9 and chromatin accessibility profiles^10–12. The identification and characterization of cell types in health and disease has further illuminated the potential of regenerative medicine enabled by the ability reprogram cells into arbitrary types. In order to facilitate the identification of reprogramming factors that can be deployed with any cell type of interest, we have systematically compared computational methods to identify the most robust means for identification of potential reprogramming factors from gene expression and chromatin accessibility data, where we define successful identification of reprogramming factors as the identification of the target transcription factor motif or transcription factor.

Reprogramming strategies typically either employ small-molecules that target signaling pathways or transcription factor-based reprogramming^13–15. Transcription factor over-expression has been a successful method for reprogramming cells such as fibroblasts or stem cells into many specialized cell types¹⁶. Transcription factor protocols are also advantageous as they can decrease protocol time and increase efficiency¹⁷. Identification of reprogramming transcription factors has generally combined expert knowledge and large-scale screens to test many possibilities and experimentally identify a set of key regulatory factors. For example, the combination of transcription factors that induce pluripotency were discovered using a candidate set of 24 transcription factors curated from the literature that were narrowed down to four factors using successive leave-one-out experiments to reprogram fibroblasts into stem cells¹⁸.

The ability to easily generate gene expression and chromatin accessibility data from primary cells has led to the development of computational methods that can potentially discover reprogramming factors in the absence of extensive developmental studies. For example, though it is not their primary purpose, methods for motif discovery and differential expression analysis can be used to rank transcription factors or transcription factor motifs as candidates for reprogramming protocols. We include these methods in our comparison of methods specifically developed for identifying reprogramming factors through modeling gene regulatory networks. Our comparison uses the same source data for all methods where possible to provide principled results.

Our analysis in this paper focuses on methods that can be readily applied to new data and thus we were not able to comprehensively evaluate certain methods. For example, Inferelator¹⁹, Mogrify²⁰, IRENE²¹, and CellNet^22,23 rely upon large repositories of cell-type specific data. We did not include Mogrify, Inferelator, and IRENE in our study because they cannot be readily applied to identify reprogramming factors for new cell types. Moreover, the biological networks built for these methods are for human whereas we use mouse data in this study.

We comprehensively and uniformly evaluate nine methods: CellNet^22–24, GarNet^25,26, EBSeq²⁷, AME²⁸, DREME²⁹, HOMER³⁰, KMAC³¹, DeepAccess³², and diffTF³³ on their ability to recover and rank eight known sets reprogramming transcription factors and factor motifs.

Gene expression data have been a long-standing basis for the identification of reprogramming factors^20,34–37. The differential expression of transcription factors in a target cell type is an indicator of their potential as reprogramming factors. However, there may be both biological and experimental confounders in using expression data for identification of reprogramming factors. For example, it has been shown that sensory neurons of the dorsal root ganglia co-express many transcription factors as they are being specified but change their expression patterns to only express select transcription factors as they mature³⁸. Additionally, gene expression data do not provide information about whether proteins are present and actively binding DNA to control transcription. When gene expression datasets generated in different studies are used together, experimental confounders arise such as the differences in measurements that result from nuclear or whole cell mRNA, or the use of different RNA amplification methods for sequencing. Network methods such as CellNet^22–24 and Inferelator^19,39 have been shown to better prioritize transcription factor candidates through the incorporation of gene expression and transcription factor-gene interaction networks, but rely on massive repositories of RNA-seq data measured from perturbed cells to confidently learn these biological networks. As a result, these network-based methods are not generally applicable to novel data from a small number of experiments.

Motif enrichment and discovery methods, such as HOMER, AME, DREME, and KMAC, can be used to identify candidate reprogramming factors by characterizing the over-represented transcription factor binding motifs in accessible chromatin in a target cell type. Other methods learn to predict the relationship between chromatin accessibility and DNA sequence (DeepAccess) or measure the differential accessibility of transcription factor sites (diffTF). ATAC-seq can measure the accessibility of chromatin in small cell populations and thus can be used with cells derived from in vivo samples, while DNase-seq requires a larger number of cells^40,41. DNA sequences that are over-represented in accessible genomic regions typically contain informative transcription factor binding motifs. Known motif enrichment or de novo motif discovery methods are commonly applied to ATAC-seq data, and require the selection of differentially accessible regions in the starting and target cell types. In using these methods, parameters such as choice of accessibility or histone mark genomic data, number and choice of genomic regions from target cells, and choice of background sequences using shuffled or natural genomic sequences must be chosen carefully to the generate the best quality results. GarNet^25,26 combines ATAC-seq and RNA-seq with the goal of identifying transcription factors which are likely to control differential gene expression, though methods that combine gene expression and chromatin accessibility may be subject to the same biological and experimental confounders faced by methods which rank transcription factors using differential gene expression.

Overall, we find that methods that use chromatin accessibility have superior reprogramming factor discovery performance when compared with gene expression methods. We identify optimal accessible region selection strategies for sequence-based methods and using these optimal strategies, we find that AME and diffTF have the most robust performance for transcription factor recovery. We also find that histone mark and EP300 annotation do not significantly improve transcription factor recovery which suggests accessibility alone is sufficient to identify reprogramming factors for new target cell types.

Results

Chosen methods use gene expression or chromatin accessibility

The nine methods we evaluated for reprogramming factor discovery used gene expression (EBSeq, CellNet), chromatin accessibility (DREME, AME, Homer, KMAC, diffTF, DeepAccess), or a combination of the two (GarNet) to identify transcription factors as reprogramming candidates based on their popularity and spanning a diverse set of methodological approaches for reprograming factor discovery. We evaluated the methods on their ability to identify transcription factors that reprogram cells from a starting cell type (stem cells or fibroblasts), to eight possible target cell types (induced pluripotent stem cells, skeletal muscle cells, cardiomyocytes, definitive endoderm cells, hepatocyte cells, pancreatic beta cells, dopaminergic midbrain neurons, or spinal motor neurons) (Fig. 1a). We started with RNA-seq and ATAC-seq data from primary cells with the exception of definitive endoderm where the endoderm cells were differentiated with small molecules⁴², and the basis of our evaluation was the reproduction of known reprogramming factors for each target cell type (Fig. 1b). Both RNA-seq and ATAC-seq was collected from the same lab for each cell type with the exception of (Table S1) stem cells where our ATAC-seq and RNA-seq were from different sources. We uniformly processed the RNA-seq and ATAC-seq data (Methods).

Figure 1. Identifying transcription factors that reprogram starting cells to target cell types.

a, Two common starting cell types, fibroblasts (skin cells) and pluripotent stem cells can be reprogrammed to 8 cell types through over-expression of reprogramming transcription factors. b, Transcription factors that have been previously implicated in reprogramming protocols. c, Methods for identifying reprogramming transcription factors from gene expression (RNA-seq) or chromatin accessibility (ATAC-seq). Output of these methods is either a ranked list of transcription factors or a rank representation of DNA binding domain of a transcription factor by a PWM. Most methods can be readily applied to new data, except CellNet which requires many samples from the same cell type to build cell type-specific regulatory networks. All methods that use chromatin accessibility data take in accessible regions, typically generated from a peak calling algorithm such as MACS2 as input. In contrast to more traditional approaches for identifying transcription factors, diffTF models more complex regulation by taking into account read counts, and DeepAccess can also predict accessibility of DNA sequences unseen in the genome for other downstream analyses such as cross-species prediction. Methods operate under different assumptions of what are important features for ranking reprogramming factors.

We used EBseq and CellNet as our methods for discovering reprogramming factors from gene expression data. EBseq²⁷ ranks transcription factor differential expression between a starting and target cell type (Fig. 1c). CellNet^22–24 ranks candidate factors based on their importance in a cell type specific regulatory network derived from perturbation-based gene expression datasets. We attempted to train a new CellNet model using our data from the replicated RNA-seq samples from our eight target cell types, but found the experimental data did not represent sufficiently informative perturbations to build cell type specific regulatory networks. Thus, we evaluated CellNet for five cell types (stem cell, hepatocyte, cardiomyocyte, skeletal muscle) with preexisting regulatory networks. Consequently, CellNet did not fit our criteria for methods which can be used to predict reprogramming transcription factors from new, perhaps difficult to collect, cell types.

We examined if the combination of differential gene expression and chromatin accessibility could improve reprogramming factor prediction with GarNet^25,26. Using a list of transcription factor putative binding sites, which can be derived from motif scanning or ChIP-seq, GarNet assigns binding sites that are present in accessible regions to their closest gene within a constrained distance. It then uses transcription factor scores per genomic site (such as ChIP binding signal or the strength of a motif match) to train a regression model to predict differential gene expression. The weights associated with a particular motif then indicate its potential importance in driving differential gene expression.

Finally, we explored methods for transcription factor motif discovery from chromatin accessibility data. We selected methods that are both widely adopted and varied in approach. From the MEME suite, DREME²⁹ performs de novo motif discovery by first identifying seed sequences as the top 100 significantly enriched words relative to background sequences, then performs a beam search based on the seed sequences to generalize words to PWMs. AME⁴³ performs discriminative motif enrichment from an existing database of motifs represented by PWMs. HOMER³⁰ first identifies the most globally enriched oligos (similar to DREME) then transforms them into PWMs which get further optimized with a sensitive local optimization algorithm. KMAC³¹ also performs de novo motif discovery, but uses a k-mer representation of DNA binding motifs that was shown to better represent actual DNA binding sites by including features such as flanking nucleotides. For the de novo motif discovery methods (DREME, HOMER and KMAC), their output is the form of a list of position weight matrices representing enriched DNA binding motifs. To link this to transcription factor activity, we used Tomtom to determine whether each PWM significantly matched one or more known transcription factor motifs (Methods). For all chromatin-based methods which rely on transcription factor motif identification, we count an identification of the correct motif family as a successful recovery of the transcription factor motif, and provide a mapping from motif family to transcription factors as Supplemental Material and on our website. As motif discovery methods rely on input choices that can affect their performance, we extensively tested (1) the number of accessible regions input, (2) if differential regions were from the starting cell type or the top most accessible regions, and (3) the choice of background sequences.

In addition to traditional motif discovery methods, we applied two complex methods for evaluating motif enrichment, diffTF and DeepAccess. We define complex methods to be those that incorporate read count information or model more complex regulatory grammars. Both complex methods also require access to high-performance computing. diffTF³³ uses the accessibility sequencing read counts within putative transcription factor binding sites (either from motif instances or ChIP-seq) to estimate the fold change in accessibility between two conditions for a given transcription factor. We also tested a deep learning estimate of transcription factor activity using DeepAccess, an ensemble of convolutional neural networks trained to predict chromatin accessibility across multiple cell types^32,44. In previous work, we found that DeepAccess was successful in identifying DNA sequences driving differential accessibility between stem cells and definitive endoderm, as measured by a high-throughput reporter assay for chromatin accessibility³². In order to rank transcription factors with DeepAccess, we utilize the Differential Expected Pattern Effect⁴⁴ to estimate transcription factor impact on chromatin accessibility using computational DNA sequence perturbation, where we simulate in silico an experiment where the DNA binding motif for a given transcription factor is inserted at a previously closed genomic locus. To rank transcription factors as reprogramming candidates, we estimate the Differential Expected Pattern Effect of these transcription factor motifs between the starting and target cell types.

A consensus database of mouse transcription factor motifs

We used a custom clustered database of 107 mouse transcription factor motifs to simplify downstream analysis with transcription factor families that share highly similar motifs (Extended Data Fig. 1a). Using the HOCOMOCOv11 database⁴⁵ of 356 mouse transcription factor motifs, we computed the pairwise similarity between motifs using Tomtom⁴⁶. We then applied Pearson’s correlation and affinity propagation clustering to obtain motif clusters (Extended Data Fig. 1b). Affinity propagation clustering is a message passing algorithm that assigns a representative data point to each cluster, which can be weighted to select for ideal characteristics⁴⁷. In this case, the similarity metric was based on the information content of the motif. This resulted in 107 transcription factor motifs representing major families of transcription factors such as the OCT/SOX heterodimer (Extended Data Fig. 1c) and LIM motifs (Extended Data Fig. 1d).

Optimization of accessible region selection

In order to identify transcription factors motifs that are enriched in accessible genomic regions of target cell types, motif discovery algorithms typically compare the prevalence of motifs in sets of positive (target) vs. negative (background) sequences. We sought to identify the most effective parameters for motif discovery by using a grid search to explore optimal combinations of three attributes of input regions: 1) for positive sequences we used either the most significant accessible regions ranked by MACS2 in the target cell type or the most significant accessible regions that are accessible in the target cell type and not accessible in the starting cell type of stem cells or fibroblasts (see Methods for details), 2) for negative background sequences we used randomly shuffled positive sequences, enhancers shared across multiple cell types, or GC-content-matched randomly sampled genome sequences, and 3) the number of significant accessible regions to use as input (Fig. 2a). For our enhancer background sequences, we used Mouse ENCODE project candidate enhancer annotations from 18 tissues, where enhancers were defined by both the presence of H3K4me1 and absence of H3K4me3⁴⁸. To be included in our background sequences, the enhancer had to be present in 15 out of 18 tissues resulting in a total of 2,609 general enhancers. For GC percentage content-matched genome-sampled sequences, we used HOMER to generate sequences then ran AME, DREME, or KMAC using these sequences as background. Over all methods and input attributes, we ran a total of 150 experiments for transcription factor recall over our eight target cell types. Based on the area under the transcription factor recall curve (AURC) within the top 10 ranked motifs averaged over the eight target cell types, we selected optimal strategies for each motif discovery method (Fig. 2b; Table S2–6).

Figure 2. Selection of genomic regions impacts traditional DNA sequence-based methods for identification of transcription factors from chromatin accessibility.

a, The 3 axes for selection of genomic regions are choice of top regions or regions that are target cell type-specific relative to starting cell type, choice of background sequences for discriminative motif discovery, and number of regions. b, For each traditional method, the best set of genomic regions are chosen based on transcription factor recovery of all 8 cell types within in the top 10 ranked transcription factor motifs. c, Normalized area under the recall curve (AURC) for top 10 ranked factors averaged over 8 cell types for each choice of negative sequences for discriminative motifs discovery of either genome-sampled GC-content matched, universal enhancer sequences, and random di-nucleotide preserving shuffled positive sequences marginalized over number of regions and discriminative region selection axes (n = 150). d, Normalized area under the recall curve (AURC) for top 10 ranked factors averaged over 8 cell types for each choice of differentially accessible using stem cells or fibroblasts as starting cell type as well as top accessible regions in the target cell type cell type marginalized over number of regions and negative background sequence selection (n = 150). e, Normalized area under the recall curve for top 10 ranked factors averaged over 8 cell types stratified by number of regions, marginalizing over negative background sequence selection and discriminative region selection (n = 150). Box plots show median and quartile values. Whiskers extend to represent the rest of the data distribution with the exception of outliers that are defined as values greater than 1.5 times the inter-quartile range and are plotted as individual points. Linear regression model predicting normalized recovery were used to estimate weights and 95% confidence interval of decision axis on reprogramming factor recovery at rank less than 10 for f, AME (n = 345) g, DREME (n = 345) h, HOMER (n = 230), and I, KMAC (n = 230). Data are presented as parameter weights and 95% confidence intervals. Feature p-values are reported if significantly nonzero (p < 0.05) after Bonferroni correction for multiple hypothesis testing of parameters for each method regression model.

We also examine the distribution in normalized AURC averaged over the eight target cell types under each decision axis of choice of background sequences (Fig. 2c), differential regions (Fig. 2d), and number of regions (Fig. 2e), marginalized over the other decision axes for the top 10 ranked factors (Fig. 2c–e) and for the top 100 ranked factors (Extended Data Fig. 2a; See Methods for details). We investigated the performance given the choice of background sequences for discriminative motif discovery. GC%-matched genome-sampled background sequences improved performance in AURC for the top 10 motifs for AME (Fig. 2f), DREME (Fig. 2g), and HOMER (Fig. 2h) over shuffled or multi-tissue enhancer sequences. We were unable to consistently run KMAC using the 40,000 – 50,000 genome-matched background sequences generated by HOMER because of the memory required for oligo-based motif discovery, so we excluded these from our analysis. Between shuffled and enhancer background sequences, KMAC had a non-significant preference for shuffled control sequences over enhancer sequences (Fig. 2i). We also found that all methods were improved by the use of regions that were accessible in the target cell type and not in stem cells or fibroblasts, with stem cells as a preferred starting cell type for 3 out of 4 methods (Fig. 2b,d,f–i). One notable difference was that enhancer-based background sequences negatively impacted performance for DREME (Fig. 2g) or HOMER (Fig. 2h). One possible reason is the first seed oligo selection step that is common to both DREME and HOMER that may be affected when the number of background sequences is significantly different from the input sequences. In contrast to other input attributes, the number of input sequences had little significant effect on performance for most methods (Fig. 2e–i). Overall, using 5,000–10,000 input sequences yielded robust success across all methods, as taking the top 5–10% of regions typically also fell in this range (Fig. 2b). TODO area under the recall curve ranking the 100 factors (Extended Data Fig. 2b; See Methods for Details).

Additional epigenomic data do not impact performance

We explored if the addition of histone mark or EP300 data, which both implicate cell type-specific cis-regulatory elements, would improve performance for transcription factor recovery. Such histone marks have helped to improve enhancer recall in previous work⁴⁹. Since matched histone mark data were not available for all cell types, we focused on the liver where we could use ATAC-seq, H3K27ac, H3K4me3, EP300, and H3K4me1. H3K27ac, H3K4me1, and EP300 are epigenomic signals which can indicate enhancer activity whereas H3K4me3 is typically a mark for active promoters. We conditioned all on accessible (ATAC-seq) regions the with overlap of H3K27ac, H3K4me3, EP300, or H3K4me1 as well as tested accessible regions that overlapped all three active enhancer marks (H3K27ac, H3K4me1, and EP300). We investigated these region sets by varying the choice of differential regions or top regions, selection of background sequences, and number of regions as well as for the four motif discovery algorithms. The normalized area under the recall curve for the top 10 ranked motifs when examined over all methods and input choices only resulted in a significant decrease in performance when conditioning on accessible regions containing H3K4me3 marks (Fig. 3a; p < 0.05 for ATAC, H3K27ac, EP300, and EP300_H3K27ac_H3K4me1, by Rank Sum Test with Bonferroni correction for multiple hypotheses). The decrease in performance using H3K4me3 data is unsurprising given its known role in predicting active promoters which may exclude enhancer regions that contain important transcription factor binding sites⁵⁰. Similarly, our finding that conditioning on predictive enhancer marks does not improve reprogramming factor recovery over chromatin accessibility is consistent with previous work suggesting that accessibility and H3K27ac have similar levels of accuracy in predicting enhancers that are validated by transgenic mouse assays⁴⁹. When separated by method, the trends in the area under the recall curve (Fig. 3b, Extended Data Fig. 3, Table S7–9) and the area under the recall curve for transcription factors within the top 10 (Fig. 3c) were consistent with the aggregate findings, though no differences were significant. We also examined the particular transcription factors found with each histone mark (Fig. 3d). Gata transcription factor motifs are consistently and strongly ranked by all methods and almost all epigenomic signals, indicating these motifs are highly prevalent and easily recovered from liver epigenomic signal. For AME, only ATAC-seq and ATAC-seq overlapping H3K27ac and H3K4me1 were able to recover the Fox transcription motifs. Similarly, for DREME and HOMER ATAC-seq and ATAC-seq with H3K27ac give stronger signal for Fox motifs, and KMAC only discovers Fox motifs with ATAC-seq, ATAC-seq overlapping H3K27ac, EP300 or the 3 enhancer marks. This may suggest that Fox motifs are more significantly over-represented in enhancer regions. Hnf1 motifs are ranked near the top by HOMER and AME and is 1 of 3 motifs ranked significant by DREME when chromatin accessibility is overlapping H3K4me3, indicating these factors may be more prevalent within liver promoter regions over enhancers.

Figure 3. Use of histone mark and EP300 annotation does not significantly impact transcription factor recovery in liver cells.

a, Normalized area under the recall curve within the top 10 motifs on hepatocyte transcription factor recovery for accessible regions with addition of 5 epigenomic markers, each dot represents a unique combination of one of the 4 methods (KMAC, AME, HOMER, and DREME), selection of differential regions, selection of background sequences, and number of sequences. Area under the recall curve shows an overall robust performance for all ATAC-seq alone along with conditioning on accessibility with presence of active enhancer marks H3K27ac, EP300, H3K4me1, and presence of all 3 active enhancer marks, with decrease in reprogramming recovery when tested against overlap of H3K4me3 (n = 150) for ATAC (n = 150; Rank Sum statistic = 2.896; p = 0.00378), H3K27ac (n = 150; Rank Sum statistic = 2.831;p = 0.00463), EP300 (n = 150; Rank Sum statistic = 3.221;p = 0.00128), EP300_H3K27ac_H3K4me1 (n = 150; Rank Sum statistic 3.612;p = 0.000394), H3K4me1 (n = 150; Rank sum statistic 2.500;p = 0.0124). Significance reported as p-values by Rank Sum Test under Bonferroni correction with star indicating significant under adjusted threshold p < 0.0083 for multiple hypotheses (n = 6), b, Normalized area under the recall curve for 4 methods (KMAC, AME, HOMER, and DREME) on hepatocyte transcription factor recovery (theoretical maximum normalized area under the recall curve 1.0; range 0–0.953; n = 900) shows an overall robust performance for all histone marks, consistent with overall trends with H3K4me3 the worst performing across methods. c, Area under the recall curve at rank within the top 10 motifs for 4 methods (KMAC, AME, HOMER, and DREME) on hepatocyte transcription factor recovery (theoretical maximum normalized area under the recall curve 1.0; range 0–0.942; n = 900). All box plots show median and quartile values. Whiskers extend to represent the rest of the data distribution with the exception of outliers that are defined as values greater than 1.5 times the inter-quartile range and are plotted as individual points. d, Rank of each reprogramming transcription factor for each method and each genomic signal shows some trends that may relate to promoter (Hnf1) or enhancer (Fox) bias of transcription factors, and some highly consistent transcription factors (Gata).

Transcription factor recovery and significance ranking

We then evaluated all RNA-seq and ATAC-seq methods on their ability to identify known reprogramming factors. For all methods for which we were able to evaluate the choice of stem cells or fibroblast as the starting cell type for differential gene expression or differential motif discovery, we used the starting cell type (stem cell or fibroblast) that resulted in the highest area under the recall curve (AURC) for the 7 non-stem-cell target cells (Table S4). When stem cells were our target cell type, we used fibroblasts as our starting cells for all methods. Overall, we find that chromatin accessibility methods have higher average factor recall over the 8 cell types for the top 100 ranked factors relative to gene expression-based methods (Fig. 4a), and the performance is similar among all methods for the area under the recall curve for the top 10 ranked factors (Fig. 4b). We developed regression models to predict the normalized AURC scores as a function of the 9 methods, and found that AME and diffTF have a statistically significant higher recall than all other methods for the top 100 factors (Fig. 4c; see Methods for details), and while AME and HOMER have higher effects on recall within the top 10 factors, these effects are not significant (Fig. 4d). Of the chromatin-based methods, AME, diffTF, GarNet, and DeepAccess are able to obtain high AURC in the top 100 factors than HOMER, DREME, or KMAC, which may be because AME, diffTF, GarNet, and DeepAccess place a prior on the discovery of motifs by using a transcription factor database. These priors due to the use of a motif database appear to have less effect when the recall curve is restricted to the top 10 ranked motifs, whereas HOMER, a de novo motif method, has the highest median performance over all cell types (Fig. 4b,d; Table S4). We found that there is variability in recovery across cell types, with liver having the highest median area under the recall curve (AURC) rank top 10 over all methods (median AURC rank top 10 = 0.653) and stem cells having the lowest with a median AURC rank top 10 of 0, and similarly in the top 100 ranked factors skeletal muscle had the highest median AURC at 0.94, and stem cells the lowest at 0.428 (Fig. 4e). For 5 out of the 9 algorithms tested, using stem cells as the starting cell state resulted in a better recall of transcription factors over fibroblast (Extended Data Fig. 4a; Cell type averaged area under recall top 100 – DREME: 0.426 (fibroblast), 0.339 (stem cell); AME: 0.435 (fibroblast), 0.506 (stem cell); KMAC: 0.156 (fibroblast), 0. 131 (stem cell); HOMER 0.408 (fibroblast), 0.520 (stem cell); DeepAccess: 0.336 (fibroblast), 0.204 (stem cell); diffTF: 0.344 (fibroblast), 0.356 (stem cell); EBseq 0.215 (fibroblast), 0.306 (stem cell); GarNet: 0.078 (fibroblast), 0.079 (stem cell)) though these performance differences were overall subtle in ranking the top 10 (Extended Data Fig. 4b) or top 100 factors (Extended Data Fig. 4c).

Figure 4. Complex chromatin methods are top performers for transcription factor recovery and significance ranking.

a, Evaluation of 9 methods (2 entirely RNA-seq-based, 1 RNA-seq and ATAC-seq, 4 traditional ATAC-seq, 2 complex ATAC-seq) for each cell type (n = 8; CellNet n = 4) using normalized area under the recall curve for top 100 ranked factors. b, Normalized area under recall curve for the top 10 ranked motifs for each method in each cell type (n = 8; CellNet n = 4). Box plots show median and quartile values. Whiskers extend to represent the rest of the data distribution with the exception of outliers that are defined as values greater than 1.5 times the inter-quartile range and are plotted as individual points. C, Linear regression models used to estimate effect size and 95% confidence intervals of method on normalized area under recall curve for top 100 ranked factors (n = 68). d, Linear regression models used to estimate effect size and 95% confidence intervals of method on area normalized under recall curve for top 10 ranked factors (n = 68). Data are presented as parameter weights and 95% confidence intervals. Feature adjusted p-values are reported if significantly nonzero (p < 0.05) after Bonferroni correction for multiple hypothesis testing of parameters within the regression model. e, Reprogramming factor recall plots for each of the eight cell types for all nine methods. f, Rank of reprogramming transcription factors for each method and each cell type. g, Correlation between reprogramming factors ranked by methods and ranked by significance based on literature.

Against our expectations, GarNet’s incorporation of gene expression with chromatin accessibility appeared to degrade rather than improve transcription factor recovery. There are several possible explanations for this observation that are not mutually exclusive: 1) by conditioning on proximity to genes, we eliminate distal accessible regions which have important regulatory function in reprogramming or accessible regions that may be cell type-specific but regulate genes that are not differential between starting and target cells, 2) transcription factor binding sites cannot be assumed to regulate their closest genes, or 3) transcription factor motif match score is not a good quantifier of impact of the transcription factor on gene expression. In order to test whether conditioning on proximity to genes had an impact on GarNet’s performance, we tested GarNet’s ability to identify transcription factors using 2kb, 10kb, or 100kb distance thresholds as limits to transcription factor:gene relationships. We found that the choice of distance threshold had little impact on GarNet’s performance (Extended Data Fig. 5).

When examining transcription factor rank within each cell type, it appeared that certain motifs were more robustly found across many methods, though only a few motifs were found by the best performer for all methods (Fig. 4f). diffTF and DeepAccess were the only two methods that highly ranked the Oct4/Sox2 heterodimer motif, which is notable as a well-known transcription factor pair in both reprogramming^18,51 and in manipulating nucleosomes to increase chromatin accessibility⁵². The disagreement in ranking between different methods led us to question whether methods were able to predict the relative significance of known reprogramming factors. From the literature, we found evidence of reprogramming factor significance for liver, pan-neuronal, and motor neuron by impact of over-expression of factors on reprogramming efficiency^51,53 and evidence of significance in cardiomyocyte from tuned expression vector of Gata4, Tbx5, and Mef2c⁵⁴ as well as evidence of the significance of Gata4 in cardiomyocyte reprogramming from leave-one-out experiments⁵⁵. We then computed the correlation between true rank and observed rank for each method. Overall, we found that DeepAccess had a strong correlation between true and observed rank (Fig. 4g), though this effect was not statistically significant (0.091 probability seeing mean correlation higher by chance under permutation test; see Methods for details).

Discussion

Efficient and accurate transcription factor candidate prioritization for cellular reprogramming is an important unsolved problem. We evaluated nine complementary methods for this task on their ability to rediscover existing reprogramming protocols. These methods fell into three broad categories: gene expression-based, traditional epigenomic-based, and complex epigenomic-based. Among the expression-based methods, CellNet represents a class of methods that use perturbation experiments to build a cell type-specific gene regulatory network. From these networks, hub genes are identified as potential targetable candidates for reprogramming. However, we found that while CellNet may have performance benefits relative to EBSeq in reprogramming factor recovery using differential gene expression (Fig. 4b), it was impossible to apply CellNet to new gene expression data with only few unperturbed replicates of the experiment. Another method that builds a regulatory network is GarNet, which links genes to proximal transcription factor binding sites within accessible genomic regions to build a transcription factor:gene regulatory network. GarNet then uses sparse linear regression to rank transcription factor candidates. While GarNet is a compelling approach as it combines gene expression information and accessibility information, we found that it did not outperform other chromatin accessibility methods (Fig. 4a–d) and had a lower average performance ranking top 10 factors than CellNet and EBseq (Fig. 4b,d) which suggests that GarNet’s assumptions may not be optimal. For example, GarNet assumes that transcription factors within enhancers regulate their closest genes, and that factor binding strength is represented by PWM match score. Future research could apply GarNet using known enhancer:gene interactions from 3D interactome data and transcription factor binding strength from ChIP-seq to determine whether modifications to these assumptions can improve its performance.

We then compared all nine methods on their ability to recover transcription factors from eight cell types with successful differentiation protocols (Fig. 4a–d). Overall, we found that AME, DeepAccess, and diffTF were able to outperform expression-based methods (Fig. 4a–d). We found that AME was the top performer in reprogramming factor recovery when looking at the top 100 ranked motifs for each method (Fig. 4a,c). Methods such as AME, DeepAccess, GarNet, and diffTF rely on pre-existing motif databases that may be missing certain transcription factors entirely and cannot discover novel motifs such as a motif resulting from the binding of a heterodimer of two transcription factors. We also noticed a high variance for all methods in transcription factor recovery ranking the top 10 motifs or factors (Fig. 4b), which may be related to data quality but is difficult to disentangle from possible biological confounders like the developmental timepoint of the collected data or the complexity of the regulation of differentiation for different cell types. We also note that there is a potential bias in that we performed optimization of input selection for the four chromatin accessibility methods on the target data from the same eight cell types that we then use in evaluating all nine methods. However, we also performed optimization of the parameters for GarNet using the same eight cell types and the other methods (DeepAccess, diffTF, CellNet, and EBSeq) did not have an obvious parameter space for optimization beyond starting cell type. Therefore, we anticipate there is little bias as a result of optimizing the input to the four chromatin methods prior to evaluating on full set of nine methods. We provide an overall summary of the recommended workflow for reprogramming factor recovery from chromatin accessibility data for a desired target cell type for practioners in Extended Data Fig. 6.

Certain transcription factors were uniformly highly ranked across methods, while transcription factors such as Oct4/Sox2 motif were highly ranked only by the complex chromatin accessibility methods, diffTF and DeepAccess (Fig. 4f). Observation of rank differences between methods lead us to ask whether certain methods were better at identifying the most significant transcription factor. We found evidence from the literature that supported ranking transcription factors within a reprogramming protocol, either through reprogramming efficiency of single transcription factor over-expression or through expression level optimization and leave-one-out experiments (Fig. 4g). While metrics for reprogramming efficiency and specific cell type can make these comparisons difficult, significance ranking is an important additional metric as the transcription factors that are chosen to be included in a reprogramming protocol by ad-hoc methods may bias the results seen from evaluating recovery performance alone. DeepAccess had the lowest probability of seeing the correlation between ranked the significance of transcription factors within reprogramming protocols by chance at p = 0.091, but this did not pass significance thresholding (Fig. 4g). Given both diffTF and DeepAccess had higher correlations than other chromatin accessibility-based approaches, it suggests that there are advantages in using complex methods for identifying transcription factors from chromatin accessibility. Both methods also provide additional value beyond transcription factor discovery or enrichment. Predictive deep learning models like DeepAccess have been used to predict causal variants^56–58, interpret complex genomic grammar^59–61, and identify evolutionarily orthologous enhancers^62,63. In contrast with all other approaches, DeepAccess estimations of transcription factor effect on cell type-specific chromatin accessibility come from designed genomic sequences containing transcription factor motifs. Therefore, DeepAccess can used to probe more complex regulatory grammar such as transcription factor combinations or spacing. Despite a tradeoff in computational expense of training neural networks, DeepAccess may be preferred if the user intends to investigate regulatory logic beyond the activity of single transcription factors. In contrast, diffTF may desirable for its statistical approach if another goal is to be able to identify whether individual binding sites are differentially accessible.

An additional metric that we did not evaluate was the number of ranked factors produced by each method that are necessary to cover all known reprogramming transcription factors for a given cell type. As we rely on literature to determine our set of reprogramming transcription factors from studies that may have different definitions of successful reprogramming, determining a set of necessary transcription factors for each of our eight target cell types was not possible. Uniform studies across multiple cell types testing the success of combinations of transcription factors for reprogramming could allow for a deeper evaluation of these methods.

While we found that expression-based methods performed worse than accessibility-based methods for transcription factor recovery, we note that comparing performance between chromatin accessibility methods and gene expression methods is imperfect. We require perfect transcription factor matches when evaluating CellNet and EBseq, while we require transcription factor family matches for GarNet, HOMER, AME, DREME, KMAC, diffTF, and DeepAccess. Ultimately, the best approach is likely to incorporate expression and accessibility data, but rather than taking GarNet’s approach to using expression to filter accessible regions and rank transcription factors, our results would suggest it is best to first use chromatin accessibility alone to select transcription factor families and then use other measurements of transcription factor activity such as proteomic, transcriptomic, or gene regulatory network information sources combined with expert knowledge to select transcription factor candidates within a family.

In evaluating methods, the best methods only reached 50–60% recall of reprogramming factors within the top 10 motifs, indicating there is still room for methodological advancements. Single cell expression data may provide sufficient information for methods such as CellNet to be applied with less experimental burden. Methods like GarNet that use both expression and accessibility may require further exploration to determine optimal approaches to combine these data. Methods such as diffTF and GarNet that rely on input of transcription factor binding sites may suffer from issues in determining an appropriate binding cutoff based on PWM information alone, and could be combined with more complex binding prediction models like DeepBind⁶⁴ to improve performance.

A limitation of our work is that we assumed there were no alternative factors to the ones present in the reference reprogramming protocols we utilized. This assumption ignores potentially better candidates that are experimentally uncharacterized. We examined whether there were any potentially novel reprogramming targets that were consistently ranked by HOMER, AME, DeepAccess, and diffTF (Table S10–16). Among these, we found NFI half-motif which was identified by 3 out of the 4 methods as a potential skeletal muscle regulator. Indeed, NFIX has been previously implicated in skeletal muscle development^65,66. Hnf1 and Rfx were ranked by all methods within the top 10 motifs for beta pancreatic cells, with Hnf1 having a role in early progenitor development of endocrine cells⁶⁷ and Rfx transcription factors playing an important role in adult pancreatic cell functional development^68,69. Experimental methods to evaluate transcription factors in parallel through transcription factor screens^70–75 or high-throughput reporter assays for functional transcription factor activity such as chromatin accessibility³² would allow us to expand our evaluation of methods to their ability to propose novel factors.

Overall, we found that chromatin accessibility-based methods recover the most known reprogramming transcription factors. We suggested methods for optimal region selection for traditional accessibility-based methods, and suggest that there are some performance benefits to using complex accessibility-based methods that perform more statistical or complex sequencing modeling of epigenomic data. We hope that our comprehensive evaluation of transcription factor reprogramming ranking and recovery will contribute to a basis for motif analysis procedures and a standard evaluation metric in the development of novel computational methods.

Methods

ATAC-seq processing

Reads were trimmed for adaptors and low-quality positions using Trimgalore (Cutadapt v0.6.2). Reads were aligned to the mouse genome⁷⁶ (mm10) with bwa mem⁷⁷ (v0.7.1.7) with default parameters. Properly paired mapped reads were filtered, and accessible regions were called using MACS2⁷⁸ (v2.2.7.1) with the parameters -f BAMPE -g mm -p 0.01 --shift −36 --extsize 73 --nomodel --keep-dup all --call-summits. Accessible regions that overlapped genome blacklist regions (encode blacklist regions ENCSR535HHO) were excluded from downstream analysis.

RNA-seq processing

Reads were trimmed for adaptors and low-quality positions using Trimgalore (Cutadapt⁷⁹ v0.6.2). Reads were aligned to the mouse genome (mm10) and gene-level counts were quantified using RSEM⁸⁰ (v1.3.0) rsem-calculate-expression using default parameters and STAR⁸¹ (v2.5.2b) for alignment.

Accessible regions selection

Regions are considered to be accessible in the target cell type and not in the source cell type if they are accessible in the target cell type (peak signal from MACS2 fdr < 0.05) and have 0 nucleotides overlapping with any accessible region (peak signal from MACS2 fdr < 0.05) in the source cell type.

Selection of best combinations of sequence-based input decision axes

For plotting the recall curves for each of the 4 traditional sequence based methods and comparing to the full set of 9 methods, or in the case of evaluation of the epigenomic features in Figure 3 for plotting the individual recall curves in Figure S3, we select the method with the best area under the recall curve within the top 10 ranked factors. These selected best combinations of features to use as input for each method/mark are listed in Table S3 and Table S4.

DeepAccess Training and Interpretation

We trained DeepAccess on data from ten cell types: stem cell, fibroblast, hepatocyte, endoderm, beta pancreatic cell, alpha pancreatic cell, cardiomyocyte, skeletal muscle, dopaminergic midbrain neuron, and spinal motor neuron using binary crossentropy loss (multitask classification) with 4,812,987 genomic regions for training: 3,133,509 regions were open in at least 1 cell type, and 1,679,478 regions were closed in all cell types (sampled with HOMER to match GC content % of accessible regions in genome). Chromosome 18 and chromosome 19 were held out for validation and training, respectively. To define training regions for DeepAccess, we generate 100bp genomic windows across the entire mouse genome. We define a region as accessible in a given cell type if more that 50% of the 100bp region overlaps a MACS2 accessible region from that cell type. We then compute the Differential Expected Pattern Effect⁴⁴ for all transcription factor motifs within the consensus database between the target and starting cell types.

CellNet analysis

We ran CellNet^23,24 using the provided mouse network and our fibroblast RNA-seq sample replicates as input to obtain a list of ranked genes.

Transcription factor list curation and differential expression with EBseq

After RSEM quantification, EBSeq²⁷ was run with the RSEM wrapper rsem-run-ebseq. Genes were filtered to fdr p < 0.05. Analysis was limited to a list of 1,374 transcription factors derived from⁸², which were sorted by EBSeq estimates of change in expression between target and starting cell types. List of mouse transcription factors is available at https://cgs.csail.mit.edu/ReprogrammingRecovery/.

GarNet analysis

PWMScan⁸³ (v1.1.0) was used to identify transcription factor motif instances in the genome for the 107 consensus transcription factors. Since GarNet^25,26 builds transcription factor:gene networks using proximity assignment, we built 3 GarNet networks where transcription factor influence was set to a threshold of 2kb, 10kb, or 100kb from transcription start sites. Then, GarNet was run for each starting cell type, target cell type pair with an input of differential expression (fold change) for genes with fdr < 0.05 for differential gene expression as estimated with EBSeq, and target cell type MACS2 accessible regions. Transcription factor rank was determined by the value of the slope.

Traditional Motif Enrichment

A script to perform motif enrichment using the four traditional enrichment methods, AME, DREME, HOMER, and KMAC is available at https://cgs.csail.mit.edu/ReprogrammingRecovery/.

Briefly, the steps in the script are to first sort accessible regions by the significance using the p-value column within MACS2 narrow peak calls, and take the top N or top P% of regions for motif enrichment. Then, we generate a fasta file containing genomic sequences from accessible regions. If the discriminative negative sequences used as a background in motif discovery are homer-matched, HOMER is run to obtain GC% matched randomly sampled sequences. Otherwise, either method-specific default (GC-matched genome for HOMER or dinucleotide shuffled for KMAC, AME, DREME) or a fasta file input by the user is used as negative sequences for discriminative motif discovery. The user may then specify one or more of AME, HOMER, DREME (MEME), or KMAC to be run.

AME analysis

AME⁴³ was run with default parameters with “--control -–shuffle--” for a dinucleotide shuffled (default) background or enhancer background sequences were provided.

DREME analysis

DREME²⁹ was run with default parameters.

HOMER analysis

For default background, HOMER³⁰ was run using the command findMotifsGenome.pl target.bed mm10 target-homer -size given. For enhancer background, HOMER was run using the command findMotifs.pl target.fa fasta target-homer -fasta enhancer.fa.

KMAC analysis

KMAC³¹ was run with the parameters --k_win 100 --k_min 4 --k_max 13 --t 1 --k_seqs 10000 --k_top 10 --gap 4.

diffTF analysis

PWMScan⁸³ (v1.1.0) was used to identify transcription factor motif instances in the genome for the 107 consensus transcription factors. Then, diffTF³³ was run with pairs of starting cell type, target cell type data sets with default parameters. Input was MACS2 accessible peaks and ATAC-seq aligned reads for sample replicates.

De novo method transcription factor matching

To match transcription factor motifs to their best motif match for a known reprogramming transcription factor, we ran Tomtom⁴⁶ (v5.0.5) with default parameters and set a threshold of a motif match with q-value < 0.05. The lowest rank (most enriched) motif that a given reprogramming transcription factor is assigned to that transcription factor.

Correlation Computation and Significance of Ordering of Transcription Factors

We compute the spearman rank correlation between the rankings for each method and the rankings based on significance of factors to reprogramming protocols from the literature. In cases where some factors were unranked by a given method, they were given a rank of 107 (max rank + 1). Then to compute significance, we performed a permutation test with 2000 permutations of the method labels (DeepAccess, EBSeq, diffTF, HOMER, GarNet, AME, CellNet, KMAC, and DREME) to the correlations to determine the probability, then computed the random method average and compared to the true method average to determine for each method the probability of seeing as high of an average by chance (DeepAccess p = 0.091; EBSeq p = 0.7655; diffTF p = 0.177; HOMER p = 0.8385; GarNet p = 0.86; AME p = 0.311; CellNet p = 0.1115; KMAC p = 0.4885; DREME p = 0.908).

Normalized Area Under the Recall Curve

We compute the area under the recall curve (AURC) for the first 100 reprogramming factors, normalized by the theoretical maximum AURC for each cell type ($M^{\mathit{100}}{}_{celltype}$), such that the normalized AURC will sum to 1. R(i) is the number of reprogramming factors recovered at rank less than or equal to $i$, and $T_{celltype}$ is the total number of reprogramming factors for a given cell type:

$$\text{Normalized AURC}={\displaystyle \sum }_{\text{celltype}}\frac{1}{{\text{M}}^{100}{}_{\text{celltype}}}{\displaystyle \sum }_{\text{i}=1}^{100}\frac{\text{R}\left(\text{i}\right)}{{\text{T}}_{\text{celltype}}}$$

and $M^{\mathit{100}}{}_{celltype}$ is computed as the total AURC if all $T$ reprogramming factors are recovered as the first $T$ factors within the top 100 ranked factors for a given method:

$${\text{M}}^{100}{}_{\text{celltype}}={\displaystyle \sum }_{\text{i}=1}^{100}\frac{\text{min}\left(\text{i},{\text{T}}_{\text{celltype}}\right)}{{\text{T}}_{\text{celltype}}}$$

Similarly, we compute this for the factors ranked within the top 10 for a method:

$$\text{Normalized AURC Rank Top 10}={\displaystyle \sum }_{\text{celltype}}\frac{1}{{\text{M}}^{10}{}_{\text{celltype}}}{\displaystyle \sum }_{\text{i}=1}^{10}\frac{\text{R}\left(\text{i}\right)}{{\text{T}}_{\text{celltype}}}$$

$${\text{M}}^{10}{}_{\text{celltype}}={\displaystyle \sum }_{\text{i}=1}^{10}\frac{\text{min(i},{\text{T}}_{\text{celltype}})}{{\text{T}}_{\text{celltype}}}$$

Statistics and Visualization

To compute statistical significance and confidence intervals for the parameters within the chromatin accessibility methods, regression models were built for each of the 4 motif discovery algorithms. Regression models take the form of:

$$\text{y}=\text{Xδ}+\text{Z}\varnothing +\text{Wβ}$$

where $\text{X},\text{Z},\text{W}$ represent indicator design matrices representing the choice of differential sequences, background sequences, and number of sequences used to perform motif enrichment / discovery. The dependent variable y is the normalized area under the recall curve averaged over the 8 cell types. Parameter estimation and confidence intervals were derived using the OLS function in statsmodels (v0.10.1).

To compute statistical significance of the effect of method choice on area under the recall curve (AURC) regression models were built that take the form of:

$$\text{y}=\text{Mβ}$$

where M represents indicator design matrices representing which of the nine methods the computed AURC value was derived using (DeepAccess, diffTF, AME, HOMER, DREME, KMAC, GarNet, CellNet, or EBSeq). The dependent variable y is the normalized area under the recall curve which come from the computation reprogramming factor recover on one of the eight cell types. Parameter estimation and confidence intervals using the OLS function in statsmodels (v0.10.1).

AUC was computed using the trapezoidal rule using the scikit-learn⁸⁴ (v0.22) function auc in python (v3.6.9). All graphs were made using the seaborn⁸⁵ (v0.9.0) library available for python with all python packages installed using bioconda⁸⁶ and conda package managers (v4.9). Versions of all software used in this work are reported in Table S17.

Data Availability

Normalized area under rank recall curve values for all methods are available in Supplementary Tables 1–9.

The consensus mouse transcription factor motif database derived from the mouse HOCOMOCOv11 database⁴⁵, shared mouse enhancer sequences, and a list of mouse transcription factors are available at: https://cgs.csail.mit.edu/ReprogrammingRecovery/.

Publicly available ATAC-seq and RNA-seq samples were downloaded as fastqs from Nucleotide Read Archive (Table S1) and processed as described in sections ATAC-seq processing and RNA-seq processing below. Uniformly processed gene count and peak files are also available at https://cgs.csail.mit.edu/ReprogrammingRecovery/.

Data collection software conda / bioconda (v4.9.0), bedtools (v2.29.2), trimgalore ) (v21032019), cutadapt (v0.6.2), samtools (v1.7), bwa (v0.7.17), MACS2 (V2.2.7.1), FASTQC (V0.11.8), STAR (v2.5.2b), RSEM (v1.3.0), R (3.6.1), python (v3.6.9), DeepAccess (v0.0.1), EBSeq (v1.2.0), CellNet (v0.1.0), GarNet (v0.5.0), HOMER (v4.9.1), AME / DREME / TomTom (v5.0.5), KMAC (GEM v3.4), diffTF (v1.7.1), PWMScan (v1.1.1), HOCOMOCO (v11), GENCODE (vm24), mouse genome (mm10) are cited in Table S17.

Code Availability

The custom script for performing motif discovery with AME, DREME, HOMER, and KMAC is available at: https://cgs.csail.mit.edu/ReprogrammingRecovery/

Extended Data

Extended Data Fig. 1.

A consensus database of 107 transcription factor motifs.

a, HOCOMOCO v11 mouse transcription factor core motif database is used as input. Motif PWM similarity to the HOCOMOCO database is computed using Tomtom. b, For each pair of motifs, Pearson correlation between Tomtom scores is computed, resulting in a symmetric correlation matrix. Affinity propagation clustering is applied to the correlation matrix, resulting in 107 clusters of transcription factor motifs with one motif being selected as the representative motif of the cluster. c, Cluster representing OCT/SOX heterodimer-like motifs with SOX2 motif selected as the representative. d, Cluster representing LIM-like motifs with LHX3 motif selected as the representative.

Extended Data Fig. 2.

Comparing input features and methods for transcription factor recovery from chromatin accessibility data.

a, Reprogramming recovery effected estimated by linear models for decision axes in input to chromatin models for AURC of top 100 ranked factor motifs, excluding predicting stem cell reprogramming factors to estimate effect of use of fibroblast or stem cell as source cell type is selection of cell type-specific regions, or selection of top regions without eliminating regions that are accessible in the source cell type. b, Cell type AURC for top 100 ranked factor motifs stratified by decision axis and marginalized over other axes. Box plots show median and quartile values. Whiskers extend to represent the rest of the data distribution with the exception of outliers that are defined as values greater than 1.5 times the inter-quartile range and are plotted as individual points.

Extended Data Fig. 3.

Comparing chromatin accessibility overlapping histone mark and EP300 epigenomic data for transcription factor recovery.

AURC for top 100 ranked factor motifs in liver using overlaps between chromatin accessibility (ATAC-seq) and overlap of chromatin accessibility with H3K27ac, EP300, H3K4me1, H3K4me3, and 3 enhancer markers (EP300, H3K27ac, and H3K4me1) per method identifies for DREME, HOMER, and KMAC worst performance using ATAC + H3K4me3 which is correlated with promoter activity, and for all methods we see similar performance levels with ATAC, ATAC + H3K27ac, and ATAC + H3K4me1 which mark enhancers.

Extended Data Fig. 4.

Fibroblast and stem cell as starting cell type comparing each method.

Chromatin methods use optimal input features for each background a, Normalized area under the rank recall curve for top 10 ranked motifs averaged over cell types, b, scatter plot of normalized area under the recall curve for fibroblast (x-axis) and stem cell (y-axis) each dot represents the normalized area under the rank recall curve for top 10 ranked motifs for one cell type and one method where color represents the method, c, scatter plot of normalized area under the recall curve for fibroblast (x-axis) and stem cell (y-axis) each dot represents the normalized area under the rank recall curve for top 100 ranked motifs for one cell type and one method where color represents the method.

Extended Data Fig. 5.

GarNet distance thresholds do not majorly impact performance for transcription factor recovery.

GarNet fraction of reprogramming factors over eight target cell types recovered by 2 kb, 10 kb, and 100 kb thresholds for maximum distance between transcription factor binding site and gene transcription factor start site.

Extended Data Fig. 6.

Deciding input and methods for ranking reprogramming transcription factors.

Decision chart for performing optimal reprogramming factor recovery given chromatin accessibility data for a desired target reprogramming factor cell type.