Non-local modeling of enhancer-promoter interactions, a correspondence on “LOCO-EPI: Leave-one-chromosome-out (LOCO) as a benchmarking paradigm for deep learning based prediction of enhancer-promoter interactions”

Abstract

A recent paper by Tahir et al 2024 in Applied Intelligence reported a computational model of enhancer promoter interactions without realizing that many of their conclusions were previously published in 2018. In addition to correcting this record, the authors appear to be unaware of an additional body of previous work on enhancer-promoter interactions, which can explain why their computational model performs poorly. We describe how the weak predictive power of their model is consistent with new insights gained from substantial recent progress in the area of detecting and modeling enhancer promoter interactions constrained by DNA looping, extrusion by cohesin, and CTCF.

Gene regulation is the process by which cells transcribe a subset of genes that confer specific biological functions to a cell type. In normal development, control of this process specifies the complement of cell types within an organ or tissue. Gene regulation also controls how cells can respond to varying extracellular environments or stresses, such as changes in oxygen or glucose levels, and immune responses. Disruption of normal gene regulatory mechanisms can lead to increased risk of disease such as cancer or diabetes. At individual genes, expression is controlled by DNA sequences called enhancers, typically 300bp long, which are cooperatively bound by transcription factor (TF) proteins. Multiple enhancers interact with a separate complex of transcription factor proteins bound at the promoter of the gene to activate transcription. The processes controlling which enhancers interact with which promoters is therefore a key component of the gene regulatory networks controlling cellular behavior. Although these interactions are usually local (within 300kb), an enhancer does not necessarily interact with the nearest promoters. MicroRNAs also appear to be transcriptionally regulated through enhancer-promoter interactions, and can contribute to gene regulatory networks via translational repression of TF target genes [1]. Thus, enhancer-promoter interactions, and the transcription factor binding sites within specific enhancers, dictate the gene-to-gene connections in gene regulatory networks which control complex cellular development and behavior [2].

A recent article by Tahir et al [3] addresses the challenging problem of predicting these enhancer-promoter interactions using computational models. A key point of [3] is that improper test and training set design can allow information leakage between training and test sets and lead to undiagnosed overfitting and failure of the models to generalize to other datasets, a continuing and serious problem in the machine learning literature. The authors suggest chromosomal test and training set splits as way to eliminate the possibility of this leakage, and describe it as a “paradigm” for prediction of enhancer-promoter interactions. We agree that for enhancer-promoter interaction prediction, and in many other cases, chromosomal test sets should be, and already are, standard practice in the field. The claim was made in [3] that, “In 2020, Belokopytova et al [4] were the first to analyze that the random splitting of datasets into training and testing subsets, as done in the released TargetFinder datasets, causes EP pairs from the same genomic regions to be present in both the training and testing subsets. This overlap leads to information leakage and an overestimation of performance for the reported EPI prediction models.” The point about leakage is correct, but the statement that this was first published in 2020 is not. In fact, this conclusion was first published in Xi and Beer 2018 [5], a paper which was cited in Belokopytova et al 2020 [4], but in a way which obscured the fact that the mechanism proposed in [4] was also clearly described in [5]. The abstract of Xi and Beer 2018 [5] states, “We report an experimental design issue in recent machine learning formulations of the enhancer-promoter interaction problem arising from the fact that many enhancer-promoter pairs share features. Cross-fold validation schemes which do not correctly separate these feature sharing enhancer-promoter pairs into one test set report high accuracy, which is actually arising from high training set accuracy and a failure to properly evaluate generalization performance. Cross-fold validation schemes which properly segregate pairs with shared features show markedly reduced ability to predict enhancer-promoter interactions from epigenomic state.” The simplest class of specific cross-fold validation schemes used in [5] which properly segregate pairs with shared features were, in fact, chromosomal test sets, as clearly shown in Fig 2 of [5]. Thus the primary claims in Tahir et al 2024 [3] were already published in 2018 [5]. This is not the only oversight in the Tahir et al paper. In fact, an additional correspondence pointing out the error in the TargetFinder paper was also published by Cao and Fullwood in 2019 [6], which also cited [5], and the only reason [5] was published before [6] is that our paper was not delayed by the response from the original TargetFinder authors, a response which does not contain a compelling counter-argument to the test set design error in TargetFinder. The Cao and Fullwood correspondence [6] was also not cited by Belokopytova et al [4]. Although they noted the overfitting in TargetFinder, Cao and Fullwood [6] detected it through randomization of the input features, showing that this should have random predictive power, but also generated models with “high performance,” thus they clearly implicated but did not directly assess training-test set leakage. Although Tahir et al comment extensively on the problems with TargetFinder, they appear to be completely unaware of the published correspondence from 2019 [6].

Tahir et al [3] go on to develop a deep neural network (DNN) model to predict enhancer-promoter interactions from DNA sequence, using both PWM-like convolution kernels and k-mer features within the 3kbp enhancer and 2kbp promoter bins. This approach is interesting, as k-mers can directly resolve correlated dependencies between nucleotide or amino-acid features [7–11], but standard convolution kernels do not, but Tahir et al seem unaware of a highly cited body of work showing that gapped-kmer features perform better than k-mers [10, 12–14]. Tahir et al show that their DNN approach cannot achieve high predictive performance (AUROC ~ 0.6), a point also previously made in [5] using epigenomic features. Thus, the two main points of the Tahir et al [3] paper were published six years earlier, but not cited. Recall that AUROC=0.5 is random guessing, and a perfect model is AUROC=1.0. The biologically relevant consequence of the low predictive power of this computational task, as designed, and as noted in [5], is that the local sequence features at the enhancers and promoters alone are insufficient to specify which enhancers interact with which promoters, and an additional non-local mechanism is needed. This is in contrast to TF binding, which can be predicted with high accuracy by the local enhancer or promoter sequence [12, 15] (including cooperative binding sites flanking the TF). So, while TF binding is similar to a lock-and-key interaction between proteins and DNA, enhancer-promoter interaction is not. At least in mammalian genomes, it does not appear that most enhancers are bound by TFs which specifically interact with a subset of promoters which bind a ‘receptive’ set of promoter TFs, but which do not bind with other ‘non-receptive’ promoters. Based on both epigenomic state and sequence feature analysis, enhancer promoter interactions appear to be more permissive.[2, 5, 15]

Since [5] was published, a much clearer picture of the mechanisms which do control enhancer-promoter interactions has emerged. The most compelling additional non-local mechanism is the formation of topologically associated domains (TADs) [16, 17] or insulated neighborhoods [18], which are regions of the genome with high intra-TAD interaction frequency, and low interaction frequency with regions outside the TAD. TADs range in length scale up to ~1MB, but the average TAD size is 300kb [16, 19, 20]. It has been known for a very long time that the protein CTCF can act to block or insulate enhancers from interacting with promoters, first shown in the β-globin [21] and H19/Igf2 [22, 23] loci, and it was shown in some cases that this insulation is mediated by the formation of chromatin loops [24]. The development of Hi-C [25] revealed that these DNA interaction loops existed throughout the genome, and that the genome was partitioned into topologically associated domains, or TADs, [16, 17]. Further experiments showed that targeted degradation of CTCF [26] or cohesin [27] eliminated these domains. However, the impact of TADs on enhancer-promoter interactions was unclear, as the effects on transcription were not as rapid as the loss of the TAD domains. A possible explanation of the slower transcriptional response to TAD disruption may be provided by nonlinear hysteresis in the establishment of enhancer-promoter interactions [28], as described in more detail below.

The leading explanation of the mechanism of TAD loop formation is through the process of loop extrusion [29, 30], whereby the ring-shaped cohesin complexes land randomly on DNA and begin extruding chromatin through the ring in an ATP dependent process to form a growing DNA loop. The progression of loop extrusion proceeds until it is blocked by CTCF binding sites, which were initially observed to be enriched near the boundaries, or anchors, of TADs [16, 31], and later it was shown that CTCF directly interacts with cohesin [32], providing a mechanism for CTCF’s orientation dependent extrusion blocking activity. Thus, while the process is to some degree dynamic, CTCF binding seems to play a central role in the demarcation of cohesin loop extruded TADs.

Since CTCF anchored loops seem to be nearly synonymous with TADs, modeling CTCF interactions is a way to predict the more complicated processes of TAD formation, and is a much more promising approach for enhancer-promoter interaction prediction than the strategy pursued by Tahir et al. However, the problem is still challenging because only a small fraction of CTCF bound regions interact [20]. CTCF looping interactions can be measured with higher spatial resolution (relative to most Hi-C defined TADs) with ChIA-PET [19, 33]. The high spatial resolution of this CTCF interaction data allowed the development of successful non-local CTCF interaction prediction methods [34]. Based on the success of [34], we were able to develop a simpler model of CTCF interaction prediction by incorporating known features of loop extrusion and adding a loop-competition component [20]. This model can predict CTCF loops with reasonable accuracy by only considering CTCF binding strength, orientation, the distance between CTCF binding sites, and their linear arrangement along the genome, which contributes to the loop-competition process described in the model [20].

The development of CRISPRi enhancer perturbation [35, 36] enabled much more precise assessment of the effects of TADs on enhancer-promoter interactions. We performed a high resolution and sensitive CRISPRi screen in the transition of embryonic stem cells to endoderm, and found a set of 24 responsive enhancers which interacted with a set of four gene promoters [28], and a set of adjacent 136 enhancers which did not interact. Using our CTCF looping model [20], it was readily apparent that the positive enhancer-promoter interactions were constrained to be within CTCF loops, and we showed with logistic-regression modeling that the positive enhancer-promoter interactions detected in our CRISPRi screen were most accurately described by co-localization within a TAD or CTCF loop [28]. We produced a summary score to predict CTCF looping, P_inloop, [28] which could more accurately predict transcriptionally productive interactions than direct measurements of Hi-C contact frequency [37]. Because of the strong predictive power of CTCF, we called this model the CTCF-loop-constrained Interaction Activity (CIA) model, but we also emphasized that the CTCF loops are consistent with Hi-C TADs [28]. An additional component of this modeling is a highly nonlinear transcriptional response through auto-regulatory enhancer binding of TF proteins, which may explain some of the observed hysteresis in enhancer-promoter interactions upon TAD disruption. When cells are transitioning and TF protein concentration is low, TADs may be critical for the initial establishment of enhancer-promoter interactions, but TAD disruption may have slower effects once the interactions have been established in a stable cell state [28].

These models of TAD formation and enhancer-promoter interactions are under intense further development [38–40]. It would be ideal to test these competing computational models in blind assessments, such as the Critical Assessment of Genomic Interpretation (CAGI) [41], as has been informative for testing enhancer prediction models [15, 42, 43]. A complication in the case of enhancer-promoter interactions is that CTCF binding appears to be quite independent of cell type, with only a small subset of CTCF binding sites changing occupancy in different cell types. This is further evidenced by the fact that sequence based models of the strongest CTCF ChIP-seq peaks do not require co-factors [7, 12]. Cell-type independence implies that cross-cell type comparisons are likely not actually independent, introducing additional potential for overfitting.

The simplified understanding of enhancer-promoter interactions provided by CTCF looping and encoded in the CIA model P_inloop has already aided the detection of altered enhancer activity in gastric cancer [44–48]. The fact that CTCF binding regions (and by implication, CTCF loops) are much more conserved between human and mouse than enhancers [49] should also help identify which enhancers are functional in mouse, and aid the validation of disease associated enhancers and enhancer variants in animal models [50–52].