Machine Learning in Epigenomics: Insights into Cancer Biology and Medicine

Abstract

The recent deluge of genome-wide technologies for the mapping of the epigenome and resulting data in cancer samples has provided the opportunity for gaining insights into and understanding the roles of epigenetic processes in cancer. However, the complexity, high-dimensionality, sparsity, and noise associated with these data pose challenges for extensive integrative analyses. Machine Learning (ML) algorithms are particularly suited for epigenomic data analyses due to their flexibility and ability to learn underlying hidden structures. We will discuss four overlapping but distinct major categories under ML: dimensionality reduction, unsupervised methods, supervised methods, and deep learning (DL). We review the preferred use cases of these algorithms in analyses of cancer epigenomics data with the hope to provide an overview of how ML approaches can be used to explore fundamental questions on the roles of epigenome in cancer biology and medicine.

Background

The epigenome consists of a diverse repertoire of covalent histone modifications and nucleic acids that cooperatively regulate chromatin structure and gene expression. Epigenetic modifications are reversible and dynamically regulated, initially added and subsequently removed by specialized chromatin-modifying enzymes known as epigenetic ‘writers’ and ‘erasers’, respectively^1,2. Epigenetic mechanisms, acting in conjunction with transcription factors, play a critical role in orchestrating the transcriptional changes associated with differentiation of a variety of cell types, during normal development^3–8. Specifically, they allow signal transduction cascades acting through common transcription factors to drive cell type-specific transcriptional responses, and they provide a mechanism for the heritable maintenance of cell type-specific gene expression after inciting signals have dissipated. While contributions of specific epigenetic elements in isolation has been studied extensively, how multiple epigenetic elements together, and in conjunction with other features such as DNA sequence, precisely control gene expression patterns is not completely understood.

It is becoming increasingly evident that the epigenome plays a major role in the etiology of various diseases, including cancer, diabetes, autoimmune disorders. Epigenome aberrations occur extensively across multiple cancer types and are thought to play a major role in establishing the neoplastic cellular state in conjunction with genetic anomalies. While the precise roles of individual cancer-specific epigenetic elements are currently being investigated, an important and poorly understand element of this process is the interplay of various epigenomic elements with other modules such as metabolic, proteomic, and transcriptomic states.

The advancements in next generation sequencing, clubbed with classical biochemical approaches, have led to innovation in mapping techniques for many epigenetic elements. For example, genome-wide profiles of histone modifications and transcription factor binding can be mapped by ChIP-Seq^9–11, or more recently CUT&Tag¹² among other methods; chromatin accessibility can be determined using ATAC-Seq¹³ and DNaseI-Seq¹⁴. Higher-order chromatin structure is determined using methods such as Hi-C¹⁵. DNA methylation is widely determined using RRBS¹⁶, WGBS^17,18, or array-based technologies. These technologies have provided massive amounts of new data in various cancer systems that can provide insights into how epigenome may regulate the evolution of cancer cells during progression and response to therapies. Furthermore, the epigenomic landscape of cancer tissues can be used to stratify patients into groups for precision medicine. To achieve these objectives, we need to implement automated decision systems that can help generate novel approaches for prevention, diagnosis, and treatment.

Overview

Machine Learning (ML) enables computers to learn from data without being explicitly programmed and make accurate predictions. ML models have created unprecedented momentum in different domains, including epigenomics studies. In this review, we categorize traditional ML frameworks based on principal focus of ML applications in epigenomics. We will review several conventional supervised and unsupervised learning methods that have been used for the examination of epigenomics data.

Machine learning applications have played crucial roles in addressing multiple questions related to the basic biology of epigenetic elements, their role in gene regulation, and the utility of the epigenome in cancer diagnostics and treatment. We present specific studies from the literature where conventional ML tools have been used to address these fundamentally important biological questions. Figure 1 presents the overall structure of this review. The paper is organized as follows: Dimensionality reduction section presents an overview of several feature extraction and feature selection algorithms used in epigenomics studies. We review major supervised learning and unsupervised learning algorithms and their application in the epigenomics field. Next, several deep learning (DL) methods in epigenomics analysis are presented in detail. Even all deep learning models follow a supervised or unsupervised learning strategy, we separated the deep learning section to discuss many methods in detail. Finally, we present a brief discussion on several challenges in epigenomics analysis and close with a focused discussion of future directions in the field.

Figure 1. Overview of the Machine Learning in Epigenomics.

Machine Learning methods can be used for analyses of datasets defining maps of higher-order chromatin structure, chromatin accessibility, DNA Methylation, and histone modification in various cancer systems to address fundamentally important biological questions.

Dimensionality Reduction

The dimension of a dataset is the number of features (also called attributes or variables in the ML literature). If the dimension p is much larger than the number of samples N (p >> N), it is called high-dimensional data¹⁹. The sizeable gap between the number of features and the observation size in high-dimensional epigenomic datasets leads to the curse of dimensionality²⁰, where several fundamental problems arise such as increased computational complexity, multicollinearity²¹, and decrease in expected predictive power (Hughes phenomenon)²². Applying a dimensionality reduction algorithm before training a classification or clustering method is crucial to identify relevant epigenomic signatures and avoid overfitting. Dimensionality reduction methods are also valuable to visually inspect the data and check the distribution of samples/cells based on their epigenomic background. There are two main approaches for dimensionality reduction: feature extraction and feature selection.

Feature Extraction (FE) methods transform the original high-dimensional data to a lower dimensionality using linear or non-linear operations. The primary goal of a FE model is to preserve as much information as possible with fewer variables. Principal Component Analysis (PCA), t-distributed Stochastic Neighbor Embedding (t-SNE), and Uniform Manifold Approximation and Projection (UMAP) are some of many FE methods in epigenomics analysis. PCA is a linear transformation based on an eigenvector search^23,24. Principal components (PCs) are directions that capture the maximum variances from the original data and are orthogonal to each other. PCA is a classical technique for exploratory data analysis (EDA) as its probabilistic formulation is computationally efficient, especially for very high-dimensional datasets like epigenomic datasets²⁵. While there are more than ten variants of the original PCA applied in different domains^26–36, only a couple are implemented in epigenomic analysis^37,38. The major limitation of PCA is its inability to capture non-linear relationships in the original data. t-SNE is a powerful FE method to capture local and non-linear patterns from the original high-dimensional data. It extends the Stochastic Neighbor Embedding (SNE) method³⁹ which maps objects that represented by high-dimensional vectors to a low-dimensional space while preserving neighbor identities, by introducing a symmetrized version of the cost function and Student-t distribution to measure similarities between observation pairs⁴⁰. UMAP uncovers a low-dimensional graph representation of the original data⁴¹ and is widely used to visualize high-dimensional genomics and epigenomics datasets. UMAP’s solid theoretical framework built on manifold theory and topological data analysis serves as a fast FE method that capture nonlinear structure in the high-dimensional representation⁴¹. Several studies, focused on comparing tSNE and UMAP, have concluded UMAP is faster⁴² and provides better scaling than t-SNE⁴¹, yet a more recent study shows no difference of preserving the global structure of the high-dimensional data between UMAP and t-SNE and the implementation of FIt-SNE is as fast as UMAP. An issue for t-SNE is choosing the right perplexity^43–46 as different perplexities often produce reasonably different outcomes⁴⁷.

Feature selection (FS) is a straightforward approach to reduce the number of features before any ML step. FS algorithms select a few features from the entire feature space and discard the rest. Choosing the right FS method can improve prediction performance, generalizability, and scalability. Many FS methods exist to efficiently reduce the number of input features, but we will focus on the most well-known and widely used ones. Correlation-based Feature Selection (CFS) ranks features according to their correlation with the class label and chooses the most correlated features by assuming that features with a low correlation with the label are irrelevant⁴⁸. Information Gain (IG) evaluates the relevance of each feature to a corresponding class by mutual information. Higher mutual information indicates a higher relevance of variable to the label⁴⁹. IG does not depend on multivariate analysis, making it possible to identify redundant features. Gain Ratio (GR) extends IG via normalizing with the entropy of the variable (intrinsic information) and reducing the symmetric measurement bias⁵⁰. ReliefF⁵¹ is a descendant of the original Relief Algorithm⁵² that weights each feature according to their skill to agree on the same labeled closest instance and differentiate on closest different class observation, where the F represents the sixth variation (A-F) of the original algorithm. ReliefF chooses a random sample and finds the nearest same class k samples (Near Hits) and the nearest other class k samples (Near Misses) and then searches for attributes that are preserved within the same class and differentiates between distinct classes⁵¹. ReliefF also extends the original idea to handle missing data and multi-class labels⁵³.

A few recent FE or FS methods have focused on addressing the high dimensionality of epigenomic datasets. For example, Alkuhlani et al.⁵⁴ introduce a multistage FS strategy to identify the optimal CpG sites for breast, colon, and lung cancer datasets. Han et al.⁵⁵ propose a dynamic Recursive Feature Elimination (dRFE) based FS method. In general, deep learning-based pipelines skip a FE or FS step in epigenomics but applying an FS method before any deep neural network model can significantly improve the performance⁵⁶.

A dimensionality reduction approach is a must for all epigenomic data analysis pipelines. A FS method selects the most relevant epigenomic features, that are the most discriminative for a particular experimental setup. A FE method helps to project the epigenomic data from hundreds of thousands features to several hundreds while preserving most insight and visualize the high-dimensional epigenomic data in 2D or 3D. Detailed mathematical explanations of feature selection and feature extraction methods are beyond the scope of this paper, but comprehensive FE and FS reviews are available^57–59.

Supervised Learning

Supervised Learning is a widely used ML strategy to solve various real-world problems^60–62. The existence of labels associated with the data is the fundamental requirement for a supervised learning setup. Supervised Learning algorithms search for linear/non-linear patterns from data-label pairs and estimate the output for newly presented input data. They answer several fundamental biological questions such as diagnosing cancer^63,64, predicting the stage of cancer⁶⁵, understanding factors affecting the aging⁶⁶, and reporting oncogenes and tumor suppressor genes⁶⁷. There are two subcategories of this branch of ML: Classification and Regression. Classification tasks predict categorical labels (e.g., estimating cancer stage via ATAC-Seq), while numerical outputs are estimated by regression tasks (e.g., evaluating gene expression values using histone modification marks). Support Vector Machine (SVM), Decision Trees, Random Forest, Naïve Bayes (NB), and Logistic Regression are some of many off-the-shelf supervised learning algorithms. SVM relies on finding the hyperplane that separates two classes of input data (binary classification)⁶⁸. The hyperplane serves as the decision boundary which needs to be equally distant from the closest points of the two categories. The ‘Support Vectors’ term refers to the closest samples to the hyperplane from each category/label⁶⁹. Decision Trees (DTs) are flexible and easy-to-use classification methods. The algorithm builds a tree, splitting based on feature values in each layer, and repeats this partitioning until a leaf node has all the same class observations⁷⁰. A large number of relatively uncorrelated DTs build the Random Forest (RF) classifier⁷¹ which uses the wisdom of the crowd. The ensemble structure of the RF reduces the contribution of individual DT errors. Naïve Bayes (NB) is another widely used supervised learning method that is fast and easy to interpret supervised learning method, which depends on conditional probabilities from Bayes Theorem⁷². NB’s principal assumption is the independence of all attributes from each other, which is unrealistic in many real-world applications. Logistic Regression (LR) models the relationship between input data and labels using a logistic function that has an S-like shape and projects any real-valued number between 0 and 1. LR predicts a dichotomous output by using a threshold on the projected range. Linear Regression takes features (explanatory variable) as input and estimates continuous output (dependent variable) by fitting a linear equation⁷³. It is fast and easy to interpret as estimated weights for each variable reveals the importance of each feature to estimate the output. Despite its popularity in data analysis, Linear regression is prone to over-fitting, outliers, and multicollinearity⁷³. Lasso Regression⁷⁴ and Ridge Regression⁷⁵ methods mitigate mentioned problems and reduce the complexity of the model. Lasso Regression performs L1 regularization that adds the absolute sum of the estimated coefficients as the penalty term to the sum of the squared error objective function whereas Ridge Regression uses L2 regularization where the penalty term is the squared absolute sum of coefficients.

Diagnoses of most cancer types are based on biopsies, computerized imaging scans, or even autopsies. While highly effective, these approaches sometimes lead to errors for an accurate diagnosis. For example, pathological examination of needle biopsies to diagnose prostate cancer has high false-negative rates⁶³. Therefore, exploiting the molecular features of tumors could be useful in such cases. For this, cost-efficient and time-efficient diagnostic tools are crucial. Vast amounts of epigenomic data generated from several cancer cells are great candidates for a supervised learning framework as each observation has label corresponding to epigenomics features. We can potentially find epi-signatures for those cancer types by training classifiers. It has been shown that by using the genome-wide DNA methylation changes as input, an L1 regularized logistic regression model can differentiate tumors from benign samples with high sensitivity and specificity rates⁶³. A striking example of the utility of cancer epigenomes to classify subtypes of tumors was shown in Capper et al.⁶⁴, where DNA methylation was used as input to train an RF classifier that predicted 100 known tumor types across all entities and various age groups. Large cohorts in autopsy studies show that up to 26% of cancer patients develop Brain Metastases (BM)⁷⁶ that emphasizes the importance of an accurate diagnosis before an optimal treatment plan. The BrainMETH⁷⁷ algorithm was developed for the accurate diagnosis of BM in a three-step RF-based classifier that uses DNA methylation as the input. Class A -first step- classify primary and metastatic brain tumors, Class B -second step-determine the tissue of origin, and finally Class C -third step- focusing on characterizing of breast cancer BM (BCBM) subtypes. High prediction performance over a large cohort (n = 165) has made BrainMETH a reliable diagnostic tool. In another study, Kaur et al.⁶⁵ used SVM, RF, NB, and DTs as classifiers and DNA methylation as input for the clinical staging prediction of Liver Hepatocellular Carcinoma (LIHC). Finally, Deng et al.⁷⁸ use combined SVM-RFE and LASSO to predict a higher risk of progression for colorectal cancer (CRC) patients and listed 222 candidate epigenetic CpGs related to CRC progression.

It is well known that several potent tumor suppressors are silenced by DNA methylation^1,2,79–84, whereas enhancers activate oncogenes^85–89. Therefore, DNA methylation, histone modification marks, chromatin openness/closeness, and 3D chromatin interaction datasets can be highly useful in identifying oncogenes or tumor suppressor genes^90–92. The DORGE (Discovery of Oncogenes and tumor suppressoR genes using Genetic and Epigenetic features)⁶⁷ ML model, which is an elastic net–based logistic regression classifier, was proposed with this purpose. Authors show that histone modifications effectively predict tumor suppressor gene expression while super-enhancers and methylation change effectively predict oncogene expression. Li et al. use five conventional supervised learning classifiers to find differentially expressed genes (DEGs) via CpG methylation, histone modification marks (H3K4me3, H3K27me3, and H3K36me3), and genomic nucleotide sequence datasets for lung cancer samples⁹³. The study shows the predictive power of histone modifications and CpG methylation for DEG estimation. Dixon et al.⁹⁴ use chromatin state features and the RF classifier to predict changes in interaction frequency. Based on their feature ranking, H3K4me1 has the most predictive power on long-range chromatin interactions estimation. Their integrative analysis that combines Hi-C, histone modifications, and CTCF binding data reveals dynamic chromatin structure reorganization during embryonic stem cell differentiation.

Enhancers are shown to be activated in multiple cancers. However, identification of enhancers is not a straightforward task as enhancers can be several hundred kilo- or mega-bases away from the gene body, or even be in a different chromosome⁹⁵. Sethi and Gu⁹⁶ developed a framework that uses a linear SVM as the classifier. The model gets self-transcribing active regulatory region sequencing (STARR-Seq), DNase-I hypersensitive sites (DHS), and H3K27ac in the first step (matched-filter scores step) and integrates matched-filter scores with H3K27ac, H3K4me1, H3K4me2, H3K4me3, and H3K9ac histone modification marks in Drosophila. Their findings on the test set confirmed the high enhancer prediction performance in this model. Authors also show their framework can predict enhancers in mammals with no re-parametrization. TargetFinder⁹⁷, which ensembles several classifiers, integrates DNA methylation, ChIP-Seq, DNase-Seq, and Cap Analysis of Gene Expression (CAGE) to estimate promoter-enhancer contacts.

Supervised Learning ML methods can help to predict missing epigenomic signals via other epigenomic datasets. Fu et al.⁹⁸ use logistic regression to understand the relationship between DNA methylation and histone modifications. They show the importance of five core histone marks (H3K4me1, H3K4me3, H3K27me3, H3K36me3, and H3K9me3) to explain the variance that was observed across methylomes. 3DEpiLoop⁹⁹ takes H3K4me1, H3K27me3, H3K36me3, CTCF, and RAD21 as input and predicts chromatin looping interactions within topologically associating domains (TADs). The authors applied several classifiers such as AdaBoost classification trees, Neural Networks, SVM, and Stochastic Gradient Boosting and chose an RF classifier for the model. ChromImpute¹⁰⁰ uses an ensemble of regression trees to impute a genome-wide epigenomic signal at 25-bp resolution. Authors show that motif analysis of these divergent locations agrees with cell type–specific regulatory mechanisms. PREDICTD (PaRallel Epigenomics Data Imputation with Cloud-based Tensor Decomposition) is a tensor factorization-based model to impute epigenomic data¹⁰¹. It is a computationally efficient method and is one of few methods^102,103 that uses tensor decomposition in epigenomics analysis.

Unsupervised Learning

Unsupervised ML methods assign samples/observations to several clusters (groups) based on their feature set similarity¹⁰⁴. The power of unsupervised learning is that it can discover underlying biology unbiasedly as it does not need labeled data. Unsupervised ML algorithms reveal novel biological findings such as discovering novel molecular subgroups of patients for a specific histologically defined cancer type.

We will discuss traditional unsupervised learning methods and novel clustering algorithms proposed for epigenomics datasets. Hierarchical Clustering (HC) groups samples through the similarity matrix and recursively clusters a pair of observations at a time. Most of the HC methods used in epigenomics are agglomerative HC models. In the beginning, each observation starts in its own cluster followed by merging of closer units until one robust cluster is identified. Wiwie et al.¹⁰⁵ show hierarchical clustering is one of the best-unsupervised ML methods among 13 well-known clustering methods that analyze 24 biomedical datasets. A significant drawback of the HC is that the algorithm needs a cut-off value to define the number of final clusters¹⁰⁶. Lin and Chen et al.¹⁰⁷ applied HC on DNA methylation and find tumor-specific hypermethylated clusters for breast cancer cell lines. Virmani et al.¹⁰⁸ analyze the methylation status of 89 cell lines. The dendrogram of the HC has two major groups that are consistent with small cell lung cancer (SCLC) and non-small cell lung cancer cell lines (NSCLC) cell lines. Lin and Wang¹⁰⁹ integrated gene promoter methylation and gene expression profiles to find candidate genes in NSCLC. Hierarchical clustering of 578 candidate genes agreed with the Epithelial-to-Mesenchymal Transition (EMT) phenotype segregation of cell lines.

k-means is a simple yet one of the widely-used clustering methods. This unsupervised learning method partitions all samples into a fixed number ‘k’ of clusters. The algorithm starts with a random initialization (assigning some individuals as ‘means’), associates every observation to the closest ‘mean’, then assigns centroids of each cluster as new ‘means’, and returns to the second (associating all samples to the new ‘means’) step¹¹⁰. This recursive setup is repeated until getting a final stable clustering. k-means is sensitive to its random initialization. Different initializations lead to having diverse final clusters¹¹¹. K-means and HC methods have been extensively used by the TCGA and other genomic studies to identify patient subgroups based on their DNA methylation profiles. Examples include classical hypermethylated CIMP phenotype in colorectal cancer¹¹² and G-CIMP in glioblastoma¹¹³. Substantial biological insights have been since gained on these subgroups of patients and specific therapeutic strategies are being designed^114,115. Zhang et al.¹¹⁶ use an extension of k-means algorithm¹¹⁷ to cluster TCGA ovarian cancer samples by DNA methylation, microRNA expression, and copy number alteration datasets. Authors uncover seven subtypes of ovarian cancer, which have significantly different survival rates. They also list ‘driver’ genes for each subtype that could be targeted for a treatment plan.

Non-negative Matrix Factorization (NMF) is a popular clustering and dimensionality reduction method in epigenomics. It was proposed to get a low-rank representation of a non-negative matrix¹¹⁸. The first clustering attempt via NMF in document clustering showed NMF can be used as a powerful unsupervised learning method¹¹⁹. After a year of the successful document clustering attempt, the NMF was applied to biological data¹²⁰ and has since become very popular in genomics and epigenomics analyses. Mishra and Guda¹²¹ use NMF clustering on differentially methylated sites between healthy and Pancreatic Cancer samples. They identify three molecular subtypes in Pancreatic Cancer that have different enrichment of neoplasm histological grades and pathologic T-stages. Through the NMF on H3K27ac data, we identified four clusters (EpiCs) of colorectal cancer samples based on their underlying enhancer patterns¹²². Importantly, these EpiC subtypes allowed stratification of these patients into groups that were vulnerable to specific combinations of targeted therapeutics where one agent was enhancer-blocking bromodomain inhibitors.

Ernst and Kellis proposed ChromHMM¹²³ that is based on a multivariate hidden Markov model as an unsupervised ML method to analyze epigenomic marks. ChromHMM assigns chromatin states to the corresponding genomic location. ChromHMM method permeates the epigenomic unsupervised analysis literature^124–128. One may think ChromHMM takes binarized epigenomic signals and predicts chromatin states as in a supervised learning scheme, yet the predictions are not ground-truth labels (cancer type/stage, age, expression of a gene, etc.) associated with the given data, but candidate-state annotations. Other similar methods are Segway¹²⁹ and EpiCSeg¹³⁰. Segway is a dynamic Bayesian network (DBN) that models the transformed data (inverse hyperbolic sin) with the Gaussian distribution and trains the model with the expectation-maximization (EM) algorithm. EpiCSeg¹³⁰ passes read counts from several histone marks to a Hidden Markov Model (HMM) like ChromHMM.

Deep Learning

Deep Learning is a subfield of Machine Learning that uses artificial neural networks (ANN or NN) for analysis¹²³. As a general rule, NNs are more flexible than other algorithms, can handle highly complex non-linear data, and have shown highly accurate empirical performances. However, they also require more data for training and can be more difficult to train and interpret than traditional machine learning algorithms. NNs are composed of multiple sequential layers that each consist of two components: a set of artificial neurons and a non-linear activation function. This structure allows each layer to flexibly model relatively simple non-linear data with practically no assumptions. Stacking these layers sequentially nests these simple but flexible non-linear functions to flexibly model complex non-linear data. All models make some assumptions about input data. Stricter assumptions decrease training data requirements, but possibly limit model accuracy if the data does not match those assumptions. Because of their architecture, NNs can analyze highly complex data with minimal assumptions. This makes NNs well suited to analyzing high-dimensional and highly complex epigenomic data to arbitrarily high theoretical accuracy when sufficient training data is available. Because NNs use a wide variety of techniques that are specific to the subfield of deep learning and every NN uses a slightly different combination of those techniques, in depth descriptions will focus on a small number of papers per data type, each of which introduces one or more techniques or approaches. We will then introduce a larger variety of papers in decreased depth to provide a general awareness of current research.

Neural Networks in Epigenomic Data Analysis

Prediction of gene expression from epigenomic data:

A number of epigenetic elements function together to exert precise control of the expression of target genes. The complex interaction of these elements makes gene expression difficult to predict, and the regulatory contribution of each epigenetic element even more difficult to interpret. Due to their ability to handle highly complex non-linear data, NNs are being increasingly used to predict gene expression from epigenetic data, with tools being developed to inspect NNs internal functioning and interpret the contribution of individual epigenetic elements. An outline of how an NN could analyze epigenetic data is shown in Figure 2.

Figure 2. Neural networks function through pattern recognition.

A hypothetical small NN is shown which has learned to use the local ATAC-seq pileup pattern to predict expression and is predicting expression of the highlighted gene. Each circle represents a neuron, and each row of neurons represents a layer. Arrows represent connections between neurons in adjacent layers. Each neuron learns from training data to identify a specific combination of its inputs that represents a relevant biologically pattern. The images on the neurons represent possible learned patterns, and the numbers next to each neuron represent how well the input data matches the neuron’s learned pattern on a scale from 0 (no match) to 1 (perfect match). Because neurons in later layers take patterns from previous layers as input, learned patterns become more complex in later layers, allowing a “deep” NN with many layers to learn highly complex patterns. Real NNs have hundreds of neurons per layer and many tens of layers and are expected to learn much more complex patterns than those shown here.

Methylation and DNA Sequence Data:

MRCNN used convolutional NNs (CNN) to predict genome-wide methylation levels based on the nearby DNA sequences¹²⁴. CNNs prioritize local connections by using sliding windows that look for patterns in small, limited regions. Where each layer in a typical NN has artificial neurons and a non-linear activation function, CNNs add a compression step, called “pooling”, that decreases the size of the data after each layer. By compressing the data after each layer and using a window that is the same size, the sliding window at each layer effectively covers larger and larger portions of the input data, but in lower resolution, learning more complex patterns that span larger portions of the genome. The final layer of a CNN connects all of the patterns found across the region of interest in previous layers to detect highly complex patterns that would require many more connections to identify in a classic NN. This means CNN¹²⁵ could be useful in predicting gene expression patterns from DNA methylation and histone modification patterns as well as the underlying DNA sequence.

MRCNN uses a 400bp window around the methylation site in the first layer, using the raw DNA sequence to learn 400bp motifs associated with hyper-, or hypo-, methylation in their samples. This method predicted methylated vs unmethylated regions with an accuracy of 93.2%, outperforming previous methods such as DeepCpG (approximately 87%)¹²⁶.

Methylation and Variation Autoencoders:

Autoencoders (AE) are NN-based dimensionality reduction models which consist of two NNs: an encoder, which creates the compressed representation called an encoding, and the decoder, which recreates the original data from the encoding. Forcing the decoding to match the original data incentivizes AEs to produce informative compressed encodings. Compressed representation are expected to resemble principal component analysis (PCA)¹²⁸, wherein regions with high variability between samples take priority in the encoding over those that are nearly identical between all samples and multiple highly correlated regions are compressed into a single value. The primary advantage of AEs over PCA is the ability to learn non-linear correlations, allowing AEs to learn biologically relevant patterns that PCAs may miss, such as buffer effects, synergistic effects, and antagonistic effects. Variational AEs (VAE¹²⁹) extend the AE model by putting additional constraints on the distribution of the lower-dimensional encoding. In the VAE, some noise is added to the encoding and it is required to match some pre-defined distribution, typically a multidimensional normal/gaussian distribution. These constraints make the encoding process probabilistic, which improves the correlation between sample similarity and encoding similarity. These probabilistic NNs are hence called “generative models”, because the enforced structure of their encodings gives them the ability to regenerate the original dataset. This allows them to be used more widely for varied forms of statistical modeling.

Wang and Wang, 2019, used a variational autoencoder (VAE) alongside t-SNE to compress 450K methylation data for logistic regression classification¹²⁷. Wang and Wang used a VAE to reduce the dimensionality of their data from 300,000 probes to 100 compressed dimensions. For visualization, t-SNE was used to further compress the data to two dimensions. The data consisted of two lung cancer subtypes (LUAD-01 and LUSC-01) and two matching adjacent normal groups (LUAD-11 AND LUSC-11). The two-dimensional non-linearly compressed representation split the four groups well, implying that the features learned by the VAE and t-SNE mapping represent biologically relevant features to separate the four sample groups. Additionally, expected biology features were recapitulated in the compressed representation: non-cancer samples produced tighter groupings than cancer samples and the greatest overlap between groups occurred between cancer samples and their matched non-cancer samples. A logistic regression model used to separate the 4 groups obtained classification precisions of 0.92 (LUAD-01), 0.99 (LUSC-01), 0.75 (LUAD-11), and 1.00 (LUSC-11) with many errors mixing a minority of cancer samples with their matched noncancer samples and a few LUSC-01 samples showing LUAD-01 profiles. This paper shows the value of VAE encodings for interpreting complex, high dimensional non-linear data.

Histone modification data (ChIP-seq):

We will discuss three models that have been developed to predict transcription from raw ChIP-seq pipeline values, with progressively more features improving performance. The first, DeepChrome, used a CNN to predict transcription from local ChIP-seq pileup values¹³⁰. DeepChrome out-performed baseline models in predicting transcription (Linear Regression, Support Vector Machines, Random Forest, a custom Rule-Based algorithm), implying that it learned biologically relevant regulatory patterns. However, the model did not include a procedure to view and assess these patterns, limiting biological interpretability.

The second method, AttentiveChrome, used a recurrent NN (RNN)¹³¹. Unlike typical NNs, RNNs effectively analyze sequential data because they preserve information from previous inputs. This allows them to analyze an input from a sequence in the context of previously provided inputs. RNNs are described in greater detailed in the referenced review, Lipton et al 2015^132,133. AttentiveChrome used a specific form of an RNN, a Long Short-Term Memory (LSTM) model which uses multiple NNs to intelligently update only small portions of the saved state, the preserved information, each time an input is provided. This limits the loss of older data and improves its ability to learn complex long-range interactions. AttentiveChrome also included an “attention module” which explicitly models the importance of specific inputs thus increasing the “attention” paid to highly important inputs on a sample-by-sample basis. This decreases the noise influencing predictions, thereby increasing model accuracy on future samples by decreasing the contribution of inputs with likely spurious correlations. Secondly, the outputs of the attention module can be extracted and viewed, elucidating the regions of the ChIP-seq pileup that most correlate with expression on a sample-by-sample basis. This method identified regions that may play mechanistic roles in regulating expression, providing targets for further mechanistic studies. The attention module, and its contribution to biological interpretability, was the primary contribution from the AttentiveChrome module.

The third method, DeepDiff, is an LSTM NN that instead predicts differential genes between two samples based on their local ChIP-seq profiles¹³⁴. This model further improves prediction accuracy through two additional techniques: Multitask learning¹³⁵ and Siamese contrastive loss^136–138. Multitask learning improves prediction accuracy by forcing the network to make multiple predictions per sample on input data. DeepDiff learned to predict differential expression as the main task and learned two auxiliary tasks. First, it learned to classify the sample’s cell type, next it learned to minimize the “siamese contrastive loss” on its sample encoding. The siamese contrastive loss forces a single NN to learn to identify similar encodings in similarly expressed genes and different encodings in differentially expressed genes. It does this by passing two ChIP-seq profiles for the same gene in different samples through the same network to generate an. The model then measures the difference in the encoding and compares it to the difference in expression, punishing the NN if a differentially expressed gene has two similar encodings or vice versa. DeepDiff performed slightly better than AttentiveChrome in predicting gene expression, however, DeepDiff’s primary contribution was the combination of three tasks that, taken together with its attention module, improve biological interpretation of the model. Attention mechanisms showed multiple expected correlations: the enhancer mark H3K4me1 and the activating mark H3K4me3 were associated with increased expression, while the heterochromatin mark H4K9me3 and inhibitory mark H3K27me3 were associated with decreased expression. However, attention also unexpectedly showed the gene body-associated mark H3K36me3 associated with decreased expression. This could result from a failure of the model or could point to some yet unknown regulatory relationship, highlighting the potential to uncover previously unknown biological relationships using highly flexible models such as an NNs.

Higher order chromatin structure (Hi-C):

DeepExpression predicts gene expression from promoter DNA sequence and Hi-C promoter-enhancer interactions¹³⁹. DeepExpression is composed of three NNs, a “densely connected CNN”¹⁴⁰ that analyzes promoter DNA sequence, a fully connected NN that analyzes enhancer-promoter (EP) interactions, and a fully connected NN that integrates output from the previous two NN modules. DeepExpression outperformed random forest, linear regression, and lasso-regularized linear regression in predicting transcription. To interpret model results, researchers implemented “model ablation analysis”, in which the model is run multiple times with different portions of the input hidden from the model, to assess the contribution of different portions of the input to prediction accuracy. They showed that the majority of the effect of the promoter sequence could be predicted using just the sequence within 1kb of the promoter whereas the contribution of enhancer-promoter contacts (predicted by Hi-C data) was less than that of the DNA sequence. Lastly, researchers extracted learned sequences from the promoter and EP interaction modules and used these sequences for motif analysis, identifying transcription factors associated with promoter and enhancer regions. DeepExpressions shows the power of NNs to learn complex patterns in input data and extract actionable information to direct downstream experiments.

Neural networks for patient stratification based on epigenomic data

ATAC-seq Sample Clustering: Wasserstein Autoencoder, Generative Adversarial Network

Using an Encoder-GAN based on the “Wassertein autoencoder” (WAE)¹⁴¹, a recent study developed a model, ClusterATAC, to accurately cluster 401 TCGA tumor samples based on the chromatin accessibility profiles mapped by ATAC-seq¹⁴². Wasserstein AEs, similarly to VAEs, extend the AE model by putting additional constraints on the distribution of the lower-dimensional encoding. The difference between Encoder-GAN and VAEs is complex and beyond the scope of this review, but both are probabilistic generative models. In ClusterATAC, the NN was used to compress the data down to an encoding of 200 values between approximately −10 and 10. Sample encodings were then clustered using a gaussian mixture model, producing 22 cancer subgroups from 401 TCGA samples. These clusters had high correlations with Kaplan Meier survival curves and cancer type, implying that learned sample clusters effectively represented real underlying biological differences between TCGA samples. This appears to be an effective technique for biological interpretation of sparse, noisy, and high-dimensional epigenomic data and will be seen again as a popular strategy for analyzing multiOmics data.

MultiOmics analyses using NNs

Integrative analyses of mRNA, miRNA, and Methylation using AutoEncoders

AEs are highly useful in analyses of multiOmics data. Whereas linear techniques would suffice for analysis if combined regulatory effects were purely additive, the non-linear nature of AEs effectively models expected synergistic and antagonistic effects when combining multiple mechanisms of expression regulation. As an example, Chaudhary et al. 2018 used an AE as a pre-processing step for k-means clustering to predict survival in liver cancer¹⁴³. In this study, the partially pre-processed matrices from the three data types (mRNA, miRNA, methylation) for each sample were stacked and input into the network. The multidimensional compressed representations of each sample were then passed through a univariate Cox proportional hazards (Cox-PH) model to select compressed features for which a significant Cox-PH model was obtained. These compressed features function similarly to components in PCA, however, they represent the presence or absence of some non-linear pattern. These significantly survival-associated, compressed features were then used to cluster samples using the K-means clustering algorithm. Lastly, to classify samples to each cluster for downstream analysis, mRNA, miRNA, and CpG sites most associated with each cluster were selected by ANOVA to train classifiers for each cluster. These survival-associated profiles were also shown to correlated with risk factors known to be associated with HCC survival, which implies an association with meaningful underlying biological processes.

Integrative analyses of miRNA, Methylation, and CNV using AutoEncoders, SHAP Values

PathME, 2020, used multi-modal sparse denoising AEs in concert with NCI Pathway Ineraction Database pathways to encode a pathway score for each pathway for each sample¹⁴⁴. The matrix of scores for every pathway-sample combination was then clustered using NMF clustering to generate sample clusters and pathway clusters simultaneously. In addition, researchers used Shapley Additive exPlanations (SHAP) values to interpret the contribution of specific epigenomic elements to pathway scores, providing insight into the regulatory processes underlying known biological processes. As previously described, AEs are non-linear dimensionality reduction NN models that use an encoder to compress data and a decoder to reconstruct the initial input. PathME uses a two-step encoder to produce pathway-specific encodings. The first step uses datatype-specific encoders to encode epigenetic features for all genes in a specific pathway. The second step concatenates these encodings and further compresses to a single pathway score for the sample.

Researchers tested the model on four multiOmic TCGA datasets: Colorectal cancer (CRC), lung squamous cell carcinoma (LSCC), glioblastoma multiforme (GBM), and breast cancer (BRCA). PathME appears to cluster data well based on multiple metrics that can be viewed in the original paper. Interesting, clustering of GBM samples showed three clusters with similar survival durations and a single cluster with significantly longer survival duration, implying that these clusters represent biologically meaningful biology. Lastly, researchers used SHAP values to interpret predictions, obtaining greater insight into model predictions than other papers represented here. SHAP values are a metric based on game theory that attempt to quantify the importance of specific inputs to the final prediction. Researchers used SHAP value absolute values, meaning that larger SHAP values correspond with a larger effect on the prediction. The calculation of SHAP values is highly complex and described in Lundberg et al, 2017¹⁴⁵. SHAP values were used to identify specific epigenetic features in specific genes that majorly contributed to scores for biologically relevant pathways. In CRC, the fibroblast growth factor signaling pathway played a major role, which was expected biologically. SHAP values showed that known prognostic markers of FGF19, FGFR2 and miR-31–3p/5p expression majorly contributed to this pathway’s score.

PathME incorporated biological knowledge through pathway-specific encodings and improved model interpretability by using SHAP values to interrogate the contribution of individual epigenetic features to pathway scores. These methods circumvent some of the major limitations of NN models and show new ways that NNs can be incorporated into biological analysis pipelines.

Integrative analyses of multiOmic single cell datasets:

scMVAE¹⁴⁶ incorporated the expected structure of single cell RNA-seq and ATAC-seq data into its model using generative technique inspired by scVI¹⁴⁷, SCALE¹⁴⁸, and MVAE¹⁴⁹. The contributions of these models will be discussed when the techniques they inspired are introduced. There are three versions of the model, each of which uses a different encoder model. Inspired by SCALE, every version of the model encodes to a gaussian mixture rather than a single Gaussian. The Gaussian mixture encoding distribution has the advantage of allowing multiple modes of cell state in the data. In complex samples, these modes tend to represent different clusters of cells, such as different cell types, or cancer vs noncancer cells. This tends to improve model accuracy when multiple clusters of cells are expected because the model is not required to squish the data into the single-mode assumed by classical generative NNs such as VAEs and WAEs. Based on the scVI model, every version of the model decodes the encoding to predict a distribution of mRNA expression levels and ATAC-seq pileup levels rather than a specific value. The distribution used, the zero-inflated negative binomial distribution (ZINB), builds assumptions about the distribution of scRNA-seq and scATAC-seq into the model. Allowing the model to explicitly predict a ZINB distribution when decoding rather than a specific value is expected to improve model robustness and generalizability by allowing the model to seamlessly incorporate noisy values without being thrown off by extreme values. For reasons beyond the scope of this article, typical NN predictions using least squares error are vulnerable to extreme values.

The three encoding models used in scMVAE will be discussed in order of increasing complexity, with the simplest encoding making minimal assumptions about the data and the most complex model making the greatest assumptions. As discussed in the introduction, models that make stringent assumptions about input data often require less training data to make accurate predictions, but may be inaccurate in spite of infinite training data if model assumptions are incorrect. This means that the most complex version of the scMVAE model may be expected to make the most accurate predictions in the presence of limited data, but only if model assumptions are empirically shown to fit the data. The first version of the encoder stacks all input data formats to put directly into the encoder. Given the limited amount of available data, this version, called scMVAE-direct, had the worst results among the three versions. The second version, called scMVAE-NN, used a single encoder for each data type to encode data type-specific patterns, followed by an encoder that combined the two single data type encodings. By allowing the final encoder to combine already encoded representations of each data type, this version likely improved the final encoder’s ability to find complex multiOmic patterns on a limited dataset. The last encoder, called scMVAE-POE, is based on the MVAE model. This model encodes each data type independently to a single multivariate Gaussian distribution, similar to the VAE. To combine the encodings from each data type, all data type-specific encodings are input into a Product of Experts (PoE) model, a special kind of inference network that limits the amount of data required for efficient training¹⁵⁰. scMVAE-POE is more complex and makes greater assumptions than previous NN models. However, using generative assumptions inspired by bayesian generative modeling in tandem with the flexibility of NNs, this model minimizes the data required for accurate modeling.

The models were tested on simulated data generated by SPLATTER¹⁵¹ and on cancer samples. The models were compared against clustering using the non-linear singleOmics methods scVI and Seurat (on scRNA-seq data and scATAC-seq data separately), and the linear multiOmic techniques intNMF and MOFA. Most models performed well clustering simple simulated data, however scMVAE-NN and scMVAE-POE performed better on complex simulated data. On real data most models performed well, however as dropout increased, the performance of other models degraded more rapidly than the performance of scMVAE. These results should be confirmed on additional datasets, however they are expected due to the structure of scMVAE-NN and scMVAE-POE, which have the flexibility of NN models while building in structure to reduce the required training data. Lastly, researchers showed that scMVAE also recovered correlations in expression between transcription factors (TFs) and their target genes (TGs). This implies that scMVAE may be useful for imputing expression values for biological interpretation of single cell multiOmics experiments.

A word on Performance Evaluation

Evaluation metrics are crucial to assess the performance of ML models. Accuracy is the ratio of the number of correct predictions to the total number of samples for classification tasks. Sensitivity is the proportion of the number of indices labeled as positive by the model to all positive cases. Specificity measures the ability to classify negative results. The Receiver Operating Characteristic (ROC) curve plots the performance of a classification model at different thresholds using False Positive Rate (1-specificity) and TPR (sensitivity). The area under the ROC curve (AUC) is widely used as a metric to measure performance. Maximizing generalizability of the ML model is the primary goal in an ML model development process. To test accuracy, one ideally needs enough observations to split the data into training and testing sets. Most of the time, there are not enough samples in epigenomics analysis and therefore k-fold cross validation^152,153 is used. Selecting the ‘k’ of k-fold cross-validation associates a well-known ML phenomenon which is the bias-variance trade-off. Although there is no formal rule to choose the ‘k’ and the number of samples affects the choice, choosing 5-fold and 10-fold cross-validation is common practice to balance the bias and the variance^152,153. Unfortunately, there is no consensus in selecting the ‘k’ of k-fold cv in epigenomics analysis. There are papers that use 2-fold^66,154, 3-fold^62,155, 4-fold^154,156,157, 5-fold^91,154, 8-fold^154,158, and 10-fold^159–161 cross-validation to investigate the model’s predictive validity.

Challenges and Future Directions

Despite numerous opportunities with ML frameworks, epigenomics data analysis and machine learning applications in epigenomics face several significant challenges. We will first discuss the most critical challenges and highlight some future research directions. The unfavorable ratio of epigenomic features to sample size makes extracting reliable knowledge challenging. This statistical barrier occurs due to the high-dimensional nature of epigenomic datasets and it relates to the curse of dimensionality. Typically, there are hundreds of thousands of features compared with 10–100 samples in most epigenomics experimental designs. Feature extraction and feature selection methods, discussed under the dimensionality reduction section, help to remove irrelevant and redundant features’ effects before downstream analysis.

Imbalanced data is a common problem, wherein the number of target labels is not the same or even close in many epigenomics studies¹⁸⁹. While the number of methods for epigenomic profiling has exploded in the past few years, they are limited by the availability of the patient material for profiling. This imbalanced class problem further worsens for rare diseases or those where biopsies are difficult to perform (such as GBM, prostate cancers). Another contribution to imbalance of the data is the number of genomic regions with specific epigenetic elements (e.g. enhancers, polycomb elements or TF binding regions) that associate with clinical or molecular phenotype of interest^190,191. An ML model that was trained on an imbalanced dataset may have a bias towards the majority class labels, whereas the actual target is the rare disease or cancer type or condition. Minority over-sampling¹⁹² and boosting¹⁹³ are major solutions to overcome this problem.

Bayesian analysis provides appropriate mathematical modeling to overcome the sample size problem which is common for many epigenomics datasets. Bayesian approaches combine the prior probability (expert’s knowledge on the domain) and the likelihood (observed data) to get the posterior distribution and interpret the results¹⁹⁴. Several genomics analyses showed the Bayesian setting’s superiority over frequentist methods^195–199. Bayesian frameworks are getting more popular in epigenomics analysis^200,201 but there are still opportunities for further enhancements. In addition, generative NN models inspired by Bayesian approaches may provide an opportunity for a synthesis of these techniques.

Maximizing generalizability of the ML model is the primary goal in an ML model development process. To test accuracy, one ideally needs enough observations to split the data into training and testing sets. Most of the time, there are not enough samples in epigenomics analysis and therefore k-fold cross validation^202,203 is used. Selecting the ‘k’ of k-fold cross-validation associates a well-known ML phenomenon which is the bias-variance trade-off. Although there is no formal rule to choose the ‘k’ and the number of samples affects the choice, choosing 5-fold and 10-fold cross-validation is common practice to balance the bias and the variance^202,203. Unfortunately, there is no consensus in selecting the ‘k’ of k-fold cv in epigenomics analysis. Various studies have used 2-fold^78,204, 3-fold^64,205, 4-fold^66,204,206, 5-fold 99,204, 8-fold^204,207, and 10-fold^208–210 cross-validation to investigate the model’s predictive validity. We also want to point out nested cross-validation²¹¹, which is a recent modification of the cv, to machine learning community in epigenomics.

ML methods in epigenomics should be designed to help explain the mechanisms and design new clinical protocols; however, the prevalent practice is to propose accuracy-oriented or efficiency-oriented ML methods. The interpretability and explainability of ML models suffer from the black-box nature of the models²¹². In addition, nonlinear classification boundaries in many complex supervised learning methods hinder the explanation of how epigenomic markers affect diseases, limiting planning for further clinical purposes. Clear reasoning must be established for any ML model before it can be trusted in high stakes clinical settings.

Ensuring reproducibility of any computational analyses is absolutely essential, specially in the genomics and epigenomics field²¹³. The limited ability to replicate the results due to some version changes of the utilized packages may cause consistency and reliability issues for biological findings. Many supervised and unsupervised algorithms depend on several packages, libraries, and modules in epigenomics. However, sometimes reproducibility is not achieved even by using the same data and the same pipeline²¹⁴. Future research must address this critical need with novel ML/DL algorithms. In addition, many genomics and epigenomics data analyses suffer from the exclusion of sample/feature outliers. Outliers decrease the ability to extract real biological signals and mislead many downstream analyses. Mallik and Zhao²⁰⁴ use Density-based Clustering of applications with reducing Noise (DBSCAN) as the outlier detector followed by hierarchical clustering.

Many research studies use Random Forest among many off-the-shelf classifiers for their prediction analysis. There are many ML methods alternative to RF that we may want to use for further epigenomics analysis. For instance, the gradient boosting machine (GBM), a part of the Netflix Prize winning solution²¹⁵, XGBoost²¹⁶ has the highest prediction performance for many Kaggle competitions, and SPORF (Sparse Projection Oblique Randomer Forests) addresses many issues related to the RF²¹⁷.

To gain a comprehensive understanding of the biological roles of epigenetic processes, integration of epigenomic data with genomics, proteomics, metabolomics, pharmacogenomics, and clinical responses is warranted²¹⁸. It is a non-trivial task to harmonize even the same data type generated by different labs or various scientists since the distribution of epigenomic signals varies across labs and batches. The more difficult problem will be integrating several heterogeneous data types²¹⁹. There are some current multi-modal integration approaches with a range of success rates however, they suffer various limitations²²⁰. Another hurdle is data privacy. Ching et al.²²¹ discussed the shortcomings of electronic health records (EHRs) and data privacy issues thoroughly. Patient privacy restrictions may critically affect the power of research studies by limiting the sample sizes²²².

Transfer learning (TL) is an ML technique where a model trained on one task is re-purposed on a second related task. TL is a popular approach in limited data resource computer vision^223–225 and natural language processing tasks^226–228 where the pre-trained model is used as the starting point to improve analysis performance. The TL models have become more popular in genomics^229,230 and epigenomics^231,232 data analysis recently. This area of ML enables researchers to ask new biological questions linking several phenotypes, diseases, or cancer types. TL will need more attention with the integration of several data types over the coming years in epigenomic analysis.

NNs are highly flexible, non-linear “function approximation” algorithms, meaning that they find patterns in high dimensional, highly complex data that allow them to model said data more accurately than the vast majority of competing algorithms. There are NN models for dimensionality reduction, clustering data, learning regulatory correlations between data types, and many other tasks. Due to the flexibility and empirically shown accuracy of NNs, these models have gained widespread use across these tasks. This is especially true in multiOmics analysis, where complex, non-linear models may be necessary to uncover the complex relationships between multiple epigenomic modalities. However, NNs are not without limitations. Compared to simpler models such as PCA, SVM, or UMAP, they can be slower, require more expertise, and be difficult to interpret when digging into both expected and unexpected results. Over time, we expect usage of NNs to increase as improved tools make development and model interpretation easier. This will grant greater access to NN’s results and insights, allowing them to provide biological insight that is not yet accessible to specialized and generalized labs alike.

Overall, machine learning is being increasingly utilized in our understanding of cancer epigenomic datasets to gain novel biological insights. We expect cutting-edge deep learning tools will find various uses in the future as number of epigenomic datasets in cancer systems become available to the community which will enhance our understanding of epigenome contribution to cancer progression.

Table 1.

Overview of the Literature on Machine Learning in Epigenomics Analysis

Cancer Type	Machine Learning Method	ML Task	Epigenomics Data Type	Ref
Liver Hepatocellular Carcinoma, Lung	Support Vector Machine	Supervised Learning	450K Methylation array, ChIP-Seq from ENCODE, Illumina Human Methylation27	65,93,131
Pan-Cancer, Liver Hepatocellular Carcinoma, Lung, Colon, Breast, Prostate, Brain Metastasis	Random Forest	Supervised Learning	ChIP-Seq, 450K Methylation array, Illumina Human Methylation27	64,65,67,77,93,131–135
Pan-Cancer	Logistic regression	Supervised Learning	450K Methylation array, ChIP-Seq from ENCODE, ATAC-Seq	67,136,137
Prostate Cancer	Linear Regression	Supervised Learning	450K Methylation array	63
Pan-Cancer	Decision Tree	Supervised Learning	ATAC-Seq	136
Breast, Lung, Colorectal, Glioblastoma	Hierarchical Clustering	Unsupervised Learning	450K Methylation array, ChIP-Seq, Illumina GoldenGate	107–109,112,113
Glioblastoma, Pan-Cancer, Ovarian	k-means Clustering	Unsupervised Learning	450K Methylation array, Illumina Human Methylation27, Illumina GoldenGate	113,116,138
Pancreatic, Colorectal	NMF	Unsupervised Learning	450K Methylation array, ChIP-Seq	121,122
Breast, Prostate	ChromHMM	Unsupervised Learning	ChIP-Seq, Illumina HiSeq 2500	139–141
Glioma, Pan-Cancer, HeLa, Breast, GM12878, Gastric	Fully Connected Deep Networks	Supervised Learning	450K Methylation Array, ChIP-Seq, DNA sequence, Affymetrix SNP 6.0, CNV, WGBS	142–149
Hepatocellular carcinoma, Breast, K562, Ovarian, Lung, Colon, CRC, GBM	AE	Unsupervised Learning	450K Methylation Array, Affymetrix SNP 6.0 CNV, SNARE-seq, scCAT-seq	150–159
Lung, Esophageal	VAE	Unsupervised Learning	450K Methylation array, ChIP-Seq	160,161
Acute myeloid leukemia, K562, HCT116	CNN	Supervised Learning	WGBS, ChIP-Seq, H3K27ac HiChIP, Dnase	162–166
Breast	LSTM	Supervised Learning	450K Methylation array, ChIP-Seq	167–169
Hepatoblastoma	GRU	Unsupervised Learning	Single cell WGBS, Single cell RRBS	170
Pan-Cancer	GAN	Unsupervised Learning	ATAC-Seq	171

Abbreviations: NMF = Non-negative matrix factorization, NN = Neural Network, DNN = Deep Neural Network, here referring to classic NNs, CNN = Convolutional Neural Network, DBN = Deep Belief Network, GAN = Generative Adversarial Network, AE = Autoencoder, VAE = Variation Autoencoder, LSTM = Long Short Term Memory, GRU = Gate Recurrent Unit, WGBS = Whole Genome Bisulfite Sequencing, RRBS = Reduced Representation Bisulfite Sequencing, GBM = Glioblastoma Multiforme, 450K Methylation array = Illumina Infinium Human Methylation 450k