Lean Big Data Integration in Systems Biology and Systems Pharmacology

Abstract

Datasets from recent large-scale projects can be integrated into one unified puzzle that can provide new insights into how drugs and genetic perturbations applied to human cells are linked to whole organism phenotypes. Data that report how drugs affect the phenotype of human cell-lines, and how drugs induce changes in gene and protein expression in human cell-lines, can be combined with knowledge about human disease, side effects induced by drugs, and mouse phenotypes. Such data integration effort can be achieved through the conversion of data from the various resources into single-node-type networks, gene-set libraries, or multi-partite graphs. This approach can lead us to the identification of more relationships between genes, drugs and phenotypes, as well as benchmark computational and experimental methods. Overall this lean “Big Data” integration strategy will bring us closer toward the goal of realizing personalized medicine.

Bringing together data from open databases and data collected by large-scale team science projects

Biomedical research is moving toward large-scale team-based projects that bring together interdisciplinary teams, using high-end equipment for measuring in high-throughput the genome-wide molecular composition of human cells and tissues. This is coupled with experiments that record cellular and organismal phenotypes, as well as the development of databases and analysis tools that provide easy and open access to the masses of the accumulating data. The collection, organization, and analysis of this Big Data in the fields of systems biology and systems pharmacology are enabled by the Internet and the rapid development of biotechnologies that can be used to measure more variables faster and more accurately.

There are many large-scale projects and popular databases (Table 1 and Fig. 1) that individually already provide many new insights about how human cells work from a system-level perspective with molecular detail. Data is also collected to explain how drugs affect the phenotype of human cells, and how drugs induce changes in gene and protein expression in human cells on a global scale. Human diseases, side effects induced by drugs, and mouse phenotypes are increasingly linked to individual drugs or genes or their combinations (Fig. 1, Table 1). Putting all this information together in an intelligent way and asking the right questions can lead to the identification of more relationships between genes, drugs and phenotypes, as well as explain new functions for genes and drugs with the ultimate goal of improved personalized medicine and significantly increasing human health-span and life-span. These concepts are central to the direction in which systems biology and systems pharmacology are heading. Data integration of molecular genomic data with clinical data is the focus of the Big Data to Knowledge (BD2K) initiative of the National Institutes of Health (NIH) in the USA (1,2). Data abstraction and organization for data integration, together with machine learning strategies that include supervised and unsupervised learning, are fundamental for successful applications in this emerging domain. Big Data thinking also requires a change in attitude; letting go of the obsession with causality and exactitude while embracing correlation analysis and inexactitude. This may seem like a step backward, but it is actually a step forward (3). Furthermore, more fruitful results can come from discovery based approaches that generate hypotheses, rather than ad-hoc hypothesis testing which dominated biomedical research for decades.

Table 1.

Resources that can be processed and integrated for building a global puzzle of systems biology and systems pharmacology.

Resource	Description	Data Structures
Mouse Genome Informatics Mammalian Phenotype Ontology (MGI-MPO) (41,42)	Associations between mouse phenotypes and gene knock-outs	BPG, GSL (phenotype: gene knock-outs), Similarity network (gene-gene)
Online Mendelian Inheritance In Man (OMIM) (43,44)	Associations between heritable diseases and gene sequence aberrations	BPG, GSL (disease: genes), N (gene-gene, disease-disease)
Encyclopedia of DNA Elements (ENCODE) (26,45,46)	Genome map of transcription factor (TF) binding and histone mod. sites	BPG, GSL (TFs: putative target genes), N (TF-TF, gene-gene)
Roadmap Epigenomics (47,48)	Genome map of histone modifications (HMs)	BPG, GSL (HMs: putative target genes), N (HM-HM, gene-gene)
Cancer Target Discovery and Development (CTD2) (49–52)	Fitness of cell lines screened against pharmacologic and genetic perturbations	BPG, DSL (cell line: lethal drugs), GSL (cell line: lethal gene knockdowns), N (drug-drug, cell-cell, gene-gene)
Cancer Cell Line Encyclopedia (CCLE) (53); Cancer Genome Project (CGP) Genomics of Drug Sensitivity in Cancer (8)	Fitness of cell lines screened against pharmacologic perturbations, Profiles of genomic aberrations, Gene expression signatures	BPG (cell line-drug/genomic aberration-diff. expressed genes), DGL (cell line: lethal drugs), GSL (cell line: differentially expressed genes), N (drug-drug, cell-cell, gene-gene)
Genotype-Tissue Expression Project (GTEx) (54)	Genomic aberrations and gene expression profiled across tissues and across the population	BPG (tissue/patients-eQTLs), GSL (tissue/patients-genes), N (patients, eQTL, genes, tissues)
The Cancer Genome Atlas (TCGA) (55–64)	Genomic aberrations, gene expression, DNA methylation, and miRNA expression, proteomics profiled across tumor samples, clinical data	BPG (patients/genomic aberration/genes/clinical outcome), GSL (patients-genes), N (patients, genes, miRNAs)
FDA Adverse Event Reporting System (FAERS) (65–67)	Reports of side effects experienced by patients following drug treatment	BPG (side effects: drugs), DSL (side effects: drugs), N (drug-drug, side-effect-side-effect)
Side Effect Resource (SIDER) (68)	Side effects of drugs reported on package label inserts	BPG (side effects: drugs), DSL (side effects: drugs), N (drug-drug, side-effect-side-effect)
Pharmacogenomics Knowledgebase (PharmGKB-OFFSIDES) (69,70)	Side effects of drugs derived from FAERS reports after bias correction and filtering	BPG (side effects: drugs), DSL (side effects: drugs), N (drug-drug, side-effect-side-effect)
DrugBank (16)	Many drug properties including indications, structure and targets	BPG and N connecting (drug/structures/indications/targets)
ChEMBL (71)	Small molecule biological activities	BPG (chemical structures/targets/data)
The Gene Ontology (72)	Functional annotations for the roles of genes across species	GSL (terms: genes), N (gene-gene, term-term)
WikiPathways (73), BioCarta, KEGG (74,75), Reactome (76)	Collections of curated cell signaling pathways	GSL (pathway: participating molecules), N (genes/proteins)
Biological Repository for Interaction Datasets (BioGRID) (77)	Repository for protein-protein interactions	N (protein-protein), GSL (protein: interacting proteins)
HPRD (78,79), MINT (80,81), IntAct (82,83), STRING (84)	Manually curated literature-derived protein-protein interactions	N (protein-protein), GSL (protein: interacting proteins)
KEA (85), PhosphositePlus (86), Phospho.ELM (87), NetworKIN (88)	Manually curated literature-derived kinase-substrate interactions	Gene-set library (kinase: substrates, kinase: interacting proteins)
NURSA (89), CORUM (90)	Protein complexes from IP-MS	GSL (complex: participating proteins), N (protein-protein)
Gene Expression Omnibus (GEO) (91)	Gene expression signatures including pharmacologic or genetic perturbations in tissues and cells	GSL (perturbation: differentially expressed genes), N (perturbation-perturbation, gene-gene)
Connectivity Map (CMAP) and LINCS L1000 (6,92)	Expression signatures including pharmacologic or genetic perturbations applied to cancer cell lines	GSL (perturbation: differentially expressed genes), N (perturbation-perturbation, gene-gene)

Abbreviations: BPG- bi-partite graphs, GSL- gene-set library, DSL- drug-set library, N- networks.

Fig. 1.

Over 40 large-scale projects, databases and other resources can be integrated in a coherent map to link human and mouse phenotypes to drugs, genes, expression signatures, protein-protein interaction networks and gene-gene functional association networks. This map is helpful for assembling the puzzle pieces needed to significantly enhance our understanding of the effects of drugs on the individual human phenotype at the molecular level. This diagram brings various datasets and resources together for the purpose of identifying non-trivial associations and relationship enabling discovery and assisting in converting data to knowledge.

The diagram in Figure 1 displays various relevant data types, their relationships, and the online open access resources for establishing links between those data types. These resources can be integrated into one coherent puzzle. Starting from left to right, cancer cell lines are made from human tumors and are profiled by methods such as RNA-seq, microarrays, or the newly introduced L1000 technology, for measuring gene expression at the genome-wide scale. At the same time, DNA sequence variations are profiled with whole genome DNA sequencing, exome-seq, CNV and SNP arrays and other methods. In addition, the chromatin status of cells is measured by histone modifications and transcription factor binding profiling with ChIP-seq, DNA methylation arrays, or Hi-C and many other methods. Moreover, methods such as proteomics and phosphoproteomics can measure the cell signaling networks activity levels within those cancer cells. These cancer cell-lines are also interrogated for their sensitivity to drugs and to genetic perturbations such as knock-down or over-expression of single genes. Gene expression measurements after drug or single gene perturbation, when done in high throughput, can produce expression signatures. Gene expression signatures are vectors of differential expression, and these signatures can be related to a biological condition, for example, the progression of a disease, or the classification of disease subtypes (4). Such signatures can be considered comparable high dimensional vectors that point to different directions in N-dimensional space. Such directions can be due to drug or single gene genetic perturbations (Figure 2). A collection of signatures produced in high throughput is supported by the NIH Common Fund project Library of Integrated Network-based Cellular Signatures (LINCS) (https://commonfund.nih.gov/LINCS). The gene expression based high-throughput screening concept (5) was elevated with the construction of the first version of the Connectivity Map. The first version of the Connectivity Map over 6000 gene expression microarray experiments where over one thousand drugs, mostly FDA approved, were applied to four human cell lines in different concentrations and changes in gene expression were measured after six hours (6). The next generation of the Connectivity Map (CMAP) is producing many more signatures using the L1000 technology. Members of the CMAP team at the Broad Institute have already conducted over one million experiments using the L1000 technology. They applied over 20,000 small molecules to stimulate many human cancer cell-lines at different concentrations and where gene expression was measured at different time points. Together with the CMAP team we developed methods to visualize a query this huge dataset (7).

Fig. 2.

Eighty eight drugs were applied at four doses: 0.08μm, 0.4μm, 2μm and 10μm to MCF10A breast cancer cell-line and then gene expression changes were measured at two time points: 6 and 24 hours, resulting in 352 experiments in total. The gene expression values of 978 landmark genes responding to these treatment conditions and control experiments were measured using the L1000 platform. Characteristic direction analysis (33) was applied on the normalized gene expression values to calculate a gene expression signature for each experiment. A gene expression signature is a vector made of the 978-genes occupying 978-dimensional-space whose direction represents in which way the experiment deviates from control and whose norm represents strength of the experiment. The strengths were compared to a null distribution so that each gene expression signature was assigned a z-score to quantify the strength of each perturbation. Only those highly significant experiments whose z-scores are greater than 5 were reserved for visualization. The gene expression vectors of these highly significant experiments were transformed into the first three principal components using PCA analysis and then plotted. The directions of the perturbation vectors fall into four groups (I–IV). Examining the labels of these experiments, we observe a time-dependency: experiments of different time points do not fall into the same group. The drug targets of the visualized experiments are also grouped. Drugs in groups I and IV target kinases within the growth factor activated pathways, whereas drugs in group II and III mainly targeting CDKs.

At the same time, high content drug sensitivity assays that measure phenotypic response of cell lines to drugs, when the drugs are applied in different concentrations, are gradually emerging. These assays can be performed in high-throughput to measure cell phenotypes, for example cell viability, as a function of dose for panels of cancer cell-lines in response to panels of relevant drugs (8–10). Such data provide another type of link that connects drugs and cell-lines. Furthermore, clinical documentation of FDA approved drugs, many of which are also used to profile cancer cell-lines in those high content assays mentioned above, connects drugs to the human phenotype. These links can be indications for the drug, or side effects induced by the drug. Knowledge about side-effects can come from several sources including the two primary open resources: Side Effect Resource (SIDER) (11) and the FDA Adverse Events Reporting System (FAERS) (http://www.fda.gov/CDER/aers). SIDER provides a computational format of drug label inserts containing information about side effects, whereas FAERS provides post-marketing reports of side effects. FAERS contains millions of records from spontaneous reports entered by physicians reporting side effects observed by their anonymized patients for the drugs that the patients took.

Databases such as PharmGKB (12) provide processed data extracted from FAERS and other resources for further analysis (13). Side effects and diseases can overlap, be similar and sometimes even indistinguishable. For example, common severe drug side effects such as neuropathy or long QT syndrome are also genetically inheritable diseases (14). At the same time, side effects and diseases often can be observed in knock-out mouse models. The MGI mouse phenotype ontology is a well-organized dataset containing accumulated knowledge about knockout mouse phenotypes (15). By integrating knockout mouse phenotypes with side effect resources and with drug target information, which can be found at open databases such as DrugBank (16), it is possible to match knockout genes to drug targets and ultimately connect disease phenotypes observed in mice to side effects observed in patients. Such links provide a possible molecular explanation for observed side effects. This analysis can be extended to include human disease genes found in databases such as the Online Mendelian Inheritance in Man (OMIM) (17) for human inheritable diseases, or the Catalogue of Somatic Mutations in Cancer (COSMIC) (18) for disease genes mutated in cancer. Once genes that cause an analogous human disease phenotype in mice have been identified, human gene-gene and protein-protein interactions networks can be used to identify the disease and side-effect connected modules of gene or proteins within these networks. Such modules potentially mediate the observed diseases and phenotypes. However, our current understanding of the human interactome is still partial and suffers from research biases that skew the connectivity distribution in those networks toward the well-studied proteins. Hence, molecular interaction networks extracted from many publications that present results from low-throughput studies should be used with caution when reused and integrated with other data.

One consideration that should be noted is that cells are part of tissues, and different tissues respond differently to drugs or genetic perturbations because they are wired differently at the intra- and inter-cellular levels, containing different network configurations and concentrations of the drug targets in the various cells that constitute the tissue. Efforts such as the Genotype-Tissue Expression (GTEx) project (19) are trying to link gene expression signatures in various tissues to mutations within specific human genes. A related issue is the heterogeneity of tumors, in which several cell types at different stages of tumor development can co-exist within a single tumor.

The ultimate goal is to discover how individual patients fit into this complex network of relationships and how this can be leveraged for their personal therapeutic gain. Efforts such as The Cancer Genome Atlas (TCGA) (http://cancergenome.nih.gov/) collect longitudinal clinical data of patient outcome and drugs used together with gene expression, protein expression and mutation status of tumors collected from those patients at time of diagnosis. Understanding similarities and differences among individuals when it comes to drug response and side effects is the ultimate challenge. Such understanding will lead us to better personalized data driven therapeutic strategies which are gradually becoming a reality.

Data Integration with single node type networks, gene-set libraries and bi-partite graphs

To integrate data from the various relevant resources described in Figure 1, there are three relatively simple and related data structures that can be used: single-node-type networks, gene-set libraries, and bi-partite graphs (Figure 3). Single-node-type networks are the most obvious and commonly applied data structure to abstract and integrate gene-gene and protein-protein interaction networks. Furthermore, drug-drug, disease-disease, patient-patient, side-effect/side-effect, phenotype-phenotype, and cell-line/cell-line association networks can also be created based on correlation or some other measure of similarity based on the attributes of the entities in each network. The edges or connections between entities within those single-node-type networks can be weighted and can be of different types. For example, an edge that connects two transcription factors can be based on similar target genes, physical protein-protein interaction between the factors, correlation observed by their co-expression, or a binding site for one factor found at the promoter of the gene that encodes for the other factor. Each of these types of edges may produce a different network, created from one of these various independent or dependent sources. So far most networks created from such data have unweighted edges, throwing away useful information for the purpose of data abstraction, but keeping the weights can be useful for improving accuracy. Cluster identification algorithms can be used to identify modules within each of those networks. Such module identification procedures fall under the umbrella of unsupervised learning. Modules are typically defined as dense regions within the network connectivity which are loosely connected to other modules. Such module identification process and the modular architecture of the networks under investigation can often better explain known functions or discover new functions. The other common use of single-node-type networks is the ability to use these networks to find connections between two entities. For example, lists of genes or proteins identified experimentally can be connected into a subnetwork with seeding algorithms such as the short path algorithm (20,21) or the mean first passage time method (14).

Fig. 3.

Resources from systems biology and systems pharmacology can be integrated by first identifying the various objects, their relations, and their data types, and then converting the data into single entity weighted networks, fuzzy set libraries, or weighted multi-partite graphs.

To integrate two or more data types, bi-partite or multi-partite graphs, or gene-set libraries can be developed from almost all the data resources listed in Figures 1 and 3, and Table 1. A gene-set library is a data structure that contains a family of sets. Each set is associated with a label that describes what is common about the genes within the set. By definition each set can be of different length, i.e., containing different numbers of genes. These sets are semantically related where the labels cover a common resource belonging to a specific knowledge domain. For example, a gene set library created from the KEGG pathways database has labels that are pathway names and the sets contain the genes within each pathway (22). Although initially such gene-set libraries were developed for the purpose of performing gene-set enrichment analyses (GSEA) (23), they can be used to organize entities such as drugs, diseases, patients, side effects, mouse phenotypes, and cell-lines. For developing the web-based software application Enrichr (24) we compiled 36 gene set libraries from many of the resources mentioned above. For another tool called Network2Canvas (25), we developed six different types of drug-set libraries. These drug-set libraries contain FDA approved drugs instead of genes. Such drug-set libraries can be used to perform drug-set enrichment analysis (DSEA).

As stated above, the initial primary use of gene-set libraries was for GSEA (23). Given a list of genes identified experimentally, GSEA uses gene-set libraries to identify and rank prior knowledge gene-sets that overlap with an input list of experimentally identified genes. Such method is a powerful approach to detect the collective functions of newly identified lists of genes. However, gene-set libraries can be used for other types of analyses. For example, comparing and combining two gene-set libraries can be used to detect unexpected relationships between different sources of data types. For instance, measuring the overlap between all sets from a gene-set library created from differentially expressed genes in tumors from patients and a gene-set library created from the ENCODE ChIP-seq dataset that profiled human transcription factors in cancer cell lines (26) can be used to assign putative transcription factors as potential drivers of gene expression changes in the individual patient tumor (27) (Figure 4). Gene-set libraries so far have been unweighted. This means that the membership level of genes within the sets is the same for all genes across the entire library. However, this can be done differently by considering “fuzzy” gene-set libraries. With fuzzy gene-set libraries, genes within the library can have different membership levels which vary between full-membership to no membership, making gene-set libraries weighted. Such addition of information should make gene set enrichment analysis, and other uses of gene-set libraries, more accurate.

Fig. 4.

Computing gene set overlap between a gene-set library made of up-regulated genes in individual tumors from breast cancer patients from TCGA with a gene set library created by identifying the putative transcription factor target genes from ChIP-seq experiments conducted by ENCODE. Clusters of gene set that significantly overlap (brown spots) are labeled based on the most common transcription factor/s identified within the cluster. To create the breast cancer patients gene set library, Affymetrix microarray gene expression data from 536 breast cancer patients were downloaded from the TCGA. Average expression was computed for genes with multiple probes. For each gene, a Z score was computed by take the average and standard deviation across all patients. Genes that are highly and significantly expressed in each tumor were retained for constructing the gene set library containing 536 patients as the labels and the genes that passed the threshold (p<0.01) as the gene sets associated with each patient. To create the ENCODE gene set library 920 experiments applied to 44 cell lines profiling 160 transcription factors were processed. We retain the target genes that had significant peaks within +2k bp of their transcription starting site (TSS). Since most experiments have replicates, we only kept genes identified in both replicates. The ENCODE gene-set library contains 434 unique experiments. Gene set overlap was computed using the Fisher exact test. The hierarchical clustering plot was created with MATLAB using the Bioinformatics ToolBox.

The third data structure is bi-partite or multi-partite graphs. This data structure fits very well with the data integration goal. Many of the connections in the plot shown in Figure 1 can be made into bi-partite graphs. For example, drugs and their targets (28), drugs and the cell lines that show highest sensitivity to the drugs, drugs and side effects, genes and diseases, drugs and signatures, and more. Such bi-partite networks can then be connected into tri- and other multi-partite graphs, and this enables the identification of non-trivial relationships. For example, drug/cell-line bipartite networks can be combined with cell-line/patient-tumor networks and this can lead to potential assignment of drugs to patients (29) (Figure 5).

Fig. 5.

Tri-partite network that integrates gene expression data from cancer cell-lines and patient tumors with drug response data for cancer cell lines. The network connects groups of patients, cell-lines, and drugs to suggest drugs for patients. Edges between patient groups and cell-lines are colored based on higher (red) or lower (green) expression correlation. Edges between cell-lines and drugs are colored based on higher (magenta/purple) to lower (cyan) drug sensitivity.

The single-node-type network, gene-set library, and bi-partite graph data structures are related. For example, a single-node-type network can be created from a gene-set library or a bi-partite graph. In the past, we showed how a gene-set library can be converted to a network using the Sets2Networks algorithm (30). In addition, bi-partite graphs can be converted to gene-set libraries and vice versa, and gene-set libraries can be transposed. For example, a kinase-substrate bi-partite network can be converted into a kinase-substrate gene-set library in which the set labels are kinases and the genes within the sets are the kinase substrates. A kinase-kinase network can also be created from such kinase-substrate bi-partite graphs by connecting kinases based on their common substrates. If the kinase-substrate gene-set library is transposed, the substrates can be connected based on the kinases they share. The kinase-substrate example can be a bit confusing since kinases can be substrates of other kinases, but this example is valid, and the principle is universal and can be applied to many other types of data.

Data management and integration projects in systems biology and systems pharmacology best practices have embraced the creation of ontologies (31) and their application using standard protocols such as Extensible Markup Language (XML). Such an approach to data integration separates data from metadata, captures the hierarchical structure of the granularity of objects within a domain, as well as removes the requirement for silo-specific data standards and protocols. Although these concepts have been useful and are now becoming standards, there is still room for leaner data organization and integration strategies that further simplify data integration and exchange. One important consideration for data integration, regardless of whether the data is converted to fit ontologies, networks or gene-sets, is the harmonization of identifiers. Dictionaries that map name spaces and convert IDs are important to establish, share and maintain. Another consideration is the quality of data curation. The extraction of computable data records from the multiple sources mentioned above requires knowledge domain expertise combined with knowledge of data structures, statistics and computer programming. The extraction of data from databases, research publication text, medical records, or supplementary materials of publications is not always trivial. The quality of the resulting extracted data is highly dependent on the quality of the human curator.

Benchmarking methods for data gathering and processing by data integration

The inference of biological meaning from data depends crucially on the applied methods of data processing, normalization, curation and analysis. However, identifying the optimal methods to apply can be difficult because an absence of a gold-standard to which each method can be compared. For example, different computational methods exist for performing gene-set enrichment analyses or for identifying differentially expressed genes but evaluating the performance of each is difficult. For example, until recently, there is no clear and unbiased way to tell which genes should be considered as the right differentially expressed genes. With gene-set enrichment analysis methods, each method may result in different rankings of terms from a gene-set library given differentially expressed genes. However, how can we tell which rankings is more correct? In the case of determining the correct differentially expressed genes, given control and treatment condition samples, different statistical methods and cutoffs result in different lists of up and down genes. How can we tell which computational method is detecting the right differentially expressed gene, reflecting the real biological state change of the cells or tissue under investigation?

Frequently, this issue is sidestepped by comparing methods in the setting of synthetic data. Synthetic data is created by a computer program that generates data, based on few simple rules, with the aim of producing a tunable artificial dataset that mimics general properties of real experimental data. However, synthetic data is unsatisfactory because models of biological data do not contain all the rich structure of real biological data. Others have attempted to compare methods of analysis by considering the degree of agreement of each method with the others to be a proxy for the quality of the computational method (32). In our recent publication which describes a novel method to identify differentially expressed genes (33), we compared our new approach, called the Characteristic Direction, to the other currently most popular methods: SAM (34), limma (35), and DESeq (36,37) as well as other less sophisticated methods such as the t-test of fold change. Using real biological data obtained through lean data integration, we were able to benchmark the various methods that prioritize differentially expressed genes by examining gene expression profiles before and after transcription factor perturbations: knockdowns, knock-outs and over-expressions, and compare the gene lists called by the various methods with data from ChIP-seq studies that profiled the binding location of those same transcription factors to the proximity of transcribed genes. More overlap between the independently collected mRNA expression data and the ChIP-seq data is likely an indicator that the method used to call the differentially expressed genes is likely better (Figure 6).

Fig. 6.

Genes were ranked according to their significance of differential expression after transcription factor perturbation and this was scaled such that the most significant gene received a scaled rank of r=0 and the least significant gene has r=1. The scaled ranks of all genes associated with binding sites of the perturbed transcription factor in ENCODE experiments were identified and this process was repeated over 73 experiments in which transcription factor perturbations were followed by gene expression profiling, and the cumulative distribution D(r) was calculated. After subtracting the expected cumulative distribution in the cases of a random uniform distribution, corresponding to the null hypothesis of no enrichment of the genes associated with the perturbed transcription factor from the ENCODE data, we plot the cumulative distributions for each of the five differential expression approaches. In order to indicate the significance of the deviations form a uniform random distribution we also indicate the values scaled by the expected standard deviation ψ. Note that the greater the peak of the resulting curve, the greater priority the method assigns to genes associated with the perturbed transcription factor in independent ENCODE experiments.

We make the reasonable assumption that the genes associated with the DNA binding sites identified in the ChIP-seq experiments of the perturbed transcription factor, are more likely to be differentially expressed after perturbation of that transcription factor, regardless of the cell type and even the mammalian organism. This way we are able to use the collective degree of prioritization of these genes as a measure of the quality of the analysis method. The significance of these comparisons increases by application to large number of transcription factor perturbation experiments. In this way we used the apparent degree of consistency of the gene expression data and ChIP-seq data to quantify the quality of the various analysis methods. This example provides a workable solution to determining the optimal methodology for data processing and analysis. This approach has the potential to significantly improve the resolution of biological insight of this type of data but is also relevant beyond this specific example. For example, this same data and setup can also help benchmarking methods that call target genes from ChIP-seq experiments that profile individual transcription factors to tell us how to best process data from ChIP-seq studies.

Combining multiple sources of evidence to make predictions and improve predictions

The example of benchmarking computational methods described above can be viewed from another perspective: We essentially created a transcription-factor/transcription-factor network from the knockdown followed by expression data, and also created another transcription-factor/transcription-factor network from the ChIP-seq data and then examined the consistency between these two networks. The more consistent these two networks are the assumption is that the better the methods used to create such networks, computationally as well as experimentally. In other words, greater similarity between these networks implies a greater apparent consistency between transcription-factor binding data and gene expression data; this may reasonably be taken as a measure of the combined quality of the experimental procedures and analysis methods. Hence, networks created for connecting the same type of objects: genes, drugs, cell, or phenotypes, using different independent methods can be used to validate one another. This is true regarding both the computational and experimental methods used to collect and process the data. For example, drug/side-effect networks can connect drugs to form a drug-drug network where the edges are shared side-effects. At the same time, a drug-drug chemical structure similarity network can be created to connect drugs to other drugs to form a network. These are two independent sources to create drug-drug networks. If there is some overlap within specific regions of those two networks, then we can identify drug chemical structural elements that can predict side effects. Finally, if a third drug-drug similarity network can be established from another independent resource, the combination of evidence from two networks can be used to improve overlap with the third network. This means that combined evidence from two independent sources can potentially improve predictability of novel relationships between entities connected by a third source. Hence, networks created from independent sources can be used as attributes that when combined, can better predict connections for a third, less determined network. Such predictions can then be validated with more data or empirically. As an example, we show how side-effect prediction for FDA-approved drugs can be significantly improved when combining a drug-drug chemical structure similarity network with a drug-drug similarity network created from gene expression profile similarity when these drugs are applied to human cells and gene expression is measured by the L1000 technology after drug treatment. Programs such as ChemmineR (38) can be used to convert drug SMILES strings to a binary vector that can be used to compute drug-drug similarity but also used as a set of attributes for classification. Drugs chemical structure is known to only provide a weak signal when it is attempted to be correlated with human phenotypes. However, when combined with drug induced expression signatures (LINCS L1000 data), which also on their own have weak relationship with side-effects, the two together can predict side-effect profiles much better (Figure 7). This type of classifier can be useful for predicting side-effect profiles for new drugs. The only thing that is needed is the expression signature of the new drug in human cell lines and the molecular structure of the drug, both can be relatively easily obtained. Similarly, desired phenotypes can be predicted for new drugs using the flip side of the coin of this method.

Fig. 7.

Two drug-drug similarity networks, one based on chemical structure and the other based on gene expression signatures, are used to predict a drug-drug similarity network that is based on shared side-effects. The R library ChemmineR was used to create the drug-drug similarity network based on structure (38). The simplified molecular-input line-entry system (SMILES) strings of 1,409 FDA-approved drugs were converted to a binary string representing 166 Molecular ACCess System (MACCS) structural elements of the drugs. Then, to create a drug-drug similarity network, Jaccard index was used to measure the overlap between shared structural elements for each pair of drugs. Gene expression data from the LINCS L1000 project were obtained from the CPC and CPD batches by selecting the most significant perturbation amongst all dosages, time points, and cell types for each compounds treatment based on the signature strength as defined by the documentation on the lincscloud.org web-site (‘distil_ss’ value). To create a drug-drug similarity network, the Pearson’s correlation coefficient was computed between all pairs of drugs applied to the expression values of the 978 landmark genes. The side-effect drug-drug similarity network was created from SIDER by first creating a gene-set library where the drugs are the terms and the side-effects are the set elements. The Sets2Networks algorithm (30) was used to compute similarity between drugs. The interactions between drugs were sorted and ROC curves were plotted based on matched interaction. To combine the L1000 and MACCS scores, the normalized scores were simply added.

Concluding remarks

Data management in biological and biomedical research in the past three decades has been messy. This is particularly true for low throughput molecular and cell biology studies, of which it was recently suggested that only a small fraction of published results are reproducible (39), and research focus is biased towards the well-studied genes and proteins (40). Low-throughput biology is driven by research biases to study specific molecular components, and where community level quality control remains largely the burden of journal editors and ad-hoc peer-reviewers. As biological and biomedical research moves towards Big-Data, data rich science, and high-throughput methods that can measure the levels of molecular species in cells all at once in a single experiment, we have the opportunity and challenge to fix the messy, biased, non-reproducible problems that have plagued the field for decades. To achieve this goal, we need to develop methods to evaluate new data in the context of existing data and fit new data in the puzzle of cells, assays, phenotypes, drugs, genes, proteins, and signatures. Here we provided only few examples how this can be achieved, but in addition we set the stage for many potential projects that can explore different aspects of the entire puzzle. In addition, the framework discussed provides a template for inclusion of new data that will be gathered in the coming years.

Highlights.

• Many relevant resources from systems biology and pharmacology can be combined into a general map

• Data from the resources can be converted to networks, gene-set libraries, and bi-partite graphs

• Indirect relationships, e.g. matching drugs to patients, can be identified by data integration

• Combining independent sources can be used to validate computational and experimental methods

• Predictions can be improved when two independent networks are used to predict a third network