Navigating transcriptomic connectivity mapping workflows to link chemicals with bioactivities

Abstract

Screening new compounds for potential bioactivities against cellular targets is vital for drug discovery and chemical safety. Transcriptomics offers an efficient approach for assessing global gene expression changes but interpreting chemical mechanisms from these data is often challenging. Connectivity mapping is a potential data-driven avenue for linking chemicals to mechanisms based on the observation that many biological processes are associated with unique gene expression signatures (gene signatures). However, mining the effects of a chemical on gene signatures for biological mechanisms is challenging because transcriptomic data contain thousands of noisy genes. New connectivity mapping approaches seeking to distinguish signal from noise continue to be developed, spurred by the promise of discovering chemical mechanisms, new drugs, and disease targets from burgeoning transcriptomic data. Here, we analyze these approaches in terms of diverse transcriptomic technologies, public databases, gene signatures, pattern-matching algorithms, and statistical evaluation criteria. To navigate the complexity of connectivity mapping, we propose a harmonized scheme to coherently organize and compare published workflows. We first standardize concepts underlying transcriptomic profiles and gene signatures based on various transcriptomic technologies such as microarrays, RNA-Seq, and L1000 and discuss the widely used data sources such as Gene Expression Omnibus (GEO), ArrayExpress, and MSigDB. Next, we generalize connectivity mapping as a pattern-matching task for finding similarity between a query (e.g., transcriptomic profile for new chemical) and a reference (e.g., gene signature of known target). Published pattern-matching approaches fall into two main categories: vector-based use metrics like correlation, Jaccard index, etc., and aggregation-based use parametric and non-parametric statistics (e.g., gene set enrichment analysis). The statistical methods for evaluating the performance of different approaches are described, along with comparisons reported in the literature on benchmark transcriptomic data sets. Lastly, we review connectivity mapping applications in toxicology, and offer guidance on evaluating chemical-induced toxicity with concentration-response transcriptomic data. In addition to serving as a high-level guide and tutorial for understanding and implementing connectivity mapping workflows, we hope this review will stimulate new algorithms for evaluating chemical safety and drug discovery using transcriptomic data.

Graphical Abstract

Introduction

Drug discovery and chemical safety require effective tools for screening new compounds for potential bioactivities against cellular targets. Transcriptomics is one of the widely used techniques for assessing the biological effects of chemicals through their impact on global gene expression.¹ Because chemicals induce gene expression changes by interacting directly via receptor binding² or indirectly by disrupting cellular homeostasis,³ inferring their targets from transcriptomic data is challenging. Connectivity mapping addresses this issue by measuring the similarity between transcriptomic profiles and gene signatures related to cellular targets using the “universal language” of genes.^4,5 It assumes that transcriptomic profiles fingerprint biological samples, and similarity between profiles implies a common mechanism. Transcriptomics evolved rapidly in the last two decades after sequencing the human genome.⁶ Beginning with cDNA spotted arrays,⁷ followed by high-density oligonucleotide arrays,⁸ and most recently RNA sequencing technology,⁹ transcriptomics has become more reproducible, reliable, and cost-effective.¹⁰ As a result, millions of transcriptomic profiles are now available in public domain repositories^11,12 for thousands of conditions.^4,13 Innovative tools are needed to uncover new relationships between chemicals, pathways, and diseases using this wealth of transcriptomic data. Connectivity mapping⁴ is an example of such a tool that can facilitate drug discovery,¹⁴ help repurpose existing drugs,¹⁵ and produce safer chemicals.¹⁶

Connectivity mapping with transcriptomic data is one of many techniques in a rich landscape of computational methods for inferring the putative interactions between chemicals and biological targets or pathways. This landscape can be broadly divided into approaches based on binding, similarity, and machine learning (ML). Binding-based methods attempt to model physico-chemical interactions between a chemical and a protein target with three-dimensional structure data using molecular dynamics^17,18 or, more recently, using ML.¹⁹ There have been impressive advances predicting new ligands for specific protein targets,²⁰ and with predicted three-dimensional structures for all known proteins,²¹ virtually screening all chemicals against thousands of protein targets could be within reach.²²

Connectivity mapping is conceptually related to other similarity-based approaches, which attempt to infer the properties of a new chemical using pair-wise similarity with chemicals of known properties, including physico-chemical properties or biological activities. If two chemicals have significant structural similarities, then similarity-based approaches assume they also have similar properties. Similarity-based approaches have two essential ingredients: a vector of attributes and a measure of similarity based on the attributes. Similarity-based pattern-matching techniques are also considered instance-based learning methods²³ in ML, which includes approaches like k-nearest neighbor (KNN) classification. Chemical similarity-based approaches use molecular structure descriptors (such as extended connectivity fingerprints²⁴) to represent chemicals and measure similarity using set operations (for a review of similarity measures, see Bero et al.²⁵). For example, a query chemical can be searched against a database to find other structurally similar chemicals from which the unknown biological role can be inferred. Chemical structure-based similarity is widely used to infer molecular targets.²⁶ One of the problems with using chemical similarity-based techniques is that minor alterations in structure can lead to drastic changes in their affinity for the same target, which are known as “activity cliffs” in structure-activity relationship (SAR) research.²⁷ Another issue is that new structural categories of chemicals can be discovered or synthesized that have no existing analogues. If they bear insufficient resemblance to known chemicals, it is not possible to infer their properties based on structural similarity alone. Despite these limitations, structure-based automated prediction approaches²⁸ are routinely used to fill data gaps for untested chemicals based on the known properties of analogues in the same local domains. More recently, structural and bioactivity similarity between chemicals has been used to infer the toxicity of untested chemicals.^29–32

Finding pair-wise similarities using biological and chemical descriptors is a practical strategy for inferring the properties of untested chemicals; however, if hundreds of chemicals are associated with different classes of biological activities (e.g., protein target, pathway activation, toxicity, etc.), then ML can be more effective. ML algorithms systematically mine patterns in data (i.e., vector representations of data derived from biological and chemical descriptors) to build accurate predictive models of various biological activities.³³ For example, ML algorithms mine chemical structure representations to build models, referred to as quantitative structure-activity relations (QSARs).³⁴ QSAR models have been used to classify potential nuclear receptor activators,^35,36 cellular stress responses,^37,38 and toxicities.^39,40 Similarly, ML algorithms mine transcriptomic data on chemicals (derived from different cellular contexts) to build models of biological mechanisms,^41–44 and toxicities.^45,46 Models derived by ML can predict the bioactivity or toxicity of new or untested chemicals using vector representations of data (i.e. attribute-value vectors that are used to train the model). Different ML methods have varying requirements for training data to produce reliable predictive models. Whereas similarity-based approaches such as KNN may only require a few examples because of their simplicity, more complex ML algorithms need varying amounts of training data to tune model parameters reliably. Furthermore, for in vivo toxicity prediction, it is also essential to consider the chemical dose, duration, and route of exposure. A systematic comparison of similarity-based and other ML algorithms is beyond the scope of this review.

Connectivity mapping may be considered an automated biological read-across^47,48 technique to infer properties of untested substances using transcriptomic profiles in place of chemical structure representations. Gene-based descriptors in transcriptomic profiles measure the expression of specific genes in the genome, just like structure descriptors capture the presence of substructural moieties in chemicals. Transcriptomic profiles, however, can capture the biological response to chemical treatments, genetic perturbations, or pathological conditions using continuous expression levels of genes in ways that chemical structure descriptors cannot. The ability of transcriptomics to capture a diverse array of physiological states also makes it a powerful tool for finding similarity-based connections. This review is a guide for navigating connectivity mapping in terms of the diverse array of technologies to generate transcriptomic profiles, define biological states using gene-based descriptors, and organize the plethora of algorithms to measure transcriptomic similarity.

Historical Background

Connectivity mapping originates from functional discovery studies,⁴⁹ which aimed to interpret the molecular phenotypes of biological samples using transcriptomics.^50,51 A pivotal study by Hughes et al. produced one of the earliest and largest compendia of transcriptomic profiles for 300 genetic and chemical perturbations in yeast.⁵² The authors used similarity between transcriptomic profiles to cluster known mutants, uncharacterized mutants, and pharmacologic agents. For example, deleting YER044c, an uncharacterized yeast open reading frame (ORF), produced transcriptional profiles similar to the sterol isomerase (ERG2) deletion mutant. Further experiments determined that the YER044c ORF encoded the endoplasmic reticulum protein (ERG28). Because ERG2 and ERG28 are both involved in ergosterol biosynthesis, their deletion mutants produced similar transcriptomic profiles. Hughes et al. also showed transcriptomic responses to the drug fenpropimorph were similar to the responses due to ERG2 deletion mutants. This is not surprising as fenpropimorph is a fungicide that disrupts eukaryotic sterol biosynthesis pathways. Surprisingly, fenpropimorph was also a potent mammalian antagonist of sigma-1 receptor (SIGMAR1), which is involved in neuromodulatory pathways involved in pain. SIGMAR1 antagonists are being explored as a novel class of analgesic agents for treating pain.⁵³ There is growing evidence that ERG2 disruptors in yeast are SIGMAR1 antagonists,⁵⁴ and such pharmacological agents can be identified by connectivity mapping. The ability to link chemicals to mechanisms within and across species showed the value of transcriptomics as a “universal phenotype” for fingerprinting global biological states and of transcriptomic similarity to uncover novel relationships between chemicals and their targets.

Before connectivity mapping approaches, transcriptomics mainly identified differentially expressed genes between cases and controls using p-value and fold-change thresholds. Lists of differentially expressed genes helped identify statistically over-represented pathways (e.g., using Fisher’s Exact Test⁵⁵) and provided insight into putative biological mechanisms (see Khatri and Draghici⁵⁶, and Rivals et al.⁵⁷)). However, because gene lists are sensitive to the choice of differential expression thresholds, using varying statistical cut-offs can produce inconsistent biological interpretations. Mootha et al. showed over-representation analysis of gene lists ignored the subtle yet coordinated regulation of gene sets relevant to a pathway. They found a gene set for the oxidative phosphorylation pathway “enriched” in diabetic versus healthy muscle tissues even though individual genes in the set were not significantly differentially expressed.^58,59 Mootha et al.⁵⁸ and Subramanian et al.⁵⁹ called this approach gene set enrichment analysis (GSEA). Other gene set analysis (GSA) approaches subsequently used for pathway, and function enrichment^60–62 have been reviewed extensively elsewhere.^56,57

The connectivity map (CMap) project, which gave rise to the eponymous “connectivity mapping” approach, was the first publicly available large-scale compendium of transcriptomic profiles generated by treating human cells with a library of small molecules.⁵ Connectivity mapping used this compendium of 564 unique transcriptomic profiles for 164 chemicals (Build 01 of the CMap database, which we refer to as CMap v1). Connectivity, or similarity, was measured using a modified version of GSEA for analyzing “gene signatures” derived from highly up- and down-regulated genes in the transcriptomic profiles (see Figure 1). For example, Lamb et al.⁵ searched a signature of histone deacetylase (HDAC) inhibitors against the CMap v1 reference database. The HDAC inhibitor signature was derived from an independent study of HDAC inhibitors in bladder and breast cancer cells,⁶³ which comprised eight up-regulated and five down-regulated genes (illustrated in Figure 1(a)). Searching the entire CMap v1 database (illustrated in Figure 1(e)) with this HDAC signature using GSEA (illustrated in Figure 1(c)), identified the most robust connections with vorinostat and trichostatin A (an example of such a match is shown in Figure (1(f)), both HDAC inhibitors. The ability of GSEA to link signatures of HDAC inhibitors from disparate experiments provided compelling evidence for the utility of connectivity mapping approaches.

Figure 1.

Overview of connectivity mapping as a pattern matching between (a) query and (b) reference using a (c) similarity measure illustrating different types of matches (f, g, and h). (a) The query is a directional gene signature ($DS=\{S^{+},S^{-}\}$) signified by a set of up-regulated genes ($S^{+}$ shown as red circles) and a set of down-regulated genes ($S^{-}$ shown as blue circles). (b) The reference is a transcriptomic profile $\mathit{x}$ shown as a vector of log2 transformed fold-change (L2FC) values for each gene (blue and red colors represent down- and up-regulation, respectively). (c) The similarity measure ($SM$) for scoring the match between $DS$ and $\mathit{x}$. (d) A collection of predefined signatures representing sets of genes (e.g., involved in pathways). (e) A collection of transcriptomic profiles for a set of perturbagens. (f) “Positive connection” between $DS$ and $\mathit{x}$ when $S^{+}$ and $S^{-}$ are correlated with up- and down-regulated genes in $\mathit{x}$. A positive connection is a match found when $SM\left(DS,\mathit{x}\right)>0$. (g) “No connection” between $DS$ and $\mathit{x}$ when $S^{+}$ and $S^{-}$ are uncorrelated with up- and down-regulated genes in $\mathit{x}$ (where $SM\left(DS,\mathit{x}\right)\approx 0)$. (h) “Negative connection” between $DS$ and $\mathit{x}$ when $S^{+}$ and $S^{-}$ are anti-correlated with up- and down-regulated genes in $\mathit{x}$. A negative connection is a match found when $SM\left(DS,\mathit{x}\right)<0$.

In the second example, López et al. searched a gene signature of diet-induced obesity in rats.⁶⁴ Using GSEA, they found a strong match between this signature and transcriptomic profiles for troglitazone, rosiglitazone, and indomethacin, all peroxisome proliferator-activated receptor gamma (PPARG) agonists. However, the directions of gene expression in the diet-induced obesity signature (i.e., up- and down-regulated genes) were found to be opposite to the directions of the genes in the profile for PPARG agonists. Such matches are referred to as “negative connections” as they have negative GSEA scores (see Figure 1(i) for a visual example of a negative connection). Interestingly, PPARG agonists are prescribed as hypolipidemic agents for the treatment of diabetes but can produce weight gain and liver injury as unwanted side effects. Thus, connectivity mapping revealed that the biological state of diet-induced obesity is “negatively connected” with PPARG-mediated hypolipidemic activity, notwithstanding differences in cells, treatment conditions, and gene expression assaying technologies. Finding negative connections between disease gene signatures and transcriptomic profiles of approved drugs forms the basis of some drug-repurposing approaches.¹⁵ These findings further demonstrated the utility of transcriptomic connectivity mapping for linking disease phenotypes with putative chemical treatments based on gene signatures. The initial success of connectivity mapping led to an expansion of the CMap (Build 02 of the CMap database, which we refer to as v2) to cover 1,309 chemicals and 6,100 transcriptomic profiles.⁶⁵

Connectivity mapping and toxicology

A key challenge in toxicology is evaluating the safety of chemicals by determining their potency and potential for activating molecular targets that can lead to adverse health outcomes.⁶⁶ In computational toxicology, transcriptomic profiling is used to rapidly screen thousands of untested chemicals to identify their putative targets, mechanism of action, or other effects.^10,67–69 This is because high-throughput transcriptomic profiling using mRNA sequencing (RNA-Seq),⁹ and more recently targeted RNA-Seq,⁷⁰ are extremely promising and cost-effective approaches for generating transcriptomic profiles for tens of thousands of chemical treatments. Whether evaluating new chemical entities for drug discovery or untested environmental chemicals for public health protection, transcriptomic connectivity mapping is a robust and high-throughput alternative to the existing techniques.¹⁶ Therefore, it is essential to examine the landscape of connectivity mapping approaches, understand their operation transparently, and assess their utility for specific toxicology applications.

Harmonizing connectivity mapping approaches

Dozens of refinements or alternatives to connectivity mapping have been proposed and are reviewed elsewhere.^71,72 In this review, we develop a coherent view of various connectivity mapping approaches with an emphasis on three main ingredients: a transcriptomic profile produced by a perturbagen, a gene signature associated with a biological state, and an approach for matching the profile with the signature. The connectivity mapping workflow can be generalized as a database search and retrieval operation (see Figure 1) in which a “query” object (Figure (1a)) is compared with an extensive collection of “reference” (Figure 1(b)) objects (from a reference database (Figure 1(e))) using a pattern matching algorithm (Figure 1(c)) to find the most similar “hits” (Figure 1(f) and (h))). We employ the generic term “object” to cover several kinds of gene set-based inputs (summarized visually in Figure 2) for pattern matching. The variation between the connectivity mapping approaches is explained by the differences in the choice of the query, the reference database, and the pattern-matching algorithm. For example, Mootha et al.⁵⁸ used a transcriptomic profile (derived from diabetic versus healthy muscle tissue) (Figure 2(a)) as the query, pathway-based gene sets (Figure 2 (e) and (d)) for the reference database, and GSEA for pattern-matching. On the other hand, Lamb et al.⁴ used gene signatures of HDAC inhibitors as the query, transcriptomic profiles as the reference database, and a modified version of GSEA. Although the workflow used by Mootha et al. is generally referred to as “pathway enrichment,” using the harmonized scheme presented here, we discuss how “enrichment” and “connectivity mapping” may be considered different types of similarity measures for comparing gene set objects comprised of gene signatures and transcriptomic profiles.

Figure 2.

Representing transcriptomic profiles and gene signatures. (a) A transcriptomic profile $\mathit{x}$ shown as a vector of log2 transformed fold-change (L2FC) values for each gene (blue and red colors represent down- and up-regulation, respectively). (b) An extreme transcriptomic profile ${\mathit{x}}^n$ is defined by selecting the $n$ most up- and down-regulated genes in $\mathit{x}$ (shown as red and blue squares, respectively). (c) A directional signature ($D{S}^n$) is defined by transforming all up- and down-regulated genes in ${\mathit{x}}^n$ to 1 and −1, respectively. The directional signature ($D{S}^n=\{S^{+},S^{-}\}$), is signified by a set of up-regulated genes ($S^{+}$ shown as red circles) and a set of down-regulated genes ($S^{-}$ shown as blue circles). (d) A non-directional signature ($S^n$) is derived from $D{S}^n$ by ignoring the direction of expression changes for all genes (all genes are shown as black circles). (e) A pathway containing a collection of proteins can be represented by a set of genes (which encode the proteins) and defined as a non-directional signature ($S$). (f) A causal network comprised of interacting proteins can be represented simply by a collection of genes (which can be represented as $S$. (g) A transcriptomic database is a collection of $\mathit{x}$. (h) A bioactivity signature database is a collection formed by one or more of the following types of signatures: ${\mathit{x}}^n,DS,D{S}^n,S$, and $S^n$.

For example, the query object used by Lamb et al.⁵ was a gene signature derived from transcriptomic profiles of HDAC inhibitors in bladder and breast cancer cells.⁶³ This gene signature for “HDAC inhibition” was defined by a set of up- and down-regulated genes (visualized in Figure 2(c)). In contrast, the reference objects were transcriptomic profiles (HYPERLINK Figure 2(a)) in the CMap v2 database. The similarity between the query gene signature for HDAC inhibition and each CMap v2 reference transcriptomic profile was measured using the same scoring metric as GSEA. Top-scoring matches with vorinostat and trichostatin A are both well-known HDAC inhibitors. In other words, GSEA “connected” the biological state of the query object, represented by a gene signature, with HDAC inhibition. This approach for connectivity mapping can be used in toxicity testing for new chemicals by generating gene signatures using transcriptomics, matching signatures with transcriptomic profiles for previously tested chemicals, and inferring putative connections with known chemicals’ mechanisms. Though connectivity mapping approaches measure the similarity between gene sets and transcriptomic profiles, there are subtle differences between them. First, Mootha et al. used a transcriptomic profile as the query object, whereas Lamb et al. used a gene signature. Second, Mootha et al. used pathway gene signatures, whereas Lamb et al. used transcriptomic profiles to define the reference database. Third, both approaches used slightly different similarity measures because they were comparing different types of objects. We believe that a harmonized scheme for encompassing the diverse array of published transcriptomic connectivity mapping approaches can be developed by formalizing the definition of query objects, reference objects, and similarity scoring measures.

Review outline

Although connectivity mapping can elucidate the mechanisms of action or toxicity of chemicals, the relative advantages of different approaches have not been discussed before. This review analyzes the gene set-based connectivity analysis pipeline in terms of reference database construction, gene signature generation, and similarity scoring measures. First, we provide a standardized terminology to describe the critical elements of gene set-based approaches to compare and highlight their unique contributions. Second, we introduce several transcriptomic and other data sources and outline the development of reference databases. Third, we discuss some of the main approaches for generating transcriptomic signatures. Fourth, we propose a detailed classification of connectivity scoring measures, including GSEA and other variants. Fifth, we describe approaches for evaluating confidence in results from connectivity mapping. Sixth, we consider different approaches for comparing the performance of different connectivity mapping algorithms. Finally, we discuss some of the opportunities and challenges of applying connectivity mapping approaches to toxicology. Our objective is to provide a conceptual overview of the entire connectivity mapping process at multiple levels, including high-level visual summaries, a formal terminology for comparing all connectivity mapping approaches, and detailed explanations of algorithms.

Key concepts and terminology

To compare the diversity of gene set-based connectivity mapping approaches, we define the key concepts and introduce terminology used in the remainder of this review. We begin with a description of different types of technologies routinely used for transcriptomic data generation. Next, we define a transcriptomic profile as the global differential gene expression data and discuss how gene signatures are created from transcriptomic profiles. Then we describe the different types of gene signatures that encompass transcriptomic profiles and gene sets that define canonical pathways. Lastly, we introduce key concepts about the relationships between gene signatures and transcriptomic profiles, and then use them to formalize similarity scoring measures. A visual overview of the different terms and their definitions are provided in Figure 1 and Table 1, respectively.

Table 1.

Glossary of transcriptomic terminology presented in this review.

Transcriptomics	Transcriptomics is defined as the measurement of large-scale (“global”) gene ($g$) expression in a biological sample including all genes or a representative subset of genes in the genome of a species (denoted as, $\mathit{G}=\left\{g_1,g_2,g_3,\dots ,g_j,\dots \right\}$). Each gene is transcribed into one or more messenger RNA (mRNA) molecules.
Transcriptomic technologies	Transcriptomic technologies use different approaches to measure global gene expression by quantifying individual mRNA molecules. Most technologies synthesize complementary DNA (cDNA) from mRNA and use complementary oligonucleotide probes to specifically detect cDNA by hybridization. Examples of transcriptomic technologies include: microarrays, L1000 and RNA-Seq.
Transcriptomic data	Transcriptomic data (or gene expression data) from different technologies is generated from biological samples under different experimental conditions including normal vs. diseased, control vs. treated, etc. Two frequent types of treatments are chemical and genetic perturbations involving the knock-out or over-expression of specific genes. Transcriptomic data can be represented by at least four main levels where the higher levels of data are derived from the lower levels: raw data specific to the assay technology (L0), unnormalized mRNA data derived from L0 data using assay-specific processing (L1), normalized mRNA data that captures the absolute levels of expression for genes and is comparable across the study (L2), differential expression data that captures the change in mRNA levels from the control (and may have associated statistical significance scores) that is comparable across studies (L3).
Transcriptomic profile	We define the transcriptomic profile ($\mathit{x}$) as the L3 transcriptomic data that captures differential expression values, e.g. log2 fold-changes (L2FC), Z-scores, p-values, q-values, etc. $\mathit{x}=[x_1,x_2,\dots ,x_i,\dots ,x_N]$ where $x_i$ is the differential expression of one gene $g_i$, there are $N$ genes in the profile and $g_i\in \mathit{G}$. If available, the significance scores associated with differential expression values for genes can also be represented as a vector denoted as $\mathit{p}$, where $\mathit{p}=[p_1,p_2,p_3,\dots p_j,\dots p_N]$, $p_i$ is the p-value associated with the change in expression $x_i$ for gene $g_i$.
Extreme transcriptomic profile	We define the extreme transcriptomic profile (${\mathit{x}}^n$) as one that contains the most differentially expressed genes in $\mathit{x}$. The $n$ most up-regulated genes ($n^{+}$) and most down-regulated genes ($n^{-}$) are denoted as $S^{+}$ and $S^{-}$, respectively. Then ${\mathit{x}}^n=\{{\mathit{x}}_i|i\in \left\{S^{n-}\cup S^{n+}\right\}\}$.
Directional gene signature	A directional gene signature ($D{S}^n$) can be a list of the most differentially expressed genes in $\mathit{x}$ (in other words, the genes in ${\mathit{x}}^n$} associated with a biological state. Therefore, it is defined as: $D{S}^n=\left\{S^{n-}\cup S^{n+}\right\}$. Directional gene signatures can be defined based on other approaches and may not contain an equal set of up- and down-regulated genes.
Gene signature	A gene signature ($S$) is a list of genes whose activity defines a biological state. $S=\left\{g_{100},g_2,g_{32},\dots ,g_i,\dots \right\};g_i\in \mathit{G}$. Therefore, $S$ can be defined as a list of genes/proteins in a canonical pathway or by the genes in ${\mathit{x}}^n$ or $D{S}^n$.
Gene set object	We define a gene set object ($O$) as the most general concept for a transcriptomic profile ($\mathit{x}$), an extreme transcriptomic profile (${\mathit{x}}^n$), a directional gene signature ($D{S}^n$) or a gene signature ($S$) i.e. $O=\{\mathit{x},{{\mathit{x}}^n,DS}^n,S\}$. Although we have introduced $\left\{{DS}^n,S\right\}$ as lists and $\mathit{x}$ as vectors all can be represented as vectors. For example, a hypothetical non-directional pathway signature $S_A=\left\{g_1,g_2,g_5,\dots ,g_i,\dots ,g_n\right\}$ only contains a subset of the genes in $\mathit{G}$ and can be represented as a binary vector ${\mathit{x}}_A=[\mathrm{1,1},\mathrm{0,0},1,\dots ,1,..]$ (that is, $\mathit{x}\left[i\right]=1ifi\in S_A$). Similarly, a hypothetical directional signature ${DS}_B^n={\{S}_B^{+}:\left\{g_3,g_4,g_8,\dots \right\}\cup S_B^{-}=\left\{g_2,g_5,g_7,\dots \right\}\}$, can be represented as the vector ${\mathit{x}}_B=[0,-\mathrm{1,1},1,-\mathrm{1,0},-\mathrm{1,1},\dots ]$ (that is, $\mathit{x}\left[i\right]=1ifi\in S_B^{+};\mathit{x}\left[i\right]=-1ifi\in S_B^{-}$).

Transcriptomic technologies and data

Transcriptomics measures global gene expression in a biological specimen, and the term was coined for RNA-Seq technology.⁷³ Here we use transcriptomics to refer to any high-throughput gene expression technology including, but not limited to, Affymetrix microarrays (used for building the CMap databases), the L1000 platform (using in the Library of Integrated Network-based Cellular Signatures (LINCS) database), and RNA-Seq technology. Transcriptomic data from different technologies are generally represented at four different levels: (L0) level 0 raw data specific to the assay technology, (L1) level 1 unnormalized mRNA data derived from raw data using assay-specific processes, (L2) level 2 normalized mRNA data, and (L3) level 3 differential gene expression data obtained by analyzing mRNA data between cases and controls. Each technology has varying needs for RNA purification but may or may not require complementary DNA (cDNA) synthesis. Affymetrix high-density microarrays hybridize each mRNA in a sample with thousands of oligonucleotide probes. Each mRNA sequence is mapped to a set of probes (called a probeset) designed to optimize the sensitivity and specificity of measurements. Affymetrix microarray L0 data are images, called cell intensity files (CEL), which are processes using image analysis tools to produce L1 data. L1 data is normalized⁷⁴ using one of the available approaches^75,76 to estimate L2 data as logarithm (base 2) intensity value for each transcript. LINCS data are generated using the L1000 platform.⁷⁷ The LINCS platform employs flow cytometry to measure the relative abundance of mRNAs that hybridize to Luminex beads tagged with fluorescent oligonucleotide probes (producing Luminex bead array data as L0 files). The L0 flow cytometry data are deconvoluted to obtain L1 data, which are quantile-normalized to get L2 data on landmark genes (978) and imputed transcripts (12,336). RNA-Seq uses high-throughput DNA-sequencing technology to directly read the cDNA in samples producing mRNA sequence fragments, known as “reads,” as raw FASTQ files (L0). Each read in L0 is aligned with the known sequences of gene products or the entire genome to generate L1 data (for a review of best practices, see Conesa et al.⁷⁸). RNA-Seq technology continues to evolve rapidly, and several modifications have been proposed to avoid cDNA synthesis (by using RNA directly) and to target specific genes (instead of sequencing all genes), including NPSeq,⁷⁹ RASL-Seq,⁸⁰ DRUG-Seq,⁸¹ and TempO-Seq.⁷⁰ Such targeted RNA-Seq approaches can be cost-effective as they use oligonucleotide templates (also called probes) derived from specific regions in individual genes to measure their expression to produce L0 and L1 data. Validation studies have demonstrated the concordance between L1000 and microarrays,⁸² and targeted RNA-Seq, RNA-Seq, and microarrays⁸³ for evaluating reference chemicals. The choice of transcriptomic platform comes down to reproducibility and cost. Therefore, new transcriptomic technologies that promise to lower the cost and efficiency of transcriptomic data generation continue to be developed (for example, see NanoString⁸⁴).

Differential gene expression analysis

The L2 normalized mRNA data for each gene in the “cases” are compared with “controls” to calculate each gene’s differential expression. There are many statistical approaches for estimating confidence in each gene’s differential expression based on distributional intensity and count data assumptions.^85–87 Batch-correction approaches can also improve differential expression estimates.^88–90 The differential expression values for genes are reported as the ratio (or difference for log-transformed L2 data) between the cases (e.g., chemical treatments or diseased subjects) and the controls (e.g., untreated samples or normal subjects). Differential gene expression is generally reported in log2 fold-change (L2FC) units. When batch effects are significant, Z-scores offer another approach to estimating treatment effects. They can be averaged over batches and reported as moderated Z-scores.⁷⁷ L2FC values or Z-scores have the following interpretation: positive/negative values mean that a gene is up-/down-regulated in a case versus control. The set of differential expression values for all genes defines the L3 transcriptomics data.

Transcriptomic profile

We define L3 data as the differential transcriptomic profile ($\mathit{x}$) in which up- and down-regulation of all genes are quantified (e.g., L2FC values or Z-scores) (Figure 2(a)). Transcriptomic technologies (e.g., Affymetrix, L1000, and RNA-Seq) measure mRNA in samples using different probes or sets of reads mapped to a unique set of genes (denoted as, $g$). Unique gene names, standardized using Entrez gene identifiers⁹¹ and Human Gene Nomenclature Committee (HGNC) codes⁹² (or other species-specific genome databases), enable a comparison of L3 transcriptomic data from diverse technologies. Each transcriptomic technology attempts to capture a broad subset of the entire list of genes for each species’ genome (denoted as, $G=\left\{g_1,g_2,g_3\dots ,g_{j,\dots }\right\})$ For the remainder of this review we ignore the differences in coverage of the genome and assume that the transcriptomic profile from any assay technology can be represented generally as a vector denoted as $\mathit{x}$, where $\mathit{x}=[x_1,x_2,\dots ,x_i,\dots ,x_N]$, $x_i$ is the differential expression of one gene $g_i,N$ is the number of genes in the profile and $g_i\in \mathit{G}$. Similarly, the significance scores associated with differential expression values for genes can also be represented as a vector denoted as $\mathit{p}$, where $\mathit{p}=[p_1,p_2,p_3,\dots p_j,\dots p_N]$, $p_i$ is the p-value associated with change in expression $x_i$ for gene $g_i$.

Extreme transcriptomic profile

Highly differentially expressed genes may be more informative than less differentially expressed ones. The extreme transcriptomic profile (${\mathit{x}}^n$) has been proposed as one possible approach for summarizing the most highly differentially expressed genes in $\mathit{x}$. Cheng et al. refer to this subspace of $\mathit{x}$ as the “eXtreme” transcriptomic profile, where extreme connotes the selection of the most up- and down-regulated genes.⁷² First, $\mathit{x}$ (Figure 2(a)) is sorted in order of differential expression values, which places the genes with the greatest up-/down-regulation at the extremes. If the $n^{+}$ and $n^{-}$ most up- and down-regulated genes in $\mathit{x}$ are $S^{+}$ and $S^{-}$, respectively then the extreme transcriptomic profile ${\mathit{x}}^n=\{{\mathit{x}}_i|i\in \left\{S^{n-}\cup S^{n+}\right\}\}$ (Figure 2(b). $S^{+}$ and $S^{-}$ may be further restricted to genes in $\mathit{x}$ that meet a p-value and/or fold-change threshold (Figure 2(b)). For example, one can define a 2-fold threshold for the magnitude of differential expression to identify $S^{+}$ (i.e. log2(FC)>1) and a 2-fold decrease (i.e. log2(FC)<−1) to identify $S^{-}$. In other words, an extreme transcriptomic profile can also be defined by selecting the top up- or down-regulated genes. In this case $S^{+}$ is comprised of the n-most up-regulated genes (where $n^{+}$=100, 200, 300, etc.), while $S^{-}$ is formed by the n-most down-regulated genes ($n^{-}$=100, 200, 300, etc.).

Gene signatures

A gene signature is a list of genes (also known as a “gene set”) whose collective activity represents a fingerprint of a biological state (mechanism, pathway, disease, etc.). We discuss two main types of gene signatures in this review. First, we define a “directional” gene signature ($D{S}^n$) as a set of up- and down-regulated genes derived from a transcriptomic profile. $D{S}^n$ can be formed by $S^{n+}$ and $S^{n-}$ (which are the $n^{+}$ and $n^{-}$ most up- and down-regulated genes in $\mathit{x}$) used to define ${\mathit{x}}^n$ (i.e., $D{S}^n=\left\{up:S^{n+},down:S^{n-}\right\}$) (Figure 2(c)). For example, the eight up-regulated and five down-regulated genes associated with HDAC inhibition⁶³ are an example of a directional signature (albeit with a different number of up- and down-regulated genes). Others have also used transcriptomic profiles in CMap v1 to generate gene signatures for treatments by including the 250 most up- and down-regulated genes.⁹³ Second, a “non-directional” gene signature ($S$) (Figure 2(d)) is a collection of genes associated with a treatment, a pathway or phenotype in which the direction of differential expression is not known (or not considered). A non-directional gene signature can be formed by combining the up- and down-regulated genes in $D{S}^n$ (i.e., $S=\left\{S^{+}\cup S^{-}\right\}$ (Figure 2e). Alternatively, it can be created from a canonical pathway (Figure 2(e)), which captures expert knowledge-based descriptions of biological networks in terms of the interactions between small molecules, proteins, and genes⁹⁴ (Figure 2(f)). So, a non-directional gene signature can be a list of genes in any given pathway ($S=\{g_{100},g_2,g_{32},\dots ,g_i,\dots \};g_i\in Pathway$). For example, Mootha et al. used the non-directional gene sets of canonical pathways to compare the transcriptomic profiles for diabetic vs healthy tissues, and identified a match with the oxidative phosphorylation pathway.⁵⁸ For the remainder of this review, we refer to non-directional signatures simply as gene signatures or gene sets.

Harmonizing gene signatures and transcriptomic profiles

Thus far, we have discussed four different concepts for representing biological states using gene expression: two types of transcriptomic profiles ({$\mathit{x},{\mathit{x}}^n$}) and two types of gene signatures ({$DS,S$}). We introduce the notion of a gene set object (denoted as $O$) to capture these four possible representations of biological state in terms of genes (i.e., $O\in \{\mathit{x},{\mathit{x}}^n,{DS}^n,S\}$), and to encapsulate different mathematical operations that are needed to calculate transcriptomic similarity. As illustrated in Figure 2, a transcriptomic profile can be translated to a signature, but a signature can also be translated to a profile. For example, a hypothetical non-directional pathway signature $S_A=\left\{g_1,g_2,g_5,\dots ,g_i,\dots ,g_n\right\}$ contains a subset of the genes in $\mathit{G}$ and can be represented as a binary vector ${\mathit{x}}_A=[\mathrm{1,1},\mathrm{0,0},1,\dots ,1,..]$ (that is, $\left\{{\mathit{x}}_A\right|{\mathit{x}}_i=1ifi\in S_A\}$). Similarly, a hypothetical directional signature ${DS}_B^n={\{S}_B^{+}:\left\{g_3,g_4,g_8,\dots \right\}\cup S_B^{-}=\left\{g_2,g_5,g_7,\dots \right\}\}$, can be represented as the vector ${\mathit{x}}_B=[0,-\mathrm{1,1},1,-\mathrm{1,0},-\mathrm{1,1},\dots ]$ (that is, ${\mathit{x}}_B=\mathit{ }\{{\mathit{x}}_i|{{\mathit{x}}_i=1ifi\in S_B^{+},\mathit{ }\mathit{x}}_i=-1ifi\in S_B^{-}\}$).

Transcriptomic similarity measures

We can now define a similarity measure ($SM$) as a pattern-matching operation on a query object ($O_q$) and a reference object ($O_r)\mathrm{ }(SM(O_q,O_r))$ to produce a similarity (or connectivity) score ($s$) and, optionally, an associated significance score (p) (i.e., $\left(s,p\right)=SM(O_q,O_r)$). The values of $s$ and p measure the strength and the significance of the match between $O_q$ and $O_r$. The values of $s$ and p are interpreted as follows: (i) high magnitude values of $s$ suggest a high degree of similarity (positive) or dissimilarity (negative) whereas low absolute values of $s$ imply low similarity/dissimilarity; and (ii) low values of $p(p\ll 1$) are statistically significant whereas, high values of $p$ have lower confidence. Therefore, a combination of the connectivity score and significance score determines the strength and the relevance of the match (also referred to as a “hit”). Most $SM$ use mathematical (set and vector) operations to calculate $s$ by comparing $O_q$ and $O_r$ (which are discussed later and shown in Table 2). All $SM$ use information about the genes that are in common between $O_q$ and $O_r$ (using the set operation $O_q\bigcap O_r$). Some approaches also use information about genes that are not in common between $O_q$ and $O_r$ and we refer to these as the complement of $O$ (denoted as $O\text{'}$), which is the set difference between $O_r\mathrm{a}\mathrm{n}\mathrm{d}{O}_q$($O_q^{\prime }=O_q\backslash O_r$) and similarly, the complement of $O_r\mathrm{ }(O_r^{\prime }=O_r\backslash O_q$). The number of genes in $O_q$ and $O_r$ are referred to as $n_q$ and $n_r$, respectively. If $O_q$ and $O_r$ are $DS$ then $\left\{n^{+},n^{-}\right\}$ are denoted as $\{n_q^{+},n_q^{-}\}$ and $\{n_r^{+},n_r^{-}\}$, respectively.

Table 2.

Summary of connectivity mapping methods

Type	Method	Query	Ref	Publications
Aggregation-based   $S{M}_a$	eXtreme Sum (XS)	$S_q$	${\mathit{x}}_r$ or ${\mathit{x}}_r^{\mathit{n}}$	Cheng et al. 2014
	eXtreme Mean	$S_q$	${\mathit{x}}_r$ or ${\mathit{x}}_r^{\mathit{n}}$
	T-statistic (TS)	$S_q$	${\mathit{x}}_r$	Tian et al. 2005; Goeman et al. 2004, 2005
	Ranksum statistic (RS)	$S_q$	${\mathit{x}}_r$	Barry, Nobel, and Wright 2005; Gower, Spira, and Lenburg 2011
	GSEAa	${\mathit{x}}_q$	$S_r$	Mootha et al. 2003
	GSEAb	${\mathit{x}}_q$	$S_r$	Subramanian et al. 2005
	GSEAc	$D{S}_q$	${\mathit{x}}_r$	Subramanian et al. 2007
	Total enrichment score (TES)	$D{S}_q$	${\mathit{x}}_r$	Iorio, Tagliaferri, and Bernardo 2009
Vector-based   $S{M}_v$	Pearson correlation	${\mathit{x}}_q$ or ${\mathit{x}}_q^{\mathit{n}}$	${\mathit{x}}_r$ or ${\mathit{x}}_r^{\mathit{n}}$	Tenenbaum et al. 2008
	Spearman Correlation	${\mathit{x}}_q$ or ${\mathit{x}}_q^{\mathit{n}}$	${\mathit{x}}_r$ or ${\mathit{x}}_r^{\mathit{n}}$	Tanner and Agarwal 2008; Zhang et al. 2009
	Cosine	${\mathit{x}}_q$ or ${\mathit{x}}_q^{\mathit{n}}$	${\mathit{x}}_r$ or ${\mathit{x}}_r^{\mathit{n}}$	Cheng et al. 2012
	Jaccard index (JI)	$S_q$	$S_r$
	Signed Jaccard (SJI)	$\left\{S_q^{+},S_q^{-}\right\}$	$\left\{S_r^{+},S_r^{-}\right\}$	Zichen Wang et al. 2016

Reference databases

We consider two main types of reference databases ($\mathit{R}$): transcriptomic databases that are comprised of a large compendium of transcriptomic profiles (${\mathit{R}}_{\mathit{x}}=\{{\mathit{x}}_1,{\mathit{x}}_2,{\mathit{x}}_3,\dots \}\mathit{ }$) (Figure 4(d)), and gene signature databases ${\mathit{R}}_{\mathit{s}}=\{S_1,S_2,S_3,\dots \}\mathit{ }$ that contain extensive collections of gene sets ($S$ or $DS$). First, we discuss large-scale reference transcriptomic databases that have been developed for connectivity mapping, including CMap,⁵ LINCS,⁹⁵ and the Gene Expression Omnibus (GEO),¹¹ which is another source of millions of transcriptomic profiles from thousands of experiments (albeit with diverse transcriptomic technologies). Second, we discuss reference gene set databases focusing on knowledge-based canonical pathway gene sets and experimentally-derived directional gene sets.

Figure 4.

Overview of connectivity mapping for estimating chemical concentration-dependent scores for a signature. (a) The query is a non-directional gene signature ($S$) signified by a set of genes (shown as black circles). (b) The reference is a transcriptomic profile $\mathit{x}$ shown as a vector of log2 transformed fold-change (L2FC) values for each gene (blue and red colors represent down- and up-regulation, respectively). (c) The similarity measure ($SM$) for scoring the match between $S$ and $\mathit{x}$. (d) A transcriptomic database (${\mathit{R}}_{\mathit{x}}$) comprised of a collection of $\mathit{x}$ for multiple chemicals and concentrations (Conc). ${\mathit{R}}_{\mathit{x}}$ is visualized as a matrix in which the rows represent genes, the columns show chemical concentrations, and the values in each column are $\mathit{x}$. For example, the outlined box in the matrix signifies eight $\mathit{x}$ for each of the concentrations of a chemical. (e) Concentration-response analysis of similarity scores between $S$ and $\mathit{x}(SM\left(S,\mathit{x}\right)$) for each $\mathit{x}$ of chemical shown in (d). The ordinate and abscissa show the similarity scores and the concentrations of the chemical, respectively. A null distribution (Null Dist.) of similarity scores (shown on the right of the graph along the ordinate axis) is generated by permuting ${\mathit{R}}_{\mathit{x}}$ and calculating $SM\left(S,\mathit{x}\right)$ for all random profiles. The standardized similarity scores (Z) (calculated using the null distribution and shown as “+” symbols) are analyzed by curve-fitting. The fitted concentration-response curve (blue) is used to estimate the benchmark concentration (BMC) corresponding to the benchmark response (BMR) value of Z=1.

Connectivity Map (CMap)

CMap v1 was initially developed to find relationships between chemicals, genes, and diseases.⁴ This dataset was produced by treating MCF7 (breast cancer), HL60 (leukemia), and SKMEL5 (melanoma) cell lines with 164 diverse chemicals at a 10 μM concentration for both 6 and 12 h. Transcriptional profiles were generated using Affymetrix GeneChip HGU133 measuring the levels of ^~ 22,000 transcripts. In all, there were 564 unique transcriptomic profiles and 453 differential expression profiles (after comparing treatments and controls). Following the success of this approach, the same group produced CMap v2,⁶⁵ in which three cell lines (MCF7, PC3, and HL60) were treated with 1,309 chemicals for 6 h to produce 6,100 differential expression profiles using the Affymetrix U133A GeneChip⁹⁶ containing 22,215 transcripts associated with 13,609 genes. The entire CMap v2 database contains 1,294 chemical differential expression profiles in MCF7 cells, 1,182 profiles in PC3 cells, and 1,078 profiles in HL60 cells and is available for download from The Broad Institute. The raw Affymetrix data are normalized and processed (as described earlier) to generate a set of transcriptomic profiles, ${\mathit{R}}_{\mathit{x}}$.

Library of Integrated Network-based Cellular Signatures (LINCS)

Following the success of the CMap v2 project, the U.S. National Institutes for Health (NIH) funded the LINCS Consortium to expand the reference transcriptome database to study genetic (single gene over-expression or knockdown) and chemical perturbations producing a database containing more than 1,000,000 profiles.⁷⁷ To achieve this 1000-fold scale-up of CMap v2, the LINCS Consortium developed computational methods to analyze a large compendium of expression data (12,031 Affymetrix gene expression profiles from GEO) to identify a subset of genes that could predict the entire transcriptome. Their analysis showed that using just 978 “landmark” transcripts could predict the expression of 82% of all genes. The L1000 platform measures these 978 genes (or 1,058 probes) using Luminex bead-based technology. It is, therefore, possible to infer the expression levels of 12,336 genes from the landmark 978 genes, and the resulting transcriptomic profiles are available as moderated Z-scores. The LINCS project has produced 1,319,138 L1000 profiles for 19,811 chemicals and 7,494 genetic perturbations. The L0, L1, L2, and L3 LINCS data are available from GEO as dataset GSE92742 and can also be interactively (or programmatically) analyzed via a cloud-based system (http://clue.io).

Gene Expression Omnibus

The US National Center for Biotechnology Information (NCBI) Gene Expression Omnibus (GEO)¹¹ database is a public domain repository of author-submitted transcriptomics data that conforms to the MIAME (minimum information about a microarray experiment) standard.⁹⁷ Unlike CMap and LINCS, GEO is a repository of published experimental studies and transcriptomics platforms. In GEO, highly multiplexed transcriptomics assays (like the Affymetrix HGU133 GeneChip and The Broad Institute’s L1000 array) are “platforms,” individual transcriptomics profiles are “samples,” and large-scale experiments are stored as a “series” of samples (like CMap v1, v2, and LINCS). A platform contains a set of probes directly linked to genes. The transcriptomic profile for a sample measures the levels for all probes. After substantial curation, a collection of biologically and statistically comparable samples in a series can be made available as a GEO DataSet. As of September 2021, GEO contains 4,348 DataSets, 160,597 series records, from 22,587 platforms and including 4,628,210 samples. All data in GEO can be freely downloaded and analyzed using the R programming language⁹⁸ via the GEOquery⁹⁹ package or using the Python programming language using the BioPython¹⁰⁰ package.

ArrayExpress

The European Bioinformatics Institute (EBI) ArrayExpress^12,101,102 database is the European counterpart to GEO, which also stores transcriptomics data in a MIAME-compliant format provided by authors in support of publications. In addition to transcriptomic data, ArrayExpress also maintains data from other molecule profiling technologies that include measurements of small molecules (metabolomics) and proteins (proteomics). Because metadata for transcriptomic profiles, including experimental model and treatment-related factors, are annotated using a controlled vocabulary,¹⁰³ ArrayExpress can be more suitable for building automated computational workflows. Although initially intended as an integrated resource for all gene expression studies from GEO and the DNA Data Bank of Japan,¹⁰⁴ the rapid growth of transcriptomics data has made this challenging. The lack of a central “index” of all transcriptomic studies makes it necessary to query ArrayExpress and GEO separately; however, some efforts have been undertaken to address this issue.¹⁰⁵ Data in ArrayExpress can be searched and retrieved using the R ArrayExpress package¹⁰⁶ and the Python BioServices package.¹⁰⁷

Directional gene sets

With the availability of thousands of transcriptomics profiles in CMap, LINCS, GEO, and ArrayExpress,¹⁰² it is also feasible to automatically generate gene signatures for a rich range of biological contexts. Gene signatures have been automatically generated for many GEO datasets using detailed annotations of treatments.^108–111 The specific genes included in a signature depend on the annotation of control (normal) and case (perturbed) samples in a study. Automated sample class interpretation (i.e., normal vs. perturbed) can be error-prone due to inconsistent annotations. Therefore, it is essential to manually-curate sample annotations in large public transcriptomic databases to generate valid gene signatures, which is a resource-intensive task. A crowd-sourcing approach has been recently used to develop CRowd Extracted Expression of Differential Signatures (CREEDS),¹¹² which contains $DS$ associated with 2,176 single gene perturbations, 828 disease signatures, and 875 single drug perturbations. The molecular signatures database (MSigDB) is another resource for $DS$ associated with thousands of experimental perturbations ^113,114. A subset of the C2 collection of MSigDB v7.0 includes thousands of $DS$ associated with chemical and genetic perturbations. The LINCS database also provides several predefined signatures for each perturbation derived from the landmark 978 genes and the complete set of inferred genes. One of the key challenges is identifying the parameters for constructing optimal gene signatures from transcriptomic profiles to find biologically relevant connections. We discuss how signature size is one of the factors used in evaluating connectivity mapping approaches for addressing different biological questions.

Pathway and other gene sets

Pathway gene set databases can be constructed from canonical pathways that capture expert knowledge-based descriptions of biological processes. This has also been done comprehensively in MSigDB,^113,114 which includes several canonical databases such as Reactome,¹¹⁵ Kyoto Encyclopedia for Genes and Genomes (KEGG),¹¹⁶ the National Cancer Institute (NCI) Pathway database¹¹⁷ and Gene Ontology.¹¹⁸ MSigDB also contains gene sets related to genetic and chemical perturbations, gene co-expression modules, transcription factor targets, etc., which are represented by $S$ (or $DS$ in some cases).

Connectivity scoring approaches

As stated earlier, gene set-based scoring can be stated generally as $SM\left(O_q,O_r\right)$, where $O_q$ is the input query, and $O_r$ is from a reference database, $\mathit{R}$. This operation can be categorized based on three main attributes: (i) the types of input arguments $O_q$ and $O_r$, (ii) the type of $SM,$ and (iii) the source of $\mathit{R}$. First, the types of $O_q$ and $O_r$ are defined by the possible pairs of inputs from $\left\{\mathit{x},\mathit{ }DS,S\right\}$ (where we assume that $\mathit{x}$ covers ${\mathit{x}}^n$ for brevity), which are expressed as (${\{O}_q,O_r\})$ including: (a) $\left\{{\mathit{x}}_q,\mathit{ }{S}_r\right\}$ or $\left\{S_q,{\mathit{x}}_r\right\}$ (b) $\left\{D{S}_q,{\mathit{x}}_r\right\}$ or $\left\{{\mathit{x}}_q,D{S}_r\right\}$, (c)$\left\{{\mathit{x}}_q,{\mathit{x}}_r\right\}$, (d) $\left\{S_q,\mathit{ }{S}_r\right\}$, (e) $\left\{S_q,D{S}_r\right\}$ or $\left\{D{S}_q,\mathit{ }{S}_r\right\}$, and (f) $\left\{D{S}_q,D{S}_r\right\}$. Although the order of the inputs distinguishes between the query and the reference, we add the subscript “q” or “r” to make the distinction explicit. Second, we categorize $SM$ (Table 2) into two main groups, including “enrichment”-based statistical aggregation approaches, borrowing the terminology used by Irizarry et al.¹¹⁹ (denoted as $S{M}_a$), and “vector”-based similarity metrics, which term was introduced first by Tanner and Agarwal¹²⁰ (denoted as $S{M}_v$). Aggregation-based approaches generally operate on signatures and profiles (i.e., input types (a) and (b)). On the other hand, vector-based methods operate on vectors (inputs of type (c)). Since $S$ and $DS$ can be transformed into vector representations (as described earlier), vector-based approaches can be applied to all kinds of inputs. Lastly, the reference databases are usually defined by ${\mathit{R}}_{\mathit{x}}$ (LINCS, CMap (v1 or v2), or GEO) and ${\mathit{R}}_{\mathit{S}}$ (MSigDB, CREEDS, etc.). We summarize some of the details of the approaches in Table 2, including the mathematical formulas for different $SM$. Lastly, we focus on the nature of the measures used to calculate $s$ and discuss strategies for estimating $p$ in the following section.

Aggregation-based enrichment scoring approaches

Matching transcriptomic profiles to pathway signatures: $S{M}_a({\mathit{x}}_q,\mathrm{ }{S}_r)$

These SM_a match transcriptomic profiles of samples against reference pathway databases (Illustrated in Figure 3). They calculate differences in the distributions of differential expression values in ${\mathit{x}}_q$ for genes in $S_r$ and genes in $S_r^{\prime }$ (where $S_r^{\prime }$ contains genes absent in $S_r$). These measures “aggregate” the values for differential expression in $\mathit{ }{\mathit{x}}_q$ across $S_r$ and compare them with $S_r^{\prime }$ using different summary statistics to produce the output, $s$, which is referred to as an enrichment score (ES) because it determines whether the gene set represented by $S_r$ is “enriched” in ${\mathit{x}}_q$. Different scenarios for the matches between ${\mathit{x}}_q$ and $S_r$ are shown visually in Figure 3 to illustrate examples of high and low-scoring matches. If genes in $S_r$ are concordant with highly up- and down-regulated genes in ${\mathit{x}}_q$ then the $S{M}_a$ produces a high $s$, which is shown by four examples: (i) strong connections: genes in the $S_r$ are concordant with highly up- and down-regulated genes in ${\mathit{x}}_q$, (ii) positive connections: genes in the $S_r$ are concordant with highly up-regulated genes in ${\mathit{x}}_q$, (iii) negative connections: genes in the $S_r$ are concordant with highly down-regulated genes in ${\mathit{x}}_q$, and (iv) no connection: genes in $S_r$ are discordant with up- and down-regulated genes in ${\mathit{x}}_q$ then the $SM.\mathrm{ }S{M}_a({\mathit{x}}_q,\mathit{ }{S}_r)$ distinguish between positive and negative connections and aim to produce $s$ that can rank connections by strength in descending order of magnitude.

Figure 3.

Overview of connectivity mapping as a pattern matching between (a) transcriptomic profile query and (b) non-directional signature reference using a (c) similarity measure showing illustrative examples of matches (f, g, h, and i). (a) The query is a transcriptomic profile $\mathit{x}$ shown as a vector of log2 transformed fold-change (L2FC) values for each gene (blue and red colors represent down- and up-regulation, respectively). (b) The reference is a non-directional gene signature ($S$) signified by a set of genes (shown as black circles). (c) The similarity measure ($SM$) for scoring the match between $\mathit{x}$ and $S$. (d) A collection of transcriptomic profiles for a set of perturbagens. (e) A collection of predefined signatures representing sets of genes (e.g., involved in pathways). (f) “Connection” between $\mathit{x}$ and $S$ is when most up- and down-regulated genes in $\mathit{x}$ match $S$ (observed when $|SM\left(\mathit{x},S\right)|>0$). (g) A “positive connection” between $\mathit{x}$ and $S$ is when mostly up-regulated genes in $\mathit{x}$ are present in $S$ (observed when $SM\left(\mathit{x},S\right)>0$). (h) “No connection” between $\mathit{x}$ and $S$ is when genes in S are randomly distributed across $\mathit{x}$ (where $SM\left(\mathit{x},S\right)\approx 0)$. (i) A “negative connection” between $\mathit{x}$ and $S$ is when mostly down-regulated genes in $\mathit{x}$ are present in $S$ (observed when $SM\left(\mathit{x},S\right)<0$).

Different $S{M}_a$ are distinguished by the approaches for scoring $S_r$ and ${\mathit{x}}_q$. For example, the absolute sum of values for differentially expressed genes values in ${\mathit{x}}_q$ may be much greater for $S_r$ than in $S_r\prime$¹²¹. Alternatively, the (absolute) mean value of the differential expression in ${\mathit{x}}_q$ may be much greater or lower for $S_r$ than in $S_r\prime$. In calculating aggregate scores for $S_r$ and $S_r\prime$, expression values can be measured using q-values (differential expression p-values adjusted for multiple testing) as is the case in the expression signature analysis tool (EXALT).¹²² The two sample Student’s t-test is another approach for determining whether the differential expression values for genes in ${\mathit{x}}_q$ are statistically significantly different between $S_r$ and $S_r^{\prime }$.¹¹⁹ Many parametric aggregation-based enrichment scoring approaches use t-tests,^123–125 ANOVA,¹²⁶ Z-scores,¹²⁷ logistic regression,¹²⁸ random-sets,¹²⁹ and standardized Chi-Squared scores.¹¹⁹ Significance analysis of function and expression (SAFE) implements several parametric methods.¹³⁰ If the differential expression values for genes in ${\mathit{x}}_q$ between $S_r$ and $S_r^{\prime }$ are not distributed normally then parametric aggregation-based enrichment scoring methods may not work. In such cases using non-parametric approaches such as the Wilcoxon rank sum statistics^130,131 or the max-mean statistic¹³² may be more appropriate.

There are three main versions of GSEA, and they are all based on a non-parametric aggregation approach. We discuss the first two versions of GSEA here as they are both of type $S{M}_a({\mathit{x}}_q,S_r)$. The first version, which we refer to as GSEA_a,⁵⁸ calculated a Kolmogrov-Smirnov (KS)-like statistic (“running sum statistic”) by comparing the sorted ${\mathit{x}}_q$ with $S_r$ as follows: (i) create a vector $\mathit{y}$ and update its values using the sorted ${\mathit{x}}_q$, (ii) if gene $g_i\in {\mathit{x}}_q\wedge g_i\in S_r\to {\mathit{y}}_{\mathit{i}}=w$, where $w=1$ (iii) if gene $g_i\in {\mathit{x}}_q\wedge g_i\notin S_r\to {\mathit{y}}_i=w^{\prime }$ where $w^{\prime }<0$ is a penalty, and (iv) calculate the empirical cumulative distribution of $\mathit{y}$ (or “running sum”), $yc={\sum }_{i=1}^k{\sum }_{j<i}^{⬚}{\mathit{y}}_j$, and (v) then $E{S}_a=\max \left(\mathit{y}\mathit{c}\right)$. We refer to the output $s$ of GSEA_a as $E{S}_a$, high values of which suggest enrichment of $S_r$ in ${\mathit{x}}_q$. It is important to note that GSEA_a only used information about the rank of genes in ${\mathit{x}}_q$ based on differential expression (using signal-to-noise ratio and not L2FC values). The second version, GSEA_b,⁵⁹ extended GSEA_a by using the magnitude of differential expression to calculate $w(w=\frac{{|x_j|}^b}{{\sum }_{i\in S_r}{|x_i|}^b}$, where $b=1$) and using a penalty adjusted to the size of the gene set ($w^{\prime }=\frac{1}{N-n}$, where there are $N$ genes in ${\mathit{x}}_q$ and $n$ genes in $S_r$). We refer to the output $s$ of GSEA_b as ES_b. The GSEA_b approach is implemented in sscMap¹³³ and in GSVA.¹³⁴

Connectivity scoring: $S{M}_a(D{S}_q,{\mathit{x}}_r)$

The development of CMap required a new type of similarity measure for matching directional signatures ($D{S}_q$) with transcriptomic profiles (${\mathit{x}}_r$) (illustrated in Figure 1). The core ideas in $S{M}_a(D{S}_q,{\mathit{x}}_r)$, which we refer to as GSEA_c, are an extension of $S{M}_a({\mathit{x}}_q,\mathbf{ }{S}_r)$, but the aggregation-based metrics compare differences in the distributions of differential expression values in ${\mathit{x}}_r$ for genes in $S_q^{+}$ vs $S_q^{+\prime }$ (i.e., up-regulated) with $S_q^{-}$ vs $S_q^{-\prime }$ (i.e., down-regulated), respectively. Therefore, GSEA_c is a similarity measure of the form $S{M}_a(D{S}_q,{\mathit{x}}_r)$, and its output $s$ is called a “connectivity” score.⁵ Three scenarios for the matches between $S_q^d$ and ${\mathit{x}}_r$ are shown in Figure 1 to illustrate three extreme cases of connectivity. First, if genes in $S_q^{+}$ and $S_q^{-}$ are concordant with highly up- and down-regulated genes in ${\mathit{x}}_r$, respectively, then the $S{M}_a$ produces $s>0$ (labeled “positive connection” in Figure 1(f)). Second, if genes in $S_q^{-}$ and $S_q^{+}$ are concordant with highly up- and down-regulated genes in ${\mathit{x}}_r$, respectively, then the $S{M}_a$ produces $s<0$ (labeled “negative connection” in Figure 1(h)). Third, if genes in $S_q^{-}$ and $S_q^{+}$ are distributed randomly with respect to highly up- and down-regulated genes in ${\mathit{x}}_r$, respectively, then the $S{M}_a$ produces $s=0$ (labeled “no connection” in Figure 1(g)). The description of the GSEA_c approach⁵ suggests that GSEA_b was used to calculate enrichment for the up- (${ES}_b^{+}=S{M}_a(\mathit{x},S^{+})$) and down-regulated genes (${ES}_b^{-}=S{M}_a(\mathit{x},S^{-})$) separately to determine overall connectivity score, $E{S}_c={ES}_b^{+}-E{S}_b^{-}$. Connections between the inputs are positive if $E{S}_c>0$ and negative if $E{S}_c<0.$ If ${ES}_b^{+}$ and ${ES}_b^{-}$ have the same sign, however, then weak or no connections are implied. Iorio et al proposed the Inverse Total Enrichment Score (TES)⁹³ using GSEA_c ($TES=1-\frac{E{S}_c}{2}$). Iorio et al. used the TES to search $D{S}_q\mathrm{ }(n^{+}=250,n^{-}=250$) against CMap v2 to calculate drug-induced gene expression profile similarity (DIPS). A weighted connectivity score (WTCS) based on GSEA_c is also used to analyze connections between a signature and the LINCS reference database. The WTCS is like the TES in that a separate ES is calculated for the up- and down-regulated genes.¹³⁵

Vector-based similarity scoring approaches $S{M}_v({\mathit{x}}_q,{\mathit{x}}_r)$

Vector-based approaches use different similarity measures to calculate $s$ between vector representations of the gene set objects, $O_q$ and $O_r$. Most types of similarity measures¹³⁶ can be used to measure the similarity between ${\mathit{x}}_q$ and ${\mathit{x}}_r$. For instance, the dot product of ${\mathit{x}}_q$ and ${\mathit{x}}_r\mathrm{ }({\mathit{x}}_q\cdot {\mathit{x}}_r)$ is the most straightforward measure of similarity. Cosine similarity scales the dot product by the product of the magnitudes, which can be interpreted geometrically as the angle between ${\mathit{x}}_q$ and ${\mathit{x}}_r$ (Cos=$\frac{{\mathit{x}}_q\cdot {\mathit{x}}_r}{{|\mathit{x}}_q||{\mathit{x}}_r|}$). The extreme values for cosine similarity are −1, 0, 1, which correspond to the situation in which the input vectors are antiparallel, orthogonal (unrelated), and parallel, respectively. Most connectivity scores produced by vector-based similarity approaches can be interpreted in the same way as enrichment scores. First, when the up- and down-regulated genes in the query and the reference profiles are highly concordant, then ${\mathit{x}}_q$ and ${\mathit{x}}_r$ are parallel, producing high positive scores (like “positive connections” in Figure 1(f)). Second, when the up- and down-regulated genes in the query and the reference profiles have no overlap, then ${\mathit{x}}_q$ and ${\mathit{x}}_r$ are orthogonal (or independent), producing low scores (like “no connections” shown in Figure 1(g)). Third, when the up- and down-regulated genes in the query profile are matched with the down- and up-regulated genes, respectively, in the reference profile, high negative connectivity scores are produced (like “negative connections” shown in Figure 1(h)).

The first application of cosine similarity was based on using “extreme” transcriptomic profiles (described earlier), and the corresponding similarity measure was referred to as the extreme cosine score (XCos).¹³⁷ Instead of cosine similarity, the correlation has also been used by several groups to calculate connectivity. Geneva¹²⁰ uses Pearson and Spearman correlation coefficients between ${\mathit{x}}_q$ and ${\mathit{x}}_r$ and Zhang et al. use rank sum correlation to calculate connectivity scores.¹³⁸ ProfileChaser¹⁰⁸ also uses Pearson’s correlation but uses the p-value of differentially expressed genes in ${\mathit{x}}_q$ as weights.

The Jaccard index¹³⁹ (or Jaccard similarity) is an even more straightforward approach for calculating the distance between binary representations of ${\mathit{x}}_q$ and ${\mathit{x}}_r$ (calculated using the Jaccard index as $J\left({\mathit{x}}_q,{\mathit{x}}_r\right)=\frac{{\mathit{x}}_q\cap {\mathit{x}}_r}{{\mathit{x}}_q\cup {\mathit{x}}_r};\mathrm{ }0\le s\le 1$) or using the equivalent gene signatures, $S_q$ and $S_r$ (calculated as $J\left(S_q,S_r\right)=\frac{S_q\cap S_r}{S_q\cup S_r}$). Because the Jaccard index does not consider directional fingerprints, it only produces positive values of $s$. In CREEDS,¹¹² on the other hand, the authors describe an approach to evaluate directional signatures using the signed Jaccard index (SJI) (given in Table 2).

Estimating significance of transcriptomic similarity scores

Evaluating confidence in similarity scores is important for determining the biological relevance of matches between $O_q$ and $O_r$. Confidence in each similarity score can be estimated as the probability ($p$) of observing a value of $s\mathrm{ }=\mathrm{ }SM(O_q,O_r)$ given the background distribution of s ($\psi (s)$). $\psi (s)$ can be estimated empirically by permutation testing. There are several different ways in which $\mathit{R}$ can be permuted, and they depend on the choice of a null hypothesis. If the null hypothesis is based on genes in $O_q$ alone the approach is called “self-contained,” but if the genes include the complement $O_q^{\prime }$ then the approach is called “competitive”.¹³⁸ A self-contained null hypothesis states that no genes in a signature are differentially expressed. On the other hand, a competitive hypothesis tests whether genes outside the signature are at least as differentially expressed as the genes within the signature. Another important question is whether the randomization procedure breaks the correlations between genes,¹⁴⁰ leading to apparently more statistically significant but less biologically relevant matches.

Instead of permutation-testing, it is also possible to derive $\psi (s)$ using an extensive transcriptomic profile database. For instance, in Geneva¹²⁰ $s$ is calculated (separately for each vector similarity measure) using the CMap v1 database to derive a true $\psi (s)$, which is used to estimate $p_0$. In connectivity mapping based on LINCS, the authors avoid deriving $p_0$ based on permutation testing to preserve the correlation structure between genes. Instead they use the entire $\mathit{R}$ to determine $\psi (s)$ and then calculate the proportion of reference profiles with a similar connectivity score, which they call the tau score (τ).¹⁴¹

Evaluating transcriptomic similarity matching approaches

Given the breadth of techniques involved in connectivity mapping, one approach for evaluating their utility is to compare their performance using objective criteria. For most new connectivity mapping methods, this means evaluating performance by scoring hits between gene set objects from reference data sets using one of the baseline approaches (which are generally GSEA_b and GSEA_c). Alternatively, the performance of connectivity mapping can also be framed as a classification problem. For example, Mootha et al.⁵⁸ analyzed differentially expressed genes from diabetic muscle samples and “classified” them as relevant for oxidative phosphorylation. This approach can be generalized for evaluating connectivity mapping; however, it requires an annotated set of positive and negative examples in reference databases.

One of the challenges in evaluating connectivity mapping approaches is that there are no gold-standard data sets of chemicals and targets. For example, the Anatomical Therapeutic Chemical (ATC) classification¹⁴² organizes drugs hierarchically based on four levels: the organ system at the top level, then therapeutic characteristics, and then the specific mechanism at level 4. One approach for classifying mechanisms by connectivity mapping developed by Iorio et al.⁹³ used TES to identify similar drugs in CMap v2. They constructed a “drug network” based on connectivity scores and evaluated significance based on random pairs in CMap v2. Next, they determined whether similarity in the network predicted similarity in drug mechanisms labeled by ATC codes. After evaluating performance using the area under the receiver operating characteristic (AUROC) curve, they showed that close neighbors in the network shared mechanisms. While this analysis did not objectively compare different $SM$, it paved the way for using ATC codes and AUROC curve analysis for evaluating the results. Similarity scores between transcriptomic profiles of chemicals labeled with ATC scores have been objectively based on their known mechanisms or relationships with protein targets.^15,121 However, the ATC scheme is specific to drugs and may not help evaluate the broader environmental chemical space.

The first evaluation of connectivity mapping approaches for predicting drug mechanisms was based on CMap v2 as the reference database with drug mechanisms labeled using ATC codes.⁷² This work compared mean-centering and differential expression analysis (based on treated and control samples) for creating the transcriptomic reference database, a range of signature sizes, different connectivity scoring methods (TES, GSEA, XCos), and used AUROC curves to compare the performance of these approaches for classifying ATC level 4 codes. Instead of using the area under the entire ROC curve, Cheng et al. measured the partial area under the curve (AUC) for a false positive rate of less than 0.1 (AUC0.1). Based on AUC0.1, the XCos vector similarity method outperformed KS and TES using gene signatures comprised of 100 up- and 100 down-regulated genes.

Transcriptomic Connectivity Mapping in Toxicology

Determining the potency and potential of chemicals for activating molecular targets that can lead to adverse health outcomes is a key challenge in toxicology.⁶⁶ Though high-throughput screening (HTS) formats such as ToxCast ^143,144 are more cost-effective than animal testing; there are far too many chemicals in commerce to evaluate using multiple HTS assays. High-throughput approaches such as RNA-Seq,⁹ and, more recently, TempO-Seq,⁷⁰ are extremely promising and cost-effective for generating transcriptomic profiles for tens of thousands of chemical treatments. Connectivity mapping can be used to evaluate the potential targets and the off-target effects of a new drug or a chemical. Whether considering new chemical entities for drug discovery or untested environmental chemicals for public health protection, transcriptomic connectivity mapping is a robust and high-throughput alternative to the existing techniques.¹⁶ With a deadline of eliminating the use of mammalian test results by 2035,¹⁴⁵ developing new approach methodologies (NAMs), which could efficiently provide information about chemical hazards and risks without using whole animals,¹⁴⁶ is imperative for protecting public health and the environment. Connectivity mapping using transcriptomic data is a NAM-based methodology that will aid in realizing this vision.

Connectivity mapping has been used to characterize ecotoxicological chemical stressors using fish transcriptomic data.¹⁴⁷ The authors constructed a reference database for 55 treatment conditions using transcriptomics data from fathead minnow and zebrafish (using a variety of gene expression platforms). Chemicals in the reference database were annotated with mechanisms using molecular initiating events (MIEs) in adverse outcome pathways (AOPs). Then they used sscMap¹³³ to find connections between gene signatures derived from new samples and profiles in the reference database. De Abrew et al. investigated the mode of action (MOA) for 34 different chemicals using transcriptomic profiles measured in MCF7, Ishikawa, HepaRG, and HepG2 cells by comparing them with the CMap v2 database by similarity using hierarchical clustering and identified biologically-relevant connections.⁴⁸ More recently, we have used multiple connectivity mapping methods presented in this review to solve three problems. First, we evaluated the reproducibility of transcriptional effects for reference chemicals in primary rat hepatocytes using TempO-Seq and Affymetrix data from OpenTG Gates.⁶⁹ We found that Jaccard and cosine similarity was more accurate than GSEA_c for correctly matching extreme transcriptomic profiles of chemicals produced by TempO-Seq and Affymetrix technologies. Second, we successfully used GSEA_c to calculate the concentration-dependent effects of chemicals on pathways and directional signatures⁶⁸ using an approach illustrated in Figure 4. Third, we developed consensus signatures of stress response pathways and used GSEA_c to match them with transcriptomic profiles for reference perturbagens.¹⁴⁸ These studies suggest the feasibility of applying connectivity mapping to evaluate environmental chemical toxicities using transcriptomics data.

Using connectivity mapping to evaluate the potential off-target effects of drugs or effects of environmental chemicals requires a computational pipeline with six components. First, a database containing gene signatures corresponding to the transcriptional effects of reference chemicals and canonical pathways. Large-scale transcriptomic data sets such as CMap,⁴ LINCS,⁹⁵ and GEO¹¹ can be used to create a reference gene signature (${\mathit{R}}_s=\{D{S}_1,D{S}_2,\dots \}$) database annotated with (a) uniquely identified chemicals¹⁴⁹ and known mechanisms,¹⁵⁰ or (b) thousands of genetic perturbations. To cover a broad set of transcriptional perturbations, gene signatures ($S$) associated with canonical pathways, expression modules, and transcription factors (e.g., MSigDB¹¹⁴) should also be included in ${\mathit{R}}_s$. Second, the transcriptomic profiles and signatures for untested environmental chemicals must be produced using one of the high-throughput transcriptomic technologies. Third, multiple connectivity mapping tools should be used to search the signatures against the reference transcriptomic databases to identify potential hits and to infer their putative targets. Fourth, for each chemical, the concentration-dependence of $s$ (and significance) must be analyzed to confidently establish a relationship with each putative target (if the transcriptomic data have been generated across multiple concentrations of test chemicals) (Figure 4). Fifth, the performance of the entire connectivity mapping approach must be systematically evaluated with a benchmark set of chemicals with known targets in order to develop best practices to ensure confidence in predictions. Lastly, in vitro potency values estimated from concentration-response analysis of chemicals must be extrapolated to oral equivalent doses using toxicokinetic modeling. Recently, Harrill et al. have successfully implemented and tested many components of this workflow in a pilot study in which 44 chemicals were tested in MCF7 cells in concentration-response using TempO-Seq.⁶⁸

The choice of similarity measure is essential in applying connectivity mapping to toxicology. Systematically comparing GSEA with other vector and aggregation-based approaches using benchmark data suggests that it has either similar or lower performance.^{119,120,123,126,127} Our initial findings also suggest that vector-based approaches combined with extreme transcriptomic profiles match chemicals to their known molecular targets more accurately than GSEA.^69,151 On the other hand, aggregation-based approaches could be more suitable for concentration-response modeling (see Figure 4) to estimate biological pathway activating concentrations.⁶⁸ Given the many steps involved in connectivity mapping workflows, further research on the contribution of different factors, including the choice of similarity measures, is necessary.

Discussion

The reproducibility,¹⁵² scalability,^9,70 and broad biological coverage of transcriptomics make it feasible to profile millions of biological samples for chemical treatments, genetic manipulations, and diseases.^5,11,95 In toxicology, connectivity mapping is being used to identify the putative targets of new chemicals,^48,69,147 to determine their impact on ecological and human health via adverse outcome pathways,¹⁵³ to demonstrate the robustness and reproducibility of transcriptional effects across different studies and technology platforms,⁶⁹ and to estimate biological pathway activating concentrations.⁶⁸ This makes transcriptomic data-driven connectivity mapping a powerful tool for screening the many thousands of chemicals in commerce¹⁵⁴ for putative effects and potency estimates for a broad array of cellular pathways.

One of the potential novel applications of connectivity mapping is to provide potential biological analogues^47,48 when a new chemical lacks structurally similar substances. In such cases, using transcriptomic profiles in place of structure descriptors can rapidly identify possible mechanisms, and other properties, for untested chemicals using reference databases.⁶⁹

This review provides a systematic analysis of the key elements of connectivity mapping and a detailed tutorial to build workflows for evaluating chemical-induced toxicity using transcriptomic data. First, we proposed a coherent terminology to formalize diverse transcriptomic technologies (microarrays, L1000 and RNA-Seq). This terminology forms the basis of a uniform framework to represent transcriptomic profiles and produce gene signatures from them. Second, we used our proposed terminology to discuss some of the most widely used data sources for transcriptomic profiles (CMap, GEO, and LINCS) and gene signatures (MSigDB and CREEDS). Third, we formalized the connectivity mapping workflow in terms of a database search and retrieval task in which a query object is searched against a reference database using similarity measures to identify “hits.” Fourth, we classified published connectivity mapping approaches into two broad categories of similarity measures: vector-based and aggregation-based approaches. While vector-based approaches resemble similarity metrics used in other domains (e.g., cheminformatics), aggregation-based approaches are a new class of algorithms for measuring the similarity between transcriptomic profiles and gene signatures. Fifth, we reviewed the performance of aggregation and vector approaches reported in the literature on benchmark transcriptomic data sets. Beyond serving as a review of connectivity mapping approaches, we believe this manuscript can serve as a practical guide for implementing workflows by integrating public domain data and computational tools.

There are many limitations of connectivity mapping methods that should be addressed by future research to improve the utility of these approaches to toxicology and related disciplines. We summarize these into four broad categories: defining signatures of biological states, curating new benchmark data sets for evaluating connectivity mapping approaches, devising novel sensitive and specific similarity measures, and developing computational workflows that standardize and enable reproducible workflows. New strategies for developing reliable gene signatures will be vital for successfully applying connectivity mapping to interpret biological effects from transcriptomic profiles. First, there is considerable redundancy in published signatures (e.g., MSigDB¹¹³), which can produce multiple hits when searching transcriptomic profiles. Searching transcriptomic profiles against redundant signatures produces related hits that can hide the subtle but more biologically relevant effects. For example, there are dozens of signatures in MSigDB associated with DNA damage response, oxidative stress response, unfolded protein response, and other cellular stress response pathways. One approach for reducing redundancy is to aggregate related signatures into “consensus” signatures.¹⁵⁵ Indeed, the idea of developing such consensus signatures is embodied in the Hallmark Signature collection of MSigDB.¹¹⁴ Second, published signatures may not wholly encompass the full range of phenomena observed in transcriptomic data. Using transcriptomic profiles for more extensive collections of chemical and genetic perturbations (e.g., from CMap, LINCS, etc.), it may be possible to fingerprint a more comprehensive array of chemical mechanisms or putative biomarkers of toxicity¹⁵⁶ as gene signatures. Early transcriptomic technologies were limited by data reproducibility,¹⁵⁷ which reduced their utility in finding reliable biomarkers. Although dealing with biological variability is still a challenge,¹⁵⁸ technological advancements are improving data quality,^68,69,152 making it feasible to transform transcriptomic profiles for specific perturbagens into gene signatures that can be used for connectivity mapping more reliably (e.g., shown visually in Figure 2). Therefore, additional research is necessary for combining profiles for a chemical (e.g., for different concentrations and time points) or for chemicals with similar mechanisms to develop new gene signatures. ML techniques could aid the development of such signatures when there are a sufficient number of profiles,¹⁵⁹ simple statistical approaches may be adequate if there are just a handful of profiles to form such signatures.¹⁵⁵ The problem is that similarity-based methods, such as connectivity mapping, require a sufficient number of relevant descriptors but are notoriously sensitive to irrelevant descriptors.²³ The number of genes necessary for producing an accurate signature for identifying a specific mechanism from transcriptomic data can only be defined by empirical evaluation. For example, Lee et al. compared chemical-induced gene signatures (from primary rat hepatocytes using TempO-Seq data) of increasing sizes using different connectivity scoring approaches with a legacy data set (Affymetrix data from Open TG-GATES¹⁶⁰) with varying results by chemical, mechanism, and treatment concentration.⁶⁹ As expected, 8 and 200 μM of WY14643, which is a peroxisome-proliferator activated receptor alpha (PPARα) agonist, produced hits with other PPARα activators in Open TG-GATES. Similarly, valproic acid (400 and 10,000 μM) produced hits with other PPARα activators in Open TG-GATES using different vector-based approaches. On the other hand, even a 1000 μM acetaminophen treatment did not find any relevant matches. While using concentration-response data could avoid false negatives, optimizing the choice and number of genes included in signatures plays an important role in finding mechanisms of chemicals with subtle transcriptional effects.

The robustness of connectivity mapping approaches for differentiating actual biological signals from the noise requires further systematic evaluations using benchmark transcriptomics data sets derived from chemicals with known bioactivities studied in appropriate cell types using relevant concentrations and exposure durations. Although the ATC classification¹⁴² helps annotate drugs, this scheme is not ideal for annotating environmental chemicals. Other efforts are underway to integrate evidence about chemical bioactivity from disparate structured¹⁵⁰ and unstructured¹⁵¹ sources to define potential reference chemicals for specific molecular and cellular targets. Once chemicals have been assigned to different classes of targets, then transcriptomic profiles for these chemicals can be retrieved from CMap, LINCS, or GEO to build gene signatures using the approaches discussed earlier. While connectivity mapping with these reference signatures can help assign putative targets or pathways to untested chemicals with high sensitivity, confident assessment of specificity is still challenging. Therefore, further work on improving specificity, by either finding additional perturbagens that do not activate a given target (i.e., negatives) or by using different randomization strategies to create “null” chemicals, is also important. Lastly, new software tools are needed for implementing different connectivity mapping workflows that can be tailored to specific problems and executed efficiently for large-scale data sets.

We have attempted to organize the spectrum of connectivity mapping approaches into vector- and aggregation-based approaches. Vector-based similarity measures are widely used in cheminformatics for finding chemical analogs using structure descriptors. Interestingly, vector approaches are also helpful in matching transcriptomic profiles and gene signatures. On the other hand, aggregation-based methods are unique to analyzing transcriptomic data, with GSEA as one of the most widely used techniques. Systematic comparisons of GSEA with other vector and aggregation-based approaches using benchmark data suggest that it may not always be the most accurate method.^{119,120,123,126,127} For toxicology applications, our findings to date suggest that vector-based approaches are more effective for identifying putative molecular targets,^69,151 but aggregation-based approaches are more suitable for concentration-response modeling (see Figure 4) to estimate biological pathway activating concentrations.⁶⁸ Further research into the relative merits of vector- and aggregation-based approaches for toxicology applications could address some of these questions.

Novel connectivity mapping approaches beyond vector- and aggregation-based techniques are under active development, and two promising research areas are worth mentioning. First, dimensionality reduction of transcriptomic profiles to find more biologically meaningful latent representations (e.g., for molecular targets) could overcome some challenges in finding optimal gene signatures from noisy data. For instance, probabilistic connectivity mapping (ProbCMap¹⁶¹) uses latent factor models, which aggregate information across genes using different statistical models to construct a low-dimension representation of transcriptomic profiles. ProbCMap uses group factor analysis,¹⁶² sparse factor analysis,¹⁶³ and Bayesian principal components analysis to generate low-dimensional vector representations of transcriptomic profiles and measure similarity between them using Pearson correlation. More recently, deep learning methods¹⁶⁴ based on multilayer artificial neural networks are enabling an exciting wave of novel data-driven approaches to elucidate latent representations of biological mechanisms from transcriptomic data ¹⁶⁵ and even predict transcriptomic signatures based on transcription factor activity.¹⁶⁶ Second, traditional gene signatures can be enhanced with computational approaches for analyzing genetic regulatory and signaling networks^167–169 to predict the activation of transcription factors from transcriptomic data.¹⁷⁰ Although we have not described these two approaches systematically in this review, the background and formal basis for analyzing connectivity mapping will help readers to place these in context. Our future work will expand on new connectivity mapping strategies based on deep learning and network analysis for identifying molecular targets of chemicals and drugs using transcriptomic data.

Conclusion

Connectivity mapping assumes that if two transcriptomic profiles are similar, it is due to a common biological state or process. If two chemicals produce similar transcriptomic profiles, then it could mean that they act via similar mechanisms. Therefore, connectivity mapping can infer the putative molecular targets of new chemicals based on existing chemicals or the toxicological properties of new chemicals based on known toxicants.¹⁶ Connectivity mapping can be used for finding biological analogues,⁴⁸ for determining the mechanism of action,^69,147 and estimating pathway activating concentrations for chemicals by coupling similarity scores with concentration-response modeling.⁶⁸ This review provides the relevant background and in-depth explanation of connectivity mapping workflows to address toxicology problems. It also lays out a roadmap for future research to address current challenges. This work is a conceptual overview for those interested in learning about the utility of connectivity mapping in the context of NAMs, practitioners interested in using connectivity mapping pipelines in their workflows, and researchers interested in developing novel approaches that advance the state-of-the-art.

Biographies

Author Biographies

Dr. Imran Shah is a computational systems biologist at the US EPA Center for Computational Toxicology and Exposure (CCTE), where he develops new data-driven methods to predict chemical effects and toxicological tipping points. His research aims to reduce dependence on animal testing using artificial Intelligence, machine learning, and multi-scale models trained with large-scale chemical and bioactivity data. He also translates these models into interactive tools to support chemical safety decisions. He completed his B.Sc. in physics from Imperial College and a Ph.D. in computational biology from George Mason University.

Dr. Joseph Bundy is a biologist at the US EPA CCTE. He has experience analyzing and integrating high-dimensional molecular data sets focusing on transcriptomics. His current work involves leveraging machine learning techniques to predict the activation of Molecular Initiating Events in biological systems perturbed by chemical exposure.

Dr. Bryant Chambers is a post-doctoral research fellow in the US EPA CCTE. He is a computational biologist with experience in high-throughput transcriptomic chemical hazard screening. His research focuses on the role of stress response networks in toxicological outcomes, adaptive antimicrobial resistance, and mechanisms of nanotoxicity.

Dr. Logan E. Everett is a bioinformatics scientist in the US EPA CCTE. His expertise includes genomics, statistics, genetics, and molecular biology. His research at EPA is focused on advancing the application of high-throughput transcriptomics in chemical safety screening.

Dr. Derik Haggard is a scientific analyst at the US EPA CCTE. He has expertise in computational toxicology, developing high-throughput transcriptomics analysis pipelines, and assisting in developing tools, applications, software, and databases used in the chemical hazard characterization and risk assessment process.

Dr. Joshua Harrill is a cellular and molecular toxicologist at the US EPA CCTE. His expertise includes in vitro toxicology, specifically the application of transcriptomics, high-content imaging and other complementary technologies for high-throughput chemical hazard screening, characterization, and informing chemical risk assessment.

Dr. Richard S. Judson is a bioinformatics scientist at the US EPA CCTE, where he is developing computer models and databases to predict toxicological effects of chemicals. His current research includes in vitro and in vivo database development and using these to build models to predict the behavior of new chemicals, and deriving pathway-based approaches using high-throughput transcriptomics data. He has published in computational biology and chemistry, bioinformatics, genomics, human genetics, toxicology, and applied mathematics. He has a BA in Chemistry and Chemical Physics from Rice University and an MA and PhD in Chemistry from Princeton University.

Dr. Johanna Nyffeler is a post-doctoral research fellow at the US EPA CCTE. She focuses on developing new approach methodologies (NAMs) for toxicity testing, including high-content imaging and cell painting. She won the Lush Prize (2020) for her innovative research on assessing the developmental neurotoxicity of environmental chemicals using cell painting in neurons. Dr. Nyffeler completed her BS in biochemistry from the University of Fribourg, MS in genetics from the University of Zurich, and PhD in toxicology from the University of Konstanz.

Dr. Grace Patlewicz is currently a research chemist at the US EPA CCTE. She started her career at Unilever UK, before moving to the EC Joint Research Centre in Italy and then to DuPont in the US. A chemist and toxicologist by training, her research interests have focused on developing and applying QSARs and read-across for regulatory purposes. She has authored over 150 journal publications and book chapters, chaired various industry groups, and contributed to developing technical guidance for QSARs and chemical categories under various OECD programs.