Enhancing Reproducibility in Cancer Drug Screening: How Do We Move Forward?

Abstract

Large-scale pharmacogenomic high throughput screening (HTS) studies hold great potential for generating robust genomic predictors of drug response. Two recent large-scale HTS studies have reported results of such screens, revealing several known and novel drug sensitivities and biomarkers. Subsequent evaluation however found only moderate inter-laboratory concordance in the drug response phenotypes, possibly due to differences in the experimental protocols used in the two studies. This highlights the need for community-wide implementation of standardized assays for measuring drug response phenotypes so that the full potential of HTS is realized. We suggest that the path forward is to establish best practices and standardization of the critical steps in these assays through a collective effort to ensure that the data produced from large-scale screens would not only be of high intra-study consistency, so that they could be replicated and compared successfully across multiple laboratories.

Summary

Pharmacogenomic high throughput screening offers tremendous promise in the rational development of targeted therapies, but its full potential will not be realized unless experimental protocols and analysis methods are standardized through a collective effort.

With dropping costs of genetic testing, cutting-edge sequencing technologies are now at the forefront of new cancer drug development. Yet, how to fully incorporate the presence of individual mutations and other specific genomic features to inform personalized therapeutic decisions remains a major challenge. Pharmacogenomics has emerged as a promising strategy to rapidly elucidate the link between genes and drugs by systematically characterizing the effects of pharmacologic agents on whole biological systems, allowing concurrent identification of therapeutic targets and discovery of drug candidates [1].

Advancements in molecular biology and the genomic sciences have had a profound impact on drug discovery, and together with the advent of automated high-throughput screening (HTS) have dramatically changed the drug development endeavor [2]. Pharmacogenomic HTS presents both opportunities and challenges. The goal is to measure the response of hundreds or thousands of cell lines to drugs or other perturbations and to associate the response with the genomic characteristics of each cell line. The extensive data generated from such efforts provide opportunities to search for biomarkers and to explore the mechanism associated with the underlying response. Two recent pharmacogenomic HTS studies, the Cancer Cell Line Encyclopedia (CCLE) [3] and Cancer Genome Project (CGP) [4] evaluated an impressive array of cell lines and drugs (1,036 cell lines and 24 drugs, and 727 cell lines and 138 drugs, respectively), generating gene-expression profiles and drug-sensitivity data for each combination. A subsequent comparative analysis of the CCLE and CGP found that while the gene expression profiles were highly concordant between studies, the measured cell line drug sensitivities were inconsistent [5], which has been further confirmed by an independent research group [6]. While the apparent variability in drug response presents a serious barrier to the ultimate goal of such studies – to develop signatures predictive of response – the high degree of correlation in the gene expression measures provides hope for a potential path forward. Here, we review potential explanations for these findings and extend recommendations for improving the reproducibility of pharmacogenomic HTS studies.

Pharmacogenomics in Cancer Drug Development

With more than 900 new cancer drugs in clinical testing [7] and better technological tools to characterize patient populations and modulation of drug targets, the paradigm of oncology drug development is shifting [8, 9]. A new drug must now demonstrate proof of concept that it can be beneficial as early as possible in clinical development. Pre-clinical pharmacogenomic HTS could identify genomic predictors allowing investigators to enrich early-phase clinical studies for those patients most likely to receive benefit. An early signal of proof of concept activity, such as vemurafenib in BRAF V600 mutant melanoma [10] and crizotinib in ALK fusion non-small cell lung cancer [11], can rapidly reduce the time to clinical testing and subsequent market approval. With a growing repertoire of potential drug combination partners, HTS could help to prioritize genotype-selective drug combinations for clinical testing [12]. Genomic predictors developed from HTS data can support this new paradigm when the number of false positive leads is contained.

HTS has become one of the primary scientific tools used in the pharmaceutical industry to generate new leads [13]. Very large compound libraries are screened for “hits” against a target [14]. Hits are generally target-specific active small molecules, which are further characterized for dose-response effects in secondary screens. Functional characterization and validation in animal models leads to selection of candidates for clinical testing. Pharmacogenomic screening extends the conventional HTS screening by using high-throughput approaches to evaluate the effect of compounds on large panels of cancer cell lines; this approach enables to identify simultaneously druggable targets and biologically active compounds [1, 15]. Effectiveness of a compound is assessed by functional assays that measure changes in such activities as cell proliferation, DNA replication, or cell death. Associating these changes with drug-induced transcriptomic alterations can identify cellular processes that are affected by a given treatment. A key advantage of pharmacogenomic screening is that it can compress the timeline of conventional HTS. By using systems biology methods, at least partial functional characterization and initial validation of the specificity/selectivity of a given compound can be obtained.

Given the complexity of HTS assay, there are many sources of experimental noise that should be carefully controlled to provide sufficient power to detect reliable “hits” [16]. Occasional false positive hits passing through the primary screen will likely be caught in secondary screening steps. However, the biological outcomes measured by pharmacogenomic HTS assays are often associated with increased variability, which reduces our ability to detect weaker target-drug-genotype associations in a single step. Several sources of variation affect these assays and may limit their power to reliably detect a broad range of biological associations, especially where a compound is effective against multiple potential targets in one or more pathways, or when the overall cytotoxic effect reflects crosstalk among interconnected biological pathways.

Lack of Concordance in Large Pharmacogenomic Studies

For pharmacogenomics to realize its full potential, the assessment of drug response needs to be robust, reliable, and reproducible. The CCLE and CGP drug screening studies are the largest public pharmacogenomic screens to date. To the credit of the investigators, they shared important details of their experimental protocols and all the pharmacogenomic data generated, allowing a careful reanalysis. Fifteen drugs and 471 cancer cell lines were screened in both studies (Figure 1A). These studies provided a unique opportunity to assess the consistency of both the raw data and the set of significant associations between genomic features, defined by gene expression measurements or mutation calls, and measures of drug response [5].

Figure 1.

(A) Comparison of the Cancer Genome Project [4] and The Cancer Cell Line Encyclopedia [3] as published by Haibe-Kains et al. [5]. Our comparative study revealed a high degree of concordance between the gene- expression datasets, moderate concordance for the mutation data, but low concordance between drug-sensitivity assays; such inconsistency propagates into the gene–drug associations found by the two studies. (B) Main steps in the experimental protocols used in the CCLE and CGP studies; each step is further described in Supplementary Table 1. The choice of experimental procedures vastly differs between the two studies, illustrating the lack of standardization in large pharmacogenomic studies.

The experimental protocols used to profile the transcriptomes at baseline prior to drug treatment were very similar in both studies, with the exception of the different density of the Affymetrix chips used (Figure 1A). Transcriptional profiles from replicate samples within the same study and between identical cell lines across studies were highly correlated, suggesting that the high level of standardization for gene expression profiling and analysis enabled reproducible findings. Only moderate concordance was observed in mutations detected in 64 genes in the 471 common cell lines between CCLE and CGP, which could be due to differences in the platforms used for mutational profiling (Figure 1A), or as a result of genetic divergence of the cell lines during passaging.

CCLE and CGP studies generated large amounts of drug sensitivity data using experimental protocols that differed in many important aspects (Figure 1B; Supplementary Table 1, Supplementary Information), including the specific pharmacological assay used (ATP-based vs. reductase-based metabolic activity for CCLE and CGP, respectively). Although the two studies were not designed specifically to address inter-laboratory reproducibility, the differences reflect a lack of consensus for pharmacogenomic HTS protocols. For 13 out of the 15 drugs tested in common in CCLE and CGP (87%) a high level of inconsistency between drug sensitivity measurements was observed (Spearman’s rank correlation < 0.5), regardless of which characteristic of the drug-dose response curve was used to summarize drug potency (Figure 1A). Intriguingly, when the same protocol was used at two different sites in the CGP study, only a fair inter-site correlation for sensitivity to camptothecin was observed (Spearman’s rank correlation for IC₅₀ of 0.57). These results led us to conclude that the two experimental protocols are not equivalent and that measuring drug response is a complex and poorly reproducible process. While compounds were assayed at least in duplicate for each cell line, unfortunately, the raw data from these technical replicates were not publicly released by either study, preventing a direct assessment of assay reproducibility.

To confirm our findings, we compared the data for two drugs, lapatinib and paclitaxel, against a third large pharmacogenomic dataset generated by GlaxoSmithKline (GSK) that used the same ATP-based assay as in the CCLE study [17]. As expected, the correlation was higher between CCLE and GSK compared with CGP and GSK, suggesting that the choice of assay has a major impact on drug response measurements. However, the correlation between GSK and CCLE drug sensitivity data was still only fair (Spearman correlation < 0.51), implying that the drug sensitivity assay used is not the only source of inconsistency [5].

The primary goal of CCLE and CGP studies was to discover new associations between genomic and transcriptomic features of cancer cells and drug response that could advance our understanding of drug sensitivity and allow development of robust genomic predictors of response. Thus, we tested whether these associations were consistent across studies. We observed that, although we were able to validate the strongest associations identified in both studies, coupling consistent transcriptomic data with inconsistent drug sensitivity measurements resulted in inconsistent pharmacogenomic associations for most of the investigated drugs. Hence, any single study may not be able to identify robust predictors of drug response because a strong association observed in one study may not necessarily be evident in independent studies [5].

In vitro Cell-Based Drug Screening: What Can Affect Assay Consistency?

When considered independently, both the CCLE and CGP studies approached their experimental design, implementation, and data analysis with apparently adequate care and rigor; however, there were methodological differences between the two studies (Supplementary Information). Several factors could introduce variability in the results of in vitro cell-based assays. It is important to establish best practices on how the chemical compound collection is managed, maintained, and delivered to the assay plates for screening [18]. Critical factors include compound handling to ensure accuracy in the amount of the compound transferred to the master solution, maximal solubility in the solvent used (usually DMSO), optimal storage conditions to ensure compound integrity, and to minimize liquid loss through evaporation or leaching of the test compounds through the storage containers [19]. Because of such potential artifacts, the purity, integrity and concentration of the compound stock solutions should be verified prior to being used in the assay. Of the two studies, only CCLE reported pre-assay compound library validation. Furthermore, the type of the liquid transfer system can affect the accuracy of the amount of compound and cell culture stocks delivered to the recipient plates [20]. Tip-based serial dilution and dispensing could lead to larger IC₅₀ values for some compounds, resulting in more than 100-fold underestimation of their potency compared with acoustic dispensing methods [21].

Minor variations in cell culture conditions including seeding density, plating efficiency, growth rate and cell cycle distribution may affect cell metabolism and drug responsiveness. The number of cells per well at the time of pharmacological testing also has a major impact on the dynamic range and sensitivity of the assay, and this variable depends both on the seeding density and on the grow rate of the cells. Several of these variables are affected by cell culture conditions including substrate, choice of cell culture medium (including relative concentrations of glucose, glutamine, and amino acids), and serum (final concentration, type, source, and batch effects). Other factors affecting the cell lines include their reference source, passage number, and the quality of the stock cultures used to seed the experiments. Even cell lines obtained from the same reference source may diverge enormously unless similar passages from the reference culture are used. Both the CCLE and CGP groups appropriately attempted to reduce the contribution of several of these variables, which would help control intra-study variability. However, these actions do not necessarily improve inter-study comparability.

Adaptive responses to the stress of drug treatment can occur within the transcriptome, proteome, metabolome, kinome, and methylome; some of these changes may be detected in hours or less of the initiation of exposure to a drug. Where rapid drug-induced changes in cells reflect common and/or evolutionarily conserved stress response pathways, key features that drive profile changes may appear qualitatively more similar than different across studies. Thus, it is likely that, provided the cell lines are of the same origin, the genomic data will appear broadly similar. If the same cell lines respond in the same way to the same drugs, selective features of the transcriptomes might be expected to exhibit at least qualitatively similar changes. These observations are broadly consistent with the findings of our comparative study [5].

The type of cytotoxicity assay used in assessing drug sensitivity could significantly impact the measured phenotypic response. Among the several assays compatible with HTS [22, 23], bioluminescent detection of adenosine triphosphate (ATP) is one of the most sensitive and reliable methods [24] and has been used extensively in HTS studies (Table 1). CCLE used an ATP detection cell proliferation assay, but CGP used two different cell viability assays, a fluorescent nucleic acids stain and a redox indicator dye to assess drug inhibition (for details see Supplementary Information). A recent study by Chan et al. directly compared cell viability assays based on quantifying total amount of nucleic acid using fluorescent DNA-binding dyes (similar to the SYTO 60 assay used in the CGP study) vs. ATP-dependent luminescence (CellTiter-Glo assay also used in the CCLE study) [25]. The study shows that the ATP-dependent luminescence assay is prone to underestimation of drug potency and efficacy, particularly for DNA synthesis-targeting agents [25]. The ATP-dependent luminescence and fluorescent DNA-binding assays are measuring different aspects of the drug response phenotype, and therefore it is not surprising that the assays show only moderate correlation in our comparative analysis [5]. Given the limitations of each assay, it has been suggested that multi-parameter testing, incorporating complementary cell-viability assays yields the most robust and informative phenotypic measures [23].

Table 1.

Comparison of available pharmacological assays for measuring cell viability.

Cell viability assay	Type of Assay	Pros	Cons	Suitable for HTS	Used in*
Cell Titer Glo (Promega)	Viability, Membrane integrity, ATP	Non-toxic, does not require fixing or washing steps	ATP could be reduced by metabolic interferences, leading to false positive results; underestimation of drug effect by DNA-targeting agents, where cells may be incapacitated but intact	Yes	CCLE, GSK, FIMM, SU2C, NTP
Syto60 (Invitrogen)	Proliferation; Fluorescent DNA stain	Simple	requires fixation, washing; could stain DNA of intact but not-viable cells	Yes	CGP
Bromodeoxyur idine-ELISA (CytoSelect, Cell BioLabs)	Proliferation, DNA synthesis	Only live and replicating cells can incorporate BrDU	Assay is time consuming and cost prohibitive in high throughput format	No
DAPI	Proliferation, nucleic acids stain	Simple	Could stain non viable cells if DNA is intact	Yes	GSK
MTT, MTS, XTT	Viability, Metabolic reduction of tetrazolium dye	Simple colorimetric assay	formazan product insoluble in aqueous media; secondary step required to solubilize it before optical detection	No	NCI60
SRB	Proliferation, anionic general biomass stain	Simple Colorimetric assay; stable for extended periods; differentiates cell kill from growth inhibition	requires fixation, washing steps	Yes	NCI60
Resazurin (Cell Titer-Bue, Promega)	Viability, cell permeable redox indicator	Simple and inexpensive; more sensitive than tetrazolium assays	Potentially cytotoxic; requires long incubation 1–4 h; could underestimate toxic effect	Yes
GF-AFC (Cell Titer-Fluor, Promega)	Viability; penetrates live cells where cytoplasmic cytopeptidase activity releaces AFC that fluoresces; Protease becomes inactive upon cell death	Non-toxic, does not require fixing or washing steps; short incubation time < 1h	potential underestimation of drug effect by DNA-targeting agents, where cells may be incapaciated but intact	Yes
Ruthenium Dye	Proliferation; the fluoresecence of the dye is quenced by oxygen; proliferating cells reduce external oxygen and increase fluorescence	Non-toxic, simple, rapid assay	Narrow dynamic range; signal depends on metabolic activity	Yes
LDH (Cytotox-ONE, Promega)	Cytotoxity; released LDH is measured by enzymatically coupled reagent chemistry	Reagents compatible with viable cells; quick incubation; cost effective	Susceptible to background signal from serum sources of LDH in growth media or from compounds inhibiting LDH activity	Yes

^*CCLE: Cancer Cell Line Encyclopedia (Barretina et al, Nature 2012, 483:603-7); CGP: Cancer Genome Project (Garnett et al, Nature 2012, 483:570-5); FIMM: Institute for Molecular Medicine, Finland (Pemovskaet al, Cancer Discovery 2013, 3:1416-29); GSK: Glaxo Smith Kline Study (Greshock. et al, Cancer Res 2010, 70:3677-86); NCI60: National Cancer Institute (Shoemaker, Nature Rev Cancer 2006, 6:813-23); NTP: National Toxicology Program (Xia et al, Environ Health Perspect 2008, 116:284-91); SU2C: Lawrence Berkeley National Lab (Heiser et al, Genome Biol 2009, 10:R31)

Primary phenotypic responses measured in pharmacogenomic studies include dose response curves and two related measures of drug potency: the half maximal inhibitory concentration (IC₅₀) and the area under the dose response curve (AUC). These measures can be estimated using many different laboratory assays and analytical methods – no standard “best practices” have been shown to be either highly reproducible or generally reflective of the response in humans [26]. The shape of the dose-response curve, and thus the estimates for both AUC and IC₅₀, depends not only on the drug-cell line combination but also on the range of drug concentrations tested, which was not identical in the two studies. More broadly, the physiological definitions of AUC and IC₅₀, originally intended to describe the mass-action law of a single-substrate competitive inhibition enzyme kinetics, [27] might not provide similarly meaningful measures of growth inhibition kinetics in whole cell systems. Cell population heterogeneity and cell-to-cell variability in drug response may affect the shape of the dose response curve, yielding shallow curves with reduced steepness or Hill slope [26]. An independent reanalysis of the two studies found that in the CCLE study a drug could be assessed both as effective or ineffective against a cell line depending on whether the assessment of drug sensitivity was based on an IC₅₀ or AUC [6]. Thus, it may not be a simple task to obtain meaningful quantitative, and perhaps even qualitative, comparisons of responsiveness to many drugs across studies.

Standardization of In Vitro Cell-Based HTS

Reproducibility is a fundamental prerequisite for any scientific study, and this requirement becomes even more critical when experimental results impact medical decision-making. As large panels of genomically-annotated cancer cell lines (such as those in the CCLE and CGP studies) are being used to screen cancer drugs and guide treatment selection for patients whose cancers show similar genomic alterations, it is important to ensure the reproducibility of cell-line based drug sensitivity data from HTS assays. The unexpectedly low concordance in the results of the CCLE and CGP studies also illustrates a broader challenge. Scientifically appealing, novel high throughput analytical methods often become widely used before the factors that affect assay performance are well understood. Due to the substantial experimental costs, limited funding, and lower priority assigned to this type of technology assessment, parameters that influence the results may not be established and standardized until after widespread use of the method.

When DNA microarray analysis was first introduced in the 1990’s there was great excitement about the potential of genome-wide expression profiling. The initial euphoria soon evaporated as many studies failed to be replicated. In 1999, the Microarray Gene Expression Data Society (MGED) was formed to address the need for standardization in microarray experiments and the reporting of experimental data and metadata. The group produced the Minimal Information About a Microarray Experiment (MIAME) standard [28, 29] and, through a series of meetings and workshops, helped to catalyze development of experimental and data analysis standards. Data preprocessing tools such as dChip [30] and RMA [31] provided the means to compare large-scale assays, methods for removing batch effects [32, 33] helped to facilitate meta-analysis across experimental datasets, and approaches to creating robust biomarkers [34–37] have provided a framework to use expression data to make more reliable predictions of disease phenotypes.

It is clear that there is a need for a similar standardization of drug-response measurements in addition to the development of new, robust drug sensitivity assays that can both be replicated across studies and that are physiologically relevant for the treatment of human disease. Lack of standardization may limit the potential of high-throughput drug screening studies for discovering novel drug sensitivities. While gene expression and genome sequencing data may be reliable measures of the individual cell lines being screened, the lack of a robust phenotypic anchor for developing and validating predictive models can limit the use of data from HTS studies. A community-based effort, similarly to that undertaken by MGED, and an investment of resources to support the development of common standards to facilitate comparisons, would pay a huge dividend in establishing high-throughput screening as a valuable tool in drug discovery and could enable a host of robust and reproducible clinical and translational applications.

Community-wide Assessment for Standardization

Given the complexity of HTS assays, a community-wide consortium effort with multiple stakeholders from industry, academia, and government working together may be needed to reach consensus on best practices and strategies for objective quality assessment and inter-site reproducibility of drug sensitivity data. HTS assays are complex, multi-step processes with many options; however, the respective advantages, limitations, and inter-site reproducibility of the alternative protocols have not been adequately evaluated. Noise introduced at any step could seriously limit the robustness of an entire experiment. For the objective assessment of quality control of cell-based HTS assays, a community-wide consortium effort ("HTSQC") similar to the MAQC-I [37] could be invaluable.

We offer some initial ideas for further consideration. A common panel of cancer cell lines whose identity would be verified genomically, and a collection of standardized anticancer compounds could be used as common "reference materials" for assessing the performance of each participating HTS platform and testing laboratory, providing a common ground to support meaningful comparisons. The reference materials (cell lines and compounds) could be maintained under strict quality control and could be used with defined protocols at a centralized location before being distributed to testing laboratories. Specifying defined cultured media and conditions, number of passages, seeding densities, and duration of assays could help establish common assay protocols. Similarly, reagents required for a given HTS platform could be kept as consistent as possible among the testing laboratories.

The same HTS platform could be tested in at least three independent laboratories using the same reference materials and the same lots of reagents with multiple replicates in the same laboratory, allowing both inter- and intra-laboratory reproducibility, and cross-platform comparability to be assessed. All study protocols and experimental details would be recorded and submitted along with raw experimental data to designated data analysis teams for independent analysis, allowing variability among data analysis methods to be assessed.

Finally, members of the consortium should share and discuss the results transparently, and publish the results in a peer-reviewed journal as a consortium consensus. Raw data and experimental details should be deposited in a public repository such as PubChem, to allow the community to replicate, further evaluate, and improve HTS assays and data analysis approaches. Such a consortium effort would provide an estimate of the overall reproducibility of current HTS assays, develop best practices, and provide an upper limit of the robustness of measured drug response phenotypes. If executed properly, this project could provide critical information for industry, academia, and regulatory agencies to objectively assess the implications of the limits of consistency of HTS assay results in drug discovery and development.

Conclusion

While our evaluation suggests caution when considering the data generated from large-scale pharmacogenomic HTS studies, our results do not undermine the value of such studies. Indeed large-scale pharmacogenomic HTS studies hold great potential for generating robust genomic predictors of drug response. However this potential will not be fully realized until improvements are made in the design, application, and community-wide implementation of robust, standardized assays for measuring drug response phenotypes. Efforts should be made to promote transparency and availability of raw assay data for HTS studies. These data will be critical to identify the sources and magnitudes of experimental uncertainties, and for developing methods to reduce their impact. One proven path forward is the establishment of best practices and standardization of the critical steps in these assays through a collective effort. Such an approach could ensure that the data produced from large-scale efforts can be replicated over a broad range of conditions and compared successfully across multiple laboratories.

Definitions (Box).

• Pharmacogenomics: the use of genetic information from populations to inform drug design and development or from individual patients to inform the clinical management of pharmacotherapy, including drug selection, dosing and analysis of drug toxicities. Here this term is used more broadly to include experimentation on model systems.

• (Conventional) High-Throughput Screening: highly automated methodology that heavily relies on robotics, liquid handling devices and automated detection to quickly conduct millions of pharmacological tests, typically on specific protein targets

• in vitro cell-based/pharmacogenomic HTS: extension of conventional HTS to include large panel of cancer cell lines to assess in vitro drug response.

• IC₅₀: half maximal inhibitory concentration as a measure of the effectiveness of a substance in inhibiting a specific biological or biochemical function. This quantitative measure is estimated from the drug dose-response curve.

• AUC: Area under the drug dose-response curve.

• DMSO: Dimethyl sulfoxide is an organosulfur compound with the formula (CH₃)₂SO. This colorless liquid is an important polar aprotic solvent that dissolves both polar and nonpolar compounds and is miscible in a wide range of organic solvents as well as water.