A Proposal Regarding Reporting of in Vitro Testing Results

Abstract

The high rate of negative clinical trials and failed drug development programs calls into question the utility of preclinical testing as currently practiced.

An important issue for the in vitro testing of agents that have advanced into the clinic is the use of clinically irrelevant concentrations in reports making claims for anticancer activity, as illustrated by publications for sorafenib, vorinostat, and metformin. For sorafenib, high protein binding leads to a dichotomy between concentrations active in the 10% serum conditions commonly used for in vitro testing and concentrations active in plasma. Failure to recognize this distinction leads to inappropriate claims of activity for sorafenib based on the micromolar concentrations commonly used for in vitro testing in low serum conditions. For vorinostat and metformin, results using in vitro concentrations higher than those achievable in patients are reported despite the availability of publications describing human pharmacokinetic data for each agent.

We encourage journal editors and reviewers to pay greater attention to clinically relevant concentrations when considering reports that include in vitro testing of agents for which human pharmacokinetic data are available. Steps taken to more carefully scrutinize activity claims based on in vitro results can help direct researchers away from clinically irrelevant lines of research and toward lines of research that are more likely to lead to positive clinical trials and to improved treatments for cancer patients.

Preclinical and translational research, both from academia and from pharmaceutical companies, is under scrutiny. On the one hand, translational research conducted by academic researchers and evaluated by pharmaceutical companies has been found to often lack reproducibility, and claimed conclusions have not been supported by further independent testing (1, 2). At the same time, preclinical testing conducted by pharmaceutical sponsors has been found lacking, as many agents with preclinical packages considered promising have subsequently failed in definitive clinical trials.

There are surely a number of potential explanations for this discouraging perspective on the utility of preclinical testing. In this commentary we address one specific research setting, that of in vitro testing of agents that are advanced in clinical testing and for which clinically relevant drug exposures are established. This is a common setting for academic investigators who use these in vitro results to help support proposals to funding agencies and pharmaceutical companies for clinical trials of these agents. Similar in vitro data are used to support the “repurposing” of non-oncology drugs for cancer indications.

The example of sorafenib highlights some of the key problems related to in vitro testing of anticancer agents that have advanced into clinical testing, and so we use it to illustrate uses and misuses of in vitro testing data. A cottage industry has developed around studying the in vitro effects of sorafenib at micromolar level concentrations, with scores of papers describing its effects against a variety of cancer cell lines at these concentrations and with no indication that an end to such publications is near. There are advantages to studying sorafenib in vitro at micromolar level concentrations, as at these concentrations it consistently kills cancer cell lines and induces a number of biologically interesting effects on signaling pathways associated with cancer. For example, sorafenib at micromolar concentrations reduces phosphorylation levels of eIF4E (3, 4), reduces levels of Mcl-1 (4-6), induces STAT3 activation (7), promotes autophagy (8-10), and inhibits MAP kinase signaling (11). Unfortunately, the effects of sorafenib at these concentrations have little or no relevance to the clinical setting for the reasons summarized in the paragraphs below.

Sorafenib shows remarkable consistency in killing cancer cell lines at concentrations in the 1 to 10 μM range. This activity is unrelated to histotype and spans a wide range of adult and pediatric cancer cell lines, including cell lines for hepatocellular carcinoma, papillary thyroid cancer with BRAF mutations, breast cancer, lung carcinoma, cholangiocarcinoma, melanoma, neuroblastoma, medulloblastoma, rhabdomyosarcoma, and various types of leukemias (see supplemental materials for references). The uniformity of sorafenib cytotoxic effects at micromolar levels is illustrated by the results of the Pediatric Preclinical Testing Program (PPTP), which showed that 21 of 23 cell lines derived from a range of childhood cancers had IC₅₀ values ranging from 3.2 μM to 10 μM (median 4.3 μM), with only a single cell line not achieving IC₅₀ by 10 μM. This uniformity creates a sense of unease, as it is the type of response associated with non-specific cytotoxic agents as well as the type of response observed at supra-physiologic concentrations for many kinase inhibitors. This unease increases with the observation that one cell line tested by the PPTP, an AML cell line with an activating KIT mutation, demonstrated an IC₅₀ of 20 nM. Other cell lines with certain KIT activating mutations are also inhibited by sorafenib in the 100 nM range (12). These observations support the position that the targeted effects of sorafenib occur in the low nanomolar range, while effects in the micromolar range represent non-specific effects on a range of biological processes that lead to cell death.

There is a body of published data documenting that the micromolar concentrations of sorafenib required to kill most cancer cell lines exceed by 2 to 3 logs the concentrations at which sorafenib inhibits cell lines with activated kinases that are its more specific molecular targets. For example, sorafenib inhibits FLT3-ITD autophosphorylation with IC₅₀ values in the low nanomolar range (13, 14), and AML cell lines with FLT3-ITD show low nanomolar IC₅₀ values (14, 15). Cell lines with oncogenic RET show IC₅₀ values in the low to mid nanomolar range (16), and cell lines driven by selected KIT activating mutations show IC₅₀ values < 100 nM (12, 17). Inhibition of VEGFR2 signaling is also observed at sub-micromolar concentrations (11). Cell lines driven by platelet-derived growth factor receptor (PDGFR) signaling have IC₅₀ values in the 10 to 100 nM range (15, 18), with concentrations of 100 nM providing virtually complete inhibition of phospho-PDGFRA and phospho-PDGFRB expression (15, 18). This activity of sorafenib at low nanomolar concentrations translates into tumor regressions in xenograft models with activating mutations in sorafenib target gene products, as exemplified by the high level of in vivo activity of sorafenib against FLT3-ITD AML xenografts (14). By contrast, xenografts lacking activating genomic alterations in these target genes show slowing of tumor growth (consistent with inhibition of VEGFR2 signaling), but not tumor regressions (17).

Importantly, the in vitro activity of sorafenib at these nanomolar level concentrations has clear clinical significance as illustrated by the activity of sorafenib against FLT3-ITD AML (19-22) and its VEGFR2-related anti-angiogenic activity as evidenced by its FDA approval for renal cell carcinoma (a VEGF pathway driven tumor for which VEGFR2-targeted agents show class activity) (23). The rarity of PDGFR-driven leukemias and the effectiveness of imatinib as upfront therapy have limited evaluations of sorafenib against this molecularly defined subset of patients (24, 25), although one patient showed a brief hematologic response prior to outgrowth of a resistant mutation (24).

Conversely, clinical experience with sorafenib supports the lack of positive predictive power for in vitro results obtained at micromolar concentrations. Low to nil objective response rates for sorafenib have been observed for multiple tumor types (most of which have cell lines showing in vitro sensitivity at micromolar concentrations), including: non-small cell lung cancer, breast cancer, melanoma, prostate cancer, squamous cell carcinoma of the head and neck, biliary tract carcinoma, and uterine carcinoma (see supplemental materials for references). Sorafenib response rates for sarcomas were also low, although 5 of 37 patients with angiosarcoma did show responses (26). This tumor type has shown responsiveness to other VEGFR2 targeted agents (27). The paucity of objective responses to sorafenib for most cancers in which the agent has been studied (outside of cancers with the activated kinases described above) despite in vitro cytotoxic activity for sorafenib at micromolar concentrations against cell lines derived from these same cancers supports the lack of clinical relevance for sorafenib in vitro effects observed at micromolar concentrations in 10% serum conditions.

Given the significant clinical activity of sorafenib that can be related to its in vitro activity at nanomolar concentrations against cell lines dependent upon its target kinases and given the otherwise limited clinical activity, what has maintained enthusiasm within the research community for studying sorafenib against cell lines that show no effect until micromolar concentrations are reached? One factor is the clinically achievable sorafenib drug levels, which are indeed in the 10 μM range (28). However, sorafenib is highly protein bound (99.7%) (29), meaning that sorafenib concentrations that are effective in 10% fetal bovine serum (FBS) are ineffective in plasma conditions. For example, FLT3-ITD driven cell lines are inhibited by 50% at 3 nM sorafenib in 10% FBS but require greater than 100-fold higher sorafenib concentrations (approximately 500 nM) in plasma for a comparable level of inhibition (13). Because clinically achievable sorafenib levels are only in the 10 μM range, sorafenib effects observed in vitro in 10% FBS at 1 to 10 μM concentrations lack plausibility for successful clinical translation.

Evaluation of sorafenib at micromolar concentrations is extending into “next generation” in vitro testing. The “CancerCell Line Encyclopedia” project evaluated sorafenib at concentrations ranging from 2.5 nM to 8.0 μM, and 98% of the 504 tested cell lines showed IC₅₀ values > 1 μM (30). For the reasons outlined above, these data at higher sorafenib concentrations lack clinical relevance. Analyses that incorporate these data to identify insights into sorafenib mechanism of action or mechanism of synergy against human cancers will produce misleading findings regarding potential clinical applications of sorafenib.

Disconnects between in vitro observations and clinical reality extend beyond the realm of kinase inhibitors. For example, vorinostat inhibits the growth of most cancer cell lines, but with IC₅₀ values generally exceeding 1 μM (31-33). Concentrations of vorinostat from 2.5 μM to 10.0 μM are commonly used to demonstrate the in vitro effects of vorinostat on cellular processes related to its enhancement of protein acetylation. However, maximum achievable drug concentrations in humans are in the 1-2 μM range at the standard vorinostat dose of 400 mg (34-37). Because of the short half-life of vorinostat (1.5 to 2.0 hours), concentrations > 1 μM are maintained for at most a few hours of each treatment day with concentrations < 0.1 μM for most of each treatment day (36, 38). Thus, the clinical relevance of the large body of literature on vorinostat’s in vitro effects with prolonged exposure to concentrations exceeding 1-2 μM is highly questionable, and indeed 3 or 6 hour exposure to 1-10 μM vorinostat in vitro fails to elicit significant growth inhibition and exposure for 24 or more hours is required for significant effects (39). This skepticism about the clinical relevance of in vitro results using micromolar concentrations of vorinostat is supported by the disconnect between the broad in vitro cytotoxic activity of vorinostat and its narrow range of therapeutic activity in the clinic (primarily cutaneous T-cell lymphoma) (37). In fairness to vorinostat, other HDAC inhibitors show similar relationships between concentrations that are effective in vitro and those studied in the clinic. Panobinostat, which was described as a “broadly active compound” in a report describing the CancerCell Line Encyclopedia, showed an IC₅₀ of approximately 60 nM against the approximately 500 cell lines against which it was tested (30). However, at the commonly used dose and schedule of this agent (20 mg administered orally 2-3 times per week), C_max values are only in the 40- 70 nM range (40-42), and concentrations above 15 nM are maintained for less than 12 hours out of every 48 hour dosing interval (40). Of note, in vitro exposure to panobinostat for 12 hours or less produces limited in vitro effects (39). The lack of clinical relevance for these nanomolar range IC₅₀ values for panobinostat is illustrated by its low single agent activity for patients with multiple myeloma (42), despite myeloma cell lines demonstrating a median IC₅₀ value of approximately 15 nM (30).

In vitro testing results are also being used to support “repurposing” of non-oncology drugs for anticancer indications. Since the observation that the anti-diabetic agent metformin could reduce the overall incidence of cancer in diabetics and reduce the risk of pancreatic and other cancers (43-45), a plethora of reports have appeared demonstrating the biochemical effects of metformin, and effects on inhibition of proliferation, survival, invasion, motility, DNA repair, and synergy with cytotoxic agents in vitro. A review of recent publications through PubMed reveals that the vast majority of these studies used metformin at 1 to 20 mM (usually > 5mM) (see supplemental materials for references). The concern about the value of such in vitro studies is not because metformin binds to plasma proteins (it has very low or negligible binding), but that accumulation in cells is mediated by organic ion transporters (OCTs) SLC22A1, SLC22A2 or SLC22A3) (46, 47). These transporters are highly restricted to expression in liver and kidney, and are dramatically reduced during malignant transformation, even in hepatocellular carcinoma (48). Population pharmacokinetics modeling in humans shows plasma levels of metformin are low with median C_max of approximately 10 μM (CI 95% 2.8-21.6 μM) following an oral dose of 663 mg (49). Some biochemical effects of metformin have been demonstrated at 50 – 100 μM, under conditions of physiologic glucose, in vitro, however these effects are relatively small (50), and even these exposures to metformin exceed those attainable in patients. While it is clear that metformin can retard progression of some cancers in rodents, it is unlikely that the mechanism of action is a consequence of direct effects on tumor cell metabolism. The antitumor effects are more likely via indirect effects on gluconeogenesis or secretion of insulin-like growth factors from liver (51). Thus, biochemical changes, or physiological changes (proliferation/survival) observed in vitro at metformin concentrations in the millimolar range appear irrelevant for understanding the in vivo activity of metformin.

A basic challenge in interpreting in vitro testing results is the difficulty in assessing therapeutic window. The obvious point is that 96 and 386 well plates don’t develop mucositis, neutropenia, or diarrhea, making it virtually impossible to assess whether the concentration of a novel agent showing in vitro activity is clinically relevant. Uniform cytotoxic activity, as shown for sorafenib at micromolar concentrations, can imply that the agent at these concentrations is extraordinarily promising or alternatively that it is a non-discriminant cytotoxic agent at these concentrations with no greater therapeutic relevance than Clorox. Attempts at using non-transformed cell lines as a surrogate for the host have not been effective at assessing therapeutic window. The one consistent in vitro predictive factor for clinical success for a candidate oncologic agent is wide variation in sensitivity across cell lines, with most cell lines requiring high concentrations and with a select few cell lines with distinctive molecular characteristics being sensitive at much lower concentrations. Examples of the successful application of this principle include small molecule kinase inhibitors that show specificity for BRAF V600E mutated cell lines, or ALK-translocated cell lines, or EGFR mutated cell lines, to name a few.

There are costs to publishing results claiming anticancer activity based on clinically unachievable exposures, the most important being the potential for misallocation of clinical research resources and patients to clinical trials with little or no chance of success. Referring back to sorafenib again, over 500 clinical trials for sorafenib are listed in “ClinicalTrials.gov”, and nearly 200 clinical trials are listed as currently active across a wide range of cancer types. While many of these sorafenib trials are for either renal cell carcinoma or hepatocellular carcinoma (cancers for which sorafenib is licensed by regulatory agencies) or for cancers for which an antiangiogenic effect is targeted, other trials are for cancers for which the justification is based in part on sorafenib’s micromolar level in vitro activity. Similarly, approximately 50 cancer-related metformin clinical trials are listed. Would these trials have been initiated if there had been greater scrutiny of the clinical relevance of the drug concentrations used in the in vitro testing experiments that contributed to the rationale for these clinical trials?

What can be done to address the issues described above? One step that journals can take is to reject without review manuscripts that make activity claims for sorafenib at concentrations exceeding 1 μM, metformin at concentrations exceeding 50 μM, and vorinostat at concentrations exceeding 1 μM. More general steps that journal editors and reviewers can take include:

• - Requiring that manuscripts describing the in vitro activity of agents that have advanced into clinical evaluation specifically address and provide convincing rationale with appropriate reference to the published literature for the potential clinical relevance of the concentrations explored.

• - Rejecting manuscripts that make activity claims for targeted agents at concentrations that substantially exceed (e.g., ≥ 10-fold) the concentrations at which the agents are effective against cell lines expressing the agents’ molecular targets.

• - Rejecting manuscripts that make activity claims for agents based on “clinically achievable drug levels”, when the concentrations used in vitro either exceed those observed in humans or else are present for only a short period of each treatment day.

• - Requiring that manuscripts describing the in vitro testing of agents with high protein binding adjust for low serum versus plasma protein concentrations whenever comparisons to clinically achievable concentrations are made.

We recognize that exceptions to the above “rules” will be justifiable in some cases (e.g., for agents that have active metabolites that achieve sufficiently high levels in patients), but authors proposing exceptions should provide clear rationale to justify their position. Importantly, we believe that these proposals, if consistently applied, can help direct investigators away from lines of research that lack clinical relevance and toward lines of research that are more likely to lead to positive clinical trials and to improved treatments for patients with cancer.

Statement of translational relevance.

Preclinical and translational cancer research is under scrutiny because of high rates of negative clinical trials and failed drug development programs. Regarding the in vitro testing of agents that have advanced past phase 1 testing, an important issue is the use of clinically irrelevant concentrations to support lines of clinical research, as illustrated by publications describing preclinical testing results for sorafenib, vorinostat, and metformin. To the extent that these publications provide rationale for clinical trials with an inherently low likelihood of success, policies limiting such publications can contribute by directing investigators away from clinically irrelevant lines of research and toward lines of research that are more likely to lead to positive clinical trials and to improved treatments for patients with cancer.