Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2012 Dec 1.
Published in final edited form as: Mol Cancer Ther. 2011 Oct 12;10(12):2350–2362. doi: 10.1158/1535-7163.MCT-11-0497

Dual inhibition of Tumor Energy Pathway by 2-deoxy glucose and metformin Is Effective Against a Broad Spectrum of Preclinical Cancer Models

Jae-Ho Cheong 1,6,*, Eun Sung Park 1,6, Jiyong Liang 1, Jennifer B Dennison 1, Dimitra Tsavachidou 2, Catherine Nguyen-Charles 1, Kwai Wa Cheng 1, Hassan Hall 1, Dong Zhang 1, Yiling Lu 1, Murali Ravoori 3, Vikas Kundra 3, Jaffer Ajani 4, Ju-Seog Lee 1, Waun Ki Hong 5, Gordon B Mills 1
PMCID: PMC3237863  NIHMSID: NIHMS331271  PMID: 21992792

Abstract

Tumor cell proliferation requires both growth signals and sufficient cellular bioenergetics.The AMP-activated kinase (AMPK) pathway appears dominant over the oncogenic signaling pathway suppressing cell proliferation. This study investigated the preclinical efficacy of targeting the tumor bioenergetic pathway using a glycolysis inhibitor 2-deoxy glucose (2DG) and AMPK agonists, AICAR and metformin. We evaluated the in vitro anti-tumor activity of 2DG, metformin or AICAR alone, and 2DG in combination either with metformin or AICAR. We examined in vivo efficacy using xenograft mouse models. 2DG alone was not sufficient to promote tumor cell death, reflecting the limited efficacy demonstrated in clinical trials. A combined use of 2DG and AICAR also failed to induce cell death. However, 2DG and metformin led to significant cell death associated with decrease in cellular ATP, prolonged activation of AMPK, and sustained autophagy. Gene expression analysis and functional assays revealed that the selective AMPK agonist AICAR augments mitochondrial energy transduction (OXPHOS) while metformin compromises OXPHOS. Importantly, forced energy restoration with methylpyruvate reversed the cell death induced by 2DG and metformin, suggesting a critical role of energetic deprivation in the underlying mechanism of cell death. The combination of 2DG and metformin inhibited tumor growth in mouse xenograft models. Deprivation of tumor bioenergetics by dual inhibition of energy pathways might be an effective novel therapeutic approach for a broad spectrum of human tumors.

Keywords: Tumor bioenergetics, Targeted therapy, Cancer energy metabolic pathway

Introduction

Proliferation of cancer cells requires oncogenic growth signals as well as sufficient metabolic energy for biogenesis of cellular constituents. The “Warburg effect” (1), a metabolic derangement in cancer cells resulting in increased glucose uptake and glycolysis, provides a selective advantage to rapidly proliferating tumor cells by supplying cellular bioenergetics required to support tumor progression. Clinically, tumors with high glucose uptake detected by the FDG-PET scan demonstrate a worsened outcome (2-4), underscoring the therapeutic potential of inhibition of glycolysis.

The mammalian target of rapamycin complex 1 (mTORC1) is a key downstream effector of the phosphatidylinositol 3-kinase (PI3K)-AKT signaling pathway (5) that regulates and promotes cell proliferation, growth, and survival in many cancer lineages (6-8). The mTORC1 complex integrates oncogenic growth signaling with the glycolytic switch (9). The growth promoting effects of mTORC1 including protein synthesis and cell cycle progression are highly energy consuming (10, 11) suggesting that mTORC1 activity would be dependent on cellular energy status. Indeed, mTORC1 is directly inhibited by phosphorylation of raptor as a consequence of activation of the AMP-activated protein kinase (AMPK), a key cellular energy sensor which is activated by bioenergetic stress (12-15).

Remarkably, the energy sensing AMPK pathway appears dominant over the growth promoting effects of the PI3K-AKT pathway determining cellular functional outcomes. This ensures that proliferation does not occur in the absence of adequate nutrients, energy and cellular building blocks, providing a novel “bioenergetics checkpoint”. Therefore, manipulation of the AMPK signaling pathway, thus mimicking energy stress, could represent a therapeutic target potentially overriding the oncogenic effects of the PI3K-AKT-mTORC1 pathway.

Herein, we describe the functional outcomes of glycolysis inhibition by 2DG in combination with AMPK agonists, AICAR and metformin. Unexpectedly, we found that combined use of 2DG and metformin, but not 2DG and AICAR, exhibits significant anti-tumor effects in vitro and in vivo. Gene expression analysis and subsequent functional assays suggest that AMPK activation protects tumor cells in energetically stressed conditions by augmenting mitochondrial respiration. Metformin, unlike AMP-mimetic AICAR, inhibits mitochondrial energy generation thereby explaining the observed anti-tumor effect in combination with 2DG. Finally, the in vivo efficacy in mouse xenograft models provides the rationale for the clinical evaluation of this novel strategy for the treatment of cancer patients.

Materials and Methods

Cell culture

Human gastric and esophageal cancer cell lines p-SK4 and OE33 were kindly provided in June 2006 by Dr. Julie Izzo (The University of Texas MD Anderson Cancer Center) and cultured in DMEM/F12 50:50 supplemented with 10% FBS in a humidified incubator containing 5% CO2 at 37°C. U2OS, MCF-7, MDA-MB-468, MDA-MB-231 and MCF10A were obtained in May 2007 from the American Type Culture Collection (ATCC) and grown in medium RPMI-1640 with 5% FBS. The identities of all cell lines were validated by STR DNA fingerprinting using the AmpF_STR Identifiler kit according to manufacturer’s instructions (Applied Biosystems, Foster City, CA cat 4322288) at Characterized Cell Line Core Facility (All the cells were last tested in October 2009). The STR profiles were compared to known ATCC fingerprints (ATCC.org), and to the Cell Line Integrated Molecular Authentication database (CLIMA) version 0.1.200808 (http://bioinformatics.istge.it/clima/) (Nucleic Acids Research 37:D925-D932 PMCID: PMC2686526). The STR profiles matched known DNA fingerprints or were unique.

Cell viability assay

Cell viability was determined by Trypan blue dye exclusion. For the assay, 0.3 × 106 cells were plated in 6-well plates and treated the next day. Methyl pyruvate (MP, 10mM) was added 2 h before treatment where indicated. Cells were trypsinized, resuspended and mixed with 1:1 0.4% trypan blue. Percentage cell death = No. of stained cells / (No. of stained + unstained cells) × 100.

Reverse phase protein array (RPPA)

RPPA was processed as previously described (16, 17).Briefly, serially diluted lysates were spotted on FAST slides (Schleicher & Schuell BioSciences, Keene, NH) using a robotic GeneTAC arrayer (Genomic Solutions, Ann Arbor, MI). After printing, slides were blotted sequentially with Re-Blot (Chemicon, Temecula, CA), I-Block and biotin blocking system (Dako, Carpinteria, CA), probed with primary antibodies and incubated with biotin-conjugated secondary antibodies. The signals were then amplified using a catalyzed signal amplification kit (DakoCytomation, Carpinteria, CA) according to the manufacturer’s instructions. The processed slides were scanned and quantified using MicroVigene software (VigeneTech Inc., North Billerica, MA).

Measurement of intracellular ATP levels and mitochondrial transmembrane potential (ΔΨm)

Intracellular ATP was measured using a luciferin/luciferase-based assay. Cells were grown under each experimental condition for indicated times, harvested, and counted. Aliquots containing equal number of cells were processed following manufacturer’s guidelines (Roche).

Rhodamine-123, a cationic voltage-sensitive mitochondrial probe, was used to detect changes in mitochondrial transmembrane potential (ΔΨm). Cells were incubated as indicated and labeled with 1μM rhodamine-123 at 37°C for 30 min. After washing, the samples were analyzed by flow cytometry.

Immunoblotting

Cell lysis and immunoblotting were performed as previously described (18). A total of 50μg protein was used for the immunoblotting, unless otherwise indicated. β-actin or GAPDH were used as loading controls. Anti-LC3 antibody was a gift from Dr. S. Kondo. All other antibodies were purchased from Cell Signaling.

Transmission electron microscopy

Samples were fixed with a solution containing 3% glutaraldehyde plus 2% paraformaldehyde in 0.1 M cacodylate buffer, pH 7.3, for 1 hour. After fixation, samples were washed and treated with 0.1% Millipore-filtered cacodylate buffered tannic acid, postfixed with 1% buffered osmium tetroxide for 30 min, and stained en bloc with 1% Millipore-filtered uranyl acetate. Samples were dehydrated in increasing concentrations of ethanol, infiltrated, and embedded in LX-112 medium. Samples were polymerized in a 70 °C oven for 2 days. Ultrathin sections were cut in a Leica Ultracut microtome (Leica, Deerfield, IL), stained with uranyl acetate and lead citrate in a Leica EM stainer, and examined in a JEM 1010 transmission electron microscope (JEOL, USA, Inc., Peabody, MA) at an accelerating voltage of 80kV. Digital images were obtained using AMT Imaging System (Advanced Microscopy Techniques Corp, Danvers, MA).

Animal protocol

Athymic female nude mice (CD-1 nu/nu from the NIH National Cancer Institute repository) were used for in vivo tumor growth studies. All of the in vivo studies were carried out under approved institutional experimental animal care and use protocols. For the breast tumor orthotopic model, MDA-MD-231 cells were resuspended at 6.5 × 106 cells/200 μL in PBS and injected into the seventh mammary gland fat pad of 4 week old athymic nude mice. When tumors were approximately 50mm3 in size (14 days), the animals were randomly allocated into four groups. Following randomization, vehicle, 0.2ml of 2DG (500mg/Kg), metformin (250mg/Kg, equivalent to a human dose of 20mg/Kg by normalization to surface area) or both were delivered via intraperitoneal administration (IP) daily for the duration of the experiment. For the esophago-gastric cancer xenograft model, nude mice received subcutaneous (s.c.) implantation with 200μL of the s-SK4 (selected from p-SK4 cells) at 4×106 cells/200 μL in PBS. When tumors were approximately 100mm3 in size (one week later), the animals were randomized. 0.2ml of PBS or 2DG (500mg/Kg) + metformin (250mg/Kg) were delivered via IP administration, daily for the duration of the experiment. Magnetic resonance Image (MRI) was performed and tumor volumes in vivo were assessed.

Tumor size were measured using digital calipers two to three times weekly. Tumor measurements were converted to tumor volume using the formula L × S2/2 (where L; longest diameter, S; shortest diameter). Mice were killed when either L or S exceeded 15 mm in control group. At sacrifice, tumors were excised and assessed histologically for verification of tumor growth. Statistical significance was determined using Student t-test.

Microarray experiment and data analysis

Total RNA was isolated from cells harvested after each treatment by using mirVanaTM miRNA Isolation Kit (Ambion Inc., Austin, TX) according to manufacturer’s protocol. Biotin labeled cRNA samples were prepared by using the Illumina TotalPrep RNA Ampliification Kit (Ambion Inc., Austin, TX). 500ng of total RNA was used for the synthesis of cDNA and then followed by amplification and biotin labeling as recommended by the manufacturer. 1.5μg of biotinylated cRNA per sample was hybridized to Illumina Human-6 BeadChip v.2 microarray and signals were developed by Amersham fluorolink streptavidin-Cy3 (GE Healthcare Bio-Sciences, Little Chalfont, UK). Data were analyzed by using Illumina Bead Studio v3.0 after scanning with Illumina bead Array Reader confocal scanner (BeadStation 500GXDW; Illumina Inc.) All statistical analysis was performed using R 2.3.0 and BRB Arraytools Version 3.5 (http://linus.nci.nih.gov./BRB-ArrayTools.html).To minimize the effect of variation from non-biological factors, the values of each sample were normalized using a quantile normalization method. Random-variance t-test was applied for the calculation of significance of each gene in the comparison of two classes and one-way ANOVA was applied for the evaluation of significance in multi group comparison. Cluster analysis was performed with Cluster and Treeview (http://rana.lbl.gov/EigenSoftware.htm). For cluster analysis, log base 2 transformed data were centered to mean values of each gene’s expression. Gene Set Enrichment Analysis was performed against GO (Gene Ontology) of Biological Process and Kolmogorov-Smirnov statistic was applied for the evaluation of statistical significance of each GO category.

Results

Inhibition of glycolysis by 2DG is not sufficient to activate AMPK or promote cell death

To investigate the consequences of metabolic stress on key signaling pathways, a panel of cancer cell lines of multiple lineages was subjected to glucose deprivation and combinations of 2DG and known AMPK activators (AICAR and metformin). The cell lysates were subject to multiplexed reverse phase protein arrays (RPPA) (16, 17) to evaluate multiple protein expression levels simultaneously (Fig. 1A and B, Supplementary Fig. S1).

Figure 1. AICAR and the combination of 2DG and metformin activate AMPK and reduce phosphorylation of mTORC1-regulated proteins.

Figure 1

Open in a new tab

A and B, Protein levels of lysates from multiple cell lines were quantified by RPPA after treatment with the indicated compounds for 24 h. Phosphorylations of S6 and ACC were quantified to evaluate signaling changes downstream of mTORC1 and AMPK, respectively. Values were compared to the untreated controls (broken lines) using a two-tailed t-test. * p<0.05, **p<0.01, ***p<0.001 C, Western blot analysis of p-SK4 cells lysate.

Incubation in glucose-free medium consistently inactivated PI3K-AKT-mTORC1 downstream signals (p-p70S6K, p-S6, and cyclin D1) and reduced expression of cell cycle regulators (cyclin D1, cyclin B1, p-Rb) (P≤0.01, Supplementary Fig. S1). p-S6 levels were markedly reduced by glucose deprivation representing the most sensitive biomarker (85% decrease, Fig. 1A). Interestingly, p-AMPK and p-ACC levels, markers of cellular bioenergetic stress, were essentially unchanged at 24 h despite glucose deprivation or the addition of 2DG, suggesting that the cells were able to compensate for the loss of glucose-derived ATP production (Fig. 1B and C).

Of note, metformin which is known to activate AMPK did not induce AMPK activation or subsequent phosphorylation of ACC and Raptor when high concentrations of glucose (25mM) were present (Fig. 1C). In contrast, the combination of 2DG and metformin markedly activated AMPK and subsequently deactivated mTORC1 downstream signaling. These coordinated alterations were compatible with activation of the TSC2 by AMPK resulting in decreased signaling through mTORC1 and downstream targets (19-22). The AMP-mimetic AICAR, however, markedly activated AMPK and inhibited mTORC1 downstream effectors even in the presence of 25mM glucose and these effects were not augmented when combined with 2DG. These results suggest that the pharmaco-biological outcomes between metformin and AICAR would be different when high level of glucose is present or glucose is deprived.

Based on these results, we predicted that inhibition of glycolysis alone would not be sufficient to promote bioenergetic stress. Indeed, clinically achievable concentrations of 3~4 mM 2DG (23) did not substantially activate AMPK and affect cell signaling in the majority of the cell lines assessed (Fig. 1B). Further, inhibition of glycolysis with 2DG at clinically achievable doses was not sufficient to induce a significant degree of cell death, even with prolonged incubation (Fig. 2A and B).

Figure 2. 2DG combined with metformin selectively promotes death of cancer cells.

Figure 2

Open in a new tab

Cell viability was assessed by trypan blue exclusion.Data are the mean ± S.D. (n=3) of at least 100 cells from one representative experiment of at least four independent experiments. A, p-SK4 cells were treated with indicated agents and viability was assessed. B, Viability of breast (MDA-231 and MCF-7) and osteosarcoma (U2OS) cell lines was assessed at 72 hours. C, p-SK4 cells were treated with 2DG+Met or 2DG +AICAR for the indicated times and representative photomicrographs taken. Scale bar, 100μm. D, GFP-LC3 transfected U2OS cells were incubated for 72h with the indicated compounds (Upper panel, fluorescence analysis; lower panel, bright fields; Scale bars, 100μm).

Metformin but not AICAR induces cell death in combination with 2DG

Based on the alterations in signaling pathways (Fig. 1C), we next compared the effects of each agent on tumor cell viability. We hypothesized that the AMP-mimetic AICAR, 2DG+metformin, and 2DG+AICAR should reduce cell viability based on the substantial changes in AMPK and mTORC1 downstream signals (Fig. 1C).

Surprisingly, neither AICAR nor 2DG+AICAR induced cell death while the 2DG+Met significantly reduced cell viability in a time-dependent manner (Fig. 2A and C). Further, the combination effect of 2DG and metformin resulted in a marked increase in cell death compared to either agent alone (Fig. 2A, B and D). Unexpectedly, AICAR as a single agent or in combination with 2DG failed to mimic the effects of metformin in combination with 2DG on cell death (Fig. 2A-D). This suggests that activation of AMPK was not sufficient to induce cancer cell death irrespective of the presence of 2DG. Instead, it suggests that enhanced cell death observed with metformin in combination with 2DG might be caused by AMPK-independent mechanisms or through mechanisms in addition to AMPK activation.

Metformin and AICAR induce markedly different gene sets controlling mitochondrial energy transduction

Although metformin and AICAR have been reported to activate AMPK (24-29), our data indicated that the administration of each alone or in combination with 2DG leads to markedly different biological outcomes. Further, when glucose was sufficient, metformin did not substantially activate AMPK at least in the cell lines tested. To explore potential mechanisms underlying the ability of metformin, but not AICAR, to induce cell death when combined with 2DG in cancer cells, we performed genome-wide transcriptional profile analysis. Gene expression patterns of each treatment group were dramatically different (Fig. 3A; 2527 significant genes; P<10−6 one way-ANOVA analysis). Further, while 2DG and control groups clustered closely together, metformin and AICAR demonstrated distinct and distant clusters independent of the presence of 2DG, indicative of markedly different mechanisms of action (Fig. 3A).

Figure 3. AICAR and metformin effects on gene expression are distinct and opposing for genes related to the mitochondrial energy transduction.

Figure 3

Open in a new tab

p-SK4 cells were treated with 2DG (4mM), metformin (5mM), AICAR (2mM), 2DG+Met, or 2DG+AICAR for 12 hours. A, Class comparisons were performed between the six treatment groups using genes filtered with one-way ANOVA. Clustering analysis of genes showed significant differences between AICAR and metformin treatment (2527 significant genes, p-value < 10−6). B,C,D and E, Supervised clustering analyses were performed for genes related to the mitochondrial energy transduction, gluconeogensis, and the TCA cycle.* p<0.05, **p<0.01, ***p<0.001. F, TEM of p-SK4 cells after incubation with indicated treatments for 12 hr (Upper panel) and 24 hr (Lower panel). Normal (arrowhead) and abnormally shrunken electron dense mitochondria (arrow) are indicated. Scale bar, 500nm. G, Mitochondrial transmembrane potential (ΔΨm) of p-SK4 cells incubated with indicated compounds (*p<0.05 compared to control, 2DG, AICAR, or 2DG+AICAR). H, Intracellular ATP levels were measured at each time point after treatment of indicated agents with a luciferase-based assay. Values were normalized based on cell numbers. Data are mean ± S.D. (n=3) of one representative of at least three independent experiments.

To estimate thefunctional and biological consequences of differences in gene expression between the treatments, weperformed gene set enrichment analysis (GSEA, Kolmogorov-Smirnov (KS) permutation test, p-value <0.001) against the biological process (BP) category of gene ontology (GO). Of note, metformin in the presence or absence of 2DG significantly downregulated expression of 12 components of the electron transfer chain complex 1 (ETC1) in the mitochondria without significantly altering expression of other components of the electron transfer chain complexes (Supplementary Table S1 and Fig. 3B). Other components of the ETC or TCA cycle were unaffected (Supplementary Tables S1, 2 and Fig. 3C). This selective effect of metformin on ETC1 could limit the extraction of electrons from NADH thereby suppressing the mitochondrial ATP-generating capacity.

In contrast, AICAR alone or in combination with 2DG significantly induced expression ofmultiple genes in the TCA cycle and the ETC complex I, II, III, and IV and F0F1-ATPases that could coordinately increase the efficiency of OXPHOS (Supplementary Table S1 and Fig. 3D and E). Further, expression of a set of genes involved in gluconeogenesis and vacuolar type ATPases that consume ATP was concomitantly decreased (Fig. 3D and E, Supplementary Table S2). This suggests that activation of AMPK by an AMP-mimeticdrug may increase mitochondrial ATP production by augmenting the efficiency of the TCA cycle, ETC, and OXPHOS, enabling cells to maintain bioenergetic homeostasis.

Metformin and AICAR exert opposite effects on mitochondrial energy generation

The disparate effects of AICAR and metformin on expression of components of the mitochondrial respiratory chain suggested that they would have differential effects on mitochondrial energy generation capability such as transmembrane potential (ΔΨm), the proton motive force that drives mitochondrial ATP production (30, 31). Consistent with the mRNA microarray data (Fig. 3D and E), AICAR alone or combined with 2DG increased mitochondrial function as shown by a significant increase in ΔΨm (Fig. 3G). In contrast, metformin in the presence or absence of 2DG induced a time-dependent decrease in ΔΨm (Fig. 3G) and mitochondrial shrinking with acquisition of an electron-dense matrix (Fig. 3F). Similar to the effects of metformin, accumulation of mitochondrial electron-dense deposits has been previously reported after a decrease in ETC complex I and III functions (32).

Consistent with the alterations in mitochondrial ΔΨm, we found that 2DG induced a modest time-dependent decrease in cellular ATP levels (Fig. 3H). As predicted, metformin induced a decrease in cellular ATP levels whereas AICAR did not significantly alter cellular ATP levels (Fig. 3H). The combination of 2DG and metformin reduced cellular ATP by 70% suggesting that decreasing ATP levels below a critical threshold could result in the cell death associated with 2DG and metformin (Fig. 2A and B). Strikingly, AICAR significantly reversed the effects of 2DG on ATP levels at delayed time points (48h), compatible with the changes in gene expression and augmented mitochondrial function described above.

Methylpyruvate rescues energetic stress-induced cell death by metformin plus 2DG

Because energetic stress activates AMPK and suppresses mTORC1 thereby inducing autophagy (19), we examined cellular ATP levels, AMPK and mTORC1 signals, and autophagy over time of treatment with metformin and 2DG. The combination of metformin and 2DG induced progressive depletion of ATP (Fig. 4A) and increased LC3-II levels indicative of sustained autophagy (Fig. 4B) compatible with the prolonged activation of AMPK and suppression of mTORC1 signals (Fig. 4C). These changes were reflected by a time-dependent decrease in cell size, an accumulation of autophagocytic vesicles and electron dense bodies, and a lack of characteristics typical of apoptosis or necrosis (Fig. 4D and E).

Figure 4. Prolonged incubation with 2DG and metformin depletes cellular ATP, activates AMPK, suppresses mTORC1 downstream signaling, and sustains autophagy.

Figure 4

Open in a new tab

p-SK4 cells were incubated with 2DG and metformin for the indicated times. A, Cellular ATP levels determined. Data are the mean ± S.D. (n=3) B, Western blotting for LC3 expression. β-actin was used as loading control. C, Kinetics of signaling molecules in AMPK and mTORC1 downstream. Western blots of samples from 4B. D, TEM analysis of morphologic alteration of cells treated with 2DG in combination with metformin. Severe cytoplasmic contraction and resultant membrane blebbing is evident (48 and 60h). Note most of the cytoplasm is replaced with autophagosomes (60h). Neither typical apoptotic nor necrotic cells are evident (N, nucleus; arrowheads, autophagosome; Scale bar, 10μm for upper panel; 500nm for lower panel). E, Autophagic vesicles were counted for three randomly selected cells from each electron microscopy section. Data presented mean ± S.D.

Metformin compromises but does not completely block mitochondrial ATP production, because it selectively inhibits ETC1 while leaving ETC complex II intact (33, 34). Thus, administering high levels of a substrate that can provide electrons to the ETC complex II should reverse, at least in part, the effects of metformin. We exploited methylpyruvate (MP), a growth factor independent exogenous cell permeable energy substrate that bypasses glycolysis and directly enters the mitochondrial TCA cycle (35-37). In theory, MP should lead to the production of FADH2 in the TCA cycle with an electron being extracted from complex II and transferred in the ETC, thus producing ATP even when ETC1 is compromised by metformin. Indeed, MP increased intracellular ATP levels in cells treated with 2DG and metformin, (Fig. 5A), indicating a partial bypass of the effects of metformin. Compatible with the increase in ATP levels, MP markedly suppressed the increase in phosphorylation of AMPK and ACC induced by 2DG and metformin (Fig. 5B), indicating that AMPK activation by 2DG and metformin is likely a consequence of a compromise in energy production rather than due to a direct effect of either compound on AMPK activity. In agreement with the ability of MP to restore cellular ATP levels and attenuate AMPK activity, MP reduced the formation of GFP-LC3 punctate vesicles as well as the number of autophagosomes (Fig. 5C and D). Consistent with the effects on GFP-LC3, there was a reduced accumulation of LC3-II on western blotting in cells treated with MP (Fig. 5E). Importantly, MP decreased cell death induced by 2DG plus metformin (Fig. 5F). The ability of MP to attenuate the effects of 2DG plus metformin were confirmed in U2OS cells in terms of autophagy, cellular morphology, cell death,and ATP levels (Supplementary Fig. S2). Thus, the ability of the combination of 2DG and metformin to induce autophagy, AMPK activation, and cell death can be attributed to decreased energy production due to a partial but coordinated compromise of glycolysis and mitochondrial function.

Figure 5. Exogenous energy substrate methyl pyruvate (MP) increases cellular ATP, decreases AMPK activation, and reduces autophagy and cell death.

Figure 5

Open in a new tab

p-SK4 cells were treated with 2DG (4mM) and metformin (5mM) with and without methylpyruvate (MP, 10mM). A, Intracellular ATP levels were measured at 72 hours *p<0.01. B, Western blotting (lysates from 5A) to assess AMPK activation. β-actin was used as loading control. C, Autophagy analsysis by fluorescence microscopy of GFP-LC3 punctate patterns (upper panel) and TEM (lower panel). (N, nucleus; arrowheads, autophagosome; Scale bar, 10μm for upper panel; 500nm for lower panel). D, Quantitation of autophagosomes on TEM. Autophagic vesicles were counted for three randomly selected cells from each electron microscopy section. Data presented mean ± S.D. *p<0.01. E, Cell lysates (samples from5A) were subject to western blotting to assess LC3 expression and lipidation (LC3-II), an autophagy marker. F, Cell death assay by trypan blue exclusion. *p<0.05. All assays were performed at 72 hours except 6C, which was 60 hours. Data are mean ± S.D. (n=3).

Metforminplus 2DG treatment inhibits tumor growth and metastasis in xenograft tumor models

We next examined the in vivo efficacy of 2DG and metformin by using two different xenograft mouse models. First, MDA-MB-231 cells were injected into the mammary fat pad of nude mice and allowed to establish measurable tumors for 14 days. Mice were randomized to different treatment groups and treated intraperitoneally (IP) daily for 36 days at which time the experiment was terminated due to excessive tumor volume in control mice. As indicated in Figure 6A, 2DG or metformin, at doses delivered, did not alter tumor growth. In contrast, the 2DG and metformin combination significantly decreased tumor growth from day 21 to the termination of the study at day 36 (Fig. 6B). Secondly, using a tumorigenic line selected from p-SK4 cells (s-SK4), we determined whether the effect of 2DG and metformin on in vivo tumor growth of MDA-MB-231 cells could be generalized to other lineages. Two weeks of treatment with metformin and 2DG decreased subcutaneous (SC) tumor growth of s-SK4 (Fig. 6C). The decrease in tumor volume was confirmed by magnetic resonance imaging (MRI) during the course of treatment (See Figure 6, D for a representative pair). As with the MDA MB 231 model, the decrease in tumor growth induced by 2DG and metformin was confirmed by weighing the tumor at the termination of the study (Fig. 6E). Notably, immunohistochemical analysis of the tumor sections revealed substantial decreases in p-S6 and cyclin D1 expression levels, recapitulating the effects of metformin and 2DG observed in vitro (Fig. 6F).

Figure 6. Effects of 2DG and metformin on xenograft tumor growth.

Figure 6

Open in a new tab

A, Tumors were established in athymic nude female mice by mammary fat pad injection of 6.5×106 MDA-MB-231 cells and treated as described in Methods. The mean tumor volume ± S.D. is presented. * p<0.05 compared to control, 2DG alone or metformin alone. B, Ex vivo tumor weight of tumors from 6 A. * p<0.05, ** p<0.01 and *** p=0.01. C, Tumors were established in athymic nude female mice by subcutaneous injection of 4×106 s-SK4 cells and treated as described in Methods. Mean tumor volume ± S.D is presented. * p<0.05. D, Representative MR images. E, Tumor weight by 2DG+metformin as assessed by in vivo MRI and excised tumors. * p < 0.05. F, Immunohistochemical analysis of p-S6 and cyclin D1 levels in paraffin embedded xenograft tumor tissues. Scale bar, 100μm.

DISCUSSION

The Warburg effect is an attractive therapeutic target because many cancers are highly glycolytic and is associated with tumor aggressiveness and clinical outcomes. Despite exciting preclinical studies, however, 2DG failed to fulfill its expectations in clinical trials. Likewise in our studies, 2DG at clinically achievable doses did not substantially activate AMPK reflecting insufficient suppression of tumor bioenergetics.These effects were inadequate to induce death of most cancer cell lines assessed or to inhibit tumor growth in vivo. This prompted us to explore whether the activation of AMPK, thus mimicking energy stress, would lead to increases in cell death. We therefore used metformin and AICAR, pharmacological mediators that have been reported to activate AMPK (24-29).

Activation of AMPK has been reported to lead to suppression of mTORC1 signaling (10, 11), cell cycle exit due to stabilization of p27KIP1(18), and a decrease in energy consumption through diminishing protein and lipid biosynthesis (13-15, 38). Our study suggests an additional role for AMPK activation in tumor bioenergetics. Direct activation of AMPK by the AMP-mimetic AICAR decreased expression of mRNA for enzymes involved in gluconeogenesis, induced expression of mRNA for multiple proteins involved in the ETC and OXPHOS, and increased mitochondrial respiration. This suggests that AMPK induces a switch from dependence on glycolysis for ATP production to energy transduction through mitochondrial respiration. In the current study, we directly demonstrated a role for AMPK in the ability to bypass effects of inhibition of glycolysis; MEFs lacking AMPK were exquisitely sensitive to the effects of glucose deprivation as well as to 2DG (Supplementary Fig. S3A and B). Thus, in the context of energy stress, activation of AMPK could limit the effects of glycolysis inhibitors, suggesting a novel therapeutic combination would be required to override the energetic protective effect of AMPK.

In the absence of glycolysis, mitochondrial energy production is the only potential cellular source of ATP. As described above, inhibition of glycolysis and subsequent activation of AMPK results in a coordinated increase in mRNA levels of genes involved in mitochondrial energy production, potentially triggering a switch to dependence on OXPHOS. Metformin, an orally available anti-diabetic drug, has been proposed to inhibit complex I of the respiratory chain in the mitochondria (33, 34). Indeed, our data demonstrate that metformin coordinately decreases mRNA levels for ETC1 components and mitochondrial membrane potential compatible with compromising mitochondrial energy transduction. While metformin has been known to activate AMPK in intact cells (39, 40), this could be a consequence of its inhibitory effects on mitochondrial ATP production as demonstrated here. Further, the ability of metformin and other biguanide analogs such as phenformin to induce lactic acidosis is compatible with inhibition of mitochondrial function and consequent production of lactate due to elevated levels of pyruvate.

In light of this, we assessed whether inhibition of glycolysis with 2DG combined with a compromise of mitochondrial function with the clinically applicable metformin would result in cell death. Strikingly, in 13 of the 15 cancer cell lines assessed regardless of mutational status, the combination of 2DG and metformin was more efficacious at inhibiting cell growth and inducing cell death than either drug alone (Data not shown).

Recently, Ben Sahra et al. reported that the combination of metformin and 2DG induced p53-dependent apoptosis in prostate cancer cells (41). In their study, apoptosis was AMPK-mediated and p53 was required. In contrast, our results demonstrate that AMPK itself prevents cell death during glucose deprivation or 2DG treatment (See Supplementary Fig. S3A and B). Further, AICAR combined with 2DG failed to induce cell death in a number of cancer cell lines, suggesting a critical role of AMPK as a guardian of cellular bioenergetics. Although, the distinct results between the two studies may be due to the different tissue of origin or characteristics of cell lines used, our data suggest that prolonged activation of AMPK by 2DG and metformin might reflect sustained bioenergetics stress due to failure of mitochondrial compensation rather than direct activation of AMPK.

Consistent with the ability of the combination of 2DG and metformin to induce cell death in vitro, the combination significantly suppressed tumor growth in two xenograft models in vivo. These results suggest that the tumor cell bioenergetics can be targeted, and the combination of 2DG and metformin warrants further clinical evaluation. Recently, metformin has gained much attention due to its anti-tumor activity in some cancer types in preclinical assays (41, 42) with AMPK activation being suggested as a major mechanism of action. As our data indicate, however, AMPK activation can promote cancer cell survival in energetically stressed conditions. A recent study showing that metformin can exert its inhibitory action directly on mTORC1 (43) further reinforces the notion that the effects of metformin can be mediated independently of AMPK.

In summary, realizing the clinical benefit of blockade of the Warburg effect may require concomitant inhibition of multiple components of cellular energy pathways. A preemptive blockade of the Warburg effect and compensatory mechanisms may prove to be dominant over the survival and growth promoting effects of growth factors or activated oncogenes. Since 2DG and metformin are already widely used in PET scan or seizure disorders and in type II diabetes, respectively, the expedited assessment of clinical effect of deprivation of cancer bioenergetics could be immediately tested in cancer patients.

Supplementary Material

1

ACKNOWLEDGMENTS

J-H.C., as an Odyssey Fellow, is supported by the Odyssey Program of the Theodore N. Law Endowment for Scientific Achievement at the MD Anderson Cancer Center. J.B.D. is supported by a TRIUMPH GlaxoSmithKline and the American Cancer Society Joe and Jessie Crump Biomedical Research postdoctoral fellowships at the MD Anderson Cancer Center. We thank Dr. Seiji Kondo and Keishi Fujiwara for critical discussion, Doris Siwak and Qinghua Yu for assistance with RPPA, Dr. Peng Huang and Zhao Chen for the mitochondrial transmembrane potential analysis, Kenneth Dunner, Jr., for conducting electron microscopy, and Maurice J. Dufilho IV for assistance with mouse necropsy. STR DNA fingerprinting was done by the Cancer Center Support grant funded Characterized Cell Line core, NCI # CA16672.

Financial support: This work was supported by National Institutes of Health SPORE (P50-CA83639), PO1 CA64602, PO1 CA099031, CCSG grant P30 CA16672, and DAMD (17-02-01-0694) to G.B.M. and Institutional Core Grant #CA16672 High Resolution Electron Microscopy Facility, UTMDACC.

Abbreviations

AMPK

AMP-activated Kinase

2DG

2-deoxy-glucose

AICAR

5-Aminoimidazole-4-carboxamide ribotide

OXPHOS

Oxidative phosphorylation

FDG-PET

Fluoro-deoxy-glucose-Positron Emission Tomography

mTORC1

Mammalian target of rapamycin complex 1

RPPA

Reverse Phase Protein Array

ACC

Acetyl-CoA carboxylase

TSC2

Tuberous sclerosis protein 2

ETC

Electron transfer chain complex

TCA

Tricarboxylic acid

MP

Methyl pyruvate

TEM

Transmission electron microscopy

Footnotes

Conflict of interest: The authors have declared that no conflict of interest exists.

REFERENCES

  • 1.Warburg O. On the origin of cancer cells. Science. 1956;123:309–14. doi: 10.1126/science.123.3191.309. [DOI] [PubMed] [Google Scholar]
  • 2.Downey RJ, Akhurst T, Gonen M, Vincent A, Bains MS, Larson S, et al. Preoperative F-18 fluorodeoxyglucose-positron emission tomography maximal standardized uptake value predicts survival after lung cancer resection. J Clin Oncol. 2004;22:3255–60. doi: 10.1200/JCO.2004.11.109. [DOI] [PubMed] [Google Scholar]
  • 3.Gatenby RA, Gillies RJ. Why do cancers have high aerobic glycolysis? Nat Rev Cancer. 2004;4:891–9. doi: 10.1038/nrc1478. [DOI] [PubMed] [Google Scholar]
  • 4.Kumar R, Mavi A, Bural G, Alavi A. Fluorodeoxyglucose-PET in the management of malignant melanoma. Radiol Clin North Am. 2005;43:23–33. doi: 10.1016/j.rcl.2004.09.011. [DOI] [PubMed] [Google Scholar]
  • 5.DeBerardinis RJ, Lum JJ, Hatzivassiliou G, Thompson CB. The biology of cancer: metabolic reprogramming fuels cell growth and proliferation. Cell Metab. 2008;7:11–20. doi: 10.1016/j.cmet.2007.10.002. [DOI] [PubMed] [Google Scholar]
  • 6.Brugge J, Hung MC, Mills GB. A new mutational AKTivation in the PI3K pathway. Cancer Cell. 2007;12:104–7. doi: 10.1016/j.ccr.2007.07.014. [DOI] [PubMed] [Google Scholar]
  • 7.Samuels Y, Wang Z, Bardelli A, Silliman N, Ptak J, Szabo S, et al. High frequency of mutations of the PIK3CA gene in human cancers. Science. 2004;304:554. doi: 10.1126/science.1096502. [DOI] [PubMed] [Google Scholar]
  • 8.Engelman JA, Luo J, Cantley LC. The evolution of phosphatidylinositol 3-kinases as regulators of growth and metabolism. Nat Rev Genet. 2006;7:606–19. doi: 10.1038/nrg1879. [DOI] [PubMed] [Google Scholar]
  • 9.Shamji AF, Nghiem P, Schreiber SL. Integration of growth factor and nutrient signaling: implications for cancer biology. Mol Cell. 2003;12:271–80. doi: 10.1016/j.molcel.2003.08.016. [DOI] [PubMed] [Google Scholar]
  • 10.Brenman JE. AMPK/LKB1 signaling in epithelial cell polarity and cell division. Cell Cycle. 2007;6:2755–9. doi: 10.4161/cc.6.22.4927. [DOI] [PubMed] [Google Scholar]
  • 11.Buttgereit F, Brand MD. A hierarchy of ATP-consuming processes in mammalian cells. Biochem J. 1995;312:163–7. doi: 10.1042/bj3120163. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Bolster DR, Crozier SJ, Kimball SR, Jefferson LS. AMP-activated Protein Kinase Suppresses Protein Synthesis in Rat Skeletal Muscle through Down-regulated Mammalian Target of Rapamycin (mTOR) Signaling. J Biol Chem. 2002;277:23977–80. doi: 10.1074/jbc.C200171200. [DOI] [PubMed] [Google Scholar]
  • 13.Gwinn DM, Shackelford DB, Egan DF, Mihaylova MM, Mery A, Vasquez DS, et al. AMPK phosphorylation of raptor mediates a metabolic checkpoint. Mol Cell. 2008;30:214–26. doi: 10.1016/j.molcel.2008.03.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Hardie DG. AMP-activated/SNF1 protein kinases: conserved guardians of cellular energy. Nat Rev Mol Cell Biol. 2007;8:774–85. doi: 10.1038/nrm2249. [DOI] [PubMed] [Google Scholar]
  • 15.Kahn BB, Alquier T, Carling D, Hardie DG. AMP-activated protein kinase: ancient energy gauge provides clues to modern understanding of metabolism. Cell Metab. 2005;1:15–25. doi: 10.1016/j.cmet.2004.12.003. [DOI] [PubMed] [Google Scholar]
  • 16.Sheehan KM, Calvert VS, Kay EW, Lu Y, Fishman D, Espina V, et al. Use of reverse phase protein microarrays and reference standard development for molecular network analysis of metastatic ovarian carcinoma. Mol Cell Proteomics. 2005;4:346–55. doi: 10.1074/mcp.T500003-MCP200. [DOI] [PubMed] [Google Scholar]
  • 17.Tibes R, Qiu Y, Lu Y, Hennessy B, Andreeff M, Mills GB, et al. Reverse phase protein array: validation of a novel proteomic technology and utility for analysis of primary leukemia specimens and hematopoietic stem cells. Mol Cancer Ther. 2006;5:2512–21. doi: 10.1158/1535-7163.MCT-06-0334. [DOI] [PubMed] [Google Scholar]
  • 18.Liang J, Shao SH, Xu ZX, Hennessy B, Ding Z, Larrea M, et al. The energy sensing LKB1-AMPK pathway regulates p27(kip1) phosphorylation mediating the decision to enter autophagy or apoptosis. Nat Cell Biol. 2007;9:218–24. doi: 10.1038/ncb1537. [DOI] [PubMed] [Google Scholar]
  • 19.Lum JJ, DeBerardinis RJ, Thompson CB. Autophagy in metazoans: cell survival in the land of plenty. Nat Rev Mol Cell Biol. 2005;6:439–48. doi: 10.1038/nrm1660. [DOI] [PubMed] [Google Scholar]
  • 20.Inoki K, Zhu T, Guan K-L. TSC2 Mediates Cellular Energy Response to Control Cell Growth and Survival. Cell. 2003;115:577–90. doi: 10.1016/s0092-8674(03)00929-2. [DOI] [PubMed] [Google Scholar]
  • 21.Sarbassov DD, Ali SM, Sabatini DM. Growing roles for the mTOR pathway. Curr Opin Cell Biol. 2005;17:596–603. doi: 10.1016/j.ceb.2005.09.009. [DOI] [PubMed] [Google Scholar]
  • 22.Wang W, Guan KL. AMP-activated protein kinase and cancer. Acta Physiol (Oxf) 2009;196:55–63. doi: 10.1111/j.1748-1716.2009.01980.x. [DOI] [PubMed] [Google Scholar]
  • 23.Mohanti BK, Rath GK, Anantha N, Kannan V, Das BS, Chandramouli BA, et al. Improving cancer radiotherapy with 2-deoxy-D-glucose: phase I/II clinical trials on human cerebral gliomas. Int J Radiat Oncol Biol Phys. 1996;35:103–11. doi: 10.1016/s0360-3016(96)85017-6. [DOI] [PubMed] [Google Scholar]
  • 24.Fryer LG, Parbu-Patel A, Carling D. The Anti-diabetic drugs rosiglitazone and metformin stimulate AMP-activated protein kinase through distinct signaling pathways. J Biol Chem. 2002;277:25226–32. doi: 10.1074/jbc.M202489200. [DOI] [PubMed] [Google Scholar]
  • 25.Zhou G, Myers R, Li Y, Chen Y, Shen X, Fenyk-Melody J, et al. Role of AMP-activated protein kinase in mechanism of metformin action. J Clin Invest. 2001;108:1167–74. doi: 10.1172/JCI13505. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Hawley SA, Gadalla AE, Olsen GS, Hardie DG. The antidiabetic drug metformin activates the AMP-activated protein kinase cascade via an adenine nucleotide-independent mechanism. Diabetes. 2002;51:2420–5. doi: 10.2337/diabetes.51.8.2420. [DOI] [PubMed] [Google Scholar]
  • 27.Young ME, Radda GK, Leighton B. Activation of glycogen phosphorylase and glycogenolysis in rat skeletal muscle by AICAR--an activator of AMP-activated protein kinase. FEBS Lett. 1996;382:43–7. doi: 10.1016/0014-5793(96)00129-9. [DOI] [PubMed] [Google Scholar]
  • 28.Corton JM, Gillespie JG, Hawley SA, Hardie DG. 5-aminoimidazole-4-carboxamide ribonucleoside. A specific method for activating AMP-activated protein kinase in intact cells? Eur J Biochem. 1995;229:558–65. doi: 10.1111/j.1432-1033.1995.tb20498.x. [DOI] [PubMed] [Google Scholar]
  • 29.Sullivan JE, Brocklehurst KJ, Marley AE, Carey F, Carling D, Beri RK. Inhibition of lipolysis and lipogenesis in isolated rat adipocytes with AICAR, a cell-permeable activator of AMP-activated protein kinase. FEBS Lett. 1994;353:33–6. doi: 10.1016/0014-5793(94)01006-4. [DOI] [PubMed] [Google Scholar]
  • 30.Labajova A, Vojtiskova A, Krivakova P, Kofranek J, Drahota Z, Houstek J. Evaluation of mitochondrial membrane potential using a computerized device with a tetraphenylphosphonium-selective electrode. Anal Biochem. 2006;353:37–42. doi: 10.1016/j.ab.2006.03.032. [DOI] [PubMed] [Google Scholar]
  • 31.Waterhouse NJ, Goldstein JC, von Ahsen O, Schuler M, Newmeyer DD, Green DR. Cytochrome c maintains mitochondrial transmembrane potential and ATP generation after outer mitochondrial membrane permeabilization during the apoptotic process. J Cell Biol. 2001;153:319–28. doi: 10.1083/jcb.153.2.319. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Lamperth L, Dalakas MC, Dagani F, Anderson J, Ferrari R. Abnormal skeletal and cardiac muscle mitochondria induced by zidovudine (AZT) in human muscle in vitro and in an animal model. Lab Invest. 1991;65:742–51. [PubMed] [Google Scholar]
  • 33.Owen MR, Doran E, Halestrap AP. Evidence that metformin exerts its anti-diabetic effects through inhibition of complex 1 of the mitochondrial respiratory chain. Biochemical Journal. 2000;348:607–14. [PMC free article] [PubMed] [Google Scholar]
  • 34.El-Mir MY, Nogueira V, Fontaine E, Averet N, Rigoulet M, Leverve X. Dimethylbiguanide inhibits cell respiration via an indirect effect targeted on the respiratory chain complex I. J Biol Chem. 2000;275:223–8. doi: 10.1074/jbc.275.1.223. [DOI] [PubMed] [Google Scholar]
  • 35.Mertz RJ, Worley JF, Spencer B, Johnson JH, Dukes ID. Activation of stimulus-secretion coupling in pancreatic beta-cells by specific products of glucose metabolism. Evidence for privileged signaling by glycolysis. J Biol Chem. 1996;271:4838–45. doi: 10.1074/jbc.271.9.4838. [DOI] [PubMed] [Google Scholar]
  • 36.Sener A, Kawazu S, Hutton JC, Boschero AC, Devis G, Somers G, et al. The stimulus-secretion coupling of glucose-induced insulin release. Effect of exogenous pyruvate on islet function. Biochem J. 1978;176:217–32. doi: 10.1042/bj1760217. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Zawalich WS, Zawalich KC. Influence of pyruvic acid methyl ester on rat pancreatic islets. Effects on insulin secretion, phosphoinositide hydrolysis, and sensitization of the beta cell. J Biol Chem. 1997;272:3527–31. doi: 10.1074/jbc.272.6.3527. [DOI] [PubMed] [Google Scholar]
  • 38.Jones RG, Plas DR, Kubek S, Buzzai M, Mu J, Xu Y, et al. AMP-activated protein kinase induces a p53-dependent metabolic checkpoint. Mol Cell. 2005;18:283–93. doi: 10.1016/j.molcel.2005.03.027. [DOI] [PubMed] [Google Scholar]
  • 39.Shaw RJ, Lamia KA, Vasquez D, Koo SH, Bardeesy N, Depinho RA, et al. The kinase LKB1 mediates glucose homeostasis in liver and therapeutic effects of metformin. Science. 2005;310:1642–6. doi: 10.1126/science.1120781. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Hawley SA, Gadalla AE, Olsen GS, Hardie DG. The antidiabetic drug metformin activates the AMP-activated protein kinase cascade via an adenine nucleotide-independent mechanism. Diabetes. 2002;51:2420–5. doi: 10.2337/diabetes.51.8.2420. [DOI] [PubMed] [Google Scholar]
  • 41.Ben Sahra I, Laurent K, Giuliano S, Larbret F, Ponzio G, Gounon P, et al. Targeting Cancer Cell Metabolism: The Combination of Metformin and 2-Deoxyglucose Induces p53-Dependent Apoptosis in Prostate Cancer Cells. Cancer Res. 2010;70:2465–75. doi: 10.1158/0008-5472.CAN-09-2782. [DOI] [PubMed] [Google Scholar]
  • 42.Buzzai M, Jones RG, Amaravadi RK, Lum JJ, DeBerardinis RJ, Zhao F, et al. Systemic treatment with the antidiabetic drug metformin selectively impairs p53-deficient tumor cell growth. Cancer Research. 2007;67:6745–52. doi: 10.1158/0008-5472.CAN-06-4447. [DOI] [PubMed] [Google Scholar]
  • 43.Kalender A, Selvaraj A, Kim SY, Gulati P, Brûlé S, Viollet B, et al. Metformin, independent of AMPK, inhibits mTORC1 in a rag GTPase-dependent manner. Cell Metab. 2010;11:390–401. doi: 10.1016/j.cmet.2010.03.014. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS331271-supplement-1.doc (2.9MB, doc)

RESOURCES