Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Sep 1.
Published in final edited form as: Cancer Immunol Res. 2014 Jun 9;2(9):901–910. doi: 10.1158/2326-6066.CIR-13-0123

STING ligand c-di-GMP improves cancer vaccination against metastatic breast cancer

Dinesh Chandra 1,$, Wilber Quispe-Tintaya 1,$, Arthee Jahangir 1, Denise Asafu-Adjei 1, Ilyssa Ramos 1, Herman O Sintim 3,5, Jie Zhou 3,5, Yoshihiro Hayakawa 4, David KR Karaolis 2, Claudia Gravekamp 1,*
PMCID: PMC4264585  NIHMSID: NIHMS604210  PMID: 24913717

Abstract

Cancer vaccination may be our best and most benign option for preventing or treating metastatic cancer. However, breakthroughs are hampered by immune suppression in the tumor microenvironment (TME). In this study, we analyzed whether cyclic di-guanylate (c-di-GMP), a ligand for stimulator of interferon genes (STING), could overcome immune suppression and improve vaccination against metastatic breast cancer. Mice with metastatic breast cancer (4T1 model) were therapeutically immunized with an attenuated Listeria monocytogenes (LM)-based vaccine, expressing tumor-associated antigen Mage-b (LM-Mb), followed by multiple low doses of c-di-GMP (0.01 nmol). This resulted in a striking and near elimination of all metastases. Experiments revealed that c-di-GMP targets myeloid-derived suppressor cells (MDSC) and tumor cells. Low doses of c-di-GMP significantly increased the production of IL-12 by MDSCs, in correlation with improved T-cell responses to Mage-b, while high dose of c-di-GMP (range 15–150 nmol) activated caspase-3 in the 4T1 tumor cells and killed the tumor cells directly. Based on these results we tested one administration of high dose c-di-GMP (150 nmol) followed by repeated administrations of low dose c-di-GMP (0.01 nmol) in the 4T1 model, and found equal efficacy compared to the combination of LM-Mb and c-di-GMP. This correlated with a mechanism of improved CD8 T-cell responses to tumor-associated antigens (TAA) Mage-b and Survivin, most likely through cross-presentation of these TAAs from c-di-GMP-killed 4T1 tumor cells, and through c-di-GMP-activated TAA-specific T cells. Our results demonstrate that activation of STING-dependent pathways by c-di-GMP is highly attractive for cancer immunotherapy.

Keywords: STING ligand, c-di-GMP, Listeria-Mage-b, metastatic breast cancer, IL-12

Introduction

Until today, there is no cure for metastatic cancer. Cancer vaccination has shown great promise for preventing or treating metastatic cancer in mice and humans (1–3). However, immune suppression in the tumor microenvironment (TME) remains a potential limitation to immunotherapy. Myeloid-derived suppressor cells (MDSC) are one of the most important players in mediating TME-associated immune suppression because they strongly inhibit T-cell and NK-cell responses (4–6), with tumor-associated macrophages (TAM), Tregs, and M2 macrophages also playing a role (4–8).

Adjuvants reducing immune suppression are of great value for cancer vaccination. c-di-GMP (3′,5′-cyclic diguanylic acid), also known as cyclic diguanylate, could be such an adjuvant. c-di-GMP is a bacterial intracellular signaling molecule that was initially identified by the Benziman laboratory in the bacterium Acetobacter xylinum (renamed Gluconacetobacter xylinus)(9). Various in vitro and in vivo animal model studies using chemically synthesized c-di-GMP demonstrated that c-di-GMP has potent immunomodulatory effects on cellular components of both innate and adaptive immunity in bacterial infections such as K. pneumoniae (10–12). Recently, stimulator of interferon genes (STING) has been identified as the sensor for c-di-GMP (13). STING is a transmembrane protein expressed in macrophages and dendritic cells (14–16). STING is mainly expressed in the thymus, heart, spleen, placenta, lung and peripheral leukocytes but is poorly expressed in the brain, skeletal muscle colon, small intestine, liver, and kidneys (14). Because of the strong immunomodulatory effects of c-di-GMP, we evaluated whether STING-dependent c-di-GMP could improve cancer vaccination through bypassing immune suppression and stimulating T-cell responses in mice with metastatic breast cancer.

As vaccine, we used a highly attenuated Listeria monocytogenes (LM) bacterium expressing tumor-associated antigen (TAA) Mage-b (Mb), which was developed in an earlier study (3). This attenuated LM is different from wild type LM (17, 18) in that the attenuated LM does not multiply in normal tissues and is naturally cleared by the immune system within three to five days (5, 19, 20). Mage-b is highly expressed in metastases and primary breast tumors of the 4T1 model (3), and is homologous with human MAGE (21). MAGE is expressed in 90% of all breast cancers (22). LM is an intracellular pathogen that delivers the vaccine antigen directly into antigen-presenting cells (APC) such as macrophages with high efficiency (23). The vaccine antigen produced by LM is processed and presented as short peptides via the MHC class I and class II pathways generating both CD4 and CD8 T-cell responses (24). Killing of tumor cells occurs through CD8 T cells. While semi-prophylactic immunizations with LM-Mb (one before and two after tumor development) were highly effective against metastatic breast cancer, this effect was less abundant with a more clinically relevant immunization protocol of three exclusive therapeutic vaccinations (after tumor development) (20) due to the strong immune suppression in the TME. Therefore, reducing immune suppression and improving T-cell responses to TAAs in the TME was the most important goal in this study, and c-di-GMP seemed an extremely suitable candidate.

Here, we demonstrate that c-di-GMP exhibits various mechanisms to combat metastatic breast cancer. Low doses of c-di-GMP provided strong adjuvant effects in LM-Mb vaccinations by reducing the MDSC population (highly expressing STING), by converting a subpopulation of immune-suppressing MDSCs into an immune-stimulating phenotype producing IL-12, and by improving CD8 T-cell responses to tumor-associated antigen Mb delivered through LM. High doses of c-di-GMP activated caspase-3 and killed tumor cells directly. This unique combination of therapeutic low doses of c-di-GMP and LM-Mb resulted in an almost complete elimination of the metastases. Moreover, one high dose c-di-GMP followed by multiple low doses of c-di-GMP in a therapeutic setting was equally effective compared to LM-Mb + c-di-GMP, and showed improved CD8 T-cell responses to Mage-b and Survivin, most likely through cross-presentation of TAAs of c-di-GMP-killed tumor cells and through activation of the T cells by multiple low doses of c-di-GMP. These dramatic results with c-di-GMP are highly promising for human clinical application.

Materials and Methods

Mice

Normal female BALB/c mice aged 3 months were obtained from Charles River, and maintained in the animal husbandry facility of Albert Einstein College of Medicine according to the guidelines of the Association for Assessment and Accreditation of Laboratory Animal Care (AAALAC), and the guidelines of the Albert Einstein Institute for Animal Studies. All mice were kept under biosafety level 2 conditions, as required for LM.

Cells and cell culture

The 4T1 cell line, derived from a spontaneous mammary carcinoma in a BALB/c mouse (25), was cultured in Dulbecco’s Modified Eagle’s Medium (DMEM) supplemented with 10% fetal bovine serum (FBS), 1 mM mixed nonessential amino acids, 2 mM L-glutamine, insulin (0.5 HSP units/ml) penicillin (100 units/ml) and streptomycin (100 μg/ml). This cell line was developed by Dr. Fred Miller, Karmanos Cancer Institute, Detroit, MI, and an early passage (passage 25) was kindly provided in June 2004. The 4T1 cell line is extremely metastatic in BALB/c mice (syngeneic background strain). This extreme metastatic character has been verified in BALB/c mice within the last two months. The cell line has a characteristic growth pattern, i.e. it forms mammosphere-like balls in cell culture. This has been verified within the last two months by inverted light microscopy. The 4T1 cell line expresses TAA Mage-b, and has been verified within the last 6 months by RT-PCR. The 4T1 cell line is mycoplasma-free.

The HEK293T cell line was purchased from ATCC (CRL-11268) in 2009. This is a human epithelial kidney cell line. This cell line is often used as a negative control for STING expression in western blots. In the manuscript presented here, HEK293T was negative for STING expression by western blotting. The HEK293T cell line is mycoplasma-free.

Plasmids, Listeria monocytogenes, and c-di-GMP

The LM-Mb strain was developed in an earlier study (3). This was constructed in the prfA negative XFL-7 strain (referred to as LM in this study), which lacks the positive regulatory factor A that is a central mediator of virulence (26). The vaccine strain was transformed with LM plasmid pGG-34, which encodes prfA and amino acids 311–660 of murine Mage-b fused to a non-cytolytic form of Listeriolysin O (LLO) (3, 27). Complementation of prfA expression by the plasmid does not fully restore virulence, but enforces retention of the plasmid during infection (26, 27).

c-di-GMP (provided by D. Karaolis, Karagen Pharmaceuticals, Frederick, MD), used in these studies was synthesized and prepared as previously described (28). Stock solutions of c-di-GMP were generated at 4 mM in Saline (equals 200 nmol in 50 μl). For in vivo experiments 150 nmol or 0.01 nmol per mouse was used, and for in vitro experiments, a range of 1.87–15 nmol was used as indicated in the text.

Immunization and tumor challenge

Various immunization protocols have been tested in a metastatic breast tumor model 4T1, as described previously (3). Semi-prophylactic Protocol A: mice were immunized intraperitoneally (ip) with c-di-GMP (150 nmol) on days 0, 7, and 14, and with LM-Mb (107 CFU) ip on days 1, 8, and 15, while 4T1 tumor cells (105) were injected into the mammary fat pad on day 3. Therapeutic Protocol B (high dose LM-Mb and c-di-GMP): mice were injected with 4T1 tumor cells (105) in the mammary fat pad on day 0, immunized ip with c-di-GMP (150 nmol) on days 3, 10, and 17, and with LM-Mb (107 CFU) (ip) on days 4, 11, and 18. Therapeutic Protocol C (low dose LM-Mb and c-di-GMP): 4T1 tumor cells (0.5×105) were injected into the mammary fat pad on day 0, and c-di-GMP (0.01 nmol) was administered every day ip, starting on day 3, while LM-Mb (104 CFU) was administered ip on days 4, 7, 10, 13, and 16. Therapeutic Protocol D (high vs low dose c-di-GMP): 4T1 tumor cells (0.5×105) were injected into the mammary fat pad on day 0, and one high dose of c-di-GMP (150 nmol) was administered on day 4, followed by ip low dose c-di-GMP (0.01 nmol) every day for 16 days. A detailed schematic view of all immunization protocols is shown in Figure S1ABCD. All mice were euthanized on day 19 and analyzed for the number of metastases and tumor growth. Primary tumors extend to the chest cavity lining and metastases predominantly to the mesenteric lymph nodes (MLN), and less frequently to the diaphragm, portal liver, spleen, and kidneys (20).

Isolation of MDSCs

Monocytic and granulocytic MDSCs (mMDSC and gMDSC, respectively) were isolated from spleen cell-suspensions according the manufacturer’s instructions (Myeloid-Derived Suppressor Cell Isolation Kit, Miltenyi Biotec) as described previously (5). For separation of the magnetically labeled cells, the Automacs Proseparator (Miltenyi Biotech) was used. As determined by flow cytometry, the purity of the isolated gMDSC was ≥90% and of mMDSC ≥85%.

ELISPOT and cytokine ELISA

Spleen cells were isolated from vaccinated and control mice with 4T1 tumors for analysis by ELISPOT as described previously (3). To detect LM-induced T-cell responses, 2×105 spleen cells from vaccinated or control mice were infected with 2×105 CFU of LM for 1 hour, and subsequently treated with gentamicin (50 μg/ml) until the end of re-stimulation (72 hours). To detect TAA-specific T-cell responses, 4×105 spleen cells from vaccinated or control mice were transfected with pcDNA3.1-Mage-b using lipofectamin 2000, and cultured for 72 hours as described previously (3). Also Survivin66–74 peptide (GWEPDDNPI) was used for restimulation of the spleen cells from c-di-GMP-treated and saline-control mice in the presence or absence of MHC class I antibodies (EBioscience, San Diego, CA). Briefly, 4×105 spleen cells were mixed with or without anti-MHC class I antibodies (1 μg/ml) and then restimulated with Survivin66–74 peptide 100 μg/ml for 72 hours. After 24 hours, IL-2 (25 U/ml)(BD Pharmingen, San Diego, CA) was added to both spleen cultures, to enrich for TAA-specific T cells. At the end of the 72 hours, the frequency of IFNγ-producing cells was determined by ELISPOT according to standard protocols (BD Pharmingen) using an ELISPOT reader (CTL Immunospot S4 analyzer, Cellular Technology, Ltd, Cleveland, OH). To determine the frequency of IFNγ-producing CD8 T cells, spleen cells were depleted for CD8 T cells using magnetic bead depletion techniques according to the manufacturer’s instructions (Miltenyi Biotech, Auburn, CA). All antibodies were purchased from BD Pharmingen.

The in vitro production of IL-12p40 by gMDSC and mMDSC was measured by ELISA as described previously (5). For this purpose, 500,000 gMDSC or mMDSC were incubated with different concentrations of c-di-GMP. After 72 hours, the levels of IL-12 in the culture supernatant were determined by ELISA according to manufacturer’s instructions (BD Pharmingen).

Depletion of CD8 T cells in vivo

CD8 T cells were depleted in 4T1 tumor-bearing mice with 400 μg of anti-CD8 antibodies (H35)(29) (kindly provided by Dr. Lauvau, Department of Microbiology and Immunology of Einstein) during c-di-GMP treatment (five injections three days apart). All mice were euthanized 2 days after the last anti-CD8 treatment, and analyzed for tumor weight and number of metastases. After depletion, the percentage of CD8 T cells in the spleen was less than 1%. As control, isotype-matched rat antibodies against HRPN were used.

Flow cytometry analysis

Immune cells from spleen, blood, lymph nodes, or tumors of treated and control mice were isolated as described previously (30). To identify MDSCs, anti-CD11b-Alexa488/PerCP-cy5.5 and anti-Gr1-PerCP-cy5.5 (clone RB6-8C5) antibodies were used. The CD11b+Gr1high population represents the gMDSC population and the CD11b+Gr1low the mMDSC population. To analyze various subsets of immune cells of 4T1 tumor-bearing mice involved in antitumor responses in spleens, tumors and lymph nodes, anti-CD3-Alexa488, anti-CD4- PerCP-cy5.5, anti-CD8-APC, anti-CD49d-PE (NK cells), anti-CD19-FITC (B cells), anti-CD11c-FITC antibodies were used. All cell populations in the tumor were analyzed within the CD45-positive population using anti-CD45-FITC/APC antibody. For the maturation of MDSCs and DCs in the spleens of tumor-bearing mice we used anti-CD80-APC, anti-CD86-PE, and anti-MHC class II-PE antibodies. To detect the production of intracellular cytokines, the cytofix/cytoperm kit from BD Pharmingen was used according to manufacturer’s instructions, and antibodies to IFNγ and IL-12 were used. Appropriate isotype controls were used for each sample. Depending on the sample size, 10,000–50,000 cells were acquired by scanning using a Fluorescence Activated Cell Sorter (flow cytometry) (Beckton and Dickinson; Excalibur), and analyzed using FlowJo 7.6 software. Cell debris and dead cells were excluded from the analysis based on scatter signals and use of Fixable Blue or Green Live/Dead Cell Stain Kit (Invitrogen). All antibodies were purchased from BD Pharmingen or eBiosciences.

MTT test and cell death

4T1 cells or HEK293T cells (2000 cells in 0.1 ml) were cultured with different doses of c-di-GMP as indicated in the text for 24 hours, then cell viability was analyzed by the 3-(4, 5-dimethylthiazolyl-2)-2, 5-diphenyltetrazolium bromide (MTT) method at a wave length of 570 nm. Cell death in c-di-GMP-treated cultures was calculated by subtracting % live cells from non-treated cells (100%).

Caspase-3 analysis

4T1 tumor cells were treated with different concentrations of c-di-GMP as indicated in the text, fixed in 10% neutral buffered formalin, and permeabilized with 0.1% Triton X100 for 30 min. Endogenous peroxidase activity was quenched using 3% hydrogen peroxide for 10 minutes, and blocked by 5% normal donkey serum and 2% BSA for 1hour. The cells were then stained by immunohistochemistry methods, using a primary antibody to active caspase-3 (rabbit anti-mouse IgG Cell Signaling, Billerica, MA) 1:50 dilution for 1hour at RT, followed by a secondary antibody conjugated with HRP for 1hour at RT (Invitrogen, Grand Island, NY), and then followed with diaminobenzidine as the final chromogen. All slides were briefly counterstained with hematoxylin.

STING analysis

Expression of STING was analyzed by western blotting. Tissues were homogenized with 1 ml of PBS containing 0.1% Triton-X100, 2mM EDTA and protease inhibitors by using Mini Bead Beater for 2–5 × 30sec. Cultures of 4T1 tumor cells and MDSCs were centrifuged and resuspended in 400 μl of ice-cold lysis buffer (1M HEPES, 0.5 M EDTA, 0.1M EGTA, 2M KCl 0.1M DTT, cocktail of protease inhibitors 5μg/ml, and 10% NP-40). The lysates (20 μg) were electrophoresed in 10% SDS-polyacrylamide gels and proteins were electro-transferred to PVDF filters membrane. Membranes were probed with rabbit antibodies against STING (Cell Signaling Technology, Danvers, MA, USA). Antibody to β-actin was used to ensure equal loading. Membranes were incubated with HRP-conjugated anti-rabbit goat IgG secondary antibodies (Cell Signaling Technology, Danvers, MA) and detection was obtained using a chemiluminescence detection kit (Thermo scientific, Rockford, IL).

Statistical analysis

To statistically analyze the effects of LM-Mb and/or c-di-GMP on the growth of metastases and tumors and on immune responses in the mouse models, the unpaired t test and the Mann-Whitney were used. *p<0.05, **<0.01, ***<0.001, ****<0.0001. Values p<0.05 were considered statistically significant.

Results

High doses of c-di-GMP and LM-Mb completely eliminated metastatic breast cancer without T-cell contribution in a semi-prophylactic setting

Immune suppression in the TME is a major problem in cancer vaccination (5, 7), and adjuvants that can reduce or convert immune suppression are greatly needed. c-di-GMP has shown promising immune stimulatory effects on innate and adaptive immune responses in bacterial infections (10). To analyze whether c-di-GMP could also induce immune stimulatory effects on the immune system in cancer vaccination, tumor-bearing mice were immunized with LM-Mb and c-di-GMP in semi-prophylactic setting (Protocol A). This combination protocol completely eliminated the metastatic breast cancer (Figure 1AB), which could not be obtained with c-di-GMP or LM-Mb alone. Most interestingly, this striking result was not due to improved CD8 T-cell responses to Mage-b and LM. In contrast, it appeared that c-di-GMP did not increase but decreased Mage-b-specific (Figure 1C) or LM-specific (Figure 1D) CD8 T-cell responses in the spleen of vaccinated and control mice. Since c-di-GMP alone was highly effective against metastases but did not improve T-cell responses, we further investigated the mechanism of action of c-di-GMP. We also tested LM-Mb and c-di-GMP therapeutically, and found that the number of metastases was reduced by 73% and tumor growth by 33% compared to the Saline only group (Figure S2AB).

Figure 1. Semi-prophylactic immunizations with high doses of c-di-GMP and LM-Mb completely eliminated the metastatic breast cancer without improving T-cell responses.

Figure 1

Open in a new tab

BALB/c mice received one preventive followed by two therapeutic immunizations with high doses of c-di-GMP and LM-Mb according to Immunization protocol A. All mice were euthanized nineteen days after the first immunization and analyzed for the number of metastases (A) and tumor weight (B). In the same experiments, Mage-b-specific (C) and LM-specific (D) CD8 T-cell responses were analyzed in the spleen (pooled) by ELISPOT in vitro. The results shown here are the averages of three independent experiments (n=5 mice per group). The error bars represent standard error of the mean (SEM). In A and B, all groups were compared to LM-Mb + c-di-GMP. In C and D, all groups not depleted for CD8 T cells were compared to LM-Mb + c-di-GMP (star), while the CD8 T cell-depleted groups were compared to the same groups without CD8-depletion (rhombus). Mann-Whitney test. *p<0.05, **<0.01, ***<0.001, ****<0.0001. Values p<0.05 were considered statistically significant.

High doses of c-di-GMP activated caspase-3 in 4T1 tumor cells

Since high doses of c-di-GMP had an effect on the metastases and primary tumor in vivo, we determined whether c-di-GMP could kill tumor cells directly. We found that 15 nmol of c-di-GMP reduced the growth of 4T1 tumor cells in vitro by 70% (MTT test) (Figure 2A), and 150 nm by 92% (Figure S3). To analyze the mechanism of tumor cell killing in more detail, we determined caspase-3 expression at various doses of c-di-GMP. We found that 7.5 and 15 nmol of c-di-GMP induced caspase-3 expression in 40% and 20% of the 4T1 tumor cells, respectively (Figure 2BC). Moreover, most of the remaining tumor cells died. Since c-di-GMP acts through interaction with its sensor STING, we analyzed 4T1 tumor cells for STING expression by western blotting. As shown here, STING is expressed in 4T1 tumor cells and bone marrow (BM) cells (positive control), and even at much higher levels in the 4T1 metastases and primary tumor, while STING could not be detected in HEK293T cells (negative control) (Figure 2D). c-di-GMP had significantly less effect on the viability and cell death of STING-negative HEK293 cells than of STING-positive 4T1 at doses of 15 and 7.5 nmol (Figure S4).

Figure 2. High doses of c-di-GMP killed 4T1 tumor cells directly and activated caspase-3.

Figure 2

Open in a new tab

4T1 tumor cells were incubated with various doses of c-di-GMP for 24hours. The viability of the 4T1 tumor cells was assayed by MTT assay (A), or the number of 4T1 tumor cells with caspase-3 activation was quantified by immunohistochemistry (IHC) (B). The results shown here is the average of three independent experiments. The number of caspase-3-positive cells was counted per 100 cells. The error bars represent the SEM. An example of caspase-3 activation is shown by IHC (C). Also the expression of STING was analyzed in the primary tumor, metastases and the 4T1 cell line by western blotting (D). Bone marrow (BM) cells and HEK293 were used as a positive and negative control, respectively.

Low doses of c-di-GMP improved efficacy of LM-Mb and T-cell responses to Mage-b in a therapeutic setting

Based on the results described above, we hypothesized that if high doses could kill tumor cells directly, it may be toxic for T cells as well. Therefore, we tested whether low doses of c-di-GMP were effective against the metastases and could improve T-cell responses. In addition, we combined this with a clinically more relevant therapeutic immunization protocol (Protocol B). Various lower doses of c-di-GMP (range 25–100 nmol) were effective against metastases and the primary tumor (Figure S5AB), but when combined with LM-Mb, c-di-GMP did not improve T-cell responses to Mage-b. Therefore, we further decreased the dose of c-di-GMP (range 0.01–10 nmol), and found that the administration of 0.01 nmol c-di-GMP every day for two weeks was the most effective. Also LM appeared to be more effective therapeutically at a lower dose (104 CFU) once every three days than a high dose (107 CFU) once per week (Figure S5CD). Based on these results, we administered 0.01 nmol c-di-GMP daily with 104 CFU of LM-Mb every three days in a therapeutic setting (Figure S1C) in all remaining studies. As shown in Figure 3AB, this combination was highly effective against the metastases but less vigorous against the primary tumor. The number of metastases in the group of LM-Mb + c-di-GMP was significantly reduced compared to all other groups, while the tumor weight in the group with LM-Mb + c-di-GMP was significantly reduced compared to the saline only group. Finally, we found that therapeutic treatment of tumor-bearing mice with 0.01 nmol of c-di-GMP significantly improved T-cell responses to LM-Mb in vivo and in vitro after re-stimulation with Mage-b (Figure 3CD).

Figure 3. Therapeutic immunizations with low doses of c-di-GMP and LM-Mb almost completely eliminated all metastases and improved T-cell responses in vitro and in vivo.

Figure 3

Open in a new tab

BALB/c mice received five therapeutic immunizations with low doses of c-di-GMP every day and LM-Mb every three days, according to Immunization protocol C. Nineteen days after tumor challenge mice were euthanized and analyzed for the number of metastases (A), tumor weight (B). In the same experiments, CD8 T-cell responses to Mage-b were analyzed in the spleen (pooled) in vitro by ELISPOT (C), or in blood (pooled) in vivo without any restimulation by flow cytometry (D). The results shown here are the averages of three independent experiments (n=5 mice per group). The error bars represent the SEM. In A and B, all groups were compared to LM-Mb + c-d-GMP. In C, all groups not depleted for CD8 T cells were compared to LM-Mb + c-di-GMP (star), and the CD8 T cell-depleted groups were compared to the same groups without CD8-depletion (rhombus). In D, all groups were compared to LM-Mb + c-di-GMP (*). Mann-Whitney test. *p<0.05, **<0.01, ***<0.001, ****<0.0001. Values p<0.05 were considered statistically significant.

c-di-GMP targets MDSCs

Myeloid derived suppressor cells (MDSC) are a major population in the TME that strongly suppress T-cell activation (4–6). Here, we tested whether low doses of c-di-GMP had an effect on MDSCs. We found a significant reduction in the number of MDSCs (35%) in blood compared to that in the saline group when combined with LM-Mb, but separately this effect was less abundant and not significant (Figure 4A). Since T-cell responses were improved by the administration of low doses of c-di-GMP we analyzed the production of IL-12 by MDSCs. IL-12 is known for stimulating naïve and mature T cells (31). Low concentrations of c-di-GMP (range 0.01–2 nmol) increased the IL-12 secretion in vitro by the granulocytic MDSCs (CD11b+Gr1high) and even more by monocytic MDSCs (CD11b+Gr1low) isolated from 4T1 tumor-bearing mice (Figure 4BC). Since c-di-GMP interacts with STING, we also analyzed STING expression. As shown in Figure 4D, both mMDSCs and gMDSCs expressed STING, but the expression was more abundant in mMDSCs.

Figure 4. Therapeutic immunizations with low doses of c-di-GMP and LM-Mb reduced the percentage of MDSCs and c-di-GMP induced IL-12 production by MDSCs.

Figure 4

Open in a new tab

BALB/c mice were immunized with low doses c-di-GMP and LM-Mb and challenged with 4T1 tumor cells according to Immunization protocol C. Nineteen days after tumor challenge mice were euthanized and the percentage of MDSCs (CD11b+Gr1+) was determined in blood in vivo by flow cytometry (A). An example of gating MDSC in blood is shown in Figure S10. In addition, gMDSCs (B) and mMDSCs (C) were isolated from 4T1 tumor-bearing mice, incubated with various concentrations of c-di-GMP in vitro, and then analyzed for IL-12 secretion in the culture supernatant by ELISA (C). The results shown here are the averages of three independent experiments (n=5 mice per group). In A, all groups were compared to LM-Mb + c-di-GMP. The error bars represent the SEM. Mann-Whitney test. *p<0.05, **<0.01, ***<0.001, ****<0.0001. Values p<0.05 were considered statistically significant. Also the expression of STING was analyzed in MDSCs by western blotting. Bone marrow (BM) cells and HEK293 cells were used as positive and negative controls, respectively.

One high dose followed by multiple low doses c-di-GMP is highly effective against metastases in a therapeutic setting

Our results showed that high doses of c-di-GMP killed tumor cells directly and that low doses of c-di-GMP activated T-cell responses in vivo. Here, we combined one high dose c-di-GMP (150 nmol) with multiple injections of low dose c-di-GMP (0.01 nmol) every day for 2 weeks, and found similar efficacy in reducing the number of metastases and tumor weight (Figure 5AB) and tumor size (Figure S6) compared to the combination of low doses of LM-Mb and c-di-GMP (Figure 3AB). Compared to the saline group, the c-di-GMP-treated mice had significantly improved CD8 T-cell responses to Mage-b in vivo (Figure 5C) and in vitro (Figure 5D). We then analyzed T-cell responses to TAAs in more detail and re-stimulated spleen cells from c-di-GMP-treated and control mice with immunodominant Survivin66–74 peptide (GWEPDDNPI), matching the H2-D haplotype of BALB/c mice, in the absence or presence of anti-MHC class I antibodies. Survivin is strongly expressed by 4T1 tumor cells (32). We found a significant increase in the number of IFNγ-producing CD8 T cells to TAA Survivin compared to that of the saline group, which was almost completely reduced when blocked with anti-MHC class I antibodies (Figure 5E). Most importantly, we demonstrated a significant increase in the growth of tumor and metastases in mice that received c-di-GMP plus anti-CD8 antibodies compared to mice that received c-di-GMP alone, but not compared to the saline and isotype control mice (Figure 5FG).

Figure 5. One therapeutic high dose followed by multiple low doses of c-di-GMP almost completely eliminated the metastases and improved T-cell responses to TAAs.

Figure 5

Open in a new tab

BALB/c mice received one high dose of c-di-GMP followed by multiple low doses of c-di-GMP every day according to Immunization protocol D. Nineteen days after tumor challenge mice were euthanized and analyzed for the number of metastases (A) and tumor weight (B). In the same experiments, CD8 T-cell responses to Mage-b were analyzed in the spleen (pooled) in vitro by ELISPOT (C), or in blood (pooled) in vivo without any restimulation by flow cytometry (D). The results shown here are the averages of three independent experiments. In each experiment n=5 mice per group. Unpaired t test. *p<0.05, **<0.01, ***<0.001, ****<0.0001. Values p<0.05 were considered statistically significant. Spleen cells of c-di-GMP-treated and saline-control mice were re-stimulated with Survivin66–74 peptide (GWEPDDNPI) in the absence or presence of anti-MHC class I antibodies (E). Tumor-bearing mice were treated with c-di-GMP plus anti-CD8 antibodies (H35), and the number of metastases (F) and tumor weight (G) were compared to treatments with c-di-GMP alone, saline or isotype controls. In this experiment n=7 mice per group. The error bars in all graphs represent the standard error of the mean (SEM). Mann-Whitney test. *p<0.05, **<0.01, ***<0.001, ****<0.0001. Values p<0.05 were considered statistically significant. The error bars in all graphs represent the standard SEM.

Discussion

In the study presented here, we report the role of STING-dependent pathways in cancer immunotherapy. For this purpose we used LM-Mb as the vaccine and c-di-GMP as the STING ligand in a metastatic breast cancer model 4T1. In the past we have shown that LM-based vaccination is highly effective in stimulating innate and adaptive immune responses in tumor-bearing mice when used in a semi-prophylactic setting (one immunization before and two after tumor development)(20). However, when used in a therapeutic setting (after tumor development), which is clinically more relevant, LM-based vaccination alone was not able to overcome immune suppression in the TME (19, 20), and clearly needs help to bypass this problem. Here we demonstrate that STING ligand c-di-GMP overcomes immune suppression and stimulates TAA-specific T cells in tumor-bearing mice.

To analyze the potential pathways of c-di-GMP in cancer vaccination we tested various immunization protocols of LM-Mb and c-di-GMP in relation to efficacy and Mage-b-specific CD8 T-cell responses in 4T1 tumor-bearing mice. First, we tested LM-Mb and c-di-GMP in a semi-prophylactic setting. The results were spectacular. For the first time we obtained a complete eradication of the metastatic breast cancer. This was not possible with either LM-Mb or c-di-GMP alone. Despite the complete elimination of the metastatic breast cancer, we found that c-di-GMP did not stimulate but inhibited Mage-b- and LM-specific CD8 T-cell responses. Since c-di-GMP alone was also highly effective against metastases and even against primary tumors, we concluded that c-di-GMP itself had an effect on the tumor cells. Indeed, we found that c-di-GMP strongly reduced the viability of 4T1 tumor cells and induced activation of caspase-3 in vitro at a dose of 15 nmol. In support of our results, Karaolis and colleagues found that c-di-GMP had an effect on growth factor-stimulated colon cancer cells in vitro (28). Based on these results we hypothesized that if c-di-GMP is toxic for tumor cells, it may be also toxic for T cells and lower doses may stimulate the T cells. Therefore, we analyzed the effect of various low doses of c-di-GMP on vaccine efficacy and on Mage-b-specific CD8 T cells. We found that 0.01 nmol of c-di-GMP administered daily, significantly improved CD8 T-cell responses to Mage-b in vivo and in vitro. Since c-di-GMP are secreted by various types of bacteria such as Acetobacter xylinum, Pseudomonas aeruginosa, Vibrio cholerae (33), and since c-di-GMP serves as a danger signal for the early detection of microbes (16), it may not be surprising that extremely low doses can be sensed by the immune system.

One mechanism of the STING-dependent pathways is the production of cytokine IFNβ in CD11c+ cells and macrophages (14). Recently, it has been shown that IFNβ is involved in the intratumoral accumulation of CD8α+ DC, which is required for T-cell stimulation (34). DCs highly express STING (14–16). We found that c-di-GMP increased the expression levels of maturation markers CD80/CD86 on DCs isolated from spleens of 4T1 tumor-bearing mice, which is important for presentation of TAAs and activation of TAA-specific T cells (Fig S7). In addition, we analyzed the effect of c-di-GMP on various subsets of immune cells in 4T1 tumor-bearing mice. The most prominent effect was the decrease in the CD4 T-cell population and increase in the CD8 T-cell population in the LN due to c-di-GMP treatment in vivo compared to that of the Saline group (Fig S8, and Table S1). Since MDSCs are present in large numbers in cancer patients and in mice with cancer, and because MDSCs play an important role in immune suppression, we also analyzed the effect of c-di-GMP on MDSCs. First we demonstrated that STING is highly expressed on MDSCs. Moreover, we found that a low dose of STING ligand c-di-GMP induced the production of IL-12 by MDSCs, and improved T-cell responses to TAA Mage-b when combined with LM-Mb vaccinations in tumor-bearing mice. Since IL-12 is known to stimulate naïve and mature T cells (31), it is possible that c-di-GMP in the 4T1 model have activated CD8 T cells through the generation of IL-12 by MDSCs. We also found that c-di-GMP-treated MDSCs could reduce immune suppression and partly restore CD8 T-cell responses to CD3/CD28 stimulation compared to non-treated MDSCs (Figure S9). Interestingly, the combination of c-di-GMP and LM-Mb significantly reduced the MDSC population in blood, but less so with either c-di-GMP or LM-Mb alone. Reduction in the MDSC population may also contribute to the reduction in immune suppression and consequently to reduced growth of tumor and metastases. Since MDSCs and immune suppression are present in almost all cancers (6), c-di-GMP might be used as an adjuvant against other cancers and with other cancer vaccines as well.

The exact mechanism of how the MDSC population in blood was decreased by LM-Mb and c-di-GMP immunizations is not yet clear, but various mechanisms are possible. One option is that the combination of LM-Mb and c-di-GMP may have destroyed the tumor cells in an early stage of treatment and prevented tumor growth and thus the recruitment of MDSCs. Another option could be that since LM-Mb infects MDSCs (5) and c-di-GMP activates T cells, the LM-Mb-infected MDSCs were eliminated by the LM- and Mage-b-activated T cells. Therefore, both reduced migration of MDSCs towards smaller tumors in treated compared to non-treated mice, and elimination of LM-Mb-infected MDSCs by c-di-GMP-activated T cells may contribute to the therapeutic efficacy but more detailed analysis is required.

Based on our results with high and low doses of c-di-GMP, we hypothesized that one administration of high dose c-di-GMP could induce immunogenic tumor cell death resulting in cross-presentation of TAAs from c-di-GMP-killed tumor cells by APCs to the immune system and that repeated low doses of c-di-GMP could activate the TAA-specific T cells. We demonstrate that one high dose followed by multiple low doses of c-di-GMP, without LM-Mb vaccination, induced CD8 T-cell responses to Mage-b in vivo and in vitro, and that the treatment was equally effective against metastases compared to the combination of c-di-GMP and LM-Mb. We found a significant increase in CD8 T-cell responses to Survivin in spleen cultures from mice that received one high dose followed by multiple low doses of c-di-GMP compared to that from saline-treated control mice (4T1 tumor cells highly express Survivin (32), and the cultures were re-stimulated with immunodominant peptide Survivin66–74). These CD8 T-cell responses to Survivin were strongly reduced by blocking the interaction of the Survivin peptide/MHC class I complex in the restimulation assay with anti-MHC class I antibodies, indicating that c-di-GMP induced CD8 T-cell responses to TAA Survivin in vivo. The most convincing result demonstrating that c-di-GMP reduced growth of tumors and metastases through CD8 T-cell responses was shown by CD8 T cell-depletions in vivo. We found significantly larger tumors and more metastases in 4T1-tumor-bearing mice that received c-di-GMP and anti-CD8 antibodies compared to mice that received c-di-GMP or isotype control alone. In conclusion, these results strongly support our hypothesis that one administration of a high dose of c-di-GMP could induce immunogenic tumor cell death resulting in cross-presentation of TAAs of c-di-GMP-killed tumor cells by APCs to the immune system and that repeated low doses of c-di-GMP could activate the TAA-specific T cells.

In summary, we have demonstrated that the STING-activating ligand c-di-GMP improves vaccination or immunotherapy against metastatic breast cancer through multiple pathways. We believe that results from this novel study provides a rationale for the development of new directions in cancer immunotherapy and in combination with agents capable of inducing immunogenic tumor cell death such as chemotherapy, radiotherapy and other therapies.

Supplementary Material

1
10
11
12
2
3
4
5
6
7
8
9

Acknowledgments

Financial Support

This work was supported by NIH grant 1RO1 AG023096-01, NCI grant R21 AI090652-01, The Paul F Glenn Center for the Biology of Human Aging Research 34118A, and NSF 1307218 and NSF (Dr. Sintim).

Footnotes

Disclosure of Potential Conflicts of Interests

David Karaolis is the inventor for several patents on the use of cyclic dinucleotides, such as c-di-GMP, as vaccine adjuvants to prevent cancer and infectious disease

References

  • 1.Thurner B, Haendle I, Roder C, Dieckmann D, Keikavoussi P, Jonuleit H, et al. Vaccination with mage-3A1 peptide-pulsed mature, monocyte-derived dendritic cells expands specific cytotoxic T cells and induces regression of some metastases in advanced stage IV melanoma. J Exp Med. 1999;190:1669–78. doi: 10.1084/jem.190.11.1669. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Kruit WH, van Ojik HH, Brichard VG, Escudier B, Dorval T, Dreno B, et al. Phase 1/2 study of subcutaneous and intradermal immunization with a recombinant MAGE-3 protein in patients with detectable metastatic melanoma. Int J Cancer. 2005;117:596–604. doi: 10.1002/ijc.21264. [DOI] [PubMed] [Google Scholar]
  • 3.Kim SH, Castro F, Gonzalez D, Maciag PC, Paterson Y, Gravekamp C. Mage-b vaccine delivered by recombinant Listeria monocytogenes is highly effective against breast cancer metastases. Br J Cancer. 2008;99:741–9. doi: 10.1038/sj.bjc.6604526. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Gabrilovich DI, Nagaraj S. Myeloid-derived suppressor cells as regulators of the immune system. Nat Rev Immunol. 2009;9:162–74. doi: 10.1038/nri2506. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Chandra D, Jahangir A, Quispe-Tintaya W, Einstein MH, Gravekamp C. Myeloid-derived suppressor cells have a central role in attenuated Listeria monocytogenes-based immunotherapy against metastatic breast cancer in young and old mice. Br J Cancer. 2013;108:2281–90. doi: 10.1038/bjc.2013.206. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Ostrand-Rosenberg S, Sinha P. Myeloid-derived suppressor cells: linking inflammation and cancer. J Immunol. 2009;182:4499–506. doi: 10.4049/jimmunol.0802740. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Gajewski TF, Meng Y, Harlin H. Immune suppression in the tumor microenvironment. J Immunother. 2006;29:233–40. doi: 10.1097/01.cji.0000199193.29048.56. [DOI] [PubMed] [Google Scholar]
  • 8.Curiel TJ. Tregs and rethinking cancer immunotherapy. J Clin Invest. 2007;117:1167–74. doi: 10.1172/JCI31202. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Amikam D, Benziman M. Cyclic diguanylic acid and cellulose synthesis in Agrobacterium tumefaciens. J Bacteriol. 1989;171:6649–55. doi: 10.1128/jb.171.12.6649-6655.1989. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Karaolis DK, Newstead MW, Zeng X, Hyodo M, Hayakawa Y, Bhan U, et al. Cyclic di-GMP stimulates protective innate immunity in bacterial pneumonia. Infect Immun. 2007;75:4942–50. doi: 10.1128/IAI.01762-06. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Karaolis DK, Means TK, Yang D, Takahashi M, Yoshimura T, Muraille E, et al. Bacterial c-di-GMP is an immunostimulatory molecule. J Immunol. 2007;178:2171–81. doi: 10.4049/jimmunol.178.4.2171. [DOI] [PubMed] [Google Scholar]
  • 12.Chen W, Kuolee R, Yan H. The potential of 3′,5′-cyclic diguanylic acid (c-di- GMP) as an effective vaccine adjuvant. Vaccine. 2010;28:3080–5. doi: 10.1016/j.vaccine.2010.02.081. [DOI] [PubMed] [Google Scholar]
  • 13.Burdette DL, Monroe KM, Sotelo-Troha K, Iwig JS, Eckert B, Hyodo M, et al. STING is a direct innate immune sensor of cyclic di-GMP. Nature. 2011;478:515–8. doi: 10.1038/nature10429. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Ishikawa H, Barber GN. STING is an endoplasmic reticulum adaptor that facilitates innate immune signalling. Nature. 2008;455:674–8. doi: 10.1038/nature07317. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Yin Q, Tian Y, Kabaleeswaran V, Jiang X, Tu D, Eck MJ, et al. Cyclic di-GMP sensing via the innate immune signaling protein STING. Mol Cell. 2012;46:735–45. doi: 10.1016/j.molcel.2012.05.029. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Barber GN. Innate immune DNA sensing pathways: STING, AIMII and the regulation of interferon production and inflammatory responses. Curr Opin Immunol. 2011;23:10–20. doi: 10.1016/j.coi.2010.12.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Racz P, Tenner K, Mero E. Experimental Listeria enteritis. I. An electron microscopic study of the epithelial phase in experimental listeria infection. Lab Invest. 1972;26:694–700. [PubMed] [Google Scholar]
  • 18.Rosen H, Gordon S. Monoclonal antibody to the murine type 3 complement receptor inhibits adhesion of myelomonocytic cells in vitro and inflammatory cell recruitment in vivo. J Exp Med. 1987;166:1685–701. doi: 10.1084/jem.166.6.1685. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Quispe-Tintaya W, Chandra D, Jahangir A, Harris M, Casadevall A, Dadachova E, et al. Nontoxic radioactive Listeriaat is a highly effective therapy against metastatic pancreatic cancer. Proc Natl Acad Sci U S A. 2013;110:8668–73. doi: 10.1073/pnas.1211287110. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Kim SH, Castro F, Paterson Y, Gravekamp C. High efficacy of a Listeria-based vaccine against metastatic breast cancer reveals a dual mode of action. Cancer Res. 2009;69:5860–6. doi: 10.1158/0008-5472.CAN-08-4855. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.De Plaen E, De Backer O, Arnaud D, Bonjean B, Chomez P, Martelange V, et al. A new family of mouse genes homologous to the human MAGE genes. Genomics. 1999;55:176–84. doi: 10.1006/geno.1998.5638. [DOI] [PubMed] [Google Scholar]
  • 22.Park JW, Kwon TK, Kim IH, Sohn SS, Kim YS, Kim CI, et al. A new strategy for the diagnosis of MAGE-expressing cancers. J Immunol Methods. 2002;266:79–86. doi: 10.1016/s0022-1759(02)00105-9. [DOI] [PubMed] [Google Scholar]
  • 23.Paterson Y, Maciag PC. Listeria-based vaccines for cancer treatment. Curr Opin Mol Ther. 2005;7:454–60. [PubMed] [Google Scholar]
  • 24.Paterson Y, Ikonomidis G. Recombinant Listeria monocytogenes cancer vaccines. Curr Opin Immunol. 1996;8:664–9. doi: 10.1016/s0952-7915(96)80083-5. [DOI] [PubMed] [Google Scholar]
  • 25.Aslakson CJ, Miller FR. Selective events in the metastatic process defined by analysis of the sequential dissemination of subpopulations of a mouse mammary tumor. Cancer Res. 1992;52:1399–405. [PubMed] [Google Scholar]
  • 26.Singh R, Dominiecki ME, Jaffee EM, Paterson Y. Fusion to Listeriolysin O and delivery by Listeria monocytogenes enhances the immunogenicity of HER-2/neu and reveals subdominant epitopes in the FVB/N mouse. J Immunol. 2005;175:3663–73. doi: 10.4049/jimmunol.175.6.3663. [DOI] [PubMed] [Google Scholar]
  • 27.Gunn GR, Zubair A, Peters C, Pan ZK, Wu TC, Paterson Y. Two Listeria monocytogenes vaccine vectors that express different molecular forms of human papilloma virus-16 (HPV-16) E7 induce qualitatively different T cell immunity that correlates with their ability to induce regression of established tumors immortalized by HPV-16. J Immunol. 2001;167:6471–9. doi: 10.4049/jimmunol.167.11.6471. [DOI] [PubMed] [Google Scholar]
  • 28.Karaolis DK, Cheng K, Lipsky M, Elnabawi A, Catalano J, Hyodo M, et al. 3′,5′-Cyclic diguanylic acid (c-di-GMP) inhibits basal and growth factor-stimulated human colon cancer cell proliferation. Biochem Biophys Res Commun. 2005;329:40–5. doi: 10.1016/j.bbrc.2005.01.093. [DOI] [PubMed] [Google Scholar]
  • 29.Letourneur F, Gabert J, Cosson P, Blanc D, Davoust J, Malissen B. A signaling role for the cytoplasmic segment of the CD8 alpha chain detected under limiting stimulatory conditions. Proc Natl Acad Sci U S A. 1990;87:2339–43. doi: 10.1073/pnas.87.6.2339. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Castro F, Leal B, Denny A, Bahar R, Lampkin S, Reddick R, et al. Vaccination with Mage-b DNA induces CD8 T-cell responses at young but not old age in mice with metastatic breast cancer. Br J Cancer. 2009;101:1329–37. doi: 10.1038/sj.bjc.6605329. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Kim TS, Kang BY, Cho D, Kim SH. Induction of interleukin-12 production in mouse macrophages by berberine, a benzodioxoloquinolizine alkaloid, deviates CD4+ T cells from a Th2 to a Th1 response. Immunology. 2003;109:407–14. doi: 10.1046/j.1365-2567.2003.01673.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Gomez-Cabrero A, Wrasidlo W, Reisfeld RA. IMD-0354 targets breast cancer stem cells: a novel approach for an adjuvant to chemotherapy to prevent multidrug resistance in a murine model. PloS one. 2013;8:e73607. doi: 10.1371/journal.pone.0073607. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Romling U, Galperin MY, Gomelsky M. Cyclic di-GMP: the first 25 years of a universal bacterial second messenger. Microbiol Mol Biol Rev. 2013;77:1–52. doi: 10.1128/MMBR.00043-12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Fuertes MB, Kacha AK, Kline J, Woo SR, Kranz DM, Murphy KM, et al. Host type I IFN signals are required for antitumor CD8+ T cell responses through CD8{alpha}+ dendritic cells. J Exp Med. 2011;208:2005–16. doi: 10.1084/jem.20101159. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS604210-supplement-1.pdf (114.6KB, pdf)
10
NIHMS604210-supplement-10.pdf (146.1KB, pdf)
11
NIHMS604210-supplement-11.pdf (165KB, pdf)
12
NIHMS604210-supplement-12.docx (89.9KB, docx)
2
NIHMS604210-supplement-2.pdf (98.5KB, pdf)
3
NIHMS604210-supplement-3.pdf (77.5KB, pdf)
4
NIHMS604210-supplement-4.pdf (39.9KB, pdf)
5
NIHMS604210-supplement-5.pdf (140.5KB, pdf)
6
NIHMS604210-supplement-6.pdf (164.4KB, pdf)
7
NIHMS604210-supplement-7.pdf (36.6KB, pdf)
8
NIHMS604210-supplement-8.pdf (287KB, pdf)
9
NIHMS604210-supplement-9.pdf (98.2KB, pdf)

RESOURCES