Skip to main content
PMC home page
eLife logoLink to eLife
. 2018 Dec 11;7:e39944. doi: 10.7554/eLife.39944

Replication Study: Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET

Jeewon Kim 1, Amirali Afshari 2,†, Ranjita Sengupta 2,†, Vittorio Sebastiano 1,3,4, Archana Gupta 2, Young H Kim 2; Reproducibility Project: Cancer Biology✉
PMCID: PMC6289570  PMID: 30526855

Abstract

As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Lesnik et al., 2016) that described how we intended to replicate selected experiments from the paper ‘Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET’ (Peinado et al., 2012). Here we report the results. We regenerated tumor cells stably expressing a short hairpin to reduce Met expression (shMet) using the same highly metastatic mouse melanoma cell line (B16-F10) as the original study, which efficiently downregulated Met in B16F10 cells similar to the original study (Supplementary Figure 5A; Peinado et al., 2012). Exosomes from control cells expressed Met, which was reduced in exosomes from shMet cells; however, we were unable to reliably detect phosphorylated Met in exosomes. We tested the effect of exosome-dependent Met signaling on primary tumor growth and metastasis. Similar to the results in the original study, we did not find a statistically significant change in primary tumor growth. Measuring lung and femur metastases, we found a small increase in metastatic burden with exosomes from control cells that was diminished when Met expression was reduced; however, while the effects were in the same direction as the original study (Figure 4E; Peinado et al., 2012), they were not statistically significant. Differences between the original study and this replication attempt, such as level of knockdown efficiency, cell line genetic drift, sample sizes, study endpoints, and variability of observed metastatic burden, are factors that might have influenced the outcomes. Finally, we report meta-analyses for each result.

Research organism: Mouse

Introduction

The Reproducibility Project: Cancer Biology (RP:CB) is a collaboration between the Center for Open Science and Science Exchange that seeks to address concerns about reproducibility in scientific research by conducting replications of selected experiments from a number of high-profile papers in the field of cancer biology (Errington et al., 2014). For each of these papers a Registered Report detailing the proposed experimental designs and protocols for the replications was peer reviewed and published prior to data collection. The present paper is a Replication Study that reports the results of the replication experiments detailed in the Registered Report (Lesnik et al., 2016) for a paper by Peinado et al., and uses a number of approaches to compare the outcomes of the original experiments and the replications.

In 2012, Peinado et al. reported that exosomes isolated from highly metastatic murine melanoma cells (B16-F10) increased the metastatic burden of the primary tumors compared to exosomes from poorly metastatic melanomas (B16-F1) or control liposomes. Hepatocyte growth factor receptor (Met) was identified as a highly expressed protein in B16-F10 exosomes. Reduction of Met, and phosphorylated Met (pMet) by shRNA in B16-F10 exosomes resulted in a statistically significant decrease in lung and bone metastatic burden compared to controls (Peinado et al., 2012). Exosomes derived from melanoma cells were proposed to promote metastasis by education of bone marrow-derived cells through horizontal transfer of exosomal Met in order to prime the pre-metastatic niche and increase vascularization (Peinado et al., 2012).

The Registered Report for the paper by Peinado et al. described the experiments to be replicated (Figure 4E and Supplementary Figures 1C and 5A), and summarized the current evidence for these findings (Lesnik et al., 2016). Since that publication, additional studies have reported in different models that tumor derived exosomes administered to mice prior to tumor cell injection increased metastatic burden by inducing pre-metastatic niche formation (Costa-Silva et al., 2015; Fong et al., 2015; Hoshino et al., 2015; Liu et al., 2016; Plebanek et al., 2017; Zhou et al., 2014). Additionally, using the same tumor model as Peinado et al., 2012, Met expression was found to be heterogeneous in B16-F10 cells, with higher lung metastatic burden, and lower primary tumor burden, in cells expressing high levels of Met compared to cells expressing low levels of Met (Adachi et al., 2016). Injection of exosomes from high Met expressing cells increased the metastatic burden of low Met expressing cells (Adachi et al., 2016). The molecules present on tumor-derived exosomes, such as the specific repertoire of integrins, are important in dictating metastatic organotropism (Hoshino et al., 2015). There have been numerous studies that have aimed to identify exosomal cargo (e.g. proteins) (Keerthikumar et al., 2016). Similar to the study by Peinado et al., 2012, other studies, using various techniques (e.g. proteomic profiling, reverse phase protein array), have identified MET in exosomes from melanoma cells (Lazar et al., 2015; Steenbeek et al., 2018) as well as in hepatocellular carcinoma cells (He et al., 2015), neuroblastoma cells (Keerthikumar et al., 2015), ovarian cancer cells (Liang et al., 2013), sera from healthy donors and prostate cancer patients (Cannistraci et al., 2017), 293 T cells (Li et al., 2016), and as a fusion protein (PTPRZ1-MET) in glioblastoma cells (Zeng et al., 2017), while other studies did not identify MET in exosomes from breast cancer patients (Chen et al., 2017) nor in some melanoma cells (Lazar et al., 2015).

The outcome measures reported in this Replication Study will be aggregated with those from the other Replication Studies to create a dataset that will be examined to provide evidence about reproducibility of cancer biology research, and to identify factors that influence reproducibility more generally.

Results and discussion

Generation and characterization of shMet B16-F10 cells and exosomes

To test the effect exosome-dependent Met signaling has on primary tumor growth and metastasis, we used the same highly metastatic mouse melanoma cell line (B16-F10) and the same lentiviral system as the original study to make B16-F10 cells stably expressing an shRNA targeting Met (shMet) or a control shRNA (shScr) using the same targeting sequences as the original study. The experimental approach to generate and characterize the stable cells and isolated exosomes was described in Protocol 1 and 2 of the Registered Report (Lesnik et al., 2016). We tested various multiplicity of infection (MOI) ratios, all of which displayed expression of the shRNA with corresponding decreased Met and Met levels in shMet cells compared to shScr cells (Figure 1—figure supplement 1). We planned to utilize cells generated with an MOI of 10, similar to the original study, but observed that the Met levels in the shScr cells at this MOI were, for unknown reasons, decreased when compared to the shScr cells generated at the other MOI ratios (Figure 1—figure supplement 1C). Thus, we proceeded with the stable cells generated with an MOI of 20, which had 22.6% Met expression, and 25.1% phosphorylated Met (pMet) expression in the shMet cells relative to shScr cells (Figure 1A–C). The stable cell lines generated in the original study were reported to have 64.1% Met expression and 23.4% pMet expression in the shMet cells relative to shScr cells (Peinado et al., 2012).

Figure 1. Characterization of shMet B16-F10 cells and exosomes.

B16-F10 cells engineered to express shScr or shMet were used to purify exosomes. (A) Representative Western blots of exosomes and B16-F10 cells expressing the indicated shRNA were probed with antibodies specific for total Met (top panel) and Gapdh (bottom panel). Membranes were cut at ~75 kDa so that Met and Gapdh could be probed in parallel. Repeat indicates the number of independently isolated exosome and cell lysate preparations from the same batch of infected cells. The fourth lane, labeled ‘Cells’ are lysate from B16-F10 cells expressing shScr. (B) Representative Western blots of exosomes and B16-F10 cells expressing the indicated shRNA were probed with antibodies specific for phosphorylated (Tyr 1234/1235) Met (pMet) (top panel) and Gapdh (bottom panel). Membranes were cut at ~75 kDa so that pMet and Gapdh could be probed in parallel. Repeat indicates the number of independently isolated exosome preparations from the same batch of infected cells. (C) Western blot bands were quantified for cells. Met or pMet levels were normalized to Gapdh, and protein expression presented relative to shScr conditions. Expression level of shScr condition was assigned a value of 1. Means reported and error bars represent SD. Results are from 3 independent biological repeats for Met expression and 4 independent biological repeats for pMet expression. Exploratory analysis: one-sample t-test on Met levels (Met/Gapdh) in shMet cells compared to a constant of 1 (shScr cells): t(2) = 8.41, p = 0.014, Bonferroni corrected p = 0.028, Cohen’s d = 4.85, 95% CI [0.55, 9.43]; one-sample t-test on pMet levels (Met/Gapdh) in shMet cells compared to a constant of 1 (shScr cells): t(3) = 8.94, p = 0.003, Bonferroni corrected p = 0.006, Cohen’s d = 4.47, 95% CI [1.02, 8.01]. (D) Representative Western blot of exosomes isolated from cells expressing the indicated shRNA probed with exosome markers Hsc70, Tsg101, and Cd63 specific antibodies. Experiment performed on 3 independent biological repeats for each condition from the same batch of infected cells. Additional details for this experiment can be found at https://osf.io/aqm2m/.

Figure 1.

Figure 1—figure supplement 1. Multiplicity of infection (MOI) ratios tested for stable cell line generation.

Figure 1—figure supplement 1.

B16-F10 cells infected with various MOI ratios were characterized for shMet, Met, and Met expression. (A) Relative expression levels of shMet (normalized to U6) was determined by qRT-PCR for each cell line. For each MOI tested, fold change in shMet expression in shMet cells relative to shScr cells was determined. Expression level of shScr cells was assigned a value of 1, which is indicated by the dashed line. Means reported from one biological repeat. (B) Relative expression levels of Met (normalized to Gapdh) was determined by qRT-PCR for each cell line. For each MOI tested, fold change in Met expression in shMet cells relative to shScr cells was determined. Expression levels of shScr cells was assigned a value of 1, which is indicated by the dashed line. Means reported from one biological repeat. (C) Western blots using anti-Met (top panel) and anti-Gapdh (bottom panel) antibodies. Membranes were cut at ~75 kDa so that Met and Gapdh could be probed in parallel. Repeat indicates the number of independently isolated cell lysate preparations from the same batch of infected cells. Additional details for this experiment can be found at https://osf.io/aqm2m/.
Open in a new tab

Using the same ultracentrifugation approach as the original study, we isolated exosomes from shScr and shMet cells. We confirmed the presence of the known exosome markers Hsc70, Tsg101, and Cd63 (Figure 1D) and found exosome number and size distribution quantified by NanoSight analyses to be similar between the two groups (Table 1). To test if Met and pMet were expressed in B16-F10 exosomes we performed Western blots and found that compared to shScr exosomes the amount of Met expressed in shMet exosomes was decreased (Figure 1A); however, we were unable to reliably detect pMet expression in exosomes (Figure 1B). Importantly, this was not due to the inability of the antibody to detect pMet by Western blot, since pMet was detected in cell lysates run on the same gel. Additionally, as a preventative measure to block any residual phosphatase activity, we added inhibitors when preparing the isolated exosomes for electrophoresis. However, if there were carryover phosphatases present during isolation of exosomes by ultracentrifugation the pMet levels could have been diminished before the inhibitors were added. Furthermore, the exosomes were prepared for electrophoresis after first being stored at −20˚C, instead of immediately after isolation, which might be critical for the detection of phospho-protein detection in exosomes. Inclusion of untransduced B16-F10 controls could have also indicated if unintended changes occurred in the stable cell lines utilized, especially given the potential impact puromycin selection can have on transcriptome profiles (Guo et al., 2017), and should be considered in the experimental design of future studies. The original study reported Met and pMet expression by Western blot in B16-F10 exosomes; however, the level of knockdown achieved in shMet exosomes was not reported. To summarize, we observed reduced Met and pMet expression in shMet cells compared to shScr cells, consistent with the stable cells reported in the original study; however, while we detected Met in exosomes from shScr cells, that was reduced in exosomes from shMet cells, we were unable to reliably detect pMet expression in isolated exosomes.

Table 1. Nanosight analysis of exosomes.

Summary of exosome number and size distribution, with or without the finite track length adjustment (FTLA) algorithm, quantified by NanoSight analysis. All values (mean, SD, median, span, concentration (particles/ml)) are given as averages for preps generated during this study (n = 15 per cell line).

Mean Median Sd Span Particles/mL
shMet FTLA size distribution 91.07 72.9 62.89 1.253 6.253e+11
size distribution 91 75 65.44 1.334 6.253e+11
shScr FTLA size distribution 88.47 71.8 56.04 1.211 9.477e+11
size distribution 88.67 73.67 61.8 1.321 9.477e+11
Open in a new tab

Exosome-dependent Met signaling on primary tumor growth and metastasis

We next used shScr and shMet exosomes to replicate an experiment that tested whether exosome-dependent Met signaling impacted primary tumor growth and metastasis. Synthetic unilamellar 100 nm liposomes were used as a control to test if tumor exosomes enhance metastasis. This experiment is similar to what was reported in Figure 4E of Peinado et al., 2012 and described in Protocol 3 in the Registered Report (Lesnik et al., 2016). Freshly isolated exosomes from shScr or shMet cells, or synthetic liposomes, were injected into C57BL/6 female mice three times a week for a total of four weeks, thereafter we implanted B16-F10 tumor cells engineered to express luciferase (B16-F10-luc). The planned study design involved waiting 21 days after B16-F10-luc tumor cell implantation before sacrificing the mice for analysis; however, two animals were found dead before this time point was reached (17 days after implantation), and of the surviving mice, the largest tumors reached >1 cm3 prompting us to stop the experiment early (18 days after implantation). We measured primary tumor growth during the length of the study and observed increased growth among all the conditions (Figure 2A, Figure 2—figure supplement 1A–D). Interestingly, the primary tumors in mice injected with shScr exosomes were on average smaller than the primary tumors in mice injected with shMet or synthetic liposomes. This was also observed when measuring the weight of the dissected primary tumors at the end of the study (Figure 2B). There are a number of factors that can affect tumor growth, such as availability of nutrients, oxygen, and space that influence initial and continued growth of the tumor (Cornelis et al., 2013; Talkington and Durrett, 2015). There was not a statistically significant difference between the tumor weights of the three groups (Kruskal-Wallis: H(2) = 2.85, p=0.24). Additionally, we conducted two planned comparisons (shScr vs shMet; shScr vs synthetic liposomes), which were not statistically significant (see Figure 2 figure legend). This is consistent with the original study that stated no differences in primary tumor growth between the three groups; however, those data were not shown, preventing direct comparison of the original study results and the results from this replication attempt.

Figure 2. Primary tumor growth and metastatic burden of mice injected with exosomes.

Female C57BL/6 mice were subcutaneously injected with B16-F10-luciferase cells after 4 weeks of intravenous injections (3 times a week) of shScr exosomes, shMet exosomes, or synthetic liposomes. (A) Following primary tumor detection caliper measurements were taken 3 times a week and used to calculate tumor volume. Line graph of tumor volume (y-axis is natural log scale) with means reported and error bars representing s.e.m. Number of mice monitored per group: synthetic liposomes = 7, shScr = 7, shMet = 7. Of note, not all mice had detectable primary tumors at the first measurement, one of the mice injected with synthetic liposomes never formed a detectable primary tumor despite the presence of metastatic burden, and two mice (one injected with B16-F19 shScr exosomes and one injected with synthetic liposomes) were found dead before the end of the study (day 17). Individual mouse tumor volume data is reported in Figure 2 - figure supplement 1. (B) At the end of the experiment (day 18), primary tumors were excised and weighed. Dot plot with means reported as crossbars and error bars represent s.e.m. Number of primary tumor weights per group: synthetic liposomes = 5, shScr = 6, shMet = 7. Kruskal-Wallis test on all three groups: H(2) = 2.85, p = 0.24. Planned Wilcoxon-Mann-Whitney comparison between shScr and synthetic liposomes: U = 22, p = 0.247, Cliff’s d = 0.47, 95% CI [-0.28, 0.86]. Planned Wilcoxon-Mann-Whitney comparison between shScr and shMet: U = 32, p = 0.138, Cliff’s d = 0.52, 95% CI [-0.10, 0.85]. (C) Metastatic burden in lungs quantified by luciferin photon flux at 18 days after B16-F10-luc tumor cell injections. Box and whisker plot (y-axis is natural log scale) with median represented as the line through the box and whiskers representing values within 1.5 IQR of the first and third quartile. Individual data points represented as dots. Number of mice per group: synthetic liposomes = 6, shScr = 6, shMet = 7. One-way ANOVA on the luciferin photon flux values (natural log-transformed); F(2,16) = 0.226, uncorrected p = 0.800 with a priori alpha level of 0.025, Bonferroni corrected p > 0.99. Planned contrast between shScr and synthetic liposomes; Fisher’s LSD test; t(16) = 0.543, p = 0.594 with a priori alpha level of 0.025, Bonferroni corrected p > 0.99, Cohen’s d = 0.31, 95% CI [-0.83, 1.45]. Planned contrast between shScr and shMet; Fisher’s LSD test; t(16) = 0.620, p = 0.544 with a priori alpha level of 0.025, Bonferroni corrected p > 0.99, Cohen’s d = 0.34, 95% CI [-0.76, 1.44]. (D) Metastatic burden in femurs quantified by luciferin photon flux at 18 days after B16-F10-luc tumor cell injections. Box and whisker plot (y-axis is natural log scale) with median represented as the line through the box and whiskers representing values within 1.5 IQR of the first and third quartile. Individual data points represented as dots. Number of mice per group: synthetic liposomes = 6, shScr = 6, shMet = 7. One-way ANOVA on the luciferin photon flux values (natural log-transformed); F(2,16) = 0.190, uncorrected p = 0.829 with a priori alpha level of 0.025, Bonferroni corrected p > 0.99. Planned contrast between shScr and synthetic liposomes; Fisher’s LSD test; t(16) = 0.573, p = 0.575 with a priori alpha level of 0.025, Bonferroni corrected p > 0.99, Cohen’s d = 0.33, 95% CI [-0.82, 1.46]. Planned contrast between shScr and shMet; Fisher’s LSD test; t(16) = 0.105, p = 0.918 with a priori alpha level of 0.025, Bonferroni corrected p > 0.99, Cohen’s d = 0.06, 95% CI [-1.03, 1.15]. Additional details for this experiment can be found at https://osf.io/mzywk/.

Figure 2.

Figure 2—figure supplement 1. Alternative visualizations of tumor growth and metastatic burden.

Figure 2—figure supplement 1.

This is the same experiment as Figure 2. (A-C) Line graphs (y-axis is natural log scale) of primary tumor volume plotted for each animal rather than averages. Number of mice same as Figure 2A. (D) Line graph of primary tumor volume data plotted on a linear scale with means reported and error bars representing s.e.m. Number of mice same as Figure 2A. (E) Lung metastatic burden data or (F) femur metastatic burden data with means reported and error bars representing s.e.m. Number of mice same as Figure 2C,D. Of note, although the data are not normal on a linear scale, which means presenting mean and s.e.m. is not appropriate, these box plots are presented to allow a direct comparison to how the original study data were presented. Additional details for this experiment can be found at https://osf.io/mzywk/.
Open in a new tab

Metastatic burden was quantified by luciferin photon flux in the lungs and femurs of the mice from each group at the end of the study. Mice injected with the shScr exosomes achieved an average of 1.0×105 photons/sec in the lungs, which was reduced to 4.2×104 photons/sec in mice injected with shMet exosomes and 5.3×104 photons/sec in mice injected with synthetic liposomes (Figure 2—figure supplement 1E). Thus, the group means of the natural log transformed lung metastatic burden data were 10.8, 10.4, and 10.5 log units, respectively (Figure 2C). On average, there was a 36.9% natural log based percentage difference in lung metastatic burden between mice injected with shScr exosomes and shMet exosomes, and a 33.6% difference between mice injected with shScr and synthetic liposomes. To test if shMet exosomes or synthetic liposomes had decreased metastatic burden compared to shScr exosomes, we performed a one-way ANOVA on the luciferin photon flux values (natural log-transformed), which was not statistically significant (F(2,16) = 0.226, uncorrected p=0.800, Bonferroni corrected p>0.99). Additionally, we conducted two planned comparisons (shScr vs synthetic liposomes; shScr vs shMet), which were not statistically significant (see Figure 2 figure legend). The original study reported metastatic burden in the lungs of mice injected with shScr exosomes [M = 5.1×104 photons/sec; 10.8 log units] were reduced by 165.3% [M = 1.3×104 photons/sec; 9.1 log units] and 80.0% [M = 2.3×104 photons/sec; 10.0 log units] with shMet exosomes and synthetic liposomes, respectively (Peinado et al., 2012). The range of metastatic lung values reported in the original study had a relative standard deviation (RSD) that were smaller (shScr = 2.9%; shMet = 9.9%; synthetic liposome = 3.8%) than the RSD observed in this replication attempt (shScr = 13%; shMet = 7.1%; synthetic liposome = 9.7%), particularly among the shScr and synthetic liposome groups.

For femurs, the average metastatic burden of mice injected with the shScr exosomes achieved an average of 7.4×105 photons/sec (12.2 log units), which was reduced to 3.3×105 photons/sec (12.1 log units) in mice injected with shMet exosomes and 1.9×105 photons/sec (11.7 log units) in mice injected with synthetic liposomes (Figure 2D, Figure 2—figure supplement 1F). So, on average, there was a 7.8% natural log based percentage difference in femur metastatic burden between mice injected with shScr exosomes and shMet exosomes, and a 44.3% difference between mice injected with shScr and synthetic liposomes, which was not statistically significant (see Figure 2 figure legend). While the original study reported metastatic burden in the femurs of mice injected with shScr exosomes [M = 4.5×104 photons/sec; 10.4 log units], there was no reported metastatic burden in the femurs of mice injected with shMet exosomes or synthetic liposomes (Peinado et al., 2012). Additionally, similar to the lung values, the RSD for the metastatic femur values were smaller in the original study (shScr = 8.4%; shMet = 0%; synthetic liposome = 0%) than this replication attempt (shScr = 14%; shMet = 10%; synthetic liposome = 8.4%).

The higher RSD observed in this study is one of the factors that could influence if statistical significance is reached, particularly since the sample size of this replication attempt was determined a priori to detect the effect based on the originally reported data. Importantly, these results should take into consideration the experimental endpoint of this replication attempt, which was shorter than the original study and what was indicated in the Registered Report (18 days instead of 21 days), which could account for less macro-metastasis. Also, as mentioned above, inclusion of untransduced B16-F10 controls could have indicated if unintended changes to the cargo of exosomes from the stable cell lines utilized occurred and should be considered in the experimental design of future studies. To summarize, for this experiment we found results that were in the same direction as the original study and not statistically significant.

Meta-analysis of original and replication effects

We performed a meta-analysis using a random-effects model, where possible, to combine each of the effects described above as pre-specified in the confirmatory analysis plan (Lesnik et al., 2016). To provide a standardized measure of the effect, a common effect size was calculated for each effect from the original and replication studies. Cliff’s delta (d) is a non-parametric estimate of effect size that measures how often a value in one group is larger than the values from another group. It is used in this case because of violations to the assumptions of normality and equal variance in the original and replication studies. Importantly, the confidence interval around Cliff’s d is asymmetric, while the p value is calculated using the normal distribution and is thus not well defined; however, there is no agreement on how to compute p values from an asymmetric distribution (Dunne et al., 1996; Rohatgi and Saleh, 2000). The estimate of the effect size of one study, as well as the associated uncertainty (i.e. confidence interval), compared to the effect size of the other study provides another approach to compare the original and replication results (Errington et al., 2014; Valentine et al., 2011). Importantly, the width of the confidence interval for each study is a reflection of not only the confidence level (e.g. 95%), but also variability of the sample (e.g. SD) and sample size.

There were a total of four comparisons made with the metastatic burden data from lungs and femurs of mice injected with shScr exosomes, shMet exosomes, or synthetic liposomes (Figure 3). For all comparisons, the results of the original study and this replication were consistent when considering the direction of the effect; however, the point estimate of the replication effect size was not within the confidence interval of the original result, and vice versa. The meta-analyses were not statistically significant for any of the effects (see Figure 3 figure legend). Furthermore, the large confidence intervals of the meta-analyses along with statistically significant Cochran’s Q tests (lung: shScr and liposomes, p=0.045; shScr and shMet, p=0.020; femur: shScr and liposomes, p=0.0084; shScr and shMet, p=0.0046) suggest heterogeneity between the original and replication studies.

Figure 3. Meta-analyses of each effect.

Figure 3.

Open in a new tab

Effect size (Cliff’s delta) and 95% confidence interval are presented for Peinado et al., 2012, this replication attempt (RP:CB), and a meta-analysis to combine the two effects. Cliff’s delta is a standardized measure of how often a value in one group is larger than the values from another group, with a larger positive value indicating a decrease in metastatic burden in animals injected with either shMet exosomes or liposomes compared to animals injected with shScr exosomes. Random effects meta-analysis: lung metastatic burden (shScr v. liposome, p = 0.149; shScr v. shMet, p = 0.226) and femur metastatic burden (shScr v. liposome, p = 0.264; shScr v. shMet, p = 0.317). Sample sizes used in Peinado et al., 2012 and this replication attempt are reported under the study name. Additional details for these meta-analyses can be found at https://osf.io/c69jx/.

This direct replication provides an opportunity to understand the present evidence of these effects. Any known differences, including reagents and protocol differences, were identified prior to conducting the experimental work and described in the Registered Report (Lesnik et al., 2016). However, this is limited to what was obtainable from the original paper and through communication with the original authors, which means there might be particular features of the original experimental protocol that could be critical, but unidentified. So while some aspects, such as cell line, mouse strain, and technique for exosome isolation were maintained, others were changed during the execution of the replication that could affect results, such as the time from cell injection after exosome education until euthanasia, which was shorter in this replication attempt than what was conducted in the original study (18 days instead of 21 days). Additionally, others were unknown or not easily controlled for. These include variables such as cell-intrinsic changes in Met expression and the associated gene expression profiles (Adachi et al., 2016), cell line genetic drift (Ben-David et al., 2018; Hughes et al., 2007; Kleensang et al., 2016), non-random genetic/transcriptional drift in the heterogeneous stable cells (Guo et al., 2017; Shearer and Saunders, 2015), genetic heterogeneity of mouse inbred strains (Casellas, 2011), the microbiome of recipient mice (Macpherson and McCoy, 2015), housing temperature in mouse facilities (Kokolus et al., 2013), and lot variability and quality of key reagents, such as the lentiviral particles (Leek et al., 2010). Environmental differences such as husbandry staff, bedding type and source, light levels, and other intangibles, all of which, by necessity, differed between the studies also affect experimental outcomes with mice (Howard, 2002; Jensen and Ritskes-Hoitinga, 2007; Nevalainen, 2014; Sorge et al., 2014). The difference in sample sizes between the studies is also a factor that could limit the opportunity to detect a statistically significant difference, especially considering the higher variability observed in this replication attempt, as described above. The difference in achieved knockdown of Met in exosomes between the original study and this replication attempt is another important factor to consider. A higher level of knockdown might be required to yield a given phenotype with this experimental design. Importantly, observing and reporting all outcomes are informative to establish the range of conditions under which a given phenotype can be observed (Bailoo et al., 2014). Whether these or other factors influence the outcomes of this study is open to hypothesizing and further investigation, which is facilitated by direct replications and transparent reporting.

Materials and methods

Key resources table.

Reagent type
(species) or
resource		 Designation Source or
reference		 Identifiers Additional
information
Strain, strain
background
(Mus musculus,
C57BL/6, female)		 C57BL/6 Charles River Strain code:
027; RRID:
IMSR_CRL:27
Cell line
(M. musculus)		 B16-F10 doi:10.1038/nm.2753 RRID:
CVCL_0159
Cell line
(M. musculus)		 B16-F10-luc doi:10.1038/nm.2753 expresses
firefly
luciferase
Other synthetic
uniloamellar
100 nm liposomes		 Encapsula
NanoSciences		 13 mg/ml
L-a-Phosplatydilcholine,
2.78 mg/ml cholesterol
(7:3 molar ratio P:C)
Sequence-based
reagent		 pGIPZ
non-silencing
shRNA lentiviral
control particles		 Dharmacon/GE Life Sciences cat# RHS4348 sense sequence:
5’-ATCTCGCTTGGGCGAGAGTAAG-3’
Sequence-based

reagent		 pGIPZ mouse
Met shRNA
lentiviral particles		 Dharmacon/
GE Life Sciences		 cat# VGM5520-
200377256;
clone ID: V3LMM_
456078		 sense sequence:
5’-CCAGACTTTTCATACAAGA-3’
Antibody mouse
anti-Hsc70		 Enzo Life
Sciences		 cat#
ALX-804–067; RRID:
AB_10538284 		1:500 dilution in
SuperBlock, O/N at 4 ˚C
Antibody mouse
anti-Tsg101		 Santa Cruz
Biotechnology		 cat# sc-7964; RRID:
AB_671392 		1:500 dilution
in SuperBlock,
O/N at 4 ˚C
Antibody rabbit
anti-Cd63		 System
Biosciences		 cat#
EXOAB-CD63A-1;
RRID:AB_2561274 		1:1000 dilution in
SuperBlock, O/N at 4 ˚C
Antibody mouse anti-Met Cell Signaling
Technology		 cat# 3127;
RRID:AB_331361 		1:1000 dilution in
SuperBlock, O/N at 4 ˚C
Antibody rabbit anti-pMet
(Tyr 1234/5)		 Cell Signaling
Technology		 cat# 3077;
RRID:AB_2315156 		1:1000 dilution in
SuperBlock, O/N at 4 ˚C
Antibody rabbit anti-Gapdh Santa Cruz
Biotechnology		 cat# sc-25778;
RRID:AB_10167668 		1:1000 dilution in
SuperBlock, O/N at 4 ˚C
Antibody rabbit anti-pMet
(Tyr 1234/5)		 Thermo Fisher
Scientific		 cat# MA5-15083;
RRID:AB_10983015 		1:1000 dilution
in SuperBlock,
O/N at 4 ˚C
Antibody HRP-conjugated
sheep anti-mouse		 GE Healthcare cat# NA931;
RRID:AB_772210 		1:15,000 dilution in
SuperBlock, 1 hr at RT
Antibody HRP-conjugated donkey anti-rabbit GE Healthcare cat# NA934;
RRID:AB_772206 		1:15,000 dilution in
SuperBlock, 1 hr at RT
Sequence-based
reagent		 shMet primer this paper Forward: 5’-CCAGACTTT
TCATACAAGAATA-3’
Sequence-based
reagent		 universal
reverse primer		 QuantiMir Kit,
System Biosciences		 cat# RA420A-1
Sequence-based reagent U6 primer LncRNA Profiler
qPCR Array Kit,
System Biosciences		 cat# RA900A-1
Sequence-based
reagent		 Met primer this paper Forward: 5’-CTGGACA
GATTGTGGGAGTAAG −3’
Sequence-based
reagent		 Gapdh primer LncRNA Profiler qPCR Array Kit, System Biosciences cat# RA900A-1
Software,
algorithm		 ImageJ doi:10.1038/nmeth.2089 RRID:SCR_003070 version 1.49 u
Software,
algorithm		 SDS Applied
Biosystems		 RRID:SCR_015806 version 2.4
Software,
algorithm		 Nanoparticle
tracking Analysis
(NTA)		 Nanosight RRID:SCR_014239 version 2.3
Software,
algorithm		 Living Image Caliper Life Sciences RRID:SCR_014247 version 4.4
Software,
algorithm		 R Project for
statistical
computing		 https://www.r-project.org RRID:SCR_001905 version 3.5.1
Open in a new tab

As described in the Registered Report (Lesnik et al., 2016), we attempted a replication of the experiments reported in Figure 4E and Supplementary Figures 1C and 5A of Peinado et al., 2012. A detailed description of all protocols can be found in the Registered Report (Lesnik et al., 2016) and are described below with additional information not listed in the Registered Report, but needed during experimentation.

Cell culture

B16-F10 (RRID:CVCL_0159) and B16-F10-luciferase (expressing firefly luciferase) (shared by Dr. David Lyden, Weill Medical College of Cornell University) were maintained in DMEM (Thermo Fisher Scientific, cat# MT10013CV) supplemented with 10% exosome-depleted fetal bovine serum (FBS) and 100 U/ml penicillin-streptomycin (Thermo Fisher Scientific, cat# 15-140-122) at 37°C in a humidified atmosphere at 5% CO2. FBS (Hyclone, cat# SH30088.03) was depleted of bovine exosomes by ultracentrifugation (Beckman Coulter, Optima L-90K) at 100,000xg (Beckman Coulter, Type 70 Ti Rotor) for 70 min at 4°C. Quality control data for both cell lines are available at https://osf.io/3x58z/. This includes results confirming the cell lines were free of mycoplasma contamination as well as STR DNA profiling of the cell lines (B16-F10 cells: DDC Medical, Fairfield, Ohio; B16-F10-luciferase cells: IDEXX BioResearch, Columbia, Missouri), which were confirmed to be the indicated cell lines when queried against STR profile databases. Additionally, B16-F10-luciferase cells were confirmed free of common mouse pathogens (IDEXX BioResearch).

Stable cell generation

B16-F10 cells (2.5×104 cells/well of 24 well plate) grown in culture medium supplemented with 1X TransDux were transduced with either pGIPZ mouse Met shRNA lentiviral particles (Dharmacon/GE Life Sciences, cat# VGM5520-200377256, clone ID: V3LMM_456078, lot# V16042101, sense sequence: 5’-CCAGACTTTTCATACAAGA-3’) or pGIPZ non-silencing shRNA lentiviral control particles (Dharmacon/GE Life Sciences, cat# RHS4348, lot# 150529606, sense sequence: 5’-ATCTCGCTTGGGCGAGAGTAAG-3’) at MOIs of 10, 20, or 50, incubated overnight (16 hr), then growth medium was replaced. Transduction efficiency was measured by GFP expression 72 hr later and then transduced cells were selected with 1.5 µg/ml puromycin for 28 days. Cells resistant to puromycin were checked for GFP expression. Growth rates were similar among all stable cell populations generated based on qualitative observations documented during passaging. Cells were frozen down and stored for further use. Thawed cells were maintained in 1.5 µg/ml puromycin, except when cells were plated for experiments. Microscopy images of cells including laboratory notes on cell culturing of stable cell lines are available at https://osf.io/yp5g6/.

Exosome purification and tracking analysis

Supernatant from B16-F10 shMet and B16-F10 shScr cells collected 48–72 hr after initial plating (two 15 cm plates at ~5×106 cells/plate) were pelleted at 500xg (Beckman Coulter, Allegra X-14R) for 10 min at 4°C. The supernatant was clarified by ultracentrifugation (Beckman Coulter, Optima L-90K) at 20,000xg (Beckman Coulter, Type 70 Ti Rotor) for 20 min at 4°C. Exosomes were then collected by ultracentrifugation at 100,000xg for 70 min at 4°C. Protein concentration of exosomes were determined with a BCA assay using a standard curve (data available at https://osf.io/9xnbf/) and either suspended in PBS and used directly for experiments (in vivo animal experiments) or stored at −20°C until analysis (Western blots and Nanosight). Characterization of exosomes was performed with a nanoparticle analysis system (Nanosight, LM10) equipped with a blue laser (405 nm) and Nanoparticle tracking Analysis (NTA) software (RRID:SCR_014239), version 2.3. Summary of exosome number and size distribution, with or without the finite track length adjustment (FTLA) algorithm, are provided in Table 1 as averaged values among the multiple independent exosome preparations generated during this study. Span was calculated using the formula: Span = (D90-D10)/D50, where D90 is the point in the size distribution where 90% of the sample is contained, D10 is where 10% of the sample is contained, and D50 is where 50% of the sample is contained (i.e. median). Batch analysis reports, including short video clips, are available at https://osf.io/vczn7/ for exosomes purified for in vitro experiments and https://osf.io/9xnbf/ for exosomes purified for in vivo experiments.

Western blots

Cell lysate was generated from B16-F10 shMet and B16-F10 shScr cells by removing medium from plates, washing 2X with 1X PBS, followed by addition of 1X RIPA buffer (Sigma-Aldrich, cat# R0278) supplemented with protease inhibitors (Sigma-Aldrich, cat# 4693116001) and phosphatase inhibitors (Sigma-Aldrich, cat# P5726) at manufacturer recommended concentrations. Lysed cells were scraped from plates, incubated on ice for 15 min followed by transfer to −80°C. Cell lysates were thawed on ice, then centrifuged at 12,000xg at 4°C before protein concentration of the supernatant was quantified using a BCA protein assay kit following manufacturer’s instructions. Previously frozen purified exosomes used for Western blots were resuspended in 1X RIPA buffer supplemented with protease and phosphatase inhibitors. Laemmli sample buffer was added to the purified exosomes or cell lysates and incubated at 95˚C for 5 min. 30 µg of purified exosome or cell lysates (or 10 µg of purified exosomes for gels examining exosome markers), along with a protein ladder (Bio-Rad Laboratories, cat# 161–0377), was resolved by SDS-PAGE and transferred to PVDF membrane as described in the Registered Report (Lesnik et al., 2016). Membranes were blocked with SuperBlock (Thermo Fisher Scientific, cat# 37516) following manufacturer’s instructions. Membranes were cut at ~75 kDa to allow for parallel probing. Incubation with primary antibody, diluted in SuperBlock, was conducted overnight at 4˚C. Membranes were probed with: mouse anti-Hsc70 (Enzo Life Sciences, cat# ALX-804–067, RRID:AB_10538284), 1:500 dilution; mouse anti-Tsg101 (Santa Cruz Biotechnology, cat# sc-7964, RRID:AB_671392), 1:500 dilution; rabbit anti-Cd63 (System Biosciences, cat# EXOAB-CD63A-1, RRID:AB_2561274), 1:1000 dilution; mouse anti-Met (Cell Signaling Technology, cat# 3127, RRID:AB_331361), 1:1000 dilution; rabbit anti-pMet (Tyr 1234/5) (Cell Signaling Technology, cat# 3077, RRID:AB_2315156), 1:1000 dilution; rabbit anti-Gapdh (Santa Cruz Biotechnology, cat# sc-25778, RRID:AB_10167668), 1:1000 dilution. Incubations were followed by washes with 1X TBS supplemented with 0.1% tween (TBST) and then the appropriate secondary antibody diluted in SuperBlock for 1 hr at room temperature: HRP-conjugated sheep anti-mouse (GE Healthcare, cat# NA931, RRID:AB_772210), 1:15,000 dilution; HRP-conjugated donkey anti-rabbit (GE Healthcare, cat# NA934, RRID:AB_772206), 1:15,000 dilution. Membranes were washed with TBST and incubated with SuperSignal West Femto Maximum Sensitivity Substrate (Thermo Fisher Scientific, cat# 34095) according to the manufacturer’s instructions. Scanned Western blots were quantified using ImageJ software (RRID:SCR_003070), version 1.49u (Schneider et al., 2012). We also used an additional antibody targeting pMet (rabbit anti-pMet (Tyr 1234/5) (Thermo Fisher Scientific, cat# MA5-15083, RRID:AB_10983015), 1:1000 dilution) due to an inability to detect a reliable Western blot signal in exosomes with the pMet antibody (RRID:AB_331361) stated in the Registered Report and used in the original study. Both antibodies gave similar results (RRID:AB_10983015: https://osf.io/96tcs/; RRID:AB_331361: Figure 1B). Additional method details and image data are available at https://osf.io/aqm2m/.

Quantitative PCR

Total RNA was isolated from B16-F10 shMet and B16-F10 shScr cells using TRIzol reagent (Thermo Fisher Scientific, cat# 15596026) according to manufacturer’s instructions. Total RNA was reverse transcribed into cDNA using a QuantiMir Kit (System Biosciences, cat# RA420A-1) according to manufacturer’s instructions. To detect the shMet shRNA expression a primer against the shMet sequence (Forward: 5’-CCAGACTTTTCATACAAGAATA-3’) and a universal reverse primer (QuantiMir Kit, System Biosciences, cat# RA420A-1) were used along with a U6 specific primer (LncRNA Profiler qPCR Array Kit, System Biosciences, cat# RA900A-1) and universal reverse primer that were used for normalization. To detect Met expression a Met specific primer (Forward: 5’-CTGGACAGATTGTGGGAGTAAG −3’) and universal reverse primer were used along with a Gapdh specific primer (LncRNA Profiler qPCR Array Kit) and universal reverse primer that were used for normalization. qRT-PCR reactions were performed in technical triplicate with Power SYBR Green PCR Master Mix (Thermo Fisher Scientific, cat# 4368577) according to manufacturer’s instructions. PCR cycling conditions were [1 cycle 50°C for 2 min, 95°C for 10 min – 40 cycles 95°C for 15 s, 60°C for 60 s – dissociation stage: 95°C for 15 s, 60°C for 15 s] using an Applied Biosystems real-time PCR system (Applied Biosystems, 7900 HT Fast Real-Time PCR System) and SDS software (RRID:SCR_015806), version 2.4. Negative controls containing no cDNA template were included. Relative expression levels were determined using the ∆∆Ct method.

Animals

All animal procedures were approved by the Stanford University IACUC# 30226 and were in accordance with the Stanford University policies on the care, welfare, and treatment of laboratory animals. No blinding occurred during the experiments.

Six-week old female C57BL/6 mice (Charles River, Strain code: 027, RRID:IMSR_CRL:27) were housed in sterile conditions under standard temperature, humidity, and timed lighting conditions with 12 hr light/dark cycles, and were provided with sterile rodent chow and water ad libitum. Mice were randomized and injected, via retro-orbital and alternating injections between left and right eyes, three times a week for a total of 28 days with either 5 µg of freshly-isolated B16-F10 shScr exosomes, 5 µg of freshly-isolated B16-F10 shMet exosomes, or 1.25 µg synthetic 100 nm liposomes (Encapsula NanoSciences) (mimics 5 µg exosome protein, based on a theoretical 4:1 protein:L-α-phosphatidylcholine ratio) in 100 µl filtered Phosphate Buffered Saline (PBS) (Thermo Fisher Scientific, cat# MT21040CM). The exosomes were resuspended right before each individual injection by pipetting 3–4 times, tapping the tube, inverted 3–4 times, and then tapping the syringe right before each injection. After exosome injections (12 total), mice were inoculated subcutaneously (s.c.) with 1×106 B16-F10-luciferase cells in 100 µl of PBS in the dorsal area. Primary tumor (caliper measurements) and body weight were measured three times a week. The planned study design involved waiting 21 days after B16-F10-luciferase tumor cell implantation before sacrificing the mice for analysis; however, two animals were found dead (one injected with B16-F19 shScr exosomes and one injected with synthetic liposomes) before this time point was reached (17 days after implantation), which in addition the largest tumors of the surviving mice having reached >1000 mm3, prompted us to stop the experiment early (18 days after implantation). To measure metastasis, mice were anesthetized (using isoflurane and O2) and 50 mg/kg of D-luciferin in 50–100 µl PBS was injected retro-orbitally. Five minutes later, mice were euthanized by cervical dislocation and primary tumors and organs (femurs and lungs) were dissected. Primary tumors were weighed (Delta Range scale: Metler Toledo, Model # PB303-S) and organs (femurs and lungs) were analyzed for luciferase expression using the IVIS Spectrum system (Caliper, Xenogen). Anesthesia, luciferin injections, euthanasia, dissection, and imaging/weighing were performed with mice from different groups in parallel (i.e. one from each of the three groups) so variations during the procedure were equal across groups.

IVIS imaging

Images were acquired in a Xenogen IVIS Spectrum at a medium binning level (8) and a 22.6 cm field of view. Acquisition times were set to auto-exposure and were required at 1 and 4 min. There was high concordance between the two exposures (correlation coefficient (ρ)=0.97). The 4 min exposure is shown in figures and used in the statistical analysis. Additionally, after this imaging was complete, the organs were soaked for 10–15 min in 4 ml of 50 mg/ml D-luciferin in six well tissue culture plates and imaged again at 1 and 3 min, which had high concordance with the first set of images taken. Living Image software (RRID:SCR_014247), version 4.4, was used for quantitative analysis. Image files are available at https://osf.io/9xnbf/.

Statistical analysis

Statistical analysis was performed with R software (RRID:SCR_001905), version 3.5.1 (Core TeamR, 2018). All data, csv files, and analysis scripts are available on the OSF (https://osf.io/ewqzf/). Confirmatory statistical analysis was pre-registered (https://osf.io/g5tzn/) before the experimental work began as outlined in the Registered Report (Lesnik et al., 2016). Data were checked to ensure assumptions of statistical tests were met and in the case of the metastatic data were natural log transformed to achieve a normal distribution and equal variance while also allowing for comparisons on a modified percentage scale (Cole and Altman, 2017). When described in the results, the Bonferroni correction, to account for multiple testings, was applied to the alpha error or the p-value. The Bonferroni corrected value was determined by divided the uncorrected value (0.05) by the number of tests performed. A meta-analysis of a common original and replication effect size was performed with a random effects model and the metafor R package (Viechtbauer, 2010) (https://osf.io/c69jx/). Meta-analyses were performed without weighting, since unweighted Cliff’s d has been reported to reduce bias (Kromrey et al., 2005). The asymmetric confidence intervals for the overall Cliff’s d estimate was determined using the normal deviate corresponding to the (1 - alpha/2)th percentile of the normal distribution (Cliff, 1993). The raw data pertaining to Figure 4E and Supplementary Figure 5A of Peinado et al., 2012 were shared by the original authors. The summary data was published in the Registered Report (Lesnik et al., 2016) and used in the power calculations to determine the sample size for this study.

Data availability

Additional detailed experimental notes, data, and analysis are available on OSF (RRID:SCR_003238) (https://osf.io/ewqzf/; Kim et al., 2018). This includes the R Markdown file (https://osf.io/hz3k7/) that was used to compose this manuscript, which is a reproducible document linking the results in the article directly to the data and code that produced them (Hartgerink, 2017).

Deviations from registered report

We planned to generated B16-F10 shMet and shScr stable cells with an MOI of 10, similar to the original study, but observed that the Met levels in the shScr cells at this MOI were, for unknown reasons, low when compared to the shScr cells generated at the other MOI ratios (Figure 1—figure supplement 1C). Experiments reported in this study were with stable cells generated with an MOI of 20. We also included additional tests (qRT-PCR) to confirm the presence of shMet, and corresponding decrease in Met expression, in the stable cells generated. The cell lysis buffer was additionally supplemented with phosphatase inhibitors in an attempt to increase detection of pMet. We also tried an additional antibody targeted against pMet (RRID:AB_10983015) due to an inability to detect a reliable Western blot signal in exosomes with the p-Met antibody stated in the Registered Report and used in the original study (RRID:AB_2315156). Both antibodies gave similar results (RRID:AB_10983015: https://osf.io/96tcs/; RRID:AB_331361: Figure 1B). For the in vivo experimentation, the planned study design involved waiting 21 days after B16-F10-luciferase tumor cell implantation before sacrificing the mice for analysis; however, two animals were found dead (one injected with B16-F19 shScr exosomes and one injected with synthetic liposomes) before this time point was reached, which in addition the largest tumors of the surviving mice having reached >1000 mm3, prompted us to stop the experiment early (18 days after implantation). Additional materials and instrumentation not listed in the Registered Report, but needed during experimentation are also listed.

Acknowledgements

The Reproducibility Project: Cancer Biology would like to thank Dr. David Lyden (Weill Medical College of Cornell University) and Hector Peinado (Spanish National Cancer Research Centre) for sharing critical reagents, data, and protocol information during preparation of the Registered Report, specifically the B16-F10 and B16-F10-luciferase cells. We would also like to thank Courtney Soderberg at the Center for Open Science for assistance with statistical analyses, Jacob Lesnik for managing resources and collaboration details at System Biosciences LLC, the Stanford Transgenic, Knockout and Tumor model Center at the Stanford Cancer Institute, and the following companies for generously donating reagents to the Reproducibility Project: Cancer Biology; American Type and Tissue Collection (ATCC), Applied Biological Materials, BioLegend, Charles River Laboratories, Corning Incorporated, DDC Medical, EMD Millipore, Harlan Laboratories, LI-COR Biosciences, Mirus Bio, Novus Biologicals, Sigma-Aldrich, and System Biosciences (SBI).

Funding Statement

The funder had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Contributor Information

Reproducibility Project: Cancer Biology, Email: tim@cos.io, nicole@scienceexchange.com.

Reproducibility Project: Cancer Biology:

Elizabeth Iorns, Rachel Tsui, Alexandria Denis, Nicole Perfito, Timothy M Errington, Elizabeth Iorns, Rachel Tsui, Alexandria Denis, Nicole Perfito, and Timothy M Errington

Funding Information

This paper was supported by the following grant:

  • Laura and John Arnold Foundation to .

Additional information

Competing interests

Stanford Transgenic, Knockout and Tumor model Center is a Science Exchange associated lab.

System Biosciences LLC is a Science Exchange associated lab and offers products and services for exosome research. Exosome detection products, specifically exosome specific antibodies, and nanoparticle tracking analysis (NTA) service were used during this study; however, ultracentrifugation-free exosome isolation products and services were not used.

System Biosciences LLC is a Science Exchange associated lab and offers products and services for exosome research. Exosome detection products, specifically exosome specific antibodies, and nanoparticle tracking analysis (NTA) service were used during this study; however, ultracentrifugation-free exosome isolation products and services were not used.

Stanford Transgenic, Knockout and Tumor model Center is a Science Exchange associated lab.

System Biosciences LLC is a Science Exchange associated lab and offers products and services for exosome research. Exosome detection products, specifically exosome specific antibodies, and nanoparticle tracking analysis (NTA) service were used during this study; however, ultracentrifugation-free exosome isolation products and services were not used.

System Biosciences LLC is a Science Exchange associated lab and offers products and services for exosome research. Exosome detection products, specifically exosome specific antibodies, and nanoparticle tracking analysis (NTA) service were used during this study; however, ultracentrifugation-free exosome isolation products and services were not used.

EI, RT, NP: Employed by and hold shares in Science Exchange Inc.

Author contributions

Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Performed all exosome purification and generation of exosome-free FBS, qPCR, Animal experiments and IVIS imaging.

Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Performed stable cell line generation, Western blots, qPCR, Nanosight analysis.

Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Performed stable cell line generation, Western blots, qPCR, Nanosight analysis.

Analysis and interpretation of data, Drafting or revising the article.

Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Performed stable cell line generation, Western blots, qPCR, Nanosight analysis.

Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Performed stable cell line generation, Western blots, qPCR, Nanosight analysis.

Ethics

Animal experimentation: All animal procedures were approved by the Stanford University IACUC# 30226 and were in accordance with the Stanford University policies on the care, welfare, and treatment of laboratory animals.

Additional files

Transparent reporting form
elife-39944-transrepform.docx (246.6KB, docx)
DOI: 10.7554/eLife.39944.008
Reporting standard 1. The ARRIVE guidelines checklist.
elife-39944-repstand1.pdf (219.1KB, pdf)
DOI: 10.7554/eLife.39944.009

Data availability

Additional detailed experimental notes, data, and analysis are available on OSF (RRID:SCR_003238) (https://osf.io/ewqzf/; Kim et al., 2018). This includes the R Markdown file (https://osf.io/hz3k7/) that was used to compose this manuscript, which is a reproducible document linking the results in the article directly to the data and code that produced them (Hartgerink, 2017).

The following dataset was generated:

Kim J, Afshari A, Sengupta R, Sebastiano V, Gupta A, Kim YH, Iorns E, Tsui R, Denis A, Perfito N, Errington TM. 2018. Study 42: Replication of Peinado et al., 2012 (Nature Medicine) Open Science Framework.

References

  1. Adachi E, Sakai K, Nishiuchi T, Imamura R, Sato H, Matsumoto K. Different growth and metastatic phenotypes associated with a cell-intrinsic change of Met in metastatic melanoma. Oncotarget. 2016;7:70779–70793. doi: 10.18632/oncotarget.12221. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bailoo JD, Reichlin TS, Würbel H. Refinement of experimental design and conduct in laboratory animal research. ILAR Journal. 2014;55:383–391. doi: 10.1093/ilar/ilu037. [DOI] [PubMed] [Google Scholar]
  3. Ben-David U, Siranosian B, Ha G, Tang H, Oren Y, Hinohara K, Strathdee CA, Dempster J, Lyons NJ, Burns R, Nag A, Kugener G, Cimini B, Tsvetkov P, Maruvka YE, O'Rourke R, Garrity A, Tubelli AA, Bandopadhayay P, Tsherniak A, Vazquez F, Wong B, Birger C, Ghandi M, Thorner AR, Bittker JA, Meyerson M, Getz G, Beroukhim R, Golub TR. Genetic and transcriptional evolution alters cancer cell line drug response. Nature. 2018;560:325–330. doi: 10.1038/s41586-018-0409-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Cannistraci A, Federici G, Addario A, Di Pace AL, Grassi L, Muto G, Collura D, Signore M, De Salvo L, Sentinelli S, Simone G, Costantini M, Nanni S, Farsetti A, Coppola V, De Maria R, Bonci D. C-Met/miR-130b axis as novel mechanism and biomarker for castration resistance state acquisition. Oncogene. 2017;36:3718–3728. doi: 10.1038/onc.2016.505. [DOI] [PubMed] [Google Scholar]
  5. Casellas J. Inbred mouse strains and genetic stability: a review. Animal. 2011;5:1–7. doi: 10.1017/S1751731110001667. [DOI] [PubMed] [Google Scholar]
  6. Chen IH, Xue L, Hsu CC, Paez JS, Pan L, Andaluz H, Wendt MK, Iliuk AB, Zhu JK, Tao WA. Phosphoproteins in extracellular vesicles as candidate markers for breast cancer. PNAS. 2017;114:3175–3180. doi: 10.1073/pnas.1618088114. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Cliff N. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin. 1993;114:494–509. doi: 10.1037/0033-2909.114.3.494. [DOI] [Google Scholar]
  8. Cole TJ, Altman DG. Statistics Notes: Percentage differences, symmetry, and natural logarithms. Bmj. 2017;358:j3683. doi: 10.1136/bmj.j3683. [DOI] [PubMed] [Google Scholar]
  9. Core TeamR . R Foundation for Statistical Computing.  Austria: 2018. [Google Scholar]
  10. Cornelis F, Saut O, Cumsille P, Lombardi D, Iollo A, Palussiere J, Colin T. In vivo mathematical modeling of tumor growth from imaging data: soon to come in the future? Diagnostic and Interventional Imaging. 2013;94:593–600. doi: 10.1016/j.diii.2013.03.001. [DOI] [PubMed] [Google Scholar]
  11. Costa-Silva B, Aiello NM, Ocean AJ, Singh S, Zhang H, Thakur BK, Becker A, Hoshino A, Mark MT, Molina H, Xiang J, Zhang T, Theilen TM, García-Santos G, Williams C, Ararso Y, Huang Y, Rodrigues G, Shen TL, Labori KJ, Lothe IM, Kure EH, Hernandez J, Doussot A, Ebbesen SH, Grandgenett PM, Hollingsworth MA, Jain M, Mallya K, Batra SK, Jarnagin WR, Schwartz RE, Matei I, Peinado H, Stanger BZ, Bromberg J, Lyden D. Pancreatic cancer exosomes initiate pre-metastatic niche formation in the liver. Nature Cell Biology. 2015;17:816–826. doi: 10.1038/ncb3169. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Dunne A, Pawitan Y, Doody L. Two-Sided P-Values from Discrete Asymmetric Distributions Based on Uniformly Most Powerful Unbiased Tests. The Statistician. 1996;45:397. doi: 10.2307/2988542. [DOI] [Google Scholar]
  13. Errington TM, Iorns E, Gunn W, Tan FE, Lomax J, Nosek BA. An open investigation of the reproducibility of cancer biology research. eLife. 2014;3:04333. doi: 10.7554/eLife.04333. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Fong MY, Zhou W, Liu L, Alontaga AY, Chandra M, Ashby J, Chow A, O'Connor ST, Li S, Chin AR, Somlo G, Palomares M, Li Z, Tremblay JR, Tsuyada A, Sun G, Reid MA, Wu X, Swiderski P, Ren X, Shi Y, Kong M, Zhong W, Chen Y, Wang SE. Breast-cancer-secreted miR-122 reprograms glucose metabolism in premetastatic niche to promote metastasis. Nature Cell Biology. 2015;17:183–194. doi: 10.1038/ncb3094. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Guo R, Lee Y, Byrnes C, Miller J. Puromycin Selection Confounds the RNA-Seq Profiles of Primary Human Erythroblasts. Transcriptomics: Open Access. 2017;5 doi: 10.4172/2329-8936.1000140. [DOI] [Google Scholar]
  16. Hartgerink CHJ. Composing reproducible manuscripts using R Markdown. [November 20, 2017];eLife. 2017 https://elifesciences.org/labs/cad57bcf/composing-reproducible-manuscripts-using-r-markdown
  17. He M, Qin H, Poon TC, Sze SC, Ding X, Co NN, Ngai SM, Chan TF, Wong N. Hepatocellular carcinoma-derived exosomes promote motility of immortalized hepatocyte through transfer of oncogenic proteins and RNAs. Carcinogenesis. 2015;36:1008–1018. doi: 10.1093/carcin/bgv081. [DOI] [PubMed] [Google Scholar]
  18. Hoshino A, Costa-Silva B, Shen TL, Rodrigues G, Hashimoto A, Tesic Mark M, Molina H, Kohsaka S, Di Giannatale A, Ceder S, Singh S, Williams C, Soplop N, Uryu K, Pharmer L, King T, Bojmar L, Davies AE, Ararso Y, Zhang T, Zhang H, Hernandez J, Weiss JM, Dumont-Cole VD, Kramer K, Wexler LH, Narendran A, Schwartz GK, Healey JH, Sandstrom P, Labori KJ, Kure EH, Grandgenett PM, Hollingsworth MA, de Sousa M, Kaur S, Jain M, Mallya K, Batra SK, Jarnagin WR, Brady MS, Fodstad O, Muller V, Pantel K, Minn AJ, Bissell MJ, Garcia BA, Kang Y, Rajasekhar VK, Ghajar CM, Matei I, Peinado H, Bromberg J, Lyden D. Tumour exosome integrins determine organotropic metastasis. Nature. 2015;527:329–335. doi: 10.1038/nature15756. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Howard BR. Control of variability. ILAR Journal. 2002;43:194–201. doi: 10.1093/ilar.43.4.194. [DOI] [PubMed] [Google Scholar]
  20. Hughes P, Marshall D, Reid Y, Parkes H, Gelber C. The costs of using unauthenticated, over-passaged cell lines: how much more data do we need? BioTechniques. 2007;43:575–586. doi: 10.2144/000112598. [DOI] [PubMed] [Google Scholar]
  21. Jensen MN, Ritskes-Hoitinga M. How isoflavone levels in common rodent diets can interfere with the value of animal models and with experimental results. Laboratory Animals. 2007;41:1–18. doi: 10.1258/002367707779399428. [DOI] [PubMed] [Google Scholar]
  22. Keerthikumar S, Gangoda L, Liem M, Fonseka P, Atukorala I, Ozcitti C, Mechler A, Adda CG, Ang CS, Mathivanan S. Proteogenomic analysis reveals exosomes are more oncogenic than ectosomes. Oncotarget. 2015;6:15375–15396. doi: 10.18632/oncotarget.3801. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Keerthikumar S, Chisanga D, Ariyaratne D, Al Saffar H, Anand S, Zhao K, Samuel M, Pathan M, Jois M, Chilamkurti N, Gangoda L, Mathivanan S. ExoCarta: A Web-Based Compendium of Exosomal Cargo. Journal of Molecular Biology. 2016;428:688–692. doi: 10.1016/j.jmb.2015.09.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kim J, Afshari A, Sengupta R, Sebastiano V, Gupta A, Kim YH, Iorns E, Tsui R, Denis A, Perfito N, Errington TM. Study 42: Replication of Peinado et al., 2012 (Nature Medicine) Open Science Framework. 2018 doi: 10.17605/OSF.IO/EWQZF. [DOI] [Google Scholar]
  25. Kleensang A, Vantangoli MM, Odwin-DaCosta S, Andersen ME, Boekelheide K, Bouhifd M, Fornace AJ, Li HH, Livi CB, Madnick S, Maertens A, Rosenberg M, Yager JD, Zhao L, Hartung T. Genetic variability in a frozen batch of MCF-7 cells invisible in routine authentication affecting cell function. Scientific Reports. 2016;6:28994. doi: 10.1038/srep28994. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Kokolus KM, Capitano ML, Lee CT, Eng JW, Waight JD, Hylander BL, Sexton S, Hong CC, Gordon CJ, Abrams SI, Repasky EA. Baseline tumor growth and immune control in laboratory mice are significantly influenced by subthermoneutral housing temperature. PNAS. 2013;110:20176–20181. doi: 10.1073/pnas.1304291110. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Kromrey JD, Hogarty KY, Ferron JM, Hines CV, Hess MR. Robustness in meta-analysis: An empirical comparison of point and interval estimates of standardized mean differences and Cliff’s delta. Minneapolis  2005 [Google Scholar]
  28. Lazar I, Clement E, Ducoux-Petit M, Denat L, Soldan V, Dauvillier S, Balor S, Burlet-Schiltz O, Larue L, Muller C, Nieto L. Proteome characterization of melanoma exosomes reveals a specific signature for metastatic cell lines. Pigment Cell & Melanoma Research. 2015;28:464–475. doi: 10.1111/pcmr.12380. [DOI] [PubMed] [Google Scholar]
  29. Leek JT, Scharpf RB, Bravo HC, Simcha D, Langmead B, Johnson WE, Geman D, Baggerly K, Irizarry RA. Tackling the widespread and critical impact of batch effects in high-throughput data. Nature Reviews Genetics. 2010;11:733–739. doi: 10.1038/nrg2825. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Lesnik J, Antes T, Kim J, Griner E, Pedro L, Reproducibility Project Cancer Biology Registered report: Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET. eLife. 2016;5:e07383. doi: 10.7554/eLife.07383. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Li J, Chen X, Yi J, Liu Y, Li D, Wang J, Hou D, Jiang X, Zhang J, Wang J, Zen K, Yang F, Zhang CY, Zhang Y. Identification and Characterization of 293T Cell-Derived Exosomes by Profiling the Protein, mRNA and MicroRNA Components. PLOS ONE. 2016;11:e0163043. doi: 10.1371/journal.pone.0163043. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Liang B, Peng P, Chen S, Li L, Zhang M, Cao D, Yang J, Li H, Gui T, Li X, Shen K. Characterization and proteomic analysis of ovarian cancer-derived exosomes. Journal of Proteomics. 2013;80:171–182. doi: 10.1016/j.jprot.2012.12.029. [DOI] [PubMed] [Google Scholar]
  33. Liu Y, Gu Y, Han Y, Zhang Q, Jiang Z, Zhang X, Huang B, Xu X, Zheng J, Cao X. Tumor Exosomal RNAs Promote Lung Pre-metastatic Niche Formation by Activating Alveolar Epithelial TLR3 to Recruit Neutrophils. Cancer Cell. 2016;30:243–256. doi: 10.1016/j.ccell.2016.06.021. [DOI] [PubMed] [Google Scholar]
  34. Macpherson AJ, McCoy KD. Standardised animal models of host microbial mutualism. Mucosal Immunology. 2015;8:476–486. doi: 10.1038/mi.2014.113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Nevalainen T. Animal husbandry and experimental design. ILAR Journal. 2014;55:392–398. doi: 10.1093/ilar/ilu035. [DOI] [PubMed] [Google Scholar]
  36. Peinado H, Alečković M, Lavotshkin S, Matei I, Costa-Silva B, Moreno-Bueno G, Hergueta-Redondo M, Williams C, García-Santos G, Ghajar C, Nitadori-Hoshino A, Hoffman C, Badal K, Garcia BA, Callahan MK, Yuan J, Martins VR, Skog J, Kaplan RN, Brady MS, Wolchok JD, Chapman PB, Kang Y, Bromberg J, Lyden D. Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET. Nature Medicine. 2012;18:883–891. doi: 10.1038/nm.2753. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Plebanek MP, Angeloni NL, Vinokour E, Li J, Henkin A, Martinez-Marin D, Filleur S, Bhowmick R, Henkin J, Miller SD, Ifergan I, Lee Y, Osman I, Thaxton CS, Volpert OV. Pre-metastatic cancer exosomes induce immune surveillance by patrolling monocytes at the metastatic niche. Nature Communications. 2017;8:1319. doi: 10.1038/s41467-017-01433-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Rohatgi VK, Saleh A. An Introduction to Probability and Statistics: Rohatgi/An Introduction, Wiley Series in Probability and Statistics. Wiley Online Library. 2000 doi: 10.1002/9781118165676. [DOI] [Google Scholar]
  39. Schneider CA, Rasband WS, Eliceiri KW. NIH Image to ImageJ: 25 years of image analysis. Nature Methods. 2012;9:671–675. doi: 10.1038/nmeth.2089. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Shearer RF, Saunders DN. Experimental design for stable genetic manipulation in mammalian cell lines: lentivirus and alternatives. Genes to Cells. 2015;20:1–10. doi: 10.1111/gtc.12183. [DOI] [PubMed] [Google Scholar]
  41. Sorge RE, Martin LJ, Isbester KA, Sotocinal SG, Rosen S, Tuttle AH, Wieskopf JS, Acland EL, Dokova A, Kadoura B, Leger P, Mapplebeck JC, McPhail M, Delaney A, Wigerblad G, Schumann AP, Quinn T, Frasnelli J, Svensson CI, Sternberg WF, Mogil JS. Olfactory exposure to males, including men, causes stress and related analgesia in rodents. Nature Methods. 2014;11:629–632. doi: 10.1038/nmeth.2935. [DOI] [PubMed] [Google Scholar]
  42. Steenbeek SC, Pham TV, de Ligt J, Zomer A, Knol JC, Piersma SR, Schelfhorst T, Huisjes R, Schiffelers RM, Cuppen E, Jimenez CR, van Rheenen J. Cancer cells copy migratory behavior and exchange signaling networks via extracellular vesicles. The EMBO Journal. 2018;37:e98357. doi: 10.15252/embj.201798357. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Talkington A, Durrett R. Estimating Tumor Growth Rates In Vivo. Bulletin of Mathematical Biology. 2015;77:1934–1954. doi: 10.1007/s11538-015-0110-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Valentine JC, Biglan A, Boruch RF, Castro FG, Collins LM, Flay BR, Kellam S, Mościcki EK, Schinke SP. Replication in prevention science. Prevention Science. 2011;12:103–117. doi: 10.1007/s11121-011-0217-6. [DOI] [PubMed] [Google Scholar]
  45. Viechtbauer W. Conducting Meta-Analyses inRwith themetaforPackage. Journal of Statistical Software. 2010;36 doi: 10.18637/jss.v036.i03. [DOI] [Google Scholar]
  46. Zeng AL, Yan W, Liu YW, Wang Z, Hu Q, Nie E, Zhou X, Li R, Wang XF, Jiang T, You YP. Tumour exosomes from cells harbouring PTPRZ1-MET fusion contribute to a malignant phenotype and temozolomide chemoresistance in glioblastoma. Oncogene. 2017;36:5369–5381. doi: 10.1038/onc.2017.134. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Zhou W, Fong MY, Min Y, Somlo G, Liu L, Palomares MR, Yu Y, Chow A, O'Connor ST, Chin AR, Yen Y, Wang Y, Marcusson EG, Chu P, Wu J, Wu X, Li AX, Li Z, Gao H, Ren X, Boldin MP, Lin PC, Wang SE. Cancer-secreted miR-105 destroys vascular endothelial barriers to promote metastasis. Cancer Cell. 2014;25:501–515. doi: 10.1016/j.ccr.2014.03.007. [DOI] [PMC free article] [PubMed] [Google Scholar]

Decision letter

In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.

Thank you for submitting your work entitled "Replication Study: Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET" for consideration at eLife. Your article has been evaluated by Charles Sawyers as the Senior Editor, Michael Green as the Reviewing Editor, and two expert reviewers.

There are a number of issues that need to be seriously addressed as outlined below in the separate reviews. We realise that additional work that was not outlined in the Registered Report is outside the scope of the project, but we would ask you to respond to each point, revising the text accordingly, and stating the necessary caveats in response (rather than performing additional experiments).

We would like to emphasize that in addition to changes in the main text, the Abstract must be modified to accurately reflect the data generated and their issues. Reviewer 1 had this to add about the Abstract:

"Most importantly, the abstract needs to be revised to specifically state the issues with animal death, failure of tumors to grow and off target effects of the control encountered in this report but not the original study. Moreover, the Abstract should clearly state that any lack of reproducibility is due to the aforementioned issues. Specifically, it should state that the failure to reproduce the results of the original study is because the author's own controls and experiments failed technically. Thus, no conclusions can be drawn."

Reviewer #1:

We would like to thank the authors for transparency in reporting results and analyses. However, upon reviewing the report, we identified several concerns that make it impossible to draw definitive conclusions from the presented data. Therefore, the study should not be published as is, and suggest that the authors perform additional work to address the major problems they have encountered that are not otherwise reported in other studies in the field, including the original study. Moreover, the authors need to include additional controls that would allow direct comparison with previous studies.

First and foremost, the Abstract, as written, is misleading and should be re-written before publication is considered. In the current version, the Abstract states that the authors 1) could not "reliably detect phospho-Met (pMet) in exosomes" and that 2) and that "while the effects [on metastasis] were in the same direction as the original study (Figure 4E; Peinado et al., 2012), they were not statistically significant". These conclusions cannot be drawn based on the data presented in the report due to experimental design and technical issues detailed below. Therefore, the statements in the Abstract need to be amended to reflect the measurement variability, lack of tumor growth in some animals, animal death and off target effects evident in this report.

The most significant issue encountered in this report is the fact that the authors' data on both Met expression and primary tumor growth indicates there are off-target effect(s) with their "non-targeting" scrambled shRNA. A more appropriate control, at least done in parallel, would have been to use the shEmpty vector that does not contain a scrambled sequence. An even better and more relevant control would have been to use the unmodified/untransduced, parental B16F10 cell line-derived exosomes for education and ensure that a level of primary tumor growth and metastasis equivalent to the original study is obtained under these conditions.

Specifically, in Figure 1A, the shScr control cells should express Met, while the shMet cells should not express Met. The data presented by the authors shows the contrary – is this a labeling mistake? Moreover, it is unclear from the data presented in Figure 1A and B if the replicates are technical or biological replicates (if so, are they independent exosome collections from the same batch of shScr or shMet lentivirus-infected cells, or are they independent infections?).

Unfortunately, it is impossible to determine if the authors' inability to detect pMet in shScr and scMet exosomes is a technical issue or a problem with the cell models, because the essential control B16F10 cells and exosomes that should have robust expression of Met and pMet was not used. The authors need to revise their experimental design and repeat their experiments including this control, which was present in the original Peinado et al. paper as well as in numerous other publications before any conclusions can be drawn. Importantly, the authors should highlight that, consistent with original report by Peinado et al., the shMet construct efficiently downregulated Met in B16F10 cells and their exosomes.

There could be several explanations that need to be explored that could account for the lack of pMet detection in exosomes in this report. First of all, it is unclear if fresh exosomes had been used in these Western Blot experiments (authors only indicate the use of freshly-isolated exosomes in in vivo studies) but it would also be critical that fresh exosomes be used for phospho-protein detection in exosomes. The authors actually state that exosomes were "stored at -20°C until analysis” therefore, the analysis of phosphoproteins should be repeated immediately following isolation, resuspension in PBS, measurement of protein and resuspension in Laemmli buffer. This should all be performed using fresh exosomes.

Importantly, many published studies, in additional to the original report from Peinado et al., 2012, have since validated and were able to detect Met as well as pMet in B16F10 exosomes and other exosomes. These studies are listed below, they should be included in the reference list of this report and discussed in the context of the authors' results.

• Steenbeek et al., 2018 – Figure 4 shows Met expression in B16F10 exosomes

• Barrow-McGee R et al., Nature Communications, PMID: 27336951 – shows cMet and pMet in endosomes throughout the paper

• Tripolitsioti D et al., Oncotarget, PMID: 29796184 – throughout the paper and Supplementary Figure 5D (Met), 6B (pMet)

• Adachi et al., 2016 – shows both Met and pMet in Figure 1C

• Cannistraci et al., 2017 – shows both Met and pMet in Figure 2D

• Plebanek MP et al., 2017

• Zeng et al., 2017 – shows both Met and pMet

• He et al., 2015 – shows both Met and pMet in Figure 4E.

Another cause for concern with this report is the off target effect present in the only control used in this study, the shScr control. As shown in Figure 2A of the report and pointed out by the authors in their results, using a lower multiplicity of infection (MOI) of 10, the same used in the original Peinado et al. paper, the authors of this report find there is a significant on target effect of the control shScr, as Met levels are downregulated. Strangely, the authors are able to reduce the on target effect of the control shScr by increasing the MOI to 20 or 50. Most likely, this is due to the oligomers binding to other targets, and thus an increase in off target effects. Since these are commercial lentiviral particles, and the time passed since the initial report, it would be critical for the authors to verify with the company that they still use the same sequence as control as reported in the original paper. In this manuscript the authors do not define the targeting sequence as in the original report (Peinado et al., specified that sense sequence used, obtained from Thermo was: 5′-ATCTCGCTTGGGCGAGAGTAAG-3′) but the authors in the current report do not describe the sequence and use pGIPZ non-silencing shRNA lentiviral control particles from Dharmacon/GE Life 281 Sciences, cat# RHS4348, lot# 150529606. Moreover, the difference in results could be due to lot variability or quality of commercial virus (this should be discussed by the authors in the report). Minor concerns regarding Figure 2 are the inconsistency of GAPDH loading control levels as well as the presentation of MOIs out of order (10, then 50, then 20). The authors need to provide empty vector and non-transduced cell controls for these experiments. Moreover, to investigate the off target effects of the shScr they should perform transcriptomics analyses of the non-transduced, empty vector, shScr and shMet B16-F10 cells. They can also perform unbiased proteomics analysis of shScr versus shMet versus B16F10 exosomes to determine if/how the cargo of shScr exosomes is affected by off target effects. The authors should test proliferation and apoptosis of shScr cells, to account for any consequences of off target effects of the shScr construct on the health of these cells. This is critical because of the issues evident in Figure 3.

The other major cause of concern, that precludes the data from this report being interpretable and the drawing of any conclusions, is, as evident from Figure 3A, B), the fact that the primary tumors growing in animals educated with shScr exosomes are smaller than those in the liposome-treated controls, could this be because of their issues with the control particles? Additionally, whereas the size of primary tumors growing in the liposome or shMET treated groups is reproducible and similar, there is variability in the growth of the tumors in the shScr treated mice, with some of the tumors as small as 0.025 g, which is 32-fold smaller than the mean of the other groups, which is 0.8 g. Consistent with the original report by Peinado et al., the authors do not find differences in tumor growth between liposome treated mice and shMet mice. Given the dramatic reduction in tumor growth in the shScr control treated mice, and the large variability in the growth of these tumors, it is not surprising that the authors of this report fail to find statistically significant differences in lung and bone metastasis between the shScr control and shMet exosome educated mice. Therefore, before any conclusions can be drawn, and to account for the large variability and off target effect of the shScr control and in this experiment overall, the authors need to repeat the experiment increasing animal numbers and including animals educated with un-transduced, parental B16F10 exosomes to demonstrate that, in their hands, they can obtain consistent primary tumor growth and metastasis levels without the confounding contribution of the off-target effects in the shScr. Moreover, to reiterate, the differences between the shScr and shMet cannot be deemed non-significant since the lack of significance stems from the high variance and lower than expected primary tumor size in the shScr control group.

Importantly, there is lack of tumor growth in some animals and death of animals injected with exosomes. B16F10 are aggressive tumors with robust growth that do not fail to implant or grow, therefore the fact that the authors of this study encounter challenges in having tumors grow in all animals injected with control cells reflect either 1) technical issues with cell viability, injection, etc. and/or 2) the fact that the off target effects of the shScr exosomes used for education are effectively inhibiting tumor growth. These issues need to be resolved before this report can be published.

Additionally, it is concerning that education of animals with 5 μg of shScr B16F10 exosomes leads to the death of some animals. This was not reported in publications with either 5 μg or 10 μg of B16F10 exosomes (for example in the original publication or in Dr. Olga Volpert's Nature Communications, 2018 report injected 10 μg of B16F10 exosomes repeatedly with no death). The deaths observed by the authors of the present study upon exosome injection are worrisome. Have the authors verified that they are measuring exosomal protein accurately after isolation? The authors should define the buffer used for exosome resuspension (in the original report was PBS as stated "The floating exosome fraction was collected again by ultracentrifugation as above, and the final pellet was resuspended in PBS"). Mouse lethality could reflect inaccurate exosome protein quantification. Alternatively, the presence of exosome aggregates in the preparation would lead to animal death, to avoid this exosomes should be vigorously resuspended before injection (10 times at least). Last but not least, the health of the cells from which the exosomes are collected can significantly influence the quality of the exosomes and could affect animal health (especially in the context of the shScr off target effects).

Reviewer #2:

Overall this replication study was carried out appropriately. I have some concerns about the quality of the data presented (in point 1) and some other comments, as follows:

1) In Figure 1, The Western blots give a variety of concerns. First, they are all very dark and overblown. I would also like to see the full blots – they should probably be included in the published paper. Second, for Figure 1A, the blot comparing the shScr and shMet in the cells shows much higher Met in the shMet cells then in the controls. I presume that is a mistake? Third, is GAPDH a relevant loading control or does it change in exosomes with Met?

2) For Figure 1—figure supplement 1, I also don't see how a shMet can be compared to shSc when they are on different blots and it seems strange to compare the MOI to just each other and not untreated cells. This is a relatively minor concern – the assessment of Met KD in Figure 1 is the most important.

3) For the analysis of tumor metastasis, are the data normal or not normal? It seems like statistical analyses were used that are appropriate for data with a non-Gaussian distribution. Therefore, I'm not sure why mean and SEM are used to represent the data, which are only appropriate for data with a Gaussian distribution.

4) For differences between this study and Peinado that are listed in the last paragraph of the Results/Discussion section of the paper, I think differences in Met signaling and in the effectiveness of knockdown (which I also doubt based on the quality of the Western blots here) should be mentioned. In addition, the study was not carried out as long as the Peinado study (18 days instead of 21 days) after exosome education, so this should be mentioned in the last paragraph too. There were also fewer mice, which could definitely affect detection of statistical significance, which should be mentioned.

eLife. 2018 Dec 11;7:e39944. doi: 10.7554/eLife.39944.015

Author response

There are a number of issues that need to be seriously addressed as outlined below in the separate reviews. We realise that additional work that was not outlined in the Registered Report is outside the scope of the project, but we would ask you to respond to each point, revising the text accordingly, and stating the necessary caveats in response (rather than performing additional experiments).

We would like to emphasize that in addition to changes in the main text, the Abstract must be modified to accurately reflect the data generated and their issues. Reviewer 1 had this to add about the Abstract:

"Most importantly, the abstract needs to be revised to specifically state the issues with animal death, failure of tumors to grow and off target effects of the control encountered in this report but not the original study. Moreover, the Abstract should clearly state that any lack of reproducibility is due to the aforementioned issues. Specifically, it should state that the failure to reproduce the results of the original study is because the author's own controls and experiments failed technically. Thus, no conclusions can be drawn."

We disagree with the statement that ‘failure to reproduce the results of the original study is because the author’s own controls and experiments failed technically’. The replication was performed with the same controls, cells, shRNA sequences, etc. as reported in the original study for these experiments with input from the original authors and peer review of the Registered Report. The specific concerns raised have been addressed below and as appropriately the text has been revised. Additionally, while technical concerns and hidden moderators might be factors that influence both the original and replication results, as Reviewer #1 suggests, another likely factor is biological variation. We agree that the results we reported could be attributed to observable, or hypothetical, differences between the original study and this replication and we have revised the Abstract to highlight this.

Reviewer #1:

We would like to thank the authors for transparency in reporting results and analyses. However, upon reviewing the report, we identified several concerns that make it impossible to draw definitive conclusions from the presented data. Therefore, the study should not be published as is, and suggest that the authors perform additional work to address the major problems they have encountered that are not otherwise reported in other studies in the field, including the original study. Moreover, the authors need to include additional controls that would allow direct comparison with previous studies.

First and foremost, the Abstract, as written, is misleading and should be re-written before publication is considered. In the current version, the Abstract states that the authors 1) could not "reliably detect phospho-Met (pMet) in exosomes" and that 2) and that "while the effects [on metastasis] were in the same direction as the original study (Figure 4E; Peinado et al., 2012), they were not statistically significant". These conclusions cannot be drawn based on the data presented in the report due to experimental design and technical issues detailed below. Therefore, the statements in the Abstract need to be amended to reflect the measurement variability, lack of tumor growth in some animals, animal death and off target effects evident in this report.

The most significant issue encountered in this report is the fact that the authors' data on both Met expression and primary tumor growth indicates there are off-target effect(s) with their "non-targeting" scrambled shRNA. A more appropriate control, at least done in parallel, would have been to use the shEmpty vector that does not contain a scrambled sequence. An even better and more relevant control would have been to use the unmodified/untransduced, parental B16F10 cell line-derived exosomes for education and ensure that a level of primary tumor growth and metastasis equivalent to the original study is obtained under these conditions.

The ‘non-targeting’ scrambled shRNA used in this replication was the same control used in the original study per the original paper methods and our communication with the original authors. The same sequence was confirmed from the commercial supplier before the replication was conducted and we included additional details of the sequence in the revised manuscript.

We agree that inclusion of untransduced B16-F10 cell line-derived exosomes would provide insight about possible effects that might have occurred during infection, such as genome instability. We have revised the manuscript to highlight the addition of untransduced B16-F10 cells as a control to consider for future experiments. For the experiments replicated, however, this additional control was not reported in the original study to allow for a comparison between untransduced B16-F10 and shScr cells (Figure 4A of the original study is between B16-F10 and B16-F1 cells and exosomes, Supplementary Figure 5A is between B16-F10, B16-F1, and B16-F10 shMet cells and exosomes, and Figure 4E of the metastasis results is between B16-F10, B16-F10 shMet, and liposome control). It was unclear from the original paper if the B16-F10 cells and exosomes used for the education experiment that was replicated were from untransduced cells or were shScramble; however, during peer review of the Registered Report it was suggested that the B16-F10 cells reported were shScramble transduced. Thus, we conducted this experiment similar to the original experimental design, which was reviewed by the original authors and peer reviewed prior to beginning the experimental work.

Specifically, in Figure 1A, the shScr control cells should express Met, while the shMet cells should not express Met. The data presented by the authors shows the contrary – is this a labeling mistake? Moreover, it is unclear from the data presented in Figure 1A and B if the replicates are technical or biological replicates (if so, are they independent exosome collections from the same batch of shScr or shMet lentivirus-infected cells, or are they independent infections?).

Unfortunately, it is impossible to determine if the authors' inability to detect pMet in shScr and scMet exosomes is a technical issue or a problem with the cell models, because the essential control B16F10 cells and exosomes that should have robust expression of Met and pMet was not used. The authors need to revise their experimental design and repeat their experiments including this control, which was present in the original Peinado et al. paper as well as in numerous other publications before any conclusions can be drawn. Importantly, the authors should highlight that, consistent with original report by Peinado et al., the shMet construct efficiently downregulated Met in B16F10 cells and their exosomes.

Figure 1A in the original submission was a labeling mistake that has been fixed in the revised manuscript. Regarding the data presented in Figure 1A and B, these are repeats from ‘independently isolated exosome preparations’ as stated in the figure legend. However, these are from the same batch of shScr or shMet lentivirus-infected cells and assessed after 28 days of puromycin selection as suggested during peer review of the Registered Report. We have added additional details to the revised manuscript to further clarify this point.

As stated above, the suggested additional control (untransduced B16-F10 cells) was not included in the Registered Report, but we have revised the manuscript to suggest the inclusion of this control in future experiments. Importantly, the parental B16-F10 cells (as well the B16-F10-luciferase cells) were shared by the original authors. We have also highlighted that the shMet construct reduced Met expression in B16-F10 cells and exosomes, similar to the original study in the Abstract and at the end of the “Generation and characterization of shMet B16-F10 cells and exosomes” section of the Results. However, the level of knockdown achieved in shMet exosomes was not reported in the original study.

There could be several explanations that need to be explored that could account for the lack of pMet detection in exosomes in this report. First of all, it is unclear if fresh exosomes had been used in these Western Blot experiments (authors only indicate the use of freshly-isolated exosomes in in vivo studies) but it would also be critical that fresh exosomes be used for phospho-protein detection in exosomes. The authors actually state that exosomes were "stored at -20°C until analysis” therefore, the analysis of phosphoproteins should be repeated immediately following isolation, resuspension in PBS, measurement of protein and resuspension in Laemmli buffer. This should all be performed using fresh exosomes.

We agree there are many possible explanations that could account for the lack of pMet detection in exosomes. As the reviewer highlighted, we did not use fresh exosomes for Western blot analysis (as stated in the Registered Report as well), but did for the education experiment. This is because in our correspondence with the original author for the experiment in question (Protocol 2 of the Registered Report we were told ‘Exosomes for WB can be stored at -20 for 2-3 weeks’.). We agree this is a reasonable explanation, which we have included in the Discussion of the revised manuscript.

Importantly, many published studies, in additional to the original report from Peinado et al., 2012, have since validated and were able to detect Met as well as pMet in B16F10 exosomes and other exosomes. These studies are listed below, they should be included in the reference list of this report and discussed in the context of the authors' results.

• Steenbeek et al., 2018 – Figure 4 shows Met expression in B16F10 exosomes

• Barrow-McGee R et al., Nature Communications, PMID: 27336951 – shows cMet and pMet in endosomes throughout the paper

• Tripolitsioti D et al., Oncotarget, PMID: 29796184 – throughout the paper and Supplementary Figure 5D (Met), 6B (pMet)

• Adachi et al., 2016 – shows both Met and pMet in Figure 1C

• Cannistraci et al., 2017 – shows both Met and pMet in Figure 2D

• Plebanek MP et al., 2017

• Zeng et al., 2017 – shows both Met and pMet

• He et al., 2015 – shows both Met and pMet in Figure 4E.

We have included additional references in the revised manuscript in addition to the references that were already included on the identification of Met and pMet in exosomes. Some of these references reported the presence of Met in other vesicles and were not included (e.g. Barrow-McGee et al., 2016).

Another cause for concern with this report is the off target effect present in the only control used in this study, the shScr control. As shown in Figure 2A of the report and pointed out by the authors in their results, using a lower multiplicity of infection (MOI) of 10, the same used in the original Peinado et al. paper, the authors of this report find there is a significant on target effect of the control shScr, as Met levels are downregulated. Strangely, the authors are able to reduce the on target effect of the control shScr by increasing the MOI to 20 or 50. Most likely, this is due to the oligomers binding to other targets, and thus an increase in off target effects. Since these are commercial lentiviral particles, and the time passed since the initial report, it would be critical for the authors to verify with the company that they still use the same sequence as control as reported in the original paper. In this manuscript the authors do not define the targeting sequence as in the original report (Peinado et al., specified that sense sequence used, obtained from Thermo was: 5′-ATCTCGCTTGGGCGAGAGTAAG-3′) but the authors in the current report do not describe the sequence and use pGIPZ non-silencing shRNA lentiviral control particles from Dharmacon/GE Life 281 Sciences, cat# RHS4348, lot# 150529606. Moreover, the difference in results could be due to lot variability or quality of commercial virus (this should be discussed by the authors in the report). Minor concerns regarding Figure 2 are the inconsistency of GAPDH loading control levels as well as the presentation of MOIs out of order (10, then 50, then 20). The authors need to provide empty vector and non-transduced cell controls for these experiments. Moreover, to investigate the off target effects of the shScr they should perform transcriptomics analyses of the non-transduced, empty vector, shScr and shMet B16-F10 cells. They can also perform unbiased proteomics analysis of shScr versus shMet versus B16F10 exosomes to determine if/how the cargo of shScr exosomes is affected by off target effects. The authors should test proliferation and apoptosis of shScr cells, to account for any consequences of off target effects of the shScr construct on the health of these cells. This is critical because of the issues evident in Figure 3.

We disagree with the reviewer that an effect on Met expression could be due to the oligomers preferentially targeting Met in the shScr condition. For this to occur it would suggest overexpression of a random sequence would divert the transcriptional/translational machinery away from making Met mRNA/protein or random integration into Met or a gene that regulates Met. Of note, these concerns also exist for the original study since the same oligo sequence was used. It is more likely that non-random genetic/transcriptional drift, which could have been caused by selective pressure from puromycin, affected gene expression of Met (as well as other genes). This has been highlighted as a possible explanation of the results in the revised manuscript.

As mentioned above, the ‘non-targeting’ scrambled shRNA used in this replication was the same control used in the original study per the original paper methods and our communication with the original authors. The same sequence as the original study was confirmed from the commercial supplier before the replication was conducted and we have included additional details on the sequence in the revised manuscript. Additionally, this was shared from the commercial supplier, “this sequence does not match any known mammalian genes and had at least 3 or more mismatches against any gene as determined via nucleotide alignment/BLAST of the 22mer sense sequence”. While the supplier is different, because of company acquisitions that have occurred between the original study and this replication, the vectors used to generate the lentiviral particles were the same as the original study. We agree lot variability is another source that is different between the two studies, although we do not know the lot used in the original study, and we have included this in the revised manuscript.

As mentioned above, the suggested additional control (untransduced B16-F10 cells) was not included in the Registered Report, but we have revised the manuscript to suggest the inclusion of this control in future experiments. Regarding the comment about Gapdh loading, we loaded equal amounts of protein (30 µg) for each sample analyzed by Western blot, opposed to adjusting amounts based on Gapdh levels.

We agree that if there were differences in cell growth, or apoptosis rates could confound the results. This is also true for the original study, which does not report what the growth, or apoptosis rates were for the experiments reported. While we did not perform an experiment to quantify growth or apoptosis rates, we did monitor the health of the cultures and qualitatively kept track of the cell growth rates among the conditions. Growth rates were similar among all stable cell populations generated based on qualitative observations documented during passaging.

The other major cause of concern, that precludes the data from this report being interpretable and the drawing of any conclusions, is, as evident from Figure 3A, B), the fact that the primary tumors growing in animals educated with shScr exosomes are smaller than those in the liposome-treated controls, could this be because of their issues with the control particles? Additionally, whereas the size of primary tumors growing in the liposome or shMET treated groups is reproducible and similar, there is variability in the growth of the tumors in the shScr treated mice, with some of the tumors as small as 0.025 g, which is 32-fold smaller than the mean of the other groups, which is 0.8 g. Consistent with the original report by Peinado et al., the authors do not find differences in tumor growth between liposome treated mice and shMet mice. Given the dramatic reduction in tumor growth in the shScr control treated mice, and the large variability in the growth of these tumors, it is not surprising that the authors of this report fail to find statistically significant differences in lung and bone metastasis between the shScr control and shMet exosome educated mice. Therefore, before any conclusions can be drawn, and to account for the large variability and off target effect of the shScr control and in this experiment overall, the authors need to repeat the experiment increasing animal numbers and including animals educated with un-transduced, parental B16F10 exosomes to demonstrate that, in their hands, they can obtain consistent primary tumor growth and metastasis levels without the confounding contribution of the off-target effects in the shScr. Moreover, to reiterate, the differences between the shScr and shMet cannot be deemed non-significant since the lack of significance stems from the high variance and lower than expected primary tumor size in the shScr control group.

We agree that there was a higher relative standard deviation (RSD) among the results reported in the original study and this replication attempt for the metastatic burden. We included the RSD for both the original and replication results to provide a direct comparison. Importantly, though we cannot compare the primary tumor growth variability as this was not reported in the original study. Furthermore, the variability should also be viewed in the context that cell growth is exponential under an ideal scenario, thus the RSD is reported in natural log units (Cole and Altman, 2017). There are factors that influence and alter the growth of the tumor initially compared to the continued growth of the tumor in vivo, such as availability of nutrients, oxygen, and space. These points have been included in the discussion of the revised manuscript.

This replication attempt, like all of the replication attempts in the Reproducibility Project: Cancer Biology, are designed to perform independent replications with a calculated sample size to detect the originally reported effect size with at least 80% power based on the originally reported data, which was followed in this replication study. We agree the higher variance in the replication is a factor influencing if statistical significance is reached and include this as a consideration when discussing the effect sizes. While technical concerns and hidden moderators might be factors that influence both the original and replication results, as the reviewer suggests, another likely factor is biological variation. That is, one would not expect tumors in individual animals to grow at the exact same rate as there are many influencing factors that cannot be controlled or explained.

We also agree performing another attempt of this experiment with modifications would begin to explore if these, or other, factors influence the outcome of this study. While, it’s not within the scope of this project, or as part of this publishing model, to also conduct these experiments, the results of this replication bring variables not previously thought to influence the experiment into question (size of the control tumors at the end of the study, length of treatment, etc.). Importantly though, it is because of the results that these and other aspects now become targets for hypothesizing and investigation.

Importantly, there is lack of tumor growth in some animals and death of animals injected with exosomes. B16F10 are aggressive tumors with robust growth that do not fail to implant or grow, therefore the fact that the authors of this study encounter challenges in having tumors grow in all animals injected with control cells reflect either 1) technical issues with cell viability, injection, etc. and/or 2) the fact that the off target effects of the shScr exosomes used for education are effectively inhibiting tumor growth. These issues need to be resolved before this report can be published.

The lack of tumor growth was observed in one animal, however in this animal metastatic burden was detected, indicating the B16-F10-luciferase cells were injected and aggressively spread throughout the animal. Additionally, the death observed was likely due to the aggressive growth of the B16-F10-luciferase cells. The mice that died (17 days after implantation) had tumors >1000 mm3 at their last measurement, which along with other surviving mice that had tumors that reached >1000 mm3, prompted us to stop the experiment early (18 days after implantation). As suggested by reviewer #2 we have included the shorter experimental time difference between the original study and this replication attempt in the Discussion of the revised manuscript.

Additionally, it is concerning that education of animals with 5 μg of shScr B16F10 exosomes leads to the death of some animals. This was not reported in publications with either 5 μg or 10 μg of B16F10 exosomes (for example in the original publication or in Dr. Olga Volpert's Nature Communications, 2018 report injected 10 μg of B16F10 exosomes repeatedly with no death). The deaths observed by the authors of the present study upon exosome injection are worrisome. Have the authors verified that they are measuring exosomal protein accurately after isolation? The authors should define the buffer used for exosome resuspension (in the original report was PBS as stated "The floating exosome fraction was collected again by ultracentrifugation as above, and the final pellet was resuspended in PBS"). Mouse lethality could reflect inaccurate exosome protein quantification. Alternatively, the presence of exosome aggregates in the preparation would lead to animal death, to avoid this exosomes should be vigorously resuspended before injection (10 times at least). Last but not least, the health of the cells from which the exosomes are collected can significantly influence the quality of the exosomes and could affect animal health (especially in the context of the shScr off target effects).

As mentioned above, the deaths observed were not upon exosome injection, but 17 days after B16-F10-luciferase implantation (which occurred after the 28 days of 3 times a week exosome injections). It is a possibility that exosome, or synthetic liposome, injection might have influenced this, but this does not seem likely. The more likely possibility is the aggressive growth of the B16-F10-luciferase cells, which was more severe in some animals than others.

Regarding the technical questions raised as possibly factors. Yes, we used a BCA assay, with standard curves, to quantify the amount of exosome protein after isolation using, similar to the original study. Furthermore, we characterized exosomes using Nanosight analysis. The methods defined the buffer the exosomes were resuspended in “100 µl filtered PBS (Thermo Fisher Scientific, cat# MT21040CM)”, the same buffer as the original study. While no information was included in the original paper, or during communication with the original authors regarding a specific way the exosome should be resuspended prior to injection (i.e. 10 times at least), the exosomes were resuspended right before each individual injection by pipetting 3-4 times, tapping the tube, inverted 3-4 times, and then tapping the syringe right before each injection. We have added these additional details to the Materials and methods in the revised manuscript. Also, as mentioned above, we monitored the health and qualitatively kept track of the cell growth rates of the cultures. There were also no complications during exosome injections.

Reviewer #2:

Overall this replication study was carried out appropriately. I have some concerns about the quality of the data presented (in point 1) and some other comments, as follows:

1) In Figure 1, The Western blots give a variety of concerns. First, they are all very dark and overblown. I would also like to see the full blots – they should probably be included in the published paper. Second, for Figure 1A, the blot comparing the shScr and shMet in the cells shows much higher Met in the shMet cells then in the controls. I presume that is a mistake? Third, is GAPDH a relevant loading control or does it change in exosomes with Met?

We have revised Figure 1, and the figure supplement, to increase the area displayed and the brightness/contrast of the images to allow for a full representation of the results. Figure 1A in the original submission was a labeling mistake that has been fixed in the revised manuscript. Gadph was the same loading control used in the original study, which is why it was used in this replication attempt. Gapdh is one of the most often identified proteins identified in exosomes (http://exocarta.org/exosome_markers). Of course, whether Gapdh is a relevant loading control, just like any other ‘housekeeping’ gene, is tricky since they vary widely, especially across sample types (e.g. Barber et al., 2005). We are not aware of any research suggesting Gapdh expression changes based on Met expression. Importantly, though we loaded equal amounts of protein (30 µg) for each sample analyzed by Western blot, opposed to adjusting amounts based on Gapdh levels.

Barber, R.D., Harmer, D.W., Coleman, R.A., Clark, B.J., 2005. GAPDH as a housekeeping gene: analysis of GAPDH mRNA expression in a panel of 72 human tissues. Physiological Genomics 21, 389–395. https://doi.org/10.1152/physiolgenomics.00025.2005

2) For Figure 1—figure supplement 1, I also don't see how a shMet can be compared to shSc when they are on different blots and it seems strange to compare the MOI to just each other and not untreated cells. This is a relatively minor concern – the assessment of Met KD in Figure 1 is the most important.

We agree that a comparison of shScr and shMet cannot be made since they are on different blots. These blots, however, illustrate that, for unknown reasons, Met levels in the shScr cells generated with an MOI of 10 were lower when compared to the shScr cells generated at the other MOI ratios.

We also agree that a comparison of shScramble and shMet cells to untransduced B16-F10 cells and exosomes would be ideal, especially considering the results observed in this replication attempt; however, this was not reported in the original study (Figure 4A of the original study is between B16-F10 and B16-F1 cells and exosomes, Supplementary Figure 5A is between B16-F10, B16-F1, and B16-F10 shMet cells and exosomes, and Figure 4E of the metastasis results is between B16-F10, B16-F10 shMet, and liposome control). It was unclear from the original paper if the B16-F10 cells and exosomes were from untransduced cells or were shScramble; however, during peer review of the Registered Report it was suggested that the B16-F10 cells reported in Figure 4E and Supplementary Figure 5A were shScramble transduced. Thus, we conducted this experiment similar to the original study design. We have revised the manuscript to highlight that the addition of untransduced B16-F10 cells would be a valuable additional control to include for future experiments to determine if any undesirable effects occurred, since infection can cause genome instability.

3) For the analysis of tumor metastasis, are the data normal or not normal? It seems like statistical analyses were used that are appropriate for data with a non-Gaussian distribution. Therefore, I'm not sure why mean and SEM are used to represent the data, which are only appropriate for data with a Gaussian distribution.

The data were normal when natural log transformed, which is what the statistical analysis was performed on. Additionally, Figure 2—figure supplement 1E, F represent the data as box and whisker plots on a natural log transformed y-axis. We agree that representing the data as mean and SEM are not appropriate (Figure 2C, D); however, with this project we presented the replication the same as the original data to allow users to directly compare results. To avoid confusion, we have reversed the figures in the revised manuscript and added a note in the figure legend. That is, the box and whisker plots are presented in the main figure and the box plots are presented in the supplement to provide a comparison to how the original data were presented.

4) For differences between this study and Peinado that are listed in the last paragraph of the Results/Discussion section of the paper, I think differences in Met signaling and in the effectiveness of knockdown (which I also doubt based on the quality of the Western blots here) should be mentioned. In addition, the study was not carried out as long as the Peinado study (18 days instead of 21 days) after exosome education, so this should be mentioned in the last paragraph too. There were also fewer mice, which could definitely affect detection of statistical significance, which should be mentioned.

We have included additional differences in the last paragraph of the revised manuscript. This replication attempt, like all of the replication attempts in the Reproducibility Project: Cancer Biology, are designed to perform independent replications with a calculated sample size to detect the originally reported effect size with at least 80% power based on the originally reported data (Errington et al., 2014). While we used fewer mice, the higher observed variance in the replication is a factor likely influencing if statistical significance is reached. This is included as a consideration when discussing the effect sizes and the RSD comparison between the original study and this replication.

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Citations

    1. Kim J, Afshari A, Sengupta R, Sebastiano V, Gupta A, Kim YH, Iorns E, Tsui R, Denis A, Perfito N, Errington TM. 2018. Study 42: Replication of Peinado et al., 2012 (Nature Medicine) Open Science Framework. [DOI]

    Supplementary Materials

    Transparent reporting form
    elife-39944-transrepform.docx (246.6KB, docx)
    DOI: 10.7554/eLife.39944.008
    Reporting standard 1. The ARRIVE guidelines checklist.
    elife-39944-repstand1.pdf (219.1KB, pdf)
    DOI: 10.7554/eLife.39944.009

    Data Availability Statement

    Additional detailed experimental notes, data, and analysis are available on OSF (RRID:SCR_003238) (https://osf.io/ewqzf/; Kim et al., 2018). This includes the R Markdown file (https://osf.io/hz3k7/) that was used to compose this manuscript, which is a reproducible document linking the results in the article directly to the data and code that produced them (Hartgerink, 2017).

    The following dataset was generated:

    Kim J, Afshari A, Sengupta R, Sebastiano V, Gupta A, Kim YH, Iorns E, Tsui R, Denis A, Perfito N, Errington TM. 2018. Study 42: Replication of Peinado et al., 2012 (Nature Medicine) Open Science Framework.

    Articles from eLife are provided here courtesy of eLife Sciences Publications, Ltd

    RESOURCES