Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Jul 1.
Published in final edited form as: JAMA Oncol. 2016 Jul 1;2(7):875–881. doi: 10.1001/jamaoncol.2015.6487

The Scientific Impact of Positive and Negative Phase III Cancer Clinical Trials

Joseph M Unger 1, Willliam E Barlow 1, Scott D Ramsey 2, Michael LeBlanc 2, Charles D Blanke 3, Dawn L Hershman 4
PMCID: PMC4945370  NIHMSID: NIHMS774291  PMID: 26967260

Abstract

Importance

Positive Phase III cancer clinical trials are widely hailed, while negative trials are often interpreted as scientific failures. We hypothesized that these interpretations would be reflected in the scientific literature.

Objective

To compare the scientific impact of positive versus negative phase III cancer clinical treatment trials.

Design

We examined the phase III trial history of SWOG over a 30-year period (1985–2014). Scientific impact was assessed according to multiple publication and citation outcomes. Citation data were obtained using Google Scholar™. Citation counts were compared using generalized estimating equations for Poisson regression.

Setting

SWOG, a national cancer clinical trials consortium.

Participants

Any trial that was formally evaluated for the randomized treatment comparison was included for analysis of publication and citation outcomes.

Exposures

Trials were categorized as positive if they achieved a statistically significant result in favor of the new treatment for the protocol-specified primary endpoint, and negative otherwise.

Main Outcomes

The main outcomes were the impact factors for the journals publishing the primary trial results, and the number of citations for the primary trial manuscripts and all secondary manuscripts associated with the trials.

Results

Ninety-four studies enrolling n=46,424 patients were analyzed. Twenty-eight percent of trials were positive (26/94). The primary publications from positive trials were published in journals with higher average impact factors (28.4 vs. 17.5, p=.007) and were cited twice as often as negative trials (average per year, 43 vs. 21, p=.03). However, the number of citations from all primary and secondary manuscripts did not differ between positive and negative trials (average per year, 55 vs. 45, p=.53).

Conclusions and Relevance

The scientific impact of the primary manuscripts from positive phase III randomized cancer clinical trials was twice as great as for negative trials. But when all of the manuscripts associated with the trials were considered, the scientific impact between positive and negative trials was similar. Positive trials indicate clinical advances, but negative trials also have a sizeable scientific impact by generating important scientific observations and new hypotheses and by showing what new treatments should not be used.

Introduction

Phase III trials provide the highest level of evidence for showing the efficacy of new treatments or interventions. The phase III trial programs of the National Cancer Institute’s (NCI’s) National Clinical Trials Network (NCTN) and the NCI’s Community Oncology Research Program (NCORP) are vital national resources and represent a substantial investment on the part of federal agencies. Given the size of the investment, negative trials – that is, those which fail to show that a new treatment is superior to standard treatment – may be incorrectly regarded as poor investments. However negative trials are also important if they show that new treatments, which might otherwise be adopted into clinical practice, in fact do not work.

The rate at which trials are positive has previously been examined, as has the relationship between trial results and publication rates in the context of publication bias.1–4 But the comparative scientific impact of positive versus negative trials using citation data has not been investigated. In this paper, we utilize the phase III trial database of SWOG, a major national cooperative group, in combination with its trial publication database and citation data from Google Scholar, to compare the scientific impact of positive and negative cancer clinical trials.

METHODS

SWOG is a member of the National Cancer Institute’s funded NCTN and NCORP programs. We surveyed the SWOG clinical trial database over the 30 year period 1985–2014 (inclusive). Any phase III clinical treatment trial which opened for enrollment during the period was included. Each trial was previously approved by an Institutional Review Board.

SWOG maintains a comprehensive database of all publications associated with its trials. All SWOG trial publications are required to be vetted through the SWOG publications office for processing and data capture prior to submission. PubMed is periodically searched by the publications office staff to identify any SWOG-related publication that may have been missed. Publication lists are periodically reviewed by SWOG disease committee chairs and senior statisticians for completeness and accuracy.

This report is based on articles published in scientific journals.

Study Outcome Definitions

A trial was defined as “completed” if it was closed to accrual. Any trial that was analyzed in the context of a formal interim analysis, or after achieving ≥50% accrual, was considered “evaluated”. An evaluated trial was defined as “full accrual” if ≥90% of protocol-specified enrollment was achieved;5 “futility” if the trial was closed early following a recommendation by the Data and Safety Monitoring Committee (DSMC) that the experimental therapy would never achieve its protocol-specified endpoint or was too toxic; “efficacy” if the trial was closed early following a recommendation by the DSMC that the trial had shown the benefit of the experimental treatment; and “sufficient” if the trial did not reach full accrual but was analyzed for the treatment comparison. Trials that were not evaluated were either closed early due to poor accrual or for other reasons (such as withdrawal of NCI or pharmaceutical industry support) and were counted as negative trials.

Definition of Positive and Negative Trials

To be categorized as a positive trial, trial results must have achieved a statistically significant finding in favor of the new, experimental therapy for a designated primary endpoint according to the statistical design pre-specified in the study protocol. A trial with a negative results indicated there was a statistically significant finding in favor of standard therapy. A trial with a null result indicated there was no statistically significant benefit for either the experimental or standard therapy. (Of note, a null result does not equate to proving that the treatments were the same, only that there was no evidence they were different.) For presentation purposes, both null and negative trials are regarded as failed experiments and are referred to, collectively, as negative trials.

We also considered the rate of positive trials based on an assessment of net benefit. In this calculation, trials that were positive according to the protocol-specified primary endpoint, but the toxicity for the new therapy was too high to recommend the new treatment, were considered negative. Conversely, trials that did not achieve their protocol-specified primary endpoint for the new treatment but that had other substantial advantages (i.e. lower cost, reduced toxicity) leading the authors to consider the new treatment an advance over standard care, were considered positive.

Trial Descriptive Factors

Trials were described according to the following baseline characteristics: type of cancer, type of design, type of endpoint, the number of randomized comparisons, and type of treatment.

Publication Data

We identified manuscript publications generated from phase III trials. Only evaluated trials were examined for publication outcomes, including citation counts, since the outcome from the randomized comparison in trials that closed early (mostly due to poor accrual) was unknown, so could not be categorized as positive or negative. The “primary” manuscript was the manuscript reporting the results of the analysis for the primary protocol-specified endpoint by randomized treatment arm. A “secondary” manuscript was any other manuscript that relied on data from a given phase III trial, such as subset or meta-analyses. Letters to the editor were excluded.

Scientific impact was evaluated according to multiple manuscript publication endpoints. To assess the potential for publication bias, we calculated the rate at which primary manuscripts were published in any journal. We also calculated the total number of primary and secondary manuscripts associated with each trial. To account for time, for each trial, we calculated the publication rate as the number of publications divided by the number of years from accrual closure date through December 31, 2014. Finally, we calculated the mean impact factor related to the primary manuscript. Since historical impact factors were not fully available, we used current (2015) scientific journal 2-year and 5-year impact factor levels.

Citation Analysis

We also examined scientific impact using citation analysis, a bibliometric tool.6,7 We created a database of the total number of articles in the biomedical literature that cited the SWOG manuscripts in each year following its initial publication using Google Scholar™, a web-based scholarly search engine. Citation data were obtained in July, 2015 for citations that occurred through December 31, 2014. The Web of Science™ search engine was also used to conduct a sensitivity analysis for primary manuscript citation data.

Statistical Considerations

The total number of primary and secondary publications associated with each trial and the mean impact factor were compared between positive and negative trials using t-tests. Secondary manuscripts (and their citation counts) associated with multiple trials were ascribed a weight equal to the inverse of the number of associated trials.

The citation rates for positive and negative trials are a function of the number of trials in each category and the number of years since publication for each trial manuscript. Also, yearly citation counts are likely correlated; that is, a manuscript is more likely to be cited if it has previously been cited. For these reasons, we modeled citation counts using generalized estimating equations (GEEs) for Poisson regression.8,9 The GEE models fit the number of citations by study over time (in yearly intervals). To account for correlation between citation counts, we specified an independence working correlation using robust standard errors. A manuscript was considered “at risk” of being cited in the literature starting the calendar year in which it was published until the end of the period on December 31, 2014. For unpublished trials, the primary manuscript date was set at the year when enrollment completed, and citation counts were assigned as zero for all at-risk years. Citation counts up to 20 years after the primary manuscript publication were analyzed. Model-based expected citation counts for positive versus negative trials are provided, as are relative-risk estimates and their 95% confidence intervals.

We fit models with citation count as the dependent variable and trial outcome status (positive versus negative) as the main independent variable. We also modeled citation patterns over time, including, separately, a linear time variable, a quadratic time variable, and each time variable interacting with trial outcome status. The best fit models were identified by comparing the QICu goodness-of-fit statistic.10 To account for potential secular trends, a trial-level covariate representing the year the primary manuscript was published (<2002 vs. ≥2002) was included in separate analyses. A separate analysis of primary manuscript citations incorporated the trial-level impact factor as a covariate. The GEE Poisson regression analyses were conducted using the SAS procedure PROC GENMOD (SAS v. 9.4; SAS Institute, Cary, NC). All statistical tests where alpha≤.05 were considered significant.

RESULTS

One-hundred seven phase III trials were activated from 1985–2014. Of these, six finished enrollment but the data were still maturing, and seven were still open for enrollment (eFigure 1 in the Supplement). Therefore n=94 studies involving 46,424 patients were examined. Trial characteristics are shown in eTable 1 in the Supplement. Breast, gastrointestinal, genitourinary, leukemia, lung, and lymphoma cancer trials were most common (≥10% each). Nearly all trials had superiority designs (96%) and included evaluation of systemic therapy (96%). The majority were designed to assess either survival alone (37%) or some other time-to-event endpoint alone (28%). There were no observed differences between positive and negative trials in factors that might influence citations rates, such as the incidence of the cancer type or the nature of the treatment.

Accrual Outcomes

Among the 94 completed studies, nearly half reached full accrual (46/94, 49%), five (5%) closed early due to an efficacy finding and 10 (11%) due to a futility finding, and 5 (5%) were analyzed after reaching sufficient accrual (eFigure 1 in the Supplement). Among the trials closed early for futility, nine closed because the experimental treatment was worse – or would never be better – than standard treatment, and one because the experimental treatment was too toxic. In no case did a study reach full accrual and the experimental treatment was found to be statistically significantly worse than standard treatment. Twenty-eight percent of trials were closed early due to poor accrual.

Rate of Positive Phase III Trials

Overall, 26 of 94 completed trials (28%) were positive according to the protocol-specified primary endpoint analysis. Given the likelihood that trials that fail are more likely to fail quickly, we excluded seven trials activated in the most recent 10 years (2005–2014); the rate of positive trials was 30% (26/87). Among the 66 evaluated trials, the rate of positive trials was 39% (26/66).

Two trials were positive according to the protocol-specified primary endpoint, but the new therapy was sufficiently toxic that the authors did not recommend it over standard therapy. Four other trials did not achieve their protocol-specified primary endpoint but were considered “positive” by the study authors based on other treatment benefits. Therefore in 28/94 trials (30%), the overall risk-benefit profile was considered to be in favor of the new treatment.

Publication Outcomes

Among the n=66 trials that were evaluated for their primary endpoint, 38 out of 40 negative trials had a manuscript publication (95%) and all 26 positive trials had a manuscript publication (100%). There was no statistically significant evidence of publication bias (p=.25).

In total, 273 unique manuscripts were published during the period, including 229 manuscripts associated with a single trial and 44 manuscripts associated with two or more trials. Among the 209 secondary manuscripts, the most common types pertained to biomarker predictors of outcomes (52, 25%); clinical, treatment, or demographic predictors of outcomes (52, 25%); and investigations of cancer biology (43, 21%). Twenty-four of the secondary manuscripts (11%) were from collaborative database or meta-analyses with other research groups. Neither the mean number of publications (3.7 vs. 4.4, p=.57) nor the publication rate relative to the number of years since trial accrual completion date (0.23 vs. 0.35 per year, p=.16) differed between positive vs. negative trials (respectively).

The mean 2-year impact factor was higher for the primary manuscript publications of positive versus negative trials (28.4 vs. 17.5, p=.007). Results were similar for the 5-year impact factor. Also, the proportion of studies published in very high impact factor journals (the Journal of the American Medical Association, the Lancet, the New England Journal of Medicine) was higher for positive trials (35%) than for negative trials (13%; p=.03).

Citation Analysis

For the 66 evaluated trials, there were 24,235 citations over 783 at risk years for the primary manuscripts, including 15,064 citations over 350 years for positive trials and 9,171 citations over 433 years for negative trials (Figure 1A).

Figure 1.

Figure 1

Open in a new tab

Number of citations by calendar year for primary manuscripts (Panel A), secondary manuscripts (Panel B), and the combination of primary and secondary manuscripts (Panel C). Calendar year was truncated at 1995 for presentation purposes. The number of publications that were at risk of citation are shown below each panel.

The model-adjusted citation rate per at risk year was 43 for positive trials and 21 for negative trials. In multi-level Poisson regression, this difference was statistically significant (relative risk=2.0, 95% CI, 1.1–3.9, p=.03). Ignoring autocorrelation, the strength of the difference was much greater (p<.001). Adjusting for the journal impact factor as covariate, results were consistent but weaker (relative risk=1.5, 95% CI, 0.9–2.4, p=.12). There was no evidence that the pattern of higher citation counts for positive trials differed according to whether the primary manuscript was published in a high impact journal (rates for positive vs. negative trials: 91 vs. 67 in high impact factor journals, and 24 vs. 15 in low impact factor journals; interaction p-value=.75).

The best fit model included both a linear and quadratic interaction of time and whether the study was positive or negative (eTable 2 in the Supplement). This can be interpreted as indicating that there was a quadratic relationship between citation counts and time which differed for the positive vs. negative trials. For positive trials, average citation counts increased for about 10 years after initial publication, after which they decreased, at approximately a similar rate, out to 20 years; for negative trials, citations counts also increased out to about 10 years, but at a more modest pace, after which average citation counts slowly decreased (Figure 2A).

Figure 2.

Figure 2

Open in a new tab

Average citations by year after primary manuscript publication for primary manuscripts (Panel A) and both primary and secondary manuscripts (panel B). The solid blue lines show the model fitted results for positive trials, and the solid red lines show the model fitted results for negative trials. The blue crosses indicate observed mean number of citations for positive trials, and the red dots indicate observed mean number of citations for negative trials. The observed and fitted values occurring prior to the primary manuscript publication year were due to citations from secondary publications associated with a given trial (for instance, articles reporting on the trial design). Citation counts for the first 20 years after primary manuscript publication were included for the analysis of primary manuscripts only. For the analysis of primary and secondary manuscripts, citation counts from 5 years prior to primary manuscript publication to 20 years after primary manuscript publication were included.

For secondary manuscripts, citation counts were much higher for negative trials (12,461 vs. 6029, Figure 1B), while for the combination of primary and secondary manuscripts, citation counts were similar for positive vs. negative trials (21,093 vs. 21,632, respectively; Figure 1C). The model-adjusted citation rate per at risk year was 55 for positive trials and 45 for negative trials (relative risk=1.2, 95% CI, 0.7–2.3, p=.53; Figure 2B). The absence of a statistically significant overall difference between positive and negative trials was due to the substantial number of citations from secondary publications for negative trials. As before, the best fit model included a quadratic interaction of time and whether the study was positive or negative (eTable 2 in the Supplement), although the strength of the interaction was weaker than for the primary manuscript citations.

Sensitivity and Outlier Analyses

Using the Web of Science search engine, the total absolute number of citations for primary manuscripts was lower (n=15,828), including 9749 for positive trials and 6079 for negative trials. The model-adjusted citation rate for positive trials was 28 and for negative trials was 14 (relative risk=2.0, 95% CI, 1.0–3.8, p=.04). As before, a quadratic interaction model best fit the data (data not shown).

We assessed the potential for very highly cited positive and negative trials to influence the analysis of total primary and secondary manuscript citations by excluding the two top positive and negative trials with the greatest number of citations from all associated manuscripts. The simple citation rate per at risk year was 41 for positive trials and 36 for negative trials (relative risk=1.1, 95% CI, 0.6–2.0, p=.66; eFigure 2 in the Supplement). The observed modest early increase in total citations for negative trials (prior to Year 0; Figure 2B) is no longer evident, suggesting the early difference was due to a few negative trials with a large amount of well-cited secondary manuscript publications.

DISCUSSION

We found that the scientific impact of primary manuscripts from positive trials was greater than for negative trials, as measured by publication in a very high impact journal and average citation rates over 20 years. But negative trials also had a sizeable impact on the scientific literature. When the citation counts from both the primary and secondary publications were considered, the total scientific impact between positive and negative trials was roughly comparable.

Recent evaluations of NCI-sponsored cooperative group trials showed high rates of full publication of completed trials in journals (≥90%).1,4 Our findings were similar, showing an overall publication rate of 97%, which did not statistically significantly differ according to whether trial results were positive or negative. Earlier observations about the potential for publication bias in clinical trials, and the subsequent requirement that all NIH-funded clinical trials be registered through clinicaltrials.gov, has likely helped to reduce publication bias in NCI-sponsored cooperative group trials.2,11,12

Overall, randomized phase III trials generated positive results at a rate of 28%, consistent with prior evidence and consistent with the idea that important scientific questions should have equipoise.1,4,13 That is, a comparative phase III trial should only be conducted in an environment in which true uncertainty exists as to whether a new treatment is better than standard treatment. In this setting, the likelihood that an individual trial will be positive should lie within a certain range; if the likelihood of success is too high or too low, then patients and clinicians are unlikely to entrust their treatment choice to random assignment.13 Moreover, if the scientific question does have equipoise, then negative trial results reduce uncertainty about which treatments should direct guideline-based care. Thus their publication serves a vital scientific and societal interest. It is also essential to publish negative trial results given the costliness of trials; the high probability that the clinical experiment is unlikely to be repeated (or, conversely, to reduce the risk that a similar trial is repeated due to lack of knowledge of a negative trial’s results); and importantly, in fulfillment of the obligation to the trial participants who volunteered to participate.

The citation trends observed in this study point to patterns of advances in treatment discovery. Positive and negative trials achieved their maximum citation rates at approximately 10 years after publication. Thereafter, citation counts diminished, likely because new treatments were developed to replace prior standards. However, even 20 years after publication, both positive and negative trials remained frequently cited (15–20 per year), suggesting that successfully completed phase III trials – regardless of their outcome – set the stage for new treatment discovery for decades to come.

These findings also point to the tremendous secondary value of well-conducted Phase III clinical trials. Fully 43% of total citations (18,490/42,725) in this set of trials were from secondary manuscripts. Clinical trial patients are typically younger and healthier than non-trial patients.14,15 However, randomized Phase III trials provide large samples of uniformly staged and treated patients with prospective data collection and long term follow-up. In an age of translational medicine research, nearly all trials also collect biologic ancillary data to investigate and generate important hypotheses regarding prognostic and predictive biomarkers. Finally, patients who receive treatment on trials have access to health care in a protocol-specified treatment setting, so studies of health outcomes and health care utilization by demographic or socioeconomic groups using clinical trial cohorts are less subject to confounding by access to care.16 Indeed in some cases a negative trial may actually be more advantageous as a data resource for secondary science, since the treatment effect can also be discounted and the full trial dataset used. In total, trial data represent a powerful resource for well-designed secondary data analyses. The power of this data resource is increasingly recognized, as reflected by a recent Institute of Medicine report, by the mission of the new Cancer Care Delivery Research program of the NCI’s NCORP research base, and by the increasing efforts of pharmaceutical companies to advance their trial data sharing.17–20

This study was potentially limited by the fact that citation data are likely incomplete, although Google Scholar has been estimated to be at least as comprehensive as other scholarly search engines.21,22 The use of Google Scholar for citation data also provided a different profile of citation data than Web of Science.23–24 Although this affected absolute citation counts, it did not impact the relative citation patterns between positive and negative trials. In addition, a given citation does not represent a uniform level of scientific impact, and citation analysis may not fully represent the scientific impact of a given trial. As such, we defined scientific impact as a multidimensional construct incorporating multiple publication endpoints.25,26 Other ways to assess scientific impact are also possible, such as usage log data.27 Finally, these data represent only a single cooperative group. Publication patterns for other cooperative cancer groups of the NCI or for non-cooperative group trials may be different.

Positive phase III trials indicate clinical advances, and as such their scientific impact as reflected in the literature is substantial. But well-designed and conducted negative trials also have a sizeable scientific impact by generating important scientific observations and new hypotheses and by showing what new treatments should not be used. Inevitably, the trials with the least scientific impact are those which close early due to poor accrual or other issues. This suggests the importance of designing trials with strong support in the scientific and treatment communities. Once a trial is successfully completed, the totality of its scientific impact, considering both primary and secondary results, promises to be substantial, regardless whether the trial results are positive or negative.

Acknowledgments

Funding/Support: Supported in part by Dr. Charles A. Coltman, Jr., Fellowship Program of the Hope Foundation; and by National Institutes of Health, National Cancer Institute, NCI Community Oncology Research Program (NCORP) Research Base grant 5UG1CA189974-01

Footnotes

Author Contributions: Dr. Unger (SWOG Statistical Center, Fred Hutchinson Cancer Research Center) had full access to all of the data in the study, conducted the analysis, and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Conflict of Interest Disclosures: The authors have no disclosures.

Role of the Funder/Sponsor: The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.

References

  • 1.Djulbegovic B, Kumar A, Soares HP, et al. Treatment success in cancer: new cancer treatment successes identified in phase 3 randomized controlled trials conducted by the National Cancer Institute-sponsored cooperative oncology groups, 1955 to 2006. Arch Intern Med. 2008 Mar 24;168(6):632–42. doi: 10.1001/archinte.168.6.632. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009 Jan 21;(1):MR000006. doi: 10.1002/14651858.MR000006.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Ramsey S, Scoggins J. Commentary: practicing on the tip of an information iceberg? Evidence of underpublication of registered clinical trials in oncology. Oncologist. 2008 Sep;13(9):925–9. doi: 10.1634/theoncologist.2008-0133. Epub 2008 Sep 15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Soares HP, Kumar A, Daniels S, et al. Evaluation of new treatments in radiation oncology: are they better than standard treatments? JAMA. 2005 Feb 23;293(8):970–8. doi: 10.1001/jama.293.8.970. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Korn EL, Freidlin B, Mooney M, Abrams JS. Accrual experience of National Cancer Institute Cooperative Group phase III trials activated from 2000 to 2007. J Clin Oncol. 2010 Dec 10;28(35):5197–201. doi: 10.1200/JCO.2010.31.5382. Epub 2010 Nov 8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Garfield E. Citation analysis as a tool in journal evaluation. Science. 1972;178:471–479. doi: 10.1126/science.178.4060.471. [DOI] [PubMed] [Google Scholar]
  • 7.Narin F. Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of Scientific Activity. Washington, DC: National Science Foundation; 1976. [Google Scholar]
  • 8.Liang KY, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73(1):13–22. [Google Scholar]
  • 9.Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics. 1986 Mar;42(1):121–30. [PubMed] [Google Scholar]
  • 10.Pan W. Akaike’s information criterion in generalized estimating equations. Biometrics. 2001 Mar;57(1):120–5. doi: 10.1111/j.0006-341x.2001.00120.x. [DOI] [PubMed] [Google Scholar]
  • 11.Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986 Oct;4(10):1529–41. doi: 10.1200/JCO.1986.4.10.1529. [DOI] [PubMed] [Google Scholar]
  • 12. [Accessed August 30, 2015];NIH News Release. 2000 Feb 29; Available online at: http://www.nih.gov/news/pr/feb2000/nlm-29.htm.
  • 13.Djulbegovic B. The paradox of equipoise: the principle that drives and limits therapeutic discoveries in clinical research. Cancer Control. 2009 Oct;16(4):342–7. doi: 10.1177/107327480901600409. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Hutchins LF, Unger JM, Crowley JJ, Coltman CA, Jr, Albain KS. Underrepresentation of patients 65 years of age or older in cancer-treatment trials. N Engl J Med. 1999 Dec 30;341(27):2061–7. doi: 10.1056/NEJM199912303412706. [DOI] [PubMed] [Google Scholar]
  • 15.Unger JM, Barlow WE, Martin DP, et al. Comparison of survival outcomes among cancer patients treated in and out of clinical trials. J Natl Cancer Inst. 2014 Mar;106(3):dju002. doi: 10.1093/jnci/dju002. Epub 2014 Mar 13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Hershman DL, Unger JM. ‘Minority report’: how best to analyzed clinical trial data to address disparities. Breast Cancer Res Treat. 2009;118:519–521. [Google Scholar]
  • 17.Institute of Medicine (IOM) Sharing clinical trial data: Maximizing benefits, minimizing risk. Washington, DC: The National Academies Press; 2015. [PubMed] [Google Scholar]
  • 18.Kent EE, Mitchell SA, Castro KM, et al. Cancer Care Delivery Research: Building the Evidence Base to Support Practice Change in Community Oncology. J Clin Oncol. 2015 Aug 20;33(24):2705–11. doi: 10.1200/JCO.2014.60.6210. Epub 2015 Jul 20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19. [on December 12, 2015];User Guide, Version 4.0. ClincalStudyDataRequest.com. Accessed online at https://clinicalstudydatarequest.com.
  • 20. [on December 12, 2015];The YODA Project. Accessed online at http://yoda.yale.edu/welcome-yoda-project.
  • 21.Khabsa M, Giles CL. The number of scholarly documents on the public web. PLoS One. 2014 May 9;9(5):e93949. doi: 10.1371/journal.pone.0093949. eCollection 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Harzing A. A longitudinal study of Google Scholar coverage between 2012 and 2013. Scientometrics. 2014 Jan;98(1):565–575. [Google Scholar]
  • 23.Bakkalbasi N, Bauer K, Glover J, Wang L. Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomed Digit Libr. 2006 Jun 29;3:7. doi: 10.1186/1742-5581-3-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Meho LI, Yank K. Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. J Amer Soc Inform Science Tech. 2007;58(13):2105–2125. [Google Scholar]
  • 25.Fortin JM, Currie DJ. Big Science vs. Little Science: How Scientific Impact Scales with Funding. PLoS One. 2013 Jun 19;8(6):e65263. doi: 10.1371/journal.pone.0065263. Print 2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.King DA. The scientific impact of nations. Nature. 2004 Jul 15;430(6997):311–6. doi: 10.1038/430311a. [DOI] [PubMed] [Google Scholar]
  • 27.Bollen J, Van de Sompel H, Hagberg A, Chute R. A principal component analysis of 39 scientific impact measures. PLoS One. 2009 Jun 29;4(6):e6022. doi: 10.1371/journal.pone.0006022. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES