Practicing on the Tip of an Information Iceberg? Evidence of Underpublication of Registered Clinical Trials in Oncology

Abstract

Objective

Members of the International Committee of Medical Journal Editors require as a condition of consideration for publication, that all clinical trials be registered in a public trials registry. We evaluated the proportion of registered trials that are published in peer-reviewed literature.

Methods

After downloading the contents of the National Institutes of Health’s ClinicalTrials.gov registry, we used keywords to identify trials in oncology. We then evaluated the proportion of trials that have been published in journals listed in PubMed.gov. Among trials with published results, we determined the proportion that reported positive versus negative results

Results

Among the 2,028 trials meeting inclusion criteria, 17.6% were available in PubMed. Twenty one percent of trials registered before September 1, 2004 were published compared to 11.9% registered after this date. Trials sponsored by clinical trial networks published the greatest proportion of registered studies (59.0%); studies sponsored by industry published the fewest (5.9%). Among published studies, 64.5% reported the results as positive findings.

Conclusions

Less than 1 in 5 studies in cancer that are registered with clinicaltrials.gov have been published in peer reviewed journals. Research sponsors, researchers, and journal editors should redouble their efforts to encourage publication of registered clinical trials in oncology.

Introduction

The clinical guidelines that support cancer care practice are largely based on evidence gathered from published results of clinical trials. In recognition of the importance of unbiased publishing of these studies, the National Institutes of Health has sponsored a website since 1999--ClinicalTrials.gov—where researchers can list the details of clinical trials at their inception. In 2004, the International Committee of Medical Journal Editors (ICMJE) initiated a publication policy that requires investigators to deposit information about trial design into an accepted clinical trials register before the onset of patient enrollment.[1] Nevertheless, there is no requirement to publish registered trials. The problem of selective publication of trials has been discussed as a general problem in the medical literature,[2–6] and for specific conditions such as depression and cardiovascular disease.[7, 8] Reports in the 1980s raised the issue of possible publication bias in oncology,[9, 10] and studies have noted possible bias of reporting of cancer prognostic factor studies and biotherapies for solid tumors.[11, 12] Nevertheless, there has been little evaluation of the extent to which selective publication may be a general issue for prospective clinical trials in cancer. Accordingly, we undertook a study to examine the degree to which treatment trials in cancer that were registered with ClinicalTrials.gov have been subsequently published in the peer-reviewed literature.

Methods

The U.S. National Institutes of Health, through its National Library of Medicine, developed the ClinicalTrials.gov registry in collaboration with the Food and Drug Administration (FDA), as a result of the FDA Modernization Act, which was passed into law in November 1997. The ClinicalTrials.gov website has been registering studies since the fall of 1999. The website contains abstracts of Clinical Study Protocols that include--among other information--a summary of the purpose of the study, the disease or condition and drug or therapy under study, recruiting status, criteria for patient participation, research study design, phase of the trial, and the sponsor. All trials registered on ClinicalTrials.gov receive a unique trial identifier. Investigators are asked to list the identifier on all publications related to the trial. Since September 2005, the International Committee of Medical Journal Editors (ICMJE) has recommended that published results of registered clinical trials identify the trials’ unique identifiers.[13]

In September 2007, we downloaded the contents of the ClinicalTrials.gov registry. We next searched the database for clinical trials focused on treatments for cancer patients. To be considered a cancer trial, the diseases and conditions section describing the trial had to contain at least one of the following nine search terms: cancer, neoplasm, carcinoma, myeloma, leukemia, lymphoma, melanoma, sarcoma or mesothelioma. Only interventional trials of treatments and trials that were designated as either completed or terminated were included in our study.

Each trial registry page has a section where investigators may list publications of the trial’s results. While this section frequently contains publications, particularly for trials listed as completed, it is not known whether researchers update this section of the registry in a timely fashion relative to publication of their trials. Thus, to find additional publications of results from registered trials, in December 2007, we also searched PubMed for reports of trials that included the unique registry identifier number of each trial as listed in ClinicalTrials.gov. Using the unique identifiers, we then determined the ratio of publications to trials registered in ClinicalTrials.gov. If we found one or more publications containing the trial’s unique identifier in PubMed, we counted the trial as having a publication.

Among trial results that were identified in PubMed, we conducted an additional analysis on the abstracts linked to the registry to determine the proportion that could be considered “positive” versus “negative” studies. We defined negative studies as those that convincingly allow the null hypothesis to be accepted when a test of statistical significance is reported. When there was no such test reported, we relied on the stated primary purpose of the trial and the authors’ conclusion with regards to the reported outcome.

Results

A total of 44,232 trials were registered in the ClinicalTrials.gov as of September 2007. Of these, 11,829 trials contained one of the nine cancer-related search terms in their list of conditions. Of these cancer trials, 2,028 met our other inclusion criteria. Table 1 contains publication frequencies for different categories of trials for patients with cancer. The number of trials by treatment-type sums to more than 2,028 because some trials had more than one treatment-type. The sponsor-type was determined by the primary sponsor of the trial. All sponsor-type categories are provided by the ClinicalTrials.gov website. University/Research Organizations are post-secondary institutions of higher learning or research organizations that are affiliated with post-secondary institutions. Networks are consortiums of medical research organizations, such as the Eastern Cooperative Oncology Group or the Southwest Oncology Group.

Table 1.

Publication Rates of Cancer Trials registered at ClinicalTrials.gov, by Category.

Category	Number of Trials	Percent
		Distribution	Published	Positivea
All cancer trialsb	2,028	100.0%	17.6%	64.5%
Stage
Completed	1,791	88.3%	19.5%	64.3%
Terminated	237	11.7%	3.4%	75.0%
Randomization
Randomized	490	24.2%	19.6%	60.4%
Non-randomized	572	28.2%	4.4%	90.9%
Not specified	966	47.6%	24.4%	63.6%
Treatment-Type
Drug	1,875	92.5%	17.4%	64.5%
Procedure	970	47.8%	25.7%	60.3%
Other	63	3.1%	19.0%	81.8%
Phase
I	579	28.6%	14.9%	89.9%
II	1,076	53.1%	17.3%	53.6%
III	270	13.3%	26.3%	63.2%
IV	50	2.5%	14.0%	83.3%
Not specified	53	2.6%	13.2%	57.1%
Sponsor-Typec
University/Research Org.	853	42.1%	14.0%	70.2%
Industry	614	30.3%	5.9%	75.0%
NIH	209	10.3%	27.3%	78.8%
Network	205	10.1%	59.0%	50.0%
Other	147	7.2%	16.3%	65.2%
Registration Date
Before 9/1/04	1,277	63.0%	21.0%	64.0%
9/1/04 – 9/1/07	751	37.0%	11.9%	66.3%

^aTrial results stated as rejection of the null hypothesis or authors’ conclusions. See text for details. Based on registered trials with published abstracts.

^bInterventional trials that were either completed or terminated as of 9/1/07.

^cAll sponsor-type categories are as listed by the ClinicalTrials.gov website.

University/Research Organizations are post-secondary institutions of higher learning or research organizations that are affiliated with post-secondary institutions. Networks are international quasi-private medical research facilities.

Among cancer clinical trials completed or terminated as of September 2007, 17.6% were listed as published in PubMed or by the registry. Terminated trials had a very low publication rate (3.4%) compared to completed trials (19.5%). Stratified by treatment type, procedures had the highest proportion of publications to trials registered. Phase III trials were more likely to be published (26.3%) compared to other study types, including Phase IV studies (14.0%). Studies funded by Networks were the most likely to be published (59.0%). Industry sponsored trials were the least likely to be published (5.9%). University/Research Organization comprised the largest proportion of primary sponsorship for registered studies. Non-randomized trials were much less likely to be published (4.4%) than randomized trials (19.6%). Trials registered before September 1, 2004, the month the ICMJE began requiring trial registration, were more likely to be published (21.0%) than those published after this date (11.9%).

There were 357 trials with published results. Of those trials, we were able to judge whether the findings were positive or negative for 341 trials. Among these studies, 220 (64.5%) reported positive results. NIH-sponsored trials reported the highest percentage of positive abstracts (78.8%) and Networks reported the lowest (50.0%). Abstracts from phase I and phase IV trials are more likely to report positive results (89.9% and 83.3% respectively) than are abstracts from phase II and phase III trials (53.6% and 63.2% respectively).

Discussion

Timely publication of cancer clinical trials is important to researchers, practitioners, and patients. Evaluating cancer clinical trials registered with ClinicalTrials.gov versus publication of those trials in PubMed, we find that less than 1 in 5 completed trials registered since the inception of ClinicalTrials.gov are listed in the registry or PubMed as published manuscripts. Restricting the sample to trials registered before September 2004, less than 1 in 4 have been published in journals cited by PubMed or the registry. Among abstracts from cancer treatment trials in PubMed that are linked to registered trials, 35.5% may be considered negative studies; that is, the authors’ concluded that the outcome was disappointing or did not merit further consideration of the tested treatment.

Our results raise the concern of publication bias for cancer clinical trials. Allowing for legitimate delays between manuscript submission and publication, it appears likely that less than one-fourth of all clinical trials registered with ClinicalTrials.gov are published in widely accessible journals. Although 35.5% of the published articles could be considered negative studies, we hypothesize that the proportion of unpublished studies contains a much greater proportion of negative findings. It is likely that many unpublished studies contain important information that could influence future research and present practice policy. Of particular concern is our finding that only 6 of every 100 industry-sponsored trials have been published in a PubMed-referenced journal. The great majority of industry sponsored studies concern patented compounds, many of which are in clinical use.

We note several limitations of our analysis. We limited this analysis to studies registered in ClinicalTrials.gov. Originally the ClinicalTrials.gov was the only registry that satisfied the requirements spelled out by the ICMJE. Today there are several trials registries open to the public that are considered acceptable by this organization of journal editors. Most of the alternative registries currently contain very few studies. The one exception is the registry managed by the World Health Organization (WHO).[14] While the WHO registry has been praised as an important step in research transparency,[[15] it contains only 12% of the number of studies registered at ClinicalTrials.gov. Some trials are registered in both registries. In our evaluation of abstracts to determine the proportion of studies that reported significant associations for treatments compared to controls, we did not consider effect size. Thus, trials that showed a positive statistical association but had very modest differences in outcome that some clinicians might consider not clinically meaningful were nonetheless counted as positive studies. Finally, we did not consider published abstracts presented at national meetings. Abstracts have particular importance in shaping cancer practice policy. Nevertheless, abstracts receive a less thorough level of peer-review, are generally not as accessible as published manuscripts, and usually do not contain enough detail to sufficiently determine the quality of the study design and limitations of the findings.

There are many reasons why researchers and sponsors fail to publish clinical trials, particularly those that do not reject the null hypothesis of no relationship between the experimental treatment compared to control therapy or placebo. Authors may view negative studies as having little likelihood of altering medical care or advancing scientific understanding about treatments for a disease. Some might feel that publication of negative studies will not advance their careers or their professional standing among peers. Negative studies may be viewed as more difficult to publish. These issues may compel researchers with many constraints on their available time for manuscript writing time to defer work on manuscripts summarizing negative studies. In some cases, sponsors may either not encourage or actively discourage investigators from publishing negative studies. Finally, all of these issues may pose particular barriers if funding for the project has expired prior to the time when authors have time to write manuscripts summarizing the findings.

When manuscripts summarizing negative studies are submitted to journals, it is possible that reviewers may be less enthusiastic in their evaluations. Negative studies that contradict widely held beliefs about particular interventions or notions of cause and effect may receive harsher criticism about their methods or the authors’ interpretation of the results. Many studies that report findings as negative are, in fact, inconclusive, either due to small sample size or other limitations in methodology.

In response to these issues the ICMJE has issued a statement to journal editors noting their obligation to publish negative studies. In this statement, however, they acknowledge that publication of inconclusive studies is problematic, noting that they, “add little to biomedical knowledge and consume journal resources.”[1]

Publication of clinical trials with negative studies clearly has value to researchers and patients. Researchers benefit from not repeating a negative trial, but they also benefit from what the negative trial implies regarding the conceptual relationship between the experimental treatment and outcomes. Negative trials compel researchers to reconsider hypotheses regarding how the study intervention and related approaches impacts mechanisms of disease, patient or population behaviors, or other pathways between treatment(s) and outcome.

Unpublished trials may have special importance in oncology, due to the toxicity and/or expense of many therapies. For example, CALGB-9633, a controlled trial of adjuvant paclitaxel and carboplatin for patients with stage 1B non-small cell lung cancer, originally reported a statistically significant survival benefit, such that the trial was stopped early. The preliminary findings were the basis for revised NCCN guidelines strongly endorsing adjuvant treatment, and adjuvant therapy became standard of care in this setting.[16] Later evaluation of patients enrolled in this trial did not find a statistically significant improvement in survival.[17] While the findings of the trial have generated controversy regarding optimal management of patients with stage 1 non-small cell lung cancer, the findings from this trial has prompted thought leaders to state that the magnitude of benefit afforded by adjuvant chemotherapy remains uncertain for this population.[18, 19] Of note, the findings from CALGB-9633 have yet to be published as a peer-reviewed manuscript.

If clinical trials are not published, it can be very difficult for clinicians and other decision makers concerned with cancer practice and policy (for example, health insurers) to have an accurate picture of the potential benefits and risks for harm posed by the treatments they prescribe. To gain regulatory approval, researchers and manufacturers must register their trials with the FDA if they wish to use them in support of an application for marketing or a change in labeling.[20] The general public has very limited access to these data, particularly when it is submitted by manufacturers. Researchers sometimes have had to resort to the Freedom of Information Act to access primary data and the FDA’s statistical interpretations of the data from manufacturer-sponsored trials registered with the FDA, a cumbersome and slow process.[7, 21]

A reasonable future goal for researchers will be to identify the proportion of unpublished studies that have negative findings for proprietary and nonproprietary treatments and other interventions that are currently in use today. Such a study will be much more difficult than this evaluation, since we relied on readily available, publicly accessible documents.

Conclusion

In summary, only a minority of clinical studies in cancer registered with ClinicalTrials.gov are published in peer-reviewed and other widely accessible journals. Our findings raise concern about the completeness of available information on present and future cancer treatments. If selective publication has altered the apparent risk–benefit assessments of cancer treatments, doctors and their patients may not be making treatment decisions that are in their best interest.