Publication of Trials Funded by the National Heart, Lung, and Blood Institute

Abstract

BACKGROUND

Rapid publication of clinical trials is essential in order for the findings to yield maximal benefits for public health and scientific progress. Factors affecting the speed of publication of the main results of government-funded trials have not been well characterized.

METHODS

We analyzed 244 extramural randomized clinical trials of cardiovascular interventions that were supported by the National Heart, Lung, and Blood Institute (NHLBI). We selected trials for which data collection had been completed between January 1, 2000, and December 31, 2011. Our primary outcome measure was the time between completion of the trial and publication of the main results in a peer-reviewed journal.

RESULTS

As of March 31, 2012, the main results of 156 trials (64%) had been published (Kaplan–Meier median time to publication, 25 months, with 57% published within 30 months). Trials that focused on clinical events were published more rapidly than those that focused on surrogate measures (median, 9 months vs. 31 months; P<0.001). The only independent predictors of more rapid publication were a focus on clinical events rather than surrogate end points (adjusted publication rate ratio, 2.11; 95% confidence interval, 1.26 to 3.53; P = 0.004) and higher costs of conducting the trial, up to a threshold of approximately $5 million (P<0.001). The 37 trials that focused on clinical events and cost at least $5 million accounted for 67% of the funds spent on clinical trials but received 82% of the citations. After adjustment of the analysis for a focus on clinical events and for cost, trial results that were classified as positive were published more quickly than those classified as negative.

CONCLUSIONS

Results of less than two thirds of NHLBI-funded randomized clinical trials of cardiovascular interventions were published within 30 months after completion of the trial. Trials that focused on clinical events were published more quickly than those that focused on surrogate end points. (Funded by the National Heart, Lung, and Blood Institute.)

Rapid publication of the results of clinical trials is widely recognized as essential in order for the findings to yield maximal benefits for public health, facilitate scientific progress, and enable clinicians and other stakeholders to make decisions that reflect an accurate, balanced perspective on existing evidence.^1–4 Within the National Heart, Lung, and Blood Institute (NHLBI) Division of Cardiovascular Sciences, slightly less than half the extramural funds are used to support clinical research; a substantial proportion of those funds support trials.⁵ Some observers have offered evidence in support of the belief that the findings of federally funded trials are not always published in a timely manner.^4,6,7 This issue is of particular concern because randomized trials may be more likely to be published than the results of other kinds of clinical studies. We conducted an extensive evaluation of the publication of the results of NHLBI-funded trials of cardiovascular interventions for which data collection had been completed during the period from 2000 through 2011.

METHODS

ELIGIBLE TRIALS

We analyzed randomized clinical trials that were supported by grants or contracts from the NHLBI extramural cardiovascular divisions, were registered at ClinicalTrials.gov, and had data collection with respect to the primary end point completed between January 1, 2000, and December 31, 2011. We provisionally identified 2183 candidate studies; 244 trials met all the inclusion criteria. The last author vouches for the accuracy and completeness of the data.

OUTCOMES OF THIS STUDY AND CHARACTERISTICS OF THE TRIALS

Our primary outcome was the time from completion of the trial to publication of the main results (online or in print, whichever came first); follow-up for the primary outcome ended on March 31, 2012. Our secondary outcome was the annual citation rates for the published articles. We used the Scopus citation database to obtain annual citation counts (including self-citations) through December 31, 2012, for publications included in the analysis of the primary outcome. We calculated the annual citation rates by dividing the total number of citations by the number of years since publication or by the number of years since completion of the trial. The former metric considered only published studies and did not count publication delays against citation rates, whereas the latter considered both published and unpublished trials and penalized studies with long delays between completion and publication.

We considered eight trial characteristics as candidate predictors of the time to publication. These included the nature of the primary end point (clinical event or other), the total cost to the NHLBI, whether the award was made to multiple participating centers, whether support was provided through a contract or a cooperative agreement, the number of participants who underwent randomization, the nature of the intervention tested (behavioral or other), the unit of randomization (individual or cluster), and, to account for secular trends, the confirmed completion date of the trial. In secondary analyses, we considered whether the trial yielded a positive result, which we defined as a significant between-group difference in the primary end point favoring the investigators’ stated hypothesis (e.g., the superiority or noninferiority of one intervention to another). The last author adjudicated the results for each trial by reading the article that summarized the primary results of the trial (if there was such an article) and, when necessary, directly contacting the principal investigator, reading internal NHLBI documents and correspondence, or both.

We defined clinical end-point events as discrete events with immediate direct adverse effects, such as death, myocardial infarction, stroke, hospitalization, or bone fracture. Examples of nonclinical end-point events included surrogates (which were often continuous rather than categorical measures) such as quality of life, measures obtained by assessment of biomarkers or with the use of imaging, physiological measurements such as weight or blood pressure, and health-related behaviors such as diet, smoking, or frequency of exercise. A behavioral intervention was defined as an intervention that was aimed at altering a health-related behavior, rather than an intervention that involved administration of a drug, the use of a device, or the performance of a procedure.

We considered direct and indirect study costs to the NHLBI, but we did not consider cash or in-kind contributions from non-NHLBI sources. Approximately one third of the trials were supported by complex, multiproject grants and contracts (e.g., those through Program Project, the Specialized Centers of Clinically Oriented Research program, and research networks), and the costs of individual trials funded by those grants and contracts were not easily separated. In such cases, we personally contacted investigators and NHLBI staff to estimate approximately how the costs of the grants or contracts were apportioned among the component research aims.

STATISTICAL ANALYSIS

For descriptive purposes, we tabulated Kaplan–Meier estimates of publication rates at 12, 30, and 48 months, and we tabulated annual citation rates according to each of the aforementioned eight predictor variables. The median time to publication was estimated with the use of a Kaplan–Meier plot of the times to publication of the published manuscripts; data on articles not yet published were censored on March 31, 2012.

According to approaches described by Harrell, ⁸ we constructed univariable and multivariable Cox proportional-hazards models to quantify the associations of our eight candidate predictors with the rapidity of publication. We identified the variables that were most important by constructing random survival forests,⁹ and we confirmed the proportional-hazards assumption by means of analyses of Schoenfeld residuals.⁸ We excluded excessive collinearity by calculating variance inflation factors.¹⁰ We tested plausible interactions, but none emerged as significant. The statistical analyses were performed with the use of the Hmisc and rms packages in the R statistical package, version 2.15.1 (www.r-project.org).

RESULTS

TRIALS

Figures 1A and 1B summarize the distributions of trial costs, the duration (from the time the grant or contract was awarded to completion), and the sample size among the 244 trials included in the analysis. Most trials cost less than $5 million, included fewer than 1000 participants, and lasted less than 5 years.

Figure 1. Costs and Sample Sizes of Trials According to the Duration of the Trial.

Descriptive histograms of 244 extramural randomized trials supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health (NIH) show the associations between the cost of conducting the trial and the duration of the trial (from the time the funding was awarded to the time the study was completed) (Panel A) and between the sample size and the duration of the trial (Panel B).

As of March 31, 2012, the results of the primary end points had been published for 156 of these 244 trials (64%). As of March 11, 2013, manuscripts reporting the primary results of an additional 34 trials (14%) had been submitted to peer-reviewed journals; 8 have now been published. We have evidence that analysis of the primary end point has been completed for 27 other trials (11%). Analyses of the remaining 27 trials (11%), with study completion dates ranging from May 2001 to December 2011, are incomplete.

PREDICTORS OF TIME TO PUBLICATION

Among the 232 trials for which results were known as of January 4, 2013, a total of 98 (42%) yielded positive results, and 134 (58%) yielded negative results. Table 1 summarizes the publication and citation rates according to the trial characteristics. In univariable analyses, predictors of earlier publication included clinical events as the primary end points, higher costs of conducting the trial, multicenter support, funding by contract or cooperative agreement, larger sample size, and nonbehavioral interventions. A positive trial outcome was unrelated to the time to publication in univariable analyses, even in subsets of trials for which the outcomes of more than 98% of the trials were known (those completed before 2010, those with end points that were clinical events, and those that cost $5 million or more). Figure 2 shows Kaplan–Meier rates of publication according to whether the focus of the trial was clinical events or surrogate end points.

Table 1.

Rates of Publication and Citation and Costs of Groups of Clinical Trials.

Variable	No. of Trials	Median Time to Publication	Proportion of Trials Published			Unadjusted Rate Ratio for Likelihood of Publication (95% CI)	Cost to NHLBI	Annual Citation Rate*
			Within 12 mo	Within 30 mo	Within 48 mo
		mo		percent			$ million
All trials	244	25	23	57	75	—	2,021	56
Clinical-event end point
Yes	45	9	64	95	100	5.47 (3.74–7.98)	1,381	135
No	199	31	12	48	68	Reference	640	13
Cost
≥$5 million	60	9	63	91	97	4.92 (3.49–6.95)	1,595	117
<$5 million	184	35	9	45	66	Reference	427	10
Multiple awards
Yes	37	8	62	92	95	3.68 (2.52–5.38)	1,303	139
No	207	30	15	50	71	Reference	718	30
Contract or cooperative agreement
Yes	64	11	55	83	90	2.81 (2.02–3.90)	1,560	113
No	180	31	11	48	69	Reference	461	14
Sample size
≥1000 participants	49	11	54	88	96	3.87 (2.38–6.30)	1,426	134
150–999 participants	134	30	14	50	71	1.14 (0.94–1.74)	466	16
<150 participants	61	30	14	49	65	Reference	129	20
Behavioral intervention
Yes	135	32	11	48	72	0.59 (0.43–0.81)	932	18
No	109	20	36	68	80	Reference	1,089	79
Cluster randomization
Yes	46	29	5	55	65	0.72 (0.47–1.10)	174	11
No	198	24	27	58	77	Reference	1,848	61
Positive outcome
All trials†
Yes	98	23	22	57	76	1.06 (0.77–1.45)	363	29
No	134	25	24	60	77	Reference	1,634	73
Trials completed before January 1, 2010
Yes	71	22	21	58	76	1.13 (0.80–1.58)	287	30
No	97	25	23	59	77	Reference	1,419	75
Clinical-event end point and cost ≥$5 million
Both	37	9	70	97	100	7.55 (4.99–11.42)	1,356	154
Positive outcome	6	7	83	100	100		47	134
Negative outcome	31	9	68	97	100		1,309	158
One	31	15	46	79	92	3.46 (2.22–5.41)	264	32
Positive outcome	12	9	53	76	84		145	35
Negative outcome	19	23	39	82	100		119	29
Neither	176	35	7	44	65	Reference	402	8
Positive outcome	80	31	10	49	73		193	8
Negative outcome	84	36	5	41	61		184	8

^*Shown is the average annual citation rate per published trial, which was calculated by dividing the total citations by the number of years since publication.

^†Information about 12 of the trials was insufficient to classify the outcome as positive or negative.

Figure 2. Time to Publication According to Type of End Point.

Shown are Kaplan–Meier estimates of the time to publication, with trials classified according to whether the primary end point focused on clinical events or surrogate end points. The shading represents 95% confidence intervals.

Table 2 shows the trial characteristics according to the type of end point and the cost of the trial. The 37 trials that had both a focus on clinical events and a cost of at least $5 million were more likely to involve multiple awards, to be funded through contracts or cooperative agreements, to enroll more than 1000 patients, to evaluate nonbehavioral interventions, and to have individuals rather than clusters as the unit of randomization. Only 16% of these 37 trials, as compared with almost half of the other trials, had positive outcomes.

Table 2.

Trial Characteristics According to Status with Respect to Clinical-Event End Point and a Cost of $5 Million or More.

Variable	Both Clinical-Event End Point and Cost ≥$5 Million (N = 37)	Either Clinical- Event End Point or Cost ≥$5 Million (N = 31)	Neither Clinical- Event End Point nor Cost ≥$5 Million (N = 176)
Total cost (millions of $)	1356	264	402
Mean cost per trial (millions of $)	36.6	8.5	2.3
Characteristic of the trial (%)
Multiple awards*	57	45	1
Funded by contract or cooperative agreement	84	61	8
Sample size ≥1000	84	26	6
Behavioral intervention	14	45	66
Cluster randomization	5	10	23
Completion date before January 1, 2010	76	65	70
Primary-end-point results
Positive	16	39	45
Negative	84	61	48
Uncertain	0	0	7

^*This category does not include all multicenter trials, since some of those trials were funded through a single grant or contract.

In multivariable analyses, independent predictors of time to publication were a focus on clinical events (adjusted publication rate ratio, 2.11; 95% confidence interval, 1.26 to 3.53; P = 0.004) and higher costs up to approximately $5 million. Above $5 million, the cost of the trial was no longer a significant determinant of the time to publication (P<0.001 as a nonlinear association) (Fig. S1 in the Supplementary Appendix, available with the full text of this article at NEJM.org). Although a positive trial outcome was not predictive of the time to publication in the univariable analysis, multivariable analyses did show a publication preference for positive trials (Fig. 3A). To reduce the potential bias resulting from the 12 trials with unknown outcomes, we performed an analysis that was restricted to trials completed before January 2010 (for which only 4 of 172 results were unknown), and found essentially the same publication preference for positive trials (Fig. 3B).

Figure 3. Time to Publication According to Trial Results.

Shown are Kaplan–Meier estimates of the time to publication in any journal, with trials classified according to whether the trial results were positive (i.e., showed a significant between-group difference in the primary end point favoring the investigators’ stated hypothesis) or negative. Estimates are shown for all 232 trials for which results are known to the NHLBI (Panel A) and for trials that were completed before January 1, 2010 (Panel B).

In a supplementary analysis of trials funded by grants only, there was no significant association between the peer-review priority scores received before the trial was funded and the time to publication (adjusted P = 0.42) (Fig. S3 in the Supplementary Appendix).

CITATION RATES

The 37 trials focusing on clinical events and costing more than $5 million received 82% of all citations after publication, whereas trials that neither focused on clinical events nor cost more than $5 million received few citations (Fig. S2A and S2B in the Supplementary Appendix). Table S1 in the Supplementary Appendix lists the 31 primary-results articles for which the mean citation rates exceeded 40 per year since the date of publication. All these trials either used clinical events as the primary end points (27 trials) or cost more than $5 million (29 trials); most (25 trials) did both. Of the 31 high-impact trials listed in this table, only 8 (26%) had positive results. At the other end of the spectrum, 16 trials that were published on or before March 31, 2012, had received less than one citation per year since publication and 6 had yet to receive even one citation.

DISCUSSION

We identified 244 NHLBI-supported randomized clinical trials of cardiovascular interventions that were completed between January 2000 and December 2011. The main results of only 57% of these trials were published within 30 months after completion of the trial. Two independent predictors of more rapid publication were a focus on clinical events as the primary end point and higher costs of conducting the trial. However, higher costs predicted publication only up to a total of $5 million; above $5 million, there was no significant association between the cost of conducting the trial and the likelihood of rapid publication. Trials that focused on clinical events and had costs exceeding $5 million received 82% of the total citations of articles reporting primary end-point results, whereas they accounted for only 67% of the total funds allocated to randomized trials of cardiovascular interventions. Trials that yielded negative results accounted for a majority of the trials we analyzed — an observation consistent with a recently reported review of cancer trials.¹¹ In unadjusted analyses, the results of negative trials tended to be published just as quickly as those of positive trials; however, publication preference for positive trials became evident after we adjusted for the type of end point and the cost of the trial. This finding probably reflects the fact that most of the large, clinical-event trials had negative results.

Our findings have potentially important policy implications when they are considered in the context of the U.S. clinical research enterprise. The public benefits of clinical research are diminished or lost when the results of clinical trials are not published. A number of parties share responsibility for this situation, including funders, investigators, academic medical centers, university promotions committees, regulators, clinical research organizations, peer reviewers of grant applications and manuscripts, and journals. It is reasonable to expect all parties to increase and coordinate their efforts to correct the problem.

From the perspective of the NHLBI, the problem may well begin with our articulation of clinical research priorities and with our approach to making funding decisions. Most of the trials in our cohort were funded through relatively small investigator-initiated research grants, and these were precisely the trials that were published slowly, if at all. The NHLBI approach to funding investigator-initiated grants is arguably better suited to encouraging discovery research than to securing the delivery of a specific product, such as publication of trial results that would be expected to have a direct effect on clinical practice or policy. In many cases, publication may not occur until after a grant is concluded or nearly concluded. Nonetheless, we acknowledge that the NHLBI, working in concert with other parties, could play a more active role in better understanding the proximate and root causes of the delay in publication of trial results, in redirecting our funding priorities toward the trials that are most likely to be published quickly and to have high impact within the biomedical community, and, when appropriate, in communicating to grant and contract recipients our expectation of timely publication. The data presented in this article and elsewhere⁴ have already stimulated intensive internal policy dialogues at the highest levels of the National Institutes of Health. These dialogues have focused not only on what our responsibilities are after an award has been granted but also on how we should use our observations to inform future funding priorities during times of increasing fiscal austerity.

Our study has some limitations. First, we included only trials that were registered in ClinicalTrials.gov. Second, by focusing on trials with completed follow-up, we selected only those that achieved an important research objective, and we may have therefore created a more optimistic picture of publication performance than would be suggested by a broader analysis. Third, we did not choose to identify and credit investigators for publication of secondary findings. The primary results are of unique importance in randomized clinical trials and are the main interest of the NHLBI as a funder. Clinical trials, especially when they are large-scale and event-driven, differ fundamentally in this respect from discovery research and smaller-scale surrogate studies, for which the aims tend to be broader and the investigators have more leeway to modify their work midstream. Finally, we had to rely on non–peer-reviewed materials to evaluate the outcomes of unpublished trials.

We were gratified to confirm the rapid publication and high impact of our most expensive trials with the most direct implications for clinical care. Indeed, studies such as the Women’s Health Initiative continue to receive many hundreds of citations each year, many years after publication, and, more important, have had documented effects on clinical care.¹² However, we found that a substantial proportion of NHLBI-funded randomized clinical trials of cardiovascular interventions were not published in a timely manner and have received few if any citations. The NHLBI, along with other stakeholders in the research enterprise, should seriously examine how best to comprehend and enhance the investment value of smaller trials with surrogate end points and should consider how best to facilitate the rapid publication of all funded randomized trials.