Publication Speed, Reporting Metrics, and Citation Impact of Cardiovascular Trials Supported by the National Heart, Lung, and Blood Institute

Abstract

Background

We previously demonstrated that cardiovascular (CV) trials funded by the National Heart, Lung, and Blood Institute (NHLBI) were more likely to be published in a timely manner and receive high raw citation counts if they focused on clinical endpoints. We did not examine the metrics of trial reports, and our citation measures were limited by failure to account for topic-related citation behaviors.

Methods and Results

Of 244 CV trials completed between 2000 and 2011, we identified 184 whose main results were published by August 20, 2014. One investigator who was blinded to rapidity of publication and citation data read each publication and characterized it according to modified Delphi criteria. There were 46 trials (25%) that had Delphi scores of 8 or 9 (of a possible 9); these trials published faster (median time from trial completion to publication, 12.6 [interquartile range {IQR}, 6.7 to 23.3] vs. 21.8 [IQR, 12.1 to 34.9] months; P<0.01). They also had better normalized citation impact (median citation percentile for topic and date of publication, with 0 best and 100 worst, 1.92 [IQR, 0.64 to 7.83] vs. 8.41 [IQR, 1.80 to 24.75]; P=0.002). By random forest regression, we found that the 3 most important predictors of normalized citation percentile values were total costs, intention-to-treat analyses (as a modified Delphi quality measure), and focus on clinical (not surrogate) endpoints.

Conclusions

NHLBI CV trials were more likely to publish results quickly and yield higher topic-normalized citation impact if they reported results according to well-defined metrics, along with focus on clinical endpoints.

The National Heart, Lung, and Blood Institute (NHLBI) has allocated substantial funds to supporting randomized, clinical trials, but recent reports have suggested that the public may not “get what we pay for.”¹ For an NHLBI trial investment to be considered successful, it is reasonable to expect that main results should be published and cited in a timely manner.² We previously reported that a large proportion of NHLBI-funded trials are not published in a timely manner, a finding that was particularly marked for trials that focused on surrogate (as opposed to clinical) endpoints and that were supported by relatively small budgets.³ Our analyses were limited, however, in that we did not undertake a systematic assessment of the characteristics of trial reports and our citation measures were raw citation counts that did not account for scientific field or year of publication.⁴ We therefore measured the characteristics of published trials and assessed the association between our measures, along with endpoint type and trial costs, with speed of publication and field-normalized citation impact.

Methods

Trial Reports

We previously described our trial selection criteria. Briefly, we started with NHLBI-supported clinical trials that were completed between January 1, 2000 and December 31, 2011.³ For these analyses, we focus on those trials for which a main-results article was published in a peer-review journal by no later than August 20, 2014. Institutional review board approval was obtained for the conduct of each of these trials.

Characteristics of Trials and Main-Results Articles

As described in detail previously, we recorded information on total trial costs and type of primary endpoint (clinical vs. surrogate).³ One author (K.C.A.), who was blinded to publication timing, read all published main-results articles (n=184) and used the Delphi list to characterize trial reporting.⁵ The Delphi list is based on explicit reporting of 9 components: inclusion/exclusion criteria; randomization; allocation concealment; baseline comparability of study groups; blinding of investigator, subjects, and care providers; reporting of point estimates and variability for primary outcomes; and intention-to-treat (ITT) analysis. Those reports that explicitly mentioned 8 or 9 of these components were considered to be of potentially higher caliber. We recorded year-specific journal impact factors based on data provided by Thompson-Reuters Journal Citation Reports.

Endpoints

We previously showed that type of endpoint and total costs were predictive of more-rapid publication.³ In this analysis of published reports, we additionally assessed each of the Delphi list components as predictors of more-rapid publication.

In our previous report, we presented raw citation counts and showed that articles that focused on clinical endpoints or were based on trial that cost at least $5 million generated more citations.³ Critics have faulted raw citation counts for failing to normalize for topic, year of publication, and type of article.⁴ Therefore, we obtained normalized citation data from the Thompson-Reuters InCites database, which includes a “citation percentile” for each article.^4,6 A citation percentile value of 0 implies that an article was cited more often than every other article published in the same year on the same topic, whereas a percentile value of 100 implies that an article was cited less often than every other article (eg, the paper had zero citations, whereas some competitors had at least 1 citation). A citation percentile value of 50 would be the expected value, meaning that an article was cited more often than 50% of articles of similar topic, type, and year of publication. Some authorities argue that a percentile value <10 corresponds to a high-impact paper.^4,7

Statistical Analyses

For descriptive purposes, we divided articles into those with Delphi list scores of 8 or 9 versus all others. Characteristics were compared using chi-square or nonparametric continuous tests, as appropriate; continuous variable characteristics were presented with box plots.⁸ We used the Kaplan–Meier method to plot time to publication as a function of Delphi list score. We used random forest regressions to determine unbiased assessments of variable importance and find potentially important interactions.^9–11 Random forest variable importance measures are related to what degree of prediction can be explained by individual variables, after accounting for all other variables. All analyses were performed using R packages, including RMS, HMisc, ggplot2,¹² and randomForestSRC.¹³

Results

Characteristics of the 184 published trial main-result articles are shown in Table 1. There were 46 trials (25%) that had higher Delphi scores of 8 or 9 (of a possible 9). These 46 trials were more likely to focus on clinical endpoints, cost more, and report concealed allocation, blinding or care providers and patients, blinded outcome assessments, and use of ITT analyses. They were also much more likely to publish rapidly (median times to publication, 12.6 vs. 21.8 months; P<0.01; Table 1 and Figure 1) and to publish in journals with higher impact factors.

Table 1.

Trial and Bibliometric Characteristics According to Quality Score

Characteristic	Higher-Quality Score (N=46)	Lower-Quality Score (N=138)	P Value
Primary outcomes clinical	46% (21)	17% (24)	<0.001
Total costs, $M	5.1 (2.4 to 10.7)	3.0 (1.6 to 5.1)	0.01
Randomization reported	100% (46)	98% (135)	0.31
Concealed allocation	52% (24)	14% (20)	<0.001
Balanced baseline measures	100% (46)	83% (115)	0.003
Eligibility criteria specified	100% (46)	98% (135)	0.31
Blinded outcome assessment	89% (41)	42% (58)	<0.001
Blinded care provider	100% (46)	25% (35)	<0.001
Blinded patients	98% (45)	22% (30)	<0.001
Primary point estimates variance	100% (46)	98% (135)	0.31
Intention-to-treat analyses	98% (45)	62% (85)	<0.001
Time to publication, months	13 (7 to 23)	22 (12 to 35)	0.001
Journal impact factor	17 (8 to 37)	6 (4 to 15)	<0.001
Citation percentile (N=144)	1.92 (0.64 to 7.83)	8.41 (1.80 to 24.75)	0.002

Categorical variables are presented as percent (number). Continuous variables are presented as median (25th to 75th percentile). Categorical comparisons are based on the Pearson test, whereas continuous comparisons are based on the Wilcoxon test.

Figure 1.

Kaplan–Meier plot showing cumulative publication rate according to modified Delphi reporting score. Of a possible score of 9, there were 46 trials that had a score of 8 or 9.

Citation percentile values were available for 144 trial main-result articles (78%). Missing values were nearly entirely owing to recent publication (typically <18 months ago), not allowing enough time for citations to accrue either to the publication or to its “competitor” articles that focus on the same topic and were published in the same year. Articles with higher Delphi scores had better citation percentiles (P<0.001; Table 1 and Figure 2).

Figure 2.

Box plot showing distribution of citation percentile according to modified Delphi score. Lower values are better and reflect higher citation rates normalized for topic and year of publication. Note the skew toward poorly cited articles (reflected by the mean values exceeding the median values).

By random forest regression, the 3 most important predictors of citation percentile values were total costs, explicit reporting of ITT analyses, and focus on clinical (as opposed to surrogate) primary endpoints (Figure 3). In adjusted analyses, the association of citation percentile with total costs followed a distinctly nonlinear pattern, with percentile value substantially improving up to an “elbow” at $5 to $10 million, with essentially no improvement in percentile value as costs increased above $5 to $10 million.

Figure 3.

Results of random forest regression model. Random forest variable importance measures (x-axis) are related to what degree of prediction can be explained by individual variables, after accounting for all other variables. The 3 most important correlates of normalized citation impact (“citation percentile”) were total costs, reporting of intention-to-treat analyses, and use of a clinical (as opposed to surrogate) primary endpoint.

Random forest regression identified 1 potentially meaningful interaction, namely, between focus on clinical endpoints and total costs. Clinical endpoint articles yielded outstanding citation percentile values irrespective of total costs, whereas surrogate endpoint articles only yielded outstanding citation percentile value when total costs exceeded $5 million (Figure 4 and Table2). In Table2, we also list the 3 trials with the best citation percentile values for each of 4 categories defined by clinical endpoints and budget. Of note, 21 of 37 (57%) articles of trials with clinical endpoints and higher budgets were among the top 1% most cited for topic and year of publication; in contrast, only 3 of 118 (2.5%) articles of trials with surrogate endpoints and lower budgets were among the top 1% most cited.

Figure 4.

Box plot showing associations of normalized citation impact (“citation percentile”) with use of clinical endpoints and costs. Random forests regression discovered an important interaction whereby citation percentile values were excellent (ie, low, given that lower values are better) if trials focused on clinical endpoints or if the budget exceeded $5 million; citation percentile values were substantially worse (higher) if neither characteristic was present. Of 184 projects, 118 (64%) focused on surrogate endpoints and had lower budgets, 37 (20%) focused on clinical endpoints and had higher budgets, and 29 (16%) had 1 (but not both) characteristic.

Table 2.

Citation Percentiles According to Endpoints and Budget

Trial Category	Number of Articles	Number Top 1% Articles	Top 3 Trials
Clinical/higher budget	37	21	ACCORD-glycemia14 WHI E-P15 AFFIRM16
Clinical/lower budget	8	2	ESCAPE17 CPR feedback18 Hypertonic saline shock19
Surrogate/higher budget	21	9	DOSE-AHF20 FOCUS-CCTRN21 LateTIME-CCTRN22
Surrogate/lower budget	118	3	Statins skeletal muscle23 Faith, activity, nutrition24 RAAS, and atrial fibrillation25

RAAS indicates renin–angiotensin–aldosterone system.

In secondary analyses, we included journal impact factor as a random forest regression covariate. Journal impact factor emerged as the most important predictor of citation percentile, followed by total costs, ITT analyses, and focus on clinical (as opposed to surrogate) endpoints (Figure 5). Citation percentile improved as impact factor rose from >0 to ≈15; above values of ≈15, there was no association between impact factor and citation percentile (Figure 6).

Figure 5.

Results of secondary random forest regression model that included Journal Impact Factor as a predictor. Random forest variable importance measures (x-axis) are related to what degree of prediction can be explained by individual variables, after accounting for all other variables. The 4 most important correlates of normalized citation impact (“citation percentile”) were total costs, Journal Impact Factor, reporting of intention-to-treat analyses, and use of a clinical (as opposed to surrogate) primary endpoint.

Figure 6.

Association of the citation percentile of main results articles and journal impact factor according to the random forest regression model depicted in Figure 5. The points and LOWESS (locally weighted scatterplot smoothing) smoother adjust for all other confounders. Lower citation percentiles imply greater citation impact; note that citation impact improves up to a journal impact factor of ≈15 to 20; for higher journal impact factors, citation impact remains consistently excellent (with all values at <10th percentile).

Discussion

We have previously established that randomized trials that cost more than $5 million and used a clinical event or composite of clinical events as their primary endpoint were published more rapidly and yielded higher raw citation counts than less-expensive trials and trials of surrogate endpoints.³ We now extend these findings by showing that primary results articles, which addressed at least 8 of the 9 the Delphi metrics, were published more rapidly (Figure 1) and had higher field- and year-normalized citation rates, even after accounting for cost and endpoint type (Figure 2). Of note, we refined our analysis by normalizing raw citation rates for “expected” rates for contemporaneous articles on the same topic and showed a similar relationship of cost and endpoint type with citation impact, with reporting by ITT standing out among the Delphi metrics (Figure 3). Consistent with our previous analyses of speed to publication, we found a distinctly nonlinear association of cost with normalized citation rates, with a steep improvement up to $5 million and flat thereafter. We also observed a strong interaction between cost and endpoint type in our adjusted citation analysis (Figure 4), showing far better citation impact for studies that cost >$5 million and assessed clinical events than for those who had only one of these characteristics.

Among the Delphi metrics, the importance of analysis by ITT clearly stands out as a strong predictor of citation percentile (Figure 3). In constructing our data set, we included only trials that were characterized in ClinicalTrials.gov or grant applications as being randomized, so it is not surprising that all but 3 of the 184 articles in our data set explicitly mentioned randomization (Table 1). Also, all but 3 articles included eligibility criteria and point estimates and variance of the primary outcome. Thus, our analysis tells us little about the association of these elements or the balance of baseline characteristics (which was addressed in 161 of 184 articles) with citation impact. However, the 4 Delphi elements related to blinding were clearly not related to citation percentile (Figure 3), despite adequate representation of both positive and negative examples.

There are some important limitations to consider. We only focused on studies that were completed and published; we could not assess citations or detailed Delphi metrics for unfinished or unpublished trials. The Delphi measures cannot be equated with quality, but nonetheless were correlated with time to publication and citation percentiles. Our analyses are able to establish only associations, not cause and effect. For example, the funding that the NHLBI is willing to commit to a trial is likely to be (at least in part) a surrogate for factors such as the perceived importance of, and interest in, the study hypothesis, sample size, attraction of strong and influential collaborators, and inclusion of a skilled, well-staffed coordinating center, all of which may facilitate publication and enhance impact. In the case of the Delphi metrics, as with reporting by ITT, one cannot know whether trials that report these results are more likely to be accepted by high-impact journals or whether high-impact journals are more meticulous about how their trials are reported and insist that they be analyzed by ITT. When impact factor of the journal was added to the random forest analysis of citation percentile (Figure 5), analysis by ITT (as well as cost and endpoint type) remained independently predictive of citation percentile, although (as one might have expected) impact factor was the strongest predictor.

Although we cannot establish that the associations we observed here are causal, we have identified a profile of randomized trials that are likely to be published quickly and yield high citation impact. Large studies addressing clinically meaningful endpoints and with substantial resources devoted to creating an experienced collaborative team and using sound clinical trials methodology are likely to achieve results that attract the attention of major professional meetings and high-impact journals and yield timely publications with high field-normalized citation rates.

Sources of Funding

This work was supported by the National Heart, Lung, and Blood Institute (NHLBI) through the efforts of the 3 authors, who are full-time NHLBI employees and conducted the work as part of their official federal duties.

Disclosures

All authors are full-time employees of the NHLBI and conducted the work reported in this paper as part of their official federal duties. Otherwise there are no disclosures to report.