High-cited favorable studies for COVID-19 treatments ineffective in large trials

Abstract

Objectives

To evaluate for coronavirus disease 2019 treatments without benefits in subsequent large randomized controlled trials (RCTs) how many of their most-cited clinical studies had declared favorable results.

Study Design and Setting

Scopus searches (December 23, 2021) identified articles on lopinavir-ritonavir, hydroxychloroquine, azithromycin, remdesivir, convalescent plasma, colchicine, or interferon (index interventions) that represented clinical trials and had >150 citations. Their conclusions were correlated with study design features. The 10 most recent citations for the most-cited article on each index intervention were examined on whether they were critical to the highly cited study. Altmetric scores were also obtained.

Results

Forty eligible articles of clinical studies had received >150 citations. Twenty of forty (50%) had favorable conclusions and four were equivocal. Highly cited articles with favorable conclusions were rarely RCTs (3/20), although those without favorable conclusions were mostly RCTs (15/20, P = 0.0003). Only one RCT with favorable conclusions had >160 patients. Citation counts correlated strongly with Altmetric scores, especially news items. Only nine (15%) of 60 recent citations to the most highly cited studies with favorable or equivocal conclusions were critical.

Conclusion

Many clinical studies with favorable conclusions for largely ineffective coronavirus disease 2019 treatments are uncritically heavily cited and disseminated. Early observational studies and small randomized trials may cause spurious claims of effectiveness that get perpetuated.

What is new?

Key findings

• •Many highly cited studies with favorable conclusions for COVID-19 treatments were refuted by large trials.
• •The refuted highly cited clinical studies also attracted large media and social media attention.
• •The refuted highly cited trials continued to be uncritically heavily cited after their refutation.

What this adds to what is known?

• •COVID-19 clinical research can be highly prone to exaggerated claims.

What is the implication/what should change now?

• •Early observational studies and small randomized trials on COVID-19 treatments should be seen with great caution.

1. Introduction

The search for coronavirus disease 2019 (COVID-19) treatments has ushered in thousands of clinical studies [[1], [2], [3]], many promises, several emergency authorizations, and some excellent successes. In particular, large adaptive randomized controlled trials (RCTs) using rapid recruitment of participants from real-world clinical practice were instrumental in documenting benefits with large-scale evidence [4,5]. By the end of 2021, dexamethasone, tocilizumab, and monoclonal antibody combinations had shown convincing evidence that they reduce mortality in various patient groups and clinical settings [[6], [7], [8]]. However, the largest RCTs to date, RECOVERY [[9], [10], [11], [12], [13]] and SOLIDARITY [14], have also showed no benefit for several other treatments–lopinavir/ritonavir [9], hydroxychloroquine [10,14], azithromycin [11], remdesivir [14], convalescent plasma [12], colchicine [13], and interferon [14]. All these interventions with disappointing results in the large trials had been presented as being highly promising and effective in earlier, mostly smaller studies. Each of these treatments has been debated heavily in both scientific and lay circles, often vehemently so. The unfavorable results from large trials may upset guidelines because they may even lead to reversal of emergency authorization (eg, https://www.fda.gov/news-events/press-announcements/coronavirus-covid-19-update-fda-revokes-emergency-use-authorization-chloroquine-and).

The ability of the scientific literature to adopt the newer, more sober evidence is unknown. Important questions arise. Has the emergence of the unfavorable results from large RCTs managed to change the pervasive presence of these treatments in the scientific literature? Do authors continue to cite the early clinical studies with the favorable results? If so, are these citations critical of the original promising studies and do they cite also the well-powered RCTs that had null results?

The present analysis aimed to evaluate how many of the most-cited clinical studies in the literature have been favorable for interventions that failed to show survival benefits for COVID-19 patients in the two largest, well-powered RCTs, RECOVERY [[9], [10], [11], [12], [13]] and SOLIDARITY [14]. The analysis also aimed to evaluate whether study design features are associated with favorable conclusions, how citations have tracked against media and social media interest (as captured by the Altmetric score [www.altmetric.com]), and whether citing articles to the most-cited studies were critical and whether they cited also the refuting large RCTs.

2. Methods

2.1. Eligible studies and search strategy

A Scopus search (last update on December 23, 2021) sought articles published in 2020-2021 with lopinavir-ritonavir, hydroxychloroquine, azithromycin, remdesivir, convalescent plasma, colchicine, or interferon (title-abstract-keywords). Articles were eligible if they were clinical studies on any of these index interventions and had received more than 150 citations in Scopus. The citation threshold was prespecified but it is arbitrary. One hundred and fifty or more citations correspond approximately to the top 0.1% most-cited items among the approximately seven million items published in 2020 and 2021 and indexed in Scopus until December 2021. Therefore, these articles are exceptionally influential in the literature (or, at a minimum, they are receiving exceptional attention).

The index treatments were selected because they have been evaluated in the two largest RCTs of COVID-19 therapeutics and have shown no significant benefit for the primary outcome of mortality and no other signals of any substantive benefit for other clinically important outcomes. The relative risks for mortality in RECOVERY was 1.03 (95% confidence interval [CI], 0.91–1.17) based on 5,040 randomized participants for lopinavir-ritonavir [9], 1.09 (95% CI, 0.97–1.23) based on 4,716 participants for hydroxychloroquine [10], 0.97 (95% CI, 0.87–1.07) based on 7,763 participants for azithromycin [11], 1.00 (95% CI, 0.93–1.07) based on 11,558 participants for convalescent plasma [12], and 1.01 (95% CI, 0.93–1.10) based on 11,340 participants for colchicine [13]. The relative risks for the mortality outcome in SOLIDARITY [14], a trial sponsored by the World Health Organization (WHO) to assess repurposed drugs for COVID-19 [14], were 0.95 (95% CI, 0.81–1.11) based on 5,451 randomized participants for remdesivir, 1.19 (95% CI, 0.89–1.59) based on 1,853 participants for hydroxychloroquine, 1.00 (95% CI, 0.79–1.25) based on 2,771 participants for lopinavir, and 1.16 (95% CI, 0.96–1.39) based on 4,100 participants for interferon.

Among the highly cited articles retrieved on these index treatments, any study design was eligible (RCT, nonrandomized controlled study, and uncontrolled study [including also case reports and case series]). Clinical studies where the index intervention(s) was involved along with other treatment(s) were also eligible, unless the clinical study focused on the other treatment(s) (eg, tested a new treatment vs. standard of care and used an index intervention as common backbone for both arms) or found that a new treatment is superior to an index intervention (thus, one cannot conclude whether the index intervention by itself is effective or not). Retracted articles and retraction notices were also excluded.

2.2. Data extraction

Data extraction on the eligible highly cited studies recorded the evaluated interventions, the design (randomized or not), sample size, and number of deaths. The conclusions of the authors of each eligible highly cited study were categorized as favorable (claiming that an index treatment is beneficial or that it is safe without any mention of harm), unfavorable (claiming no benefit and/or a safety problem for an index treatment), or equivocal when there was a mixed message or potential benefit seen in some particular analysis or end point.

2.3. Media and social media impact

Media and social media impact was assessed by the Altmetric score. The score tracks the presence of an article in media and social media, for example, news, Twitter, blogs, Facebook, and more (https://www.altmetric.com/). Information was also obtained through Altmetric on the rank of the article across all scientific articles tracked by Altmetric and on the number of news items and tweets mentioning each highly cited article. Correlations between citation counts and Altmetric score, news items, and tweets were estimated with Pearson coefficients.

2.4. Citation content analysis

For each index treatment, the 10 most recent citing articles to its most highly cited study with favorable or equivocal results were probed to assess whether the citations were critical to the highly cited study and to identify whether the citing articles also cited the large trials with unfavorable results (RECOVERY and/or SOLIDARITY) [[9], [10], [11], [12], [13], [14]], either the final peer-reviewed publications, or the preprints, or at least some press release or other mention. Ten articles were selected as a convenience sample. Unfavorable results were announced by RECOVERY for hydroxychloroquine on June 5, 2020, for lopinavir-ritonavir on June 29, 2020, for azithromycin on December 14, 2020, for convalescent plasma on January 15, 2020, and for colchicine on March 5, 2021 and by SOLIDARITY on hydroxychloroquine, remdesivir, lopinavir, and interferon on October 15, 2020. Therefore, probably there was sufficient time for the authors of the examined citing articles to be aware of these results when they wrote or revised their articles. The most recent citing articles as of December 27, 2021 were retrieved from Scopus.

3. Results

3.1. Highly cited clinical studies on COVID-19 index treatments

The Scopus search yielded 63,803 results, of which 465 published items were highly cited with >150 citations. Forty five of 465 pertained to any of the index interventions being used in clinical studies. A retracted article and its retraction notice and three articles where the favored treatment (baricitinib and arbidol) was not an index intervention were excluded. The 40 eligible articles appear in Table 1 .

Table 1.

Clinical studies with more than 150 Scopus citations that assess COVID-19 treatments that have shown no benefit in large trials (RECOVERY and SOLIDARITY)

Author (reference)	Interventions	n	RCT	>200 deaths	Favorable for index treatment	Citations
Cao [15]	LPV/r vs. SOC	199	Yes	No	Equivocal (benefit in MITT analysis)	2,859
Gautret [16]	HCQ ± AZ	38	No	No	Yes	2,839
Beigel [17]	Remdesivir vs. placebo	1,062	Yes	No	Yes	2,562
Wang [18]	Remdesivir vs. placebo	237	Yes	No	Equivocal (nonsignificant trend)	1,612
Grein [19]	Remdesivir	53	No	No	Yes	1,444
Shen [20]	Convalescent plasma	5	No	No	Yes	1,331
Duan [21]	Convalescent plasma	10	No	No	Yes	1,034
Geleris [22]	HCQ	1,446	No	No	No	931
Hung [23]	LPV/r + ribavirin + interferon vs. LPV/r	127	Yes	No	Yes	772
Boulware [24]	HCQ prophylaxis vs. placebo	821	Yes	No	No	688
Pan [14],a	Four active interventions (HCQ, remdesivir, lopinavir, and interferon) vs. control	11,330	Yes	Yes	No	646
Rosenberg [25]	HCQ, AZ, both, neither	1,438	No	Yes	No	625
Li [26]	Convalescent plasma vs. SOC	103	Yes	No	Equivocal (benefit in severe disease and for PCR conversion)	615
Goldman [27]	Remdesivir five vs. 10 days	397	Yes	No	No	562
Tang [28]	HCQ vs. SOC	150	Yes	No	No	552
Cavalcanti [29]	HCQ vs. HCQ + AZ vs. SOC	667	Yes	No	No	510
Molina [30]	HCQ + AZ	11	No	No	No	448
Horby [10],a	HCQ vs. SOC	4,716	Yes	Yes	No	430
Spinner [31]	Remdesivir vs. SOC	596	Yes	No	Equivocal (uncertain clinical value)	428
Gautret [32]	HCQ + AZ	80	No	No	Yes	396
Chen [33]	HCQ vs. control	30	Yes	No	No	322
Simonovich [34]	Convalescent plasma vs. placebo	228	Yes	No	No	311
Libster [35]	Convalescent plasma vs. placebo	160	Yes	No	Yes	276
Arshad [36]	HCQ, HCQ + AZ, AZ, neither	2,541	No	Yes	Yes	267
Agarwal [37]	Convalescent plasma vs. SOC	464	Yes	No	No	262
Million [38]	HCQ + AZ	1,061	No	No	Yes	246
Horby [9],a	LPV/r vs. SOC	5,040	Yes	Yes	No	236
Mahevas [39]	HCQ, control	181	No	No	No	235
Skipper [40]	HCQ vs. placebo	491	Yes	No	No	232
Zhang [41]	Convalescent plasma	4	No	No	Yes	231
Ye [42]	Convalescent plasma	6	No	No	Yes	222
Magagnoli [43]	HCQ+/-AZ, control	807	No	No	No	203
Zhou [44]	Interferon or interferon + Arbidol	77	No	No	Yes	198
Liu [45]	Convalescent plasma, control	39	No	No	Yes	195
Ahn [46]	Convalescent plasma	2	No	No	Yes	192
Joyner [47]	Convalescent plasma	5,000	No	No	Yes	191
Joyner [48]	Convalescent plasma	20,000	No	Yes	Yes	186
Zeng [49]	Convalescent plasma	6	No	No	Yes	180
Deftereos [50]	Colchicine vs. SOC	105	Yes	No	Yes	177
Saleh [51]	(HCQ or chloroquine) ±AZ	201	No	No	Yes	162

Abbreviations: HCQ, hydroxychloroquine; AZ, azithromycin; SOC, standard of care; LPV/r, lopinavir-ritonavir; MITT, modified intention-to-treat.

^aPresenting results from one of the two large randomized trials (RECOVERY and SOLIDARITY).

3.2. Favorable conclusions

Of the 40 eligible studies, 20 (50%) had favorable conclusions for the index treatments, four were equivocal, and 16 were unfavorable. The unfavorable group included the publication of SOLIDARITY itself [14] and the publications of the hydroxychloroquine [10] and lopinavir-ritonavir [9] assessments from RECOVERY. Of the seven articles that exceeded 1,000 citations, five had favorable conclusions and two were equivocal (one described a significant benefit in a modified intention-to-treat analysis and the other mentioned a nonsignificant trend for clinical improvement).

3.3. Correlates of favorable conclusions

The highly cited articles with favorable conclusions were far less likely to be RCTs than the other highly cited articles (3/20 vs. 15/20, exact P = 0.0003). The few RCTs with favorable conclusions tended to be smaller than the others (median sample size 160 vs. 464). Of the six studies with at least 200 deaths, all three randomized trials did not reach favorable conclusions, whereas two of three nonrandomized studies did.

3.4. Altmetric scores

All 40 highly cited articles had very high Altmetric scores placing them at the top 5% of published articles and 24/40 had extraordinarily high Altmetric scores placing them among the top-2000 highest Altmetric scored articles of all science of all times (Supplementary Table 1). Five articles were among the top-200. There was a strong correlation between the Altmetric score and number of citations (r = 0.74) (Fig. 1 ). The correlation of the number of citations was stronger with the number of news items (r = 0.81) and more modest with the number of tweets (r = 0.47).

Fig. 1.

Correlation between citation counts and Altmetric scores for the eligible highly cited articles.

Favorable articles did not have higher media and social media mentions than other articles. Altmetric values in the top-2000 of all science occurred in 9 of 20 favorable, four of four equivocal, and 11/16 unfavorable articles (exact P = 0.10 for the comparison of articles with favorable conclusions vs. others).

3.5. Citation content

Only nine (15%) of 60 recent citations to the most highly cited studies with favorable or equivocal conclusions were critical to the highly cited study (Table 2 ). Citing articles uncommonly (8/60, 13%) cited the respective RECOVERY or SOLIDARITY results.

Table 2.

Qualitative analysis of recent citations to the most highly cited article for each index treatment that reached favorable or equivocal conclusions

Intervention	Highly cited article	Critical citations (among 10 recent sampled citing articles)	RECOVERY/SOLIDARITY trials citeda
LPV/r	Cao, NEJM [15]	1/10 (“So far none of these drugs have been found to be an appropriate drug for COVID-19”)	0/10
HCQ±AZ	Gautret, Intern J Antimicrob Agents [16]	3/10 (“So far none of these drugs have been found to be an appropriate drug for COVID-19” and “The excitement surrounding hydroxychloroquine was fueled early on by excessive media attention after a nonrandomized study (with questionable, hotly debated reliability) was released” and “The WHO announced the failure of the the solidarity trial, which means that hydroxychloroquine did not achieve the desired effect in the treatment of COVID-19”)	2/10 (SOLIDARITY)
Remdesivir	Beigel, NEJM [17]	2/10 (“So far none of these drugs have been found to be an appropriate drug for COVID-19… WHO have made a conditional recommendation against the use of Remdesivir for hospitalized COVID patients, regardless of the disease's severity, because of a lack of evidence showing that it improves survival rate” and “had little or no effect on overall mortality, initiation of ventilation, or duration of hospital stay”)	2/10 (SOLIDARITY)
Convalescent plasma	Shen, JAMA [20]	2/10 (“Small studies using convalescent serum for SARS-CoV-2 patients suggested that treatment was well tolerated, reduced viraemia and clinical symptoms [Shen et al., 2020, Duan et al., 2020], whereas the larger RECOVERY Collaborative Group [2021c], testing convalescent plasma as a treatment in life-threatening COVID-19 did not result in significant improvement and was discontinued early” and “the clinical effect of this CP intervention has not yet been determined, because patients could have recovered due to other treatments administrated in parallel”)	1/10 (RECOVERY)
Interferon	Hung, Lancet [23]	0/10	2/10 (SOLIDARITY)
Colchicine	Deftereos, JAMA Network Open [50]	1/10 (“in this large, well powered trial, we found no evidence of a benefit from colchicine”)b	1/10 (RECOVERY)

Abbreviations: LPV/r, lopinavir-ritonavir; HCQ, hydroxychloroquine; AZ, azithromycin.

^aCitation to either the respective publication of RECOVERY or SOLIDARITY on that intervention or to some preliminary press release or other statement about the trial results.

^bThis statement comes from the article publication of the RECOVERY colchicine trial itself.

4. Discussion

Many highly cited clinical studies favor COVID-19 treatments that have shown no benefits in large, well-powered randomized trials. Most favorable studies are not randomized or are even uncontrolled, but exercise strong, persistent influence on the scientific literature. Citation counts track well with the strong presence of these studies in media and social media. The most highly cited studies on these interventions have either entirely or partially favorable conclusions. Citations that they continue to receive are rarely critical of them.

Citations are a measure of the influence of a research article. Various manifestations of citation bias have been demonstrated in other fields before the COVID-19 era [[52], [53], [54], [55], [56], [57]]. In principle, studies with “positive” results are more heavily cited than studies with “negative” results on the same topic. The citation bias creates a distorted picture for the perception of the scientific literature at large. Repeated mention of the most favorable results gives the allusion that they are more likely to represent the truth. In addition, the COVID-19 literature is unique in terms of the massive volume of articles produced [58,59] (and thus also citations generated) within a very limited timeframe. Very few studies in the history of medicine have ever received the number of citations received by the most highly cited COVID-19–related articles.

The charged situation surrounding the COVID-19 pandemic may have further intensified the citation bias. Several treatments have received tremendous attention not only in the scientific literature but also in the wider society. Many highly cited articles analyzed here have also reached astronomical Altmetric scores from massive discussion in media and social media. Altmetric scores correlated well with citation counts. The correlation was more prominent when news items were considered, although tweets had a more modest correlation. Altmetric score analyses have shown [60] that media and social media attention may remain high even for fully retracted articles.

Some caveats need to be acknowledged. The large trials may not necessarily be a gold standard. No single clinical study can claim to possess the truth. Even large trials can be biased upwards in their beneficial estimates, for example, due to lack of blinding. However, this is less of a concern for an outcome such as mortality. The CIs of the large trials cannot exclude very small benefits on survival–or small harms. These trials have also shown no benefit on other outcomes. However, small benefits (or harms) for these outcomes cannot be excluded with perfect certainty. Moreover, beneficial effects may still exist in circumscribed, special circumstances, with different dosing regimens and in specific patient subgroups outside the eligibility criteria of the large RCTs or under-represented in these large RCTs. However, similar concerns and speculative counter arguments may be raised almost in any clinical topic, especially by those who still believe in an intervention despite its poor performance in very large trials [61].

Moreover, admittedly not all guidelines have removed these treatments from their list of recommended interventions. Remdesivir is probably the most notable example. It is not recommended by the European Respiratory Society [62] and the WHO has issued conditional recommendations against its use [63]. Conversely, the US National Institutes of Health (NIH) list remdesivir very prominently among the few treatments that they recommend [64]. Sometimes there is an overlap or connection between the institutions which perform the clinical studies and those which issue guidelines. For example, the most-cited favorable trial on remdesivir was spearheaded by NIH [17]. It is unknown whether there is a strong, generalizable relationship between sponsoring institution and study results/interpretation, but this observation warrants further evaluation across studies performed on COVID-19 therapeutics. Moreover, a consequence of trusting a treatment as being effective is that it becomes attractive, if not necessary, to use it as a background treatment or comparator when new interventions are evaluated. For example, NIH-spearheaded trials have already compared interferon + remdesivir versus remdesivir alone and claimed to find no benefit for interferon [65] and baricitinib + remdesivir versus remdesivir alone and claimed to find a benefit with adding baricitinib [66]. It is unknown whether interferon or baricitinib may have an interaction effect with remdesivir.

For treatments that do not have favorable results in large randomized trials, some evidence may provide support for consideration of a circumscribed use. For example, remdesivir recently showed promising results for an outpatient use in a randomized trial of 562 participants [67], convalescent plasma had favorable results in early outpatient intervention with high-titer dosing in a recent trial on 1,225 participants [68], and hydroxychloroquine has shown promising results for a preventive use with lower doses in a recent meta-analysis [69]. The best solution in these situations would be to consider whether there is room for conducting new large simple trials that focus on the debated settings, populations, and doses where there is continued controversy powering these trials to detect or reject the possibility of mortality benefits. The experience with SOLIDARITY and RECOVERY proves that such trials can be done quickly. The old argument that large trials take immense resources and a very long period of time to complete has been contradicted with the efficiency of the current adaptive large trial platforms. Other large trial platforms are also launched, such as the TOGETHER trial [70], and they can contribute further large-scale evidence. The procured evidence would then need to be balanced very carefully in terms of benefits and risks in different settings and populations.

Acknowledging these caveats and some residual uncertainty, it is more likely that the treatments examined in the current analysis overall do not save lives. The most recent meta-analyses are also consistent with this interpretation [[71], [72], [73], [74], [75]]. For hydroxychloroquine, a recent collaborative meta-analysis even shows nominally, statistically, significantly increased mortality [76].

Prior experience from a content analysis of persistent high citations of nonvalidated articles in other fields may offer us insights into why this situation arose also for COVID-19 treatments. One empirical evaluation [61] assessed in-depth the citation patterns of extremely highly cited articles on the benefits of beta-carotene for cancer prevention, estrogens for Alzheimer's dementia, and alpha-tocopherol for cardiovascular disease. Despite the emergence of large RCTs with unfavorable results for these interventions, the observational studies that made the original promises continued to be heavily cited long after the “negative” RCTs were published. Their citations were either ignoring the refuting RCTs (for beta-carotene) or raising numerous counterarguments against them (for estrogen and alpha-tocopherol). Similarly, in psychology, where several major claims had been found to be irreproducible in preregistered reproducibility assessments done by many teams, frequent citation of the original claims has continued unperturbed after their nonreproduction [77]. The citing articles uncommonly take a critical stance against the original claims [77]. In other areas where evidence is heavily centered on biological considerations, citation networks citing some preferred articles may create a perpetuated distortion of what is the established knowledge. This has been seen in empirical evaluations in genetics of amygdala activation [78] and in pathology of inclusion body myositis [79]. Another empirical evaluation even found evidence of so-called “affirmative citation bias” [52]. The citations to the critical articles that had thoroughly debunked prior beliefs were mostly affirmative, in favor of the original beliefs that had been debunked. The authors concluded that even criticism itself may paradoxically reinforce the establishment of debunked prior beliefs [52]. Curiously, the stronger the refutation, the stronger people stick to their beliefs, as described also by Kahnemann [80].

These observations suggest that often science is organized in cherished schools of thought that are recalcitrant to the provision and acceptance of contrarian evidence. This may have happened also in the case of the early promising COVID-19 treatments. Moreover, in COVID-19, given the vast attention devoted, the citation rates of these nonvalidated treatment benefits are even more extraordinary. Furthermore, the overall impact of their dissemination reverberates across wider societal circles, not just across scientific groups. The advent of very large RCTs did not suffice to perturb much of this intense dissemination.

In conclusion, one should avoid putting much trust to highly promising results from early observational studies and small randomized trials of new or repurposed treatments [81]. For serious diseases, like COVID-19, evidence on mortality end points should be sought. Pilot studies should not be abandoned or dejected and they do have some value in offering early insights. However, they should be seen with great caution and with tempered enthusiasm. Large trials with flexible designs that allow obtaining large-scale rigorous evidence in a timely manner have been a major success during the pandemic [[3], [4], [5]] and their use should be promoted further.