A metaresearch study revealed susceptibility of Covid-19 treatment research to white hat bias: first, do no harm

Abstract

Objective

To investigate the presence of white hat bias in Covid-19 treatment research by evaluating the effects of citation and reporting bias.

Study design and setting

Citation bias was investigated by assessing the degree of agreement between evidence provided by a remdesivir randomized controlled trial and its citing articles. The dissimilarity of outcomes derived from nonrandomized and randomized studies was tested by a meta-analysis of hydroxychloroquine effects on mortality. The differential influence of studies with beneficial over those with neutral results was evaluated by a bibliometric analysis.

Results

The articles citing the ACTT-1 remdesivir trial preferentially presented its positive outcomes in 55.83% and its negative outcomes in 6.43% of cases. The hydroxychloroquine indicated no significant effect by randomized studies, but a significant survival benefit by nonrandomized ones. Citation mapping revealed that the study reporting survival benefit from the hydroxychloroquine-azithromycin combination was the most influential, despite subsequent studies reporting potential harmful effects.

Conclusion

The present study raises concerns about citation bias and a predilection of reporting beneficial over harmful effects in the Covid-19 treatment research, potentially in the context of white hat bias. Preregistration, data sharing and avoidance of selective reporting are crucial to ensure the credibility of future research.

What is new?

Key findings

• •Citation bias is evident by the preferential reporting of positive over neutral outcomes of the same study.
• •The example of the ACTT-1 remdesivir trial indicated that the majority of citing articles opted for presenting only its positive outcomes.
• •Nonrandomized evidence about hydroxychloroquine use supported benefit from its use, which was not confirmed by randomized studies.
• •Conventional and alternative metrics demonstrated that research claiming positive effects of the hydroxychloroquine-azithromycin combination had higher influence than studies providing evidence of harm.

What this adds to what is known?

• •The quality of Covid-19 treatment research may be negatively affected by white hat bias, which comes from personal beliefs and the urgent need of an effective intervention.
• •Nonrandomized studies are prone to white hat bias due to the possibility of selective analysis and reporting.

What is the implication and what should change now?

• •Pre-registration and public data sharing should apply for both randomized and observational studies. The peer-review process should ensure the avoidance of citation bias.
• •Meta-analyses of observational studies should consider routinely implementing the credibility ceiling test to explore skepticism about the presence of potential biases.

1. Introduction

The rapid spread of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has provoked a pandemic of coronavirus disease 19 (Covid-19)-related publications. Within 1 year after its emergence, more than 85,000 research papers about Covid-19 have been published in the areas of health, life, physical and social sciences [1]. Research interest has focused on the evaluation of novel antiviral treatments, with more than 4,300 intervention studies being registered in the Clinicaltrials.gov database. The need of rapid knowledge dissemination has led to a surge of fast-track pre-prints, resulting in a complex network of academic and non-peer-reviewed publications [2]. The situation is further complicated by the existence of multiple versions of a single study in open archives and as journal articles, often presenting contradictory outcomes [3]. As a result, the process of decision-making and identification of credible evidence remains a challenge both for the researcher and the clinician.

White hat bias is defined as the distortion of evidence towards the promotion of what is widely perceived as fair ends. This type of bias has been first described in nutritional epidemiology and emanates from the motivation of scientists to practice beneficence and improve human health [4]. White hat bias differs from bias due to financial conflicts of interest since it has been assumed that scientists without industry funding tend to ignore negative or insignificant results about interventions regarded as beneficial to health. In this context, publication bias risk is augmented, potentially leading to the preferential reporting of significant outcomes [5]. Concerns about white hat bias may be also raised when citation bias is suspected with citing papers inaccurately describing the available original evidence. This effect may be extended in institutional reports and press releases, with distorted presentation and “spinning” of study outcomes leading to false claims about the effectiveness of interventions [6].

As the clinical management of Covid-19 patients remains largely supportive, there is increasing demand for the development of a novel treatment in order to alter the natural history of the disease. The aim of the present metaresearch study is to evaluate the potential presence of white hat bias in the Covid-19 treatment research; to achieve this, different aspects of reporting, citation and publication bias are assessed by examining the examples of remdesivir, hydroxychloroquine and azithromycin administration. In addition, the potential harmful effects of interventions claimed as effective are evaluated.

2. Materials and methods

2.1. Citation bias

The potential presence of citation bias was assessed by judging the degree of agreement between the results of a highly influential paper and the statements of its citing articles. Specifically, the article reporting the outcomes of Adaptive Covid-19 Treatment Trial (ACTT-1) [7] was chosen. This study was a randomized controlled trial aiming to evaluate the effectiveness of remdesivir administration in Covid-19 patients. Both positive and negative outcomes were reported; remdesivir was associated with significantly lower duration of hospital stay (11 vs. 15 days, P-value <0.001) and higher rates of clinical improvement at day 15 (odds ratio [OR]: 1.5, 95% confidence intervals [CI]: 1.2–1.9). On the other hand, no significant benefit was observed in the outcomes of survival (hazard ratio: 0.73, 95% CI: 0.52–1.03) and cumulative recovery among critically ill patients. The ACTT-1 was chosen as the study of interest since it is a highly influential randomized controlled trial that exerted significant effect on clinical practice by guiding the approval of remdesivir. In addition, mixed results were presented, thus rendering it suitable for investigating the potential presence of citation bias.

The full reference list was accessed via NEJM.org up to 25 December 2020. The citing papers were classified as accurate (category A) when both significant and insignificant results were reported. Category B was assigned to articles presenting only positive outcomes and category C to those stating only the negative ones. Category D referred to articles with misleading statements reporting outcomes with distorted statistical significance. Papers were unscorable (category E) in case they did not report the significance of outcomes or provided ambiguous statements. The citing studies were classified depending on their design as reviews, meta-analyses, nonrandomized, randomized and basic research studies (including in vitro, animal and simulation studies). Letters to the editor, commentaries, non-English papers and category E articles were excluded. The categories (A-D) were compared among different study designs (review, meta-analysis, nonrandomized, randomized, basic research) using multinomial logistic regression analysis. The category A served as the reference one. Analysis was conducted in R-3.6.3 (“nnet” package [8]).

2.2. Non-randomized vs. randomized evidence

To compare the outcomes derived from observational and randomized studies, a meta-analysis was conducted regarding the effects of hydroxychloroquine treatment on survival rates of Covid-19 patients. An update of a recent meta-analysis [9] was performed by systematically searching Medline using the following search algorithm: “("Coronavirus"[Mesh] OR covid-19 OR "SARS-CoV-2"[Mesh]) AND "Hydroxychloroquine"[Mesh])”. All randomized and observational (both prospective and retrospective) studies that evaluated mortality rates among Covid-19 patients receiving hydroxychloroquine were selected. The literature search process was crosschecked by two independent researchers and any possible conflict was resolved through discussion. Meta-analysis was conducted by fitting a random-effects model (restricted maximum likelihood - REML), which provided estimated of OR and 95%CI. Subgroup analysis was performed based on study design. In the subgroup of nonrandomized studies, the credibility ceiling test was used to test the robustness of outcomes [10]. The ceilings of 5%, 10%, and 15% were implemented. As a sensitivity analysis, the summary odds ratio was estimated using 2 one-stage generalized linear mixed models (modified Simmonds-Higgins and hypergeometric‐normal model) [11]. Meta-analysis was performed in R-3.6.3 (“metafor” package [12] ).

2.3. Harmful effects

The combination of hydroxychloroquine with azithromycin has been proposed early in the course of the pandemic an effective Covid-19 treatment, leading to its widespread prescription. However, this effect was not confirmed by further studies, indicating potentially increased risk of mortality associated with this intervention [13]. To compare the differential influence of studies evaluating the efficacy of hydroxychloroquine-azithromycin combination, the PlumX Metrics tool was used providing information about citations, mentions in news/blogs and social media coverage [14]. Specifically, the evaluated articles included the study of Gautret et al. [15] presenting positive outcomes about the hydroxychloroquine-azithromycin combination and 3 subsequent influential studies [16], [17], [18] with negative outcomes. The main outcome of Gautret et al. [15] was that the combined treatment was linked to significantly higher proportion of patients with viral load clearance within 6 days (70% vs. 12.5%, P-value: 0.001). On the contrary, Cavalcanti et al. [16] reported no significant difference in the primary outcome of clinical improvement at day 15 (OR: 0.99, 95% CI: 0.57–1.73) and the secondary one of death (OR: 0.67, 95% CI: 0.18–2.21). Similarly, Geleris et al. [18] suggested no difference in death or intubation risk (hazard ratio: 1.04, 95% CI: 0.82–1.32), while Rosenberg et al. [17] reported no significant effect on in-hospital mortality risk (hazard ratio: 1.35, 95% CI: 0.76–2.40). Moreover, a citation map of articles in the field was constructed using the VOSviewer software (version 1.6.16) [19]. To achieve this, the Scopus database was searched using the algorithm “(hydroxychloroquine OR chloroquine) AND (covid-19 OR coronavirus OR sars-cov-2)”, limited to original articles.

3. Results

3.1. Citation bias

The process of citing articles selection is schematically depicted in Fig. 1 . Overall, 798 records were identified; of them 101 were excluded due to design or language reasons. Therefore, 697 papers were retrieved as full-texts. Subsequently, 260 of them were classified to the category E and were excluded from the analysis.; hence, 437 articles (313 reviews, 14 meta-analyses, 64 non-randomized, 13 randomized and 33 basic research studies) were evaluated. The majority of articles were categorized in category B (55.83%), followed by category A (35.93%), D (6.41%) and C (1.83%) (Fig. 2 ). Compared to category A, the frequency of category B was significantly lower in meta-analyses (OR: 0.30, 95% CI: 0.09-0.99) and higher in basic research articles (OR: 3.35, 95% CI: 1.25-9.01). Moreover, compared to category A, the proportion of category C was significantly lower in reviews (OR: 0.03, 95% CI: 0.01-0.08) and higher in non-randomized studies (OR: 7.37, 95% CI: 1.54-35.31), while category D was lower in review articles (OR: 0.18, 95% CI: 0.11-0.29). No meta-analysis was categorized in category C and no randomized study in either category C or D. The outcomes of the multinomial regression logistic regression are presented in Table 1 . The full list of citing articles with their evaluations is available in Suppl. Table 1.

Fig. 1.

Flow chart of citing article selection.

Fig. 2.

Bar plot displaying the categories assigned to citing articles stratified by study design. Category A refers to accurate presentation, category B to selective reporting of positive outcomes, category C to selective reporting of negative outcomes and category D to presentation of misleading outcomes.

Table 1.

Outcomes of the multinomial logistic regression analysis

Category	Study design
	Review		Meta-analysis		Non-randomized		Randomized		Basic research
	OR	95% CI	OR	95% CI	OR	95% CI	OR	95% CI	OR	95% CI
A	Reference	Reference	Reference	Reference	Reference
B	1.49	1.18–1.89*	0.30	0.09–0.99*	1.12	0.62–2.02	0.78	0.26–2.39	3.35	1.25–9.01*
C	0.03	0.01–0.08*	-	-	7.37	1.54–35.31*	-	-	7.73	0.68–88.18
D	0.18	0.11–0.29*	0.61	0.07–5.10	1.05	0.33–3.38	-	-	2.21	0.40–12.15

OR, odds ratio; CI, confidence intervals.

^⁎P-value < 0.05

3.2. Nonrandomized vs. randomized evidence

Literature search resulted in 5 new studies [20], [21], [22], [23], [24]; hence, the meta-analysis was based on a total of 20 studies (16 non-randomized and 4 randomized). Pooling of non-randomized studies indicated that hydroxychloroquine was associated with significantly lower risk of death (OR: 0.76, 95% CI: 0.61–0.96). The credibility ceiling test resulted in loss of statistical significance by applying ceilings of 5% (OR: 0.95, 95% CI: 0.85–1.05), 10% (OR: 0.97, 95% CI: 0.87–1.07) and 15% (OR: 0.99, 95% CI: 0.89–1.10). On the other hand, pooling of randomized controlled trials demonstrated no significant difference in mortality between the two groups (OR: 1.06, 95% CI: 0.96–1.18) (Fig. 3 ). Sensitivity analysis indicated similar outcomes using the modified Simmonds-Higgins (nonrandomized OR: 0.71, 95% CI: [0.51–0.98], randomized OR: 1.07, 95% CI: [0.94–1.21]) and the hypergeometric‐normal (nonrandomized OR: 0.71, 95% CI: [0.51–0.99], randomized OR: 1.07, 95% CI: [0.93–1.22]) model.

Fig. 3.

Forest plot of the effects of hydroxychloroquine on mortality by the pooling of nonrandomized and randomized studies. CI, confidence intervals; RE, random-effects.

3.3. Harmful effects

The outcomes of citation analysis are presented in Fig. 4 . The number of citations was highest for the article of Gautret et al. (1722 citations), followed by the studies of Geleris et al. (514 citations), Rosenberg et al. (311 citations) and Cavalcanti et al (132 citations). Alternative metrics were also higher in the same study (398 mentions in news/blogs and 112,092 in social media). The difference was largest in social media coverage with 11,440, 11,115 and 7,002 mentions of Cavalcanti et al., Geleris et al. and Rosenberg et al. articles, respectively. Citation mapping indicated the highest density of citations in the positive-outcomes study of Gautret et al., followed by the negative-outcomes one of Geleris et al. Interestingly, the bibliographic map indicated high density in another 2 studies of the same research team with the study of Gautret et al. [25,26] . On the contrary, citations were remarkably lower in studies reporting the harmful effects of hydroxychloroquine-azithromycin combination due to potential QT prolongation [27], [28], [29].

Fig. 4.

(A) Outcomes of PlumX Metrics regarding citationa, news/blogs mentions and social media coverage. (B) Citation density plot of articles in the hydroxychloroquine-azithromycin combination field. Deep colors indicate high citation density.

4. Discussion

The outbreak of Covid-19 has put clinicians and researchers in a desperate need of identifying effective interventions aiming to limit disease severity and in-hospital mortality. To this end, drug repurposing has emerged as an attractive option by studying drugs with already known safety profile [30]. Quickly, several candidate pharmacological treatments were endorsed by local guidelines and were widely prescribed, although the quality of evidence remained low. For example, the U.S. Food and Drug Administration (FDA), based on preliminary observational data, have approved the use of chloroquine and hydroxychloroquine via an emergency use authorization (EUA) procedure; however, the subsequent accumulation of negative outcomes by well-designed studies led to the revocation of the EUA [31]. The present study evaluated the presence of white hat bias in Covid-19 treatment research, providing evidence of citation bias and predilection of reporting of beneficial over harmful effects.

The analysis of the ACTT-1 trial citing articles revealed that the majority of them (55.83%) tended to preferentially report the positive outcomes associated with remdesivir administration. On the contrary, only 6.41% of citing papers favored the sole presentation of negative outcomes. This effect was maximized in basic research studies and was attenuated in high-quality studies (randomized controlled trials and meta-analyses). The positive part of the study results had significant influence on clinical practice as it led to the widespread adoption of remdesivir in treatment algorithms after receiving EUA by the U.S. FDA [32]. However, the potential beneficial effects of this intervention were not confirmed by the subsequent large-scale Solidarity trial, reporting that remdesivir was not associated with significant effects on hard clinical outcomes [33]. As a result, the latest World Health Organization (WHO) guidelines [34] recommend against the use of remdesivir due to uncertainties for efficacy in critical outcomes, long-term safety and effects on specific patient subgroups. It should be also noted that remdesivir constitutes a costly intervention, with an official cost of $2340–3120 [35] for a 5-day treatment course; therefore, the feasibility of further large-scale trials aiming to prove any tiny significant survival benefit in certain patient subgroups remains problematic.

It has been hypothesized that independently funded non-randomized studies may be biased by the prior beliefs of the researchers about the possible beneficial effects of the investigated intervention [4]. In addition, the publication probability of an observational study reporting negative outcomes may be lower than that of a randomized controlled trial due to the prospective registration of the latter [36]. The results of the present meta-analysis indicated that nonrandomized studies demonstrated that hydroxychloroquine was associated with significantly lower death risk, while no significance survival benefit was evident by the pooling of randomized controlled trials. Importantly, the direction of the summary odds ratio of non-randomized studies is in the opposite direction compared to that of randomized studies. The credibility ceiling test was implemented as a statistical way to model skepticism about the potential biases of observational evidence. Statistical significance was lost by applying even the lenient ceiling of 5%, questioning the precision of the meta-analysis of non-randomized studies. This result can be translated as that the effects become insignificant if we consider that any observational study cannot provide more than 95% certainty that the intervention is beneficial or harmful. Regarding the role of funding, it may be assumed that industry-funded studies may avoid disseminating spuriously positive results since they typically opt for a randomized study design with prospective protocol registration, limiting the possibility of selective reporting. Actually, one randomized controlled trial was funded by the industry, reporting no significant clinical improvement from the administration of hydroxychloroquine [16].

The combination of hydroxychloroquine and azithromycin is one of the most representative examples of an intervention initially proposed as beneficial that was subsequently shown to be ineffective and even harmful. Specifically, the claims of a research team [15,25,26] about the remarkable effects of this treatment combination on viral clearance and mortality rates enjoyed wide acceptance by the media, providing policy makers and healthcare stakeholders a potentially life-saving tool to confront the pandemic. Therefore, as indicated by the present results, the study of Gautret et al. [15] had the highest influence on the research community, compared to subsequent well-designed studies reporting negative outcomes. Interestingly, studies demonstrating the increased risk of QT prolongation and arrhythmia by the hydroxychloroquine-azithromycin combination received remarkably less citation impact [27], [28], [29]. Importantly, the COALITION II trial [37] which was partially funded by the industry indicated no significant benefit by the use of azithromycin compared to standard care. It should be also noted that a pharmacovigilance study reported that the majority of adverse effects by the use of drugs for the management of Covid-19 came from the administration of azithromycin with or without hydroxychloroquine, mainly consisting of electrocardiographic changes, hepatitis, and diarrhea [38].

The present study has several strengths. Selective reporting was evaluated by systematically searching the reference list in of a highly influential providing a large sample of full-text citing articles. The credibility ceiling test was used in the meta-analysis of hydroxychloroquine aiming to quantify the effects of skepticism due to potential biases of observational studies. Moreover, the greater influence of studies with positive outcomes was confirmed by both conventional and alternative metrics, while a citation density map was constructed to visualize the relative impact of studies in the field of hydroxychloroquine-azithromycin combination. On the other hand, the safe recognition of white hat bias remains problematic since no validated tool exists for its evaluation; hence, the assessment of parameters remains partially subjective. In addition, white hat bias may be complicated and difficult to distinguish from several types of biases associated with observational studies, such as cherry-picking of significant results and selective analysis. Regarding the meta-analysis, the small size of studies in conjunction with the rarity of the event (i.e., death) may challenge the assumption of normality [39]. To address this issue, two one-stage models were applied as a sensitivity analysis aiming to avoid within-studies approximations [11]. However, these generalized linear mixed models use maximum likelihood estimation and may also introduce bias by underestimating the between-study variance [40]. It should be also noted that the present study addresses only specific examples of information distortion that provide potential evidence of white hat bias; therefore, it may serve as a pilot one in the field and further studies are needed to investigate the magnitude of this type of bias in other proposed Covid-19 treatments.

In conclusion, this metaresearch study presented evidence of reporting and citation bias potentially associated with white hat bias in the Covid-19 treatment field. Personal beliefs in conjunction with an urgent need of the scientific community for effective interventions may have led to a predilection for propagating beneficial over neutral or harmful outcomes. In this context, the rapidly expanding body of Covid-19-related evidence is at risk of creating a medical misinformation mess and producing more noise than clinically useful information. To strengthen the credibility of evidence, both non-randomized and randomized studies need to opt for preregistration and public data sharing aiming to reduce the degrees of freedom in the analysis and reporting of outcomes. It is crucial to disclose both financial and non-financial conflicts, while authors should be vigilant in avoiding “spinning” during reporting of evidence in order to refrain from disseminating misleading results.

Author contribution

Ioannis Bellos: Conceptualization, Methodology, Software, Data Curation, Formal Analysis, Writing.