Why the article that led to the widespread use of hydroxychloroquine in COVID-19 should be retracted

Abbreviations

ANSM

Agence nationale de sécurité du médicament et des produits de santé

AZ

azithromycin

COVID-19

coronavirus disease 2019

HCQ

hydroxychloroquine

ICU

intensive care unit

IGAS

Inspection générale des affaires sociales

IQR

interquartile range

MMR

measles, mumps, and rubella

ND

not done

NPCR

number of negative PCRs

PCR

polymerase chain reaction

In July 2020, a publication on the coronavirus disease 2019 (COVID-19) treatment with hydroxychloroquine (HCQ) and azithromycin (AZ) by Gautret et al. concluded that “hydroxychloroquine treatment is significantly associated with viral load reduction/disappearance in coronavirus disease 2019 (COVID-19) patients and its effect is reinforced by azithromycin” [1].

The public and political impacts of this research and its conclusions were large, and began as soon as its preprint was put online in March 2020. Nevertheless, the paper from Gautret et al. suffers from several methodological flaws that we summarize and discuss below.

Main issues that could be observed

Methodological aspect

Gautret et al. was a small, non-randomized study, published in July 2020. Although not planned a priori in the study protocol, as shown in a recent Inspection générale des affaires sociales (French General Inspectorate of Social Affairs – IGAS) report, a control group was set up for this study, but it was not appropriate. Indeed, the patients of this group came from a different population that was obtained in various locations (Marseille, Nice, Avignon and Briançon hospitals), and also submitted to different types of monitoring and evaluation. Moreover, the observed discrepancy in the groups repartition by age (control 34, IQR 14.5–64.3 vs. treatment 51.5, IQR 41.5–59.3) may also explain a higher viral load in the control group since peak viral load is associated with age.

Furthermore, in January 2021, Gautret et al. published a re-analysis of their data in a letter to the Editor [2]. At this stage, some changes appeared, as the number of enrolled patients (26 vs. 24 patients treated with hydroxychloroquine), of deaths (1 vs. 2), of intensive care unit (ICU) transfers (3 vs. 5), negative polymerase chain reactions (PCRs), and of days of patient exclusion (all before day 4 vs. most on day 7). All these data are inconsistent with the initial paper (Table 1 ). The number of negative PCRs was also inconsistent at all time points between the two studies, and remains impossible to reconcile even considering “Not Done” PCRs as negative. Viral load at day 7 was a time point of detection for the viral load planned in the study protocol, but the Gautret et al. initial publication reported data until day 6 instead. In this context, data at day 7 were not significant regarding treatment efficacy. The study protocol (EudraCT number 2020-000890-25) also planned a follow-up until day 14, but these results were never published. Such selective outcome reporting is clearly inconsistent with good-quality research practices and is often qualified as questionable research practices that could lead studies to be classified as “high risk of bias” according to Cochrane methodology.

Table 1.

Comparison between July 2020 [1] and January 2021 [2] excluded patients are counted separately in the July 2020 paper, but are deduced from enrolled patients in the January 2021 data.

	Gautret et al. July 2020 [1]			Gautret et al. January 2021 [2]
Drug regimen	HCQ	HCQ + AZ	Control	HCQ	HCQ + AZ	Control
Enrolled	14	6	16	16	8	18
Excluded	6 (three ICU transfers on day 2, 3 and 4, two left on day 3, one death on day 3)			1 (day 7)	2 (day 3 and day 4) One patient added (day 7)	4 (all on day 7)
Deaths	1			1	1
ICU transfer	3			2	2	1
NPCR Day 3	5 (0 ND)	5	1 (7 ND)	5	5	2
NPCR Day 4	7 (1 ND)	5	4 (3 ND)	7	5	5
NPCR Day 5	6 (1 ND)	6	3 (7 ND)	7	6	4
NPCR Day 6	7 (1 ND)	6	2 (0 ND)	8	6	3
NPCR Day 7	Unpublished	Unpublished	Unpublished	7	6	4

AZ: azithromycin; HCQ: hydroxychloroquine; ICU: intensive care unit; ND: not done; NPCR: number of negative PCRs; PCR: polymerase chain reaction.

Clinical aspect

The reported data were coming from semi-quantitative and qualitative PCRs.

According to a report from IGAS, different PCR positivity thresholds were applied to the control and treatment group. Moreover, by cross-checking the results presented in the article with the file containing the source data of the study, IGAS has identified that 6 patients from the control group were incorrectly transcribed in the article: negative according to the source data, but positive in the publication.

Furthermore, secondary endpoints are not reported in this article. The authors also report the exclusion of 4 patients from the treatment group (3 patients transferred to ICU and 1 deceased), while these patients should have been considered as treatment failures although average length of hospital stay and mortality are secondary outcomes of the submitted protocol [3].

Statistical approaches

Of course, the study was not analyzed according to an intent-to-treat principle. Considering the excluded patients, a re-analysis has been made that showed that the data were no longer significant [3]. Furthermore, the changes in planned time points raise concerns about possible p-hacking, particularly regarding azithromycin for which the conclusion is drawn based upon a series of 6 patients, without randomization. Moreover, according to repeated measures, statistical tools used were not appropriate.

Ethical aspect

Authorization was granted by Agence nationale de sécurité du médicament et des produits de santé (French National Agency for the Safety of Medicines and Health Products – ANSM) on March 6th for a study on hydroxychloroquine only while this study started on March 3rd 2020 using both hydroxychloroquine and azithromycine.

Furthermore, children below 12 years were claimed to be excluded, but two 10-year-old children were reported in this study.

It should also be noted that this article was submitted, reviewed and accepted in one single day by a journal whose editor-in-chief is a member of the same institute as the authors. While this information was previously available in the meta-data of the article, it has now been removed from the publisher's website for no apparent reason, but the extracted metadata is still available on public repositories (https://github.com/lonnibesancon/OpenSciencePandemic) of a published manuscript that analyzed the publishing time of COVID manuscript [3]. All these concerns are summarized in Table 2 .

Table 2.

Summary of the differences between authorized clinical protocol and Gautret et al.’s data.

	Authorized protocol EudraCT 2020-000890-25	Gautret et al.’s data [1]
Drug	HCQ	HCQ + AZ
Exclusion criteria	Children < 12 years excluded	Two 10-year-old children were included
Control group	Not included	Control group from other centers (Marseille, Nice, Avignon and Briançon)
Timepoints	Days 1, 4, 7 and 14	Days 1, 2, 3, 4, 5 and 6
Treatment group	5 12–17 years	0 12–17 years
	10 18–65 years	17 18–65 years
	10 ≥ 65 years	3 ≥ 65 years
Authorization	March 5th	Trial started on March 3rd
Secondary endpoints	Apyrexia, normalization of respiratory rate, and average length of hospital stay and mortality	No data on apyrexia, normalization of respiratory rate or average length of hospital stay. Mortality is not discussed

AZ: azithromycin; HCQ: hydroxychloroquine.

Discussion

All taken together, these methodological, biological, statistical and ethical concerns directly hinder the validity, trustworthiness and robustness of the final findings. Most of these shortcomings are usually detected during the peer review process, which suggests that it was not carried out with rigor and objectivity.

The consequences of this article, with results quickly spread all around the world by influencers and several politicians including presidents, were countless. Hydroxychloroquine was rapidly promoted as an easy fix, allowing government leaders to downplay the seriousness of the COVID-19 pandemic and ignore advices formulated by numerous medical experts [4]. Despite many attempts to replicate the results of the Gautret et al. paper, later systematic reviews and meta-analyses found no evidence that HCQ prophylaxis or treatment (associated or not with azithromycin) reduced COVID-19 related mortality, but rather could be associated with increased mortality in COVID-19 patients [5], [6], [7]. The sharp demand for HCQ also led to shortage of the drug for patients with diseases for which HCQ use has proven to be effective, such as lupus and inflammatory arthritis. In addition, the Gautret et al. study resulted in a tremendous waste of research money and effort trying to replicate its results (more than 250 studies worldwide), while one can also argue that it might have prevented better public health decisions, indirectly causing increased morbidity and mortality [8]. A lecture given in June 2022 evaluated that in 8 countries, “HCQ was associated with at least 9,564 deaths during first wave of COVID-19 pandemic” after taking into account the mortality increase and the prescription rates (submitted).

The flawed conclusions from Gautret et al. have quickly disseminated globally. The scientific community identified the methodological, biomedical and ethical concerns days after its publication. Nevertheless, three years and more than 6,000 citations later, the paper has not been retracted. If its influence on medical professionals and policymakers seems to be declining with time, it remains still prominent among conspiracist groups, with potentially devastating effects, leading to increased mortality directly associated with hydroxychloroquine's adverse side effects, or indirectly by strengthening anti-isolation and antivaccination opinions. Finally, with more than 6,000 citations, this paper still alters journal and researchers bibliometric indexes unreasonably and immorally. We find this lack of retraction particularly alarming, especially about the message it sends, almost 25 years after the Wakefield et al. paper which suggested that the measles, mumps, and rubella (MMR) vaccine was associated with intestinal morbidity and autism. Despite its small sample size and non-randomized design, lack of reproducibility, and findings of ethical violations and scientific fraud, it took 12 years to retract the Wakefield study, meanwhile receiving broad publicity and causing a noticeable decrease in vaccination rates and increase in measles outbreaks. Many scientists have raised concerns about the slow correction of scientific literature, in particular in the medical field [9]. The overwhelming evidence of issues with the study by Gautret et al. undeniably calls for editorial and publisher actions; promoting scientific rigor and integrity sometimes requires retracting manuscripts [10].

Conclusion

As researchers, we worry that the lack of retraction of Gautret et al. could become a dangerous precedent. We, therefore, urge the scientific community to join us in asking that the International Society of Antimicrobial Chemotherapy and the publisher of the International Journal of Antimicrobial Agents to reverse the editorial board's decision and retract the article by publicly sharing the reasons for such a decision.

Authors contributions

All authors contributed equally to this manuscript. All authors took equal parts in discussions that led to this article, to the interpretation of the data it contains, revised the manuscript according to their views and approved the final version. All authors participated equally to the proofreading of the manuscript and its finalization. Damien Barraud is the guarantor for this article but all authors support this analysis.

Disclosure of interest

EMB has received consulting and speaker fees from publishers and research institutions, and gets donations through Patreon.com. HM has shares in H2MW, the company that publishes the “Revues et Intégrité” blog. EB is a full-time Novartis employee. Views are his own.

The other authors declare that they have no competing interest.