Commentary on “Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open label non-randomized clinical trial” by Gautret et al

ABSTRACT

The results of a clinical trial comparing hydroxychloroquine with or without azithromycin to the standard of care for the treatment of COVID-19 were recently published by Philippe Gautret et al. This study provides outstanding results for the combination of hydroxychloroquine and azithromycin over the standard of care, but the evidence was deemed insufficiently robust to warrant a public health decision to widen the use of hydroxychloroquine for the treatment of COVID-19. We provide a scientific critical review of the Gautret et al. publication, put the results in the context of the current knowledge, provide an evaluation of the validity of the results (from a methodologic perspective), and discuss public health implications. The study has a number of limitations, including small sample size, lack of comparability between patients in active treatment and control arms, lack of blinding, use of interim analyses without controlling for the risk of type 1 error, use of analysis in the per-protocol population instead of the intention-to-treat population, and inconsistencies between the study protocol and article. However, none of these observations is of a nature to reverse the conclusions. The study brings useful knowledge consistent with available evidence and clinical practice from China and South Korea, which could have prompted quicker policy decision-making.

The results of a clinical trial comparing hydroxychloroquine with or without azithromycin to the standard of care for the treatment of COVID-19 were published in the International Journal of Antimicrobial Agents by Philippe Gautret et al. [1]. This study provides outstanding results for the combination of hydroxychloroquine and azithromycin over the standard of care.

This publication was considered interesting by several health authorities, including Ministers of Health, but the evidence was deemed insufficiently robust to warrant a public health decision to widen the use of hydroxychloroquine for the treatment of COVID-19[2]. The World Health Organization did not initially issue an opinion about this study or recommendation for use of hydroxychloroquine, a commonly used and well-established product for the treatment of malaria in endemic malaria regions that is also used to treat other diseases. Instead, later, the World Health Organization warned against using untested medicines without the right supporting evidence because it could raise false hope, referring obviously to hydroxychloroquine[3]. Poland, however, appears to be an outlier and has updated the summary of product characteristics of chloroquine to introduce a mention of efficacy against coronavirus[4]. Dr Krzysztof Simon has reported the successful use of chloroquine in a Wroclaw hospital in clinical practice[5]. Poland was followed by Jordan[6], and Tunisian health agency (INEAS) recommended the use of hydroxychloroquine in its first guidelines[7].

Very high concentrations of cytokines were identified in critically ill patients with coronavirus 2 infection[8]. Hydroxychloroquine is widely used in autoimmune diseases such as lupus and contribute to reduce the production of cytokine and proinflammatory factors. In vitro trials have shown the efficacy of hydroxychloroquine on other coronaviruses[9]. More recently, in vitro trials showed that hydroxychloroquine and chloroquine are very potent for inhibiting coronavirus 2 [10–13]. There is 70 years of experience with chloroquine, and it has a very well-established safety profile with some well-known serious adverse events that are preventable with appropriate pretreatment and on-treatment monitoring.

The urgency of the public health crisis, the dramatic increase in the number of fatalities in Europe, and the high risk in emerging countries that have weak hospital and intensive care unit infrastructure contrast with the lack of recommendation on the combination of hydroxychloroquine and azithromycin.

In lay media, several have criticized the quality of the study and questioned the reliability of the results [14–18]; however, no scientific critical review of the study has been published. The aim of this paper is to provide an objective scientific critical review of the Gautret et al. publication, put the results in the context of the current knowledge, provide an evaluation of the validity of the results (from a methodologic perspective), and discuss public health implications.

Critical Review of the Publication by Gautret et al

General study design

This was a prospective, open-label, nonrandomized controlled trial conducted across five study centers. One center (Institut Hospitalo-Universitaire Mediterranée [IHU]) administered hydroxychloroquine, while four centers from the same region recruited the control patients. It is unclear whether the control patients were part of the study or not in the protocol. The reported sample size power calculation applies to a comparative 2-arm study; however, the abstract refers to a single-arm study, and according to the protocol submitted to the EU Clinical trials register (clinicaltrialsregister.eu ID number, 2020-000890-25/FR), the study was originally designed as a noncontrolled study. Thus, it appears that the control arm was a later amendment of the study.

The intervention hydroxychloroquine alone or combined with azithromycin was used in addition to the standard of care for the treatment of coronavirus 2. The comparator was standard of care, which is not yet standardized – this may introduce a bias. The patients receiving the active treatment were enrolled in a specialized unit benefiting from high-level experts and the most up-to-date infrastructure. In comparison, control patients recruited through other hospitals, such as Briançon, which is a remote hospital in the French Alps, would receive fair and good quality of care but not as specialized as in the IHU.

This raises the question of whether differences in management of patients between centers, and therefore between arms, might have influenced the outcomes of viral clearance. Management of these patients is intensive to help them survive by maintaining vital functions and preventing complications (e.g., infections or heart or kidney failure). However, differences in patient management may not necessarily impact coronavirus carriage and clearance. In addition, an open study always raises the issue of observer bias (i.e., the investigator’s knowledge of a patient’s treatment may influence the study outcome assessment). When considering subjective outcomes, it represents a very important source of bias. However, in this case, the outcome measure is an automated biological measure. It is very unlikely that the outcome measure might have been influenced by the open design. Development of the study product for a double-blind trial would have required several months to provide an appropriate formulation and relevant stability data as required by good manufacturing practices/good clinical practices. It is not a feasible option in the context of the ongoing crisis.

These important methodologic biases represent clear conceptual methodologic limitations, and several could have been avoided, but it is unlikely that avoiding them would have reversed the trend seen in the trial.

Population

The study included hospitalized patients aged over 12 years with polymerase chain reaction (PCR)–documented coronavirus 2 carriage from a nasopharyngeal sample at admission, whatever their clinical status. The patients who refused to receive hydroxychloroquine were used as control patients, as were noneligible patients. Obviously, this makes the comparability of participants between the control and active treatment arms highly questionable. When comparing two treatment arms, it is important to have comparable populations. Alternatively, statistical methods should be used to control for differences in patient characteristics between arms.

There is an obvious selection bias as patients were not randomly assigned. Patients from the IHU were very different from those at the other centers and were channeled based on unknown drivers. As expected, the analysis of baseline characteristics showed clear differences between the treatment arms: the hydroxychloroquine-treated patients were older (51 years vs 37 years), were less likely to be asymptomatic (10% vs 25%), and were more likely to present with pneumonia (30% vs 12.5%). These differences are considerable, even if nonstatistically significant, but are probably a bias against the hydroxychloroquine intervention. This suggests that the study provided conservative results when considering population differences. Indeed, older and more severely ill patients are likely to have a weaker immune defense response to virus/bacterial infection. Although this may have serious impact on the patient prognosis and survival, it may have limited impact on the efficacy of hydroxychloroquine on virus clearance.

Informed consent

It is unclear in the publication whether participants in both the active and control arm gave informed consent. However, as control patients were treated according to normal practice, and the primary outcome assessment was not interfering with their normal care, they may be considered an external standard-of-care control arm that may not be part of the study. Although this point requires clarification, it is unlikely to affect the results.

Primary endpoint

Carriage of coronavirus 2 is based on nasal and oropharynx sampling and assessed through a well-established technique to detect coronavirus 2 RNA using real-time reverse transcriptase-PCR. It is a reliable technique and should not raise significant objections. However, it appears the endpoint was changed during the study, although the article is unclear about this. The planned assessment time points were days 1, 4, 7, and 14, and the reported sample size calculation refers to a reduction in viral load at day 7. This contradicts the outcome section of the article, in which the primary outcome is stated to be viral clearance at day 6. Results are reported at days 3, 4, 5, and 6. It may be that 7 days was initially intended as the main assessment endpoint, but the authors decided to report findings after 6 days when clear results were observed, considering the urgency of the situation. However, these contradictions within the article as well as between the article and the protocol raise questions about the credibility of the article. It is important to note that this endpoint is a surrogate endpoint and would not preclude clinical and survival benefit. However, with the lack of approved drugs to fight coronavirus 2 infection, which can be fatal in some patients, this information is of high importance.

Intervention

The dosing of hydroxychloroquine is well reported in the methods section: 200 mg three times a day for 10 days; however, the administration schedule of azithromycin is not reported in the description of the methods, but rather in the results section. It may also be noted that there is no mention of azithromycin in the summary protocol on the EU Clinical Trials Register. According to the abstract, the patients receiving azithromycin were selected based on clinical profile, but the specific profile is not reported. There is no clarity on this point. The review of the baseline results does not allow identification of clear criteria such as age, lower respiratory tract infection, or onset of disease. The plasma concentration of hydroxychloroquine in this combination therapy arm is not different from the monotherapy treatment arm.

Male patients were more likely to receive azithromycin. Male patients comprised 37% of the control arm, 28% of the hydroxychloroquine arm, and 66% of the combined hydroxychloroquine/azithromycin arm. There is an obvious imbalance; however, it is unclear how sex may be a confounding factor. How this may have impacted the very high rate of clearance of COVID-19 in this subgroup population still must be clarified. Although this constitutes poor practice in study reporting, it is unlikely to introduce a significant bias in the reported results.

Statistical methodology

The use of the Fisher exact test is appropriate, but statistical methods for interim analyses and early interruption of clinical trials were not used[19]. It appears daily interim statistical analyses were performed, and the trial was interrupted early without using methods to control for the probability of incorrectly rejecting the null hypothesis of no treatment differences. However, the magnitude of difference appears to be so large that the hypothesis of no difference between arms would be rejected even if appropriate methods for interim statistical had been used.

A very important point concerns the population used to assess the primary endpoint. The authors used the per-protocol population whereas intention-to-treat analysis is normally employed for clinical trials aiming to establish the superiority of a clinical treatment[20]. This is a particularly important point in this study as all patients who dropped out belonged to the hydroxychloroquine arm. This imbalance requires a careful reanalysis of the results.

Among the patients who dropped out, three patients were transferred to the intensive care unit: two were PCR positive and one was PCR negative at the time of transfer. One died while being PCR negative, one withdrew consent while being PCR negative, and one stopped because of adverse event nausea while being PCR positive. Thus, of the patients who dropped out, 50% were negative at the time of dropout and 50% were positive.

By performing a statistical analysis using the last-observation-carried-forward imputation method to account for missing data (a conservative methodology widely used by regulatory authorities)[21], we could assess the proportion of patients who were negative at 65% for hydroxychloroquine arm compared with 12.5% for the control arm, which is still statistically significant (p = 0.0012) and very clinically relevant.

If we employ a very conservative method that assumes all dropouts were PCR positive at day 6, the proportion of patients who were PCR negative would become 54% for hydroxychloroquine arm compared with 12.5% for the control arm. This is still statistically significant (p = 0.096) and very clinically relevant.

Finally, it is informative also to consider separately the subgroup of patients treated with hydroxychloroquine alone, among whom the proportion with viral clearance was 57%, compared with 12.5% for the control arm. This difference is again significant from statistical (p = 0.0187) and clinical perspectives.

Although there may be uncertainty in the effect size, ranging from 50% to 70%, the intervention has shown a very large effect size that could not be undermined by statistical considerations.

Medical writing quality

The poor quality of medical writing is also an issue. We identified more than a dozen inaccuracies, imprecisions, or mistakes in the abstract alone. They are reported in Table 1. This may be related to the time pressure to release the first non-Chinese clinical results on coronavirus 2–infected patients. However, this level of medical writing may compromise the perceived quality of the evidence reported and may lead readers to doubt the quality of investigational procedures used during the trial as well. It is important that they be corrected in the final version uploaded in the journal.

Table 1.

The inaccuracies and inconsistencies or mistakes in the abstract

1	Efficient instead of effectiveness (reported twice)
2	Patients included instead of enrolled
3	Role of hydroxychloroquine should be efficacy and safety
4	On respiratory viral load should be on nasopharyngeal viral load
5	Single arm while later in the abstract it is comparative to negative control
6	Patients included in a protocol instead of a study
7	Negative control instead of standard of care control
8	Early March to March 16th not precise. The study was approved on the 4th of March not clear when it started.
9	Viral load was tested instead of was measured
10	Hydroxychoroquine not specified on the top of standard of care
11	Azithromycin appears as a new intervention not described in the methodology nor in the protocol.
12	Untreated patients from another center should be from 4 other centers
13	Presence of virus at day 6 was the primary end point while it was supposed to be at day 7 in the protocol and the protocol specify the objective is to measure time to clearance of virus load.
14	While the study did not measure time to virus clearance it concluded at a reduction of carrying duration
15	Survey instead of study

In conclusion on the publication

This study has been poorly reported, and the trial design introduces several biases. Some of these biases could have been avoided at the time of protocol development with thoughtful consideration. The quality of medical writing is poor and suggests a quickly written manuscript and no support from a skilled methodologist. The priority obviously was given to the fast communication of the results that may have a dramatic impact in a situation of a severe pandemic.

The authors could have significantly improved the quality of their manuscript and its public health impact by discussing in a more transparent way all the limitations of the study and invested additional effort in medical writing; however, none of these observations is of a nature to reverse the results. These results appear to be acceptable in the light of the effect size. These data may have been influenced by the several biases identified in the study manuscript but not to an extent to make the intervention ineffective. It clearly works even if we cannot accurately specify the actual effect size.

How This Study Adds to Current Knowledge

In China, 483 studies on COVID-19 were registered on the national clinical trial registry[22]. The most studied intervention was chloroquine and hydroxychloroquine as monotherapy or as part of combination therapy, represented in 17 trials. Most results are not yet published, but evidence has accumulated on the effectiveness of chloroquine and hydroxychloroquine. A recent review of 100 patients aggregated from several trials has been published and reports the benefit of chloroquine on clearing coronavirus 2 from infected patients[23]. However, it does not provide details on patient disposition and outcomes. It aims to be a signal to the community that evidence supports the use of chloroquine, but details are to follow as these trials are not yet finalized.

The US Centers for Disease Control and Prevention provides information on the use of hydroxychloroquine and chloroquine in the USA to treat coronavirus 2–infected patients. They describe a multiplicity of different hydroxychloroquine regimens currently used in the USA. It is also used as a prophylactic treatment for health-care professionals[24].

Beyond Chinese trials on chloroquine and hydroxychloroquine, several randomized trials are being conducted on efficacy and safety of hydroxychloroquine in the USA, South Korea, Norway, and Australia in coronavirus 2–infected patients (ClinicalTrials.gov ID, NCT04316377, NCT04315896, and NCT04308668). In addition, two studies are ongoing, assessing the prophylactic effect for health-care professionals exposed or not exposed to patients infected with coronavirus 2 (ClinicalTrials.gov ID, NCT04318015 and NCT04308668).

All 8 of the treatment guidelines endorsed by the National Health Commission of China, including a Chinese expert consensus report, recommended using chloroquine 500 mg twice daily for 10 days maximum (Table 2) [25–32]. However, one guideline recommended not using it in conjunction with azithromycin to avoid the risk of cardiac arrhythmia (Table 2).

Table 2.

The inclusion of chloroquine phosphate on the guidelines

	Guideline name	Publishing organization	Key information
2020.02.19	Novel coronavirus pneumonia diagnosis and treatment plan (provisional 6th ed)[24]	National Health Commission	Eligibility: general treatments for all COVID-19 cases, regardless of disease severity Dosage: 500 mg, twice per day, use no longer than 10 days
2020.02.20	Expert consensus on chloroquine phosphate for the treatment of novel coronavirus pneumonia[25]	Health Commission of Guangdong Province for chloroquine in the COVID-19 treatment	It recommended chloroquine phosphate tablet, 500 mg twice per day for 10 days for patients diagnosed as mild, moderate, and severe cases of novel coronavirus pneumonia and without contraindications to chloroquine. Contraindicated to use combined with macrolide antibiotics, including azithromycin.
2020.02.21	Close monitoring the adverse effects of chloroquine phosphate for the treatment of novel coronavirus pneumonia[26]	Health Commission of Hubei Province	Chloroquine phosphate can cause acute death. Lethal dose for adults is 2–4 g.
2020.02.28	Notifications on the adjustment of dosage of chloroquine phosphate for the treatment of novel coronavirus pneumonia[27]	National Health Commission	Chloroquine phosphate (500 mg bid for 7 days for adults aged 18–65 years with body weight over 50 kg; 500 mg bid for days 1 and 2, and 500 mg qd for days 3–7 for adults with body weight below 50 kg) Contraindicated to use combined with macrolide antibiotics, including azithromycin.
2020.03.02	Shanghai expert consensus on the integrated treatment for novel coronavirus pneumonia[28]	Shanghai Expert Panel on the Clinical Treatments for COVID-19	Hydroxychloroquine sulphate and chloroquine phosphate were both recommended.
2020. 03.03	Novel coronavirus pneumonia diagnosis and treatment plan (provisional 7th edition)[29]	National Health Commission	Chloroquine phosphate (500 mg bid for 7 days for adults aged 18–65 years with body weight over 50 kg; 500 mg bid for days 1 and 2 and 500 mg qd for days 3–7 for adults with body weight below 50 kg)
2020.03.05	Guangdong expert consensus on the Chinese integrative medicines for the prevention and treatment of COVID-19[30]	Guangdong Association of Integrative Medicine	Chloroquine phosphate (500 mg bid for 7 days for adults aged 18–65 years with body weight over 50 kg; 500 mg bid for days 1 and 2 and 500 mg qd for days 3–7 for adults with body weight below 50 kg)
2020.03.15	Shandong expert consensus on the diagnosis and treatment for novel coronavirus pneumonia[31]	Shandong Expert Panel on the Clinical Treatments for COVID-19	Hydroxychloroquine sulphate (200 mg, 3 times per day)

The Korean guidelines recommend the use of chloroquine 500 mg twice daily or hydroxychloroquine 400 mg per day[33].

The Gautret et al. study is consistent with the results of recent Chinese studies not fully reported and is very consistent with current scientific knowledge. There is a well-founded biological assumption to support the validity of the tested hypothesis. Real-world large-scale clinical practice in China and Korea supports the experimental finding even though the exact contribution of the chloroquine in control of the outbreak cannot be accurately quantified at this time in either country. All current evidence, while not as robust as that from a double-blind randomized clinical trial, points to the effectiveness of hydroxychloroquine in patients with coronavirus 2 infection.

Evidence-based medicine teaches us to consider all evidence, including case reports and expert opinions for decision-making. In the specific case of available evidence on hydroxychloroquine used in coronavirus 2 infection, the recommendation would receive an evidence level 2b or 2 c [34–36]. Such evidence grading supports the recommendation for use of hydroxychloroquine for COVID-19 infection, especially in a crisis situation.

The overall knowledge around hydroxychloroquine should translate to policy decision-making to control the COVID-19 pandemic. Especially as no alternative option currently exists in readily accessible quantities.

Why French Authorities Did Not Recommend Hydroxychloroquine

There have been numerous public questions about why hydroxychloroquine was not recommended as an early treatment in France. One of the authors of the study, Raoult, is also a member of the expert council advising the president and the government on the COVID-19 crisis. Therefore, the council is certainly well informed about this study. This raises the question of why this study is not leading to any recommendation for generalized use of hydroxychloroquine to defeat the COVID-19 outbreak.

Scientific perspective

In France, methodology tends to supersede all evidence. If the methodology is not perfect, the evidence is ignored. This is very well illustrated by the debate between two major experts in the field, Jean Luc Harousseau and Jean François Bergman, both highly respected experts with extensive experience in policy decision-making in several governmental agencies[37]. In this context, the poor reporting and the multiplicity of biases in this study prevent any French methodologist from considering the study, while a careful assessment would have shown the limited impact of poor reporting and biases on the results. French methodologists tend to disqualify studies but not to assess learning points despite biases and put them in the context of current knowledge.

The possible second scientific obstacle may be competition between scientists. There is always a very high competition among scientists to be first. Some scientists succeeded in establishing the Gautret et al. study as a hypothesis-generation study and not a hypothesis-testing study. Several other experts were convinced that the biases and poor reporting invalidated the study results.

In addition, a large European study, Discovery, will be launched next week and will enroll 3200 patients in 5 arms including placebo, remdesivir, a combination of lopinavir and ritonavir with or without interferon beta, and hydroxychloroquine (apparently without azithromycin)[38]. Six hundred and fifty patients per arm seems large, but no information is available at the time of writing concerning how the sample size was estimated. Depending on the population included, results may just be negative. Little is reported at the time we submit this manuscript. WHO is launching a similar but global study called SOLIDARITY to test the same interventions against placebo[3].

Thus, the decision on the recommendation of hydroxychloroquine may be delayed until several hypothesis-testing studies are reported, while a lot of knowledge is available from Chinese and South Korean experience. The low risk associated with a hydroxychloroquine recommendation to a specific population versus the potential high benefit would argue in favor of the use of hydroxychloroquine until more evidence brings definite results, rather than waiting for new evidence.

Political perspective

It is very difficult for politicians to make the right decision in a crisis. All decisions are scrutinized and criticized. Hydroxychloroquine has been presented as an unsafe therapy, and the study has been widely criticized in the media.

The primacy of the precautionary principle ‘primare non nocere’ after the extensive transmission of HIV via blood transfusion in France has left profound effects [39,40], and decision-makers are driven by safety first when making decisions. More recently, the massive ordering of H1N1 vaccines and stockpiling of antivirals in France has provoked considerable public controversy[41]. The European public health decision-makers appear to be risk-averse. This is very clear when comparing the US Food and Drug Administration and European Medicines Agency regulatory approval decisions.

Public health perspective

An important question will likely raise future controversy when evidence has accumulated: why did we not start the recommendation of hydroxychloroquine prescription earlier? The risk of serious adverse events with short-term administration of hydroxychloroquine is well below all the case fatality rate estimates in Europe[42]. Hydroxychloroquine may offer the potential to quickly control the COVID-19 outbreak, and associated social and economic psychiatric consequences, as well as consequences for health-care professionals, with adverse events that are manageable.

The current confinement strategy will probably have worse direct consequences on health (e.g., severe psychiatric consequences)[43]. It will also have indirect consequences on health, as economic losses entailed by this strategy will impact future funding of health systems. If a therapeutic option is available with limited evidence, one should carefully weigh the benefit and the risk of that option versus alternative options.

Hydroxychloroquine treatment with massive testing and limited confinement has successfully worked in South Korea to control the outbreak with an impressively low rate of fatalities[44].

Conclusion

Gautret et al. conducted a quick and dirty study, as methodologists would legitimately say, with poor quality reporting and several biases. However, the critical review of this study suggests that it brings useful knowledge, on the top of an already existing one, that could have prompted quicker policy decision-making.

A temporary conditional approval known in France as ‘Recommandation Temporaire d’Utilisation’ could have been granted to hydroxychloroquine and potentially saved lives.

Assuming the ongoing studies return negative results, everyone will consider it a good decision not to recommend hydroxychloroquine. But if the results are positive, which is highly probable, then there will be furious discussion over why it was not taken earlier. In that case, we will learn about the risk of delaying decisions in an uncertain environment.

Evidence-based medicine would likely allow an intermediate grade for recommending the use of hydroxychloroquine for the treatment of some specific cases of COVID-19.

So far, European decision-makers have shown very little ability to learn from China [45] and South Korea[44], the only two countries that have been able to control the outbreak. Cultural differences, language barriers, and arrogance from the old Europe may certainly explain why best practice knowledge sharing failed in this situation.

Post Scriptum

After submission of this manuscript, four new trials have become available that do not change the conclusion of this paper.

Disclosure statement

No potential conflict of interest was reported by the author.