Review of: “Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial Gautret et al 2010, DOI:10.1016/j.ijantimicag.2020.105949

Summary

This is a study of the effect of hydroxychloroquine as treatment of hospitalised patients with Covid-19. The endpoint was virus presence in nasopharyngeal swabs after six days. Consenting patients received hydroxycholoroquine, and in some cases azithromycin. A control group consisted of patients who were either from another centre, or declined this treatment. The reported results are that of twenty patients who were treated with hydroxychloroquine, 14 had negative swabs at day 6, as compared with two of 16 controls. The effect was stronger in those also receiving azithromycin, where six out of six had negative swabs at day 6. The authors conclude that hydroxychloroquine is efficient in clearing viral nasopharyngeal carriage of SARS-CoV-2 in COVID-19 patients in only three to six days, which is reinforced in those also using azithromycin.

As outlined below, this study suffers from major methodological shortcomings which make it nearly if not completely uninformative. Hence, the tone of the report, in presenting this as evidence of an effect of hydroxychloroquine and even recommending its use, is not only unfounded, but, given the desperate demand for a treatment for Covid-19, coupled with the potentially serious side-effects of hydroxychloroquine, fully irresponsible.

Major comments

Methodology

• 1.The index group and control group were drawn from different centres. The information that is given about characteristics of index group and control group is minimal, and still major differences are evident from all three variables shown (age, sex, presence of symptoms). The authors have performed statistical tests on these baseline characteristics, which is inappropriate. In the text they emphasise the absence of statistically significant differences between groups, implying that absence of statistical significance proves equality, which shows a lack of understanding of basic statistics.

  It is remarkable that in a randomised trial, when only chance may have introduced differences between groups, authors go out of their way to present a long list of baseline characteristics to lend credibility to the fairness of comparing outcome occurrence between groups, where here, in a non-randomised comparison of patients from different centres who clearly do differ, authors have not made the slightest effort to present such baseline characteristics. The reviewer can only come to the conclusion that the comparison with the control group is meaningless.

• 2.It is reported that 42 patients met the eligibility criteria, and of these 16 were in the control group, and 26 in the treated group. Of these 26, six were excluded (and incorrectly labelled as lost to follow-up): three were transferred to the ICU, one died, and two terminated treatment or were discharged. Firstly, it is noteworthy that 4/26 treated patients deteriorated and 0/16 control patients, which emphasises that the groups were different. More importantly, excluding patients who deteriorated from the analyses introduces severe selection bias, since it selectively excludes people who did not do well (as an extreme example: if 25/26 treated patients had died, and one had virus clearance at day 6, would a claim of 100% clearance be valid?).
• 3.Whereas clinical course and side effects are part of the study aim, the authors indicate they will only present these in a subsequent publication. Given the small number of patients treated with hydroxychloroquine, with or without azithromycin, all with one centre, it would have been entirely feasible to collect and present these results. In the Discussion the authors unequivocally recommend the use of hydroxychloroquine with azithromycin in patients with Covid-19. Even if we see this study as an uncontrolled case series of treated patients - since the control group is inappropriate -, and even if we concede that clinical recommendations can in rare circumstances be based on such a case series (the famous example of the effect of parachutes when jumping from airplanes, as given by Austin Bradford Hill), that is not the case here. Minimal conditions for such a recommendation would be that the prognosis of the disease without treatment is well-established and very poor (1), that an effect on clinical outcome is unequivocally shown (2), and that adverse effects of the treatment are presented, too (3). None of these conditions were met here. 
• 4.One may question the choice of viral load in nasopharyngeal swabs, as well as the accuracy of the measurement thereof, as a relevant endpoint.
• 5.An argument a contrario that could be put forward is that if anything, the control group was younger and hence of better health than the case group, and that even if the groups are different, it is remarkable that the older, treated group, did better. This argument is spurious, for young people who need hospital admission may have a particular risk profile that makes them different from older people. This may be with regard to nasopharyngeal viral presence, as well as clinical symptoms. One also wonders whether the groups of patients were included at a same stage of disease. So, all that is left is 50% viral clearance after one week in 20 patients. This may well be simply the natural course.
• 6.This study has been registered in the EU Clinical Trials register, under EudraCT: 2020-000890-25. In the registered protocol, the primary endpoint is listed as results of SARS-COV2 virus detection on Day 1, Day 4, Day 7 and Day 14. In the manuscript, results are presented for Days 3, 4, 5 and 6. This is inconsistent. The one day that is consistent is Day 4, where absence of virus is reported in 12/20 treated patients, and 4/16 of the control group (which representation is incorrect, since tests were not performed in all patients - see #9). When we add back the four patients who deteriorated in the treated group, this would become 12/24 vs 4/16, which, even if the comparison was between prognostically similar groups, would be too small a difference to base a conclusion on (p=0.18). Moreover, the authors must have had information on some patients who did reach the prespecified timepoint of seven days of follow-up, and it is unlikely that this was a substantially lower number than at Day 6. There will be a, albeit probably much smaller number of patients who reached 14 days of follow-up. The data at these time points should have been presented, since now it seems the authors chose a convenient time point to present the data, in a breach of protocol.
• 7.No attempts have been made to make any adjustment, do any sensitivity analyses, or account for missing data, so the level of sophistication does not go beyond raw percentages, a Fisher exact test and a Student t- test. Some attempt could have been made to estimate the age-effect, to capture differences in symptoms and duration of them, etc etc.
• 8.In the Supplementary material on the journal's website, it is stated that several patients were “asymptomatic”, four in the control group and two in the treated group. Yet, the manuscript describes this as a study in hospitalised patients. It seems unlikely that asymptomatic patients were admitted to hospital.
• 9.In the figure on azithromycin, it seems all patients who received azithromycin started this treatment at admission. It is more likely that, at least in some patients, this treatment was added somewhere in their clinical course.
• 10.The Supplementary material also shows that many data points were missing, including date of onset, but especially on the outcome of interest: virus presence in nasopharyngeal swabs. In the 16 controls, out of a total of 6 (days) * 16 is 96 swabs, the test was not done in 43 of those. In fact, on day 6, on which the report focuses, swabs were not performed in five of the 16 controls, and one of the treated patients. Under table 2 in the manuscript, it is stated that data points were missing particularly in controls, and that they were considered positive if they were positive the day before. This was actually true for three out of five with missing data on day 6, but two also had missing data on Day 4. Hence, the figures in table 2, that 2 out of 16 controls were negative at day 6, is a gross misrepresentation. In addition, the manuscript states that patients were included if the PCR was positive at admission. For two control patients, the test was not done at admission (Supplementary Table), so these should not have been included.

Conclusion

This is a non-informative manuscript with gross methodological shortcomings. The results do not justify the far-reaching conclusions about the efficacy of hydroxychloroquine in Covid-19, and in the view of this reviewer do not justify any conclusion at all.

Declaration of Competing Interests

The author has no competing interest relevant to this paper.