Rational for meta-analysis and randomized treatment: the COVID-19 example

My opinion is that hydroxychloroquine has become the symbol of a struggle between practising physicians and methodologists [1], and the Western world against the rest of the world [2]. This leads to great confusion in the literature between, on the one hand, the advocates of an empirical approach based on the sensitivity of bacteria, viruses or parasites to anti-infectious agents in vitro and the rational use of these anti-infectious agents in patients, and, on the other hand, the analysts who, taking up the various studies, are more specifically interested in the form of the studies to determine the existence of biases. Recently, hydroxychloroquine, from my point of view, became a paradigm of such conflict.

For example, when testing hydroxychloroquine, exclusion of patients without confirmed diagnosis is for us a major issue. In contrast, several studies do not consider this parameter to be essential [3,4], including a large study testing the efficacy of hydroxychloroquine prophylaxis [5]. This inclusion criterion is mandatory for me. In addition, there are considerable differences in dosing regimens as there is no standard dosing regimen: the Recovery Trial gave a theoretically toxic dosage at baseline (2.4 g), others use 200 mg daily, while we prescribe, in my institute, 600 mg daily, as in Q fever or Whipple's disease [6]. Also, it is important to compare the duration of treatment. Entering hospital data with a hydroxychloroquine yes/no answer does not tell much about the treatment. These are major problems with the inclusion criteria in big data studies.

The stage of the disease at which treatment is given is critical. Two reviews explain that there are four stages in the disease [6,7]. There is a first virological stage, when the antiviral drugs can be effective, a second viro-immunological stage, when the immunological reaction aggravates the patient's condition and is associated with abnormal coagulation phenomena including anti-phospholipid antibodies [8], a third stage that is exclusively or almost exclusively immune, also called the cytokine storm, and finally a fourth stage, resulting from multiple pulmonary and visceral injuries. Thus, each stage probably corresponds to different therapeutic strategies. Finally, evaluation of the therapeutic efficiency in all infections of the lung is usually performed after 3 days of treatment. Mixing patients of different stages, with different doses and durations of treatment, may result in a false result known as “Simpson's paradox” [9]: studying separate groups of patients from the same study may result in opposite conclusions. Some French studies exclude patients who are treated while they have already been hospitalized for 2 days [10], which is difficult to understand. For sure, a key element is the delay between the start of disease and implementation of the treatment.

Finally, we need outcomes that are measurable and not too dependent on circumstances. Transfer to the critical care unit is dependent on local management and possibilities. Hospitalization is not a good criterion either. The decision to hospitalize is not based on objective criteria, but rather reflects a management strategy or bed availability rather than a clear clinical reality. These are reflections that have arisen from our experience here having followed more than 6000 people with this disease [11]. Evaluation of outcome, as in other infectious diseases, can be clinical with objective measures (death) or biological. In fact, viral loads allow better appreciation of the effectiveness of an antiviral treatment (especially in AIDS and hepatitis C) than the design of the study. Indeed, the incontestable objective factors are death and viral load, which are measurable elements and which do not (or only slightly for PCR) depend on the observer or the circumstances. Of course, they may not be linked directly, but both report on some objective efficacy of the treatment.

Thus, if we look at the practice of care using two therapeutic reference books as a reference, Conn's current therapy book [12] and the Bennet Dolin Blaser book [13], there are very few cases of treatable infections where randomized studies have been able to modify or allow the management of infectious diseases. In bacteriology, antibiotic testing drives therapeutics. In virology, viral load drives therapeutics, as for AIDS or hepatitis C. In parasitology, for therapeutics, effects (for malaria) have also been biologically measured. Thus, in contrast to the asserted dogma, there are very few examples of currently followed therapeutics in infectious diseases that have been determined by randomized double-blind studies [14]. Methods of randomized trials have dogmatically become the reference standard, but are controversial [[15], [16], [17]]. Currently randomized trials have been severely criticized including in cardiology and cancer therapy [18,19]. A recent giant review by the Cochrane Library (including 1583 meta-analysis covering 228 medical conditions) fails to show any superiority of randomized studies versus observational studies in many healthcare outcomes [14]. It is not established therefore that there is evidence of a superiority of randomized studies. The very existence of meta-analysis highlights that there are discrepancies between the different randomized studies, which proves that these studies did not eliminate biases [20]. Indeed in this journal, as for hydroxychloroquine, it was reported that one meta-analysis demonstrates no effect of hydroxychloroquine on COVID-19 infection [21] and the other the opposite [22]. However, meta-analysis exemplified the role of the sponsor in the biases, as shown by us for probiotics [23].

To evaluate observational studies, after analysing the significance of each factor, including therapy, on the outcome, it is possible to carry out multivariate analysis that tests the independence of the factors or to carry out propensity scores that should in principle neutralize a number of factors [11], or at least those that are known. Indeed, it has been well accepted for a very long time, within the framework of Simpson's paradox [24], that not all equivocal factors are necessarily recognized, and that sometimes the addition of studies that are all in favour of one therapy gives an inverse result when combined, due to a bias in the number of people included. This paradox is well known and shows that it is impossible to fight bias mathematically and that it is necessary to have knowledge of the disease and formulate a hypothesis before interpreting massive studies [11].

All in all, there is no indisputable science of therapeutic trials and their evaluation. It cannot be said that significant progress has been made in the practice of care by randomized trials in infectious diseases. They have generated a new specialty, particularly in the medical world, which is that of methodologists and analysts who, by definition, are convinced that their method is the best. In principle, over the history of hydroxychloroquine, depending on the studies that one decides to exclude, one is likely to retain one hypothesis or another.

In conclusion, there are currently nearly 100 publications available in the literature evaluating, through randomized or observational studies, or big data analyses, the effect of hydroxychloroquine on patients generating opposite results. In order to be useful, a study, whatever the mode used, must first contain only patients whose diagnosis has been formally confirmed. Secondly, the stage of the disease must be specified. The effect of drugs, in most viral diseases (such as zoster, for example), is different: at the beginning, where antivirals are effective; at the time of the inflammatory reaction, where corticosteroid therapy is effective; and at the time of necrotic lesions, where no treatment is effective.

Thirdly, the dosage of the drugs used must be clearly stated, as well as the duration of treatment.

Fourth, anti-infective treatment cannot be evaluated unless there has been an opportunity to give it for 3 days or more for drugs with a relatively short half-life.

Fifth, treatment can only be compared with data that are the same, whatever the conditions and circumstances, and that do not lead to subjective or ad hoc evaluations; then, the primary objectives must be death and disappearance of the virus.

Sixth, the patients should be compared in the high-risk groups, including age, lymphopenia and level of oxygen dependence.

These elements constitute the basis, whatever the method of inclusion chosen (randomized or not) or the method of analysis. This represents, in my opinion, the medical basis of therapeutic evaluation. Moreover, the data processing and the redaction of the paper, in my opinion, must absolutely be carried out by neutral structures, having no conflict of interest, and not by the pharmaceutical industry and be available, at the latest, 1 year after the publication of the work in order to be evaluated by the other teams.

Transparency declaration

The author declares no conflict of interest.

Funding source

No funding.

Editor: L. Leibovici