The RECOVERY trial: An analysis and reflection two years on

Dear Editor,

The RECOVERY trial led the way in scientific advances during the initial stages of the coronavirus disease 2019 (COVID-19) pandemic with regards to changing clinical practice [1,2]. This was a result of, among other things, a well-designed and simple trial protocol that allowed easy implementation during a time when healthcare was most vulnerable [2], [3], [4]. We will outline the key reasons that led to RECOVERY having success as well as some things that may need further evidence or consideration while interpreting the results.

The RECOVERY trial is a multi-center randomized control trial with an adaptive platform design. The statistical analysis was performed from a frequentist perspective. The published trial that we reference is the arm focused solely on dexamethasone [1,2]. The adaptive platform design of this trial is mostly left out of the reporting. The RECOVERY group is testing multiple treatments at the same time in different study arms that are not included. The adaptive elements of this trial include the ability to change the treatment as well as sample size reassessment [1,2]. There is no use of response adaptive randomization or adaptive enrichment reported [5].

The primary outcome is all-cause mortality at 28 days. Secondary outcomes included a requirement for intubation and ventilation/extracorporeal membrane oxygenation (ECMO) and time until discharge. Statistical analysis was done on an intention to treat basis. Patients were randomized to standard care vs standard care and dexamethasone.

One of the key aspects in trial design that led to the success of the RECOVERY trial is the use of a simple and effective primary outcome measure [2]. This outcome, all-cause mortality at 28 days, had some important aspects that led to the success of the trial [6,7]. Firstly, the outcome was important for patients and clinicians in making choices regarding treatment [4,6]. There was very little evidence available at this time and anything that reduced the mortality rate significantly was going to have an important role in changing clinical practice. Secondly, the property of the outcome itself had good statistical properties for analysis [8]. The mortality rate at 28 days is easily manipulated and portrayed into easy-to-understand data for decision-makers. It is not going to lead to confusion or misinterpretation, as would be possible with a surrogate marker or composite outcome [7].

This choice of the outcome can be easily compared to the SOLIDARITY trial [9]. The researchers used in-hospital mortality as the primary outcome. This was not broken down into 28-day mortality like RECOVERY and they had stopped recording data as soon as the patient was discharged. The outcome used in RECOVERY had some noticeable benefits. Firstly, it allowed data to be collected more efficiently. As well as this it gave clinicians more clinically relevant data points in knowing that this outcome was relevant beyond admission if they were discharged before day-28.

The second aspect that led to the success of this trial is the use of a multi-centered approach with a wide inclusion criterion [2]. This allowed a large number of patients to be recruited with remarkable efficiency. RECOVERY had access to 176 different NHS sites across the UK [1,2]. The inclusion criteria are simple, a confirmed or clinically suspected case of COVID-19, that requires hospital admission and has no medical history that will put the patient at risk if they participate. This allowed for recruitment of a total of 11,303 in the platform trial with 6425 assigned to Dexamethasone vs standard care. This was achieved between March 2020 and June 2020 and concluded after an interim analysis. We will discuss more about this and the sample size below.

In comparison to trials that were being run at the same time, looking at agents such as hydroxychloroquine, vitamin D, and ivermectin, the RECOVERY trial led the way in trial design. These trials unfortunately did not make use of a multi-centered RCT approach and had low recruitment numbers leading to underpowered trials and were unable to show any true treatment effect. A database of COVID-19 trials has been set up by the world health organization (WHO) and here we can see that around 50% of registered clinical trials in the early phases of the pandemic, had recruited less than 100 patients.

There are some important critiques to consider when interpreting results from this trial. The first of which is the sample size. We cannot find clarity in the calculation of sample size within the published trial or the trial protocol [1]. From an adaptive trial design point of view, within the trial protocol or statistical analysis plan, there is no mention of how exactly the trial will be stopped. It appears to be based on a risk difference, yet we can see no further mention of this. The outcome is subsequently reported as a Hazards Ratio which we found quite confusing to understand the switch in nomenclature. Importantly, the authors do not write in concordance with the CONSORT guidelines for an adaptive design [10]. There may be very good statistics behind their adaptive design and stopping for efficacy, but if it is not reported, then it leads to a level of uncertainty when interpreting these results.

We are unable to see any reporting of a clear control of the type one error rate. Multiplicity is a potential issue in this trial due to the use of subgroup analysis [8]. Given that this trial is analyzed using frequentist statistics, there should be a clear method for controlling the Type 1 error rate. Even if the trial was analyzed using the Bayesian framework, there should be mention of using simulations to illustrate the control of the Type 1 error [5]. The authors have suggested that there could potentially be harm in the subgroup that received dexamethasone, but who were not on any ventilatory support. To us, this seems unreasonable given this was not what the trial was powered to show. As the analysis is performed from a frequentist approach, there is always a possibility that this is merely a chance finding due to the patients in the control group performing poorly “by chance.” Standard protocols now advise COVID-19 patients not requiring oxygen to not be prescribed steroids. We do feel this may require more robust evidence before a strong conclusion like this is made.