The relationship between pragmatism, timing, and study size on impact of randomized trials: a qualitative, hypothesis generating study of trials of systemic corticosteroids for COVID-19

Abstract

Objective

To explore qualitatively the relationship between selected trial design choices and proxies for a scientific and clinical uptake in a cohort of published randomized controlled trials (RCTs) of corticosteroids for COVID-19, to identify design characteristics that may result in trials with potential to eliminate equipoise, achieve uptake, and help reduce research waste.

Study Design and Setting

A systematic literature search and qualitative, narrative review of published RCTs (up to April 13, 2021) evaluating the effectiveness of systemic corticosteroids in treatment of COVID-19. We extracted information on sample size, number of centers, single-country or multi-country conduct, dates of initiation and of publication, risk of bias and pragmatism scores, and also on an impact measured by citation in scientific literature and in clinical guidelines. We qualitatively compared design features of the highest impact vs. other trials.

Results

Randomised Evaluation of COVID-19 Therapy (RECOVERY) was by the most impactful of the seven eligible RCTs as it was 10 times more frequently cited in peer-reviewed literature and influenced all the selected COVID-19 treatment guidelines. All trials started recruiting from similar dates. RECOVERY was a single-country, multicentre platform trial at low risk of bias, features which also fail to distinguish it from the other trials. RECOVERY was distinguished by more strongly pragmatic design features, more centers, and more rapid recruitment resulting in a larger sample size and early publication.

Conclusion

Higher pragmatism scores may contribute to recruiting more centers and more rapid recruitment of patients at each center, leading to larger size, earlier publication, and greater scientific and guideline uptake. By eliminating equipoise, RECOVERY rendered other simultaneous trials redundant. Further work is needed to confirm these findings in a larger quantitative study and to identify the individual contribution of each characteristic of pragmatism to conduct and impact of trials and their interaction in different national contexts. Until then, research waste might be reduced by designing trials with as many of the characteristics of RECOVERY as is feasible.

What is new?

Key findings

• •Randomised Evaluation of COVID-19 Therapy is a large multicenter, single-country, platform randomized trial of several treatments repurposed for COVID-19 care. The subtrial of corticosteroids had a low risk of bias and highly pragmatic design features that facilitated a wide implementation and rapid recruitment. This trial was cited 10 times more often than the next most cited trial and relied on all the prominent guidelines we reviewed, changing clinical practice globally. It eliminated equipoise, rendering redundant the other simultaneous trials of steroids that lacked one or more of these features. This research waste should be reduced.

What this adds to what was known?

• •Large, pragmatic, unbiassed, single-country platform trials of repurposed drugs and interventions, covering different potential pandemic conditions and multiple treatments, for a range of sociodemographic situations and healthcare capacities could be a valuable investment in readiness for future pandemics, resulting in trials with a greater scientific and clinical impact and less research waste.

What is the implication and what should change now?

• •Generic protocols for trials aimed at each kind of pandemic threat could be prepared, in advance, by a multinational consortium of research agencies and public health bodies. Each participating country team could adapt the generic protocol and maintain preapprovals from their ethics and health system committees in their own country. This could ensure rapid publication of a set of trials, tailored for local applicability, designed prospectively for metanalysis, and could speed study closure, saving lives and eliminating the research waste arising from a flurry of uncoordinated trials.

1. Introduction

The novel SARS-CoV-2 (COVID-19) virus, detected in November of 2019, has quickly caused a global pandemic with a huge public health burden [1]. Because of the long timeline for producing new tailored pharmaceuticals, many researchers focused on repurposing existing medications for treatment of COVID-19 infection [2,3]. From November 2019 to December 2021, more than 2,500 interventional trials were registered on ClinicalTrials.gov evaluating interventions for COVID-19 [4]. Fatality from COVID-19 infection appeared to be partly an immune system over-reaction and thus trials of anti-inflammatory treatments, such as corticosteroids, were a priority [4,5]. Corticosteroids have previously been used to mitigate severe organ injuries in other viral pneumonias, but the initial recommendations for using corticosteroids for COVID-19 were uncertain as small-scale, nonrandomized studies demonstrated contradictory results [[6], [7], [8]]. This uncertainty drove some to use more rigorous methods to assess the effectiveness of steroids for COVID-19 [2,3]. Owing to their high internal validity randomized controlled trials (RCTs) are considered the gold standard among designs for evaluation of interventions in healthcare [9]. The basis of an RCT is random assignment of participants into experimental and control groups with allocation concealment which, within the bounds of chance, helps to balance the characteristics of the groups between arms at baseline. Unless biases arise during the trial, the outcomes are attributable to differences in intervention [10].

The urgent circumstances of the COVID-19 pandemic provides a unique opportunity for our examination of design determinants of trial impact: a large number of RCTs on the same drug treatment for a single disease, thus likely to share a common effect size, all trials conducted at the same time, in different health systems but all under similar pressure, and with urgent demand for clinical guidance, allowing the impacts to be measured by citations and incorporation into rapidly produced national and international treatment guidelines. This allowed us to assess how trial design features contribute to impact, reducing the differences in disease, drug, timing, or other factors that would be common in comparisons between RCTs. In this hypothesis-generating study, we analyze published RCTs of corticosteroid use for COVID-19 management to identify design features that may explain their clinical and scientific impact, with the aim of guiding a future trial design.

We assessed study size, number of centers, single-country or multi-country conduct, dates of initiation and of publication, and known predictors of impact [11]. We also assessed two measures not previously shown to predict impact, risk-of-bias 2 (RoB-2) scores, and PRagmatic Explanatory Continuum Indicator Summary 2 (PRECIS-2) scores. These are widely used measures of internal and external validity, respectively, the two main vulnerabilities in trial design, which we believe may affect an uptake of RCT findings. Each trial was assessed on its scientific impact and clinical impact, determined from the number of article citations generated and number of guideline citations, respectively.

2. Materials and methods

2.1. Methodology overview

This study was based on a systematic search to ensure all relevant trials were included; the analysis of the included studies was qualitative, comparing characteristics of the highest impact trial with less influential trials.

2.2. Search strategy

A systematic review of the literature was performed by searching the databases MEDLINE, Embase, Scopus, and Cochrane up until April 13, 2021. The search strategy included database-specific keywords and Medical Subject Headings terms (Appendix A1). Studies were limited to those performed on human subjects. No limitations were placed on the publication date, language, or geographic location.

2.3. Inclusion/exclusion criteria

Studies had to employ an RCT design to investigate the use of one or more systemic corticosteroids to treat a COVID-19 infection or a COVID-19–induced condition in human participants. Studies that were not randomized or involved nonhuman participants were excluded. Platform trials testing several interventions were included if one or more of these was a systemic corticosteroid.

2.4. Study selection and screening

The studies obtained through database searches were imported into Covidence Systematic Review Software (Covidence, Veritas Health Innovation, Melbourne, Australia) [12]. The systematic screening process was performed independently by three reviewers (K.D.C., D.D., and A.L.) which involved an initial title and abstract screening followed by full-text screening (Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram, Appendix B, Fig. B.1). Disagreements were resolved by consensus meetings.

2.5. Data extraction and consensus generation

Standardized data extraction was completed independently by three reviewers (K.D.C., D.D., and A.L.). The extracted data were author(s), dates of trial initiation and of publication, the number of trial participants, and the location and number of centers. Internal validity was assessed using the Cochrane RoB 2.0 tool [13] and external validity was quantified using the PRECIS-2 tool [14], by all three reviewers. Discrepancies were resolved through consensus.

2.6. PRagmatic Explanatory Continuum Indicator Summary 2 (external validity)

The second version of PRECIS-2 [14] tool was used to assess the trial design. The scoring system of PRECIS-2 is composed of nine domains: Eligibility, Recruitment, Setting, Organization, Flexibility (delivery), Flexibility (adherence), Follow-up, Primary Outcome, and Primary Analysis—scored from 1 (very explanatory) to 5 (very pragmatic). All three reviewers independently scored each included RCT using the PRECIS-2 tool. Any discrepancies in scores for a PRECIS-2 domain were resolved during a consensus meeting with the codeveloper of the tool (M.Z.). The final consensus score for each domain was used to generate the PRECIS-2 wheel for each included study using http://precis-2.org/.

2.7. Risk of bias assessment (internal validity)

The second version of the Cochrane RoB-2 tool for randomized trials [13] was used to assess the risk of bias in all studies analyzed in this article. Included studies were assessed independently by all three reviewers (K.D.C., D.D., and A.L.) and disagreements were resolved by consensus meetings. RoB-2 assesses bias in the following five domains: (1) risk of bias arising from the randomization process, (2) bias due to deviations from intended interventions, (3) bias due to missing outcome data, (4) bias in measurement of the outcome, and (5) bias in selection of the reported result [15]. An algorithm consisting of a series of signaling questions leads to an assignment of “high risk,” “some concerns,” or “low risk,” for each domain and an overall risk-of-bias judgment.

2.8. Assessment of impact and importance

Completed trials were ranked based on both their clinical impact and scientific impact. A scientific impact was indicated by the number of “cited by” articles on PubMed at the time of data extraction, whereas a clinical impact was defined by the number of major national and transnational clinical guidelines for COVID-19 that cited the trial findings. To obtain these rankings, information regarding the number of citations in PubMed and citation in prominent national and international clinical guidelines for COVID-19 (European Medicines Agency [16], United Kingdom National Health Service [17], United Kingdom National Institute for Health and Care Excellence [NICE] [18], United States National Institute of Health [19], and World Health Organization [WHO] [20]) were extracted for each study.

3. Results

3.1. Search results

Following the completion of the systematic search and the removal of duplicates, 443 unique articles were identified and screened. After level 1 screening, 58 studies progressed to a full-text screen after which 16 articles remained. Interobserver agreement was good. Cohen's kappa (Κ) coefficient for screening and a full-text review of 0.72 and 0.87, respectively. Of these 16 studies, only seven had been published in peer-reviewed journals and were eligible for this analysis. The literature search is summarized in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram (Appendix A, Fig. A.1).

3.2. Trial characteristics

We extracted data from seven studies: Horby et al., the Randomised Evaluation of COVID-19 Therapy (RECOVERY) trial [21], Angus et al., the REMAP-CAP trial [22], Tomazini et al., the CoDEX trial [23], Dequin et al., the CAPE COVID trial [24], trial by Edalatifard et al. [25], trial by Jamaati et al. [26], and trial by Tang et al. [27]. The characteristics of the seven selected studies are provided in Table 1 (ordered by study size).

Table 1.

Extracted characteristics for included articles

Study, centers, country	Patient recruitment	Publication date	Size (N)	Pragmatism: 8 domains/40a	RoB
Horby et al. 2021 [21] 176 centers, United Kingdom	March 19 to June 8, 2020	July 17, 2020 (Preliminary report) February 25, 2021	6,420	39	Low
Angus et al., 2020 [22] 121 centers, International	March 9 to June 17, 2020	September 2, 2020	614	30	Some
Tomazini et al. 2020 [23] 41 centers, Brazil	April 17 to June 23, 2020	October 2, 2020	299	21	Some
Dequin et al. 2020 [24] 33 centers, France	March 7 to July 3, 2020	September 2, 2020	149	28	Low
Edalatifard et al. 2020 [25] two centers, Iran	April 20 to June 20, 2020	September 7, 2020	68	33	Low
Jamaati et al. 2021 [26] one center, Iran	March 2020, for 28 days	February 16, 2021	50	34	Some
Tang et al. 2021 [27] seven centers, China	February 14 to March 31, 2020	January 22, 2021	86	29	Some

^aEight of the nine PRECIS-2 domains were scored; flexibility (adherence) was excluded as the drug was administered in hospital with no patient adherence component.

3.2.1. Recruitment and publication dates

Based on participant recruitment, Tang et al. was the earliest trial and began recruiting on February 15, 2020. Jamaati et al., Horby et al. (the RECOVERY trial), Dequin et al., and Angus et al. all started recruitment in March of 2020 followed by Tomazini et al. and Edalatifard et al. in April of 2020. The start dates, unsurprizing given the pandemic, were within one or twomonths of each other, but Horby et al. published their preliminary report first, in July 2020, followed by Angus et al., Dequin et al., and Edalatifard et al., in September 2020., Tomazini et al. in October 2020 and Tang et al. and Jamaati et al. in January and February of 2021, respectively. The final result from Horby et al. was also published in February 2021.

3.2.2. Size and centers

Horby et al. (176 centers in the United Kingdom, n = 6,420, 36 patients per center) recruited by far the most participants (10 times more patients and seven times more patients per center than the next largest trial, by Angus et al. (121 centers in Australia, Canada, France, Ireland, The Netherlands, New Zealand, the United Kingdom, and the Unites States of America, n = 614). The remaining five trials were single-country and much smaller; Dequin et al. (33 centers in France, n = 149), Edalatifard et al. (two centers in Iran, n = 68), Jamaati et al. (one center in Iran, n = 50), Tang et al. (seven centers in China, n = 86), and Tomazini et al. (Brazil, n = 299).

3.2.3. Internal and external validity

Internal validity was assessed with the Cochrane RoB 2.0 tool. Dequin et al., Edalatifard et al., and Horby et al. each had a ‘low’ overall risk of bias score. The trials by Angus et al., Jamaati et al., Tang et al., and Tomazini et al. were classified as having ‘some’ risk of bias (Appendix B, Table B.1).

Eight of the nine domains of PRECIS-2 were scored: flexibility (adherence) was not scored as these trials tested a hospital-administered drug with no patient adherence component. Eligibility, recruitment, setting, organization, flexibility (delivery), follow-up, primary outcome, and primary analysis were independently scored for each trial. The final consensus scores between all three reviewers are shown in Appendix B, Table B.2 and the associated PRECIS-2 wheels are shown in Figure 1 .

Fig. 1.

PRECIS-2 score wheel.

The RECOVERY trial (Horby et al.) had the highest overall PRECIS-2 score (most pragmatic design features) with even distribution across all domains, suggesting that the designers employed a consistently pragmatic approach to their trial design. In contrast, other trials all had at least one domain that scored 3 or less, suggesting a less consistent intention toward pragmatism.

3.3. Assessment of impact

The impact of each trial was assessed based on its scientific impact and clinical impact, which were determined from the number of article citations generated and number of guideline citations, respectively. Extracted values can be found in Table 2 .

Table 2.

Extracted characteristics indicative of impact for included articles

Study	Corticosteroid	Citationsa	Influenced guidelines
Horby et al. 2021 [21]	Dexamethasone	1,887	EMA, NHS, NICE, NIH, WHO
Angus et al., 2020 [22]	Hydrocortisone	130	NHS, NICE, WHO
Tomazini et al. 2020 [23]	Dexamethasone	188	NHS, NICE, WHO
Dequin et al. 2020 [24]	Hydrocortisone	75	NHS, NICE, WHO
Edalatifard et al. 2020 [25]	Methylprednisolone	35	NICE
Jamaati et al. 2021 [26]	Dexamethasone	1	-
Tang et al. 2021 [27]	Methylprednisolon	2	-

EMA, European medicines agency; NHS, National health service; NICE, National institute for health and care excellence; NIH, National institute of health; WHO, the World health organization.

^aPubMed was used to identify citations for all trials.

As of March 27, 2021, the trial by Horby et al. was cited by 1,887 other articles, which was more than 10 times as many citations as the next most cited RCT included in our study, thus establishing it as the most important article based on this marker. The trials conducted by Tomazini et al. and Angus et al. were the second and third most cited trials, respectively, each with more than 100. Next, the trial by Dequin et al. and Edalatifard et al. had 75 and 35 citations, respectively, followed by Tang et al. and Jamaati et al. with two and one citations, respectively.

The trial by Horby et al. was the only trial that influenced all five selected guidelines; hence, its designation as the most impactful trial by this measure. The trials conducted by Tomazini et al., Sequin et al., and Angus et al. were referenced only by three treatment guidelines, published by the National Health Service, NICE, and WHO. The trial conducted by Edalatifard et al. influenced the NICE treatment guideline. Finally, the trials conducted by Jamaati et al. and Tang et al. 2021 were not cited by any of the selected treatment guidelines.

Based on these criteria, the RECOVERY trial (Horby et al.) was the most impactful (Appendix B, Table B.3).

4. Discussion

4.1. Conclusion

We assessed start of recruitment, publication timing, and study size, all of which are known predictors of impact [11] and easy to extract. We also assessed RoB-2 scores and PRECIS-2 scores, not previously reported as being associated with impact. We used the number of major guidelines influenced (as a proxy for clinical impact) and number of literature citations (as a proxy for scientific impact) because they are straight-forward, easily accessible, and accurate measures of the influence of a trial on scientific and clinical decision-making.

The principal finding of our study was that the RECOVERY trial by Horby et al. was dominant in impact compared to any of the other six: RECOVERY was the only trial mentioned in all of the five guidelines reviewed and had 10-fold more scientific citations than the next most cited. It was uniquely strong in level of pragmatism, large sample size, number of centers, and early publication but was similar to others in start date for recruitment and risk of bias.

Sample size is known to affect impact [[28], [29], [30]], also supported by our findings. Timing of publication is critical as early publication of RECOVERY's preliminary finding resulted in a loss of equipoise and early termination of the corticosteroid arm in Angus et al. (REMAP-CAP) [22], Tommazini et al. (CoDEX) [23], and Dequin et al. (CAPE COVID) [24], with research waste, a problem in biomedical research [[31], [32], [33], [34]].

All these trials started within weeks of each other, so early start-up could not have advantaged RECOVERY but is obviously necessary for early publication. Four trials were at “low” or (next category up) “some” risk (three trials) for bias, a truncated range suggesting that thorough understanding of bias prevention is now widespread and while necessary for interpretability is not sufficient to ensure early publication and impact. Similarly, all but one trial had multiple centers, necessary but not sufficient to rapidly recruit large numbers of patients.

This suggests that cooccurrence of the unique features may be responsible for the higher impact of this trial. We therefore hypothesize that the differences between RECOVERY and the other trials that may have enabled rapid recruitment and its early publication lie in its pragmatic design features (inclusive recruitment, common clinical outcomes, no restrictions on usual clinical care other than randomization, minimal monitoring, usual care comparators, and minimal extra data collection beyond what is needed for clinical care). These features underly a simple trial design, with no distortion of the usual clinical flow and needing few extra resources, which was easier to integrate into the everyday clinical context. This in turn allowed more centers to join the RECOVERY trial and improved recruitment yield per center, leading to faster recruitment and thus early publication. High levels of pragmatism may also have improved confidence in the applicability of the trial findings and added to an uptake of RECOVERY findings into clinical guidelines [35].

Of course other issues may also have contributed to the greater impact of RECOVERY. The value of large platform trials is supported by reviews analyzing existing RCTs for nonepidemic conditions such as vitamin D for infectious diseases [36] and can also be seen in the protocol of the BEAT-CF platform trial of multiple treatments for exacerbation of cystic fibrosis [37]. The platform design of the RECOVERY trial allowed simultaneous testing of multiple other COVID-19 treatments, which eliminated the need to set up separate RCTs for each one [11], thus adding to the attractiveness of the unified single-funding proposal. However, this design feature does not directly contribute to the impact of the trial in relation to corticosteroids themselves, although it undoubtedly reduces the cost and effort for testing each of the interventions in the platform. Platform trials should thus be used where possible, with the caution that they depend on sophisticated statistical skills in both design and analysis. Where the scarcity of this level of statistical skills makes this unfeasible, simpler parallel arm pragmatic, multicenter RCTs will still achieve worthwhile findings quickly, albeit one intervention at a time (or a few, in multiarm trials).

4.2. Recommendations

It took 4 months after the first acknowledgment of the pandemic for RECOVERY to be launched and a further 4 months for that trial to report its initial findings, which changed medical care outcomes for hospitalized patients with COVID-19 around the world. Had this trial been launched on recognition of the pandemic, 4 months of death and disability for many patients could have been avoided. How might we eliminate that 4 month delay in providing evidence-based care for the next pandemic illness?

Some might argue that for a question as important as care of patients in a pandemic, where many thousands of very ill patients would be cared for based on evidence from a single trial, confirmation in one or two other independent trials is reassuring, especially if conducted in different settings. We agree but note that even if equipoise is not lost, forcing their closure, these other trials might be poorly designed and contribute only low-quality evidence that confuses clinical decision-making [38], as seen with trials of ivermectin for COVID-19 whose results did not impact clinical management [39].

We therefore recommend that for epidemic and pandemic situations, where evidence-based decision support is urgently needed, important choices between interventions should be made using multiple, simultaneous, large, multicentre pragmatic randomized trials using shared protocols. Although ethics and logistics procedures are difficult to coordinate between countries, we recommend that several such trials should always be launched simultaneously, each conducted by a separate team in their own country. Organizations with international reach, such as WHO and the Gates foundation could facilitate the preparation of shared protocols with each participating country offering different socioeconomic and healthcare delivery systems. Prospectively planned meta-analyses combining these studies could produce evidence applicable to a wide range of settings, with subgroup analyses that would answer important secondary questions. These protocols should be developed cooperatively, with many shared elements, especially pragmatic features such as use of inclusive inclusion and subgroup definitions, common clinical case definitions, reduced data collection through reduced monitoring, clinically ascertainable and/or widely available test measures, simple primary outcomes, usual care comparators, and simplified trial procedures.

The protocols should be prepared in advance of any pandemic, with several generic protocols, each appropriate for one of the expected kinds and routes of spread of pandemic illness. These generic protocols, each specific to an expected type of pandemic, could be prepared, maintained, and updated centrally, working closely with each national team. In the event of a pandemic, the most appropriate generic protocol could be centrally adapted to the specifics of the actual pandemic agents, adapted in each country to their own needs and launched in several countries simultaneously, early in the pandemic. A pragmatic approach to design of these trials may avoid the usual ponderousness of research and rapidly inform global clinical practice in a pandemic [40].

4.3. Study strengths and limitations

When a pandemic arises, which interventions should be evaluated? The RECOVERY trial provided a good model by studying repurposing of widely used and readily available medications that could be easily accessed by most health systems all over the world [41].

There are several strengths to our study. First, our screen yielded a high Cohen's kappa coefficient suggesting high inter-rater reliability and minimal risk of selection bias [42]. Another strength is that we compared trials examining the same treatment for the same indication during the same time period, in a global pandemic panic. Therefore, we were able to hold constant many factors, such as whether or not the study yielded positive results, different treatments, different time periods, different health system and disease contexts, and focus only on trial design characteristics.

Our study has limitations. Owing to the limited number of clinical trials examining the effect of corticosteroids in COVID-19 patients, our study is a small qualitative analysis rather than a large meta-analysis, which limits us to hypothesis generation rather than causal attribution [43]. As well, we chose as our proxy for clinical impact the number of prominent national or international guidelines influenced but ideally we would have measured this directly with prescribing data or interviews with clinicians. Unfortunately, prescribing data are only available with a substantial time lag and interviews were not within the scope of a student project.

Overall, our finding of the importance (and interaction) of pragmatism, size, and timing points to trial design characteristics that future trial makers might find helpful to maximize the clinical and policy utility of their trials. Future research on this question should use more definitive outcomes for impact, should quantify the relationship between impact and potential study design correlates, and study this question also for nonepidemic situations [37].