Delivering rapid, up-to-date, high-quality evidence is feasible during health emergencies: PAHO living systematic review of 305 COVID-19 potential therapeutics

ABSTRACT

Objective.

To develop a living systematic review to deliver continuous, real-time evidence synthesis in the context of a rapidly evolving landscape of studies on potential therapeutic interventions during the COVID-19 pandemic.

Methods.

The living systematic review was conducted using the COVID-19 Living Overview of Evidence platform, which aggregates studies from more than 40 sources, including electronic databases and preprint servers. Daily searches identified randomized controlled trials assessing pharmacological interventions for COVID-19. Meta-analytical pooling was applied to derive precise effect estimates, and the GRADE framework was used to assess certainty. The iterative process ensured the continuous integration of new evidence and rapid updates to the review.

Results.

The review evaluated 305 interventions across 924 randomized controlled trials and included 48 updates from its launch in April 2020. This dynamic process allowed the team to respond promptly to decision-maker queries and deliver reliable information on intervention effectiveness and safety. The outputs of the review supported the development of therapeutic guidelines and informed decision-makers, playing a pivotal role in shaping clinical practices and public health strategies during the pandemic.

Conclusions.

The living systematic review approach demonstrated how dynamic evidence synthesis can meet the demands of a rapidly evolving global health crisis. By providing decision-makers with timely, high-quality evidence, the process underscored the importance of integrating living reviews into preparedness strategies for future public health emergencies or rapidly evolving fields where new evidence emerges quickly.

The COVID-19 pandemic has presented an unprecedented global challenge, prompting an urgent need for effective therapeutic interventions to mitigate its impact on public health. As the scientific community raced to uncover treatments that could prevent the infection, alleviate the severity of symptoms, and improve patient outcomes, the sheer volume of information generated proved overwhelming to decision-makers (1). It became clear that novel mechanisms to identify, assess, and summarize the huge amount of emerging information were necessary (2). Although, these methods and processes need to be able to deliver fast answers, they should remain aligned with the fundamental principles of evidence-based decision-making (3).

In the context described, conventional systematic reviews, conducted at a single point in time, quickly became outdated as new studies were published every day. To address this challenge, the concept of a living systematic review (LSR) has emerged, offering a continuous and iterative process of evidence synthesis that allows for real-time incorporation of new research findings (4, 5).

In this article we describe the process and present the results of an LSR, emphasizing its dynamic evidence synthesis capabilities, conducted by the Pan American Health Organization (PAHO) since almost the beginning of the COVID-19 pandemic. During that period, we successfully and continuously summarized the evidence on COVID-19 therapeutic interventions, providing a reliable resource that reflected the latest advancements in COVID-19 research, facilitating informed decision-making, and promoting evidence-based practice.

MATERIALS AND METHODS

Search source

We used the COVID-19 Living OVerview of Evidence (L·OVE) platform (https://app.iloveevidence.com/covid19) to identify studies for inclusion in this review (6, 7). This platform, developed by Epistemonikos Foundation, is a comprehensive repository of COVID-19 articles that is maintained through searches in more than 40 sources, including electronic databases, preprint servers, trial registries, and other resources relevant to COVID-19 (7–9). The latest version of the methods, the total number of sources screened, and a living flow diagram and report of the project are updated regularly on the methods page of the L·OVE website (10).

Search strategy

We retrieved all the articles classified as randomized trials in the COVID-19 L·OVE platform. The methods used by this platform to classify articles are described in the methods page of its website (10), which was last checked for this review on 30 November 2023. The searches covered the period from the inception date of each database, and no study design, publication status, or language restriction was applied (10).

Study selection

In the COVID-19 L·OVE platform, the results of the searches in the individual sources are de-duplicated by an algorithm that compares unique identifiers (database identification number, digital object identifier [DOI], trial registry identification number), and citation details (author names, journal, year of publication, volume, number, pages, article title, and article abstract) (7). Two authors independently screened the titles and abstracts yielded against the inclusion criteria. We obtained the full reports for all titles that appeared to meet the inclusion criteria or required further analysis and then decided about their inclusion.

Inclusion criteria

We aimed to find all available randomized controlled trials (RCTs) for potential therapeutic pharmacological interventions for COVID-19 with study designs that included head-to-head comparisons, or control groups with no intervention or a placebo. Target patient populations included both adults and children exposed to or with confirmed or suspected COVID-19. We focused on comparative effectiveness studies that provided evidence on patient-important outcomes: mortality, invasive mechanical ventilation, symptom resolution or improvement, infection (prophylaxis studies), hospitalization (studies that included patients with non-severe disease), and severe adverse events (11). For studies that assessed thromboprophylactic interventions we also assessed venous thromboembolic events and major bleeding. For the outcome “hospitalization” we included information from studies reporting the number of hospitalizations or the number of hospitalizations combined with the number of deaths without hospitalization.

Data analysis

We presented all the analysis with relative and absolute effect sizes. To assess interventions’ absolute effects, we applied relative effects to baseline risks (risks with no intervention). We extracted mortality and invasive mechanical ventilation baseline risks from the International Severe Acute Respiratory and emerging Infection Consortium (ISARIC) cohort as of 18 December 2020 (12, 13). For baseline infection risk in exposure to COVID-19 we used estimates from a systematic review on physical distancing and mask utilization (14), and for adverse events and symptom resolution/improvement we used the mean risk in the control groups from included RCTs until 18 December 2020. For venous thromboembolic events and major bleeding baseline risk we used the mean risk in the control groups from included RCTs until 25 March 2021. For hospitalization baseline risk we used the median risk in the control groups from included RCTs until 23 December 2021.

For result interpretations and imprecision assessment we used a minimally contextualized approach which considers whether the 95% confidence interval (CI) includes the null effect, or, when the point estimate is close to the null effect, whether the 95% CI lies within the boundaries of small but important benefit and harm that corresponds to every outcome assessed (15).

Based on the outcome rating exercise performed in the World Health Organization (WHO) clinical practice guideline for the management of patients with COVID-19 (16) we selected the following thresholds to define important benefits and harms: mortality, +/– 1%; mechanical ventilation, +/– 2%; symptom resolution or improvement, +/– 5%; symptomatic infection in exposed individuals, +/– 5%; hospitalization in patients with mild recent COVID-19, +/– 1.9%; severe adverse events, +/– 3%.

For some interventions when we found significant heterogeneity, we performed subgroup analysis considering: 1) risk of bias (high/moderate vs low risk of bias); 2) disease severity (mild, moderate, severe, or critical); and 3) intervention’s characteristics (i.e., different doses or administration schemes). When we observed important differences between subgroups, we presented individual subgroup’s estimates of effect and certainty of the evidence assessment.

We used the Cochrane 2.0 tool to assess risk of bias which focuses on randomization, allocation concealment, blinding, attrition, or other biases relevant to the estimates of effect (17). We used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to assess the certainty on the body of evidence for every comparison on an outcome basis (18). Risk-of-bias judgments were performed in parallel by two reviewers or compared against other similar projects (Drug treatments for covid-19: LSR and network meta-analysis; The COVID-NMA initiative) (19, 20). Significant discrepancies were discussed until a final decision was reached.

We did not formally assess publication bias; however, it remained a constant concern throughout the review process, given the unprecedented speed at which studies were being conducted and published. The rapid generation of evidence created the possibility that some findings, particularly those with negative or inconclusive results, were not being made publicly available. To mitigate this risk, we employed rigorous methods to identify emerging evidence comprehensively. These included systematic searches across multiple sources, including trial registries and preprint servers, ensuring that we captured studies at various stages of dissemination.

We used the MAGIC (Making GRADE the Irresistible Choice) authoring and publication platform (https://app.magicapp.org/) to generate the tables summarizing our findings, and developed a website to present the results of the review in a user-friendly and easy-to-navigate format (https://covid-therapy.bvsalud.org/).

Living evidence synthesis

The automated notification system available in the COVID-19 L·OVE platform provided instant notification of articles with a high likelihood of being eligible. The authors reviewed them, decided upon inclusion, and updated the living web version of the review accordingly. After selecting the new studies to be included in the review, we performed meta-analytical pooling, when possible, to derive more precise estimates of effect and gain additional statistical power. The entire process was conducted on a daily basis, ensuring the review incorporated the latest available evidence in real time. By rapidly integrating new studies, the team was able to provide the most up-to-date estimates of effects, enhancing the precision of results and the robustness of the synthesis.

Support to decision-makers

The LSR process was specifically designed to meet the urgent demands of decision-makers during the COVID-19 pandemic. Overwhelmed by an unprecedented volume of emerging information and the proliferation of false claims regarding intervention efficacy, decision-makers frequently submitted queries seeking reliable, evidence-based answers about whether specific interventions were effective. By leveraging the LSR framework, we were able to rapidly respond to these queries using continuously updated evidence.

RESULTS

The first version of the LSR was published online on 22 April 2020; it assessed 18 interventions and included 44 studies, of which only 13 were RCTs (21). Since then the review has been updated 48 times, with its latest version available online since 30 November 2023 (22). Figure 1 shows the evolution over time of the number of studies included in the LSR, interventions assessed, and interventions with high or moderate certainty evidence. This latest version of the review assessed 305 interventions and included 924 RCTs (Figure 2). The results of the latest version are available in an interactive format on the PAHO COVID-19 therapy portal, at: https://covid-therapy.bvsalud.org/.

FIGURE 1. Number of studies included in the living systematic review, interventions assessed in randomized controlled trial, and interventions with high or moderate certainty evidence over time.

Source : Prepared by the authors.

FIGURE 2. Identification of studies included in the living systematic review.

Source : Prepared by the authors.

Risk of bias

Results of risk-of-bias assessment for every included study are available in the latest publication of the full review (49th edition) (22). Overall risk of bias was variable, with some studies fulfilling high methodological standards but most showing important limitations. Consequently, the methodological limitations and small sample sizes, for most of the 305 assessed interventions resulted in very low certainty of the evidence, which means that their effect remains unknown.

Effects of interventions

The LSR identified several effective interventions for COVID-19 prevention and treatment while also highlighting those that were ineffective or harmful. For prevention, monoclonal antibodies (mAbs) such as casirivimab–imdevimab and tixagevimab–cilgavimab significantly reduced the risk of infection. In patients with mild to moderate COVID-19, interventions including nirmatrelvir–ritonavir and various mAbs demonstrated reductions in hospitalizations. For severe to critical cases, corticosteroids, tocilizumab, baricitinib, and other treatments reduced mortality and mechanical ventilation requirements. However, some interventions, such as ivermectin, hydroxychloroquine, and convalescent plasma, were found to be ineffective or harmful. The effectiveness of mAbs was noted to depend on circulating SARS-CoV-2 variants. Figure 3 provides a summary of these findings, while the complete results, including detailed estimates of effects and certainty levels, are available in the latest version of the review (22).

FIGURE 3. Summary of effective interventions identified in the living systematic review.

Note : * Probably not effective against some SARS-CoV-2 variants.

Source : Prepared by the authors.

Support to decision-makers

The LSR was a cornerstone of the rapid response program, playing a pivotal role in supporting decision-makers during the COVID-19 pandemic (23). Its outputs informed the development of national and regional guidelines, such as Guidelines for the management of patients with severe and critical COVID-19 in the Americas (24), Considerations for strengthening the first level of care in the management of the COVID-19 pandemic (25), and Guidelines for prophylaxis and management of patients with mild and moderate COVID-19 in Latin America and the Caribbean (26), which were adapted by countries including Argentina, El Salvador, and Peru. The living review also addressed specific decision-maker queries, such as the efficacy of mAbs and antivirals, ensuring that recommendations were both timely and context-sensitive. Knowledge translation strategies, including workshops, capacity- building sessions, and the dissemination of findings through platforms like the PAHO COVID-19 therapy portal, facilitated the rapid uptake of evidence. For instance, Argentina leveraged the living review to develop evidence summaries that guided clinical therapeutic protocols, while El Salvador adapted it to design pharmacological guidelines (23).

DISCUSSION

In this article, we elucidate one of PAHO’s responses to an unprecedented scenario during the COVID-19 pandemic, in which the imperative need for knowledge on effective therapeutic interventions collided with the rapid influx of vast amounts of information of varied quality within a remarkably short timeframe. We described the method implemented to rapidly synthesize the scientific information and summarized our most important findings. A complete description of the systematic review’s findings is available online (22).

The essence of an LSR lies in its ability to advance alongside the evolving body of evidence. Traditional systematic reviews are often constrained by the lengthy publication timelines and periodic update schedules, which may result in outdated information (4, 5). In contrast, our review integrated new findings in real time by employing methodological shortcuts, thus saving time and supporting a continuous and iterative process.

The findings of our LSR were largely consistent with those reported in other systematic reviews and clinical practice guidelines, including the WHO living guideline (16). However, a key advantage of our approach was its ability to anticipate these results, often by weeks or even months. By continuously integrating new evidence in real time, our review provided early signals regarding the effectiveness or ineffectiveness of interventions, addressing the urgent need of decision-makers for timely, well-synthesized evidence to guide clinical and public health responses in a rapidly evolving pandemic landscape. Additionally, unlike other reviews that focused on specific interventions or clinical scenarios, our LSR assessed all interventions investigated in at least one RCT, ensuring a more comprehensive evaluation of the evolving evidence base.

The LSR was one of the products developed by the PAHO rapid response program (RRP) that was implemented at the onset of the pandemic by the clinical management team. The RRP supported evidence-informed decision-making and policymaking by providing high-quality research evidence, contextualized and targeted to the needs of decision-makers in the Region of the Americas and in Member States (27). It was an example of how science and innovation were applied in response to COVID-19 and, although acknowledging the importance of all other elements of health care, was focused predominantly on clinical care, while reflecting the principles of equity and prioritization of vulnerable groups. A range of other products were developed from the evidence generated in the LSR, including but not limited to therapeutic and prophylaxis guidelines, additions to the essential medicines list, and evidence briefs for policy. The publication, printing, and dissemination of these products were effective in improving awareness and clinical care of patients (28). To further support the timely dissemination and uptake of the evidence, the PAHO COVID-19 therapy portal (https://covid-therapy.bvsalud.org/), a visualization tool for the LSR, was developed in 2022.

The COVID-19 response highlights the importance of institutionalizing an RRP for health policy and practice, and the role of LSRs. The level and needs of decision-makers, the setting, and available resources will influence the type of response program, noting that a close relationship with the end-user will facilitate uptake of the evidence products (28). LSRs introduce distinctive challenges, which we tackled by integrating proficient members skilled in programming and methodology into the core RRP team. Additionally, we adapted the traditional systematic review process to ensure feasibility in our case. A variety of shortcuts can be implemented in LSRs to enhance feasibility without substantially compromising certainty. These shortcuts include automating steps in the workflow, such as utilizing artificial intelligence tools for study selection, text mining, data mining, and alternative publication platforms (29–31). There remains a need for more transparency in reporting methods and in evaluating the performance and applicability of technologies used as LSRs evolve in the emergency space. Similarly, there is a focus on developing and updating living evidence maps for emergent and reemergent diseases and priority health topics (e.g., BIGG-MAP) (32) during non-emergency times, to generate recommendations and guidance for clinical practice and policy when required, in an equitable, accessible, and user-friendly manner.

One of the key strengths of our living review is its contribution to the timeliness and relevance of information available to decision-makers (33). As new studies were published and the understanding of COVID-19 therapeutics evolved, our review remained a dynamic and responsive resource. This real-time synthesis facilitated a more nuanced understanding of the effectiveness, safety, and feasibility of various therapeutic interventions, empowering healthcare professionals, policymakers, and researchers to make well-informed decisions.

We acknowledge certain limitations in our pursuit of consistently keeping our review up to date, necessitating the implementation of some shortcuts from traditional systematic reviews. To expedite the process, full-text selection and data extraction were undertaken by a single reviewer, and the robustness of risk-of-bias assessments occasionally relied on comparisons with other reviews instead of involving a second reviewer. Our focus was specifically on outcomes critical for decision-making, and it is important to note that our publication process did not include formal peer-reviewed assessments. These limitations increased the risk of errors in result interpretation or misclassification of the certainty of evidence. However, by comparing our results with other systematic reviews following more traditional processes, we corroborated that these situations were infrequent and of limited relevance.

In conclusion, the successful implementation of an LSR on COVID-19 treatments underscores its feasibility and utility as a dynamic tool for evidence synthesis. By embracing this innovative approach, we were able to continuously support evidence-based decision-making, ensuring that decision-makers had access to the most up-to-date and relevant information to navigate the complexities of treating COVID-19. Moreover, the LSR model presented here has the potential to be applied beyond emergency situations, particularly in fields where a breakthrough has been reached, leading to a rapid surge of new evidence. This could be especially valuable for assessing emerging treatments in specific diseases where ongoing research demands continuous and timely synthesis of information.

Disclaimer.

Authors hold sole responsibility for the views expressed in the manuscript, which may not necessarily reflect the opinion or policy of the RPSP/PAJPH and/or the Pan American Health Organization (PAHO).