Old dog with new tricks: An introduction to real-world evidence for pharmacists

Drug regulators require pharmaceutical manufacturers to perform clinical trials to demonstrate efficacy and safety prior to market access approval. Historically, these trials have consisted of randomized controlled trials (RCTs). In the evidence hierarchy, RCTs are considered the gold standard for demonstrating efficacy and safety due to the low risk of bias and confounding when randomization, allocation concealment, and blinding are adequate. However, the strong internal validity of RCTs is obtained in place of generalizability: Strict protocols may not reflect clinical practice, target populations may be excluded, and additional comorbidities and concomitant medications can increase complexity and the risk of drug-drug interactions. Additionally, the limited length of follow-up hinders the ability to detect uncommon but serious adverse effects as well as the duration of treatment effects. The recent growth in orphan drugs and the need to test these highly specialized drugs for rare conditions have presented an opportunity for the proliferation of real-world evidence (RWE). For these medications, RCTs may simply not be practical given that the target populations may be too small to obtain an adequate sample size. RWE is in no way new and leverages classic observational study designs and methods. However, because of the limitations of RCTs and the increased need for orphan drugs for rare conditions, along with the high cost of conducting RCTs and the recent ubiquity of electronic health records and administrative data, the Food and Drug Administration (FDA) and other regulators around the world are now accepting RWE of drug effectiveness to supplement RCT evidence—a key opportunity to leverage older methods to solve new problems.^1-3 We believe that it is essential for pharmacists to know how to use and appraise RWE in order to make the best formulary and patient care decisions. In this commentary, we aim to define and describe the role of RWE, explain the current stance of regulators and payers on the use of RWE, and emphasize important tools for pharmacists to determine high-quality RWE.

What is real-world evidence?

RWE is the study of the use and potential benefits or harms of a medical product based on the analysis of real-world data (RWD).^4,5 RWD encompasses information on patient status or the delivery of healthcare that is routinely obtained from various sources such as electronic health records, medical claims, billing, product and disease registries, health surveys, home medical devices, wearable technologies, and health applications.^4,5 Both the sources of data and design of trials are diverse. RWE includes classic nonrandomized observational studies such as cohort, case-control, and case-series as well as randomized pragmatic trials where follow-up care resembles clinical practice.^4,5 Importantly, there is a strong consensus that RWE is best used to supplement, not replace, RCT evidence and can provide additional evidence of drug effectiveness and safety, additional evidence for cost-effectiveness analyses, and additional evidence on off-label medication indications, rare diseases, target populations excluded from RCTs, and situations where it would be unethical to perform an RCT.^6,7 The study designs used in RWE are not novel but consist of familiar observational study methodology, the “old dog.” However, regulatory agencies are using “new tricks” with regards to how they are incorporating this evidence into drug approval and reimbursement.

Why is real-world evidence important for pharmacists?

RWE has a long history of being used for postmarketing drug safety surveillance to identify long-term or rare adverse effects, often cited as phase 4 studies.^1,4 In 2018, an international environmental scan on the use of RWE in single-drug assessments found that non-RCT data for benefit assessments was uncommonly used by regulators or health technology assessment agencies outside of certain areas such as oncology.⁸ Similarly, a more recent international review of the use of RWD for new drug applications and additional indications found that they have so far mostly concentrated on rare diseases in oncology and metabolism, such as for the approval of cerliponase alfa, asfotase alfa, and uridine triacetate.^3,9 However, in order to improve efficient patient access to therapies, global regulators such as FDA are now examining how to optimize the incorporation of RWE across the drug life cycle.^1,4

There are many initiatives ongoing to improve this process across the drug life cycle. For example, although the scope of FDA’s use of RWE to supplement other evidence of drug effectiveness has historically been limited to the areas of oncology and rare diseases, in 2018 the agency developed a framework for its RWE program.⁴ This framework was created to evaluate the potential use of RWE for drug effectiveness research in order to support new indications for already approved drugs, use in populations not previously studied, comparative effectiveness or safety data, and postapproval study requirements.⁴ FDA also recently developed a draft guidance document for industry stakeholders on submission of RWE for drugs, outlining acceptance of RWE submissions to support the purposes outlined in its framework.² This development highlights the fact that RWE will be important in how drugs get entry into the market moving forward.

Likewise, the inclusion of RWE will have an important impact on how medications are reimbursed, both nationally and on local pharmacy and therapeutics (P&T) committees.¹⁰ A matched cohort study using registry data demonstrated that omalizumab, a humanized monoclonal antibody used for the treatment of severe asthma, did not reduce clinically important outcomes despite a significant cost.¹¹ However, the subgroup of patients on triple inhaler therapy did benefit, demonstrating that further study of specific subgroups may help to optimize cost-effectiveness.¹¹ A 2017 study in the United States found that published RWE was uncommonly referenced in P&T reviews, contributing to only 4.8% (21 of 439) of therapeutic class review references and to none (0 of 126) of the monograph references.¹² Out of the 21 RWE studies, 12 were high-quality comparative studies.¹² However, the prevalence of RWE use by P&T committees is expected to increase, and members will need to know how to use and analyze RWE to inform formulary decisions.¹³ This trend will also impact patient care decisions as the use of RWE to support drug effectiveness increases. Pharmacists in various roles will be seeing increasing amounts of RWE and will need to know how to interpret and apply this data for decision-making.

How do you determine high-quality RWE?

With the anticipated growth of RWE, skills to understand and apply this evidence will be important. When observational studies are well designed, they can provide complementary results to RCTs,^14-17 but limitations must be noted and contextualized.^18,19 When evaluating the quality of RWE, a number of appraisal tools are available.^20,21 Importantly, any appraisal of RWE should be grounded in the assessment of data quality, study design, and methods.

Often the first step of any evaluation is understanding the quality of the RWD used—and not all data is created equal. Incomplete data (for example, missing information from patient charts), constrained access to data, and lack of universally accepted methodological standards are limitations in some data.²² In contrast, for some RWD that is extensively used, such as large administrative claims databases, there are well-validated outcome measures; this can be seen as an important strength. The use of hard outcomes such as myocardial infarction, stroke, or death may improve the validity of RWE,²³ as they are less subject to bias due to lack of blinding and are often supported by validation studies. Several initiatives, such as the Observational Medical Outcomes Partnership (OMOP) Common Data Model, are ongoing as possible solutions to transform data within disparate databases into a standardized vocabulary.^24,25 Knowledge of the type of data available from the data source is pertinent in order to be mindful of the completeness of the data. Major flaws in these foundational components of the study will limit any usability—regardless of the ingenuity of the study design or robustness of analyses.

Observational studies can also be bound by inherent limitations in design. However, there are study designs that can be used to improve internal validity. Pragmatic trials have the rigor of being randomized, which aims to remove known and unknown baseline factors that can confound results.²⁶ At the same time, pragmatic trials can provide evidence on treatment effectiveness in a routine, real-life practice setting, using RWD such as electronic health records.²⁶ The use of RWD can improve the feasibility of the study, and the practice setting may improve the generalizability of results.²⁶ Furthermore, quasi-experimental study designs can be used to infer causal relationships in nonexperimental data.²⁷ In an approximate order of lowest to highest internal validity, these include ordinary regression and panel methods, matching and reweighting estimators, instrumental variables and related methods, and regression discontinuity designs.²⁷

In addition to data quality and study design, the other foundational piece to any appraisal is the assessment of the potential impact of confounding and bias. There are various methods that can be used to address confounders in RWE. Propensity scores are a common methodological and statistical solution in RWD analyses to balance confounders between groups and minimize the risk of selection bias.²³ Other methods used to minimize confounding include restriction (specifying inclusion or exclusion criteria based on presence of confounders in order to increase similarity between groups), stratification (separating groups into smaller samples based on preidentified criteria), matching (choosing criteria for a control group based on confounders to increase similarity to an index group), and other forms of statistical modeling (such as use of confounder summary scores and multivariate analyses). Furthermore, sensitivity analyses can be conducted to assess the range of effect estimates under different assumptions.²³ However, it is important to note that all of these methods are only used to address measured and known confounders. With regards to transparency, it is important that the methods used are clear and that authors share statistical codes and any assumptions made. Selection of which method to be used can vary based on the study question, sample size, and study design.²⁸

There are a number of available tools to help guide any appraisal. For example, in collaboration with FDA, the Duke-Margolis Center for Health Policy RWE Collaborative has created documents to guide the development and use, and to improve the quality of RWE.²⁹ The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) have created a task force to make recommendations on procedural practices for RWE.³⁰ There are several tools that can be used to help assess the risk of bias in non-RCT studies. The Cochrane Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) tool can be used to assess the risk of bias in nonrandomized quantitative studies evaluating the effectiveness of an intervention, such as cohort studies, case-control studies, controlled before-and-after studies, interrupted time series studies and quasi-randomized studies.²⁰ The ROBINS-I tool addresses 7 domains where bias may be introduced: The first 2 domains pertain to baseline assessment, the third domain reviews classification of interventions, and the remaining 4 domains pertain to issues after the intervention has started.²⁰ Another tool that can be used is the Newcastle-Ottawa scale for cohort and case-control studies.²¹ With regards to reporting standards, the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) statement for observational studies,^31,32 and the CONSORT (Consolidated Standards of Reporting Trials) extension for pragmatic trials can be used.³³ If needed, organizations can consider developing online training programs for RWE methodology, appraisal, and interpretation. An example includes an online training program for comparative effectiveness research methodology.³⁴ This program was created by a collaboration between the Academy of Managed Care Pharmacy, ISPOR, and National Pharmaceutical Council to strengthen the critical appraisal skills of formulary decision makers.³⁴

Conclusion.

With the ubiquity of electronic health records and administrative data and the limitations of RCTs, global regulators including FDA along with health technology assessment agencies are investigating the application of RWE, encompassing classic observational study designs, to supplement assessment of drug effectiveness. This new regulatory shift will impact how medications are approved and reimbursed. It will have a downstream effect on the availability and type of evidence that will need to be considered by P&T committees for formulary approval and by clinicians on the front lines. Pharmacists will need the knowledge and tools to critically appraise and interpret these study designs in order to make formulary, reimbursement, and clinical decisions.

Disclosures

The authors have declared no potential conflicts of interest.