The Expanding Role of Real-World Evidence Trials in Health Care Decision Making

Abstract

Real-world evidence (RWE) is the clinical evidence about benefits or risks of medical products derived from analyzing real world data (RWD), which are data collected through routine clinical practice. This article discusses the advantages and disadvantages of RWE studies, how these studies differ from randomized controlled trials (RCTs), how to overcome barriers to current skepticism about RWE, how FDA is using RWE, how to improve the quality of RWE, and finally the future of RWE trials.

Definitions of Real-World Data and Real-World Evidence

Although the terms real-world data (RWD) and real-world evidence (RWE) are frequently used interchangeably, they have different meanings. RWD are information gathered through observations of routine clinical practice from multiple sources that can be linked together to provide meaningful patterns. RWD is based on patients and their clinicians choosing treatments according to the patients’ clinical characteristics and preferences.¹

If the RWE trial design and conduct closely approximate testing the effectiveness of the product in real-world clinical practice, then the trial is known as a pragmatic trial² (as opposed to an explanatory trial, which aims to evaluate the efficacy of an intervention in a well-defined controlled setting)³ RWE is the analysis of RWD from a study designed with a high degree of pragmatism.⁴ The 21st Century Cures Act defines RWE as evidence derived from “data regarding the usage, or potential benefits or risks, of a drug derived from sources other than randomized clinical trials.”⁵ RWE represents conclusions derived from observations of patients in health care settings (rather than subjects in a research environment).

Why More RWE Is Now Available

The US health care system in the past few years has become able to generate and interpret large new sources of data from electronic health records, billing claims, disease registries, wearable devices, and patient generated data on social media. For example, 99% of hospitals across the country now use electronic health record systems (EHRs), compared to about 31% in 2003.⁶ In the US, 11% of office-based doctors used electronic records in in 2006,⁷ whereas by 2017, 90% of office-based doctors used electronic records.⁸ Every day, billions of user-specific data points are generated by wearable devices and smartphones, but over 99% of this newly created digital data remains unanalyzed.⁹ New open data policies, such as Open FDA¹⁰ and academic data sharing initiatives^11,12 are expanding the pool of searchable data for RWE investigators. New software for interpreting the emerging tsunami of data based on emerging capabilities of artificial intelligence, machine learning, and natural language processing are simplifying the tasks of combing through large databases to assess correlations between patient features, health care professional features, diseases, and treatments.

Advantages and Disadvantages of Traditional Randomized Controlled Trials

A randomized controlled trial (RCT) in medicine is an experiment whereby the subjects participating are randomly allocated to either the group receiving the intervention under investigation or to a group receiving standard treatment (or placebo treatment) as the control. This approach to assessing the benefit of a treatment is intended to reduce bias by balancing both known and unknown confounders into the two groups. In an RCT, subjects with particular traits, who are more or less likely to result in benefit from the intervention, are distributed roughly equally through random allocation into one treatment arm or the other. Randomization allows researchers to determine the effects of the intervention versus no intervention while other variables between groups are kept constant. The RCT is often considered the gold standard for evidence¹³ to evaluate the efficacy of an intervention.

RCTs have at least six disadvantages for guiding health care decision making. These include: (1) high cost, (2) long time to complete, (3) comparison against only placebo or few alternate treatments, (4) impractical eligibility criteria, (5) underrepresentation of some types of people, and (6) trial effect, which can unduly inflate the measured benefit of an intervention.¹⁴ Furthermore the patient’s underlying condition can affect the results of a RCT in way that may be difficult to account for with the randomization process.¹⁵ An example is the tendency of diabetes subjects with very elevated A1C levels to often benefit more from interventions in RCTs than subjects with only slightly elevated A1C levels.¹⁶ If only the former subjects are randomized, then the study will likely show a sizable benefit of the intervention but the results will not necessarily be generalizable for patients with diabetes whose control is not as poor as that of the trial subjects. If the study emphasizes generalizability and both types of subjects are recruited, then likely the benefit will become diluted and the trial organizers risk reaching a conclusion that the same intervention will fail to demonstrate efficacy in a diabetes population with diverse levels of glycemic control.

Advantages of RWE

RWE is derived from analyzing data collected from a health care setting that is not part of a context of an RCT. RWE trials are useful for generating hypotheses about relationships between real-world phenomena and outcomes. RWE trials can be classified as observational or pragmatic. Observational data can be defined as data generated from experience with routine medical care that has been systematically recorded, for example as electronic medical records, registries, or administrative claims in a format that can be used for research.¹⁷ In an observational study the researcher does not control the intervention being studied. Observational research provides a mechanism to determine risk factors and mechanisms of activities or interventions that could serve as potential targets in future RCTs.¹⁸ Pragmatic trials seek to answer the question of whether an intervention works under usual conditions. Randomized trials tend to be considered as either pragmatic or explanatory. Explanatory randomized controlled trials address whether an intervention works under ideal conditions.¹⁹ Features of an optimal pragmatic RWE trial are listed in Table 1.²⁰ RWE provides a different type of information than RCTs do. RWE is useful for comparing the effectiveness of various interventions in the real world, where patients are not being closely followed or encouraged to adhere to treatment like they are in clinical trials, especially when the goal of a study is to assess the durability of effect, generalizability, and long-term safety of an intervention. For low event rate endpoints, RWE can identify rare unanticipated complications better than RCTs, which generally study fewer subjects for shorter time periods. RWE studies can also provide information about real-world adherence to an intervention and the threshold of disease severity at which point the intervention is prescribed. The cost of RWE trials is much less than RCTs (especially for RWE pertaining to a long time period), and retrospective RWE studies do not present difficulties with recruitment—the appropriate subjects are already in the database. RWE is currently being used for some regulatory approval decisions, postapproval monitoring of safety signals, payer coverage decisions, and outcomes-based contracting.

Table 1.

Features of an Optimal Pragmatic RWE Clinical Trial.²⁰

1	There are no inclusion or exclusion criteria
2	Practitioners are not constricted by guidelines on how to apply the experimental intervention
3	The experimental intervention is applied by all practitioners, thus covering the full spectrum of clinical settings
4	The best alternative treatments are used for comparison with no restrictions on their application
5	The comparative treatment is applied by all practitioners, covering the full spectrum of clinical settings
6	No formal follow-up
7	The primary outcome is a clinical meaningful one that does not require extensive training to assess
8	There are no plans to improve or alter compliance for the experimental or the comparative treatment
9	No special strategy to motivate practitioner’s adherence to the trial’s protocol
10	The analysis includes all participants in an intention-to-treat fashion

Disadvantages of RWE

In RWE studies, biases can be introduced by unrecognized confounders. For example, patient and physician factors that determine the selection of a treatment may also affect health outcomes. Confounding by indication of treatment (selection bias) means that a physicians has a tendency to prescribe more effective treatments to patients who are most likely to benefit by them. Also, patients who adhere to a prescribed treatment may be more likely to engage in other healthy behaviors (adherence bias), which can create the false conclusion that the intervention is the sole cause of an improved outcome.²¹ In some datasets, there might be an imbalance of risk factors or protective factors with patients receiving one intervention compared to another intervention because the two groups were not randomized to have equal incidences of this factor, and the lower risk patients might appear to have a better outcome because of the intervention (confounding bias), but the difference in outcomes was actually due to the confounding factors. Some potential confounding factors, (such as obesity, sedentary lifestyle, alcohol use, or drug use) as well as some outcomes (such as mortality or change in symptoms) might not be consistently captured in electronic health records or medical claims. Adherence can be high in a clinical trial that measures efficacy (which is performance under ideal conditions); whereas in the real world adherence may be low and outcomes may be less favorable. Reporting bias can occur if some outcomes or datasets in the health record are selectively revealed or withheld. Whereas a RWE trial can inform on the overall performance of a treatment, it will be difficult to identify the specific components or biases that account for its effectiveness. Therefore, a RWE study is said to suggest a hypothesis that can be tested with a RCT.

RWD datasets reviewed by researchers after they have been created by clinicians are vulnerable to systematic omissions or misclassification. For electronic health records, secular trends in coding criteria and reimbursement policies for hospital claims can impact data registration.²² Data mining occurs when analysts reexamine existing datasets to generate new information. There is a concern that an organization can repeatedly analyze RWD with various models until a desired outcome is identified. Therefore, RWD can be manipulated via repeat analyses with nondisclosure of unhelpful results.²³ Data sharing between different health care organizations might be restricted because of privacy laws, which can limit the generalizability of a dataset to the patients from a single health care organization. At this time there are no generally agreed upon standards for the design, conduct, analysis, and reporting of RWE trials, which can undermine confidence in such studies and create a suspicion that the RWD has been “cherry picked” to prove a predetermined conclusion.²⁴

According to a Cochrane Systematic Review in 2014 about results of health care outcomes assessed with observational study designs compared with those assessed in randomized trials, little evidence was found for significant effect differences between observational studies and RCTs.²⁵ In the literature however, there are examples when effects identified in observational studies were not be reproduced in RCTs.^26-29 Given advantages and disadvantages of both RCTs and RWE studies, it is likely that the two types of trials in combination can provide evidence that neither type alone can readily provide.

The FDA RWE Program

In December 2018, FDA announced that Pursuant to the 21st Century Cures Act (Cures Act) signed into law on December 13, 2016, the agency is launching a Real-World Evidence Program, to apply across its drug and biologic review process.³⁰ This framework will allow for evaluation of drug or biologic products (but not devices) using RWE to help support the approval of a new indication, an expansion of the indication to a new population,³¹ or a post approval study.³² This program will cover both prospective noninterventional clinical trials where the population receiving the intervention is identified at the start of the study, such as in pragmatic trials, and retrospective observational studies where the population receiving the intervention is studied from historical data. Although FDA uses RWE for assessing postmarket product safety, the application of RWE to support effectiveness determinations has previously been limited to a few instances.³² A risk of this new policy will be that a RWE trial with unrecognized biases could lead to approval of an ineffective or unsafe new indication whereas previously, new indications each required an RCT.³³

How to Improve the Value of RWE

Six methods for improving the value of RWE have been discussed by clinical trial experts. First, stakeholders should develop methodology standards for RWE trials to improve the quality of these studies and increase the confidence of study users. The FDA plans to encourage this type of activity. A consensus panel of database experts recently recommended disclosure of a set of inputs to define a population from a database in any RWE study to improve the study’s transparency and reproducibility.³⁴ The “active comparator, new user” study design mitigates biases in RWE observational studies that compare interventions.³⁵ Second, since about one fourth of FDA-mandated postmarket studies are currently not being registered,^36,37 a mandatory national registry for RWE trials should be created. Currently an investigator can easily data mine by changing assumptions and repeating the RWD analysis until a desired outcome is generated. Such a practice is deceptive and would be inhibited by a registry that could be established to resemble www.clinicalatrials.gov. Third, since results of about one fourth of FDA-mandated studies are currently not being reported,^36,37 formal restrictions should be established that data from an unregistered RWE study could not be being published or submitted for regulatory purposes.³⁸ Fourth, for a fatal disease like cancer, a synthetic control arm accurately predicting mortality outcomes in patients not receiving an investigational intervention in a RWE prospective study should be built so that every patient in a pragmatic trial can receive active treatment, and no one would have to be randomized to the control treatment.³⁹ Such a high-quality EHR-derived mortality dataset for retrospective and prospective RWE generation was recently reported by Flatiron Health.⁴⁰ Fifth, RWE researchers should collaborate with (a) The Patient-Centered Outcomes Research Institute (PCORI) and its partner networks to link claims data with EHR data⁴¹; and (b) PCORI’s spin-off initiative, the National Patient-Centered Clinical Research Network (PCORNET), which is a network of research networks with a common data model and query system that is standardizing EHR data collected by more than 100 health institutions.⁴² Sixth, researchers should collect data from patients with the secure FDA mobile app that was released in November 2018.⁴³

Conclusion

RWE will become an increasingly important concept for determining the performance of drugs and biologics because of the exponential increase in electronic data available to researchers and the improved analytics tools that are also becoming available. RWE trials are much less costly than RCTs. For this reason, the pharmaceutical industry might want to replace many types of postmarketing RCT safety studies with RWE studies. Postmarket multiyear cardiovascular RCT outcomes trials for diabetes drugs cost in the range of $250 million dollars,⁴⁴ and RWE studies cost far less than that. RWE can measure outcomes of interventions for all patients, which is the most important measure of an intervention to patients, health care professionals, and payers.³⁹ Although, compared to RCTs, RWE has many advantages, but this approach to data collection also has potential problems with data quality, consistent methodology, and a risk of not accounting for confounding factors. Both RCTs and RWE trials have their specific weaknesses and strengths, and in combination the data from both types of trials can supplement each other.

RCTs will remain necessary for the foreseeable future for initial clearance of investigational treatments, which must initially be demonstrated to be effective in a core target group and for a core indication in an RCT before they can be introduced to clinical practice (where there is less oversight than in a RCT). However, rare complications of therapies might not be detected in the general population from an RCT with a small number of subjects, tight inclusion criteria, and a short duration of 6-12 months. RWE postmarketing studies will become increasingly used to (1) assess safety and effectiveness in real-world populations for regulatory purposes;⁴⁵ (2) compare clinical outcomes in real-world observational trials of clinical interventions (which might not necessarily require regulatory approval) to determine optimal treatment strategies;^46,47 (3) identify prescribing patterns for drugs with similar indications for population health analyses;⁴⁸ (4) measure resource utilization by patients in the real world;¹ and (5) compare economic outcomes in real-world observational trials to determine the most economically attractive treatment strategies.^49,50

In conclusion, it appears that RWE will increasingly supplement post approval RCTs, but not replace them. It is also highly unlikely that RWE trials could replace phase 1-3 trials for most therapies. Based on new RWE policies recently announced by FDA, it is expected that RWE studies will allow drugs and biologics to be approved more efficiently for new indications and to have their indications expanded if they demonstrate good postmarket performance. As the flaws in the structure of RWE trials become corrected with standards, rules, more reliable databases, and better methods for eliminating confounding factors, then the importance of this research method will continue to grow in the regulatory, clinical, and economic arenas.